Wasn't CS3 optimized for UB or something similar? That would be the biggest reason to upgrade. Other than that, there were not a lot of difference between the two. CS3 has an HDR tool built in and there's several other upgrades.
CS4 uses the computer's GPU for some functions, alleviating the strain on the CPU and memory which also allows for real time viewing of certain effects like upsizing, rotation, etc... It also allows for sizings other than 25%, 50%, 75%, and the like to be viewed without the pixelation that occurs in 2 & 3. That was one of my major annoyances. Sometimes I just didn't want to picture blown up on my screen that large to edit a certain section and now working with 21mp photography files, that's even more important.