With Activity Monitor, like you have, go to System Memory and take note of Used, Wired, Inactive and Free memory numbers. The Page Outs indicate the times when there was no memory available for an application and some part of memory was written to disk to make space for it. This is an expensive process and if you see a lot of Page Outs that indicates that you are running memory hungry applications that are affecting your system performance.
Increasing memory will greatly help in these particular situations. RAM is fairly cheap so you should determine the maximum your iMac supports and go to that even if you don't think you'll ever use it..if money is tight, then look at upgrading to 8GB at least..