Out of application Memory

Running OS X 10.9.1 Mavericks. Recently I have been receiving an error window stating that "Your System has run out of application memory. To avoid problems with your computer, quit any applications you are not running." Strange thing is, I am only running finder and my browser. All other applications are off. This happens when I use Firefox or Safari. If I persist in trying to run the browser to get help, the Force Quit Applications window pop up with the above message. I am virtually locked out; can't run anything. Also, can't remember how to force OS X to boot into save mode (what key do you press when booting up?) I really need help on this one. Otherwise, my iMac is an expensive,good looking door stop. HELP!

To start up in Safe Mode Be sure your Mac is shut down. Press the power button.
Immediately after you hear the startup tone, hold the Shift key.
The Shift key should be held as soon as possible after the startup tone, but not before the tone.
Release the Shift key when you see the grey Apple logo and the progress indicator.

As to the RAM issue, how much RAM have you installed in your iMac ?? What OS are you running as well. If you only have 2-4GB Safari has a big problem of Memory Leak and not releasing it back to the CPU, in 10.5/6/7 got better in 10.8 and has hit the sweet spot in 10.9 and the Nap feature.
You would be wanting to open up Activity Monitor and have (All Processes) or CPU as the live window and that you have Real Memory in the top bar. Now click on Real Memory once or twice to bring the highest number to the top, and then you can see what is taking up all your Memory.

If you are running Mavericks 10.9 make sure your up to date on 10.9.2 and look in the Mac App Store for a App called Memory Diag ~ by Rock Sand Studio Install that and see if it helps you out any, if you aren't on Mavericks forget this paragraph

Report back.

There's one other app from the App Store I can recommend: Memory Clean by FIPLAB
It was either free or very near free. Simple, with just one function - to free up unused memory: