I have the same computer. I doubled the RAM to 8, but it made absolutely no (noticeable) difference. I'm not a tab freak, so that's possibly why.
It was not slow at 4.
Whether someone notices a difference between 4 GB and 8 GB or more depends heavily on the tasks they are doing, software used, and how they use their machines. Having many programs open at once or running memory intensive tasks can definitely affect speed.
@OP
Here's one idea that may give some insight into whether additional memory will be helpful:
1. Reboot the Mac so you are starting with a clean slate so to speak.
2. Launch Activity Monitor (in Applications/Utilities) and click on the "Memory" tab.
3. Leave Activity Monitor open but go about business as usual on your Mac (launch programs / perform tasks that usually result in beachballs)
4. When the beachballs hit the point of being noticeable / annoying click back to Activity Monitor and look at the numbers at the bottom of the memory pane (especially the swap numbers). Let us know the results.