Gotcha, a few things to keep in mind with the way Apple does things..
The version of iOS that comes with your device (regardless of what it is) is the BEST version of iOS for that device. You can usually go to one more version after that without seeing too many issues, but jumping to the version beyond that (so iOS 5 to 6 and then 7 for example) usually causes issues.
You might think, they are just a simple number change between 5 and 6 and so on, but remember that these releases are usually a year apart and meant for HW that is coming out on a yearly cycle. If you look at the chart below you'll see the iPhone 5s' CPU performance is 40x faster than the iPhone 5 which is better than the 4S and 4 and so on.
Since the newer version of iOS is targeting the newer hardware, trying to run it on older and slower hardware is going to cause issues and slowness. Just because iOS supports the older versions of the hardware doesn't usually mean its a good idea to upgrade to it..
If you do upgrade to IPhone 5S or 6, then the newer versions of iOS will run remarkably well.
I had an iPhone 4 just like you and had it on iOS 7 which is VERY slow compared to iOS 4/5 that I ran for a long time. When I upgraded to iPhone 5S, the overall speed of the device compared to the iPhone 4 was just amazing. Recently I got a chance to see a friends iPhone 6 and the speed increase over my iPhone 5S is equally amazing..