It could also be because your cache folder is full to the gills. Just empty the cache by going to the menubar ---> Safari ---> Empty Cache when you start seeing the spinning coloured pizza icon too many times..
To speed Safari up, I have also set my favicon folder as Read Only so the folder doesn't contain bijillions of favicons before I remember to clean it up.
Simple procedure: go to Macintosh HD ---> Users ---> your account ---> Library ---> Safari
Spot the Icons folder, click on it once and do Apple + I (or Command + I or Get Info in Finder's menubar and then File, Get Info in the drop-down menu).
In the Info window that appears, you will see the Ownership & Permissions section. First item is: You can with a drop-down menu. Select Read only from it.
This will prevent the folder from storing megabytes of favicons. You will need to manually empty the Icons folder after this procedure but, good news, it will be the last time you ever need to do this! :girl: