That's the same spec has my old MacBook and my wife's current MacBook. Do you really have 4GB of RAM installed? The CoreDuo could only take 2GB - if you've installed bigger RAM, that might be the problem right there.
Streaming Flash video has always got the fans going on almost any Mac. Even my 2010 MacBook Pro i7 gets warm when streaming flash - but it's far worse if you don't have a GPU helping out. Your 2006 MB only has a GMA950 which cannot help decoding Flash.
The first thing you can do, at least on YouTube, is join the HTML5 trial. This will mean that most of the videos you watch will play back using an HTML5 container, rather than a Flash container, significantly reducing the stress on you CPU.
YouTube - Broadcast Yourself.
Secondly, you should resist playing back anything higher than 480p. So if you're watching Hulu, YouTube or other videos, don't select 'HD' (720p or higher) as your machine will struggle to play this back if it is Flash. iTunes downloads should still be fine in 720p though.
Going full screen will choke your GMA950 as well, so watch the video in a window when possible.
One final thing - your fans could be clogged with dust and if your CPU is getting too hot, it will throttle back (i.e. go slower) until it cools down again, resulting in a choppy playback experience. If you're brave enough, open it up and carefully clean it out, or get an expert to do it for you.
Don't forget, your Mac is now five years old. In 2006 when you bought it, a machine that old would have been a 500mhz G3.
