Magic Mouse Slow Response when streaming video

When I stream video (with any application) to my PS3 or 360 the magic mouse seems "heavy".
It is constantly slow to response and I find it hard to move across the screen.
The CPU is not in high use at all (usually between 2%-10%.. sometimes none at all).

Any ideas? This is super frustrating. I basically can't use my mac if video is streaming.

I'm running OS 10.6.7
3.06 GHz Intel Core 2 Duo
4 GB 1057 MHz DDR3 memory

This should not be happening...

On Lion... this is still an issue.
Is anyone else have this problem at all?

Also wanted to add to this... If I use the regular wired mouse it works just fine. Only an issue whens streaming video and using magic mouse.