I usually have Chrome running and Adium. Could either of those be waking the computer for network access?
I would just un-check "wake for network access" and see if that helps.
Nope. I'm running Lion, but had this problem back on SL as well. Any other ideas?