I know people who have worked for Apple. They've all said the same thing: refurbs are practically new.
The ONLY difference is that something was defective, which Apple then replaced, and then re-tested the entire computer. This means that in addition to the rigorous testing that your new computer is already put through, it has to pass it again.
Not only that, but you're given the exact same warranty, boxes, manuals, cables, and software you get from any other new Mac.
Unless you want the iPod nano, I'd pick up the refurb.
I realize that yes, refurbs do have a certain stigma when looked at as "used", but Apple's warranty policy is pretty cut an dry: they only take it back if it's defective. So if the computer is significantly damaged by the previous owner, chances are Apple won't take it back.