MSN Messenger for Mac does not support the camera feature as far as I know (unless there is a plug in you can install), do you have Skype or something similar?
Also, when you say closed are you quitting the program so that it no longer has a small arrow beneath it on the dock or are you just crossing the program window off? If you are just crossing the window off then the program is not closed and so potentially could be using the camera still.
Try making sure that everything in your dock that can use the camera is closed fully, then start Photo Booth and see if you get the error still.
Hope this helps,
Chris.