If you do a search, this question crops up a lot on the forums. There are various FAQs and HOWTOs for beginning users.

The stuff that's left on the desktop can be safely deleted. The icons that look like drives with the Firefox logo on will simply disappear when dragged to the trash because it is a 'mounted' disk image. The disk image itself (the thing you downloaded) can be put in the trash as normal.