Open your downloads folder and drag the item to the Trash.
Downloads folder will either be in the dock or you can locate it through Finder as well (or both ways).
What little window are you talking about? You mean the progress bar that pops up when you download a file?