When that box opens you need to drag the Firefox icon that's in that box into your Applications folder. That's the actual program.

What you're opening first is the disk image which is how the download is packaged. Once you open it and drag the program out then you can delete the disk image by dragging it to the trash.