In OS X, many apps do not need to be installed in order to run, which you have now found out.
You could think of the .dmg file sort of like a zip file in Windows.
When you open the .dmg it shows you what's inside.
Unlike windows, many apps can be opened and run right from there without an installation routine.
In order to install the app, open a new Finder window, and drag the app's icon and drop it in the Applications folder.
You can then close all the windows and eject (or drag to the trash) the opened dmg file.
You will now open the app from the Applications folder.
If you want to keep that app on the dock all the time - while you have it open, you can right click and select Keep in Dock.
To remove an app, just drag it from the Applications folder to the trash.
And you might want to read through eric's
Mac-Forums Switchers FAQ and Information located right at the top of the Switchers forum for this and other mighty fine pointers on using your new OS.