8.3 means no more than 8 characters in the first part of the name (before the ".") and the extension should not exceed 3 characters. It is a hold over from DOS, which could not exceed those limits.
File name is just that, the name of the file. File format is how the data in the file is structured. The extension is an indicator of what the format of a file is, but not a gaurantee. A jar labeled "Sugar" tends to suggest that there is sugar inside, but if you wanted to you could put flour in it, or sand for that matter. It just makes the most sense to put sugar in a jar labeled sugar, flour in a jar labeled flour, etc.
Generally for photographs jpeg format is a good compromise between size and quality. On the high end for digital photographs is RAW. Some of the professionals on the board can probably break down the various formats better than I.
EDIT: So when you ask what extension to use, jpg, jpeg, or other it is a little off. I would use "jpg" for jpeg files, "bmp" for bitmaps, "gif" for GIFS, "doc" for Word documents, etc... As to which file format to use, the paragraph above is my answer to that.