No, not for most of them. Only those that can have just one window open will do that, such as Dictionary. It's quite intentional, and I find it useful. If I close the only open Word file, then want to open another, it opens quickly, because Word is still open.
You have to get used to quitting apps instead of closing windows. Use Command-Q or the menu. Or you can go to the Dock icon and Quit. There are lots of options, but clicking the red button is not one of them.