If you uncheck "Show video within the main window" you should be able to have more than one video playing.
More than one 'instance'? I doubt it.
Just tried it. Nope. Opening a video while another is playing closes the first one.
Odd. With Quicktime and Quicktime Player 7 you can open and play more than one at a time.
Edit: There is a work-around: https://wiki.videolan.org/How_to_play_multiple_instances_of_VLC/