I put the movie into an ISO image with DVD shrink on my PC. I then transfered the image to my wife's macbook, so she could watch the movie without having to bring the disc with her. On my laptop I have Nero 7.7 and it has an option to enable Nero Image Drive. And what it does is it creates a virtual drive where you can mount either NRG or ISO images and run it to view it. It's a nice feature that way I can have a few movies on an external drive or on the on the laptop itself without having to carry the DVD's around.
The rest are right on.
OSX mounts ISO files as discs automatically, ot Image Drive-esque software needed. As a matter of fact, most OSX software downloads are supplied as ISO files (compress into a DMG file).
I have never had a ISO file made on my PC not work in OSX, but I don't doubt you are having a problem. I can think of three things.
1. The file is corrupt, but if you can get it to mount with Image Drive, than that is probably not it. Leaving,...
2. Somehow DVD Shrink is writing the file like OSX wants to see
3. I must also admit, I've never tried a ISO file of a DVD-Video disk format (I have only used data files). It may be simply that they do not work. I know DVD Video files are ignored by OSX built-in CD/DVD burning software, you can't bruna video disc with a simple drag and drop like you can data (though it is possible through Terminal). Maybe someone here can confirm that a ISO image of the UDF DVD Video disc (video_ts based) will open?
Might I suggest just using Mac the Ripper (free!) and rip the original DVD straight to the Mac?