Rank these three things in order of importance: File Size, Time Spent Encoding, and Quality.
If file size and quality are most important rip to H.264. Handbrake is my method of choice. This can take 8-12 hours to encode a movie though. With a 2 hour movie you can get decent quality at 700MB, high quality at about 1.1GB, and DVD quality at around 2.2GB. (The last is a guess but I will rip one tonight to find out.)
If file size and speed of encoding are most important. MPEG2 is probably the way I would go. Possibly a compressed DVD format using popcorn or DVD2OneX. These are never my priorities though.
If speed and quality are tantamount, rip the DVD to your hard drive and leave it. This I will do if I want a movie for a trip and it is not one of the ones I keep encoded with H.264.
Generally I would recommend handbrake and Mac The Ripper for the above operations. I have never used osex but I notice it has not seen activity for almost 3 years, which is not a good sign.