Tough to say, since mkv is a container and it could hold several different media types within it.
Download
Perian, it will allow you to play mkv files in Quicktime.
You can also try
MPlayer OS X
Perian takes time to buffer the MKV before it can be played back smoothly. Also, even with the "Play soft sounds louder" setting on the prefpane, sound is still soft, requiring the user to manually adjust it using QT Player's Movie Properties box.
As for MPlayer OS X, which one? There are so many variants of program with that name, and scattered over different sites, each appearing to be a different program, with different program icon and interface. It ranges between OK and just plain sad. The original MPlayer OS X appears to be the worst of them. There is a version with the word "Extended" tagged on to the title. But the latest update is from Jan 2011. Anyway of all the players, I find MPlayer OS X's icon(s) to be the most amateurishly drawn. And the worst thing about this program would have to be the volume control keys. You need 2 hands just to achieve that. That is just wrong.
I like to use Miro. It appears clearer and is much smoother than what Quicktime seems to be able to produce.
No skip forward/backward keys, only available as an option on menu bar. This is on top of tons of missing features, such as aspect ratio adjustment, cropping of black bars, etc. It seems the developers have mistakenly put their focus on making the program more iTunes like rather than "QT Player enriched",
I like to use Miro. It appears clearer and is much smoother than what Quicktime seems to be able to produce.
I'm not sure if Miro actually plays back 720p MKVs smoother than QT Player. But a few minutes ago I tried to check Miro's playback quality, and noticed that it did lagged a bit in the first few seconds of playback. Also, see my notes above.
This seems to be the darling of OS X users, because people swear by it. And it really doesn't seem like that terrible of a program. But if you're an Asian (read: Chinese) user who absolutely read the text below to understand the movie, you can do better with other programs out there. The reason I say this is because the Chinese font looks absolutely atrocious with the font rendering engine, reminiscent of hinted Linux font-smoothing. Some people might enjoy that, granted, but it doesn't look like OS X. And worse, the default font is Arial. But if you pick a different font, the Chinese words might turn out as boxes. And here comes my pet peeve. You still need to two-hand it when you want to adjust the volume.
[OK, I know technically, both Miro and VLC can use the scroll wheel to adjust volume. But sometimes you just want one simple key to do things.]
The best thing to do is to convert your files into a standard format like MP4, if you want the real answer.
Sure, if you have the time and inclination, knock yourself out. Whatever the case, that isn't the real answer to this thread.
One program I discovered very recently is XMBC. It is one **** of a bloated program, with a retard interface that challenges the mind-boggling mental retardation that is Windows Media Center (I note with interest that Windows 8 is going full steam in that direction). But all the bad things about its interface is more than made up for in playback quality. I think on OS X, you just cannot find MKV BDrip playback as flawless as XMBC. I'll stake what little reputation I have on that claim.
That being said, I'm using MPlayerX right now, which proves to be smooth sailing so far. It does everything you want with aplomb. And it has a single key press for speed and volume. It also lets you resize subtitles with key combo, which I find handy.
Prior to this I was using Movist, which I like a whole lot better. And the reason is simple: it has the most logical use of the cursor keys. Up and down for volume. Left and right for seek intervals. And get this, (developers, listen up) it even has user configurable seek intervals! This is crucial! We don't always want to jump 10 seconds in the video. We sometimes miss what the mumbling actor is saying, so we want to skip 3 seconds backwards. Only 3 seconds, not 10! If you don't make your player manually adjustable this way, then your player is sub-standard, I'm sorry.
It's sad that not more media player developers know or used Media Player Classic (note the word "Classic", so this is not the standard Windows component) on Windows. It has the best media playback features bar none on any platform. The only thing it fails in is basic editing functions like QT Player (by that I mean QT Player 7 and not the useless one that ships with your OS X). Media Player Classic offers such things as DXVA that facilitates hiccup-free 1080p h.264 playback with old Intel GMA, and makes full use of the keyboard. Got a bad aspect ratio DVD rip? No problem. Just press num 2 or 8. There is no player on OS X that mimics that.
[Yes, I'm aware that this is a 2009 thread. Blame flocottee for resurrecting this thread, and the others that followed suit. Thanks to them this page showed up on Google search. Since people are going to be here ad infinitum via "the devil's portal" anyway, I thought I'd write a detailed assessment on the subject.]
