you wont find any free ones, and especially on the Mac.
If you are bored of GTA, you could try True Crime: Streets of LA. Just Cause.
Fallout 3, Oblivion, Far Cry are good open world games.
of course, these are all PC games, with no Mac alternative. Sorry
If you are technically savvy you can try porting games with the WINE/Cider engines, but this is never guaranteed to work, and really tests your patience too. GTA San andreas has been successfully ported.