Which exact Mac model are you using? If it is a Mac which has one of those combined headphone and Mic jack a standard 1/8" stereo connector probably wont work. A four pole connector
such as this one might work. I actually prefer
this to avoid having to find a four pole connector. Their not hard to find but I never seem to have one when I need it.
I'd probably try something other than GarageBand for this as well. It's a good program but for general recording in this kind of situation it can be a bit tricky to get the settings set up correctly. I haven't used
this program personally but it seems simple to set up and use. The demo is free to try.
I was in a similar situation as you are some time ago and I had to essentially trick things into working. I think I did the following
1. Make sure the turntable is connected properly and you are getting good wound without the annoying hum that is sometimes present.
2. On some stereos you need to activate the tape output /monitoring function on the stereo to hear sound from the turntable. If that is the case run the audio cable from that tape output loop to the Mac.
3. If you go into System preferences and look at the audio inout you should see an indication that sound is being picked up.