hmm... Well, I don't pretend to know a lot about macs since I'm pretty new to them. I have found that the microphone input on built-in sound cards is not meant for recording of any quality. A decent audio interface is one step... You can find products from M-audio, mackie or any online music gear store. 100 bucks or more will get you something nice, but I haven't done much research for lesser quality than that.
I'm sure you also understand that if you are recording from a cassette, the quality will probably not be up to the standards that your ears like.
Another variable is the tape deck quality. Are you recording from a walkman or a boombox? These don't have extremely high quality outputs.
I don't think Garageband is the issue. It may not have all the bells and whistles, but it records a nice clean track for transferring music.
Hope this helps.