The "which distro should I use" topic has been discussed to death on various Linux forums. I don't think we should get into that in this forum.
First thing to do is to make sure whatever hardware you are going to use for networking (or anything else) is supportet in Linux. Then you need to decide what to use for filesharing. Personally I use SFTP. It's probably the easiest service to set up and secure on both OSs. Downsides are that it's not designed exactly for this purpose, there is no support in finder, and encryption kills speed.
SMB could be viable solution, though it's kinda weird to use Microsoft's protocol to network Apple and Linux.
I've never used NFS, and I'm not sure if there is GUI tools for mounting NFS shares on OS X.
I've seen an option to enable AppleTalk in Linux kernel config. I have no idea how easy/hard it is to set up. I'd quess most distros don't have it in their stock kernels.
I've never compiled kernel on RedHat or Mandrake. Some people tell not to use vanilla sources from kernel.org, but instead whatever the distro is providing.