X Server 10.4

Ok, I am setting up an X Server running 10.4 and I would like to
know if there is such thing as authentication for Mac computers
and if there is something like Group Policy in Windows to restrict
users using the X Server to deploy such policy.

Yes there is an LDAP Open Directory like Active Directory for creating user accounts and managing authentication.

As far as group policy, there are some things or settings that can be deployed and/or configured for users utilizing the Open Directory but it's not as granular as GP. Keep in mind that GP is great for turning on/off features globally, whereas alot of those settings are just not there to modify in the OS X GUI. Get used to shell scripting and defaults write commands if you're interested in locking down OS X machines with "policy".

Ok I have been reading about LDAP Open Directory and it seems to work
just like Active Directory. However, I have a big question. I have a Windows
2003 Server acting as DHCP, DNS, and Domain Controller. I am able to start
the Open Directory on the X Server but the Kerberos service is not running.
I read that to turn Kerberos manually I need to configure the X Server's DNS.
Will this affect the zones that I have in the Windows 2003 Server or can it
be just a DNS made for the macs??

Perhaps if you utilize DHCP reservations on both OS X and W2K3 servers there won't be issues with DNS.

Just a thought.

You might also have success if you bind the domains.