If you need to connect from your OSX machine to the Windows machine,
CyberDuck is a good one and it's free.
Transmit is a commercial product but is quite popular. Interestingly enough, you can use the default FTP support in OSX to read from FTP servers but not write. This is something that has been missing from OSX for a long time for reasons that have never quite been explained.
As far as AFP is concerned, you are better off using SMB between your Windows/OSX machines as there are no compatibility issues and is quite easy to setup and configure.
Windows - Ftp/afp
