If you've lived this long with Office X, there's probably no pressing need for you to upgrade to Office 2004. Besides, Office 2008 will be out later this year.
Office 2004 has a few nice features...among them, a helpful compatibility check when saving, better Microsoft Exchange compatibility for Entourage, and it fixed the 31-character limit bug. Nothing earth-shattering.
Office 2008 will introduce a new file format, the same as the one used in Office 2007 for Windows. It will also be Universal, for better performance on Intel Macs.