There is memory and there is memory. Generally, *nix systems (and this includes Mac OS X) consider that unused memory is wasted memory. In general, they will fill up available memory over time on purpose, caching stuff there to improve performance. This is not a bad thing. Memory usage will creep up over time, and this is design intent.
On the Mac, there are four types of memory buckets reported on:
Wired memory
This is memory that applications or the system needs immediate access to, so it can't be cached to disk. It will vary depending on what applications you're using.
Active memory
This is memory that is actively being used.
Inactive memory
This memory is no longer being used and has been cached to disk. It'll remain in RAM until another application needs the space.
Free memory
This memory is not being used.
It is probably the Inactive memory that you are seeing going up? This is what happens on my system over time. Inactive memory goes up and up, and this seems to have no performance implications at all - just standard *nix memory management.
If it is wired memory, that is a different matter.