I recently upgraded the RAM in my Mac Pro, and was surprised to see that Safari actually seems to take longer to load now that I have more RAM. I am baffled as to why that is.
So, setting XBench questions aside for a moment, more RAM does appear to slow a few things down. I suspect that the offsetting increases in the speeds of other things more than make up for it of course, and of course, with more RAM comes the ability to do more things without running out and having to go to virtual memory (disk based, hence slower).