You are not limited to using an Xserve if it proves not to meet your needs. For instance, I believe the user
Nethfel runs OS X Server on a Mac Mini but I could be wrong.
This article talks about one individual who ran it on a Mini as well. That said, you need to decide what machine fits your needs based on specs. A good place to start would be to browse what is available through the Apple website.
You may also want to take a look at
these videos available on the iTMS (free) if you're interested. The one's I've seen aren't too technical but will give you a good sense of how OS X Server works and what to expect. In fact, these videos show the individuals using OS X server on a MBP.
Also, depending on your needs, you may not need OS X server and might be fine with OS X. What are you planning to have this machine do?