Font Book problems

I have way too many fonts loaded into FontBook and i want to get it back to its original state, but I am not sure how to do that. I know I can delete them one by one but they are still in ~/library/fonts. Basically what I am asking is...Can I delete all the fonts in ~/library/fonts without getting rid of fonts that are needed for my system? I bought Extensis so I dont need fontbook, but I dont want to leave it full of fonts-especially because I dont want the fonts that are in there any more. Please Help!

good choice on the Extensis - I chugged my computer long ago with too many fonts in Fontbook. Without my mac in front of me I'd say in Fontbook it had sections/collections for OS, User, and All. I remember I deleted every font set but the one with the computer icon and then scrapped Fontbook entirely.