Strange Address Book Contact Import Problem

I currently have 46 contacts in my address book. I have a VCF file that I want to import which contains around 600 contacts. When I double click on the VCF file the contacts import properly and are all in the address book. The problem is, when I quit then restart the address book only my original 46 contacts are there. I even tried deleting the address book folder in Library > Application Support so that I could start with a new address and import my 600 contacts. Even after doing that I restarted address book and those same 46 contacts magically reappear.

Any ideas on why these 46 contacts are always there and are preventing me from keeping my import from staying?

Thanks very much!

Some info:
Address Book version 6.0 (1043)
Macbook running OSX 10.7