It should work, I'm able to connect to WEP networks just fine. But I'm not exactly sure what your problem is with it. It could very well be the router you're using. Which router is it?
But to be honest, WEP is about as good as having no security turned on at all. If you have any control over your network, you want to use WPA. It's more secure.
Yes, you want the second option and no, you don't need special formatting for the password, you just enter the password you create when you turn on WPA (it's no longer a HEX password).