Your router has two passwords actually. One is the one used to administer it and the other is for the WiFi. While you can share the WiFi password with friends/family and they can get an IP address, you don't want to do that with everyone.
A lot of routers these days have a Guest WiFi that they can set up. You can hand out the password of that to your friends and they can get into the Internet but can't access your machine..
So that's another layer of security..
As the others have said, just the IP address doesn't give anyone much control. The usual method of hacking actually involves finding a vulnerability in an application/service that is running.
So a website for example might have a port that is opened or be running a program/script that is vulnerable to certain type of information.
Most desktop computer seldom have any of these services running and Mac OS by default keeps most of all the ports closed so noone can actually get in that way..
Frankly, these days the thing that most "hackers" do to get access to system is social engineering or phishing. Social engineering is usually targeted at big companies and it involves the hackers calling and pretending to be someone of importance to get bits of information and then taking that and calling someone else to get more information and doing that over and over again until they get enough information to bypass all security checks. The phishing method is a more common way of accessing your information and at it's core its as simple as sending you an email that looks totally authentic about your account (doesn't matter what account, bank, email, ISP, whatever) being locked out or going through a security scan and you are required to enter your credentials to unlock it.
Of course, while the email looks authentic with all the right images and text, the URL you are directed to validate your credentials is bogus and once you complete the validation you're passed along to the real site with a failure. Without thinking about it, you enter your password a second time on the real site and boom you log in, everything looks good and you move on. Unfortunately, without you realizing you just gave your credentials away. If this was the bank, bye bye money..
[RANT]
If it wasn't already abundantly clear, a lax use of computer phrases just tips me over the edge and when I see the shows and movies show that some person can type on a keyboard for about 5 minutes boom they've gotten past the firewall of some secure network and are grabbing all the files, I just roll my eyes and smile cuz that's just BS!
My absolute favorite butchery of my chosen profession is the movie Independence Day where Jeff Goldbloom's character (the genius MIT grad, he'd have to be MIT-educated to pull this crap off) designs a virus on his Mac (no less) that he manages to inject (what the interface was, I'm not sure..do all aliens conform to our IEEE 802.11abgn standards??) the alien computer and not only corrupt their system, but also manages to show a very nice and menacing skull and crossbones on the screen with audio to boot!
In the real world we all live in, we have to jump through umpteen hoops to get a Windows program to run in Linux or Mac OS and totally forget about running Mac OS apps on Windows or Linux..but no worries, Goldbloom's Mac can talk to an alien computer with whatever architecture it has.
OK..end rant..
[/RANT]