Internet failing

I've had my imac since last July, and before that my the rest of my family used windows. Now since i got the mac, the internet connection of my house has been going down randomly, esp. when i download or load big sites. i just restart my routers to fix this, but i don't want this is getting annoying.
Is there any suggestions of what the problem is here?

What OS are you using? Do use a reliable ISP?

What router are you using? I've had issues with Linksys G routers and Macs in general. I'd also check if there's a firmware update for the router on the manufacturer's website.