Help! Recording issues!

Hi, I record a podcast using arguably the most convoluted set up you could think of and I'm having a problem that I can't seem to fix, so I'm here pleading for help. This is my set up. I have two mics hooked up through USB, one I use, the other is rigged up to a speaker (I'll get to why in a minute). I record with my co-hosts on skype so I use soundflower to route their feed through garageband. I also use a software soundboard (BZsoundboard) which is routed through a different channel in soundflower. This is where we get to the mics. I have thus far been unable to find a way to route the soundboard through skype so I've hooked up the second mic to my speakers which does work so my skype people can hear the sound effects.
This setup, while convoluted, does work but there's one problem. Every few minutes of recording either the soundboard or the skype feed glitches out and it starts sounding like the feed is recording twice but one is 1/2 second behind the other. Do you know of anyway I can fix this? My theory is the problem is coming from soundflower but I don't know.
Any help is appreciated, thx!