Window focus changes after OS X comes out of screen saver

So pretty much what it says in the title. I've been searching for someone with this problem for a while and finally decided to ask. On my Macbook Pro when I move the mouse while the screen saver is on OS X changes to the previously active window. For example: I have Chrome and Adium running. Say Chrome is active when the screen saver comes on, when I move the mouse the active window changes to Adium instead of Chrome. This becomes annoying when the 2nd to last window was on another space, I move the mouse and my desktop changes to the other space to get to the window. I know it's not my touchpad, I have touch click disabled, I can only use the button to click. It does the same thing if I use a USB mouse or turn off the screen saver with the keyboard.