My observations are pretty much equal regarding the 'stock' terminal,
iTerm2 and gnome-terminal which I installed via macports:

I am on a mac mini 2011 with 8 Gb of RAM. My previous experience is with
ubuntu 10.04 on a Gateway GT5422E. I use terminals frequently. I find that on
ubuntu, scrolling in a terminal or a terminal application is just as responsive as
in a Gui application.

On the mac mini (with Lion) I am finding the scrolling - uparrow,downarrow, rightarrow, leftarrow - is significantly slower than in Apple Gui applications.

Is there a way to correct this? Are there by-application memory settings.

I am a total noob when it comes to mac, although I am coversant with
the *nix command line.