Which version of SharePoint are you using?
There are memory leaks etc which are fixed in Service Pack 1 - if you are not on that version, you might want to consider upgrading.
Also, check that your SQL Server disks are not fradmenting, especially around tempdb. If you have not placed tempdb on a fast disk subsystem by itself, you might want to consider that. Also, if you can size it, precreate it so that SQL Server does not have to keep grabbing more disk space.
Finally, check your SharePoint and system event logs on all servers to ensure that you don't have application errors that could be slowing things down.