When you are using big program like Firefox yes, it's big, some of memory usages may go into swap. However, if your harddisk's performance is like mine, it's totally in pain. One word only can describe slow.

There is a way to move the content in swap back to main memory, run the following as root:
swapoff -a
swapon -a

Doesn't look genius to me, but I don't know if there is any way else. If it is, please leave a comment.

You may also want to change the system setting, so reduce the chance of moving back. On my Fedora 10, the default vm.swappiness is 60. You can change it on-the-fly as root:
sysctl vm.swappiness=10
And edit the /etc/sysctl.conf as root:
vm.swappiness = 10

If you don't want to use swap anymore, after you swapoff, then to remove the swap partition by removing from /etc/fstab and fdisk'd.