How much RAM do you have?
Basically if the pagefile is being used then the disk is being accessed while this happens so you can imagine stuff will slow down as you're loading data off the disk at the same time as Windows is trying to manage a pagefile of that size on the same disk. Generally this isn't a problem if you have enough RAM as pagefile usage will be lower so less disk thrashing during daily usage.
You have a few solutions.
1) Install a spare HDD that is comparable in speed and set the Pagefile to that drive while turning it off on the OS drive.
2) Install more RAM and limit the pagefile to a set min/max (1GB, for example).
3) Upgrade to Windows 7 which has much better memory management