Upping the vram with a new graphics card won't help and on a 32 bit windows system that actually takes quite a bit of ram away from the operating system and/or pse 11.
It would be much more efficient/better to update to a 64 bit version of windows and then pse 11 can use up to around 3.2 gb of ram instead of the around 1.6 gb your limited to now.
Still though, if you images are large and the area to fill is large, then even a 64 bit os might not help with the fill part in photomerge.
You might try reducing the pixel dimensions of the images, then saving them as tif files and try photomerge again.