When you run a 32-Bit Windows application Windows assigns it 4 GB of memory. This isn't real memory, it's virtual memory. Normally the application can only use half of this, the other half is reserved for the kernel. The net result is that the application has 2GB of memory to play with.
This is a limitation of 32-bit applications. It doesn't matter if you are running them on a 32-bit or a 64-bit operating system.
There is an old trick for running Exchange or SQL server on a 32-bit operating system. The trick is that you can set the /3GB and /PAE kernel flags to give the application 3GB instead of 2. In this scenario the application is still assigned 4GB of virtual memory, but the operating system packs itself into 1GB instead of 2.
I had always thought that the application had to understand large address spaces to be able to take advantage of this. Today I learned that is not necessarily the case.
There is a tool called "The Microsoft COFF Binary File Editor", cleverly named EDITBIN.EXE, that has the capability to modify executables. This tool is packaged with Visual Studio and documented here: https://technet.microsoft.com/en-us/library/xd3shwhf(v=VS.90).aspx
One of the things it can change is turning on the extensions that let the applications use this extra memory. Here's how
EditBin.exe /LARGEADDRESSAWARE NameOfTheExecutable.exe
Viola, that's it. Add this to your toolbox for fighting out-of-memory errors in old 32-bit applications. Magic.
Fair warning, this change modifies the EXE file. Make a backup of the .exe file first. Also know that if the .exe is digitally signed this modification will invalidate the .exe's digital signature and you'll get warnings about the signature being invalid. It's a bad practice to blindly ignore bad signatures, but in this case it is expected.
It's worth noting that you don't have to use the /3GB or /PAE switches on a 64-bit OS, the OS understands big addresses. All you should have to do is flip the large address aware bit in the executable and you are off to the races.
Best of luck. This won't work for every app, but it worked for me.