Look for the "bin" folder within the DevTest installation folder. There's a file named Workstation.vmoptions
There are instructions in there on how to configure it. For example, use -Xmx1024m to increase heap size to 1 GB.
If you set this value too high, however, Workstation won't launch. It depends on how much physical memory you have and what else is running.
--Mike