What is the current amount of available RAM on the device and how may of the DevTest components are you executing simultaneously? Forcing lower memory consumption using VMOPTIONS may cause other issues and eventually start throwing out of memory errors.
System requirements can be found here: System Requirements - DevTest Solutions - 10.1 - CA Technologies Documentation
Minimum requirement highlights are:
DevTest Server (Registry, Portal, etc.) - 2 GHz or faster, 8 cores minimum, 8 GB available
DevTest Workstation - 2 GHz or faster, 2 cores minimum, 4 GB available
VSE - 2 GHz or faster, two cores minimum, 2 GB available RAM for VSE, in addition to the RAM requirement for DevTest Workstation, DevTest Server, or Application Insight
Depending on what you are trying to achieve, you might check out using the Community Edition which has a smaller memory footprint.