VMWare, PAM, Java applications and resource ballooning

Discussion created by Jennifer_Jessup Employee on Jun 6, 2012
Latest reply on Oct 12, 2012 by MiguelAngelo
We have clients that run into random memory problems a lot, and a lot of the problems stem from VMWare and resource allocation. Specifically something that is called ballooning from the ESX host systems.
Here is an explanation of ballooning, and how to resolve it.

An explanation of virtual machine resource ballooning

In a virtual environment your VM Guest system is provided by an ESX Host system. This host system is what is "feeding" the resources to each guest system.

In a typical environment you may have several guest systems provided by a single ESX host. Because of this, something called "ballooning" is enabled on the ESX host.

What ballooning does is control the feed of resources to the guest systems as it is available. So for instance, if you specify Guest A to have 8G RAM, you do not get a contiguous block of 8G. You get smaller blocks of available memory (whatever the host has) that will equal 8G.

The problem with Java and setting the max heap size is that it expects this in a single block of contiguous memory. So if you tell Java to grab 4G, it will only function at that point if it has a contiguous block of 4G.
The problem being that the hypervisor is only feeding the "8G" to the guest in much smaller blocks so there is not a block of a contiguous 4G anywhere, and the application that is controlled then by Java can not start up.

We can see this in the performance graphs from the ESX host. If your VMWare Admins can get this information from the ESX host that owns this guest system, they can confirm this for you.

We do know that "ITPAM" supports 4G and even more. In fact, it's not actually the application behind Java, it's whether or not Java supports it. For a 64bit Java in a 64bit operating system environment, Java has no upper limit for memory. There are customers running CA Process Automation with 4G, 6G and even 8G of ram set for the max memory Java heap with out issue. The errors seen when you try to start Java with more memory than is contiguously available aren't going to be "out of memory" errors as it hasn't even started up to run out of memory, but more of what is seen where the service won't start up.

So there are a couple of options here.

First being to disable ballooning at the ESX host. This information is provided below, but you need to get your VMWare Admins involved in this.

Second, if disabling the balloon driver on the ESX host is not an option, try smaller increases to the max memory heap setting. Increase the amounts by 512.
So start with 1536 and run with this amount for a week. If you still see that you are having the same problems, increase the max mem to 2048, and so on.
Gradually increasing the memory heap allows you to see exactly what your contiguous limit is, if you hit it before the problem is resolved please involve your VMWare Admins as they can get the performance graphs that will confirm the ballooning of the resources.

How to disable the balloon driver on the ESX host

Connect directly to the ESX Server host where the virtual machine resides on, using Virtual Infrastructure Client (VI Client).
Log in to the ESX Server host as a user with administrative rights.
Shut down the virtual machine.
Right-click on the virtual machine listed on the Inventory panel and click Edit Settings.
Click the Options tab and select General.
Click Configuration Parameters.
Click Add row and add the parameter sched.mem.maxmemctl in the text box.
Click on the row next to it and add 0 in the text box.
Click OK to save changes.

*****Disabling balloon driver in a virtual machine results in performance issues with the ESX Server host. Refer to page 132 of "Resource Management Guide" for more information on balloon driver.