- A Gateway under load is expected to allocate more memory over time, to a point based on the heap size settings; this increase doesn’t indicate a leak
- A Gateway which becomes idle will generally not deallocate memory; this doesn’t indicate a leak
- Gateway heap size requirements are dependent on your use case; one size doesn’t fit all
- If your heap is too small, throughput is likely to suffer
- If your heap to too large, you may experience occasional high latency
Introduction to the JVM
Java applications like the CA API Gateway are run on a Java Virtual Machine (JVM). Important functions of the JVM include loading, interpreting, and executing the application, and managing its memory. Memory management includes allocation of various categories of memory and performing garbage collection. The largest category of memory is the heap, and significant space can also be used by stack memory allocated for each application thread, and Metaspace for application code and metadata.
The JVM is started by the ‘java’ command, and there are numerous arguments which can be provided to this command which configure the behaviour of the JVM, including heap sizing and garbage collection parameters. A minimum and maximum heap size can be specified, which allocates a certain amount of memory to be allocated for the heap on JVM startup, and a maximum size that the heap can grow to if needed.
Finding JVM System Resource Usage
On a Gateway appliance, the java process that started the JVM running the Gateway application will typically be the largest consumer of memory and CPU time; this usage along with other information about the process can be found in the output of the top command.
'top' output (edited to show only the gateway process):
top - 15:50:42 up 43 min, 1 user, load average: 0.04, 0.02, 0.02
Tasks: 228 total, 1 running, 227 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.5%us, 1.2%sy, 0.0%ni, 72.7%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st
Mem: 16466544k total, 7472140k used, 8994404k free, 22856k buffers
Swap: 2097148k total, 0k used, 2097148k free, 551540k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4753 gateway 20 0 14.4g 4.9g 23m S 223.9 31.0 91:43.71 java
The RES value is the resident memory usage - the memory allocated by a process that is occupying RAM. Generally, on a Gateway appliance the resident memory usage value for the Gateway java process is the total memory usage of the application (i.e. there will not be allocated memory swapped to disk, which would not show in the RES value). The resident memory usage can be calculated as:
Total Memory Usage = Heap + Thread Stacks + Metaspace + Other Memory Categories
See Further Reading for details of other memory categories.
Heap Usage and Garbage Collection Basics
The heap for the Gateway application contents are a mix of short- and long-lived items, and like almost all Java applications it is constantly changing. When an item in the heap is no longer in use it will be removed from the heap automatically by periodic Garbage Collection (GC), but until it is Garbage Collected (GC’d), it will still count as “used” heap space.
Automatic garbage collection is the process of looking at heap memory, identifying which objects are in use and which are not, and deleting the unused objects. An in use object, or a referenced object, means that some part of your program still maintains a pointer to that object. An unused object, or unreferenced object, is no longer referenced by any part of your program. So the memory used by an unreferenced object can be reclaimed.
For example, the contents which represent a Published Service are largely static and stay in memory almost indefinitely. They are only thrown away and replaced when its Policy is changed. On a production Gateway the heap contents which represent the Published Service could stay in the heap unchanged for weeks or months. Long-lived items like these are eventually moved to a special part of the Heap (the “Old Generation”) which is GC’d less frequently. These items tend to be a major component of “baseline” heap usage – the average level of heap usage after a GC has cleaned up unused items.
On the other extreme, when that Service handles a request, huge numbers of new items of varying sizes are created and stored in the heap. These items are only used for nanoseconds or a few milliseconds in the process of reading the request, inspecting and manipulating it, auditing and logging, and returning a response. These items can cause heap usage to increase rapidly but are GC’d very frequently.
When a new item needs to be added to the heap, but it is already full, a GC will be initiated to remove unused items and free up space for the new item. In the process of this, the GC algorithm may decide to allocate more memory for the heap. This usually happens either because after removing unused objects there is still not enough free space for the new item, or because the algorithm determines it will improve GC efficiency. A larger heap may require a lower frequency of GC, with a tradeoff that each GC may take longer on average because it must process more items. However, the result is less time spent in GC overall meaning more CPU time spent doing real work.
It is normal for a Gateway’s heap to grow over time (i.e. more memory will be allocated for it by the JVM) as an effect of GC, until it either levels off at a size which effectively accommodates the regular application activity or reaches the maximum heap size specified. This growth is reflected in the resident memory usage, which will also steadily increase until it reaches that maximum plus the size of the other memory categories. Under the Gateway’s configuration the heap size will very rarely shrink. However, as explained, that does not necessarily mean that as the heap grows the Gateway is “using” all that memory because the allocated physical memory for the heap does not reflect how much of the heap is in use, nor how much of that heap space would be reclaimed after the next GC.
Heap Growth Factors and Appropriate Heap Sizing
The Gateway startup script automatically sets the maximum heap size to be half of the total physical memory of the appliance e.g. a 16 GB VM will result in an 8 GB maximum heap. Container Gateways specify the heap size directly, for example in a Docker Compose file. The minimum heap size for an appliance Gateway is undefined and will use a platform- and Java version-dependent default, commonly around 2 GB. Container Gateways usually specify the minimum to be the same as the maximum.
Note that the growth rate of the heap and the maximum heap size required for good performance depends on customer and deployment-specific variables such as the Gateway’s load, the size and content type of the messages it handles, the types of manipulation and transforms it is performing, use of the Store to Cache assertion, the number and size of various connection pools, and many other factors which affect memory usage. If your heap is too small for your use case, GC will be frequent and can severely impact your performance. If your heap is very large, GC will be less frequent but can be very time consuming.
I have observed that a maximum heap size of between 8GB and 16GB satisfies most customer use cases, but 4GB may be plenty for yours. Appropriate appliance and Docker container memory sizing for your deployments should always be determined by dedicated performance testing in pre-production environments with traffic and concurrency levels which accurately reflect your production load.
- Resident memory allocation for the Gateway java process is expected to be more than the JVM heap size
- The Gateway resident memory usage is expected to increase with use (up to its configured maximum); it is not expected to shrink
- The amount of allocated memory doesn’t indicate how much of that memory is currently “in use” by the Gateway
- Set heap sizes for your Gateways according to your production traffic needs, based on pre-prod testing