Layer7 API Management

We've been working with our store and look up in cache use and wondering about its effect on system resources.  The topic is pretty well summed up by the following question.  So far I haven't found any documented explanation of how it's implemented.

"I would be curious to see how Layer7 physically stores the data. Are they writing to a database table or storing local files or saving in RAM? Does it allocate the entire spot for a large file + many sessions, or dynamically assign based on actual use? Those may help determine best practices for enabling cache assertions. "

Cache is memory, it's defined by the Cache key that's used to store the value. This value is accessible by anyone who accesses that cache using that specific key.

Thank you!  Now if I could expand on the question a little, it would really help.

What happens when a cache is, let's say "over used".  With hundreds of services potentially using a few MB's of cache space is there anything that prevents this use running the system out of memory?  Is cache space in any way limited or isolated?  Is there any way to set a threshold on how much memory can be allocated to cache?  There should be, in my opinion.

Great Question and "Yes", this is very important.

We control this by the 'Store to Cache' assertion itself,

The 'Max Entry Size', referring to the message that is being cached and the 'Max Number of Entries' should be carefully planned out here. In theory you can multiply the 2, to get max possible memory consumed ( Takes into account Max Entry Age and Cache Keys ). If the number of entries is exceeded, then the new entry will replace the oldest in the cache. ( First in First Out ).

Thanks Doyle.  Please correct me if I'm wrong on any of the following.

I'm aware of the store to cache parameters.  Problem is, what happens when we have hundreds of services using that assertion, created by multiple developers?  While I appreciate the intention of careful planning, the fact is, in a large implementation that's going to be real hard to.  In fact, it's difficult to find all services that use the cache assertion, a "where used" utility is something we've long asked for from product development.  (We wrote our own using RESTMAN.)  On one small system I find 45 services using the store to cache assertion.  I'd have to look through each one to determine how many unique cache's I have, add up each one's max size, determine an acceptable limit, all of which would have wide variability, then try to police the service author's activities to comply with the objective.  Not workable.  And that's for 45 services/policies.  Imagine 450 of them.

Worse yet, I assume the memory is constrained not by the system availability, but rather by the gateway's jvm heap space (it appears a typical VM gateway has 2 GB of heap allocated by default), so we can't really even really rely on overall system memory metrics.  It would be nice if there was a way we could:

1.  Limit the amount of heap space that can be allocated to cache.  If the limit is reached, produce a warning, don't allow any new cache (or deallocate an old one), but don't crash the system.

2.  Find out how many cache segments are resident in memory.

Yes, i see where you are coming from now. This is a valid concern, if this is a shared infrastructure and have several people developing policies and utilizing cache, it would be hard to monitor this. I do have a few comments regarding a few of your statements, but understand that this isn't a solution. i just want to clear up a few things.

Your most likely correct in regards to these Cache, being stored in Heap, and depending on how much resources the VM has, depends on how much space Heap is allocated. if 4GB or less, i think it uses 2GB as you stated, but if set up via best practices, as the vm has more than 4GB, the heap should be allocating anywhere from 1/2 to 2/3 of the hosts memory, but adding more memory isn't the solution. if bad cache management practices continue to occur, it only delays the inevitable.

I've often 'Fantasized' of a visual page showing me each Cache and the configured properties and current Utilization. This would be pretty nice. If you would like, you/I can create an Idea on Communities for something similar that fits your requirements and we can vote on it, which should get put on Product Management's Radar for consideration in a future release? I know this isn't an immediate fix, but could work out well in the future.

As for a more immediate 'Work Around', i think the way that i would approach this would be to monitor the JVM Heap Utilization. You can do this via requesting our 'Get JVM Metrics' assertion from Support. Once installed, you can use that assertion in a policy to display current Heap Utilization. with this info, you should be able to monitor this and maybe trigger certain actions to happen if the threshold maintains a certain level ?

One of the triggers could be a restman call to modify the cluster wide property  'messageCache.resetGeneration' which dumps all Cache in the Cluster or just configure this to run on a regular interval, instead of expecting the trigger from the jvm monitoring.

Other than this, the most i can say is that you can try to manage this procedurally via Code Review and having a sort of 'Governing Gateway Team' that reviews and approves what code is put into the next environment.

hope my ramblings helped

Thanks Doyle.  There's certainly some information in there I was not aware of.  I've requested the JVM Metrics assertion from support.

With regard to "set up via best practices" on memory, keep in mind the VM systems are delivered as an image in an OVA file.  So those best practices should be communicated to whoever creates that configuration.  In general we figure they know best, so don't typically mess with it.  Obviously we could change the startup to allocate more heap.  In our case the OVA seems to have specified the 1/3 allocation (per your best practices statement), as our VM's have 6 GB memory, with 2 given the heap via the -Xmx parameter.

For a visual on the cache utilization, have you ever worked with CA Introscope (APM)?  It's been a while for me and my memory is fuzzy but I know it had all sorts of 'platform' metrics like this.  Not sure it will break it down like that but it certainly shows heap utilization.  We've contemplated instrumenting the gateway JVM with Introscope, but we don't currently have any unused agent licenses with which to do so.

As you stated, since the appliance is shipped via OVA, we don't know up front how much memory will be allocated. We have our minimum recommendations, but is designed for minimum load. I've never heard of this '1/3' setting, before, with 6 GB of memory, i'd configure that to be '1/2' personally. Our new 9.0 Gateways will ship with this configuration already in place, so you won't have to worry about this going further, but it is a general best practice to apply this setting in the applianceDefs.sh file.

I have worked a little with introscope, but haven't seen anyone attempt to get JVM Data populated from the Gateway. I don't foresee it being difficult at all though. I'd use the assertion mentioned above to expose an endpoint that returns the data in the expected format the EP Agent prefers or you can look more into a solution our Tactical Team put together, which basically has the Gateway querying itself and shooting the data off to the Introscope Server.

I'd personally try to avoid anything that has an agent that requires itself to be injected into our JVM. I've seen performance degradation with this approach and would prefer a solution that our Engineering team has put together ( i.e. the assertion above ).

You should also be able to accomplish this via Splunk, ELK, or any other monitoring based tool that has a little flexibility if licensing is an issue.