Automic Workload Automation

Hi folks,

Is anyone out there running their AE in a Distributed Server Environment, like us ?

Our Production system is AE v9 SP11 and we have installed CP/WP server processes on two virtual Windows 2008 R2 servers (VMWare).
The two Windows servers are located in separate Data Centers in order to increase system stability, e.g. in case one Data Center fails we can still operate from the other site.
We have four CP's, two on each node. I recently found out (INC0348544) that in order to be able to handle all my Agents if one node fails, I will have to create more CPs on my system (because each CP can handle a maximum of 2030 connections).

If you are also running in a Distributed Server Environment, I curious to know:

How many server processes (CPs and WPs) do you have?
How many Agents do you have ?

Regards

Hi Keld,

We have a similar setup as you. 2 VM app nodes, 2 CPs on each, for a total of 4 CPs. We have roughly 400 connections per CP, so way under the limit, although glad I came across your post as I did not know of the limit until now.

We've just doubled our WP count from 8 (4 on each host) to 16 (8 on each). This improved performance A LOT for us, as we have many concurrent users accessing the platform and issuing UI actions through DWPs. We generally have 10 DWPs (up from 3-4) and 5 WPs and 1 PWP. Increasing WPs is easy (just need to add in .ini entries in ucsrv.ini), I believe there is more to it for CPs (possibly need to load DB? Don't quote me, but pretty sure there is more to it than with WPs).

Our 2 app nodes are not in different data centers, however.

Hi Eric,

Can you share some details regarding you VMs? What VM version, number of cores, CPU and so on...have you had any performance issue related to VM ?

10 DWPs ?? That's a lot..., how many concurrent users do you have? (in average we have 50 concurrent users connected to our AE Prod system)
How many jobs do you execute on a daily basis ?

Were your CPs/WPs very busy since you decided to double the number ?? did you monitor the busy time over time (System Overview -> Automation Engine -> b.01, b.10, b.60) ?

Adding WPs is quite easy and can be done on the fly, but if you need more than the standard 5 CPs, then you need to execute the SQL statement to add them to the database..I have done that a couple of times...

Best practice is that you "double everything" when running in a distributed server environment, because one node should be able to handle the load if the other node goes down...

I'm always looking for ways to improve the performance, and the number of server processes plays a big role. Currently we have 12 WPs (8 WPs, 3 DWPs and 1 PWP). 
The Million Dollar question is: How many CPs/WP can be added to gain optimal performance without causing administrative overhead on the System? 

Regards
Keld.

We are running CentOS 6.3 currently. 16GB mem each. AE v9 sp11 hf2, with eyes on getting to 11.2 predominately for zero-downtime future upgrades.

We have 8 cores per app node, but most go lightly/un used. There is another thread about this HERE. Maybe it's expected that only 1 CPU will take brunt of load, but makes our infrastructure resource folks anxious seeing 7/8 cores barely being used. The graph Chip pasted on that other thread was before we doubled our WPs. Having checked today, average usage for cpu0 has increased significantly to 40-50% (up from 20-25%), while others still hover below 5%. Not ideal, but cpu0 can be explained by doubling WPs I suppose. Other than this, can't think of any issue we have had regarding performance at the VM/OS level. Memory does seem to climb a bit over time, but as they say, unused memory is wasted memory and we're fine so far.

We run about 100,000 jobs per day.

We increased our DWPs because we were having frequent complaints of user actions taking a very long time to complete. Indeed, we noticed it too, and yes you can see this in System Overview > AE--we would see "100" for all DWPs. Part of it was a known bug in the software version we were in, which we've patched now.  So, if you have a few people who ask for stats at the same time and it takes 10-20 seconds or more, those actions bottleneck your DWPs and no other user requests will be processed in the mean time. 10 DWPs + the recent patch has shown a dramatic improvement. We have 50-75 users typically.

Our WP activity spikes a bit on the hour, since many teams "Force adjustment on timeframe," but otherwise only a couple WPs seems like enough.

For us, the database seems like where we need to focus to increase performance. Even after the patch, we think user requests could be more responsive. So we're working on tuning the DB now to get getstatistics() calls and getactivities() calls lower.We've added some indexes to improve DB Maintenance. We've shrinked some tables, and still working on reorganizing/rebuilding. Apparently it's tricky to push these kinds of things through on an active DB.

What do you mean by administrative overhead?

Hope this helps!

Hi Eric,

We are planning on installing ILM (http://docs.automic.com/documentation/AE/9.13/english/AE_WEBHELP/uc4.htm#ucacqa.htm?Highlight=ilm) - Are you considering that too?
We are on Oracle 11.2, what about you?

In theory you would gain 100% performance if you double the WPs/CPs, but only in theory, because the System spends relatively more time on internal tasks than on regular work due to the increased number of server processes. This is what I call 'Administrative overhead'.

Hi Eric,
Automic describes the 'Administrative overhead' like this:

"Automic recommends setting  up the number of your AE system's server processes in accordance with  the above listing. There is no limit but with the number of server processes  increasing, the internal administration effort becomes higher than the  benefit gained from load distribution."

http://docs.automic.com/documentation/AE/11.2/english/AE_WEBHELP/help.htm?product=awa#ucabnp.htm