Poor AE performance following DB load of transport case

Discussion created by Michael_Lowry on Oct 10, 2017
Latest reply on Feb 7, 2018 by Michael_Lowry
Recently, we have observed the following problem: in the minutes following the load of a transport case file, there is a substantial spike in the usage of work processes. This can have a negative impact on the performance of the AE system. In particular, the Java User Interface becomes sluggish or unresponsive. (The AWI may also be affected.)

Automic Support initially claimed that the problem was due to poor DB performance. We’re not sure whether this is a correct or adequate explanation. In any event, we have learned more about the problem.

If several (typically more than 2) instances of the AE DB Load utility ucybdbld run at the same time, the following message appears in the log:
U00038129 Waiting 1 minute for the generation of Version Control Objects. Processing will be continued afterwards.
Message U00038129 is mentioned (using the old numbering system) in this old v9/v10 technical article:

000009614  Loading Transport Case takes long time and receive U0038129 Waiting message

I will paraphrase the article below:
If a transport case is loaded into a client that has VERSION_MANAGEMENT enabled (set to YES), the AE creates entries in the internal task list (ITL) table (ITL_TType=GEN_OX) to create version management entries for the objects loaded from the transport case. Those entries are then processed by the Dialog Work Processes (DWPs), independently of the DB load. This processing can take from minutes to hours, and can last longer than the DB load itself. Each DWP can create one version management object every 4 seconds.

If a transport case is loaded while the DWPs are still processing entries in the ITL (e.g., from a previous DB load), message U00038129 will appear and ucybdbld will wait a minute before trying again. Once the ITL is empty, ucybdbld will proceed with the DB load.
We have version management enabled, and we suspect that the poor AE performance we have seen (and particularly poor GUI performance) is caused by the processing of these ITL entries. The problem appears to be exacerbated by the following factors:
  • The AE system has many connected users
  • Multiple deployments (DB loads) are run simultaneously or in quick succession
Additionally, we have seen an increase in the frequency and severity of instances of the problem since we upgraded from v11.2. to v12.0.3.

The documentation page describing the Transport Case states:
Transporting objects to a system/client with running activities might impact performance temporarily.
A variant of this warning has appeared in the documentation going back at least as far as v9.13. My guess is that with the introduction of the dedicated Dialog Work Process, the overall AE performance of ITL processing is limited, because processing of the ITL table is handled exclusively by the DWPs. We largely confirmed this conjecture. In the moments following the load of a transport case, the usage of dialog work processes jumps dramatically. I just ran a quick test:
  1. Before the test, no WP was above 40% usage (the B.01 column).
  2. I loaded a transport case containing a single empty DOCU object. (This file contains 14 records for the OH table, plus a folder path, and that’s it.)
  3. By 15 seconds after the DB load had completed, the usage of DWPs began to spike.
  4. By 30 seconds after the DB load had completed, 14 DWPs and 6 WPs were at 100% usage.
  5. By 1 minute after the DB load had completed, the usage levels had begun to drop again.
  6. By two minutes after the DB load had completed, the usage level had returned to normal.
Even during the peak, there were still a few DWPs that were not at 100%. So the GUI continued to be usable, if a bit sluggish. But remember, this was just an essentially empty transport case. Normal deployments are much bigger, involving many objects with records in many tables. Large deployments can lead to extended periods during which the AE is so busy it can barely serve GUI users.

In the test above, the transport case contained an object that already existed in the target system. It was replaced, meaning that version management came into play. I did another test using a transport case containing an object that had never existed in the target system. (So version management should not need to be involved.) Nevertheless, this DB load caused the same high usage of DWPs & WPs. So it appears that these ITL entries are written (and processed) regardless of whether the object is new or a replacement of an existing object.

Has anyone else seen this problem?