Automic Workload Automation

Expand all | Collapse all

Poor AE performance following DB load of transport case

  • 1.  Poor AE performance following DB load of transport case

    Posted Oct 10, 2017 10:12 AM

    Recently, we have observed the following problem: in the minutes following the load of a transport case file, there is a substantial spike in the usage of work processes. This can have a negative impact on the performance of the AE system. In particular, the Java User Interface becomes sluggish or unresponsive. (The AWI may also be affected.)

    Automic Support initially claimed that the problem was due to poor DB performance. We’re not sure whether this is a correct or adequate explanation. In any event, we have learned more about the problem.

    If several (typically more than 2) instances of the AE DB Load utility ucybdbld run at the same time, the following message appears in the log:

    U00038129 Waiting 1 minute for the generation of Version Control Objects. Processing will be continued afterwards.

    Message U00038129 is mentioned (using the old numbering system) in this old v9/v10 technical article:

    000009614  Loading Transport Case takes long time and receive U0038129 Waiting message

    I will paraphrase the article below:

    If a transport case is loaded into a client that has VERSION_MANAGEMENT enabled (set to YES), the AE creates entries in the internal task list (ITL) table (ITL_TType=GEN_OX) to create version management entries for the objects loaded from the transport case. Those entries are then processed by the Dialog Work Processes (DWPs), independently of the DB load. This processing can take from minutes to hours, and can last longer than the DB load itself. Each DWP can create one version management object every 4 seconds.

    If a transport case is loaded while the DWPs are still processing entries in the ITL (e.g., from a previous DB load), message U00038129 will appear and ucybdbld will wait a minute before trying again. Once the ITL is empty, ucybdbld will proceed with the DB load.

    We have version management enabled, and we suspect that the poor AE performance we have seen (and particularly poor GUI performance) is caused by the processing of these ITL entries. The problem appears to be exacerbated by the following factors:

    • The AE system has many connected users
    • Multiple deployments (DB loads) are run simultaneously or in quick succession

    Additionally, we have seen an increase in the frequency and severity of instances of the problem since we upgraded from v11.2. to v12.0.3.

    The documentation page describing the Transport Case states:

    Transporting objects to a system/client with running activities might impact performance temporarily.

    A variant of this warning has appeared in the documentation going back at least as far as v9.13. My guess is that with the introduction of the dedicated Dialog Work Process, the overall AE performance of ITL processing is limited, because processing of the ITL table is handled exclusively by the DWPs. We largely confirmed this conjecture. In the moments following the load of a transport case, the usage of dialog work processes jumps dramatically. I just ran a quick test:

    1. Before the test, no WP was above 40% usage (the B.01 column).
    2. I loaded a transport case containing a single empty DOCU object. (This file contains 14 records for the OH table, plus a folder path, and that’s it.)
    3. By 15 seconds after the DB load had completed, the usage of DWPs began to spike.
    4. By 30 seconds after the DB load had completed, 14 DWPs and 6 WPs were at 100% usage.
    5. By 1 minute after the DB load had completed, the usage levels had begun to drop again.
    6. By two minutes after the DB load had completed, the usage level had returned to normal.

    Even during the peak, there were still a few DWPs that were not at 100%. So the GUI continued to be usable, if a bit sluggish. But remember, this was just an essentially empty transport case. Normal deployments are much bigger, involving many objects with records in many tables. Large deployments can lead to extended periods during which the AE is so busy it can barely serve GUI users.

    In the test above, the transport case contained an object that already existed in the target system. It was replaced, meaning that version management came into play. I did another test using a transport case containing an object that had never existed in the target system. (So version management should not need to be involved.) Nevertheless, this DB load caused the same high usage of DWPs & WPs. So it appears that these ITL entries are written (and processed) regardless of whether the object is new or a replacement of an existing object.

    Has anyone else seen this problem?



  • 2.  Poor AE performance following DB load of transport case

    Posted Oct 10, 2017 10:15 AM
    I noticed this remark in known error article 9164:
    If a new system is being loaded with a transport case containing objects from another system, the load utility will wait regardless of whether Version Management is active or not.
    It would be nice to get a clear explanation of what this means.


  • 3.  Poor AE performance following DB load of transport case

    Posted Oct 11, 2017 07:47 AM
    Immediately after running another DB load of the same empty DOCU object, I listed the entries in the ITL table.
    select * from ITL
    Here are the results:

                           
    ITL_IDNRITL_TTYPEITL_TIDNR
    281113RE_US_CA1
    281114GEN_VNME1666002
    281115GEN_OX1665001

    OH_IDnr 1665001 refers to the object just updated. 1666002 is the archived version (the version of the object from before the DB load). The object and the archived version were visible in the AE before the ITL entries had been processed, so the ITL entries do not have to be processed before the updated object and the version in the Version Management tab are available. The ITL table entries disappeared within a few seconds after the DB load had completed, but the high level of DWP usage persisted for several minutes.

    We will run a DB trace on the DWPs to see what they’re actually doing after they read & delete the ITL entries.


  • 4.  Poor AE performance following DB load of transport case

    Posted Oct 11, 2017 08:09 AM

    I did another DB load, and this time captured the names of the referenced objects.

    select ITL_Idnr, ITL_TType, ITL_TIdnr, OH_Name
    from ITL left join OH on ITL_TIdnr=OH_IDnr
    ITL_IdnrITL_TTypeITL_TIdnrOH_Name
    281113RE_US_CA1
    281114GEN_VNME1666002000000005.UC0.TEST_DEPLOYMENT.1.DOCU
    281115GEN_OX1665001UC0.TEST_DEPLOYMENT.1.DOCU


    Note the temporary name assigned to the archived version. Once the ITL entries had been processed, the AE renamed this object to UC0.TEST_DEPLOYMENT.1.DOCU.VERSION.20171011.105830.



  • 5.  Poor AE performance following DB load of transport case

    Posted Oct 11, 2017 08:53 AM

    I wrote a script to submit an arbitrary number (up to 10) of simultaneous DB loads. Each of the 10 transport case files used contains a uniquely-named empty DOCU object.

    #!/bin/bash
    ae_utilities_dir="/usr/local/ebm/uc4/utilities"
    deployment_dir="/tmp/Deployment_Tests"
    batch_name="
    UC0.TEST_DEPLOYMENT"
    function usage(){
            echo "Usage: run_simultaneous_deployments.sh <target environment> <number of deployments>"
            echo "where number of deployments is a number between 1 and 5"
            exit 1
    }
    if [[ $# -ne 2 ]];then
            echo "Error: Invalid number of arguments specified."
            usage
    fi
    target_environment=$1
    number_of_deployments=$2
    case ${target_environment} in
    "EXP")
            target_client="0001"
            ;;
    "DEV")
            target_client="0100"
            ;;
    "ITE")
            target_client="0110"
            ;;
    "PROD")
            echo "Target environment ${target_environment} is not allowed."
            usage
            ;;
    *)
            echo "Target environment ${target_environment} is unknown."
            usage
            ;;
    esac
    echo "Target environment    : ${target_environment}"
    echo "Number of deployments : ${number_of_deployments}"

    if [[ $number_of_deployments -lt 1 ]] || [[ $number_of_deployments -gt 5 ]];then
            echo "Error: Specified number of deployments out of range."
            usage
    fi
    counter=1
    while [[ $counter -le $number_of_deployments ]]; do
            echo "Running command: ${ae_utilities_dir}/bin/ucybdbld -B -FOT -C${target_client} -X${deployment_dir}/${batch_name}.${counter}.txt -Ii${ae_utilities_dir}/bin/ucybdbld${target_environment}.ini -EREPLACE > ${deployment_dir}/${batch_name}${counter}_deployment_log.txt 2>&1 &"
            ${ae_utilities_dir}/bin/ucybdbld -B -FOT -C${target_client} -X${deployment_dir}/${batch_name}.${counter}.txt -I${ae_utilities_dir}/bin/ucybdbld${target_environment}.ini -EREPLACE > ${deployment_dir}/${batch_name}${counter}_deployment_log.txt 2>&1 &
            (( counter++ ))
    done

    If I submit several DB loads simultaneously using this script, they all run very quickly. Message U00038129 does not appear. I am guessing that ucybdbld does something like this:

    1. Check whether entries exist in the ITL table. If so, display messageU00038129, wait 1 minute, and try again.
    2. If/when no entries are found in the ITL table, proceed with the DB load.
    3. Check whether VERSION_MANAGEMENT is set to Y in UC_CLIENT_SETTINGS.
    4. If so, then for each object updated, add the corresponding entries to ITL.

    It seems that if one runs a few deployments at almost exactly the same time, they can all get past step #2 before any one of them has reached step #4.

    My understanding is that the U00038120-mechanism is designed to prevent problems that could result from simultaneous DB loads. It seems possible that the mechanism is not 100% reliable.



  • 6.  Poor AE performance following DB load of transport case

    Posted Oct 12, 2017 10:28 AM

    I just used the script to perform 5 simultaneous DB loads. None of these resulted in message U00038129. The ITL table briefly contained several rows:

    ITL_IdnrITL_TTypeITL_TIdnrOH_Name
    311113GEN_OX1726001UC0.TEST_DEPLOYMENT.4.DOCU
    311114RE_US_CA1
    314113RE_US_CA1
    314114GEN_VNME1729002000000010.UC0.TEST_DEPLOYMENT.1.DOCU
    314115GEN_OX1665001UC0.TEST_DEPLOYMENT.1.DOCU
    313114GEN_VNME1728002000000002.UC0.TEST_DEPLOYMENT.2.DOCU
    313115GEN_OX1667001UC0.TEST_DEPLOYMENT.2.DOCU
    312113GEN_OX1727001UC0.TEST_DEPLOYMENT.5.DOCU
    312114RE_US_CA1
    313113RE_US_CA1
    315113RE_US_CA1
    315114GEN_VNME1730002000000001.UC0.TEST_DEPLOYMENT.3.DOCU
    315115GEN_OX1674001UC0.TEST_DEPLOYMENT.3.DOCU

    I believe this confirms my theory above about how ucybdbld works. I wonder what would happen if I re-ran the test using several transport case files containing different versions of the same object(s).



  • 7.  Poor AE performance following DB load of transport case

    Posted Oct 17, 2017 05:52 AM

    I updated my script so that it is also capable of submitting multiple simultaneous XML imports (using the ImportObject Java API class). After a few tests with very small imports, I draw the following preliminary conclusions:

    • XML import takes aboutthree times as longas an equivalent DB load usingucybdbld.
    • XML importdoes not cause records to be written to the ITL table.
    • XML importdoes notlead to high usage of DWPs after the import has completed.

    I gather that this means during XML import, the AE server processes do more of the work — locking objects, creating VCS objects, and so forth — during the import process, and that they consequently have less to do after the import is complete.

    Previously, when we have discussed the pros and cons of the https://community.automic.com/discussion/10411/how-do-you-handle-promotion-of-ae-batches-from-dev-to-test-prod]transport case and XML file formats, we have assumed that XML offered poorer performance. Now we are beginning to see that this is an oversimplification. Yes, using the transport case, the DB load is quicker; but at least with v12.0.2, the negative impact on the AE server is higher.

    I will conduct additional tests with batches containing more objects.



  • 8.  Poor AE performance following DB load of transport case

    Posted Oct 18, 2017 07:25 AM
    In the v11 documentation, the section that describes loading the transport case includes a strongly-worded injunction against using ucybdbld to load many objects into a busy system (emphasis mine):

    Keep the following performance-improving notes in mind when loading many objects to a client:

    • There should beno activities running in this clientin order to make sure that the Automation Engine is notobstructed by database locks caused by the utility AE.DB Load.
    • Turn offVersion ManagementandRevisioningas otherwise, a version is generated for each loaded object which affects performance negatively.
    In the v12 documentation, the wording has been softened considerably:

    Keep the following  in mind when loading many objects to a client:

    Was the product changed between v11 and v12 to make it better at handling DB loads into busy systems, or was only the documentation changed?


  • 9.  Poor AE performance following DB load of transport case

    Posted Oct 18, 2017 08:42 AM

    Hi Michael,

    In general I’m not aware of any changes between 11 and 12, I guess the documentation was changed, because the v11 is too strict.

    Concerning what you wort in the beginning of the thread, I think there should be a detailed analyses on the system a TC with just one object should not case DWP’s went fully utilized (==> ticket with the traces you mentioned, if you want).

    There is a fundamental difference between XML import and TC load:

    • A XML import is entirely processed by a DWP. The VC object can be created right away.
    • A TC can be loaded even the system in down. Therefore the ITL is used to queue the VC object generation. The ITL’s are processed as soon as the system is up and running again or a DWP is idle and runs the next timer (every 20 sec.).

     KR, Josef




  • 10.  Poor AE performance following DB load of transport case

    Posted Oct 18, 2017 09:56 AM
    I've seen this before.  I do not recall that we really ever got a satisfactory answer / workaround.  Rafael_Lubera_1451 might remember more.


  • 11.  Poor AE performance following DB load of transport case

    Posted Oct 18, 2017 05:23 PM
    Josef Scharl said:
    There is a fundamental difference between XML import and TC load:
    • A XML import is entirely processed by a DWP. The VC object can be created right away.
    • A TC can be loaded even the system in down. Therefore the ITL is used to queue the VC object generation. The ITL’s are processed as soon as the system is up and running again or a DWP is idle and runs the next timer (every 20 sec.).
    Thanks for the clarification. I had neglected to consider that it is possible to load a transport case when the AE is down. This must mean that ucybdbld determines whether to display display message U00038129 based not on only whether there exist rows in the ITL table, but based on whether a DWP is currently processing these entries. I suppose that the DWP writes a row to the table to indicate that it is processing the entries in the table. (Perhaps this is the RE_US_CA row?)

    We have an incident open about this already. It’s INC00138606 in case you want to take a look.


  • 12.  Poor AE performance following DB load of transport case

    Posted Oct 20, 2017 10:33 AM
    We are experiencing the same behavior. Sometimes, I want to update all clients with a new set of our library's object and I am doing this by using ucybdbld for each client. So, it could be more than 50 consecutive db load. And I can assure you that the performance are affected and all the WP and DWP processes are running at 100%. Let us know if the support team comes with a hotfix.


  • 13.  Poor AE performance following DB load of transport case

    Posted Oct 24, 2017 10:28 AM
    Automic informed us that v12.0.3 addresses a bug related to performance problems after running a DB load, so we upgraded one of our systems and continued testing. AE v12.0.3 might be a bit better, but we are still seeing a pretty big spike in DWP activity after running a DB load, and a corresponding impact on GUI responsiveness.
    ucybdbld determines whether to display display message U00038129 based ... on whether a DWP is currently processing these entries.
    I confirmed this. Message U00038129  does not appear if multiple DB loads are run when the AE server is down. It also does not appear if the DB loads run quickly enough. I ran 10 DB loads at once (really, all at the same time), and did not see any instances of message U00038129 in the logs.
                                           
    Run
    Duration
    100:47.4
    200:28.7
    300:38.3
    401:15.3
    501:51.8
    601:42.2
    700:56.6
    801:33.3
    901:05.8
    1001:24.3
    I presume that these runs avoided the forced one-minute delays of message U00038129 because they all ran before the next time a DWP began processing entries in the ITL table. (This happens every 20 seconds according to Josef.)

    Next, I inserted a two-second delay between the DB loads. The last three runs had instances of message U00038129 in the logs. As a result, these runs took several minutes longer to complete.
                                           
    Run
    Duration
    100:22.9
    200:30.2
    300:37.1
    400:43.9
    500:51.0
    600:57.9
    701:04.8
    803:20.7
    903:28.2
    1003:35.7
    I will continue testing and report more results here.


  • 14.  Poor AE performance following DB load of transport case

    Posted Oct 24, 2017 11:12 AM
    With zero activity on the system, this is what the AE server usage looked like while 10 simultaneous DB loads* were running:
    • For about 5 minutes after I started the DB loads, the usage hovered around 8%.
    • Then it shot up to 100% for about 15 seconds.
    • Next it remained at 80% for a minute.
    • Finally it leveled off at 12% and remained there.
    While the DB loads were running, the GUI was barely usable. It constantly refreshed the folder list and the double arrow indicator flickered on and off. This is 100% reproducible. Tomorrow I will try again while the system is busy and has many active user connections.

    * The transport cases contained 66 objects each — with at least one of each object type.


  • 15.  Poor AE performance following DB load of transport case

    Posted Oct 26, 2017 09:59 AM

    I’ve learned a bit more about this problem. v12.0.3 definitely improves things, at least in situations where there is only one deployment running at once. In busy systems, this alone was enough to trigger a performance degradation in v12.0.2. With v12.0.3, that does not seem to happen.

    However, when more than one deployment (DB load) is allowed to run at once, v12.0.3 is still negatively impacted during the time when DWPs are processing ITL entries, even in a system with little activity and few open connections.

    #Scenario descriptionResult
    1Simultaneous DB loads of 10 transport case files, each containing 66 objects.  5 minute-long negative impact on AE server while ITL table entries were being processed:
    • No new GUI or Java API connections could be established.
    • System Overview could not be refreshed.
    • Folder tree in Explorer view did not work.
    • High utilization of WPs and DWPs lasting until several minutes after the ITL table had been cleared.
    2Single DB load of 1 transport case file containing 660 objectsNo noticeable impact on AE system performance, GUI usability, or API connectivity. Slight increase in AE server process utilization for about a minute.

    I notified Automic Support of my findings.



  • 16.  Poor AE performance following DB load of transport case

    Posted Oct 27, 2017 05:15 AM
    When as we identified this problem a few weeks ago, we quickly put in place the following work-around: deployments are queued serially rather than being allowed to run in parallel. Also, if additional deployments are queued up while a deployment is already running, a two-minute delay is inserted between the deployments. This delay is intended to give the DWPs time to fully process the ITL entries added by each deployment, before the next deployment runs.

    Since we implemented this work-around, we have not seen any significant GUI slowdowns associated with deployments.


  • 17.  Poor AE performance following DB load of transport case

    Posted Nov 02, 2017 07:18 AM
    Michael Lowry wrote:
    [W]hen more than one deployment (DB load) is allowed to run at once, v12.0.3 is still negatively impacted during the time when DWPs are processing ITL entries, even in a system with little activity and few open connections.
    ...
    Simultaneous DB loads of 10 transport case files, each containing 66 objects:
    • No new GUI or Java API connections could be established.
    • System Overview could not be refreshed.
    • Folder tree in Explorer view did not work.
    • High utilization of WPs and DWPs lasting until several minutes after the ITL table had been cleared.
    I had another thought about this today. Perhaps the poor GUI performance and unusable folder tree associated with multiple simultaneous DB loads has to do with many kick events being sent from the AE to the GUI.


  • 18.  Poor AE performance following DB load of transport case

    Posted Nov 02, 2017 02:21 PM

    Michael Lowry wrote:

    Perhaps the poor GUI performance and unusable folder tree associated with multiple simultaneous DB loads has to do with many kick events being sent from the AE to the GUI.

    I have found some evidence to support this hypothesis. I enabled xml=2 tracing for the GUI and then looked at the resultant trace file to see what the kick events looked like. The TREE_CHANGED kick event type looks like this in the XML trace:

    <kick client="0001" idnr="0000000000" system="AE_SYS_1">OFS        </kick>

    (OFS is the table where the AE folder structure is stored.)

     

    Knowing what to look for, I monitored the trace file while running several different deployment scenarios, with both single and simultaneous deployments of both transport case and XML files.

     

    Number of of OFS kick events sent from the AE to the GUI

    6 AE processes running¹62 AE processes running²
    #Deployment description
    TCXMLTCXML
    11 file containing 660 objects31471
    25 files, each containing 66 objects1552355
    310 files, each containing 66 objects 301047010


    With just a few AE processes running, the difference in the number of kicks between the two file types is only a factor of three. However, when many AE processes are running, the difference becomes very large. It seems likely that some of these kicks are unnecessary.

     

    Notes:

    1. 1 PWP, 1 RWP, 1 OWP, 1 CP, 1 DWP, 1 JWP
    2. 1 PWP, 1 RWP, 1 OWP, 5 CPs, 30 DWPs, 14 WPs, 10 JWPs



  • 19.  Poor AE performance following DB load of transport case

    Posted Nov 03, 2017 06:34 AM
    Edited by Michael A. Lowry Mar 12, 2023 04:45 AM
    What factors influence the number of kicks sent? I believe I have figured it out.

    The number of kicks sent from the AE to the GUI, k, is determined in the following way:
    XML
    k = f
    where:
    f is the number of files imported
    Transport case
    k f × w
    where:
    f is the number of files loaded, and
    w is the number of non-Java AE work processes running
    (PWP + RWP + OWP + DWP + WP)*
    The number of objects appears to play no role whatsoever.

    A colleague suggested that the the number of folders in the transport case might play a role, but additional tests did not support this hypothesis. Whether the file contains zero O records, just a few, or many, the number of kicks is the same.

    The number of open Explorer windows in the GUI also does not matter. The same number of kicks is sent whether the GUI has zero, one, or more Explorer windows open.

    Finally, I saw that the number of kicks the same regardless of whether the DB loads are run in sequence or in parallel. The reason we notice the problem more acutely when the deployments are run simultaneously is that all of the kicks arrive within a short interval. The GUI can handle 470 kicks arriving over a 20-minute interval, but is unable to handle the same number of kicks arriving within one minute.

    * With only six AE processes running, the number of kicks is not exactly proportional to the number of WPs. (It’s three kicks per file instead of four.) I’m not sure why this is.


  • 20.  Poor AE performance following DB load of transport case

    Posted Nov 04, 2017 09:54 AM
    For each transport case file loaded, there is one kick per WP. It seems reasonable to guess that each WP sends one kick. I wonder if this is truly necessary. It seems like one kick per file (as with the XML scenario) ought to be sufficient.

    I have informed Automic Support of these findings, and I’m awaiting their reply.


  • 21.  Poor AE performance following DB load of transport case

    Posted Nov 17, 2017 12:03 PM
    I have reviewed again Automic’s recommendations for sizing AWA systems. Automic advises considering four factors when sizing an AE system:
    1. Number of concurrently connected users
    2. Number of agents
    3. Number of object definitions
    4. Number of executions per day
    As these numbers grow, so do the stated system requirements, including the receommended number of AE server processes. In particular, as the number of users grows, it is necessary to add more dedicated DWPs. (According to the sizing tables, each DWP can handle approximately two connected users.)

    As we have discovered though, adding more WPs has a second unintended & undesirable effect: it leads to UI responsiveness degradation during deployments. If the behavior we have observed is the way the AE is supposed to work, then we could soon find that these are paradoxical requirements:
    • We must increase the number of DWPs to support additional users.
    • We must reduce the number to maintain a reasonable level of performance.
    I’m still waiting for a reply from Automic to my question about WPs and OFS kicks during processing of ITL table entries. I hope to receive a reply soon.


  • 22.  Poor AE performance following DB load of transport case

    Posted Nov 17, 2017 12:26 PM
    1 DWP handles 2 users?  Wow.  I have never heard that.  How many DWP's do you have on your system?


  • 23.  Poor AE performance following DB load of transport case

    Posted Nov 17, 2017 01:28 PM

    Laura Albrecht wrote:

    1 DWP handles 2 users?  Wow.  I have never heard that. How many DWPs do you have on your system?

    LauraAlbrecht608310: For each of the four sizes (small, medium, big, and high-end), just take the number of users and divide by the number of recommended DWPs. The result is approximately 2.

    UsersDWPsUsers per DWP
    1061.67
    50301.67
    200902.22
    400*1802.22

    In an environment with many active users, the number of recommended DWPs required can grow rapidly.

     

    We do not have that many DWPs at present, and we are waiting until we have an answer from Automic before making any changes.

     

    * For high-end, because the other numbers had also doubled, I assumed double the number of users as for big, 400 users.



  • 24.  Poor AE performance following DB load of transport case

    Posted Nov 20, 2017 05:14 AM
    I just confirmed that the behavior is the same for the AWI as for the Java User Interface: for each load of a transport case file, the AE sends one OFS kick to the AWI server for each running work process.

    An examination of the AWI log (with tracing enabled) also revealed a few other interesting details. As I suspected, the AWI maintains its own cache of the folder tree.  For each OFS kick received, the AWI server posts a corresponding FolderTreeChangedEvent for each connected user.
    2017-11-20 09:46:59,712 ionID=0000000020553975 [TRACE] NOSESSION/- NOUI   [.aetracing.AutomationEngineTraceListener] - <uc-env session="">
       <kick client="0001" idnr="0000000000" system="AE_SYS1">OFS        </kick>
    </uc-env>
    2017-11-20 09:46:59,712 pool-2-thread-46       [DEBUG] NOSESSION/- NOUI   [.impl.dataservice.folder.FolderTreeCache] - Received folder tree change-event from backend
    2017-11-20 09:46:59,713 pool-2-thread-50       [DEBUG] NOSESSION/- NOUI   [.impl.dataservice.folder.FolderTreeCache] - Received folder tree change-event from backend
    2017-11-20 09:46:59,713 pool-2-thread-50       [TRACE] AE_SYS1:1/USER123/TEST NOUI 0000000020553975  [ecc.framework.entrypoint.core.Entrypoint] - Event posted: com.uc4.ecc.backends.impl.dataservice.folder.FolderTreeCache$FolderTreeChangedEvent@219b80a
    2017-11-20 09:46:59,713 pool-2-thread-46       [TRACE] AE_SYS1:1/USER123/TEST NOUI 0000000020553975  [ion.explorer.ExplorerFolderTreePresenter] - FolderTreeChangedEvent received
    2017-11-20 09:46:59,713 pool-2-thread-50       [TRACE] AE_SYS1:1/USER123/TEST NOUI 0000000020553975  [ion.explorer.ExplorerFolderTreePresenter] - FolderTreeChangedEvent received
    2017-11-20 09:46:59,713 pool-2-thread-46       [TRACE] AE_SYS1:1/USER123/TEST NOUI 0000000020553975  [ecc.framework.entrypoint.core.Entrypoint] - Event posted: com.uc4.ecc.backends.impl.dataservice.folder.FolderTreeCache$FolderTreeChangedEvent@18f80928
    But thanks to its cache, the AWI effectively shields the connected users from the folder tree changes being processed behind the scenes. Even when many OFS kicks are being handled by the AWI server, the folder tree view in the Process Assembly perspective remains usable and responsive. This is a big improvement over the behavior of the JUI, which can be rendered virtually unusable by forced refreshes of the folder tree.


  • 25.  Poor AE performance following DB load of transport case

    Posted Nov 24, 2017 09:17 AM
    Just a quick notification to those of you considering using Zero Downtime Upgrade: ZDU compatibility mode may worsen the AE performance problem during deployments.

    This morning we used ZDU to begin the process of upgrading one of our systems to v12.0.3HF3. We are upgrading from v12.0.2 to fix the bug described in PRB00135332. We are currently running in compatibility mode, meaning only the DB initial data and one of the nodes has be upgraded. The other node is still running the v12.0.2 binaries.

    Since the start of compatibility mode this morning, every time we have run a deployment of a transport case file to this system, there has been a dramatic rise in WP utilization that has lasted for several minutes. So far, it has not caused any other problems, but it is something we are monitoring carefully. We expect that the problem will go away once we commit the upgrade to v12.0.3HF3.


  • 26.  Poor AE performance following DB load of transport case

    Posted Nov 25, 2017 03:51 AM

    Yesterday Automic confirmed my finding that when the AE is processing ITL entries following the DB load of a transport case, the AE sends one OFS kick to each connected user per running WP. They agreed that this is not how it should work, and opened PRB00216238 for the bug.



  • 27.  Poor AE performance following DB load of transport case

    Posted Dec 01, 2017 04:40 AM
    This bug will be fixed in Automation Engine 12.0.4, 12.1.1, and 12.2.0.


  • 28.  Poor AE performance following DB load of transport case

    Posted Jan 08, 2018 12:09 PM
    This problem ticket has been updated. I’ll include the details below.

    Fix description: An error has been fixed where the UI gets unresponsive after multiple loads of transport cases. For every transport case every WP triggered a folder-cache refresh for all users. This has to be done only by the primary.

    Release plan:
    • Automation Engine 12.2.0 - Planned      release date: 2018-06-19
    • Automation Engine 12.1.1  - Planned      release date: 2018-01-29
    • Automation Engine 12.0.5  - Planned      release date: 2018-05-07


  • 29.  Poor AE performance following DB load of transport case

    Posted Jan 12, 2018 09:27 AM


  • 30.  Poor AE performance following DB load of transport case

    Posted Feb 01, 2018 03:25 AM
    This problem is supposed be fixed in v12.1.1, released yesterday, 31 January 2018.


  • 31.  Poor AE performance following DB load of transport case

    Posted Feb 07, 2018 11:38 AM
    I can confirm that this is fixed in v12.1.1. The AE now sends exactly one OFS kick to each connected AWI/JUI/API client, per loaded transport case file.


  • 32.  Re: Poor AE performance following DB load of transport case

    Posted May 31, 2018 08:37 AM

    AE v12.0.5 was released recently. It is supposed to fix this bug. (I have not tested it yet.)



  • 33.  Re: Poor AE performance following DB load of transport case

    Posted Jun 19, 2018 11:28 AM

    Michael_Lowry wrote:

    AE v12.0.5 was released recently. It is supposed to fix this bug. (I have not tested it yet.)

    I confirmed that the bug is fixed in v12.0.5.



  • 34.  Re: Poor AE performance following DB load of transport case

    Posted Sep 03, 2018 11:34 AM

    According to the latest information we have from CA Support, this bug was not completely fixed in v12.0.5. A more complete fix is supposed to be included in v12.0.5 HF3.



  • 35.  Re: Poor AE performance following DB load of transport case

    Broadcom Employee
    Posted Sep 05, 2018 03:09 PM

    Hi Michael,

     

    as you might know version handling related to DB.Load will be handled by DWPs. Beside the bug you have mentioned you can speed up the processing by adding additional DWPs to the system which has worked fine in v12 and former versions. In the knowledge base you'll find an article explaining the behaviour - and how much ITLs will be processed per DWP/per interval.

     

    It is good practice to start additional DWPs before loading a transport case file into the system and turn them off or switch back to WP once the table does no longer contain ITL entries.

     

    regards,

    Peter



  • 36.  Re: Poor AE performance following DB load of transport case

    Posted Sep 06, 2018 05:57 AM

    We use an automated system to handle deployments. Any end user of the Automation Engine can trigger a deployment. I suppose we could add a step before and after the DB load:

    1. Increase the number of DWPs using ChangeServerMode 
    2. Run the DB load (ucybdbld)
    3. Decrease the number of DWPs using ChangeServerMode 

     

    We run 50 DWPs by default.



  • 37.  RE: Re: Poor AE performance following DB load of transport case

    Posted May 27, 2020 01:48 PM
    Edited by Jean-Francois Anctil May 28, 2020 12:47 PM
    Hi @Michael A. Lowry,
    I know it's an old post, but I am curious about how many users are connected to your environment to have 50 DWPs? I have followed the Automic's recommendations for sizing our environments (we only have 15 DWPs...), but the server become unresponsive (100% cpu) everytime I use the transport case a couple of times in a short span of time.

    regards,

    J-F


  • 38.  RE: Poor AE performance following DB load of transport case

    Posted Nov 30, 2023 08:57 AM

    Hi Michael,

    I would like to share an additional experience. the same wait log situation happened when I loaded a transportcase output from v12 to v21. this file had only user objects and usergroup objects in it. interestingly, when I changed the place I marked with yellow in the screenshot to 009 just like the v21 output and tried it, it loaded instantly. 

    have a nica day.



    ------------------------------
    Olgun Onur Ozmen
    https://www.linkedin.com/in/olgunonurozmen/
    ------------------------------