Automic Workload Automation

Expand all | Collapse all

How to avoid build up of .TXT files on agent host

John O

John O'MullaneJul 22, 2016 04:06 AM

  • 1.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 03:37 AM
    Hi all,
    Today if a Job runs and it's report is greater that x MB's it is partially saved to our Database and the remainder stored in a file on the temp folder of our Unix host.
    This may casue problems for disk space on my host so I want the option of not saving this .TXT file on my Unix host but instead deleting it if it get's large like this.

    I have previously increases the max report size in the UC_HOSTCHAR_DEFAULT variable but I don't see an option of not keeping the .TXT file.

    Can anyone help?

    John.


  • 2.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 04:48 AM
    Hi John,

    This can be set at the job's level. Just ensure this box is unchecked:
    vawyiad81y87.pnghttps://us.v-cdn.net/5019921/uploads/editor/yj/vawyiad81y87.png" width="724">

    Should the issue still persist, then it would probably mean that the report was not transferred back to the DB when the job ended, either because of its size or because of a failure during the job's processing.

    In which case, you may find out which RunID caused the problem with the script located here.


  • 3.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 05:24 AM
    Thanks Antoine.
    Yes, I am aware of that checkbox which is unchecked in all my Job.

    You are correct, it only happens when the report is too large to write to DB.

    However, I want to see if there is a way of preventing this from happening in the first place. I was hoping these was some setting, either in the .ini or some Variable that would not create these .TXT files when such a scenarion occurs.


  • 4.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 05:43 AM
    Hi John,

    I'm afraid there is no way of preventing it as reports are always written in the /temp directory of the agents before being sent back to the DB.
    They will be deleted after being sent to the DB if the "File" box is unchecked, but only if there was no issue during the job's processing.


  • 5.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 09:47 AM
    One other direction to go is to modify the jobs to only generate on failure or to never generate at all if you're not interested in the output. 

    With neither selected, it should not ever create the output. This is one thing that can be helpful for db size if you have jobs that run often but don't care about the output at all (filewatchers, size checks, etc). 


  • 6.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 11:06 AM
    I do want to keep all my reports but I'm happy to delete those that are too large to get saved in DB if it means a stable agent host.


  • 7.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 12:56 PM
    We keep our oversized reports for a few days and then purge them, just in case there is something of value in them.  I do this manually.  This also gives me the personal incentive to work with the software staff to fix what is generating the oversized reports, so it doesn't happen very often anymore.

    You can query the MELD table where meld_msgnr = 29007 to find these too.  I have a solution set up that sends an email to my production control staff when one of these messages occur so we can stay on top of the issue.


  • 8.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 01:47 PM
    Another route if you want to keep them would be to setup a filter to look for the U0029007 in the Activation Report and kick off a job to notify and/or move them to a stable location for review. 


  • 9.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 02:05 PM
    We ran a script every day that went to our ./temp directory and deleted anything older than 14 days.


  • 10.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 04:59 PM
    thanks all. I do have a job that cleans up any TXT files remaining after 5 days but I'm worried that a Job which runs as frequently as every 10 minutes could fill up the disk space within a couple of hours if it's report is very large for whatever reason.

    I like the idea of "look for the U0029007 in the Activation Report"  and also the query on the MELD table.

    What does meld_msgnr = 29007 actually look for, oversized reports?




  • 11.  How to avoid build up of .TXT files on agent host

    Posted Jul 19, 2016 05:22 PM
    It occurs when a TXT report file is left in the temp folder.

    And it is documented here;
    http://docs.automic.com/documentation/AE/11.2/english/AE_MESSAGES/Messages.htm#290xx.htm

    It would have been useful if this message had included the runid of the object that the report belongs to, but since it is missing, it then becomes useful to either learn how to convert the report file name into a runid (which Antoine posted a link to above) or do what I do, which is go back into the message log history via System Overview, find the 29007 message, and near it you will also find the "ended normally" message for the object in question.



  • 12.  How to avoid build up of .TXT files on agent host

    Posted Jul 20, 2016 10:52 AM
    Another option could be to use an event to monitor the space.  I think you can do it on a particular directory although I've never tried this.  If the event is satisfied - space is high - you could simply automatically run your script to clean up the TXT files.  With no intervention required.


  • 13.  How to avoid build up of .TXT files on agent host

    Posted Jul 20, 2016 11:23 AM
    Laura has some cool ideas for automatically cleaning up the temp folder.  I particularly like the event idea so that it can react automatically to heavy workload activity which could occur more than once per shift.  Philosophically though, I felt that doing automated cleanup of these folders would mask the root cause of the problem which is applications that are writing ridiculously large reports that no one will ever use.   If we had not been requiring our staff to fix these offenders, we'd probably be creating oversized reports several times a day.  But since we have taken corrective action we now have maybe two or three a year, and our UC4 database is smaller.

    I also realize that it can be difficult to get applications staff to respond to requests from operations staff for what they perceive to be low priority issues like this one.   I ran into resistance in several cases where the application was writing a debug-report in production and the support staff insisted they needed the debug information 'just in case' of a production failure.  It required management intervention to force them to stop printing debug reports in production.


  • 14.  How to avoid build up of .TXT files on agent host

    Posted Jul 21, 2016 08:27 AM
    Good point Pete.  I guess that could be addressed by keeping track of the # of times the process had to be run and if it was frequent - then send out an informational email so that further investigation could be done.


  • 15.  How to avoid build up of .TXT files on agent host

    Posted Jul 22, 2016 04:06 AM
    thanks to everyone for your help on this...


  • 16.  How to avoid build up of .TXT files on agent host

    Posted Jul 22, 2016 04:12 AM
    petwir Is there a link between MELD and the actual Object name?
    I see there is a MELD_IDNR column but not sure if this is the link back to the Object.

    i see loads of entries in my MELD table where meld_msgnr = 29007 but want to link to Job if possible.


  • 17.  How to avoid build up of .TXT files on agent host

    Posted Jul 22, 2016 11:16 AM
    The database provides no linkage from MELD.  In the case of 29007, one could extract the file name (e.g. OAACIQEC.TXT) and translate it into a RUNID.   UC4 scripting language provides a function for doing so called ALPHA2RUNNR;
    http://docs.automic.com/documentation/AE/11.2/english/AE_WEBHELP/help.htm?product=awa#ucabhz.htm?Highlight=alpha2runnr

    Also Michael_Lowry has posted some more detailed information about this translation and has provided alternative coding solutions;
    https://community.automic.com/discussion/6669/convert-between-task-run-id-job-report-id



  • 18.  How to avoid build up of .TXT files on agent host

    Posted Jul 22, 2016 11:36 AM
    We set up a process that automatically archives to a large NFS file system all agent job log files older than 5 days. We run this script every day via cron on all servers where the UC4 agent is installed. The logs are compressed (gzip), and categorized in sub-directories on the NFS file system, based on the name of the server and the user who ran the job.
    #!/bin/bash
    # compress job logs older than 5 days and move to a /net share
    logger -p info "compressing job logs greater than 5 days old and moving to /net share"
    serverdir=$(hostname)
    find /var/uc4/agent/out -mtime +5 -type f| while read x
    do
    techowner=$(ls -l $x|cut -d " " -f3)
    # see if requires zipping
    if [[ "$x" != *.gz ]]
       then
    # zip it up
       logger -p info "zipping...   "$x
       gzip $x
    # reference the zipped name
       x=$x".gz"
       fi
    # ok now look for directory to put it in and create if not there
    dir=/net/hnfsuc4logs/appshare/uc4logs/$serverdir/$techowner

    if [[ ! -d $dir ]]
        then
        logger -p info "creating directory:" $dir
        mkdir -p $dir
        if [[ $? -ne 0 ]]
           then
           logger -p info "error creating directory : " $dir " Exitting immediately"
           exit 99
           fi
        chmod 755 $dir
        fi
    # directory created or we have exitted
    logger -p info "moving file : "$x" to directory : "$dir
    mv $x $dir
    chmod 644 $dir/$(basename $x)
    done
    exit 0
    We are in the process of setting up something similar on Windows agents.


  • 19.  How to avoid build up of .TXT files on agent host

    Posted Jul 23, 2016 08:29 PM
    I also built this SQLServer database query to help me find those jobs that are generating very large reports, but still not exceeding MAX_REPORT_SIZE;

    select oh_name as job_name      , dateadd(hour, datediff(hour, getutcdate(), getdate()), ah_TimeStamp2) as Start_time      , CONVERT(varchar, DATEADD(ms, ah_runtime * 1000, 0), 114) as runtime      , count(*) as report_size from uc4.dbo.rt    , uc4.dbo.ah    , uc4.dbo.oh where ah_timestamp1 > cast('20160715 00:00:00:000' as DATETIME) -- how far back in time and   ah_oh_idnr = oh_idnr and   rt_ah_idnr = ah_idnr and   not oh_name in ('APPUTILP', 'AE_PROD#WP001') -- Objects to exclude from the selection group by oh_name, ah_timestamp1, ah_timestamp2, ah_runtime having count(*) > 500 -- MAX_REPORT_SIZE is set in UC_HOSTCHAR_DEFAULT order by 2;


  • 20.  How to avoid build up of .TXT files on agent host

    Posted Jul 25, 2016 03:27 AM
    petwir we have our MAX_REPORT_SIZE at 240, I even think I doubled this recently from 120.
    Looks like you have your at 500. 
    Any reason you picked this value?