Automic Workload Automation

Expand all | Collapse all

PREP_PROCESS_REPORT only working on DB reports by design?

  • 1.  PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 13, 2019 07:59 AM

    Hi.

     

    Suppose I do this in a JOBS:

     

    "Process" tab:

    echo "Flamingo"
    echo "Chicken"
    echo "Flamingo Chick"

     

    "Post-Process" tab:

    :SET &HND# = PREP_PROCESS_REPORT("JOBS",, "REP", "*lamin*")
    :PROCESS &HND#
    : SET &RET# = GET_PROCESS_LINE(&HND#)
    : PRINT "from report: &RET#"
    :ENDPROCESS

     

    Then my post process report looks like this:

     

    2019-05-13 13:48:45 - U00020408 from report: Flamingo
    2019-05-13 13:48:45 - U00020408 from report: Flamingo Chick

     

    So far, so good.

     

    Now when I switch the JOBS to store its runtime report NOT in the database, but in a file (via the checkboxes in the UNIX tab), this immediately fails. &HND# will not contain a valid file handle for PREP_PROCESS_REPORT to read out of.

     

    I heard someone(tm) say this is by design, but can't find any written evidence of that.

     

    Does anyone know if this is indeed by design, and if so, is it written anywhere? Or should rectifying this go into an "idea" ...?

     

    p.s. if it is by design, this is dumb. I told a bunch of people to store their excessively long reports in the file system instead of shoveling jiggabytes into the database and potentially causing muchos problemos in the process, but this immediately backfired because of this nugget 

     

    p.p.s. sadly, the users can not use a simple output scan instead

     

    p.p.p.s. the 11.2 documentation says:

     

    " PREP_PROCESS_REPORT

    Comments

    This script function provides the report contents of executable objects for further processing with AE Script. The report is read from the AE database and prepared as data sequence.

     

    But this is not really conclusive and still dumb, and the 12.2 documentation contains no such thing.

     

    Thanks!



  • 2.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 13, 2019 02:57 PM

    The 11.2 documentation also says "The report is read from the AE database...".  Seems pretty explicit to me.

     

    We require our applications to write voluminous reports (or csv files, or whatever) as a separate file onto a file-server, rather than letting them be stored on our application servers.    We usually have UC4 passing the target folder/file specification as a parameter so that the file name can include the UC4 runid, which has turned out to be quite useful for us.



  • 3.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 13, 2019 02:58 PM

    And when UC4 is passing the folder/file name, it can then use PREP_PROCESS_FILE() against that output. 



  • 4.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 13, 2019 03:00 PM

    We also use the REGISTER_OUTPUTFILE() function to logically attach the output file to the UC4 statistics entry.  Then you can search the report with an OUTPUT_SCAN too if you want.



  • 5.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 14, 2019 04:12 AM

    Pete,

     

    Cheers for the hints, I'll look into REGISTER_OUTPUTFILE.

     

    Either way, our users voiced they will not change their scripts, and I have no authority to enforce that. They have reverted to putting loads of data into the DB, despite load problems. I have to somewhat agree with their reasoning: It should not be neccessary to change the scripts to totally different commands when changing the report storage option. The report retrieval is transparent when you open a report from AWA, it should be transparent when using scripting, too.

     

    The report is read from the AE database...".  Seems pretty explicit to me.


    Yeah ... that may be so, but they exorcised this from the 12.x docs, so it's not canon anymore.

     

    Br,



  • 6.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 13, 2019 04:14 PM

    Hi

    Job - WIN tab:

    Database

    The job report is handled by AE. After a job has been processed, the Automation Engine takes the process protocol which has been stored on the agent computer and stores it to the AE database. Doing so is only possible if a protocol can be created. It is not possible if a command has been used.

    File

    The job report is handled by AE. After the job execution, the process protocol is available as a file in the target system. This only works if a protocol can be created. It is not possible if a command has been used.

    Automic 

     

    If the Job report is not to be stored into the automic DB because of the "save report in DB checkbox", its on the agent only (or nowhere, because not created). So where should AE get its data from?

     

    This behavior has always been this way (could be a sentence from CA DEV :-)   ).

     

    A possible workaround would be "On error only".

     

    cheers, Wolfgang



  • 7.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 14, 2019 04:21 AM

    Wolfgang old (methaphorically) friend!

     

    So where should AE get its data from

     

    Where does it get it from when displaying a report stored on the agent? It manages to get the matching report for displaying it in AWA with no problems whatsoever, so why can't it call the exact same existing "sub-routine" to fetch it when running PREP_PROCESS_REPORT?

     

    A possible workaround would be "On error only".

    Sadly, they can't do that due to their logic. They want to parse the output for number of files processed, then do arithmetic on that, then set status codes based on that.

     

    Cheers,

    Carsten



  • 8.  Re: PREP_PROCESS_REPORT only working on DB reports by design?



  • 9.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 14, 2019 06:54 AM

    Wolfgang old (methaphorically) friend!

    Sometime I feel like an UC4 dinosaur...

     

    Where does it get it from when displaying a report stored on the agent?

    => Job messenger sends the Report to the AE (=DB)

     

    Usually (simplified)

    user presses "run", AE generates JCL (a batch or SH file), transfers it to the agent, the agent executes it and stores the report in a textfile, job messenger sends the TXT file and the status  it back to AE (=DB).

     

    Hope I didnot get you wrong...

     

    They want to parse the output for number of files processed, then do arithmetic on that, then set status codes based on that.

    What about using means of OS to count the files in your report-file and use REGISTER_VARIABLE ?

    Its a hard way but it might work....

     

    cheers, Wolfgang



  • 10.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 14, 2019 07:41 AM

    Job messenger sends the Report to the AE (=DB)

    I highly doubt the "=DB" part for when your job is told to store it in the agent file system exclusively

     

    Run a simple job and store the report on the agent file system. Open report in AWI. It will display it, and I highly doubt it gets pushed into the DB first when you merely display it in AWI. And even it it would, that would not change anything. Quite on the contrary, if it were that way, then there would already be code in AWA to pull it back into the DB (and there would already be useless redundancy), so even less reason not to have PREP_PROCESS_REPORT operate on it.

     

    Bottom line, AWA can locate the proper report for a given job run even when it's in the file system on an agent machine, so there is, to me, no reason not to make PREP_PROCESS_REPORT work the same way, no matter where the report physically is.

     

    What about using means of OS to count the files in your report-file and use REGISTER_VARIABLE ?

    Thanks. But that was in fact my very first suggestion to them (as you probably know, I'm always advocating to do stuff in shell script instead of AutomicScript ).

    But they don't want to change their jobs, and I have no authority (nor desire) to force this, nor dabble in their job objects. So any proposals for alternative coding solutions are unfortunately moot.

    I would agree that this may seem ridiculous at first, but this is how my company wants it to be: ~60 part-time Automic "users" all creating and running their own jobs (and at some point, Automic has also sold their interfaces as facilitating exactly that) and I have no final authority over what they create.

     

    And as I pointed out repeatedly: While I tell them when I think something doesn't make sense in their jobs, in this case I have to agree with them. There is no good reason why a different storage location should neccessitate a different script function.

     

    I think Automic should facilitate that, so yeah, the idea is there

     

    Cheers,

    Carsten



  • 11.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 14, 2019 12:57 PM

    If it is decided that you will be required to store very large reports in the AE database, then you'll also need to study the MAX_REPORT_SIZE setting inside of the UC_HOSTCHAR_DEFAULT variable.  This setting restricts what the largest size of report will be accepted into the AE database.  When this limit is exceeded, the report is left on the application server and it will not be imported into the AE database.

     

    There is always an upper limit.  This products primary mission is to schedule and run tasks, not to be a report repository.



  • 12.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 15, 2019 04:20 AM

    Cheers, will keep it in mind. And yes, totally agree on the "Automic is not a document management server" but alas, again, the powers that be direct what the powers that be direct.

     

    It's also somewhat sad that one can not freely use MAX_REPORT_SIZE as intended. We recently discovered that PREP_PROCESS_XML also uses MAX_REPORT_SIZE for it's limit, hence we had to increase MAX_REPORT_SIZE for a user who demanded to be able to process large XML files with Automic . The dual role of that variable is, however, officially declared as "as designed".



  • 13.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted May 15, 2019 04:23 AM

    Oh, but the good news is, MAX_REPORT_SIZE is also a UC_HOSTCHAR-capable variable. So one can set it different for certain hosts, so that may be a way arround some issues.



  • 14.  Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Broadcom Employee
    Posted May 16, 2019 03:32 PM

    maybe i can add some facts here:

     

    1) prep-process-report is AE (wp) function and using SQL to read data from the DB

    2) a job messenger is not sending any report 

    2.1) the report is sent by the agent - if store in db is used

    2.2) the report transfer is triggered by the AE (wp) 

    3) because of 1) this will only work if 2.1) is used

    4) prep-process-file is exactly the script function what is doing the expected (working with file content on agents)

    4.1) but keep in mind if the data is stored on the disk - someone must remove these files

    4.2) i can remember lots of customer tickets in the past - where this kind of logic filled the disk in a very short time...

    5) the UI/AWI is able to display the data because of the feature to look into running job outputs

    5.1) but this is not touching the AE (wp) at all

    5.2) this simple get-file function is going from the UI/AWI to the connected CP and is routed from that CP to the CP where the agent is connected and requesting a file - this file is then passed back from the agent through the CP's to the UI to display the content

    5.3) as mentioned before this feature was implemented that a user can have a look inside the output of long-running tasks - where/what is within the output - no need to login to this box and look into the report in question.

    6) if the report is stored in file and a prep-process-report would do the transfer implicit what is the benefit -> still the data must be transferred to the DB and if all jobs have this command it makes no difference if the flag is set or not - data is stored in DB for processing

    7) the only thing that might make sense is to enhance the prep-process-report with a flag - if the report is not within DB use prep-process-file - but this can already be achieved today with a simple if - do the report function or the file function

    7.1) but still the data is then transferred from the agent to the DB again for further processing

    7.2) this logic will only reduce the data to be transferred - but data is still written to DB

    7.3) as written here counts are searched - so data will always be transferred - the benefit would not be too big...

    7.4) if just counts are needed - use the  :REGISTER-VARIABLE function and no need to transfer any report

    8) but the file version might lead than very often into 4.2)

    9) 4.2) leads in a complete outage of the app server

    ...

     

    hope this helps and can be passed on to your uc4-scripting developers

    holli



  • 15.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 03, 2019 08:39 AM
    Edited by Carsten Schmitz Jul 03, 2019 08:48 AM

    Hi Holli,

    Many thanks for the elaborate explaination. Some remarks ... and sorry for the late reply, this fell victim to the forum migration for some time.

    And right at the start: I'm not sure if this can be sufficiently explained here. If you want, and have the the time, maybe we could have a quick call about this.


    > 4.1) but keep in mind if the data is stored on the disk - someone must remove these files

    Yes, we know, we have "find -mtime -delete" jobs on most servers for that reason for the "out" directory



    > 7.4) if just counts are needed - use the  :REGISTER-VARIABLE function and no need to transfer any report

    Yeah ... frankly, this whole situation is silly. The colleagues (who now filed "official" support tickets to me to get this addressed in Automic, so I have to deal with this), they could possibly use :REGISTER_VARIABLE. But for that they'd have to write shell code to do their output analysis and determine the numeric value to be fed to :REGISTER_VALUE. If they'd do that, if I had authority to tell them to change their job, it wouldn't need :REGISTER_VALUE. They could do the same output analysis in shell and do an "exit 1" and the job would be "ENDED_ERROR" and all would be fine. It would probably also well within their powers to truncate the output to a level where I'd not have to tell them to stop trashing the DB with useless output, but they don't want to do that either.

    They have this job that uses PREP_PROCESS_REPORT in a bunch of convoluted INCLUDE blocks because they know Automic Script to that extend and this was historically the obvious way to achieve the task at hand for someone, and they don't want to change the structure of the job.

    And yes, that is silly and as I type this and have to find a solution that works for everyone, I seriously wonder where my life went wrong. Normally I'd go to some manager and tell them to change their job to use :REGISTER_VARIABLE or somehow make it better and not dump noisy-looking stdout into the database, but there's one problem with this is: they do have a point with asking: If the admin tells us to store the report in the file system instead, why does PREP_PROCESS_REPORT suddenly stop working?

    The problem with PREP_PROCESS_FILE is: How do they even know which file they need to process? The name seems to be randomly chosen, it's something like (example) OAUQPMCX.TXT. Is it in any way deterministic? Is there any script function to determine that name? If so, I can probably convince them to modify their scripts to use PREP_PROCESS_FILE.

    As for the ideation idea, I am keeping that up for the moment. I still believe PREP_PROCESS_REPORT should be implemented in a way so it doesn't care where the underlying storage location is. And sorry, I still maintain that with 5.2 already implemented (and not only working for running jobs, but for ended ones as well), I still fail to see why such mechanism can't be used to make PREP_PROCESS_REPORT storage-location-agnostic, by fetching the file from the agent and NOT storing it in the db but processing it in RAM.

    But maybe I am also missing something here, I'm no expert on Automic Script.


    > 8) but the file version might lead than very often into 4.2)

    This really is a non-argument. Any Automic installation like ours that has been given to ~60 people to do with pretty much as they please *) will have experienced hard disks that fill up due to accumulating FS reports, and taken precautions to wipe old reports from disk ...


    *) I don't like it either but that's how OBI has decided it has to be.

    Best,
    Carsten




  • 16.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 03, 2019 11:55 AM
    You said: "The name seems to be randomly chosen, it's something like (example) OAUQPMCX.TXT."  Someone on these forums figured out where the file name comes from.  There is a formula that derives the file name based upon the RUNID (And I have no idea why they didn't just use the RUNID.)  I don't have that information handy though.   And this vendor has never shared that translation information with us customers either, which means they could change it at will.

    I can't imagine having dozens of people allowed to write their own UC4 solutions without a set of best practices to follow, and no one with the authority to guide/control them.  In a few years the environment could become so costly to manage, that management could start looking for a replacement product.  (I've seen this before... blame the tool for poor management.)





  • 17.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?
    Best Answer

    Posted Jul 04, 2019 01:51 AM
    Did someone say ALPHA2RUNNR / RUNNR2ALPHA? You can use These AE script elements to convert between RunID and it's string-notation.The logic behind however is not explained.

    ------------------------------
    Head of chief line manager senior hot air warmer
    Swisscom
    ------------------------------



  • 18.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 04, 2019 04:05 AM
    ​Hi Joel,

    Many thanks! RUNNR2ALPHA is exactly what I was looking for, I will propose it to the people who maintain that job.

    As for the logic behind it never being explained ... hmmm I wonder: A ten digit run id maxes out at 9.999.999.999, or ten billion minus one. A seven digit alphabetical code, wouldn't that be 26e7 ? Which, if I calculate correctly, maxes out at 8.031.810.176, or eight billion and then some?

    I can see absolutely no way how this can ever go wrong :)


  • 19.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 04, 2019 08:38 AM
    Based on my calc.exe, 26^7 is 10'460'353'203 - all good!

    ------------------------------
    Head of chief line manager senior hot air warmer
    Swisscom
    ------------------------------



  • 20.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 05, 2019 07:26 AM
    ​Are you sure about that?

    Various mobile phones, calculators (including Windows 10 calc.exe) and my Excel all say it's 8.031.810.176.

    26
    676
    17576
    456976
    11881376
    308915776
    8031810176

    :)




  • 21.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 05, 2019 08:16 AM
    Edited by Carsten Schmitz Jul 05, 2019 08:20 AM
    ​Maximum run id is actually NOT the highest ten digit number, but it's 2^31-1, or 21474483647. So there is no implied hash collision after all.

    Kudos to @Michael A. Lowry for the insider info and figuring that one out!



  • 22.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 05, 2019 08:21 AM
    Edited by Michael A. Lowry Jul 05, 2019 08:37 AM
    You can read about investigation into the relationship between task run IDs and job report IDs.
    See also this script (developed by a friend of mine) that can Convert between task run ID & job report ID. (In case you want to do this outside of AE scripting.)


  • 23.  RE: Re: PREP_PROCESS_REPORT only working on DB reports by design?

    Posted Jul 05, 2019 08:23 AM
    Edited by Michael A. Lowry Jul 05, 2019 08:37 AM