Automic Workload Automation

Did anyone else not have their schedules refresh at the period turnaround point?  We are on Automic Web Interface 12.2.1.

Same issue here. In our production client, the schedule never turned around at midnight. In lower environment it turned around at 11pm, but didn't process anything after midnight in either environment. 

Yes, we were hit with it too in all environments.  We are on v12.0.6.  All of our schedules turned around at 11pm and everything that was valid to run on Sundays took off immediately at 11pm.  The schedules were "finished" and had a the period turnaround next scheduled at 11pm Monday night.  After restarting the schedules, it either would cause processes to be skipped (if before the schedule was restarted) or duplicated (if it also ran at 11pm the night before).  

Hi

Is there any (official) Workaround to prevent this issue?

THX

Wolfgang

I'm creating a check in my environments to ensure that the schedule is submitting objects as intended. Currently scheduling a run once event that does nothing but create statistical entries at 5 past the hour in the schedule and running a recurring event at 10 past the hour in the client queue that checks to make sure the job has run. If not, it will send an alert to my group to refresh the schedule. I'm going to test further and may end up scripting it to automatically clear and submit a fresh schedule. 

It's not ideal, but I'm hoping it will provide a decent check and notify us if it comes up again.

Has anybody opened a ticket to support for the issue?

ThomasKelly613226 wrote:

 

Has anybody opened a ticket to support for the issue?

Yes, they are aware of the issue.

Quote from support:

"We are aware of this situation. To be on the safe side, please restart all your schedule objects. As soon as I have an update for you with a permanent fix I will let you know."

Thank you jkessans! What's the ticket number on that? I'm going to open one on our end as well and would like to reference that.

Ticket #01251723

This was actually opened last Fall as we noticed a time change bug then so I just added to it and bumped up the severity as it was a much bigger impact this time.

That's why this is my first time seeing it. We just upgraded from 10.0.6 to 12.2.1 in January/February. Thanks again!

We are on v12.0.6 and upgraded in September, 2018.  Sounds like it impacted multiple versions.

We're on 11.2.7 and preparing for an upgrade to 12.2 or 12.3 (depends on timing).  This would have been an absolute killer.  Please keep us informed on what support reports.  Thanks!

Good info guys, we had a similar issue.

CA - ideally someone from CA would jump into these threads and post the knowledge base ticket link that describes and acknowledges the issue so 100's customers don't need to log their own support tickets. Thoughts?

That would be wonderful. I'll give them until tomorrow before opening my own ticket and hope that happens. 

Hi

This is not a oracle specific issue, it's inherent to using timezones with DST, when going to an other offset there is always one hour (or in some cases 30 minutes) "gap" where the time moves forward ( = does not exist).

during that offset when Patch not been applied at OS level,JDBC  and DB  level we will notice bad data,which required to be deleted or updated. I have provided a KB below to start with.

Oracle support can provide the required information about the current patches available  to specific  named timezones as per the version of oracle installed on your Database.

After DST changes ensure below in the KB article:

Will the Brazil DST change in 2018 affect CA Spect - CA Knowledge 

https://comm.support.ca.com/kb/ump-ataglance-or-trend-report-fail-with-unknown-error/kb000044786

Regards,

Dhanunjay

Hi

I am afraid your KE relates to another product and is not related to the issue discussed here.

cheers, Wolfgang

Hi guys,

I got a bit more info from support today regarding the TZ Bug:

affected Versions

Automation.Engine 11.2.9 and above
Automation.Engine 12.0.6 and above
Automation.Engine 12.1.3 and above
Automation.Engine 12.2.1 and above

Affected will be every JSCH object which is active and which has a TZ set (directly or via Client TZ) and
when the TZ object has DSC change set.

[UPDATE]

Affected will be every running JSCH object which has a TZ set directly or via Client TZ and the TZ object has Daylight saving time change set not equal to the Period change time.

It will happen on the day before (!!) DSC.

If it did already happen a restart of the JSCH should help.

Automic is going (rumor) to send an official statement with more info on this.

Automic Case AE-20418 (internal)

cheers, Wolfgang

Received this today:

Automic: Daylight Saving Bug in JSCH

Dear Valued Customer,

A bug in the routine calculating the next period turnaround of Schedule Objects causes multiple or no executions of jobs after the day light saving switch.

The bug has been introduced in the following service packs per release:

Automation.Engine 11.2.9
Automation.Engine 12.0.6
Automation.Engine 12.1.3
Automation.Engine 12.2.1

Systems running on an earlier service pack of the same release are *NOT* affected. e.g.: AE 12.0.3 = not affected; AE 11.2.9 = affected

Any JSCH object in any system, independent from the system's local time zone can be affected. A Schedule Object is affected in case it has a time zone object defined which refers to a time zone in which a daylight saving switch is defined. Schedules with no time zone defined will fall back to the client's time zone. Hence, all Schedules within a client which is set to such a time zone are affected.

Daylight saving switches occur on several dates from early March to mid April depending on the country. The country the system is physically located doesn't affect the vulnerability. It's the setup of the TZ Object in Automic Automation.Engine only, which determines if the bug will occur or not.

An overview of countries with DST switch and when they are due can be found here:

http://www.webexhibits.org/daylightsaving/g.html
and here
https://www.timeanddate.com/time/dst/2019.html

A script has been created to identify affected JSCH objects. The script will also provide update statements (one per object) to fix the turnaround time, which will correct the situation.

In order to be able to import the objects, you need to have SQLVAR_INTERNAL set to YES in the UC_SYSTEM_SETTINGS.

The script needs to be run once a day as affected JSCH objects can be identified one day before the daylight saving switch only.

An e-mail address can be defined in the top of the script to have a notification sent out in case faulty JSCH objects are found.

In case the Script identifies affected JSCH Objects, please execute the provided 'update' statement to fix it.

The script will not perform the update statement automatically for safety reasons.

You can easily grab the entire report and strip the timestamp at the beginning of the lines using a text editor to execute all statements with the tool of your choice.

Please don't forget to perform a 'commit' at the end, in cases the database used requires it.

Details of the definitive fix and the script can be found in this Knowledge Base Article.

Sincerely,
Automation Team
CA Technologies, A Broadcom Company

Text of the KB:

DST: Jobs executing twice at turnover or at incorr - CA Knowledge

The generated update statement does not work for Oracle DBs. You have to change it like this:

update eh set eh_nextcheck = to_date('&correct_turnaround_utc#','YYYY-MM-DD HH24:MI:SS') where eh_ah_idnr = &runid#;"

It may also be necessary to add the where-condition

and eh_status < 1800

to the statement for the search for the JSCH. If a JSCH is not active, its field eh_nextcheck is empty. The Automic script then ends with an error.

ainda02: Who can correct the file attachment?

We try to test the fix but it doesn't seem to work reliably, at least for us: With a scheduler with the period change at midnight and a modified time zone TZ.CET the field eh_nextcheck should be updated to 23:00 yesterday. But I would expect 23:00 today. I opened case 01332071 earlier.

Has anyone successfully recreated the error and been able to apply the fix?

Although AE 11.2.6 hf1 should NOT be impacted as per the KB, I imported the script in my Production env (11.2.6 hf1) and it gave an error trying to run it:

U00020586 Runtime error in object 'CHECK_PERIOD_TURNAROUND.START', line '00022'. No valid timestamp '                     ' (YYYY-MM-DD HH:MM:SS)

why is that ?

Keld_Mollnitz: I guess you have a JSCH which is not active anymore, e.g. ENDED_OK, in your activities. Because of this you should add

and eh_status < 1800

to the where clause of the select statement.

OLD:

select eh_ah_idnr, eh_name, eh_nextcheck, OSA_StartTime, eh_client, eh_tz from eh, osa where eh_oh_idnr = osa_oh_idnr and eh_otype = 'JSCH'

 

NEW:

select eh_ah_idnr, eh_name, eh_nextcheck, OSA_StartTime, eh_client, eh_tz from eh, osa where eh_oh_idnr = osa_oh_idnr and eh_otype = 'JSCH' and eh_status < 1800

Hi,

does anyone have news on that. Is there a corrected version of the script?

Ich habe vom Support die Info, dass ein korrigiertes Script fertig ist, dass man noch die offizielle Ankündigung im Detail formuliert. Daher gehe ich davon aus, dass in Kürze eine offizielle Info kommen müsste.

The new version just arrived. Please have a look at

DST: Jobs executing twice at turnover or at incorrect time

But it has been exported from an AE version 12 which cannot be imported into an AE version 11.2

Okay, dann basteln wir mal alle schön am Sonntag vormittag an unseren Schedules rum. Und hoffen, dass am Montag noch alles heile ist. Ist ja schließlich FOKUS Tagung.

Hi Keld,

just replace the version in this line of the xml file with your own version string

<uc-export clientvers="12.1.5+low.build.10821">

Yes of course. Now it works. Why did I not think of that

Hm,

I tried to test the new script in our test environment, but when I start 2 new scheduler with a test timezone they always have a eh_nextcheck time of tomorrow 9 pm. See the database extract below.

And the script produces wrong output.

    2019-03-29 15:32:30 - U00020408 /*    **************************************************************************    */
    2019-03-29 15:32:30 - U00020408 /*    Client 100 - Schedule: JSCH.SCHELP.TEST    */
    2019-03-29 15:32:30 - U00020408 /*    Period turnaround seems to be wrong: 2019-03-30 01:00:00 (TZ.TEST) <> defined turnaround 2019-03-31 00:00:00 (TZ.TEST)    */
    2019-03-29 15:32:30 - U00020408 /*    timestamp should be: 2019-03-31 00:00:00 (TZ.TEST) => 2019-03-30 22:00:00 (UTC)    */
    2019-03-29 15:32:30 - U00020408 /*    to correct the running schedule, please execute:    */
    2019-03-29 15:32:30 - U00020408
    2019-03-29 15:32:30 - U00020408       update eh set eh_nextcheck = '2019-03-30 22:00:00' where eh_ah_idnr = <runid>;
    2019-03-29 15:32:30 - U00020408
    2019-03-29 15:32:30 - U00020408 /*    **************************************************************************    */

I set the test timezone object to have a DST switch this morning 2 am

The database entries for running schedulers look fine, so I assume/hope the script works.

Hi,

we had this problem yesterday.

All Scheduler started at Sunday 23:00 instead of Monday 00:01.

We restarted all Scheduler objects after 00:01. Will we have the issue again this evening?

We will update tomorrow from version 12.0.7 to 12.0.7HF1 which fixes this bug.

If someone of CA can answere these questions would be very nice.

Hey Tobias,

according to the Knowledge Base article it's yes but according to my tests a schedule restart doesn't solve the problem. You could check with this SQL  statement (sorry couldn't test the MSSQL statement but Oracle works)

-- MSSQL

select
  eh_ah_idnr, eh_name, eh_nextcheck, osa_starttime, eh_client, eh_tz
from
  eh, osa
where
  eh_oh_idnr = osa_oh_idnr
  and eh_status < 1800;

-- Oracle

select
  eh_ah_idnr, eh_name, to_char(eh_nextcheck,'YYYY-MM-DD HH24:MI:SS') "NEXTCHECK", to_char(osa_starttime,'YYYY-MM-DD HH24:MI:SS'), eh_client, eh_tz
from
  eh, osa
where
  eh_oh_idnr = osa_oh_idnr
  and eh_status < 1800;

If the error is fixed the "nextcheck" column should correspond to the schedulers period turn around date and time in utc.

For example. If the period turnaround is set to today 24:00:00 (midnight) the value of nextcheck should be "2019-04-01 22:00:00".

Hope that helps,

Matthias

Hi,

thanks for the answer.

I think this sql-statement has the same function as the script in the knowledgebase-entry.
Can you confirm this?

Because the script doesn't call errors now in our system.

Thanks for your help!

Yes, that is correct. It's pretty much the same except that it displays the time in the oracle version.