AutoSys Workload Automation

We are currently seeking help on the autosys issue described below, so that we can resume running our autosys batch jobs without manual intervention.

On April 2018, an AIX 5.2 WPAR was installed on AIX 7.1.
Since the hardware virtualization application batch jobs scheduled in autosys are hanging when a new container is invoked to run. The problem is intermittent and jobs resume running when resubmitted manually.

Software Versions
AIX 5.2 Versioned WPAR Runtime 1.1.2.200
AIX 7.1 TL4 SP4 on the AIX host.
Autosys 4.5
Oracle 8.1.7.4.0

The Problem

In log file SSAIX02:/autosysprd01/autotree/autouser/out/event_demon.*** the following Oracle error was found:

May 24th (Autsys Log)
[03:53:48.0979] [1] -- ORACLE error --
[03:53:48.0979] [1] ORA-01115: IO error reading block from file 5 (block # 2466) ORA-01110: data file 5: '/oradataprd03/PRD02/data01.dbf' ORA-27091: skgfqio: unable to queue I/O IBM AIX RISC System/6000 Error: 5: I/O error
[03:53:48.0980] [1] Processing OCI function not used (4)

Steps taken to resolve the issue.......

No errors in the Oracle alert log. Database review shows no issues with the database.

DB verify was clean.

DBVERIFY: Release 8.1.7.4.0 - Production on Thu May 24 11:58:54 2018

(c) Copyright 2000 Oracle Corporation. All rights reserved.

DBVERIFY - Verification starting : FILE = /oradataprd03/PRD02/data01.dbf

DBVERIFY - Verification complete

Total Pages Examined : 38400
Total Pages Processed (Data) : 20980
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing (Index): 0
Total Pages Processed (Other): 38
Total Pages Empty : 17382
Total Pages Marked Corrupt : 0
Total Pages Influx : 0

Disk controller not reporting any errors,

Referenced Oracle Knowledge Article ( Doc ID 559697.1 ) : ORA-01115 ORA-01110 ORA-27091 ORA-27072 Error: 5: I/O error

Increased maxreqs to a value greater than or equal to 8192. (NOTE: Values that fixed the errors: 8192, 16384 or 32768.)
The "MAXREQS" value currently set at 4096 was changed to 16384 and the server rebooted to take effect.
The Oracle database and the autosys software was restarted as part of the normal reboot procedure.

Moved the datafile to a different filesystem.

Despite the aforementioned troubleshooting the problem still exists (same error message).

All recommendations with regards to this issue are greatly appreciated

Hello Audrey

Still using Autosys 4.5?

Is there any plan to move to latest versions for AIX, Oracle and Autosys?

As you know Autosys does not perform any direct IO operation to the database but uses an Oracle client with OCI calls.

These Oracle errors are returned to Autosys from the Oracle dataserver through the Oracle client

As this problem is related to IO errors, I would suggest to check this Oracle document to see if it helps

ORACLE-BASE - Direct and Asynchronous I/O 

And on AIX with Sybase we had this requirement, not sure if it might help with Oracle:

When installing an Autosys 4.51 Agent on the AIX platform, 'Asynchronous IO' is an essential prerequisite.
You will need to configure "AIO" and Not POSIX AIO with smit.

Please note AutoSys does not require AIO. AIO is required by Sybase.

Regards

Jean Paul

Autosys 4.5 and Oracle 8? Did I read that properly?

Honestly, please upgrade. Your OS and your DBA will thank you.

Talk to the DBA IO errors have to deal with the database file system and the schema setup.

Good luck...

Steve C.

Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.

Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.

As much as I would like to there is no plan to upgrade AIX, Oracle and/or Autosys.

Need to maintain the application as is until it can be replaced.

AIO is enabled

$ lsdev | grep aio0
aio0       Available  Asynchronous I/O (Legacy)

Thanks for the suggestions.  Will follow up on them.

Audrey

Hi Audrey

I did some search in Google and I found the following community link

https://community.oracle.com/thread/2187707

The problem has been resolved with the Oracle patch (metalink 390656.1)

Maybe this can help.

Thanks and Best regards,

Faouzia

Audrey,

Please spare a moment to update this thread with the status. Were you able to find the cause of the error and address?

Thanks,

Chandru