Automic Workload Automation

I have some scripts that create CSVs and then powershell reads them, creates a JSON body with them, and passes them to the AE REST API. It works great except every two or three runs it gives me the following error after it waits for 60 seconds:

Invoke-RestMethod : {
"code" : 45110,
"error" : "An Automation Engine internal error occured.",
"details" : "Wait for response of request 30 timed out after 60 seconds."
}
At E:\TIMSdev\core\scripts\callapi.ps1:140 char:11
+ $APIRUN = Invoke-RestMethod -Uri $RUNURL -Method POST -Body $JSON -He ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc
eption
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

The full API call is:

Invoke-RestMethod -Uri http://servername:8088/ae/api/v1/200/executions -Method POST -Body $JSON -Headers System.Collections.Hashtable

Any help would be greatly appreciated.

Hi,

did you do any kind of debugging? Did you check the input from the csv file?

I use the same powershell call for testing but with "-Method GET" and only manually. So maybe it's a time issue. You could try to put some kind of wait before or after the REST call.

Let me know if that worked.

Matthias

I did do a lot of debugging before I got to this point.  Initially I used the uc4REST plugin but was having an issue with orphaned connections by the user for the API connection.  With that method, none of the logout/disconnect API calls were ending the session, so I worked with support to reach this point with their built-in API. 

The JSON/CSV contents are passing fine.  Automic is seeing them and pulling them into the script and assigning as variables without issue.  

I'll add a wait to see how it goes, probably 5 seconds, between the call and the end of the loop it's in and between the end of the previous "write-host" command.

I added 5 second sleeps, still getting the time out.  If I use "GET" I receive an error for body-type, I'm assuming in regards to attaching a JSON body.

Invoke-RestMethod : Cannot send a content-body with this verb-type.
At E:\TIMSdev\Core\scripts\callapi.ps1:142 char:11
+ $APIRUN = Invoke-RestMethod -Uri $RUNURL -Method GET -Body $JSON -Hea ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-RestMethod], ProtocolViolationException
+ FullyQualifiedErrorId : System.Net.ProtocolViolationException,Microsoft.PowerShell.Commands.InvokeRestMethodComm
and

I tried using the UC4REST plugin as well, it was creating orphaned users, I'm good with getting this to work either way, using UC4REST or AE REST, if someone has had a successful example of either.

Hm,

okay GET requests don't have a body. You need to set the header as I explained here:

REST calls via powershell [solution inside] 

Pleas check the passed arguments from the csv file when the REST call fails. Also, did you count the lines in the csv file and the number of results you get? Do they match?

For a deeper analysis please provide the whole script.

Matthias

So I should try placing the JSON body contents within the Headers?  That seems contrary to the AE documentation for REST calls.  

My script is pretty complex, so that's partly why I didn't share, but we'll see if I can upload it all, might be multiple posts.  I redacted the UN and PW, but the rest is as-is in the environment.

inboundeventtrigger.bat(This calls the fileattributes.ps1 script):

(attached)

fileattributes.ps1(Creates the CSVs then calls the callapi.ps1):

(attached)

Callapi.ps1(Does the work):

(attached)

Functions.ps1(The GETTOKEN function isn't currently being used):

(attached)

Variables.ps1 file:

(attached)

EDIT: Attached the files instead of posting them.  Also, added a followup with some log info that is pertinent. 

Hi,

I'll have to check that script later. Unfortunately it's pretty busy today. By the way ... there is an attachment function hidden in the extended editor you should check out :-)

Matthias

I found this today in the CPsrv_log_005 log.  

20180803/080717.948 - 45 U00003407 Client connection '11953' from '172.16.32.142:56416' has logged off from the Server.
20180803/080720.823 - 2488 U00045098 Method 'POST', URL: 'http://dc01-aedev01:8088/ae/api/v1/200/executions'
20180803/080720.823 - 2488 U00045099 The server replied with following status: '401'
20180803/080720.823 - 2488 Jetty: Error Processing URI: /ae/api/v1/200/executions - (org.jboss.resteasy.spi.UnhandledException) RESTEASY003770: Response is committed, can't handle exception
20180803/080727.073 - 31 U00003406 Client connection '11954' from '172.16.32.142:56417' has logged on to the Server.
20180803/080727.073 - 45 U00003339 Server with the name 'DEV#WP008' is already running, the connection '/172.16.32.142:56417' will be closed.
20180803/080727.088 - 45 U00003472 Connection to Server '11954' has been closed.
20180803/080727.088 - 45 U00003407 Client connection '11954' from '172.16.32.142:56417' has logged off from the Server.

I have found some issues and resolved them, so I'm back to square one in most regards.  I'm able to call the workflow fine, but it's like the API can only handle 1 call, then the next one pretty close to always fails, then the 3rd call works, the 4th nearly always fails, and so on for every call.  I tried a 30 second wait between the calls, and a 1 minute wait, no difference.

I looked in the CP log and see this.

20180803/150247.393 - 36 U00045105 Log on of 'SVCTIMSDEV/NDBH' successful.
20180803/150252.159 - 41 U00045105 Log on of 'SVCTIMSDEV/NDBH' successful.
20180803/150257.268 - 42 U00045014 Exception 'java.util.concurrent.TimeoutException: "Wait for response of request 504 timed out after 60 seconds."' at 'com.automic.kernel.impl.CpDispatcherComponent.waitForResponse():102'.
20180803/150257.268 - 42 U00045098 Method 'POST', URL: 'http://dc01-aedev01:8088/ae/api/v1/200/executions'
20180803/150257.268 - 42 U00045099 The server replied with following status: '500'
20180803/150347.425 - 36 U00045014 Exception 'java.util.concurrent.TimeoutException: "Wait for response of request 508 timed out after 60 seconds."' at 'com.automic.kernel.impl.CpDispatcherComponent.waitForResponse():102'.
20180803/150347.425 - 36 U00045098 Method 'POST', URL: 'http://dc01-aedev01:8088/ae/api/v1/200/executions'
20180803/150347.425 - 36 U00045099 The server replied with following status: '500'
20180803/150352.191 - 41 U00045014 Exception 'java.util.concurrent.TimeoutException: "Wait for response of request 509 timed out after 60 seconds."' at 'com.automic.kernel.impl.CpDispatcherComponent.waitForResponse():102'.
20180803/150352.191 - 41 U00045098 Method 'POST', URL: 'http://dc01-aedev01:8088/ae/api/v1/200/executions'
20180803/150352.191 - 41 U00045099 The server replied with following status: '500'
20180803/150358.425 - 41 U00045105 Log on of 'SVCTIMSDEV/NDBH' successful.
20180803/150359.488 - 41 U00045098 Method 'POST', URL: 'http://dc01-aedev01:8088/ae/api/v1/200/executions'
20180803/150359.488 - 41 U00045099 The server replied with following status: '200'

I'm running low on ideas.

My current workaround is to make the API call twice, so the second one intentionally fails, allowing the 3rd, 5th, 7th, etc to succeed.  This is very inefficient and will definitely take too long as we get more files, creating more calls.

$APIRUN = Invoke-RestMethod -Uri $RUNURL -Method POST -Body $JSON -Headers $HEADERS $APIRUN | out-null

Sleep 5

$APIRUN = Invoke-RestMethod -Uri $RUNURL -Method POST -Body $JSON -Headers $HEADERS $APIRUN | out-null

Sleep 5