Automic Workload Automation

  • Private Community
Back to discussions
Expand all | Collapse all

So ... how DO you use curl in the most basic way with the REST API?

  • 1.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 16, 2018 07:32 AM
    Hi.

    I've seen many threads about this, but none where a simple example using the most basic command is given: curl.

    I do not want to use POSTMAN.

    How does one go about using the REST API with curl? Here's the most basic example I can think of:

    • something that is a GET request
    • doing it right on the AE server, using localhost

    This is what I get:

    $ curl -D- http://localhost:8088/ae/api/v1/100/executions/197661022 --basic -u '100/DEPT/USER'
    Enter host password for user '100/DEPT/USER':
    HTTP/1.1 401 Unauthorized
    Content-Type: application/json
    WWW-Authenticate: Basic realm="AUTOMIC"
    Content-Length: 134
    Server: Jetty(8.2.0.v20160908)

    {
      "code" : 45107,
      "error" : "Cannot logon to Automation Engine.",
      "details" : "The credentials (user/dept/pwd) are not valid."
    }
    (I substituted "DEPT" and "USER" for the actual data ONLY in the above post, for security reasons. On the command line, I tried my actual department and username, of course).

    I tried a gazillion combinations of client, department and user name but NONE accepts my correct, confirmed user password. At some point I managed to lock my account (it's on a "lock after five unsuccessful logins" policy), so it DID identify my correct account, but even though I unlocked it, it will never accept my known-good user password.

    I'm also an Administrator. I haz all the priveleges.

    What am I missing? Is my curl broken?

    Thanks!


  • 2.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 16, 2018 11:11 AM

    Your Case interested me so I tried it out on my end and when I entered the same parameters as you:

     

    $ curl -D- http://localhost:8088/ae/api/v1/10/executions/1162017 --basic -u '10/USER'

     

    I have got the same 401 unauthorized.

    But when I changed the Parameters to:

     

    curl -D- -u USER/10 http://localhost:8088/ae/api/v1/10/executions/1162017
    } "name" : "JOBS.***.***.TO.***.***",   "type" : "JOBS",    "run_id" : 1162017,   "status" : 1900,   "status_text" : "ENDED_OK - ended normally",   "runtime" : 5,   "activation_time" : "2018-03-06T15:38:54Z",   "start_time" : "2018-03-06T15:38:54Z",   "end_time" : "2018-03-06T15:38:59Z",   "agent" : "RA_FTP_AGENT_402",   "platform" : "CIT",   "parent" : 0,   "user" : "USER/10" }
    I hope this information is helpful for you.

     



  • 3.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 16, 2018 11:18 AM
    Hi,

    many thanks for trying that!

    However, still no luck on my end. Tried any of these:

    curl -D- -u 100/USR/DEPT http://localhost:8088/ae/api/v1/100/executions/197661022
    curl -D- -u USR/DEPT http://localhost:8088/ae/api/v1/100/executions/197661022
    curl -D- -u USR http://localhost:8088/ae/api/v1/100/executions/197661022
    curl -D- -u DEPT/USR http://localhost:8088/ae/api/v1/100/executions/197661022
    curl -D- -u 100/USR http://localhost:8088/ae/api/v1/100/executions/197661022

    ... and some more. All result in 401 :(


  • 4.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 16, 2018 11:52 AM
    Sorry for asking this but does it work from the browser? when you try to access that URL?


  • 5.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 19, 2018 06:24 AM
    Eisa Heinig said:
    Sorry for asking this but does it work from the browser? when you try to access that URL?
    Hey,

    Why sorry? I'm grateful for inspiration :)

    But no, doesn't work either. I tried it from IE, I get the BasicAuth prompt, but it won't deliver either.

    Cheers,
    Carsten


  • 6.  So ... how DO you use curl in the most basic way with the REST API?
    Best Answer

    Posted Mar 19, 2018 10:44 AM
    Found it, after spending about three hours on this ...

    It's the @ character. I had that as part of my UC4 user password, because the company policy demands a complex password.

    If your password has an @ character in it, the REST API does not authenticate.

    When I dumb down my password, this works:

    curl -D- --basic -u USER/DEPT http://localhost:8088/ae/api/v1/100/executions/197661022

    I opened INC00222026 for the "

    matter (and potentially other characters).

    I could open another ticket for a documentation bug with this I also found, but I can't be bothered. In all frankness, I back Alpha software on some platforms that appears to be cleaner :(

    One can prove beyond any doubt that this is a defect in Automic, not curl, by starting a netcat:

    jive_text_macro jive_macro_quote" __jive_macro_name="quote" modifiedtitle="true">nc -l -p 1234

    then do the curl against that:

    curl -D- --basic -u USER/DEPT http://localhost:1234/ae/api/v1/100/executions/197661022

    which results in:

    GET /ae/api/v1/100/executions/197661022 HTTP/1.1
    Authorization: Basic U0NITUlUWi9TQkM6R2VoZWltIzE3QA==
    User-Agent: curl/7.29.0
    Host: localhost:1234
    Accept: */*

    and U0NITUlUWi9TQkM6R2VoZWltIzE3QA== uudecodes into my test password "Geheim#17

    , complete with

    @" character - so it is beyond any doubt encoded and sent correctly by curl.

    Thanks to all who provided input!

    Best,
    Carsten



  • 7.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 21, 2018 03:47 AM
    Hi Carsten, 

    after our call yesterday, I tried to the REST API with an @ sign in the password. However I could not reproduce the error. My password is 123@ and the REST API behaves as expected.

    I used version Automic 12.1.1 for my test.

    This is my curl command:
    curl -D- --basic -u  'REST/API' -H "Content-Type: application/json" -X GET http://<myserver>:8088/ae/api/v1/1000/executions/2726251
    Enter host password for user 'REST/API':
    HTTP/1.1 200 OK
    Content-Type: application/json
    Content-Length: 337
    Server: Jetty(8.2.0.v20160908)

    {
      "name" : "JOBP.PROCESSING",
      "type" : "JOBP",
      "run_id" : 2726251,
      "status" : 1900,
      "status_text" : "ENDED_OK - ended normally",
      "runtime" : 27,
      "activation_time" : "2018-02-21T11:12:21Z",
      "start_time" : "2018-02-21T11:12:22Z",
      "end_time" : "2018-02-21T11:12:49Z",
      "parent" : 0,
      "user" : "THU/AUTOMIC"
    }[root@vviepmkub01 ~]#


    My curl version is this:
    curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
    Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
    Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets


    Could this be something environment-specifc?


  • 8.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 21, 2018 08:26 AM
    Hello Mr. Thullner,

    I'm quite certain I narrowed it down to the "

    character some days ago, but this morning I must admit I can't replicate that particular finding. So let's go back to the original password that started the narrowing-down process. This is my troublesome original (and obviously now former) UC4 password:

    jive_text_macro jive_macro_quote" __jive_macro_name="quote" modifiedtitle="true">]}(/~}0&={@L"A$Übel

    Both my colleague and I set this multiple times now, and it consistently fails for the REST API, but is otherwise a valid UC4 password for logging into the AWI or the old client. Today, it appears it's the "Ü" character that is troublesome (which would make sense, because it's an ASCII character >127, but an ASCII character and valid UC4 password character nonetheless).

    Please kindly try to set the above password exactly as-is on your end, see if it works or fails with the RESP API. I am very curious to see your mileage with this.

    Thanks, kind regards,
    Carsten Schmitz



  • 9.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 21, 2018 08:32 AM
    Hi, 

    I now set my password to ']}(/~}0&={@L"A$Übel'

    Now the strange thing starts: The password works when I use it with Postman, but not with CURL. I will have to dig deeper into this. 


  • 10.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 21, 2018 08:36 AM
    Now the strange thing starts: The password works when I use it with Postman, but not with CURL. I will have to dig deeper into this.

    Many thanks, much appreciated. If I can find out something on my end, I'll let you know.

    For what it's worth, I ran the "complex" password through the netcat test as well. It appears to uudecode correctly after passing through curl.



  • 11.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 21, 2018 08:41 AM
    Just a workaround-suggestion:

    So the combination is... one, two, three, four, five?
    That's the stupidest combination I've ever heard in my life! That's the kind of thing an idiot would have on his luggage!
    http://www.imdb.com/title/tt0094012/quotes

    cheers, Wolfgang


  • 12.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Mar 21, 2018 08:53 AM
    Wolfgang Brueckler said:
    That's the kind of thing an idiot would have on his luggage!
    True. Him, and the government of the United States of America.

    http://www.todayifoundout.com/index.php/2013/11/nearly-two-decades-nuclear-launch-code-minuteman-silos-united-states-00000000/

    But as I recently said in another thread: Just don't put your nuke's launch codes into Automic :p

    Best,
    Carsten


  • 13.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Apr 11, 2018 04:58 AM
    Robert Thullner said:
    Hi, 

    I now set my password to ']}(/~}0&={@L"A$Übel'

    Now the strange thing starts: The password works when I use it with Postman, but not with CURL. I will have to dig deeper into this. 
    Hi Robert,

    Were you (or anyone at Automic) able to make any progress on this yet?

    Thanks!


  • 14.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Apr 11, 2018 12:37 PM
    Hi, 

    I spent quite some time to get this working with curl (also with some of our consultants). However, no one was able to get the password working. So the conclusion was that this is some encoding issues on curl/Linux, as it works with Postman and in our tests. 

    We tried several options (HTML codes, ASCII codes, several escape options) instead of using the special characters but were not successful so far.

    Did you have any success on your side? 


  • 15.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Apr 12, 2018 03:50 AM
    Hi Robert,

    cheers for that, and no - sorry, no success on my side either. But since curl appears to send the password correctly, as can be demonstrated with netcat, I'm a bit at a loss here as to why this doesn't work. Pity, with curl being the most basic and go-to tool for these kinds of things.

    Unfortunately, I am very limited in my debugging because my company doesn't allow me to install any software that isn't specificially approved, otherwise I could possibly experiment with newer curl versions, build one from source with debug options, try curl on other platforms (e.g. Windows) to narrow it down further. Alas, I am prohibited from doing many of these things.

    Will Automic continue to look into this further? I do have a support ticket still open (INC00222026), not sure what to do with that one at this point, but I never had any such problems with curl and any other web service. Did your developers already take a look, maybe seeing whats really happening under the hood of the web service when this happens? If it's truly an encoding issue, it should be possible to pin it down there?

    Best regards,
    Carsten


  • 16.  So ... how DO you use curl in the most basic way with the REST API?

    Posted Apr 12, 2018 05:04 AM
    Took me a while to figure out how, but one can also show that the password is passed correctly without any encoding issues with tcpdump, thus going one step beyond netcat and demonstrating a correctly encoded password on the receiving server on the actual, same run of curl. On the receiving server:

    tcpdump -i eth0 -w - | strings | grep -i "Authorization: Basic"
    Authorization: Basic QVBJL1RFU1Q6XX0oL359MCY9e0BMIkEkw5xiZWw=
    ^C4634 packets captured
    4642 packets received by filter
    6 packets dropped by kernel


    QVBJL1RFU1Q6XX0oL359MCY9e0BMIkEkw5xiZWw= uudecodes into:

    API/TEST:]}(/~}0&={@L"A$Übel


    So no, sorry, I still can't see any pointers for this being an encoding issue. Thus, I kinda expect Automic to have their developers look into this. I don't have the source code of the web services, nor a debug environment for jetty, but looking there is the next logical step. Automic does have those ressources.

    Kind regards,
    Carsten


  • 17.  Re: So ... how DO you use curl in the most basic way with the REST API?

    Broadcom Employee
    Posted Apr 18, 2018 10:25 AM

    After consulting with our development team, the correct curl call for that password would be:

     

    curl -D- --basic -u testuser/test:$(echo ']}(/~}0&={@L"A$Übel' | iconv -f utf8 -t iso8859-1) http://localhost:8088/ae/api/v1/0100/executions/1007006

     

    The REST API expects the password to be encoded in the target encoding of the AWA server. The curl command above will convert it appropriately, so that the password can be used.