Symantec Access Management

Expand all | Collapse all

Cannot start LLAWP on RHEL7/Apache 2.4

  • 1.  Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 07, 2017 10:01 AM

    I have many instances of CA Siteminder running on RHEL6 with Apache 2.2 without any issue at all. I have a documented process I follow and it all works great. This is my first deployment on RHEL7 with Apache 2.4 and I simply cannot get the LLAWP service to start.

     

    The error I am getting from apache's error_log is:

    CSmLowLevelAgent: No such file or directory
    [07/Feb/2017:09:32:29] [Error] SiteMinder Agent
    Failed to Start the LLAWP process.
    LowLevelAgent.LLAWPExec.Failed (Invalid argument)
    sh: nm: command not found
    sh: grep: command not found
    sh: awk: command not found
    sh: awk: command not found
    [Tue Feb 07 09:32:34.869020 2017] [sm:warn] [pid 2526] Siteminder Web Agent: restart not supported.

    The agent is installed in /opt/CA and everything under it is owned by the user apache and group apache. I made changes to the /etc/sysconfig/httpd file to make sure all of the environment variables are happy:

     

    PATH=$PATH:/opt/CA/webagent/bin
    export PATH
    CASMHOME=/opt/CA/webagent
    CAPKIHOME=$CASMHOME/CAPKI
    PATH=$PATH:$CASMHOME/bin
    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CASMHOME/bin
    export CASMHOME CAPKIHOME PATH LD_LIBRARY_PATH

     

    Can anybody offer any advice? Many thanks. I'm happy to provide other logs or configs if necessary.



  • 2.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Broadcom Employee
    Posted Feb 07, 2017 12:57 PM

    Hi Mark,


    Several items are critical in determining if this supposes to work or not.

    Agent version:?

    Agent bits level:?

    Agent web server vendor:? Redhat apache, open source, IBM apache....

    What is the agent binary installer file name:?

     

    Is there any error or warning in agent installation logs?

     

    Hongxu



  • 3.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 07, 2017 02:27 PM

    Agent version: I don't know. Based on the installer I think it's 12.52.

    Agent bits level: If you're asking if it's i686 or x86_64, it's x86_64.

    Agent web server vendor: RH apache (apache 2.4.x)

    What is the agent binary installer file name: ca-wa-12.52-sp01-cr01-linux-x86-64.bin

     

    Is there any error or warning in agent installation logs? None.



  • 4.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Broadcom Employee
    Posted Feb 07, 2017 02:57 PM

    SAP Portal Services 

    The agent installed is not certified version, see page 22, you should be using web agent 12.52sp1cr02 or later.



  • 5.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Broadcom Employee
    Posted Feb 07, 2017 01:30 PM

    Hi Mark,

     

    Please check out this knowledge base article.  Moving the web agent's environment variables to the httpd file is the correct thing to do, but the variable names provided in your post do not look right.  Please follow the example in the article:

     

    http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1340867.aspx 

     

    -Pete



  • 6.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 07, 2017 02:23 PM

    This is getting me somewhere. I changed my variables to use those linked in your article (the previous set was supplied to me by another admin some time ago).

     

    Now I am seeing the following in /var/log/httpd/error_log:

    CSmLowLevelAgent: No such file or directory
    [07/Feb/2017:14:10:29] [Error] SiteMinder Agent
    Failed to Start the LLAWP process.
    Execlp failed: 'No such file or directory'. LLAWP.exe must be callable from the system path."
    [07/Feb/2017:14:10:34] [Info] [CA WebAgent IPC] [2358] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6647862b
    [07/Feb/2017:14:10:49] [Info] [CA WebAgent IPC] [2358] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6647862b
    [07/Feb/2017:14:10:49] [Error] SiteMinder Agent
    Failed to initialize the message bus.
    [07/Feb/2017:14:10:49] [Warning] SiteMinder Agent
    SiteMinder agent has encountered initialization errors and will not service requests.
    [07/Feb/2017:14:10:49] [Info] [CA WebAgent IPC] [2358] [CSmSem::SemRm] Removed semaphore 393222

    The trouble is that LLAWP *is* callable from the system path now. From /etc/sysconfig/httpd:

    NETE_WA_ROOT=/opt/CA/webagent
    export NETE_WA_ROOT
    NETE_WA_PATH=${NETE_WA_ROOT}/bin
    export NETE_WA_PATH
    CAPKIHOME=/opt/CA/webagent/CAPKI
    export CAPKIHOME
    LD_LIBRARY_PATH=${NETE_WA_ROOT}/bin:${NETE_WA_ROOT}/bin/thirdparty:${LD_LIBRARY_PATH}
    export LD_LIBRARY_PATH
    PATH=${NETE_WA_PATH}:${PATH}
    export PATH

    LLAWP in the path:

     

    [root@devserver ~]# which LLAWP
    /opt/CA/webagent/bin/LLAWP

    The LLAWP binary also doesn't seem to start with apache, which it usually does on my RHEL6 systems. I see no errors in error_log after restarting the service. It only seems to want to start LLAWP when I make a request.

     

    I can start LLAWP manually, and it appears to connect just fine with no errors. But it doesn't start with the httpd service at all.



  • 7.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Broadcom Employee
    Posted Feb 07, 2017 07:13 PM

    Hi Mark,

    Yes supported is best, but also for your problem, it does seem to be env variable related.

    If httpd cant find the LLAWP.exe to run it, it was not in the PATH for that httpd executable.

    That can happen because a lot of env variables are restricted when you startup a httpd child process
    to handle requests PATH and LD_LIBRARY_PATH being two of them. 

    My suggestion is to try out the apache httpd PassEnv command.
    mod_env - Apache HTTP Server Version 2.4 

    How it exactly works depends if you are starting apache as root or running via the suEXEC version.

    If it worked for you previously, there may be some change in the apache build from Redhat 6 to Redhat 7.

    But that error is fairly clear, that it did not find LLAWP.exe in the PATH when it tried to run it.

    So :
        PassEnv LD_LIBRARY_PATH PATH

    may help.

    Also strace is a good way to see what is happening.

    strace -Ff -t -i -v -o strace. log -s 16384 <command>



    Cheers - Mark
    But it is best to try the supported version, sometimes it is just that was the version that
    was validated, but often it is because of a specific problem that is then fixed or worked around in that
    version.



  • 8.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 08, 2017 08:59 AM

    I'm working on getting the supported version. I just don't have an installer available to me. Once I do I will work with that.

     

    In the meantime I tried passing the environment variables. I found a `PassEnv CAPKIHOME` so I added LD_LIBRARY_PATH and PATH to that. No dice. Then I tried three separate `PassEnv` entries. Same exact error.



  • 9.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 08, 2017 12:04 PM

    Okay. I'm now running agent version 12.52.0106.2209 and I can't get Apache to start at all anymore:

    httpd: Syntax error on line 56 of /etc/httpd/conf/httpd.conf: Cannot load /opt/CA/webagent/bin/libmod_sm24.so into server: libicudata.so.49: cannot open shared object file: No such file or directory

    There's definitely some kind of path issue going on here but I cannot for the life of me figure out what it could be. The libicudata.so.49 shared object is in /opt/CA/webagent/bin. That location is in LD_LIBRARY_PATH and the LD_LIBRARY_PATH environment variable is being passed along to apache in /etc/sysconfig/httpd. I'm also using PassEnv in httpd.conf for what it's worth.



  • 10.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 08, 2017 12:11 PM

    It looks like a permission issue.  Is the user that Apache is running as have permissions to that directory and the files located there?



  • 11.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 08, 2017 12:14 PM

    Yes. The /opt/CA directory and all subfolders and files are owned by user and group apache.



  • 12.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 08, 2017 04:47 PM

    Why not try capturing the strace log ?

     

    strace -Ff -t -i -v -o strace. log -s 16384 <command>



  • 13.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 09, 2017 03:58 PM


  • 14.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Broadcom Employee
    Posted Feb 08, 2017 04:44 PM
    1) This looks a little different - with the new version - a missing dependency perhaps.  

    > httpd: Syntax error on line 56 of /etc/httpd/conf/httpd.conf: Cannot load /opt/CA/webagent/bin/libmod_sm24.so into server: libicudata.so.49: cannot open shared object file: No such file or directory

    That errors would be happening earlier, when running the cmd to start apache, not as the earlier error occurred, when the running (bg) apache
    tried to fork off a child process.

    That error while reading the initial config when starting httpd usually means a missing dependency:

    Can you try :
       source the env variables . ./ca_wa_env.sh

       ldd  /opt/CA/webagent/bin/libmod_sm24.so

    See if it is just path or something else, odd that it did not complain in the earlier version.

    I think :
    PassEnv LD_LIBRARY_PATH PATH CAPKIHOME

    Certainly the in the previous one, the child process was not getting the PATH from the parent.


    2) strace will help

    strace -Ff -t -i -v -o strace. log -s 16384 ./apachectl start


    That will show exactly what env variables are passed into the child process, and also what .so files are attempted to
    load - so usually gives some clue as to what is missing.

    Cheers - Mark



  • 15.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 09, 2017 03:58 PM


  • 16.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 09, 2017 04:00 PM

    [root@dev ~]# . /opt/CA/webagent/ca_wa_env.sh
    [root@dev ~]# ldd /opt/CA/webagent/bin/libmod_sm24.so
    linux-vdso.so.1 => (0x00007ffc32291000)
    libsmerrlog.so => /opt/CA/webagent/bin/libsmerrlog.so (0x00007fdc5c398000)
    libsmeventlogger.so => /opt/CA/webagent/bin/libsmeventlogger.so (0x00007fdc5c290000)
    libsmcommonutil.so => /opt/CA/webagent/bin/libsmcommonutil.so (0x00007fdc5c107000)
    libsmi18n.so => /opt/CA/webagent/bin/libsmi18n.so (0x00007fdc5bf74000)
    libicudata.so.49 => /opt/CA/webagent/bin/libicudata.so.49 (0x00007fdc5ad53000)
    libicui18n.so.49 => /opt/CA/webagent/bin/libicui18n.so.49 (0x00007fdc5aa28000)
    libicuio.so.49 => /opt/CA/webagent/bin/libicuio.so.49 (0x00007fdc5a91b000)
    libicuuc.so.49 => /opt/CA/webagent/bin/libicuuc.so.49 (0x00007fdc5a694000)
    libSmXlate.so => /opt/CA/webagent/bin/libSmXlate.so (0x00007fdc5a535000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdc5a308000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fdc5a103000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fdc59efb000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fdc59bf3000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fdc598f0000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdc596da000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fdc59319000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fdc5c839000)



  • 17.  Re: Cannot start LLAWP on RHEL7/Apache 2.4
    Best Answer

    Broadcom Employee
    Posted Feb 12, 2017 06:43 PM
    Hi Mark :

    I assume this is in the logs :
    httpd: Syntax error on line 56 of /etc/httpd/conf/httpd.conf: Cannot load /opt/CA/webagent/bin/libmod_sm24.so into server: libicudata.so.49: cannot open shared object file: No such file or directory

    The strace, shows startup via systemd -

    12100 15:53:47 [00007fd75cfad9f7] execve("/sbin/apachectl", ["/sbin/apachectl", "start"], 

    ...

    12102 15:53:48 [00007fcd052c59f7] execve("/usr/bin/systemctl", ["/usr/bin/systemctl", "start", "httpd.service"],

     

    And we loose all visibility once there since this process is just waiting for httpd to start. Given the startup its a little like the suid bits, where new process is started that does not inherit the env of the parent process at that point.   

     

     

    I did find this article: 

    https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec1340867.html 

     

    In RHEL 7 /etc/sysconfig/httpd, only environment variable may be defined and the source statement does not work.

     

    The resolution is to include the content of ”ca_wa_env.sh” script file in /etc/sysconfig/httpd. Beware that ${VARIABLE} must be replaced with the actual value. Especially, as for ${LD_LIBRARY_PATH} and ${PATH}, see these values by ‘env’ command beforehand. This modification to the httpd file should be logically equal to sourcing the shell script.

     

    Which seems to be on point for your problem and hopefully that will resolve your issue. 

     

    And also found this community idea, asking for better integration with systemd for webagent: 

    Start/stop Red Hat Apache 2.4 (RHEL 7) / Web Agent using systemctl command 

     

     

     

    Nevertheless, I've reserved a RedHat7 vm, to tinker with the systemd startup, and put up some more exact instructions.  The main point is we need to set those three env variable PATH, LD_LIBRARY_PATH and CAPKIHOME, before the httpd program runs.  I did find this article which I want to test as well :

    arch linux - How to set environment variable in systemd service? - Server Fault 

     

    Cheers - Mark



  • 18.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 13, 2017 11:04 AM

    Holy. ****.

     

    This was the solution. Specifically: I manually ran ca_wa_env.sh, I echoed each variable to see the completely parsed value, and replaced the contents in /etc/sysconfig/httpd. 

     

    Before I did that I double-checked and confirmed that the variable "EnvironmentFile=/etc/sysconfig/httpd" was set in the  /usr/lib/systemd/system/httpd.service file. Then I pasted the contents in into the /etc/sysconfig/httpd file. 

     

    Before I made changes I had the following at the end of the file:

    NETE_WA_ROOT=/opt/CA/webagent
    export NETE_WA_ROOT
    NETE_WA_PATH=${NETE_WA_ROOT}/bin
    export NETE_WA_PATH
    CAPKIHOME=/opt/CA/webagent/CAPKI
    export CAPKIHOME
    LD_LIBRARY_PATH=${NETE_WA_ROOT}/bin:${NETE_WA_ROOT}/bin/thirdparty:${LD_LIBRARY_PATH}
    export LD_LIBRARY_PATH
    PATH=${NETE_WA_PATH}:${NETE_WA_PATH}/bin:${PATH}
    export PATH

    After making changes, this is what I had at the end of the file:

    NETE_WA_PATH=/opt/CA/webagent/bin
    export NETE_WA_ROOT
    NETE_WA_ROOT=/opt/CA/webagent
    export NETE_WA_PATH
    CAPKIHOME=/opt/CA/webagent/CAPKI
    export CAPKIHOME
    LD_LIBRARY_PATH=/opt/CA/webagent/bin:/opt/CA/webagent/bin/thirdparty:/opt/rh/rh-php70/root/usr/lib64:/opt/CA/webagent/bin:/opt/CA/webagent/bin/thirdparty:/opt/CA/webagent/bin
    export LD_LIBRARY_PATH
    PATH=/opt/CA/webagent/bin:/opt/CA/webagent/bin/bin:/opt/rh/rh-php70/root/usr/bin:/opt/rh/rh-php70/root/usr/sbin:/opt/CA/webagent/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
    export PATH

    This is confirmed working. THANK YOU!!!!



  • 19.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 13, 2017 11:37 AM

    Just up-voted that enhancement request (sorry ideation (whatever an ideation is )).  That seemed like a no-brainer.  Actually surprised this wasn't found before and documented.



  • 20.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 07, 2017 06:56 PM

    Red Hat Apache 2.4.X is NOT CERTIFIED on RHEL 7 64 bit on 12.52 SP1CR1 (ca-wa-12.52-sp01-cr01-linux-x86-64.bin)

    You will need to install minimum 12.52SP1CR02 web agent as per the PSM.

     

    Please do not proceed with any further troubleshooting on an unsupported version.

     

    Cheers,

    Ujwol

    Ujwol's Single Sign-On Blog 



  • 21.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Feb 08, 2017 08:57 AM

    This is good information. I'm reaching out to my server admins to find out how to get the newer release.



  • 22.  Re: Cannot start LLAWP on RHEL7/Apache 2.4

    Posted Jan 08, 2019 05:28 AM

    Hello

    did you do any other changes..... because i am still getting some other error post addition of NETE_WA_PATH, NETE_WA_ROOT, PATH etc etc

     

    httpd: Syntax error on line 150 of /opt/apache/conf/httpd.conf: Cannot load /opt/apache/ca/webagent/bin/libmod_sm24.so into server: libsmerrlog.so: cannot open shared object file: No such file or director