Symantec Access Management

After upgrading an SPS server from 12.52 sP1 CR5 to CR8 the server will not start. The following message is displayed in the server.log file

[19/Apr/2018:14:02:58-937] [FATAL] - No ROOT context found.
[19/Apr/2018:14:02:58-952] [ERROR] - InitCatalina failed ('Failed to start component [StandardServer[8005]]')
[19/Apr/2018:14:02:58-952] [ERROR] - org.apache.catalina.LifecycleException: Failed to start component [StandardServer[8005]]
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:162)
[19/Apr/2018:14:02:58-952] [ERROR] -     at com.netegrity.proxy.ProxyServer.initCatalina(ProxyServer.java:799)
[19/Apr/2018:14:02:58-952] [ERROR] -     at com.netegrity.proxy.ProxyServer.init(ProxyServer.java:158)
[19/Apr/2018:14:02:58-952] [ERROR] -     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[19/Apr/2018:14:02:58-952] [ERROR] -     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[19/Apr/2018:14:02:58-952] [ERROR] -     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[19/Apr/2018:14:02:58-952] [ERROR] -     at java.lang.reflect.Method.invoke(Method.java:606)
[19/Apr/2018:14:02:58-952] [ERROR] -     at com.netegrity.proxy.ProxyBootstrap.execute(ProxyBootstrap.java:233)
[19/Apr/2018:14:02:58-952] [ERROR] -     at com.netegrity.proxy.ProxyBootstrap.main(ProxyBootstrap.java:169)
[19/Apr/2018:14:02:58-952] [ERROR] - Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardService[Catalina]]
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:162)
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:758)
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
[19/Apr/2018:14:02:58-952] [ERROR] -     ... 8 more
[19/Apr/2018:14:02:58-952] [ERROR] - Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[SPS]]
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:162)
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.core.StandardService.startInternal(StandardService.java:444)
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
[19/Apr/2018:14:02:58-952] [ERROR] -     ... 10 more
[19/Apr/2018:14:02:58-952] [ERROR] - Caused by: org.apache.catalina.LifecycleException: A child container failed during start
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1127)
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:300)
[19/Apr/2018:14:02:58-952] [ERROR] -     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
[19/Apr/2018:14:02:58-952] [ERROR] -     ... 12 more
[19/Apr/2018:14:02:58-952] [ERROR] - ProxyServer initialization failed.
[19/Apr/2018:14:02:58-952] [ERROR] - Config File: 'E:\Netegrity\secure-proxy/proxy-engine/conf/server.conf')

Any ideas?

Thanks,

Matt

By looking at the last few lines of error, it does seems something is not correct in 'server.conf'. If you have backup of this file prior to CR8 upgrade, do a quick diff and see if the upgrade have overwritten any of your custom settings.

[19/Apr/2018:14:02:58-952] [ERROR] - ProxyServer initialization failed.
[19/Apr/2018:14:02:58-952] [ERROR] - Config File: 'E:\Netegrity\secure-proxy/proxy-engine/conf/server.conf')

i moved the old server.conf back and it still fails with the same error, must be something else in the configuration.

Seems a bit odd, the directory I think hardcoded or maybe set in Tomcat/properties file.

I agree with Makesh the use of backslash in strings sometimes gives problems with string parsing (being an escape character) - but I know java also can work with it - and I assume thats the old values. 

it may be the error is not really the root config but someting else too. 

I would suggest run it with Process Monitor (from sysinternal/microsoft) running as that will show you want config files it is reading, and also what directories it is trying to read from - probabyl give you a clue (I wonder if in the upgrade to CR8, it is looking for somethign on the C:\ drive not the E:\ drive but with the same path - just a wild guess.

https://docs.microsoft.com/en-us/sysinternals/downloads/procmon 

Cheers - Mark

Matt

We have the lines here from server.log. How does the nohup.log look? Did we review the nohup.log (records the boot info) which is generated on CA AG initial startup, even before server.log kicks in.

Nohup has a bunch of exceptions, here is the first:

Apr 23, 2018 9:20:39 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Apr 23, 2018 9:20:39 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Apr 23, 2018 9:20:39 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8088"]
Apr 23, 2018 9:20:39 AM org.apache.catalina.mbeans.GlobalResourcesLifecycleListener createMBeans
SEVERE: No global naming context defined for server
Apr 23, 2018 9:20:39 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Apr 23, 2018 9:20:39 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.77
Apr 23, 2018 9:20:42 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Apr 23, 2018 9:20:42 AM org.apache.catalina.util.SessionIdGeneratorBase createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [140] milliseconds.
HeaderPreFilter.init : EXECUTING FILTER INIT  tsappsdev
SearchandReplacePostFilter.init : EXECUTING FILTER INIT WRT
HeaderPreFilter.init : EXECUTING FILTER INIT  ivcma
SearchandReplacePostFilter.init : EXECUTING FILTER INIT EPORT
SearchandReplacePostFilter.init : EXECUTING FILTER INIT SIP
HeaderPreFilter.init : EXECUTING FILTER INIT  QM
HeaderPreFilter.init : EXECUTING FILTER INIT  globalbi
HeaderPreFilter.init : EXECUTING FILTER INIT  sbx
Apr 23, 2018 9:20:43 AM org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet [forward] in web application [] threw load() exception
java.lang.NoSuchMethodError: com.netegrity.util.cookie.impl.DefaultCookieSpec.<init>([Ljava/lang/String;ZLjava/lang/String;Lcom/netegrity/util/cookie/NameValuePairFactory;Z)V
    at org.tigris.noodle.filters.CopyCookies.validateNInit(CopyCookies.java:176)
    at org.tigris.noodle.filters.CopyCookies.init(CopyCookies.java:100)
    at org.tigris.noodle.NoodleFilterLoader.createFilter(NoodleFilterLoader.java:217)
    at org.tigris.noodle.NoodleFilterLoader.createFilters(NoodleFilterLoader.java:200)
    at org.tigris.noodle.NoodleFilterLoader.newInstance(NoodleFilterLoader.java:190)
    at org.tigris.noodle.Noodle.init(Noodle.java:131)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

That looks more like the real error :

Apr 23, 2018 9:20:43 AM org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet [forward] in web application [] threw load() exception
java.lang.NoSuchMethodError: com.netegrity.util.cookie.impl.DefaultCookieSpec.<init>([Ljava/lang/String;ZLjava/lang/String;Lcom/netegrity/util/cookie/NameValuePairFactory;Z)V
    at org.tigris.noodle.filters.CopyCookies.validateNInit(CopyCookies.java:176)
    at org.tigris.noodle.filters.CopyCookies.init(CopyCookies.java:100)

It was a while ago, but new constructor was added, with extra parameter, since there was some issues with date formats in transferring expiry date of cookies from what was sent from the backend to sending it to the frontend. 

But for that to not be correct, means it kept an old .jar file rather than replacing it with a new one - something like that. 

I assume the reinstall you did below, may have skipped that (although you now seem to have another issue with the service).

Cheers - Mark

Do you know the name of the Jar. Right now I'm comparing servers trying to pick it out from the datestamps.

Hi Matt,

If you add -verbose to the SmSpsProxyEngine.properties file it will show in the nohup log file which .jar file each class isn loaded from. 

But also I see you've got custom SPS filters (as per the nohup log segment you added), it might be worth commenting those out, to see if it load correctly then - it is possibly they might be relying on the older cookie format, and need to be recompiled with the new .jar files.

I assume there is a CA support case for it, maybe send me that and I'll have a look at the logs. 

Cheers - Mark

(PS: Generally I've moved into the APIM team, but still look at the occasional SPS issue)

After trying a reinstall of the SPS server the Apache 2 service is now the only service that exists. The other SPS service has disappeared from the server.

I have a backup of the conf folder so I am thinking of an uninstall / reinstall then restore the old conf folder.

Matt - Service disappearing after the reinstall is quiet unusual. This requires further digging on the env, best is to open a support ticket to investigate this further.

Hi Matt,

I have had that happen too, it might be related to unistall/install without rebooting, sometimes if somethign is in use it seems to defer it until restart time. 

After a fair bit of searching  I found :   

For the tomcat service in SPS it can be registered via : 

    SmSpsProxyEngine.exe -service default

An unregistered via : 

     SmSpsProxyEngine.exe -UnregServer default

The name "default" can change with the different instances, but default should be the "defualt" entry.

There is a similar method for the httpd service (I think it was the httpd.exe ) .

Hope that helps. 

Cheers - Mark 

Thanks Mark, That brought the service back.

Matt