Symantec Access Management

Expand all | Collapse all

HLA Error, Exiitng with HTTP 500 Error

Chris_Stallone

Chris_StalloneFeb 24, 2015 04:18 PM

  • 1.  HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 12, 2015 11:28 AM

    We have a server that suddenly started throwing a 500 error. This is what was logged repeatedly in the Agent Log:

     

    [6232/2680][Fri Jan 30 2015 10:50:09][CSmHighLevelAgent.cpp:191][INFO] HLA: Initialization complete.

    [6232/2680][Fri Jan 30 2015 16:48:27][CSmHttpPlugin.cpp:502][ERROR] Unable to resolve URL. Exiting with HTTP 500 server error '10-0002'.

    [6232/2680][Fri Jan 30 2015 16:48:27][CSmResourceManager.cpp:155][WARNING] HLA: Missing resource data.

    [6232/2680][Fri Jan 30 2015 16:48:33][CSmHttpPlugin.cpp:502][ERROR] Unable to resolve URL. Exiting with HTTP 500 server error '10-0002'.

    [6232/2680][Fri Jan 30 2015 16:48:33][CSmResourceManager.cpp:155][WARNING] HLA: Missing resource data.

    [6232/2680][Fri Jan 30 2015 16:48:42][CSmHttpPlugin.cpp:502][ERROR] Unable to resolve URL. Exiting with HTTP 500 server error '10-0002'.

    [6232/2680][Fri Jan 30 2015 16:48:42][CSmResourceManager.cpp:155][WARNING] HLA: Missing resource data.

    [6232/2680][Fri Jan 30 2015 16:48:49][CSmHttpPlugin.cpp:502][ERROR] Unable to resolve URL. Exiting with HTTP 500 server error '10-0002'.

    [6232/2680][Fri Jan 30 2015 16:48:49][CSmResourceManager.cpp:155][WARNING] HLA: Missing resource data.


    We had to kill LLAWP and Recycle the App Pool to recover. Does anyone know what causes this error out of the blue?


    SM Policy Server 12.52 SP1

    Policy Server & Agent are both running Windows 2008 R2 64-bit.



  • 2.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 13, 2015 01:36 AM

    Hi,

     

    The message in web agent log is general and didn't help much to troubleshoot the issue. We need to check on the web agent trace log to match the tid in order to understand what request has received and possibly give us some clue on why the agent behave that way.

     

    You can find 10-0002 error code from following:

    CA SiteMinder® Integrated Documents 12.52 SP1

     

    Is this on IIS web server? Did you see any error from event viewer log? Is this issue reproducible?

     

    Regards,

    Kar Meng



  • 3.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 16, 2015 04:39 PM

    Thanks for responding. I agree the web agent log is general and doesn't help! I no longer have the Trace Log, but will grab it if this issue occurs again. I am not able to reproduce it.

     

    This is on IIS 7.5, and there were no errors or any indications in the Windows Event Log.



  • 4.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 17, 2015 12:49 AM

    Thanks for update. Web agent trace log will be a good start to investigate the issue. Beside, check if there is any DNS issue for last occurrence. Sometime DNS lookup issue might cause web agent an issue.



  • 5.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 24, 2015 04:18 PM

    Anyone able to assist Kevin further?

     

    Thank you



  • 6.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 25, 2015 01:51 PM

    Kevin kbuckle7

     

    Could you advise if the WebServer stopped responding to all request i.e. legit URL access.

     

    10-0002 suggests that Illegal Characters exist in a URL or characters defined in the BadUrlChars parameter have been detected in a URL. This is what the WebAgent is designed to do. However if these functions cause the WebServer unresponsive. Then that is a serious issue in the WebAgent. Hence my questions follow...

     

    1. Did the WebServer getting unresponsive OR throwing HTTP 500 on Browser, time exactly (happened the exact time) with the 10-0002 messages being logged in the log?
      1. The last thing we would want to do is follow a red herring. Hence it is crucial to know if both concurrences happened at same time.
    2. Did you see a huge flurry of 10-0002 suddenly out of the blue (like an attack)?
      1. Have you inspected the WebServer Access log to determine what those URLs where and where did those get initiated from?
      2. Are those legit request OR just some person trying a inject something random?

     

    Please advice on the above points. It may help to ascertain a way forward.

     

     

    Regards

     

    Hubert



  • 7.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 25, 2015 03:46 PM

    Yes, the Web Server would stop responding to all requests. (It served up the SiteMinder-specific 500 Error page). It did not work again until we killed the LLAWP process and restarted the App Pool.

     

    1. Yes, the error directly correlated with the 10-0002 message.

    2. Yes, it was suddenly, and out of the blue.

     

    I would further note, that, after upgrading from 12.5 to 12.52SP1 Policy Servers, this WebAgent began to have an app pool crash twice a day, with the SM 500 Error. The resolution was to create a custom HCO that used the IP addresses of the Policy Servers instead of the FQDNs.



  • 8.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 26, 2015 10:05 AM

     

    Since the WebAgent process is getting unresponsive; it would need some deeper investigation via Support Channels.

    Is this issue reproducable OR has occurred repeatedly over a period of time i.e. not just one time occurrance.

    Assuming the latter; my recommendation would be in different parts to investigate deeper.

     

     

    NOTE : Running PART-A, PART-B and PART-C in production could have slight performance implications. Always safe to run these on non-Prod Environments. Turn'em off once debugging is complete in Production. PART-D is more of performance tuning.

     

     

    PART-A : Run diagnostics on the LLAWP and w3wp.exe Process.

    Running diagnostics would ensure, that in the event of a Crash or Hang; CA Engineering would have substantial data of the State of process & memory when the Process is hung.

    • Run a Debug Diagnostics Tool on the LLAWP and IIS process to print a DUMP when a CRASH or HANG state occurs.
    • We could use DebugDiag or ADPlus; since WebAgent OS is Windows.
    • Make sure which ever tool is used, it is configured for a "FullDump" and not a "MiniDump".
    • Since we have configured "FullDump"; there would be lot of dumps [FirstChance Exceptions] getting generated (Each may be in size of 700MB to 900MB).
    • What is of our interest is a Dump which is called [SecondChance Exception] and one or two [FirstChance Exception] just before the [SecondChanceException].
    • Until such time there are no [SecondChance Exception] dump; we could delete all [FirstChance Exception] dumps.

     

     

    PART-B : Run PerfMon on LLAWP and w3wp.exe Process.

    Running Performance Monitor and save it in CSV format, this would ensure we are capturing the memory state of LLAWP Process. Just to make sure there are no memory leaks. Configure Performance Monitor to run every 5mins and configure it to capture the below attributes.

       - memory\available bytes

       - memory\committed bytes

       - process(LLAWP)\processor time

       - process(LLAWP)\private bytes.

       - process(LLAWP)\thread Count.

       - process(w3wp)\thread Count.

       - process(w3wp)\processor time

       - process(w3wp)\private bytes.

     

     

    PART-C : Enable WebAgent Tracing.

    Enabling Tracing does have performance implication. However need of the hour is for as much details possible, it would worth to sacrifice a bit of performance until the issue is resolved. Turn of Tracing in Production once issue is resolved OR support has all necessary information.

     

     

    PART-D : Tune the HCO, Increase the Max Sockets Per Port in the HCO.

    The trusted host and Policy Server communicate across TCP/IP connections. The number of available sockets for the authorization, authentication, and accounting ports of the Policy Server determines the number of available TCP/IP connections. The number of sockets per port controls the number of simultaneous threads accessing the Policy Server from the web server. Separate web server threads handle each user access request. Each thread requires its own socket. The web server maintains a pool of threads for requests and only creates one when there are no more available threads. As traffic increases, the number of sockets per port must increase. Several settings affect the TCP/IP connections between the trusted host and the Policy Server.

    • Maximum Sockets Per Port
      Defines the maximum number of TCP/IP connections that the trusted host uses to communicate with the Policy Server. By default, this value is set to 20, which suits low- and medium-traffic web sites. Increase this number in the following situations:
      • You are managing a high-traffic web site.
      • You have defined agent identities for virtual servers.
    • Minimum Sockets Per Port
      Determines the number of TCP/IP connections open for the Policy Server at startup. The default value is 2. If you are managing a high-traffic web site, increase this number.
    • New Socket Step
      Specifies the number of TCP/IP connections that the Agent opens when new connections are required. The default value is 2. Increase number of sockets to add at each required at each level when you require more sockets

     

     

    PART-E : Open a support Ticket.

    Unfortunately we do not have the capability to read and validate dumps on public forums. Hence when there is a dump available for a Crash or Hang. Open a Support Ticket and have all of the above information ready for Support; as this would save crucial time and to/fro communications. Trust me the to/fro comms could be a bit frustrating especially when there are Service Outages and we do not have the right data for analysis.

     

     

    PART-F : Provide a bit more deeper info on the Setup in Support Ticket.

    IIS Setup Info

         - Application Pool Bit Version.

         - Application Pool mode i.e. Integrated or Classic.

         - Is Web Gardening being used, how many processes is w3wp set to.

         - How many WebSites on IIS or just the on Default WebSite.

         - Is there any other plugins enabled on IIS e.g. any Reverse Proxy Plugins?

     


    PART-G : Check Policy Server Heatlh.

    Kindly check the Policy Server for any abnomalities being reported during the tenure when WebAgent stops responding.


     

    Regards

     

    Hubert



  • 9.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Mar 10, 2015 05:36 PM

    Thank you for your help and suggestions. We actually had a case open with CA for several months, but never got anything remotely helpful. We have had some better luck in the Communities.



  • 10.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Mar 10, 2015 05:43 PM

    Thank You Kevin kbuckle7

     

    Hope we get to the root of the problem soon. If we do find it, kindly drop a line as to what the problem was and solution if any. Sharing knowledge helps us get round issues a lot quicker by reading previous experiences. Please feel free to keep posting queries, as it a learning for all.

     

     

    Regards

     

    Hubert



  • 11.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Mar 10, 2015 05:55 PM

    Absolutely!



  • 12.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Feb 26, 2015 05:34 PM


    from my experience, these errors generally show randomly when IIS worker process recycles without LLAWP going down properly. most common cause is app pool recycling being set afte certain number of minutes.

     

    Please verify there is app pool recycle  option set to some minutes along with Idle time for IIS worker process. if they are enabled and set toa  specific value I suggest you to chang them as below to avoide those HLA errors in future

     

    1) Disable App pool recycling (after certain minutes, as far i recall recall 1720 is the default value)

    2) set idle timeout to 0 (disbaling it) in the properties of the app pool being used by the website



  • 13.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Mar 12, 2015 10:06 AM

    why do i feel like i stumbled across the blind leading  the blind?

    is there a case open? if so which support engineer is helping you?

    we should probably leave it to them if you have one. too  many cooks can cause issues.

     

    i would doubt the 10-0002 (URL HTTP METHOD ISSUE) is causing the web server to stop.

    BADURLCHARS/BADCSSCHARS are 00-0002

     

    i would expect the value of "DISALLOWOVERLAPPINGROTATION" to play a role.

    It could cause plug ins to crash into themselves depending on the setting, which causes a plug in to fail.

     

    i would start with this MS write upon the feature:

     

    https://msdn.microsoft.com/en-us/library/microsoft.web.administration.applicationpoolrecycling.disallowoverlappingrotation(v=vs.90).aspx



  • 14.  Re: HLA Error, Exiitng with HTTP 500 Error

    Posted Dec 17, 2015 10:11 AM

    We are having the same issue.  Except we are using the IBM Web Server (IHS) on windows 2008 R2 with Web Sphere.

     

    Did anyone ever work out a root cause?  What about DNS resolution(nslookup) on the web agent?  Is DNS resolution (like nslookup uses) 100% required or will hosts entries on the server suffice?

     

    What about better profile settings so the trace file will actually show what hosts is it not able to resolve.  Why would  you put out an error saying  you can't resolve something, then not tell people what you can't resolve.  Seriously.