DX Application Performance Management

Expand all | Collapse all

Connecting to Enterprise Manager

Junaid Wahab

Junaid WahabMay 22, 2018 01:35 PM

  • 1.  Connecting to Enterprise Manager

    Posted May 22, 2018 11:07 AM

    I have 10.5.2.24 with SP2 installed. I am not able to connect to MOM using workstation. Even Webview is also not able to connect to MOM. 

     

    MOM logs I am getting following:  

    5/22/18 10:47:53.073 AM EDT [DEBUG] [Acceptor Helper 1] [Manager.AcceptorHelper] Error handling incoming Connection.  This may happen normally when a Workstation logs in: java.net.SocketTimeoutException

     

    WebView logs:

    apmadm@emkhi23971:/opt/CA/APM/10.5.2.24/bin> ./WVCtrl.sh start
    Using APMHOME: /opt/CA/APM/10.5.2.24
    Using JAVA_HOME: ../jre
    Using EM Host: emkhi23971.acc.american.com
    Using EM Port: 5001
    MOM/EM is running..
    JVM PID is  20182
    ./WVCtrl.sh start: Starting APM WebView...
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    328
    Problem while starting WebView server. Please check /opt/CA/APM/10.5.2.24/logs/IntroscopeWebViewConsole.log for details.
    ./WVCtrl.sh start: APM WebView could not be started

     

     

    any idea?



  • 2.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 11:23 AM

    Try clearing the work cache on the EM by deleting the contents of <EM_HOME>/work.

    This is likely the problem if you were previously running as root and then moved to a non-root user.

     

    Also, clear your browser cache.



  • 3.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 12:04 PM

    Hiko_Davis

    I am running both MOM and Webview as apmadm since I install. I clear the cache and restart both MOM and Webview /ATC but still having same problem. My Workstation is installed in my laptop so don't need clear the browser cache. 

    any other suggestion because  I am still getting same



  • 4.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 12:27 PM

    Stop the MOM, clear the work cache, change root logger to DEBUG, restart MOM and attempt to connect. Post results of the login attempt here.



  • 5.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 11:53 AM

    In addition to what Hiko posted, the KB below will show you how to clear all the cache so that Workstation and Webview can connect.

     

    How do I clear the cache on a APM Enterprise Manag - CA Knowledge 



  • 6.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 01:35 PM
      |   view attached

    Here you go the logs

    Attachment(s)



  • 7.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 01:38 PM

    Hi,

     

    1. From your laptop, telnet to <IP_EM>:5001

    2. The services were iniciated with user root?

     

    Regards,

    Richard



  • 8.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 01:38 PM

    Thanks; I'll report back soon.



  • 9.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 01:41 PM

    Hi @junaidwily,

     

    And the logs Webview?

     

    Regards,

    Richard



  • 10.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 01:46 PM

    Not likely related, but you should check your Management Module that has this action in it and remove it temporarily:

     

    5/22/18 01:01:25.980 PM EDT [ERROR] [main] [Manager.Action] Problem constructing new action: com.wily.introscope.server.enterprise.entity.alertaction.UnknownActionException: An unknown action type "ExtendedMailAction" was encountered
    com.wily.introscope.server.enterprise.entity.alertaction.UnknownActionException: An unknown action type "ExtendedMailAction" was encountered


  • 11.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 01:59 PM

    Richard Here you go the WebView logs

     

     

    5/21/18 12:27:28.080 PM EDT [INFO] [WebView.HealthMonitor] EM/MOM status is unavailable
    5/21/18 12:27:28.081 PM EDT [INFO] [WebView.Login] Logged out user "WilyWebView"
    5/21/18 12:27:28.084 PM EDT [ERROR] [WebServer.RemoteClientConnectionManager] Unable to connect to EM emkhiabc.un.america.com:5001,com.wily.isengard.postofficehub.link.net.HttpTunnelingSocketFactory - java.net.ConnectException: Connection refused (Connection refused)
    5/21/18 12:27:28.085 PM EDT [ERROR] [WebView.Login] Error while releasing session token.
    5/21/18 05:02:49.213 PM EDT [INFO] [WebView] The WebView application has successfully stopped.
    5/21/18 05:02:49.230 PM EDT [INFO] [org.mortbay.log] Stopped SocketConnector@0.0.0.0:8080
    5/21/18 05:14:41.657 PM EDT [INFO] [WebServer] Starting Web Application Server
    5/22/18 10:53:16.097 AM EDT [INFO] [WebServer] Starting Web Application Server
    5/22/18 11:55:44.241 AM EDT [INFO] [WebServer] Starting Web Application Server



  • 12.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 01:53 PM

    Shutdown the EM, remove the module, and restart.

    After the EM has fully started, add the module back using the 'deploy' folder and fix the alert action.

     

    This particular field solution requires a EM and workstation plugin. Make sure you have both. The EM plugin goes to <EM_HOME>/product/enterprisemanager/plugins. The WS plugin goes to <EM_HOME>/ws-plugins.



  • 13.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 02:03 PM

    Right now I move this MM to different folder and restart MOM. still facing same problem



  • 14.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 02:07 PM

    Have you tried what Richard suggested and tried to telnet from your WV server to EM on port 5001?

    What does 'netstat' say? Is the port stuck open? If so, do you have the commandline tool 'fuser' installed so you can clear the TCP port?



  • 15.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 02:35 PM

    5001 port is open


    Trying 10.651.64.57...
    Connected to emabc31s016.
    Escape character is '^]'.



  • 16.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 02:59 PM

    Are your webview and EM on the same server or different servers?

     

    Check your webview properties file for the tcp.host, tcp.port and webserver.tcp.port settings and see if they are set properly.  If using localhost, try using the hostname or IP.



  • 17.  Re: Connecting to Enterprise Manager

    Posted May 22, 2018 03:06 PM

    They both are running on different machines. It was working for last 3 -4 month but suddenly start having problem. I think culprit is MOM because it is not allowing any workstation connection.

    introscope.webview.tcp.port=8080

    introscope.webview.enterprisemanager.tcp.port=5001
    introscope.webview.enterprisemanager.webserver.tcp.port=8081



  • 18.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 03:33 PM

    Have you recycled MOM and Webview?  If so and you still see the same issue, then I recommend opening an issue to debug it further.



  • 19.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 22, 2018 07:07 PM

    Yes the root cause may be the MOM performance. Recycling MOM per Matt's advice should hopefully resolve it at least temporarily. However reviewing the perflog.txt (How to interpret the values of Perflog.txt fields. - CA Knowledge) or creating a support case for deeper review would be a good idea to confirm root cause and prevent future occurrences.



  • 20.  Re: Connecting to Enterprise Manager

    Posted May 23, 2018 09:43 AM

    I run "netstat -an | grep 5001" and getting more then 50 results. some of them are showing FIN_WAIT2. My other MOM who are running fine are showing only 6 or 7 results. yesterday I change the port from 5001 to 5002 and I was able to connect to MOM using workstation also Webview was connected fine.

    I Google it and found that the number of ports are exhausted Normally this  type of problem occurs because  the web application is opening connections under the covers and then put the port into TIME_WAIT and thereby.

     

    My question is is there a way I can forcefully close the connections. I asked our Linux admin to reboot the machine but it didn't help.



  • 21.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 23, 2018 11:44 AM

    If you are able to install the package ‘psmisc', it comes with a commandline tool called ‘fuser' that can be used to free those ports without rebooting your server.

    I know on you're on SuSE, so I'm not sure how to do that on that platform. For CentOS/RHEL, you need to install the following: ‘epel-release psmisc'.

     

    Once installed, just run the following command:

     

    fuser -k -n tcp 5001



  • 22.  Re: Connecting to Enterprise Manager

    Posted May 23, 2018 01:24 PM

    Hiko_Davis its help and close all the connection also kill MOM process but when I start MOM all the tcp 5001 connection are again showing the same. Is there any other way to fix it?



  • 23.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 23, 2018 01:37 PM

    Can you account for each connection? Should be one for each Collector, WebView, and workstation.

     

    Looks like we'll need to take a deeper look at your EM and OS configurations.

    Please open a ticket and make sure to reference me as your contact.

     

    Please upload your properties files and logs.

     

    shaja15 : FYI



  • 24.  Re: Connecting to Enterprise Manager

    Posted May 23, 2018 05:21 PM

    Support case # 01095393 - Connecting to Enterprise Manager



  • 25.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 23, 2018 06:20 PM

    There will also be many open sockets for the EM->APM DB connection.

    Testing with my 10.7 MOM immediately after startup shows 24 open sockets from its java process to postgres socket 5432. You can check with command "netstat -nap|grep 5432|grep -i jav|wc -l"



  • 26.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 23, 2018 11:59 AM

    Try installing fuser this way: sudo zypper install psmisc

     

    Hopefully, it will also download any dependencies for you.



  • 27.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 23, 2018 12:01 PM

    This article may be relevant to you when you go to install 10.7 and Infrastructure Agent: https://comm.support.ca.com/kb/systemedge-agent-install-fails-on-linux-with-errors-about-missing-libraries/kb000033669

     



  • 28.  Re: Connecting to Enterprise Manager

    Posted May 30, 2018 02:06 PM

    Check the version of your workstation install and the version of the EM - MOM.  If you look in the MOM log (IntroscopeEnterpriseManager.log) there is a start up banner that has the version number.  On the Workstation, on the log in screen, it will have the version and build number.    Pretty good chance that a version 10.0.x workstation will not connect to a 10.5.2 EM.  Check the downloads for 10.5.2.24 sp2 for a new workstation install.

     

    Workstation:

    I've been having the same issues for months (support case 01086810) and I'm to the point of thinking that it is the WVCtrl.sh script itself.  So the next time this type of issue occurs, I'm going to try to start WebView from binary.

     

          nohup ./Introscope_WebView.bin >> "nohup.out" 2>&1 &

     

    Yea, using this makes it messy to stop WebView since you will need to get the pid (ps -elf|grep webview) and then kill -9 <webview pid>

     

    Also try to run workstation from the MOM  http://<mom host>:8081 in IE with Java plugin and click on launch workstation to insure the versions all match up.

     

     

    Hope this helps,

     

    Billy



  • 29.  Re: Connecting to Enterprise Manager

    Broadcom Employee
    Posted May 30, 2018 05:37 PM

    Yes in general the workstation and EM/MOM versions being used must match to ensure full compatibility because that is the only combination that has been tested. A mismatch of versions may work but not guaranteed because of changes across versions.



  • 30.  Re: Connecting to Enterprise Manager
    Best Answer

    Posted May 30, 2018 05:54 PM

    The problem start when we are moving agents from one Data Center to another Data Center APM Environment.

    I worked with Norris Graves and Hiko_Davis and find out that some DNS server are acquiring these 5001 ports. temp we are moving problematic agent to old Data center APM environment after that I will work with App owners and Linux team to fixing the problem.

    Thanks everyone for the help