DX Application Performance Management

  • 1.  EPA ORPHANED (CMD) PROCESSES & CA Support Service

    Posted Jan 26, 2011 08:45 AM
    Hi All !

    My sad history.. for other customers.. and for CA support managment team..

    26.01.2011 16:14 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello mr. Lewis !

    1. Please commit about you can not help for this case..
    2. Please send me contacts with your managment team

    Regards, Igor Matukin


    26.01.2011 15:37 DAVID LEWIS
    Hello Igor,
    As part of the EPAgent guide, CA provides the following default plugins:

    solaris/diskStats.pl . . . . . . . . . . . . . . . . . . 28
    windows/processAvailability.pl AND solaris/processAvailability.pl . . . 28
    windows/diskStats.pl . . . . . . . . . . . . . . . . . . 29
    apacheLogReader.pl, weblogicLogReader.pl, and websphereLogReader.pl . 30
    httpSvcAvailability.pl . . . . . . . . . . . . . . . . . . 31
    kstat.pl . . . . . . . . . . . . . . . . . . . . . . 32
    iisLogReader.pl . . . . . . . . . . . . . . . . . . . 33

    Please test with one of these, relevant to your system. If you see the
    memory leak with one of these, please send the heap dump.
    If any of those cause a memory leak, I will be happy to raise the issue
    with engineering.

    CA Support can only support officially tried and tested scripts. This is
    why I continue to make a distinction between our scripts and your scripts.
    Your errors relate to your own scripts.

    I do believe that using cygwin is essential to using your scripts, having
    read through them, since the excerpt from Process Explorer mentions
    processes like grep, which won't be in a standard Windows installation.

    So, I will do the following:
    1) Make sure that my management is aware of the situation. I can ask them
    to contact a suitable account representative for you, as this will be a
    good way to discuss the involvement of Professional Services.

    If you send me details of problems with one of the above default plugins,
    I will review that data, but we will not be reviewing data from your
    custom scripts.

    I'm sorry if you do not agree with our procedure but we will endeavour to
    find help for you within the whole organisation.

    Kind regards,
    David
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    26.01.2011 15:10 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello mr. Lewis !

    The Bank has no problems with "own scripts"..
    The Bank has a problems with Introscope EPagent orphaned cmd processes.. e.g.
    memory leaks..
    I am writing to you about these problems from the my first post on
    04.01.2011..
    But you are talking all the time about problems with "own scripts"..
    Once again, we have no problem with "own scripts".. "own scripts" works fine..
    According to the documentation "own scripts" can be used.. And again, the
    Bank has no problems with "own scripts".. Bank has a problems with Introscope
    EPagent orphaned cmd processes.. e.g. memory leaks.. Are you understand me ?
    If you can not understand scripts syntax and "believe it is still the result
    (problems) of you using your own scripts" you can contact with other more
    experienced colleagues.. And you had to do it the first day rather than
    contend with the customer.
    I am am happy to put you in touch with my management team if you believe
    that
    there is a support issue to be answered
    Please, do it. Please, give me please contact your management team.
    Have you had any luck with contacting Professional Services, for example?
    Please, give me please contact the Professional Services.

    Regards, Igor Matukin


    26.01.2011 14:20 DAVID LEWIS
    Hello Igor,
    I can confirm I have received the data but I believe it is still the result
    of you using your own scripts.
    I will take a quick review of this but if you are demonstrating the problem w
    ith your own scripts, I do not believe that we can offer further assistance.
    I am happy to put you in touch with my management team if you believe that
    there is a support issue to be answered.
    It may be that CA Support cannot help but another department can. Have you
    had any luck with contacting Professional Services, for example?
    Kind regards,
    David
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    25.01.2011 09:35 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hi David !
    Please confirm that you have received all necessary information..
    Regards, Igor Matukin


    21.01.2011 21:40 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hi David !

    Please, see file attachments..

    1. epagent_custom_plugins.zip - epagent custom plugins,
    installed to C:\Program Files\CA
    Wily\Introscope9.0.5.0\epagent\epaplugins\custom)

    2. epagent_procs_screenshots.zip - screenshots for demonstration orphaned
    processes problem

    3. jvisualvm_dumps_0x.zip - Java VisualVM application snapshots and heap
    dumps

    4. winperfmon_log.zip - Windows performance counters log (ids and private
    memory counters) for java processes..

    5. epagent_properties_n_log.zip - IntroscopeEPAgent.properties and
    IntroscopeEPA.log files
    (located to C:\Program Files\CA Wily\Introscope9.0.5.0) You can find errors
    in IntroscopeEPA.log file..
    For example: 1/21/11 11:48:16 AM MSK [ERROR] [EPAgent.EPAgent WinPerfDisk]
    The stateless plugin 'WinPerfDisk' has timed out.
    You can find my post from 05.01.2011 12:12
    I wrote: "..The plugins response time is small.. and equal for 2-3 seconds.."
    You can simply check it.. (run plugin as standalone script from command line)

    7. epagent_cmd_output.zip - epagent command line output..

    Regards, Igor Matukin

    P.S.
    Some custom plugins can be use
    - cygwin (v1.7.5, latest version 1.7.7, it is not important I think) tools,
    you can dowload/install (to C:\Cygwin) from http://www.cygwin.com
    - Commandline Calculator V1.10, by R.L. Veldt, you can dowload/install (to
    C:\Tools) from

    http://shareware.pcmag.com/free/Calculators-for-math-finance-etc/cc110zip/1867
    .html

    You can also dowload/install usefull Sysinternals tools.. (such as
    ProcessExplorer etc.)
    from http://technet.microsoft.com/en-us/sysinternals/bb545027

    Regards, Igor Matukin


    21.01.2011 21:33 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: epagent_properties_n_log.zip


    21.01.2011 21:33 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: winperfmon_log.zip


    21.01.2011 21:32 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: jvisualvm_dumps_03.zip


    21.01.2011 21:30 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: jvisualvm_dumps_02.zip


    21.01.2011 21:25 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: jvisualvm_dumps_01.zip


    21.01.2011 21:21 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: epagent_procs_screenshots.zip


    21.01.2011 21:21 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: epagent_custom_plugins.zip


    21.01.2011 14:09 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was deleted from the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: Email_2011_1_5_6_17_35.txt


    20.01.2011 20:24 DAVID LEWIS
    Hello Igor,
    Please show me your configuration using a script that we, CA, supply to
    demonstrate the memory leak.
    By this, I mean the plugins that you are using. In the
    IntroscopeEPAgent.properties you have sent me, you are referencing your
    BaN..bat files.
    If I try to run your scripts, I will also have to install cygwin, and
    possibly other tools.
    Also supply a logfile with this, so I can tell the exact version of Java
    (32 or 64-bit) and the particular OS.

    In that situation, I will happily reproduce your setup here, and, provided
    that we see the same results, I will escalate to our engineering team
    rapidly.

    Are you able to take a heap dump of the EPAgent process so you can see what
    is consuming so much memory?
    Are you monitoring the leak using the GC Heap information in Introscope or
    another tool?
    You can use something like visualvm, which is part of the 1.6 Sun Java JDK
    to take a heap dump on Windows.

    Just to let you know, this is a link to our technical support guidelines.
    https://support.ca.com/phpdocs/0/common/techsupp_policy.pdf
    Section 5 mentions the areas that we may not be able to help in, including
    code written by yourself.

    In which case, if the memory leak is with our code, we will happily work to
    resolve it.
    If the memory leak could be introduced by your scripts, we cannot guarantee
    that we could resolve that for you.

    Regards,
    David
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    20.01.2011 18:11 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !
    Please confirm if you are interested in me asking our product management to
    consider us supplying a Java Service Wrapper with the EPAgent..
    Yes, it is very intresting.. for any customer.. Wrapp application as windows
    service or unix daemon is good and common practice for monitoring agents..

    Regards, Igor Matukin


    20.01.2011 17:59 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !
    A customer can write any script that they like, and indeed use any script
    found on our community sites, or written by our technical consultants...
    However, those scripts do not go through any testing by our own engineers,
    so we do not know what effect they will have
    Ok, you are right, in general.. But what about dummy scripts such as echo 123
    etc. ?
    Should these scripts be tested also or no :-) ?

    Ok, how we can get confirm from your engineers about customer scripts ?
    I mean have we or not errors in customer scripts in your engineers opinion..
    Is it action for me or for you ? Shoud I open new case for confirmation from
    you engineers about echo 123 script ? Can you send our scripts for your
    engineers without many speaking ?

    David, we have strongly defined memory leaks in EPA agent.. not in customer
    scripts..
    We can reproduce problem in real time.. What is problem ?!

    Regards, Igor Matukin


    20.01.2011 16:55 DAVID LEWIS
    Hello Igor,
    I'm sorry that you have not had the responses you expect from this issue.
    I did not see all of your updates to the issue so I have not spent 20 days
    looking at your one particular issue.
    When opening the issue, from what I read, it says that you had an issue
    with using the Java Service Wrapper only.

    Further to your comments about support of EPA scripts:
    This is again a question about support.
    I have to determine what issues can be addressed by technical support, what
    issues may need other CA departments to respond to, and what issues may be
    related to 3rd party problems.

    A customer can write any script that they like, and indeed use any script
    found on our community sites, or written by our technical consultants. That
    is not my point.

    However, those scripts do not go through any testing by our own engineers,
    so we do not know what effect they will have.

    This is why we cannot provide support for custom scripts, as we have no
    idea how they behave, especially in loaded environments.

    I am researching this with our management to provide an official statement
    for this.

    In terms of engaging Professional Services, this is normally done through
    contacting your CA Account Manager.
    Since you have direct access to our support portal, I am assuming you
    purchased the product directly from us. If you purchased the product from a
    reseller, it could be that the reseller has their our Account Manager who
    could assist you further.


    Please confirm if you are interested in me asking our product management to
    consider us supplying a Java Service Wrapper with the EPAgent. A response
    on this can take months, so I cannot give you any date on feedback for this.

    Regards,
    David Lewis
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    20.01.2011 15:18 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hi David !
    I think to comment on this issue..referencing a lot of other batch
    files in order to get metrics.
    David, are you think about it about 20 days ? You could write me about this
    much earlier..
    We can exclude a lots of metrics and leave a few metrics only and I did it..
    This is not the default configuration and looking into the batch files that
    have been written by you, that is where all of the processes are coming
    from.
    Why ?! Is it your personal or official opinion ? Can you give me links to
    official documentation ?
    "Environment Performance Agent Guide" (1) p. 9 - "The CA Wily Introscope
    Environment Performance Agent (EPAgent) integrates metric data from non-Java
    sources into Introscope"
    We cannot provide support for scripts written by yourself..
    I am sorry, mr. Lewis.. we speaking about Introscope EPA agent memory leaks..
    Our simplest scrips works fine.. without any problems..

    Ok, according to your opinion, customer can use only those scripts/programms
    that are included in the default Introscope EPA agent distribution. Am I
    right ? But, according p. 11 in (1):
    "..EPAgent can receive text from any executable entity in a subprocess
    (including compiled applications), the most flexible approach is to use a
    scripting environment such as Perl, KornShell.."
    In normal support, if engineer can not help for customer, this engineer
    should forward case to more experienced engineer or service.. For 20 days
    case leaving we have nothing.. You was supposed to write about it on the
    second day..

    Please, write me again you official commit about customer can not write user
    defined programms/scripts for EPA Agent plugin and shoud use only
    programms/scripts are included in Introscope EPA agent distribution.
    14.01.2011 13:41 DAVID LEWIS
    ..I will raise an enhancement request for EPAgent to be provided with
    supported Java Service Wrapper and that is all we can offer at this stage
    What about it ?
    We cannot provide support for scripts written by yourself but our
    Professional Services team work on this..
    Please, give me contacts for this Professional Services team..

    Regards, Igor Matukin


    20.01.2011 14:26 DAVID LEWIS
    Hi,
    I think to comment on this issue, if you look at your
    IntroscopeEPAgent.properties file, it is referencing a lot of other batch
    files in order to get metrics.

    #################################
    # Stateless Plugins
    #------------------

    introscope.epagent.plugins.stateless.names=BaNCSLogonCheck,BaNCSConnCount,Ba
    NCSHostRespTime,BaNCSLinkDBRespTime,WinPerfCPU,WinPerfMemory,WinPerfDisk,Win
    PerfNerwork,MSSQLPerf


    introscope.epagent.stateless.BaNCSLogonCheck.command=./epagent/epaplugins/cu
    stom/BaNCSLogonCheck.bat
    introscope.epagent.stateless.BaNCSLogonCheck.delayInSeconds=30


    introscope.epagent.stateless.BaNCSConnCount.command=./epagent/epaplugins/cus
    tom/BaNCSConnCount.bat
    introscope.epagent.stateless.BaNCSConnCount.delayInSeconds=30


    introscope.epagent.stateless.BaNCSHostRespTime.command=./epagent/epaplugins/
    custom/BaNCSHostRespTime.bat
    introscope.epagent.stateless.BaNCSHostRespTime.delayInSeconds=30


    introscope.epagent.stateless.BaNCSLinkDBRespTime.command=./epagent/epaplugin
    s/custom/BaNCSLinkDBRespTime.bat
    introscope.epagent.stateless.BaNCSLinkDBRespTime.delayInSeconds=30


    introscope.epagent.stateless.WinPerfCPU.command=./epagent/epaplugins/custom/
    WinPerfCPU.bat
    introscope.epagent.stateless.WinPerfCPU.delayInSeconds=30


    introscope.epagent.stateless.WinPerfMemory.command=./epagent/epaplugins/cust
    om/WinPerfMemory.bat
    introscope.epagent.stateless.WinPerfMemory.delayInSeconds=30


    introscope.epagent.stateless.WinPerfDisk.command=./epagent/epaplugins/custom
    /WinPerfDisk.bat
    introscope.epagent.stateless.WinPerfDisk.delayInSeconds=30


    introscope.epagent.stateless.WinPerfNerwork.command=./epagent/epaplugins/cus
    tom/WinPerfNetwork.bat
    introscope.epagent.stateless.WinPerfNerwork.delayInSeconds=30


    introscope.epagent.stateless.MSSQLPerf.command=./epagent/epaplugins/custom/M
    SSQLPerf.bat
    introscope.epagent.stateless.MSSQLPerf.delayInSeconds=30



    This is not the default configuration and looking into the batch files that
    have been written by you, that is where all of the processes are coming
    from.
    We cannot provide support for scripts written by yourself but our
    Professional Services team work on this sort of thing all of the time.
    If there are scripts missing from the original installation, we can ask for
    product enhancement that the files may be available by default.

    Please let us know how we could help you further with this?

    Kind regards,
    David Lewis
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    14.01.2011 14:07 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    05.01.2011 14:16 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !

    New and significant information..
    I run EPAgent as interactive process from command line (epagent_start.bat..
    see attached archive 19955536-1.zip)..
    And we have same problems such as run EPAgent as windows sevice..
    ...

    wyisc_environment_performance_agent9.0.5.0.pdf
    (Environment Performance Agent Guide)

    Running the EPAgent
    The EPAgent can be run as either a standalone .jar file, or a Java
    application.
    To run the EPAgent as a standalone .jar file:
    Run a Java command-line with the appropriate -jar flag, as in:
    java -jar <Introscope_Home>/lib/EPAgent.jar
    To run the EPAgent as a Java application:
    Add the EPAgent files to the appropriate classpath as in the following
    example:
    java -classpath "<Introscope_Home>/lib/EPAgent.jar"
    com.wily.introscope.api.IntroscopeEPAgent

    epagent_start.bat contents..:
    @ECHO OFF
    SET JAVA_HOME=C:\Program Files\Java\jre6
    SET PATH=%PATH%;"%JAVA_HOME%\bin"
    SET ISCP_HOME=C:\Program Files\CA Wily\Introscope9.0.5.0
    SET EPA_CONF="%ISCP_HOME%\IntroscopeEPAgent.properties"
    SET EPA_JAR="%ISCP_HOME%\epagent\lib\EPAgent.jar"
    java -Dcom.wily.introscope.epagent.properties=%EPA_CONF% -classpath %EPA_JAR%
    com.wily.introscope.api.IntroscopeEPAgent


    14.01.2011 13:57 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello mr. Lewis !

    As stated in my last response, we have same errors also in supported
    environment..
    Are you reading my first message only ? Can you read my ALL messages ?

    Regards, Igor Matukin


    14.01.2011 13:41 DAVID LEWIS
    Hello,
    As stated in my first response, your environment is not supported so any
    errors you see are most likely a result of this.
    If you are seeing any problems with running a supported configuration,
    please raise that issue.
    I will raise an enhancement request for EPAgent to be provided with
    supported Java Service Wrapper and that is all we can offer at this stage.

    David Lewis
    CA Support
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    14.01.2011 09:35 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Environment profile related data not found!


    14.01.2011 09:35 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello mr. Lewis !

    We have no any response from you about 10 days (from 04.01.2011) about case
    "19955536-1: EPA ORPHANED (CMD) PROCESSES"..

    Regards, Igor Matukin, B&N Bank, Russia
    Tel.: +7 495 7555060 ext. 2983


    05.01.2011 14:16 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !

    New and significant information..
    I run EPAgent as interactive process from command line (epagent_start.bat..
    see attached archive 19955536-1.zip)..
    And we have same problems such as run EPAgent as windows sevice..
    C:\Program Files\CA Wily\Introscope9.0.5.0>grep ERROR IntroscopeEPA.log
    ...
    1/05/11 01:53:49 PM MSK [ERROR] [EPAgent.EPAgent MSSQLPerf] The stateless
    plugin 'MSSQLPerf' has timed out.
    New timeout settings
    (introscope.epagent.config.stalledStatelessPluginTimeoutInSeconds=60) has no
    effect..

    Regards, Igor Matukin


    05.01.2011 12:12 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !

    I find property stalledStatelessPluginTimeoutInSeconds..
    # Time period in seconds that a stateless plugin is allowed to run before it
    # is considered stalled and forcefully killed. If commented out or left
    unconfigured
    # the default is 60 seconds. A value of zero will prevent stalled plugins from
    # being detected and killed.
    #
    introscope.epagent.config.stalledStatelessPluginTimeoutInSeconds=60

    But why timeouted child process does not killed ?
    I find message
    "1/05/11 11:07:23 AM MSK [INFO] [EPAgent.EPAgent WinPerfMemory] Plugin
    WinPerfMemory is stopping..."
    in log file.. And, really, orphaned child process exists.. not killed..
    I will try increase "delayInSeconds" property for plugins up to 60 sec and
    monitor EPAgent processes..
    The situation is strange .. you can run user defined plugins manually.. The
    plugins response time is small.. and equal for 2-3 seconds

    Regards, Igor Matukin


    05.01.2011 12:07 SECURE SERVER
    Entered by SFTP Server User PCS DESK
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: Email_2011_1_5_4_6_29.html


    05.01.2011 12:07 SECURE SERVER
    Entered by SFTP Server User PCS DESK
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: Email_2011_1_5_4_6_29.txt


    05.01.2011 11:44 SECURE SERVER
    Entered by SFTP Server User PCS DESK
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: Email_2011_1_5_3_43_59.txt


    05.01.2011 11:44 SECURE SERVER
    Entered by SFTP Server User PCS DESK
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: Email_2011_1_5_3_43_59.html


    05.01.2011 11:43 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !

    I have significant information about this case problem..
    I use DEBUL level for trace EPAgent work and find errors
    "1/05/11 11:07:23 AM MSK [ERROR] [EPAgent.EPAgent WinPerfMemory] The
    stateless plugin 'WinPerfMemory' has timed out"
    in log file.. And we have orphaned plugin cmd process in this case..
    And my questions:
    1. Is it this sutiation is normal ? Why timedout cmd process do not
    terminated by parent process ?
    2. What is it time for plugin ? Can we manage (increase) this time ?

    Regards, Igor Matukin


    05.01.2011 09:51 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Hello David !
    At this stage, we could only suggest raising an enhancement request such
    that running the EPAgent as a Windows service is officially supported.
    Please, do it.. Another ways for running EPAgent declared in "Environment
    Performance Agent Guide Version 9.0" (p. 25) such as
    - The EPAgent can be run as either a standalone .jar file, or a Java
    application
    - To run the EPAgent as a Java application
    have requirements run EPAgent only as interactive application from
    interactive logged user or needs third party software..
    Wrapper for application as windows service or unix daemon is good and common
    practice for monitoring agents..

    Regards, Igor Matukin


    04.01.2011 17:43 DAVID LEWIS
    ISSUE IS NO LONGER PENDING VIA TECHNICIAN'S ACTION


    04.01.2011 17:43 DAVID LEWIS
    ISSUE IS NOW PENDING VIA TECHNICIAN'S ACTION


    04.01.2011 17:42 DAVID LEWIS
    Next Action: Customer to verify solution Due Date:01072011 NY time


    04.01.2011 17:42 DAVID LEWIS
    Hello Igor,
    The running of the EPAgent on a Windows service is not supported. We
    provide guidelines on how it can be achieved but are not able to help with
    any issues encountered.

    At this stage, we could only suggest raising an enhancement request such
    that running the EPAgent as a Windows service is officially supported.

    Kind regards,
    David Lewis
    **** SEND MAIL TO imatukin@msk.binbank.ru SOLUTION OPSYS: NONE


    04.01.2011 17:07 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: 19955536-1_02.zip


    04.01.2011 16:53 DAVID LEWIS
    Next Action: Contact Customer Due Date:01062011 NY time


    04.01.2011 16:40 DAVID LEWIS
    CALL 1-005 FROM CAI TO CLIENT CALLBACK
    Analyzing logs


    04.01.2011 14:42 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: 19955536-1.zip


    04.01.2011 14:42 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    A file was uploaded to the SFTP server:
    sftp://supportftp.ca.com/0912550/19955536-01/files_from_customer
    File name: 19955536-1.zip

    04.01.2011 12:30 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    Product: CA Wily Introscope - MULTI-PLATFORM Release:9.05

    04.01.2011 12:30 SUPPORT PORTAL
    Entered by Support Portal User SERGEY FEDOROV
    We (B&N Bank) have problems with processes managed under EPA agent..
    Often we have orphaned batch (cmd) files runned from EPA agent.. (we can see
    these cmd processes by use sysinternals process explorer tool)
    The problem can be reproduced and monotored at real time..
    The problem occured only if EPA agent java process wrapped as NT service (
    free Java Service Wrapper 3.5.6 used). The problem was not occured if we run
    EPA agent java process interactively from command line.. We can send
    additional technical information about this problem.. (processes list, batch
    files, EPA conf file..etc.) Java and Java Service Wrapper updates are not
    resolve this problem.


  • 2.  RE: EPA ORPHANED (CMD) PROCESSES & CA Support Service

    Posted Mar 16, 2011 07:30 AM
    MAR 11 2011 11:03 SERGIO MORALES CORREA APMWLY GBL SS (TCC NTF)
    Dear Igor,

    ...I have performed further research and found that
    the issue is mentioned in the README 9.0.6.0 file, page 36:
    ftp://supportftp.ca.com/0912550/19955536-01/files_from_ca
    "Stalled stateless plugins may not stop properly - 2748, 27489, 27492:
    Due to a JRE issue, stalled stateless plugins are not properly killed on
    some platforms, and they continue to consume resources even after the
    EPAgent has been stopped. These orphan processes must be stopped manually."


  • 3.  RE: EPA ORPHANED (CMD) PROCESSES & CA Support Service

    Posted Mar 16, 2011 09:03 AM
    Yeah we have seen the same problem. I cant remember how we resolved it under windows besides fixing our scripts to stop anything crashing or hanging. One thing to watch out for is if your script kicks off another process (e.g. shell script kicking of another shell script or other process). In that case the EPAgent kills the first script but not the second. This has been a problem for a while I believe. Note as the note mentions this is a Java issue - you can prove it yourself by kicking off and external process in Java then killing the main process.

    Under AIX we do the following to protect ourselves, so that if our main script gets killed it kills its children

    trap 'kill_em' 0 1 2 3 9

    kill_em() {
    kill $procid
    sleep 10
    ps -p $procid >/dev/null 2>&1 && kill -9 $procid
    }

    someothercmd&
    procid=$!

    wait


  • 4.  RE: EPA ORPHANED (CMD) PROCESSES & CA Support Service

    Broadcom Employee
    Posted Mar 16, 2011 10:01 AM
    There's also a nice cleanup routine called saferkill that you can modiy in the vmstat script in the Document Library.

    -Hiko


  • 5.  RE: EPA ORPHANED (CMD) PROCESSES & CA Support Service

    Broadcom Employee
    Posted Mar 16, 2011 10:05 AM
    Sorry everyone; that subroutine is NOT in the vmstat script.
    It's in the old EPAgent startup script.
    I'll upload it shortly.


  • 6.  RE: EPA ORPHANED (CMD) PROCESSES & CA Support Service
    Best Answer

    Broadcom Employee
    Posted Mar 16, 2011 10:43 AM

    Here's the subroutine:

    >
    # example: saferKill vmstat VMSTAT.pid
    # this function takes 3 arguments
    # $1 is the name of the process to be kill, it is case insensitive
    # $2 is the name of the file containing the pid(s) of the process to kill,
    # $3 is used to trigger logging or not. A value of "log" triggers logging
    # there can be multiple pids as long as they are separated by a space
    # return 0 if it successfully killed all the processes, 1 otherwise
    saferKill() {

    if [ "$#" != "3" ] ; then

     

    echo "ERROR: Function saferKill takes 3 arguments"

     

    exit 1
    fi

     

    PROCESS_NAME="$1"

    PID_FILE_NAME="$2"

    LOG="$3"

    RETURN_VALUE=0

     

     

    if [ -s "$PID_FILE_NAME" ]; then
    for PID in `cat $PID_FILE_NAME`; do

     

    # echo "DEBUG: Getting pid $PID from file $PID_FILE_NAME"

     

    if ps -p "$PID" 2> /dev/null | grep -i "$PROCESS_NAME" > /dev/null 2>&1 ; then
    if [ "$LOG" = "log" ] ; then echo "INFO: Attempting to stop $PROCESS_NAME process (pid=$PID), this could take a couple of seconds..."; fi

    COUNT=0

    while [ $COUNT -lt 5 ] && kill -0 "$PID" > /dev/null 2>&1 ; do

     

    if [ $COUNT -lt 2 ]; then
    # try to kill the process twice with a normal kill
    # echo "DEBUG: kill $PID"
    kill "$PID"
    else
    # kill did not work, try kill -9
    # echo "DEBUG: kill -9 $PID"
    kill -9 "$PID"
    fi

     

    sleep 2

     

    COUNT=`expr $COUNT + 1`
    done
    if kill -0 "$PID" > /dev/null 2>&1; then

     

    RETURN_VALUE=1

    if [ $LOG = "log" ] ; then echo "ERROR: Unable to stop $PROCESS_NAME process (pid=$PID)"; fi
    else
    if [ $LOG = "log" ] ; then echo "INFO: Successfully stopped $PROCESS_NAME process (pid=$PID)"; fi

    fi
    # else
    # echo "DEBUG: Unable to stop $PROCESS_NAME process (pid=$PID), the process does not exist"
    fi

     

    done
    else
    if [ $LOG = "log" ] ; then echo "INFO: Process $PROCESS_NAME is not running" ; fi

    fi

     

    # delete the pid file
    rm -f "$PID_FILE_NAME"

     

    return "$RETURN_VALUE"
    }