DX Application Performance Management

Expand all | Collapse all

Agent load balancing

Anon Anon

Anon AnonJan 16, 2014 04:49 AM

  • 1.  Agent load balancing

    Posted Jan 14, 2014 10:28 AM

    If you connect v 9.1.4 agents directly to a 9.1.4 collector (and not the MOM) are they still candidates for MOM load balancing ?

    It looks like they are.

    We have got two new collectors to put live, with agent load of around 450 agents. I have got the collectors connected to a test MOM just for visibility.

    As part of the testing we do I want to make sure that one collector handles full load without issue, but for some reason approx 1/3 of the agents were always ending up on the secondary collector. I put verbose logging on and I can see that the MOM is rebalancing them.

    When I shut the MOM down the agents didn’t rebalance and stayed on the primary collector.

    It’s not a major issue, but it just went contrary to my understanding. I thought you needed to connect the agent to the MOM directly for load balancing to be in place ?

     

     



  • 2.  RE: Agent load balancing

    Broadcom Employee
    Posted Jan 14, 2014 06:12 PM
    dave.richards:

    If you connect v 9.1.4 agents directly to a 9.1.4 collector (and not the MOM) are they still candidates for MOM load balancing ?

    It looks like they are.

    We have got two new collectors to put live, with agent load of around 450 agents. I have got the collectors connected to a test MOM just for visibility.

    As part of the testing we do I want to make sure that one collector handles full load without issue, but for some reason approx 1/3 of the agents were always ending up on the secondary collector. I put verbose logging on and I can see that the MOM is rebalancing them.

    When I shut the MOM down the agents didn’t rebalance and stayed on the primary collector.

    It’s not a major issue, but it just went contrary to my understanding. I thought you needed to connect the agent to the MOM directly for load balancing to be in place ?

     

     


    Hello Dave,

    What you are describing is expected behaviour for 9.1.x load balancing & you can effectively force the load balancing to be disabled by increasing the 2 relevant properties to high values in the EM properties file on MOM & Collector:

    • introscope.enterprisemanager.loadbalancing.threshold
    • introscope.enterprisemanager.loadbalancing.interval

    Knowledge Base Article TEC594846 has further details

    https://comm.support.ca.com/?legacyid=TEC594846

     

    Regards,

     

     



  • 3.  RE: Agent load balancing

    Posted Jan 16, 2014 03:51 AM

    Thanks for that clarification.

    Can I ask one follow up question ?

    Will the MOM only rebalance the agents to the secondary collector defined in the agent profile ? In other words, no other collector other than the 2 defined collectors in the agent profiles are candidates for agent rebalancing.

    Or can the MOM decide to balance the agents to a completely different collector in the cluster, one that is not defined in the agent profile ?

     

    The reason I ask is that these two collectors are soon to be added to an existing live cluster with 7 collectors already in place.

     

     



  • 4.  RE: Agent load balancing

    Posted Jan 16, 2014 04:49 AM

    Yes MOM will decide it.

    Regards

    Vinay

     



  • 5.  RE: Agent load balancing

    Posted Jan 16, 2014 04:54 AM

    OK. So the only way we can "pin" agents to the collectors we want is to disable the load balancing in the way that is described in the KB article ?

    It is important that these agents are only ever connected to one of these 2 collectors. We do not want the agents to go anywhere else.

     



  • 6.  RE: [CA APM General Discussion] RE: Agent load balancing

    Posted Jan 16, 2014 04:57 AM
    Agent is reporting first MOM and MOM will decide which Collector is responsible for load balancing.



    Regards,

    Vinay Nayak



    From: CA APM/Wily Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
    Sent: Thursday, January 16, 2014 3:24 PM
    To: mb_message.2307983.107626830@myca-email.ca.com
    Subject: [CA APM General Discussion] RE: Agent load balancing



    OK. So the only way we can "pin" agents to the collectors we want is to disable the load balancing in the way that is described in the KB article ?

    It is important that these agents are only ever connected to one of these 2 collectors. We do not want the agents to go anywhere else.



    Posted by:dave.richards
    --
    CA Communities Message Boards
    107629370
    mb_message.2307983.107626830@myca-email.ca.com
    https://communities.ca.com


  • 7.  RE: [CA APM General Discussion] RE: Agent load balancing

    Posted Jan 16, 2014 05:02 AM

    Vijay - we are setting the agent profile to connect to a collector, not to the MOM. So the initial connection is to a collector as far as I understand. Then the MOM decides to rebalance later.

     

    That behaviour is understood now, but what I am not clear on is if the MOM can decide to send agents to a collector that is not defined in the agent's profile ?



  • 8.  RE: [CA APM General Discussion] RE: [CA APM General Discussion] RE: Agent l

    Posted Jan 16, 2014 05:04 AM
    If you are using load balancing it will happen automatic.



    Regards,

    Vinay Nayak



    From: CA APM/Wily Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
    Sent: Thursday, January 16, 2014 3:32 PM
    To: mb_message.2307983.107626875@myca-email.ca.com
    Subject: [CA APM General Discussion] RE: [CA APM General Discussion] RE: Agent load balancing



    Vijay - we are setting the agent profile to connect to a collector, not to the MOM. So the initial connection is to a collector as far as I understand. Then the MOM decides to rebalance later.



    That behaviour is understood now, but what I am not clear on is if the MOM can decide to send agents to a collector that is not defined in the agent's profile ?

    Posted by:dave.richards
    --
    CA Communities Message Boards
    107629415
    mb_message.2307983.107626875@myca-email.ca.com
    https://communities.ca.com


  • 9.  RE: Agent load balancing

    Broadcom Employee
    Posted Jan 16, 2014 05:06 AM

    Hi Richard,

    If you want to connect particular agents to specific collectors I would suggest to use  "exclude" and "include" switches when using loadbalancing feature. Please see below examples (extracted from the Config and Admin guide):

     

    Agent Load Balancing Configuration Examples

    You can configure the MOM agent load balancing properties in varied ways. These configuration examples provide code samples:

    Assign an Agent to a Particular Collector or Set of Collectors

    The following example shows how to configure assigning an agent named MyAgent to the Collector named "myhost" on port 5001.

    <agent-collector name="Assign MyAgent to One Collector Example">
    
    <agent-specifier>.*\|.*\|MyAgent</agent-specifier>
    <include>
    
    <collector host="myhost" port="5001"/>
    
    </include>
    
    </agent-collector>
    

    The following example shows how to configure the latched property where:

    • Agents are assigned to connect to the Collector named "newCollector-ip" on port 6001.
    • When newCollector-ip is unavailable, the MOM can assign all agents with "EPAgent" in their name to either Collector1-ip or Collector2-ip.
      <agent-Collector name="Latch when Collector Unavailable Example">
      
      <agent-specifier>.*\|.*\|EPAgent.*</agent-specifier>
      <include>
      
      <Collector host="newCollector-ip" port="6001" latched=”true” />
      <Collector host="Collector1-ip" port="6001" />
      <Collector host="Collector2-ip" port="6001" />
      
      </include>
      
      </agent-Collector>
      

    The following example shows how to assign agents where:

    • Collectors are named "localhost" on ports 5001 and 5002.
    • Collector localhost:5001 is preferred over localhost:5002.
      <agent-collector name="Allow EPAgents on Two Collectors and One is Preferred Example">
      
      <agent-specifier>.*\|.*\|EPAgent.*</agent-specifier>
      <include>
      
      <collector host="localhost" port="5001" latched="true"/>
      <collector host="localhost" port="5002"/>
      
      </include>
      
      </agent-collector>
      

    The following example shows how to assign agents where:

    • Collectors are named "localhost" on ports 5001, 5002, 5003, and 5004.
    • Collectors localhost:5001 and localhost:5003 are preferred over localhost:5002 and localhost:5004.
      <agent-collector name="Allow EPAgents on Four Collectors and Two are Preferred Example">
      
      <agent-specifier>.*\|.*\|EPAgent.*</agent-specifier>
      <include>
      
      <collector host="localhost" port="5001" latched="true"/>
      <collector host="localhost" port="5002"/>
      <collector host="localhost" port="5003" latched="true"/>
      <collector host="localhost" port="5004"/>
      
      </include>
      
      </agent-collector>
      

    Exclude an Agent from a Particular Collector or Set of Collectors

    The following example shows how to exclude an agent named MyAgent from a Collector named MyHost.

    <agent-collector name="Exclude MyAgent from a Collector Example">
    
    <agent-specifier>.*\|.*\|MyAgent</agent-specifier>
    <exclude>
    
    <collector host="MyHost" port="5001"/>
    
    </exclude>
    
    </agent-collector>
    

    The following example shows how to exclude multiple agents from multiple Collectors.

    <agent-collector name="Exclude Two Agents Assigned from Two Collectors Example>
    
    <agent-specifier>.*\|.*\|MyAgent.*</agent-specifier>
    <agent-specifier>.*\|.*\|MyOtherAgent.*</agent-specifier>
    <exclude>
    
    <collector host="MyHost" port="5001"/>
    <collector host="MyOtherHost" port="5001"/>
    
    </exclude>
    
    </agent-collector>
    

    Assign Multiple Agents to Multiple Collectors

    The following example shows how to assign multiple agents to multiple Collectors.

    <agent-collector name="Two Agents Assigned to Two Collectors Example>
    
    <agent-specifier>.*\|.*\|MyAgent.*</agent-specifier>
    <agent-specifier>.*\|.*\|MyOtherAgent.*</agent-specifier>
    <include>
    
    <collector host="MyHost" port="5001"/>
    <collector host="MyOtherHost" port="5001"/>
    
    </include>
    
    </agent-collector>
    

     

     

    Thanks,

    Jose

     



  • 10.  RE: Agent load balancing

    Posted Jan 16, 2014 07:46 AM

    Thanks Jose.

    I think we will use the "crude" way of disabling MOM load balancing by modifying the values below, as per the KB article.

    introscope.enterprisemanager.loadbalancing.threshold=20000

    introscope.enterprisemanager.loadbalancing.interval=600

     

    As an observation though, I don't think the documentation on 9.1.4 is clear on this.

    The MOM config file itself expclitly states that  "Only version 8.0 or later agents that connected to the MOM are eligible for rebalancing."
    As we are NOT connecting to the MOM in our agent profile we expected the agents to connect and stay where we told them to go.

    Thanks for your help all.

     

    Extract from MOM config file.......

    # Define MOM Load Balancing properties.
    #
    # Load Balancing properties support hot config.
    #
    # The tolerance of the Load Balancer for imbalance.
    # The number of metrics in a Collector must differ from the weight-adjusted
    # cluster average (i.e. the average for all Collectors) by at least the
    # threshold before the Load Balancer rebalances the Cluster.
    #
    # Rebalancing means redistributing Agents among Collectors
    # to equalize the weight-adjusted metric load.
    # See the connection properties about setting weights.
    #
    # Only version 8.0 or later agents that connected to the MOM
    # are eligible for rebalancing.
    #



  • 11.  RE: Agent load balancing

    Posted Jan 16, 2014 08:58 AM

    I have been doing some further testing on this scenario, and I have found an undocumented MOM config setting

    introscope.enterprisemanager.loadbalancer.enable

    This became visble to me after I changed the agent max limit to 500 from 400 and the config was reloaded and this metric was shown as set to true.

    I have set to this to be false.

    I shut down the MOM and the secondary collector, so that 460 agents were on the primary.

    Started the secondary collector. Load remained the same - 463, 0

    Then I started the MOM and the agents did NOT rebalance.

    So it looks like this setting in the config file disables load balancing on the MOM, which is what we want.

     

    Can anyone on CA support confim this is OK to use, as it is undocumented and there is no information on ca.com either ?



  • 12.  RE: Agent load balancing

    Posted Jan 16, 2014 11:55 AM

    Dave,

    I am into this thread a bit late but I think that if you are pointing the agent to a collector (or set of collectors with backup connects set up in the Agent profile) then they will never particiapte in load balancing since. again to my understanding, only those agent pointed to the MoM will be load balanced.

    So turning of Loadbalancing in the MoM will turn it off for all your other collectors in the cluster so what I would do is to exclude these collectors in the loadbalancing xml (as explained above) and continue to point your agents that you want for these two collectors via the agent profile. 

    This should leave these two collectors out ot the loadbalaning from the MoM and allow you to segragate your agents as you desire.

     

    Steve Oak



  • 13.  RE: [CA APM General Discussion] RE: Agent load balancing

    Posted Jan 16, 2014 12:24 PM
    Steve,

    You wrote:
    “I think that if you are pointing the agent to a collector (or set of collectors with backup connects set up in the Agent profile) then they will never particiapte in load balancing since. again to my understanding, only those agent pointed to the MoM will be load balanced.”

    That’s not accurate anymore, this was the old behavior, nowadays Agents are getting load-balanced by default even without being pointed to the MOM.

    Regs,

    Florian.


  • 14.  RE: [CA APM General Discussion] RE: Agent load balancing

    Posted Jan 17, 2014 07:45 AM

    Hi

    Is anyone from CA ok to comment on the hidden parameter below ? It seems to do the job that we want but as its undocumented (as far as I can see) I want to know if we are OK to use this.

    We don't want agents to be loadbalanced at the moment as we have plans to move collectors to new MOMs and we want all the agents to go over as well.

    introscope.enterprisemanager.loadbalancer.enable=false

    Thanks



  • 15.  RE: [CA APM General Discussion] RE: Agent load balancing
    Best Answer

    Broadcom Employee
    Posted Jan 19, 2014 07:45 PM
    dave.richards:

    Hi

    Is anyone from CA ok to comment on the hidden parameter below ? It seems to do the job that we want but as its undocumented (as far as I can see) I want to know if we are OK to use this.

    We don't want agents to be loadbalanced at the moment as we have plans to move collectors to new MOMs and we want all the agents to go over as well.

    introscope.enterprisemanager.loadbalancer.enable=false

    Thanks

    Hi Dave,

    From what I have researched "introscope.enterprisemanager.loadbalancer.enable" is a new hidden property added in 9.1 and defaults to true except for the new Cross-Cluster Data Viewer (CDV) functionality.

    Setting it to false should disable the load balancing as you have found.

    Regards,

     



  • 16.  RE: [CA APM General Discussion] RE: Agent load balancing

    Posted Jan 20, 2014 07:29 AM

    OK, thanks Lynn and thanks to everyone else who has contributed to the thread.

     

    As an aside, the email notification feature of the CA APM messageboard is really useful so thanks also to whoever enabled that.