Openshift Infra Agent - setting min/max heap ?

Discussion created by dave.richards on Apr 10, 2018
Latest reply on Apr 11, 2018 by dave.richards

Like • Show 0 Likes0
Comment • 6

How can we modify the min/max heap settings for this agent ?We dont want to rebuiild the agent.

Looking at the startup scripts we can see that the min/max values can be passed as params, but how to do this ?

As we have a large estate the agent is crashing due to default heap of 16/256 being too small

./APMIAgent.sh startwithlogs: APM Infrastructure Agent started

APM Infrastructure Agent Start is Completed

Tue Apr 10 11:41:45 UTC 2018 [INFO] APM Infrastructure Agent Start is Completed

4/10/18 11:41:51 AM UTC [INFO] [IntroscopeAgent.OSEMetadataInfo] Operation Name = null

4/10/18 11:41:58 AM UTC [INFO] [IntroscopeAgent.IntelligentInstrumentationService] Intelligent Instrumentation Service activated

4/10/18 11:42:43 AM UTC [ERROR] [IntroscopeAgent.InfrastructureAgent Agent] Error on getting last deployment status for heapster

4/10/18 11:42:48 AM UTC [INFO] [IntroscopeAgent.InfrastructureAgent Agent] AutoProbe has reloaded all directive configuration. This is because a change to a directive configuration file was detected.

4/10/18 11:43:02 AM UTC [ERROR] [IntroscopeAgent.OutgoingRouteConnector] Closing outgoing route connector due to unrecoverable error.

java.lang.OutOfMemoryError: GC overhead limit exceeded

4/10/18 11:43:02 AM UTC [ERROR] [IntroscopeAgent.InfrastructureAgent Agent] The thread named "UnknownHub Hub Transmit 2" has unexpectedly terminated because of java.lang.OutOfMemoryError GC overhead limit exceeded

4/10/18 11:43:09 AM UTC [ERROR] [IntroscopeAgent.InfrastructureAgent Agent] The thread named "Thread-3" has unexpectedly terminated because of java.lang.OutOfMemoryError GC overhead limit exceeded

The pod remains Running. We have attempted a couple of restarts, all produce the same results.

When reviewing the container, I can see the /usr/local/openshift/apmia/run.sh calls apmia-ca-installer.sh which in turn call bin/APMIAgent.sh

APMIAgent.sh takes default min/max heap from within the script as it expects them to be passed as arguments when being executed ($2 and $3)

Is it possible to set these as ENV variables?

Outcomes

mike_cook

Apr 10, 2018 1:20 PM

Hi Dave,

Funnily enough I was looking at the same thing for the collector agent. Yes you can is the answer

If you define a couple of variables MIN and MAX than call teh script with them as follows

# export MAX=512
# export MIN=128
./*sh start $MIN $MAX
Starting /collectors/CollectorAgent-10.5.2/jre/bin/java -server -Xms128m -Xmx512m -cp lib/CollectorAgent.jar:lib/gson.jar:lib/Agent.jar -XX:ErrorFile=/collectors/CollectorAgent-10.5.2/logs/jvm_error.%p.log com.wily.introscope.agent.collector.CollectorAgent
./CollectorAgent.sh start: CollectorAgent (pid 15426) started

If they are not set and you still call it with those arguments it just uses the default as the $2 $3 are missing

# unset MIN
# unset MAX
# ./*sh start $MIN $MAX
Starting /collectors/CollectorAgent-10.5.2/jre/bin/java -server -Xms16m -Xmx256m -cp lib/CollectorAgent.jar:lib/gson.jar:lib/Agent.jar -XX:ErrorFile=/collectors/CollectorAgent-10.5.2/logs/jvm_error.%p.log com.wily.introscope.agent.collector.CollectorAgent
./CollectorAgent.sh start: CollectorAgent (pid 15478) started

hope this helps

cheers

Mike

3 people found this helpful

Actions

dave.richards @ mike_cook on Apr 10, 2018 1:45 PM

Hi Mike, its not what we want. As the supplied "run "script is already running inside the Docker image we cant pass any parameters to do it without rebuilding the Docker image locally, which we would rather not do. Ideally we thought the min/max values might be able to get picked up from the Openshift environment where we can set them, rather than as command line args that we can't set

Actions

mike_cook

@ dave.richards on Apr 10, 2018 1:54 PM

Hi Dave,

I wont pretend to be an Openshift/Docker expert but if you have a script in the docker image that references variable names like in my example MIN MAX then you can set these externally to values or not. I admit this would need a rebuild to get it to be able to read the variables in the first place, but only once to set the mechanism up, you could then change the variable values in Openshift (if I understand you correctly).

If I'm still missing something - apologies for that.

cheers

Mike

2 people found this helpful

Actions

dave.richards @ mike_cook on Apr 10, 2018 2:05 PM

Hi Mike, it looks to us like the actual agent startup script that is supplied can only take the values as command line arguments. It doesn't appear to try to get them from the environment. So if there is no way to get them from the environment, we are going to have to modify the supplied image and rebuild that. We think we have got a workaround, but just wanted to know from CA if we were missing something.

2 people found this helpful

Actions

Lynn_Williams

@ dave.richards on Apr 11, 2018 2:00 AM

Hi Dave,

Mike is saying that you will need to edit the OOTB APMIAgent.sh script to be able to process the values of MIN & MAX as environment variables. I do agree that you would also need to disable the argument checking for the heap values. The following changes work for me:

OLD:

====

# changes for passing heap values in arguments
MIN_HEAP_VAL_IN_MB=16
MAX_HEAP_VAL_IN_MB=256

MIN_ARG_PRESENT=true
if [ -z "$2" ]
then
MIN_ARG_PRESENT=false
fi

MAX_ARG_PRESENT=true
if [ -z "$3" ]
then
MAX_ARG_PRESENT=false
fi

====

NEW:

====

# modified to pass in heap values as environment variables
MIN_HEAP_VAL_IN_MB=16
MAX_HEAP_VAL_IN_MB=256

if [[ "$MIN" ]]; then
echo "Setting MIN_HEAP_VAL_IN_MB to environment variable value for MIN"
MIN_HEAP_VAL_IN_MB=$MIN
fi

if [[ "$MAX" ]]; then
echo "Setting MAX_HEAP_VAL_IN_MB to environment variable value for MAX"
MAX_HEAP_VAL_IN_MB=$MAX
fi

# disable heap argument checking

MIN_ARG_PRESENT=false
MAX_ARG_PRESENT=false

#MIN_ARG_PRESENT=true
#if [ -z "$2" ]
# then
# MIN_ARG_PRESENT=false
#fi

#MAX_ARG_PRESENT=true
#if [ -z "$3" ]
# then
# MAX_ARG_PRESENT=false
#fi

====

Hope that helps

Regards,

Lynn

1 person found this helpful

Actions

dave.richards @ Lynn_Williams on Apr 11, 2018 8:00 AM

Thanks Lynn and Mike

Ok we understand what is required now. We've spoken to Nishant also, and we understand that there will be a mod to the script released at some point so the heap size can be picked up from the environment without having to amend the supplied script.

Actions

6 Replies

mike_cook Apr 10, 2018 1:20 PM
Hi Dave,

Funnily enough I was looking at the same thing for the collector agent. Yes you can is the answer

If you define a couple of variables MIN and MAX than call teh script with them as follows

# export MAX=512
# export MIN=128
./*sh start $MIN $MAX
Starting /collectors/CollectorAgent-10.5.2/jre/bin/java -server -Xms128m -Xmx512m -cp lib/CollectorAgent.jar:lib/gson.jar:lib/Agent.jar -XX:ErrorFile=/collectors/CollectorAgent-10.5.2/logs/jvm_error.%p.log com.wily.introscope.agent.collector.CollectorAgent
./CollectorAgent.sh start: CollectorAgent (pid 15426) started

If they are not set and you still call it with those arguments it just uses the default as the $2 $3 are missing

# unset MIN
# unset MAX
# ./*sh start $MIN $MAX
Starting /collectors/CollectorAgent-10.5.2/jre/bin/java -server -Xms16m -Xmx256m -cp lib/CollectorAgent.jar:lib/gson.jar:lib/Agent.jar -XX:ErrorFile=/collectors/CollectorAgent-10.5.2/logs/jvm_error.%p.log com.wily.introscope.agent.collector.CollectorAgent
./CollectorAgent.sh start: CollectorAgent (pid 15478) started

hope this helps

cheers
Mike
3 people found this helpful
Like • Show 2 Likes2
Actions
- dave.richards @ mike_cook on Apr 10, 2018 1:45 PM
  Hi Mike, its not what we want. As the supplied "run "script is already running inside the Docker image we cant pass any parameters to do it without rebuilding the Docker image locally, which we would rather not do. Ideally we thought the min/max values might be able to get picked up from the Openshift environment where we can set them, rather than as command line args that we can't set
  Like • Show 0 Likes0
  Actions
  - mike_cook @ dave.richards on Apr 10, 2018 1:54 PM
    Hi Dave,
    
    I wont pretend to be an Openshift/Docker expert but if you have a script in the docker image that references variable names like in my example MIN MAX then you can set these externally to values or not. I admit this would need a rebuild to get it to be able to read the variables in the first place, but only once to set the mechanism up, you could then change the variable values in Openshift (if I understand you correctly).
    
    If I'm still missing something - apologies for that.
    
    cheers
    Mike
    2 people found this helpful
    Like • Show 1 Like1
    Actions
    - dave.richards @ mike_cook on Apr 10, 2018 2:05 PM
      Hi Mike, it looks to us like the actual agent startup script that is supplied can only take the values as command line arguments. It doesn't appear to try to get them from the environment. So if there is no way to get them from the environment, we are going to have to modify the supplied image and rebuild that. We think we have got a workaround, but just wanted to know from CA if we were missing something.
      2 people found this helpful
      Like • Show 1 Like1
      Actions
      - Lynn_Williams @ dave.richards on Apr 11, 2018 2:00 AM
        Hi Dave,
        Mike is saying that you will need to edit the OOTB APMIAgent.sh script to be able to process the values of MIN & MAX as environment variables. I do agree that you would also need to disable the argument checking for the heap values. The following changes work for me:
        
        OLD:
        ====
        # changes for passing heap values in arguments
        MIN_HEAP_VAL_IN_MB=16
        MAX_HEAP_VAL_IN_MB=256
        
        MIN_ARG_PRESENT=true
        if [ -z "$2" ]
        then
        MIN_ARG_PRESENT=false
        fi
        
        MAX_ARG_PRESENT=true
        if [ -z "$3" ]
        then
        MAX_ARG_PRESENT=false
        fi
        ====
        
        NEW:
        ====
        # modified to pass in heap values as environment variables
        MIN_HEAP_VAL_IN_MB=16
        MAX_HEAP_VAL_IN_MB=256
        
        if [[ "$MIN" ]]; then
        echo "Setting MIN_HEAP_VAL_IN_MB to environment variable value for MIN"
        MIN_HEAP_VAL_IN_MB=$MIN
        fi
        
        if [[ "$MAX" ]]; then
        echo "Setting MAX_HEAP_VAL_IN_MB to environment variable value for MAX"
        MAX_HEAP_VAL_IN_MB=$MAX
        fi
        
        # disable heap argument checking
        
        MIN_ARG_PRESENT=false
        MAX_ARG_PRESENT=false
        
        #MIN_ARG_PRESENT=true
        #if [ -z "$2" ]
        # then
        # MIN_ARG_PRESENT=false
        #fi
        
        #MAX_ARG_PRESENT=true
        #if [ -z "$3" ]
        # then
        # MAX_ARG_PRESENT=false
        #fi
        ====
        
        Hope that helps
        
        Regards,
        
        Lynn
        1 person found this helpful
        Like • Show 0 Likes0
        Actions
        dave.richards @ Lynn_Williams on Apr 11, 2018 8:00 AM
        Thanks Lynn and Mike
        
        Ok we understand what is required now. We've spoken to Nishant also, and we understand that there will be a mod to the script released at some point so the heap size can be picked up from the environment without having to amend the supplied script.
        Like • Show 0 Likes0
        Actions

Openshift Infra Agent - setting min/max heap ?

Outcomes

Related Content

Recommended Content

Incoming Links