DX Application Performance Management

Hi everyone.

I have a big CA Introscope deployment monitoring about 200 Tibco BusinessWorks agents running on 5 Tibco BusinessWorks servers, all this agents belongs to the same business application. As you can imagine, agent naming is an important issue, since there are hundreds of agents and assigning a name to each is a long and complicate task.

To avoid the process of individual agent naming, we're using the autoname feature for Tibco BusinessWorks, which is really good giving the agents a meaningful name (from the final user point of view) and is easy to manage for the Tibco administrators, they only have to add two properties to the bwengine.tra: [font=Courier New]java.extended.properties=-javaagent:<Agent_Home>/Agent.jar ‑Dcom.wily.introscope.agentProfile=<Agent_Home>/core/config/IntroscopeAgent.profile[font].

This configuration allows us only modify one file per Tibco server, instead of modifying 200 tra files if we were configuring each instance, and after the bwengine.tra configuration, the administrator guys only redeploy instances and automatically they are monitored by Introscope Agents.

But, this feature has a big inconvenient for our client: all agents take 2 minutes to start with their default name (UnknownAgent, AgentName, for example) and get connected to the Enterprise Manager and 8 or 10 additional minutes to get their real name, 10 minutes is the best configuration I've had. For our client, this is not a good behavior because they lose about 10 minutes of metrics, yes, after an agent renaming the metrics are lost and stay within the agent with the old (and grey) name so we can never take them in case of a root cause analysis for some problem.

In order to avoid losing this metrics, we tried to assign a name for each agent in the <application>.tra file but it is a hard and time consuming task, even worse, after an application is redeployed that property is lost and the Tibco administrators would need to repeat the configuration each time they redeploy each of the 200 applications.

Finally this is my question:

Is there any smarter way to assign names to each agent without the need of waiting 10 minutes to get good names? also without assigning individual names?, I've realized that Tibco needs about 10 minutes to get their automatic naming mechanism working, but the instances names are assigned from the creation time, I mean, the instance name is given by the administrator/developer when it's created, so could be any way to get it maybe as a variable in the bwengine.tra such like:

[font=Courier New]-Dcom.wily.introscope.agent.agentName=$InstanceName[font]

Do you have any experience with Tibco and Introscope?, if not, do you have any idea to workaround this issue?

In the worst case, is there any way to get those metrics when the agent automatically changes it's name?

Thanks in advance,

Hi,

We also have CA Introscope deployment monitoring for monitoring tibco and we have the same issue, but it started to happen when we upgrade to IBM jdk 1.6 before that we didn't had this issue unless we start all BW engines off on machine at the same time .

So from my experience we map this issue to the machine resources. Have you tested just stopping and starting one BW?

Telmo

Hi Telmo, thanks for answering.

This deployment runs over Tibco 5.8 and JVM HotSpot 1.6 and we have always had this issue. Our agents always take about 10 minutes to get their name, even if we restart only one, it always takes a long time.

Have you tried any workaround to set a static name to the instance?, I know that there under Tibco|Jobs metrics in the Investigator tree, there exists a node with the instance name; also, each application is deployed in a directory with the instance name, so I think that there are some options to set this name by using -Dcom.wily.introscope.agent.agentName, maybe something like:

-Dcom.wily.introscope.agent.agentName=$directoryName

Any idea?

Hi,

You can do this. On IntroscopeAgent.profile you can set the property "introscope.agent.agentNameSystemPropertyKey" with the name of the java sytem property that has the agent name. And on bwengine.xml you put the property to enable it to be changed on tibco administreator.

Example:
Changes on IntroscopeAgent.profile:
Set introscope.agent.agentNameSystemPropertyKey=AgentName

Changes on bwengine.xml add the fowling lines:
<property>
<name>CA APM auto naming</name>
<option>java.property.AgentName</option>
<default>BW_NAME</default>
<description>Defines CA APM agent NAME</description>
</property>

Generate again the EAR on the machine you changed the bwengine.xml, deploy the EAR on tibco, on the tibco administartor put the name you want for this BW in CA APM, start the BW engine.

Please let me know if it worked for you.

Telmo

Hi.

I checked for the bwengine.xml but for this application the file only exists per each Tibco Server, but we have a lot of instances inside. I'm not a Tibco developer, but could we create a bwengine.xml for each Tibco app we deploy into the Tibco server?

Thanks,

Hi,

You just need to change the bwengins.xml on the machine that you compile the code. The easy way is to have 1 machine to compile all EAR of tibco and then you deploy them, If you talk with a Tibco Administrator they can explain better this is a Tibco process for including tibco variables available on the administrator

Telmo

Uhh, I'll try this with the Tibco guys here and will let you know if this worked.

Thanks!

It seems that this solution will not work us. There are a lot of developers and machines to be configured and the client didn't like this approach cause it will depend on the developer to set the instance name for the agent and could lead to errors.

I've received a reply from Tibco in which they say there are not an easy solution, but they gave us an option to analyze:

Currently there is no readymade variable to get access to application instance name. But if you look at the appliction TRA file it will have the entry as below:-

application.args=%APP_ARGS% -p "C\:/TIBCO_HOME/tra/domain/<yourdomain>/application/<yourproject>/TestProject1-Process_Archive.tra" -name TestProject1-Process_Archive

The -name parameter refers to the applicationInstance name. So maybe you can use this name to pass it for your monitoring application.

I'll check if there is something that works.

Regards,

We have the same type of problem, [s]although i think our engines spin up a little faster when they're done individually[s] we have three versions of the engines running so this makes sense, and in a large environment like yours (mine's the same) you may just have to take the hit and ask your developers to stop trying to deploy applications in the same window you need metrics from :D
Have you confirmed the total loss of metrics when this happens? I was under the impression that the EMs would marry the two up so that you didnt lose actual data and just had the left over generic agent?
I should look into this as well at some point once i get through the other hurdles I have with wily and general work within tibco.

Thanks for your response.

Yes, this is a big Tibco deployment, about 80 JVM instances per TIbco server, but in my case, there are always a long time until the agent gets its name, doesn't matter if there is one deployment or twenty, even worse, just a JVM restart makes the agent lose its name and it takes 10 minutes again.

About metrics losing, I also have the impression that some test I perfomed a few weeks ago showed me the "historical" metrics attached to the new agent name. Now, in the last tests, when the agent goes form AgentName%<number> to its final name, I lost the metrics, I mean, I don't see the AgentName%<number> metrics anymore, just in the investigator for a few hours until the gray and inactive agent disappears.

BTW, this environment I'm testing is not a development one, this is preproduction with a few deployments every day.

Regards,

Hi,

as said above there are four ways to set the agent name:

[list]
[*]Agent name specified in a Java system property: -Dcom.wily.introscope.agent.agentName= (command line)
[*]Agent name specified in a system property key: introscope.agent.agentNameSystemPropertyKey= (agent profile)
[*]Agent name obtained automatically from the application server (Tibco BW)
[*]Agent name specified explicitly in the agent profile: introscope.agent.agentName=
[list]

Autonaming waits until the name is available from the app server (Tibco BW).

To remediate the "lost metrics during startup time" problem you can mount the old agent after it has grayed out: click Manager->Mount Agent... in the Investigator. You need to have the permission historical_agent_control to do that.

BR,
Guenter

Hi Grossberger, thanks for your reply.

It seems I have no luck with this feature, TIBCO says there are no variables with the instance name that we can catch at deployment time.

The issue with agent names is that we have more thatn 200 and a lot of custom dashboards with metrics grouping filtered by the agent name, so those dashs only shows metrics when the agents have their final name and previous metrics are only visible in the investigator.

Aas you could realize, we also have tons of AgentName%## agents to mount. For example, after a complete application servers restart we got 200 AgentName%# agents, after 10 minutes they get its real name and we have 200%# AgentName%# gray agents that will be unmounted in the next hour. Becase of this, is really difficult to find the proper one and mount it.

Regards,

HI Carlos,

I understand your problem but there is no simple solution in this case.

The only solution I can think of was already mentioned before: Your developers should set the name in a sytem property that the agent can refer to with introscope.agent.agentNameSystemPropertyKey=<agentNameProperty>.

This is not as simple as autonaming and requires discipline and some effort in your organization.

Sorry, can't help you more on that,
Guenter