DX Application Performance Management

Hi,

I'm having an issue trying to monitor certain PerfMon metrics.... I'm pretty certain it has to do with the Syntax.

Basically, processor is one of the key PerfMon metrics we need. If I add the additonal "out of the box" .NET and ASP metrics, this sometimes causes the Processor metric to dissappear. Is there any trick to adding additional PerfMon metrics?

Thanks all,

Mike

Try adding individual metrics one at a time to the filterpattern to see what may be causing the problem.

I was doing that... its very clear what metric added (again - out of the box) takes away Processor, but I can't figure out why.

Let me post the details.

I'll check this post again when you've updated with your filterpattern.

Thanks for the assistance.

Here is some context around the problem. The server(s) are Windows 2003 Standard x64, running MS BizTalk 2008. Two servers in a load balanced cluster.

Here is the PerfMon filter line:

introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.uNET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*

With that line and specifically on these two servers, I get these metrics:

.NET CLR Exceptions

.Net CLR Jit

.NET CLR Loading

.NET CLR Security

.NET Data Provider for SqlServer

I need to test a few more things.... but if I remove all except the |Processor|*|* line I do get the "Processor" metrics. If I start adding the other metrics i lose "processor" piece. I'll give details in a bit. I'm pretty sure its the expression in some way.

Thanks again,

Mike

Here is my filter for BizTalk 2009/2010, based upon the TechNet documentation related to monitoring bottlenecks in BizTalk tier:

introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*,|Processor|_Total|% Processor Time,|Process|BTSNTSvc|Private Bytes,|Process|BTSNTSvc|Virtual Bytes,|Process|BTSNTSvc|*Count,|PhysicalDisk|*|% Idle Time,|PhysicalDisk|*|Avg. Disk Queue Length,|BizTalk:Message Agent|*|Database size,|BizTalk:Message Agent|*|Message *throttling state,|BizTalk:Message Box:General Counters|*|Spool Size,|BizTalk:Message Box:General Counters|*|Tracking Data Size,|BizTalk:Message Box:General Counters|*|Tracking Spool Cleanup,|BizTalk:Message Box:Host Counters|*|Host Queue \- Length,|BizTalk:Messaging|*|Documents processed\/Sec,|BizTalk:Messaging|*|Documents received\/Sec,|XLANG\/s Orchestrations|*|Orchestrations completed\/sec,|XLANG\/s Orchestrations|*|Orchestrations dehydrated\/sec,|XLANG\/s Orchestrations|*|Orchestrations rehydrated\/sec,|XLANG\/s Orchestrations|*|Persistence points\/sec,|XLANG\/s Orchestrations|*|Orchestrations resident in memory,|XLANG\/s Orchestrations|*|Orchestrations suspended\/sec,|XLANG\/s Orchestrations|*|Transactional scopes committed\/sec,|XLANG\/s Orchestrations|*|Transactional scopes aborted\/sec,|XLANG\/s Orchestrations|*|Transactional scopes compensated\/sec

I'm going to attach the filter in a file since the forum may not show the entire filterpattern.

You can find more info about monitoring BizTalk in TechNet: http://msdn.microsoft.com/en-us/library/dd547397(v=bts.10).aspx

Thank you. Even using your PerfMon Filters i do not get everything... im not sure why as it does not log any reasons that I can locate.

I'm going to ask some silly questions:

1. Did you enable verbose logging in the agent?
2. Did you restart the service for PerfMon Collection Service?

Also, did you mean BizTalk 2006 R2 or 2009? There is no 2008 in TechNet.

Restart the APM PerfMon Service? Yes.

Verbose logging - can you be more specific? I'm not aware of a logging option I can location which switches verbose on/off.

I'm sorry you're right. BizTalk 2009 Sql 2008.

Thanks,

1. Can you see the BizTalk counters when going directly to perfmon?
2. Does the agent have the proper permissions to access perfmon counters for BizTalk?

Ensure the following properties are set to their defaults:

introscope.agent.perfmon.metric.limit

Specifies the maximum number of Performance Monitor metrics that can be reported in each interval.

Default

The default is 1000 metrics.

Example

introscope.agent.perfmon.metric.limit=1000

introscope.agent.perfmon.metric.pollIntervalInSeconds

Specifies how frequently the Performance Monitor Collection Agent check Performance Monitor objects, counters, and instances for new metric values. The default polling interval checks all metric values every 15 seconds.

Default

The default interval is 15 seconds.

Example

introscope.agent.perfmon.metric.pollIntervalInSeconds=15

introscope.agent.perfmon.category.browseEnabled

Enables or disables the discovery of new Performance Monitor counters.

Property settings

True or False

Default

True (enabled)

Example

introscope.agent.perfmon.category.browseEnabled=true

introscope.agent.perfmon.category.browseIntervalInSeconds

Determines how frequently the Performance Monitor Collection Agent checks for new Performance Monitor objects to be discovered.

Property settings

Positive integer representing the number of seconds in the interval.

Default

The default interval is 600 seconds (10 minutes).

Example

introscope.agent.perfmon.category.browseIntervalInSeconds=600

When you get a chance, please show me your filterpattern. The one I provided has worked successfully in a number of installations.

Hi,

I had a full response typed out and the site went into maintenance....

• I can see the BizTalk metrics in PerfMon itself so I know there are there.
• I updated the logging to do Verbose and restarted the Agent service.
• All default PerfMon configuration is set as default as you mentioned in your last post.
• "Permissions" I am not sure about... I do not see any indication that is the problem though.

With the PerfMon Filter set as you mentioned:

introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*,|Processor|_Total|% Processor Time,|Process|BTSNTSvc|Private Bytes,|Process|BTSNTSvc|Virtual Bytes,|Process|BTSNTSvc|*Count,|PhysicalDisk|*|% Idle Time,|PhysicalDisk|*|Avg. Disk Queue Length,|BizTalk:Message Agent|*|Database size,|BizTalk:Message Agent|*|Message *throttling state,|BizTalk:Message Box:General Counters|*|Spool Size,|BizTalk:Message Box:General Counters|*|Tracking Data Size,|BizTalk:Message Box:General Counters|*|Tracking Spool Cleanup,|BizTalk:Message Box:Host Counters|*|Host Queue \- Length,|BizTalk:Messaging|*|Documents processed\/Sec,|BizTalk:Messaging|*|Documents received\/Sec,|XLANG\/s Orchestrations|*|Orchestrations completed\/sec,|XLANG\/s Orchestrations|*|Orchestrations dehydrated\/sec,|XLANG\/s Orchestrations|*|Orchestrations rehydrated\/sec,|XLANG\/s Orchestrations|*|Persistence points\/sec,|XLANG\/s Orchestrations|*|Orchestrations resident in memory,|XLANG\/s Orchestrations|*|Orchestrations suspended\/sec,|XLANG\/s Orchestrations|*|Transactional scopes committed\/sec,|XLANG\/s Orchestrations|*|Transactional scopes aborted\/sec,|XLANG\/s Orchestrations|*|Transactional scopes compensated\/sec

The only PerfMon metrics that show up in investigator are:

1. .NET CLR Jit
2. .NET CLR Loading
3. .NET CLR Security
4. .NET Data Provider for SqlServer

The errors from the Introscope Log:

4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] Starting PerfMon Service
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon metric polling period is 15000ms
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon metric limit is 1000
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] Perfmon category browse is enabled
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon category browse period is 600000ms
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon agent matching expression is *|*
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] Polling interval is 15000 ms.
4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon Service has started successfully

Regards,

Mike

Hi Solis,

Increase the number of performance monitor metrics from 1000 to 1500 or 2000 by modifying the below property:-

introscope.agent.perfmon.metric.limit=1500

or

introscope.agent.perfmon.metric.limit=2000

Restart the perfmon agent after modifying the property.  Give it a try & let me know if that helps.

Thanks,

Sunil

Also switch back to the below default property along with the previously suggested property change:-

introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*

Thanks,

Sunil

Hi Sunhil,

No difference. Same metrics only appear and nothing more.

What has been odd, is the reason I started investigating this.... sometimes after a monthly patching or reboot, the server and agent will come up and the expected perfmon metrics show up. Then with another reboot they dissappear again. Makes no sense and the logs point nowhere....

Hi Solis,

Can you try pointing the perfmon agent to a different  Enterprise Manager & see the behavior ?

Thanks,

Sunil

sunilhere:

Hi Solis,

Can you try pointing the perfmon agent to a different  Enterprise Manager & see the behavior ?

Thanks,

Sunil

Unfortunatey we only have one single Enterprise Manager setup....

HI,

I have the same problem . have you decided the problem?

Thanks

agcoliveira:

HI,

I have the same problem . have you decided the problem?

Thanks

Not certain what you're asking? I was not able to get all of the metrics I was trying to get yet.

If I figure out what the issue is, I will certainly report back.

Mike Solis

Mike,

I have some time now to work on this with you.

Please let me know if you still need assistance.

I already resolved my problem. I only wanted measure the CPU and memory,and my solution went:

- Delete the other counters: introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*,|Web Service|*|*,|Memory|*

and post only the counters that I needed: introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|Memory|*

Thanks

Hi Mike,

Did your issue resolve. ?
Please share your investigation inputs.

Regards,

Vaibhav

To enable verbose logging, look for the following file in your agent directory:

introscope.agent.log.config.path=logging.config.xml

Follow the directions to enable verbose logging, save, and restart the agent.

Hi everyone.... I'm back.

I've had a little better luck with this, but not directly with the BizTalk perfmon metrics. I still have a strange issue where adding certain counter filters make other counter's stop reporting into Introscope.

We are testing out a new problematic app, so I was asked to add a bunch of IIS/Windows and .NET counters to Introscope. I was having excellent luck adding one at a time, until I got to the last one.... ASP.NET v*

Here is my filter string:

introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET CLR Memory|*Global*|*,|System|*,|Memory|*,|Process|{osprocessname}|*,|Web Service|*|*,|Web Service Cache|*,|Network*|vmxnet3*|*,|PhysicalDisk|*|*,|Internet Information Services Global|*,|ASP.NET v*|*

How it displays with the above filter string applied:

As you can see, the last filter is for ASP.NET v*. When this is added, I get the ASP.NET v2.0.50727 metrics, however Internet Information Services Global (|Internet Information Services Global|*) and Network Interface (Network*|vmxnet3*|*) disappear gray out. When I remove the |ASP.NET v*|*, Network Interface and Internet Information Services Global come right back. Obviously something about the ASP.NET v* filter is causing a problem but I cannot figure out what!

Anyone know what the issue is here?

Is there a formal/informal guide to the regular expressions used by this filter someplace that someone is aware of?

Thanks!

Marking as answered.

Just to add to this - it appears the issues I had here all related to a single server. I have no idea why some of the counters did not show up, nor why they would appear/disappear as they felt like it. On any other server I have done perfmon configuration for (CA APM) this has worked fine.

Thanks Haruhiko.

There is a known issue where sometimes Windows installation do not always build and load the counter DLLs, which then requires you to reload them.

https://support.microsoft.com/en-us/kb/300956