You can further refine your metric collection by creating custom probe definitions in the form of new PBD files. Creating custom directives, by creating probes to track application specific measurements, requires the use of specific syntax and keywords. To write custom probes, you must define:

• The directive type (indicating generically how many class(es) or method(s) to trace)
• The specific class(es) or method(s) to trace
• The type of information to trace in the class(es) or method(s) (for example, a time, a rate, or a count)
• The fully-qualified metric name (including the resource path) under which to present this information

Once a custom PBD is created, Introscope treats it as if it is an out-of-the-box PBD. The newly created metrics can used in alerts, reports, or dashboards.

Note: Be sure to choose methods to trace carefully, as more methods traced means more overhead.

Building a PBD

Creating a new PBD file can be broken down into several main steps:

• Identify components to instrument
• Create the PBD file
• Deploy the PBD file
• Verify and validate the PBD file

Each of these steps is covered in detail in its own page and is summarized in the following sections.

Identifying components to instrument

Writing a PBD file is entirely dependent on having identified a set of classes and methods to instrument. With the use of a variety of directives, an effective PBD can be generated for an entire package (or namespace in .NET), specific classes and methods, or all methods of a single class; it is up to the designer of the PBD to determine what components are added and how.

Commonly, a PBD will target specific classes, either directly or via inheritance and implementation. When identifying which set of classes to instrument, the following are good general selection criteria:

• Classes that support the key use cases of the application or framework
• Classes that transition the application across physical partitions
• Classes that are known or suspected to have performance issues

Create the PBD file

When creating a PBD file from scratch, there are a few key steps that should be performed in order to insure that the PBD is sustainable.

Note: It is always a good idea to comment your PBDs. We recommend using a comment section at the start of a PBD to describe the overall intent of the following directives. Additionally, comments, which are lines beginning with the # character, are recommended to annotate related class identification directives or method tracer directives.

Be sure to visit the Directives for more detail regarding the directives mentioned below.

PBD Creation Steps

1. Declare tracer group(s) using the SetFlag directive. This will allow you to create a set of logically organized probes associated with a group of classes with the user-defined set of monitoring instructions.
2. Associate classes with your tracer group using the IdentifyClassAs and similar directives.
3. Identify the methods to trace and how to trace them (select directive type, class / method to trace, type of tracer (counter, timer, etc), and name of resulting metric(s))
4. (Optional) Validate and verify the syntax. Before activitating the tracer, there are methods by which you can validate the syntax, thereby avoiding any errors during the activation.
5. (Optional) Active the tracer group. This can be done using the TurnOn directive, the hotdeploy directory, or by using Remote Dynamic Instrumentation.

Important! Be sure to choose which methods to trace carefully. Tracing too many methods and the monitoring probe could impact application performance. Similarly, the more times a method is called, the more it could impact performance.

Deploy the PBD file

Custom PBDs, unique to an application, can be stored in the <Agent_Home>/wily/core/config/hotdeploy directory. Any PBDs added to this directory will be implemented without having to update or modify the introscope.autoprobe.directivesFile property in the IntroscopeAgent.profile. If you have enabled dynamic instrumentation, the PBDs in the hotdeploy directory are picked up live from the folder. No restartis required.

When there are multiple agent configurations for a single agent install, there can a general hotdeploy directory as well seperate hotdeploy directories for each unique agent profile (or a unique PBL file).  Custom PBDs will typically be placed in the agent profile specific hotdeploy directoy. The exception to this is for custom framework support that can be shared between agents. Such PBDs will often deployed to the <Agent_Home>/wily/core/config directory.

There are, of course, multiple ways of deploying a PBD file and enabling it for use.

Verify and Validate the PBD File

After deploying a PBD file to an agent, AutoProbe will load the new directives either on application restart or, with Dynamic Instrumentation, at the regular polling interval.  The most common problem encountered with PBDs is when a tracer group is referenced in a directive but never declared with the SetFlag directive. While the SetFlag directive could exist in a present PBD, that PBD may not be in a hotdeploy directory, added to a PBL loaded by the agent, or added to the directives list in the agent profile - therefore, that PBD is not loaded by the agent and the SetFlag directive is never seen. 

To confirm that the PBD has been successfull loaded, the best place to check is the AutoProbe.log. This log file lives in the <Agent_Home>/wily/logs directory, and can often have the agent name appended to its file name for clarity, e.g. AutoProbe.log. This file name is specified in the relevant IntroscopeAgent.profile. 

The contents of this file will list all directives, by PBD file, that are being evaluated and loaded. Additionally, it will list out all classes that are inspected by AutoProbe and call out which are instrumented or not instrumented. Finally, the AutoProbe log file will list any error messages related to the loading of PBDs, including the name and line number of any problematic directive.

Controlling Overhead

• High call volume and low or zero response time? 
• Hitting the 50k metric clamp?
• To Blame or Not To Blame

Common Problems

• Two layer traces
• No traces
• Too many traces
• Traces that are mysteriously missing database or web services