Skip navigation
All Places > CA Unified Infrastructure Management > Blog > Authors BryanKMorrow

The function of this probe will be to provide a UIM administrator with small callback utilities to make the management of the UIM infrastructure a little easier. This probe will be under constant development with new utilities being added regularly. If you have suggestions for new automated utilities or feedback on the current state, please post in the comments of this post.

 

 

PLEASE NOTE: When using the automation_device_wiper callback with the delete_qos option enabled, this will perform a delete on all of the Raw and Summary tables, this could lead to significant load on the database server and possibly some deadlocks.

 

Probe New Features
This probe is a collection of useful callback utilities to help a UIM administrator manage their system. You can accomplish the following tasks currently with this probe:

 

NEW: First attempt at a UIM topology map (hubs with large robot counts won't work OOTB currently, still working on the dynamic sizing).
UPDATED: HTML 5 Report for top probes in use , now supports a top_n parameter to adjust the default of 15

UPDATED: MYSQL connectivity when special characters are used should now be fixed.


Probe Existing Features
• Encrypt passwords for profile usage in almost all current probes
• Update the interface alias for a specified device->interface (Usually only available in USM)
• Reset a probe’s security on a specified robot
• Retrieve a list of source->targets where data has not been received in a specific time frame
• Remove a device or list of devices from discovery with an option to delete QOS

• Delete QOS by providing a list of targets
• Clean niscache of provided list of robot names
• Generate HTML Reports for the following: License Pack Counts, UIM Users, Account Contacts, Hub Subscribers, CDM/Processes/NTServices Thresholds.

      Threshold reports can now take a USM Group Name parameter. NOTE: Needs to be child group that contains                 devices
• Generate HTML Report for UMP database health. Based on TechDoc http://search.ca.com/assets/SiteAssets/TEC1405477_External/UMPUSMSlowPerformanceGuideandTroubleshootingChecklist1.1.pdf
• Manually configure probes by providing the following information: probe name, section, key and value. Can provide a       comma-separated list of robots. Multi-threaded
• Modify single probe configuration by providing JSON
• Collect and ZIP a probe’s log files and configuration files. Could be used for support.
• Collect thresholding information based on configured probe profiles, multi-threaded (This is the migration of the     threshold_archive probe that is currently on the Communities). Multi-threaded

      Can now be limited by providing a hublist parameter. Hub name, not hub address is required.
• Generate HTML SVG report of VMware topology. This features uses the Vsphere API to collect parent-child relationships for your infrastructure. It ‘should’ generate one HTML page for each configured resource for each vmware probe instance. Multi-threaded.
      o NOTE: This connects to vSphere directly so the probe will need to be on the same physical network as the Vcenter          or ESX host. The vmware probe doesn’t collect the needed attributes out of the box, so that is why it connects                  directly to vSphere.
• Retrieve list of current USM Groups (Will be needed for future USM group creation feature)

 

 

 

TODO LIST
• Create USM groups from JSON
• Store data in H2 database for license usage over time
• Improve configuration archive and diff reports (configuration_archive functionality)
• Add more probe threshold reports (logmon)
• Improve probe configuration through JSON to support multiple robots and probes
• Continue testing the VMware topology feature

 

 

 

 

REVISION HISTORY

Date

Version

Change

July 22nd, 2016

1.00

Initial Draft

July 22nd, 2016

1.01

Added devices_with_no_data

July 25th, 2016

1.02

Devices with no data always creates a CSV, no matter the size. Initial merge of healthcheck probe with just list_subscribers creating a CSV.

July 28th, 2016

1.03

Added automation_device_wiper callback

September 9th, 2016

1.04

Removed decrypt password option, added list_accounts/users, can now provide a CSV list of devices for removal

October 4th, 2016

1.05

Added ability to manually configure probe configurations and new licensing report.

October 26th, 2016

1.10

Added niscache_clean, modify probe configurations from json, UMP/database health report, threshold gathering and reporting and vmware topology

November 8th, 2016

1.11

Added support file retrieval and processes/ntservices threshold reports

November 21st, 2016

1.12

Delete QOS by target, inactive probe report and added filtering for threshold gathering and reporting

December 7th, 2016

1.13

Top probe usage reports, configuration_archive custom probe integration

January 3rd, 2017

1.14

mysql connectivity fixed, added uim topology map, top_n parameter to top_probes report

This is a newer version of the probe_configuration_archive that was available on the CA Marketplace. This probe offers the same features, but does so with multi-threading capabilities. This post will hopefully be a living document where people will comment on the current state and also additional features they would like to see implemented. I will attach the latest documentation and probe package to this post, so please subscribe for updates.

 

Probe Features

This multi-threaded probe will archive every probe configuration in your UIM environment and automatically do configuration differentials between the current and previous configurations. This will allow an administrator to track configuration changes in their environment automatically. This probe base is derived from the previous single threaded probe_configuration_archive.

 

Prerequisites

  • UIM Server version 8.0 or greater
  • Java_jre version >= 1.7

 

Probe Configuration

This probe can only be configured using Raw Configure mode.

raw_config_v1.png

ignored_probes: Use a comma separated list of probe configurations you wish to be excluded from collection

archive_path: File system location where you wish to store the configuration files

threads: The number of threads used while doing probe configuration gathering

 

RUNNING THE ARCHIVE

The current version is driven from the rebuild_configuration_database callback, please run this whenever you want to archive your configurations. Future version will re-implement an interval based archive.

probe_utility.png

 

TECHNICAL INFORMATION

The probe uses an internal JDBC capable database (http://www.h2database.com) to store the probe configurations and also the configuration change data. This allows the data to still be accessible for custom dashboards and reports, but also avoids using the UIM database as a datastore. I chose to stick with the raw_configure option as opposed to the newer Probe Framework SDK because of the multi-threading that was implemented. The multi-threading should speed up the configuration gathering quite significantly compared to the older probe_configuration_archive.

CONFIGURATION_ARCHIVE_CURRENT TABLE

This table contains a record of each probe configuration. It stores the origin, hub, robot, probe, probe version, filesystem path to configuration file, complete configuration file in a column, a record id and archive date.

CONFIGURATION_CHANGES TABLE

This table contains a record of each probe configuration change. It stores the hub, robot, probe, configuration section, configuration key, previous and current configuration value, previous and current archive date and previous and current record ids.

HOW IT WORKS

The probe uses the native callbacks provided in many probes to get the probe configurations.

  1. It uses the gethubs callback on the hub probe to get a list of available hubs
  2. Loops through each of those hubs to get a list of robots using the hub’s getrobots callback
  3. (Multi-threaded) Grabs a list of probes and information on each probe from the controller via the probe_list callback
  4. (Multi-threaded) Saves the configuration file from the robot by using the text_file_get callback
  5. (Multi-threaded) Inserts information about the configuration into the database
  6. Compares the current and previous configuration to determine if there were any changes
  7. If there are changes, writes those changes to the database

 

JDBC CONNECTIVITY

In order to query the data from the UMP you will need to copy the h2 database driver jar file into the <Nimsoft_installation_directory>\probes\service\wasp\libs directory. The driver can be found Here. You will then need to create the data source in the Dashboard Designer.

database_configuration.png

  • Name: Logical name for the connection
  • Driver Class: org.h2.Driver
  • JDBC URL: jdbc:h2:tcp://<IPADDRESS_OF_PROBE_CONFIGURATION_ROBOT/./configuration_archive
  • User: uim
  • Password: uim

 

KNOWN BUGS

  • Ignore List function is currently not working
  • Sometimes when re-deploying you may need to deactivate->reactivate the probe for it to start properly

TODO LIST

  • Enable file and table retention
  • Add callbacks for adhoc differential between configurations based on record/date
  • Add callback option to replace current probe configuration with archived configuration

 

Again, please feel free to comment directly on this post with any feedback or suggestions for the probe. I plan to continue development on this for quite some time.

 

Thanks,

 

Bryan

As a follow-up to the Executive Level Scorecard dashboard example posted previously, this is a summary of a network device. It contains a context selector that pulls from devices being polled by SNMP Collector. It also has some predefined parameters that allow you to modify the interface utilization metrics that are collected in bytes, allowing you to convert to kilobytes, megabytes or gigabytes.

 

THE DASHBOARD

--------------------------

dashboard.png

 

As you can see its the same type of information that can be found in USM, but in what I feel is a more presentable format.

 

  • All the data is driven by the context selector at the top right.
  • All of the datasources are SQL driven to use the context selector.
  • The KiloBytes In and Kilobytes Out columns are modified using a parameter

 

THE CONTEXT SELECTOR

-------------------------------------

context_selector.png

 

select distinct source from s_Qos_Data where probe = 'pollagent' order by source asc  

 

This query retrieves all devices that are collecting data from the snmp_collector probe

 

THE GAUGES

--------------------

cpu.png

 

As you can see its driven by a SQL datasource, pulling from the S_QOS_SNAPSHOT table.

 

select s.samplevalue from s_Qos_data d join s_qos_snapshot s on d.table_id = s.table_id where d.source = '${device}' and d.qos = 'qos_cpu_utilization'  

 

 

THE INTERFACE TABLE

----------------------------------

table_widget_1.png

 

Here I am just mapping the column headers to the column output and adjusting the width so the information fits without a horizontal scrollbar

 

I am by no means an SQL expert so this query may not be very optimized but it works!

 

NOTE: This is for MSSQL, queries for MYSQL and Oracle will most likely need to be adjusted.

 

select i.interface, i.utilization_in, gg.bytes_in / ${kilo} as bytes_in, i.utilization_out, gg.bytes_out / ${kilo} as bytes_out, i.errors_in, i.errors_out, j.discards_in, j.discards_out from (select g.interface, g.utilization_in, g.utilization_out, h.errors_in, h.errors_out from   (select a.interface, a.utilization_in, b.utilization_out from   (select z.target as interface, y.samplevalue as utilization_in from s_Qos_data z   join s_qos_snapshot y   on z.table_id = y.table_id   where z.source = '${device}' and z.qos = 'qos_interface_utilizationin') a     INNER JOIN     (select x.target as interface, w.samplevalue as utilization_out from s_Qos_data x   join s_qos_snapshot w   on x.table_id = w.table_id   where x.source = '${device}' and x.qos = 'qos_interface_utilizationout') b   on a.interface=b.interface ) g   INNER JOIN     (select c.interface, c.errors_in, d.errors_out from   (select v.target as interface, u.samplevalue as errors_in from s_Qos_data v   join s_qos_snapshot u   on v.table_id = u.table_id   where v.source = '${device}' and v.qos = 'qos_interface_pcterrorsin') c     INNER JOIN     (select t.target as interface, s.samplevalue as errors_out from s_Qos_data t   join s_qos_snapshot s   on t.table_id = s.table_id   where t.source = '${device}' and t.qos = 'qos_interface_pcterrorsout') d   on c.interface=d.interface) h   on g.interface=h.interface) i   INNER JOIN     (select e.interface, e.discards_in, f.discards_out from   (select p.target as interface, o.samplevalue as discards_in from s_Qos_data p   join s_qos_snapshot o   on o.table_id = p.table_id   where p.source = '${device}' and p.qos = 'qos_interface_pctdiscardsin') e     INNER JOIN     (select n.target as interface, m.samplevalue as discards_out from s_Qos_data n   join s_qos_snapshot m   on n.table_id = m.table_id   where n.source = '${device}' and n.qos = 'qos_interface_pctdiscardsout') f   on e.interface=f.interface) j   on i.interface=j.interface   INNER JOIN     (select cc.interface, cc.bytes_in, ff.bytes_out from   (select aa.target as interface, bb.samplevalue as bytes_in from s_Qos_data aa   join s_qos_snapshot bb   on bb.table_id = aa.table_id   where aa.source = '${device}' and aa.qos = 'qos_interface_bytesin') cc     INNER JOIN     (select dd.target as interface, ee.samplevalue as bytes_out from s_Qos_data dd   join s_qos_snapshot ee   on dd.table_id = ee.table_id   where dd.source = '${device}' and dd.qos = 'qos_interface_bytesout') ff   on ff.interface=cc.interface) gg   on j.interface=gg.interface  

 

As you can see I am using ${device} to pull in the device.  As for the metric conversion you can see where I'm using the front slash to mark the division and ${kilo} as the conversion.

 

select i.interface, i.utilization_in, gg.bytes_in / ${kilo} as bytes_in, i.utilization_out, gg.bytes_out / ${kilo} as bytes_out, i.errors_in, i.errors_out, j.discards_in, j.discards_out 

 

THE PARAMETERS

----------------------------

parameters.png

 

Here I'm just entering the conversion numbers and assigning them a parameter.

 

 

Please let me know your thoughts on the dashboard and any ideas on improving it.



Thanks,


Bryan

Recently I have completed a dashboard that I feel provides both technical and business users with a very high level overview of your Infrastructure that is managed by UIM. Not only does it have the Executive level information (SLAs, Health Index, Alarm State) but it also allows a more technical user to drill down into a more detailed set of data (USM, other dashboards, etc). The Scorecard also shows very well in a NOC environment on the projector or tv. You can download the Scorecard.zip file and import it into the HTML 5 Dashboard Designer.

 

THE DASHBOARD

---------------------------

scorecard_front.png

 

As you can see there are different sets of information that each 'badge' contains.

 

  • The numbered gauge is used to represent the health of the service or application.
    • I prefer either an SLA or Health Index value.
  • The A icon represents the alarm status of the service or application.
    • A simple alarm filter is best.
  • The P icon represents a drill down dashboard (Some examples of these will follow in later posts).
    • This could also just link to a Unified Dashboard or any other direct link in the UMP.

 

 

THE DATASOURCES

------------------------------

scorecard_datasources.png

 

In order to make the dashboard generic I have created some simple SQL based datasources to be used at the time of import. These are just hard coded numbers that each query represents. You can just as easily select the SLA datasource that is appropriate for the service or application.

 

THE GAUGE

-----------------

scorecard_badge1.png

 

 

THE ALARMS

-------------------

scorecard_alarm1.png

 

THE PERFORMANCE

------------------------------

scorecard_performance1.png

 

 

Please let me know your thoughts on the dashboard and any ideas on improving it.

 

TO DO: I still need to make the different sections modular, currently its a fixed background.

 

Thanks,

 

Bryan

As some people are still new to HTML 5 Dashboard options in UIM, I wanted to give a brief example of using the Line Chart Widget.

 

First we will start with the Line Chart; this will provide you the ability to create a time series chart based on either a QOS metric or an SQL query. In this example I will be providing the SQL example.

 

Step 1 – Drag the Line Chart widget onto the dashboard canvas. This will leave give you a blank widget.

 

line_chart_blank.png

 

Step 2 – Configure the Chart

  • Give it a Y-Axis Label. In my case that is Messages Per Minute
  • Give it a Series Duration, I will be using 1 hour

 

line_chart_widget_chart.png

 

Step 3 – Configure the Series

  • Highlight the Line Chart widget on the canvas (will have a red border).
  • Press the + icon under the Series navigation section on the right column.
  • In the Series Data Source Type dropdown, select SQL.
  • Press the + icon under the SQL data source to create the datasource.
  • Give the Query a name, select the NIS option from the database dropdown.
  • Paste and Test your query.

In this example I am returning the last hour with the following query:

 

select d.sampletime, d.samplevalue as posted from S_QOS_DATA s join RN_QOS_DATA_1730 d on s.table_id=d.table_id where s.target = 'WorldWideHQ' and d.sampletime > DATEADD(HOUR, -1, GETDATE())

 

 

I like to go back and rename the Series 1 that was created earlier, this will help if you are creating your own legend.

 

       Repeat the steps required to create multiple series if needed.

 

NOTE: The format of the SQL query needs to return two columns, the first being the sampletime and the second being the value.

 

Step 4 – Create a custom legend. The current version of the dashboard does not have a legend associated with the Line Chart, so we will need to create one.

  • Drag a Rectangle Widget onto the canvas underneath your Line Chart.
  • Change the size to be appropriate to your Line Chart.
  • Change the color to match the Series.
    • Series 1: #1F77B4
    • Series 2: #FF7F0E
    • Series 3: #2CA02C
    • Series 4: #D62728
    • Series 5: #9467BD
    • Series 6: #8C564B
  • Give the Widget a Label to match the Series.

 

Clone the Rectangle Widget and repeat for each Series.

 

hub_health_line_chart_example.png