DX Unified Infrastructure Management

This is a custom java probe I have written to help alleviate some probe configuration issues.  Attached is the probe package itself (just drag to the archive), an information document and a basic .jrxml file that can be imported into Unified Reports. This will be an ongoing development process so please provide your feedback here in this thread with issues and feature requests and I will do my best to address them in future versions.

Click Here to Download Probe (Working Link)

Probe Features

This probe will archive every probe configuration in your Nimsoft environment on a customizable interval and automatically do configuration differentials between the current and previous configurations. This will allow a Nimsoft administrator to track configuration changes in their environment automatically.

TECHNICAL INFORMATION

On startup the probe will talk to the data_engine at the path you specified to detect if you have the required database tables and if not it will create them in the Nimsoft database for you. These two tables are PCA_CFG and PCA_CHANGES.

PCA_CFG TABLE

This table contains a record of each probe configuration. It stores the hub, robot, probe, probe version, filesystem path to configuration file, complete configuration file in a column, a record id and archive date.

PCA_CHANGES TABLE

This table contains a record of each probe configuration change. It stores the hub, robot, probe, configuration section, configuration key, previous and current configuration value, previous and current archive date and previous and current record ids.

HOW IT WORKS

The probe uses the native callbacks provided in many probes to get the probe configurations.

1. It uses the gethubs callback on the hub probe to get a list of available hubs
2. Loops through each of those hubs to get a list of robots using the hub’s getrobots callback
3. Grabs a list of probes and information on each probe from the controller via the probe_list callback
4. Saves the configuration file from the robot by using the text_file_get callback
5. Inserts information about the configuration into the database
6. Compares the current and previous configuration to determine if there were any changes
7. If there are changes, writes those changes to the database

NOTES

• As this is the first probe I have ever written in Java there may be some verbose logging that will be removed in future versions.

BUG LIST

• Need to change the HUB address to just hub name in PCA_CHANGES
• Turn down logging

Hi Bryan,

The package seems to be broken and can't be imported to archive

-jon

Seems Google modified the probe package somehow, I have changed the original link and included the new link in this post.

Download Probe

Hi

I have deploy the probe on my primary test machine , but I recieved next message in the log file :

Nov 19 14:13:50:380 [main, probe_configuration_archive] ****************[ Starting ]****************
Nov 19 14:13:50:380 [main, probe_configuration_archive] Version 1.0
Nov 19 14:13:50:380 [main, probe_configuration_archive] CA Technologies
Nov 19 14:13:50:392 [main, probe_configuration_archive] port=48034
Nov 19 14:13:50:431 [main, probe_configuration_archive] Login to NimBUS is OK
Nov 19 14:13:51:432 [main, probe_configuration_archive] initProbe()
Nov 19 14:13:51:432 [main, probe_configuration_archive] Setting callbacktimer
Nov 19 14:13:51:432 [main, probe_configuration_archive] Gathering Configurations
Nov 19 14:13:52:290 [main, probe_configuration_archive] Create PCA_CFG table.
Nov 19 14:13:52:290 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException

Have you a idee of the problem

Can you look into the NIS database and look to see if the PCA_CFG table was created?

HI,

Table is not created .

What is exactly the archive-directory ?

Joel

What database are you using, MSSQL, MYSQL or Oracle? I've only tested it on MSSQL and may need to make some adjustments for the others.

The archive-directory is the directory on your robot you want the configuration files copied to.

I using SQL2008r2.

Bryan:

I am getting a connection refused trying to download the probe.

I have switched hosting and cannot edit the original post, the new download location is http://192.99.166.70/nimsoft/probe_configuration_archive_1.0.zip

Thanks Bryan.  I noticed you had originally tested with MSSQL and I am wondering if this could be used for Oracle.  I tried deploying and got the Java Null Pointer exception mentioned in this thread.  Would it be enough just to create the two tables manually in Oracle or is there other queries that would need to be modified?

I would have to add the Oracle thin libraries and make some modifications on the connectivity. I am currently working on a project that connects to Oracle so adding this to the probe shouldn't be that difficult. Give me a few days and I'll try and get a new version up.

I'm also getting the NullPointerException right after I start the probe. Anyone getting this to work without the error.  I used the http://192.99.166.70/nimsoft/probe_configuration_archive_1.0.ziphttp://192.99.166.70/nimsoft/probe_configuration_archive_1.0.zip download link above.Any workaround or suggestions on what needs to be done to fix.  My database is Microsoft SQL2008R2 and my security is Windows Authentication.

Mar 20 10:43:40:439 [main, probe_configuration_archive] log level=3

Mar 20 10:43:40:439 [main, probe_configuration_archive] file retention=30

Mar 20 10:43:40:439 [main, probe_configuration_archive] table retention=90

Mar 20 10:43:40:439 [main, probe_configuration_archive] Setting probe archive directory at : C:\probe_archive

Mar 20 10:43:40:439 [main, probe_configuration_archive] Ignoring probes: [cdm, spooler, hdb, snmptd, qos_engine, qos_processor, mpse, ppm, alarm_enrichment, remote_control]

Mar 20 10:43:40:440 [main, probe_configuration_archive] ****************[ Starting ]****************

Mar 20 10:43:40:440 [main, probe_configuration_archive] Version 1.0

Mar 20 10:43:40:440 [main, probe_configuration_archive] CA Technologies

Mar 20 10:43:40:448 [main, probe_configuration_archive] port=48044

Mar 20 10:43:40:483 [main, probe_configuration_archive] Login to NimBUS is OK

Mar 20 10:43:41:485 [main, probe_configuration_archive] initProbe()

Mar 20 10:43:41:485 [main, probe_configuration_archive] Setting callbacktimer

Mar 20 10:43:41:485 [main, probe_configuration_archive] Gathering Configurations

Mar 20 10:43:42:094 [main, probe_configuration_archive] Create PCA_CFG table.

Mar 20 10:43:42:098 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException

This probe was one of my earliest Java creations, there may be some significant issues with the database connectivity. I will do my best to produce a new version with some additional debugging early next week.

I'm just excited at the thought of capturing all the configs. Thanks for putting it out there for all to use.

What are the differences in this and the audit probe?

This will archive the entire .CFG file and then run a diff against the previous configuration. A lot of people have requested the ability to easily archive their configurations, this was just my attempt at that while adding the diff feature as well.

Here is a new version of the probe, I just tested it with MSSQL 2012 and UIM 8.2 beta.

http://192.99.166.70/nimsoft/probe_configuration_archive_1.02.zip

If this doesn't work for you please let me know.

Bryan

Bryan, thanks for updating the probe. Looks like my issue is the same. I see it shows it's the updated version in the log but I'm still getting the NullPointerException when it trys to create the database. Any chance it has something to do with Microsoft Credentials vs SQL credentials?

Mar 20 10:52:08:101 [main, probe_configuration_archive] ****************[ Stopped ]****************

Mar 23 09:58:24:715 [main, probe_configuration_archive] data_engine path = data_engine

Mar 23 09:58:24:715 [main, probe_configuration_archive] log level=3

Mar 23 09:58:24:715 [main, probe_configuration_archive] file retention=30

Mar 23 09:58:24:715 [main, probe_configuration_archive] table retention=90

Mar 23 09:58:24:715 [main, probe_configuration_archive] Setting probe archive directory at : C:\probe_archive

Mar 23 09:58:24:715 [main, probe_configuration_archive] Ignoring probes: [cdm, spooler, hdb, snmptd, qos_engine, qos_processor, mpse, ppm, alarm_enrichment, remote_control]

Mar 23 09:58:24:716 [main, probe_configuration_archive] ****************[ Starting ]****************

Mar 23 09:58:24:716 [main, probe_configuration_archive] Version 1.02

Mar 23 09:58:24:716 [main, probe_configuration_archive] CA Technologies

Mar 23 09:58:24:728 [main, probe_configuration_archive] port=48044

Mar 23 09:58:24:761 [main, probe_configuration_archive] Login to NimBUS is OK

Mar 23 09:58:25:764 [main, probe_configuration_archive] initProbe()

Mar 23 09:58:25:764 [main, probe_configuration_archive] Setting callbacktimer

Mar 23 09:58:25:765 [main, probe_configuration_archive] Gathering Configurations, running inital GetConfigs method

Mar 23 09:58:25:776 [main, probe_configuration_archive] Database Driver: MS SQL Server

Mar 23 09:58:25:792 [main, probe_configuration_archive] jdbc:sqlserver://wmstsql04.cs.ball.com:1433;databaseName=NimsoftSLM

Mar 23 09:58:27:668 [main, probe_configuration_archive] Create PCA_CFG table.

Mar 23 09:58:27:669 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException

It might, I'm just grabbing the data_engine configuration information. Does the machine you're running the probe on have direct access to the database?

I'm running the Probe Configuration Archive on the Primary Hub, but my credentials to the database are required to be NT domain credentials rather than SQL credentials and the SQL database is on another server WMSTSQL04 rather than the HUB server. Have you had any more interaction with joel.ingels above? He reported the same issue.

I will test with Windows credentials and see if that is indeed the issue.

It is definitely the Windows Integrated Authentication. I will have to do some testing to see exactly how to accomplish the connectivity. Hope to have something before the end of the day.

Ok, I have made some changes to the probe and updated the sqljdbc libraries to the latest version. Integrated security should now work. You will need to put the full path to your data_engine in the configuration file, even if its on the Primary Hub (/domain/hub/robot/data_engine). The probe assumes you are running a 64 bit operating system, if you happen to be using a 32 bit OS you will need to go into the sdk/auth/x86 folder and rename the .dll file (just remove .rename). Then in the probe EDIT not configuration, change the java.library.path to x86 instead of x64.

Here is the download link again

http://192.99.166.70/nimsoft/probe_configuration_archive_1.02.zip

If you have any questions or problems you can post here or email me at bryan.morrow@ca.com

Bryan

Okay, works like a charm. Thanks for all the hard work.  Looks like Windows Authentication now works.

Danny

Glad that it works, please make sure it runs again after 24 hours and let me know.

Thank you Bryan! This works great.

Question:

1. How do you import the Unified Report file to UR? I'm not that familiar with UR. Where do I find this report w/in the tree structure w/in UR?
2. Also How can I figure out why this might not be backing up all robots under a specific node? Under one hub have 200+ robots. The probe is only creating 67 of the 200. Any ideas?

Thank you,

Daniel

Just added to the thread. To add the UR report for this probe do the following. I am doing this thru the UMP - Unified Reports Portlet.

1. Open and log into UMP.

2. Reports - Unified Reporter

3. Expand Public - Nimsoft Monitor

4. You can r-click and Add Folder to create a "Probe Configuration Report"

5. Then select the new folder and r-click Add Resource > JasperReport

6. In the "Set Up" Give this report a name:  Robot Probe Config Report.

7. In the "Locate the JRXML File: Upload a Local File" hit the Browse button and find the probe_configuration_report.jrxml

8. Select DataSource, and "Select data source from repository" then choose the Nimsoft, UIM, NimsoftSLM DB that should of previously created when setting up Unified Reports.

9. Hit Submit when done.

Hi Bryan,

Any chance to have a version for MySQL as well? It would fit perfectly to one of my customers (MSP) who is trying to reinvent this wheel by himself.

Thanks,

Fernando

This should work with MYSQL at the current version of 1.02 provided above.

Bryan, this looks good, does it require a license key?

Thank you

This version should not, if you grab it from the Marketplace it does.

Having an issue with the report coming back empty, with no data. everything looks good in the log file for the probe configuration archive and it is running. Any ideas?

It has been a while since I have used this probe so please forgive me if I am wrong. I believe the report shows the configuration changes that was made. So if you have not changed any configuration since you have added the probe the report will come back blank. It wasn't until I started changing probe configurations that I saw the report populate with data. Have you made any configuration changes since configuring the probe?

This is great, thanks!

We have it running in our test environment but a little concerned moving to PROD.

I've noticed:

1) It's not honoring the time schedule option.  For instance, setting time to 300 does not do the check every 5 minutes.  I assume it keeps the full 24 hour schedule.

2) Every time the probe restarts, it'll redo any configuration grab and rewrite the full config file regardless if there are any changes. Is this a bug or feature?

Feature request for a large environment:

1) It would be nice to limit the scope of which hubs it queries, that way we could run these on individual hubs (which could be geographically far away).  Whitelist + Blacklist for hub names would be amazing.

2) Specify time of day to actually run if possible, or only run on Sunday, etc.

Thanks again.

Erik

I was able to get it to run and then run a report from the UMP. But it isn't including all of the probes that I have deployed. I can see the data that it grabbed and saved to a flat file but since we didn't change any of the configurations, it isn't pulling the (only set up once) probes into the report. I tried running the probe again and since I haven't changed anything, it is still the same. Any ideas?

thanks!

Hi Bryan,

I just want to say that you have done a really great job.

I was wondering, I am running the probe in my Primary Hub which is separated from the DB server, I am using SQL credentials but I am getting the error below:

Apr 19 13:31:47:905 [main, probe_configuration_archive] ****************[ Starting ]****************

Apr 19 13:31:47:905 [main, probe_configuration_archive] Version 1.02

Apr 19 13:31:47:905 [main, probe_configuration_archive] CA Technologies

Apr 19 13:31:47:913 [main, probe_configuration_archive] port=48024

Apr 19 13:31:47:939 [main, probe_configuration_archive] Login to NimBUS is OK

Apr 19 13:31:48:941 [main, probe_configuration_archive] initProbe()

Apr 19 13:31:48:941 [main, probe_configuration_archive] Setting callbacktimer

Apr 19 13:31:48:941 [main, probe_configuration_archive] Gathering Configurations, running inital GetConfigs method

Apr 19 13:31:48:942 [main, probe_configuration_archive] Controller address for primary hub: /CIB-UK/CIB-UK-711-0001/LONS00112372/controller

Apr 19 13:31:49:955 [main, probe_configuration_archive] Database Driver: MS SQL Server  SQL Authentication

Apr 19 13:31:50:961 [main, probe_configuration_archive] jdbc:sqlserver://LONC00112148H:1433;databaseName=CA_UIM

Apr 19 13:32:05:795 [main, probe_configuration_archive] Create PCA_CFG table.

Apr 19 13:32:05:797 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException

As you can see, I am using SQL credential, however it does not work. Anybody has any idea about it?

I would like to use this probe because seems to be really useful.

Thank you very much in advance

I am having a similar problem as to some in this thread, I've created a separate thread: probe_configuration_archive  Using this on 8.4 and not sure what the fix may be at this point as turning up the loglevel doesn't seem to shed any light on the issue.

This is an old version of the probe, please use the latest release that includes an internal H2 database and multi-threading.

Configuration Archive - A Custom Multi-threaded UIM Probe