All,
While looking at the current ways we can integrate Jenkins with CA Application Test, I came up with the following methods:
1. If DevTest is installed on the same server as Jenkins, we can use the Ant/junitlisa integration to locally run our tests, generate a JUnit report, and integrate it into Jenkins reporting.
2. If DevTest is not installed on the same server as Jenkins, we can use a plugin like HTTP Request Plugin to leverage the DevTest Invoke API to start tests remotely and then follow up with the test results out in the DevTest Portal to find out which ones passed/failed.
What I was looking for was a third option where I could leverage the DevTest Invoke API to kick off a set of tests, but then also receive some feedback within Jenkins about how many of my tests passed/failed. This sent me on a quick path of Googling how to create Jenkins plugins. What I came up with was a relatively small Java project which builds a Jenkins plugin that does the following:
Allows you to choose your remote DevTest host, port, test suite path, and config path, and then will kick off your test suite remotely using the Invoke API. The plugin will then scrape the output of the Invoke call and provide a JUnit style xml output file that can be integrated into Jenkins reporting. I'm publishing the plugin here to solicit feedback and help others who may have a similar need, but it must be understood that:
THIS PLUGIN IS NOT PART OF THE OFFICIAL DEVTEST PRODUCT
THIS PLUGIN IS NOT SUPPORTED BY CA
IF THE OFFICIAL DEVTEST INVOKE API CHANGES, IT MAY CAUSE THIS PLUGIN TO STOP WORKING
USE AT YOUR OWN RISK
If you'd like to see the source code, or you'd like to extend this plugin to do something special, please reach out to me and I'd be happy to work with you.
I've tested this plugin against DevTest version 9.1, but I suspect it will work fine with older/newer versions that also contain the Invoke API.
If you would like to try out this prototype plugin, here are the instructions:
1. Download the attached hpi plugin file and extract it from the zip file
2. In your Jenkins instance, go to Manage Jenkins -> Manage Plugins, and on the Advanced tab, scroll down to Upload Plugin, browse to the downloaded plugin and click Upload
3. In your Jenkins projects, you now have a new option under "Build Step" which is CA DevTest Invoke Test Suite.
4. Add the requested information. For the server, be sure to include http:// or https:// in front of your hostname. For Test Suite Path and Config File Path, be sure to check the hint for those fields which says to be sure you update property lisa.portal.invoke.test.root found in lisa.properties on your registry server.
5. In order to see your test results, under Post Build Actions, add a step for "Publish Junit test results report" and include the path devtest_results/**xml
6. Now kick off a build. You'll need to actually run your build twice the first time in order to see the test report show up (this is a Jenkins thing.)
When you see your test results report, you can drill down into it. There are a couple of things to note about this prototype:
The plugin only has access to the information that comes back from the invoke API. That means it only has the overall duration of the entire suite, not individual tests. Also, it only tells us how many tests passed/failed, not which individual tests passed/failed. What this means is that in the test results report, the overall time is accurate, however the individual test times are just an average time of all the tests. Also, the drilldown which shows which tests passed/failed will be accurate as far as the numbers of each, but not which ones passed or failed. To see this information, continue drilling down into any of the tests and you will see the REST Invoke output which includes a link you can follow that will take you directly to the DevTest report for your test suite which contains all the accurate info.
If you download and use this, please let me know, and let me know what you think. I hope this helps someone since it was a fun side project for me to work on.
-Chris