Skip navigation
All Places > Rally > Blog > Authors DavidLeDeaux

Rally

3 Posts authored by: DavidLeDeaux Employee
DavidLeDeaux

Parsing Webhooks

Posted by DavidLeDeaux Employee Feb 1, 2019

Last week we touched on creating webhooks, and this week we'll cover a way to capture a webhook and pull out information that you want to use.

 

Prerequisites

For this article, we'll be relying on Google Sheets using their published application capability.

We'll be working in the Script Editor which is very javascript heavy, so a background in javascript programming is helpful.

google_published_app from GitHub - davidledeaux/RallyWebhooks: Proof of concept ways to use Rally (Agile Central) webhooks 

 

 

Setting Up

The idea here is that we'll use the scripting capability of Google Sheets to create a published application.  This means that we'll have a unique URL that we can point webhooks to and then use javascript to parse the incoming JSON payload and write some logic around what we want to do with it.

 

Log into Google and access your Sheets at https://sheets.google.com

Start a new spreadsheet and give it a name like Agile Central Webhooks

Click Tools -> Script Editor

 

At this point you should have a blinky cursor in the myFunction() function:

 

In order for Google to accept requests as a published application, there are two mandatory functions needed; doGet() and doPost().  These two functions correspond with the HTTP GET and POST methods and define the behavior of your script depending on how it is accessed.  For our example, we'll only be using doPost(), but it is interesting to see how you could return responses to data that has been collected in the backend spreadsheet if someone performs a GET request.

 

So the first thing you'll do is to replace the existing function with your own.   In our example we don't really want our app serving anything important for a GET request, so we simply respond to any requests with a generic "request received" message.  We'll be doing the bulk of our work within the doPost() function.

function doGet(e) {
return HtmlService.createHtmlOutput("request received");
}

//this is a function that fires when the webapp receives a POST request
function doPost(e) {

}

 

 

Code Overview

The first thing we want to do is add a few lines to grab the data from our POST request and assign it to a variable and convert it to JSON data that we can access as a dictionary.

var params = JSON.stringify(e.postData.contents);
params = JSON.parse(params);
var data = JSON.parse(e.postData.contents);

 

The next few lines will set up some variable to give us some shortcuts to some data that we'll reference frequently.  The bulk of the interesting data will be in the changes section of the message payload, however we'll be pulling supplemental information from other areas of the message as well.

var changes = data.message.changes;
var state = data.message.state;
var object_type = data.message.object_type

 

In this next block of code we're pulling out details about the current state of the artifact; the formatted ID of the work item that was changed, the name of the work item and the time of the update.

for (var key in state) {
if (state[key].name == "FormattedID") {
var formatted_id = state[key].value;
}

if (state[key].name == "Name") {
var artifact_name = state[key].value;
}

if (state[key].name == "LastUpdateDate") {
var update_date = state[key].value;
}
}

 

Because we can have several changes that can occur on a work item at one time, we have to iterate through all of the changes in the changes section and pull out before and after values.  One thing that complicates this section is that, depending on what was changed, the structure of the changes element might not always be the same.  This is why we see three different methods to gather old and new values. 

 

In my example, I'm simply concatenating the results into a single string to store in a spreadsheet cell later on.  This mimics the changes summary emails that are sent out by notifications, however you might want to handle each change separately for processing depending on your end goal.

var what_changed = "";
for (var key in changes) {
if (changes[key].name != "VersionId") {
var changes_key = changes[key];

if (typeof changes_key.value === 'object' && changes_key.value !== null) {
if (changes_key.value.name) {
var new_value = changes_key.value.name;
var old_value = changes_key.old_value.name;
} else {
var new_value = changes_key.value.value;
var old_value = changes_key.old_value.value;
}
} else {
var new_value = changes_key.value;
var old_value = changes_key.old_value;
}

var what_changed = what_changed + changes_key.display_name + "<br>" + "Was: <br>" + old_value + "<br><br> Now: " + new_value + "<br><br>";
}
}

 

Finally, we log the results to a spreadsheet.

var sheet = SpreadsheetApp.getActiveSheet();
var lastRow = Math.max(sheet.getLastRow(),1);
sheet.insertRowAfter(lastRow);
sheet.getRange(lastRow + 1, 1).setValue(update_date);
sheet.getRange(lastRow + 1, 2).setValue(object_type);
sheet.getRange(lastRow + 1, 3).setValue(formatted_id);
sheet.getRange(lastRow + 1, 4).setValue(artifact_name);
sheet.getRange(lastRow + 1, 5).setValue(what_changed);

SpreadsheetApp.flush();

 

And send a canned response that we received the post request.

return HtmlService.createHtmlOutput("post request received");

 

This is a very simple example of a way to get data into a spreadsheet, but it doesn't have to end there.  For example, you could have Google generate an email using 

MailApp.sendEmail('myemail@company.com', 'Work Item Changed', 'Work item was changed. Here are the changes ' + what_changed);

 

Or, you could turn around and have Google make a call to update another system with a command like

var response = UrlFetchApp.fetch('https://api.myothersite.com/someendpoint/');

 

 

Publishing the App

Once we have all of our code in place, we need to turn it into a published application and get our URL.  To start that process, we'll click Publish -> Deploy as web app...

 

Give the project a name

 

In the next dialog, we have to have the app impersonate someone who is authenticated since the webhook isn't capable of passing credentials. 

 

After clicking Deploy, you'll get a URL that you can use as a TargetURL in your webhook.  You could take this URL and PATCH your webhook you created from the  Creating a Webhook in Agile Central blog post.

Last week, I put together an article introducing webhooks.  This week we'll be covering creating your first webhook.  Creating a webhook is going to require working with our REST API.  It's a good idea to skim through our webhooks documentation page to get an idea of the topics discussed there as it will go into much more detail than this post will.

 

Prerequisites

You'll need to identify a few things prior to generating our webhook:

  1. Tools
  2. Match Expression
  3. Target URL

 

Tools

We'll be interacting with a REST API, so it's helpful to have a client that can craft these HTTP requests with the various methods needed to handle our CRUD operations.  Some people really like , however my "go to" REST client lately has been a little Chrome extension called RESTlet Client so my examples will be using that tool.  Regardless of which tool you use, the concepts are generally the same.

 

We'll also need a target URL to deliver a webhook to.  Now, we're not going to be turning around and doing anything with this payload, so a simple visual indicator of delivery is all the proof we need that the webhook is firing successfully and when we expect it to.  For that, I like to use Webhook Inbox.  Webhook Inbox is a free and simple tool to use.  You simply click one button and it gives you a unique URL to point your webhook to and it shows you the incoming webhooks in real time.  The catch with this is that is only stays active for about a day so it's only useful for debugging purposes.  You'll need to request a new URL and update your webhook if you come back to this a few days later.

 

Match Expression

Expressions will be the fine grained rules that we use to match on.  They'll usually use some form of "key", operator and value.  For example, you might choose to specify a project in your expression so that if a work item is modified in that project, we'll trigger a webhook.  I like to keep things simple, so I usually create a webhook that fires when one particular work item is modified.  That's not useful in a production scenario, but for this example we'll keep it simple and we'll fire whenever US58 is updated.

 

Target URL

This is the destination for the payload to be delivered.  This payload will always come from our datacenters, but the destination is up to you.  It could be a server listening for requests behind your firewall, it could be a serverless function call in AWS, or it could be a third party service like Zapier or Flowdock.  This target URL will typically be a unique destination designed to accept calls for a specific webhook.  I like to use a webhook test site like Webhook Inbox to test the validity of my expressions prior to moving the webhook target into its production URL and in this case, you probably already have a Webhook Inbox set up.

 

Procedure

Before jumping into the nuts and bolts, it's important to explain the relationship between HTTP methods and CRUD operations in a RESTful interface.  In general, there are four HTTP methods that get used to indicate different actions to perform with a target; GET, POST, PATCH/PUT, DELETE.  Not all targets will support all operations because it's not always relevant and in some RESTful implementations, POST and PATCH or POST and PUT are used interchangeably.

 

Our WSAPI interface is an example where the calls can use POST or PUT interchangeably to modify an artifact, however with our Pigeon API, PATCH is required to modify.

HTTP MethodCRUD Action
POSTCreate
GETRead
PATCHUpdate
DELETEDelete

 

Creating a Webhook

In order to create a webhook, we're going to use the POST method and we'll be calling the Pigeon API endpoint directly at https://rally1.rallydev.com/apps/pigeon/api/v2/webhook 

 

In the next section down we have two areas that we want to populate; headers and body. 

Headers are going to contain information about what we're sending to the server to create the webhook.  In this case we want to tell it that'we sending a Content-Type of application/json.  If you're not already logged into Agile Central, then you'll need to click "Add Authorization" to add your username and password to the request.

 

Body, is going to be the message that we send to the API.   For our basic example, our body will consist of the following JSON payload:

{
"AppName": "DavidL WebhookInbox Integrations",
"AppUrl": "http://api.webhookinbox.com/i/EF1aKWbX/in/",
"Name": "DavidL WebhookInbox Integration",
"TargetUrl": "http://api.webhookinbox.com/i/EF1aKWbX/in/",
"ObjectTypes": ["HierarchicalRequirement"],
"Expressions": [{
"AttributeName": "FormattedID",
"Operator": "=",
"Value":"US58"
}]
}

 

Then we click the "Send" button and we should get back a response from the server with a 200.

 

The response body is going to contain a lot of valuable information and it's a good idea to keep this around while you're working on getting the webhook set up.

{
"LastUpdateDate": "2019-01-21T22:37:10.360Z",
"Expressions": [
{
"AttributeID": null,
"AttributeName": "FormattedID",
"Operator": "=",
"Value": "US58"
}
],
"SubscriptionID": 209,
"LastWebhookResponseTime": null,
"_ref": "https://rally1.rallydev.com/apps/pigeon/api/v2/webhook/65016026-b215-4611-b802-c1acd7b37640",
"ObjectTypes": [
"HierarchicalRequirement"
],
"LastSuccess": null,
"LastStatus": null,
"_type": "webhook",
"OwnerID": "2d8dc501-829f-4564-bfd9-REDACTED",
"FireCount": null,
"TargetUrl": "http://api.webhookinbox.com/i/EF1aKWbX/in/",
"CreatedBy": null,
"_objectVersion": 1,
"Disabled": false,
"Security": null,
"ErrorCount": null,
"Name": "DavidL WebhookInbox Integration",
"LastFailure": null,
"AppName": "DavidL WebhookInbox Integrations",
"ObjectUUID": "65016026-b215-4611-b802-c1acd7b37640",
"CreationDate": "2019-01-21T22:37:10.360Z",
"AppUrl": "http://api.webhookinbox.com/i/EF1aKWbX/in/"
}

 

The _ref line is very valuable for coming back to this webhook for checking on the status or for making modifications later on so copy that out to a file.

 

Now, we simply go back to US58 and make a small change while watching our Webhook Inbox.  If everything was set up correctly, you'll see the webhook appear within a few seconds in your inbox:

 

Modifying a Webhook

Let's say you made a mistake and instead of firing on US58, you actually wanted it to fire on US59.  This is where that _ref URL comes in handy.  Take that URL and paste it into the address bar of the query, and change your METHOD to PATCH.

 

Then, down to the body of the request and simply change the value down there

 

Click Send, and you'll see the Expressions section has been updated to US59:

 

This method of updating works for updating any part of the payload, including target URLs, object types, expressions, etc.

 

What's Next

In the next installment, I'll be going over how to tie a webhook into a serverless technology for further processing and data manipulation.

A webhook is an interesting creature.  Many people are familiar with the idea of requesting information from a RESTful API interface, but that's still a "pull" concept.  What if the information came to you automatically?  This is the idea behind a webhook.

 

Years ago, we were promised push technologies that fed us information that was relevant to us.  While we have services that feed us information, many of these services are still very much pull based technologies that mimic push technology.  Now, with the advent of serverless computing, "if this then that" workflow processes and devops automation, we're really starting to see true push technologies coming into their own. 

 

Wehooks are a major part of this silent revolution and Agile Central supports webhooks.

 

The bad news is, the "last mile" is missing.  We can send webhooks, but there aren't many partners that are available to do anything with them.  Part of the difficulty is that the payload that is sent in a webhook isn't standard across the board.  You can't just simply look for a "message" parameter in a webhook and expect it to be there every time.  Every vendor is free to layout their payload in any way they see fit.  The Agile Central webhooks are very rich with detail.

 

There's no way I can fit an entire Agile Central webhook message in this post, but a sample of the information contained looks like this:

{
     "message": {
          ...
          "object_type": "Defect",
          "transaction": {
               ...
               "user": {
                    "uuid": "ae784947-d29d-4cf4-8ed0-e64ab75487eb",
                    "username": "REDACTED@rallydev.com",
                    "email": "REDACTED@ca.com"
               },
               ...
          },
          "state": {
               "55c5512a-1518-4944-8597-3eb91875e8d1": {
                    "value": "DE1",
                    "type": "String",
                    "name": "FormattedID",
                    "display_name": "Formatted ID",
                    "ref": null
               },
               ...
          },
          ...
          "subscription_id": 209,
          ...
          "changes": {
               ...
               "7f25a8ae-6948-49a9-92f5-b437ca213251": {
                    "value": {
                         "value": 32.0,
                         "units": "Points"
                    },
                    "old_value": {
                         "value": 16.0,
                         "units": "Points"
                    },
                    ..."display_name": "Plan Estimate",
                    "ref": null
               }
          },
          "action": "Updated"
     },
     ...
}

 

This is only a handful of the 800 lines that the original message contains.  A lot of the other information contained in the payload would be current state of all fields, details around the artifact type, object IDs, etc.  A sample payload can be seen here.

 

The trigger for these is based on a variety of configurable parameters.  For example, I have a webhook that always fires when DE1 is updated.  This gives me a predictable way to trigger a webhook.  For some, they may want a webhook that triggers any time a change is made to a User Story that is In-Progress or Defects that are in a particular project.

 

These payloads then get delivered to a listening destination that has some code capable of parsing the JSON request contained in the body and then acting upon that code.

 

In the next post, we'll go over how to create a webhook.