Rick.Brown

Service Virtualization using half-bridging (TIBCO Rendezvous)

Blog Post created by Rick.Brown Employee on Jun 1, 2017

There is the occasional requirement for virtual services to be created for protocols where DevTest does not have out-of-the-box support for the transport protocol(s) in use. There are various ways to create deployable virtual services in these cases, the most common of which is the creation of a custom Transport Protocol Handler. However, this is not the only way to design and deploy virtual services for unknown protocols. This document will explain an alternative method, which is useful for cases where we have test steps for a technology but we don’t have virtual service support. The technique is commonly known as “half-bridging”, as we will be creating a matched pair of custom virtual services, to translate (or bridge) between the unsupported protocol and a supported protocol. After activating these two virtual services, we will be able to use one of the standard supported mechanisms in DevTest for the creation, storage and deployment of virtual services, providing all the usual advantages of DevTest over other service virtualisation tools or stubs & mocks.

The Request Listener

The first step is to create a virtual service from scratch, into which we will insert a listener test step for the protocol, a transformer step for formatting the request into a supported data protocol, and a transmission step to forward the message over a supported transport protocol.

Depending on the transport requirements, the transmission step might provide the facility to record a standard virtual service, or it might provide message samples to use as “r-r pairs” for importing into a recording.

Depending on the data requirements, the transformer step might discard control fields, translate to a common standardised and supported data protocol, create meta-data information or store the entire data message.

The Response Listener

The next step is to create a matching virtual service as the listener, but for the response. It should have the same facilities as the listener.

The Responder

This is the opposite to the response listener. It should take a VSI response, translate it into the required data protocol and transmit it through a receiver step to the original protocol.

The standard virtual service

The most common virtual service to use by default is the HTTP virtual service, as it has no external requirements. Other types of virtual service can be used, depending on the requirements such as multiple responses to a single message, reliable messaging, asynchronous considerations, etc.

The custom protocol

This document will explore the requirements of the TIBCO Rendezvous protocol. I chose this for a number of reasons:

  1. I had a customer requirement to do this
  2. It can be downloaded from the vendor website quickly
  3. The installation process is simple
  4. It is trivial to begin sending and receiving data
  5. DevTest includes a test step for this protocol
  6. It needs a lot of processing to correctly replay messages
  7. The message format is documented

Any techniques explained here should be transferrable to other protocols, such as FTP. The specifics might change, but the goal should be the same and the result should be working, supportable virtual services.

If you would prefer an out-of-the-box solution, a Professional Services engagement would be better for you, where transport and data protocol customisations are all hidden in code.

Download & Install of TIBCO Rendezvous

Rendezvous downloads are available for Windows and for Linux. With a simple registration, or connection through Google+, downloads are immediately available.

Installation takes only a few minutes. After install, copy the C:\tibco\tibrv\8.4 folder (or whichever version of Rendezvous you've downloaded) to C:\tibco\tibrv so TIBCO’s execution assumptions are met.

Configuration of DevTest to support Rendezvous

The DevTest documentation explains what files are required from TIBCO to be copied into DevTest.

https://docops.ca.com/devtest-solutions/10-1/en/administering/general-administration/third-party-file-requirements#Third-PartyFileRequirements-TIBCOFileRequirements

I found that not all the mentioned JAR files were installed by TIBCO. I copied this set of files into DEVTEST_HOME/lib/shared:

So I’m missing the JMS files. No problem – I can use native communication in DevTest.

Execution of a simple TIBCO Rendezvous system is by launching the following application. I put it in a batch file for ease of not having to remember any command-line arguments:

This provides a simple web server to show the status of its running Rendezvous environment

A Simple Rendezvous message publisher & listener

Choose a subject, and start listening to it. Do this in a new shell or command prompt:

You will receive an error every few seconds about the licence being expired. I believe the installation page gave me 30 days, so I’m not sure why it complains immediately, but it doesn’t affect functionality.

Now write something to that subject, using another new shell or command prompt:

Look again at the listener window. It has been updated with that message:

Ok, everything looks good. The message looks like text, although it has a strange wrapping around it of {DATA=”…”}. This will bite us later, but for now, it’s all working.

The TIBCO web page has also been updated, with the registered listener:

TIBCO Rendezvous API Documentation

The TIBCO Rendezvous message format is called “TibrvMsg”, and is documented at:

https://docs.tibco.com/pub/rendezvous/8.3.1_january_2011/pdf/tib_rv_java_reference.pdf

This gives us lots of hints about what that “{DATA=” is doing, in the listener log.

TIBCO Rendezvous Message Requirements

We need to decode & encode the TIBCO Rendezvous message format. In the TIBCO logs, the messages look like text, but they contain more information.

A TIBCO Rendezvous message is a grouping of name-value pair objects, with the addition of “id” and “type” fields.

I need to deal with “aeRvMsg” messages, which can include name values that are not XML-safe.

When “type” is set to “TibrvMsg.MSG” in a TibrvMsg message, it is an object inside that message, so we need to consider how to use recursion to explode through the entire message and extract names, ids, types and values. In Rendezvous, these are called “name”, “id”, “type”, “data”.

One use case for TIBCO Rendezvous is fast messaging. Therefore, timestamps and other dates in TIBCO Rendezvous fields use TIBCO-specific date formats, which can include tenth-of-a-microsecond timings. General purpose operating systems are unable to specify or use dates & times to such precision, and the “TibRvMsg.DATETIME” type will notify us that we need to send a timestamp through the TibrvMsgDate() class.   

A DevTest Rendezvous message sender

We have a simple message being published and received when using built-in Rendezvous facilities. Let’s move on to DevTest.

Before reading the API documentation, I presumed that I would simply be able to send “myMessage” or “{DATA=”myMessage”}” and have it send correctly. When I did this, however, the listener showed “lisa-field {DATA=”myMessage”}”. I subscribed to “mySubject” in DevTest, and immediately republished the captured message, the listener log showed it correctly, so something more involved is happening.

After reading the API documentation, I realised that “myMessage” is the “data” part of the TibrvMsg. It also needs “name” to be set, which would make sure it doesn’t default to “DATA” or to “lisa-field”. The “id” and “type” fields can be ignored for the next couple of paragraphs.

So, I must send an object over the required subject, and this object must have both “name” and “data” associated with it. In DevTest, it’s time for scripting. Let’s create a test with two steps, the first of which will be a script step:

In my DevTest publish step, I send the object “rvMsg”, and the listener log displays it correctly.

Ok, this is what I need to expand on, but I will need to create a complete message like the ones I’ll be using on-site, to make sure I don’t forget to add support for some features that are used. Note that I have masked all potentially identifiable data, so the script shows what I’m doing and how I do it, without including PII.

But wait – there’s something extra in my “real” message. What are those date entries? Well, because Rendezvous does strange things with date precision, I find that I need to use their specific date facilities rather than assuming some compliance with Java SimpleDateTime. The TibrvMsg.add method also allows “type” to be defined, and the “type” for a date field is “TibrvMsg.DATETIME”.

I execute my test, and the listener logs it correctly. Now we’re playing with gas!

I could do with abstracting the TIBCO structure out of my Rendezvous step, so it can be controlled by DevTest config files. This will become important in the next steps, so let’s do that now.

Firstly, restart the listener on the service that I’ll be using:

Then let’s add the TIBCO service definition into project.config:

Now let’s edit the publish step to use those definitions:

Ok, now, if I want to write to different services or subjects, or send to remote Rendezvous servers, I can do that in the config file and leave the test as-is. This would also mean that, eventually, when I want to record different subjects, I can deploy a single virtual service with different config files to record multiple subjects concurrently.

DevTest includes a service called “Continuous Validation Service” (CVS). If I deploy this test to CVS, and run it on a schedule, I should be able to use it as a constant source of TIBCO Rendezvous messages, which will speed implementation and debugging of everything I need.

 

Test Harness

Now we have a publisher, we can craft a responder. Using exactly the same creation mechanisms as above, but using a response message instead of a request message, we build a service model to create and publish a message. Insert a subscriber at the start, and loop from responder to subscriber. Insert a “Messaging Virtualization Marker” step at the start, and we have a deployable responder.

If we deploy the virtual service, we will not see transactions incrementing in the Portal. This is because we’ve hand-crafted our responder, so it doesn’t include a transaction incrementer. I modified my project.config file to separate the request subject and the response subject.

Now we have a request being sent to the request subject every minute, and a response being sent to the response subject as soon as a request is read.

Design Decisions of the Half-bridge

Before we go any further, we need to make some protocol-specific design decisions.

Our requirement is to virtualise TIBCO Rendezvous. This typically uses fire-and-forget publishing, which means that, to record, we need to be subscribing at the moment of publishing, otherwise we won’t see the message.

Other systems use different mechanisms. For example, IBM MQ Series typically uses queues, which persist messages until they are read, and HTTP uses on-the-wire interception, which will fail until a server endpoint is made available.

There are a few options open to us. We need to be constantly listening to Rendezvous subjects, so we can capture the messages, but we need to decide how & where to store them, so our recorder can use a standard mechanism.

We could republish messages to some queues that we can read later, but we would have an external dependency on the queue-based ESB, and I don’t want external dependencies in this situation.

We could try to mash together listening and recording, but we would probably miss many response messages, which will make this approach unsuitable.

We could store messages on the file system, but we would need to make sure we have file system permissions, that the directory structure is known, and the recording would change to generation from message samples, which isn’t our goal.

Thinking more about it, storing messages on the file system is probably ideal in this specific instance, as we can store meta-data along with the messages, we won’t lose any messages, we can make sure they follow whatever correlation scheme we need, we can record at our leisure and everything will be reliable. Also, think times in TIBCO Rendezvous are irrelevant, and we can re-read messages as many times as we want for debugging purposes. The more I think about it, the better this approach looks, so this is what we’ll do.

The Request-Recording Half-bridge

We need to subscribe to the request subject. We need to make sure the timeout is so large that it will effectively listen forever (DevTest documentation claims that a timeout of 0 seconds will force a forever listen, but it just returns immediately, so we will set it to a large number, such as 99999 seconds). We need to receive the message and translate it into something that DevTest can consume, so it should include an operation name, all data to correlate against, all data to match against and all necessary dates, and we need to store it where we can grab it later for virtual service generation. We need to do all this inside a virtual service.

A service image shouldn’t be necessary. We need to use a service model to do this.

Messaging Virtualization Marker step allows this service model to be launched from VSE.

The RV sub step is a standard subscriber to the Rendezvous subject.

It needs a filter to store the TibrvMsg into a property

The Process TibrvMsg script stores the data that we need

 

import com.tibco.tibrv.TibrvMsg; //load the API

TibrvMsg message = testExec.getStateValue("rvMsg"); //retrieve the message

opname = message.get("^data^").get("^class^"); //create operation name from one of the fields

idObj = message.get("^tracking^").get("^id^"); //make sure we store correlation data

payload = message.get("^data^").get("XMLPeticion").get("XMLData"); //this is the data we want in the VSI request

 

payloadInsertPoint = payload.lastIndexOf("</"); //insert our correlation data

payload = payload.substring(0, payloadInsertPoint) + "<id>" + idObj + "</id>" + payload.substring(payloadInsertPoint);

 

stanza = payload.indexOf("xml version="); //If we have a stanza, we want to add our operation name after it

if(stanza > 0) {

    opnameInsertPoint = payload.indexOf("<", stanza);

    payload = payload.substring(0,opnameInsertPoint) + "<" + opname + ">" + payload.substring(opnameInsertPoint) + "</" + opname + ">";

} else {

    payload = "<" + opname + ">" + payload + "</" + opname + ">";

}

 

filenameStartPos = payload.indexOf("<SCRID>") + 7; //Another piece of correlation data. We will use this for correlating filenames

filenameEndPos = payload.indexOf("</SCRID>", filenameStartPos);

filename = payload.substring(filenameStartPos, filenameEndPos) + "-req.xml";

filename = testExec.getStateString("basePath", "") + "/" + filename;

testExec.setStateValue("filename", filename);

return payload;

The Response-Recording Half-bridge

The response half-bridge needs to be an exact representation of the response message, including all TibrvMsg field names, ids, types and data, to allow the response-replay half-bridge to reconstruct the response message exactly.

A service image shouldn’t be necessary. We need to use a service model to do this.

The Messaging Virtualization Marker and RV sub steps are identical to the Request Half-bridge (but listening to the response subject).

The Process TibrvMsg is more involved.

 

import com.tibco.tibrv.TibrvMsg;

import com.tibco.tibrv.TibrvMsgField;

import org.apache.commons.lang.StringEscapeUtils;

 

TibrvMsg rvMessage = testExec.getStateValue("rvMsg");

 

String decodeTibrvMsg(TibrvMsg rvMessage) {

    String xmlMessage = "";

    int total = rvMessage.getNumFields();

    for(int idx = 0; idx < total; idx++) {

        TibrvMsgField field = rvMessage.getFieldByIndex(idx);

        String segment = "";

        if(field.type == TibrvMsg.MSG) {

            segment = decodeTibrvMsg((TibrvMsg)field.data);

            newline = "\n";

 

        } else {

            segment = "" + field.data;

            newline = "";

            if(field.name.equals("XMLData")) {

                segment = StringEscapeUtils.escapeXml(segment);

            }

        }

        xmlMessage = xmlMessage + "<rv_" + field.name + " ID=\"" + field.id + "\" Type=\"" + field.type + "\">" + newline + segment + "</rv_" + field.name + ">\n";

 

    }

    return xmlMessage;

}

finalMessage = "<TibrvMsg>" + decodeTibrvMsg(rvMessage).replace("^", "caret").replace("(", "openBracket").replace(")", "closeBracket") + "</TibrvMsg>";

filenameStartPos = finalMessage.indexOf("lt;SCRID") + 12;

filenameEndPos = finalMessage.indexOf("lt;/SCRID", filenameStartPos) - 1;

filename = testExec.getStateString("basePath", "") + "/" + finalMessage.substring(filenameStartPos, filenameEndPos) + "-rsp.xml";

testExec.setStateValue("filename", filename);

 

return finalMessage;

The standard virtual service

A regular virtual service will be used, with minimal configuration. It is simplest, in this instance, to use a standard HTTP virtual service with XML data protocol. This affects the response-replay half-bridge, as that half-bridge needs to listen to HTTP.

There is a small amount of configuration work to be done, because TIBCO Rendezvous uses date formats that aren’t configured in lisa.properties. I won’t explain datechecker here, but an example of the date format I see in converted TIBCO Rendezvous response messages (converting a date from a TibrvMsg.DATETYPE field) is:

Mon May 29 00:38:02 BST 2017

 

The three lines I added to lisa.properties to support that date format are:

 

 lisa.vse.datechecker.rvtimestampregex=(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) (([12]\\d)|(3[01])|(0?[1-9])) (([012]?\\d)|(2[0123])):(([012345]\\d)|(60)):(([012345]\\d)|(60)) ([A-Za-z][A-Za-z][A-Za-z]) \\d\\d\\d\\d

 

lisa.vse.datechecker.rvtimestampformat=EEE MMM dd HH:mm:ss z yyyy

 

lisa.vse.datechecker.rvtimestampformat&\

 

After this, we can generate our service image from rr-pairs, using the standard generation wizard, API call or command-line utility, and it acts like any other well-defined and well-behaved protocol.

The Replay Bridge

Finally, we need to get a request, translate it to XML, forward it to our standard virtual service, get the response from our standard virtual service, translate it into a TibrvMsg and publish it to the response subject.

A service image shouldn’t be necessary. We need to use a service model to do this.

The “RV sub” and “Process TibrvMsg” steps are copied from the request-recording half bridge, but without saving the file. The “web service” step calls our standard virtual service with the XML representation of the message, storing the response. The “process response” message is a new script to translate the response XML into a TibrvMsg, and “RV pub” pushes that message to the response subject.

The “process response” script is more complicated than the recording scripts. Firstly, it needs to convert the XML string to a Document, then it needs to parse the nodes in the document, converting data types as necessary, before storing the TibrvMsg messages and combining them into one response object.

import org.xml.sax.InputSource;

import org.w3c.dom.*;

import javax.xml.parsers.*;

 

import com.tibco.tibrv.TibrvMsg;

import com.tibco.tibrv.TibrvDate;

 

import java.text.SimpleDateFormat;

import java.util.Date;

 

Object createObjectOfContent(String text, short type) {

                                    Object result = null;

                                    switch (type) {

                                    case 0:

                                                      result = Base64.decode(text);

                                                      break;

                                    case 3:

                                                      SimpleDateFormat format = new SimpleDateFormat("EEE MMM dd HH:mm:ss z yyyy");

                                                      Date date = format.parse(text);

                                                      result = new TibrvDate(date);

                                                      break;

                                    case 7:

                                                      result = Base64.decode(text);

                                                      break;

                                    case 8:

                                                      result = text;

                                                      break;

                                    case 9:

                                                      result = Boolean.valueOf(text);

                                                      break;

                                    case 14:

                                                      result = Short.valueOf(text);

                                                      break;

                                    case 15:

                                                      result = Short.valueOf(text);

                                                      break;

                                    case 16:

                                                      result = Short.valueOf(text);

                                                      break;

                                    case 18:

                                                      result = Integer.valueOf(text);

                                                      break;

                                    case 19:

                                                      result = Integer.valueOf(text);

                                                      break;

                                    case 20:

                                                      result = Integer.valueOf(text);

                                                      break;

                                    case 21:

                                                      result = Integer.valueOf(text);

                                                      break;

                                    case 24:

                                                      result = Integer.valueOf(text);

                                                      break;

                                    case 25:

                                                      result = Integer.valueOf(text);

                                                      break;

                                    case 32:

                                                      result = Integer.valueOf(text);

                                    case 64:

                                                      result = Long.valueOf(text);

                                                      break;

                                    default:

                                                      result = text;

                                                      break;

                                    }

 

                                    return result;

                  }

 

//String to document

private static Document convertStringToDocument(String xmlStr) {

    DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();

    factory.setValidating(true);

    factory.setNamespaceAware(true);

    factory.setIgnoringElementContentWhitespace(true);

    DocumentBuilder builder;

    builder = factory.newDocumentBuilder(); 

    Document doc = builder.parse( new InputSource( new StringReader( xmlStr ) ) );

    doc.getDocumentElement().normalize();

    return doc;

}

 

String unXmlSafe(String safeString) {

    safeString = safeString.substring(safeString.indexOf("_") + 1);

    return safeString.replace("caret","^").replace("openBracket","(").replace("closeBracket",")");

}

 

TibrvMsg convertDocToRvMsg(Node node) {

    TibrvMsg rvSection = new TibrvMsg();

    String thisName = thisData = "";

    int thisId = 0;

    short thisType = 0;

    var thisDataObj;

 

    NodeList nodeList = node.getChildNodes();

    for (int nodeNo = 0; nodeNo < nodeList.getLength(); nodeNo++) {

        Node currentNode = nodeList.item(nodeNo);

        thisName = unXmlSafe(currentNode.getNodeName());

        thisData = currentNode.getTextContent();

        if (currentNode.getNodeType() == Node.ELEMENT_NODE) {

            if (currentNode.hasAttributes()) {

                // get attributes names and values

                NamedNodeMap nodeMap = currentNode.getAttributes();

                for (int attrNo = 0; attrNo < nodeMap.getLength(); attrNo++) {

                    Node node = nodeMap.item(attrNo);

                    if(node.getNodeName().equals("ID")) thisId = Integer.parseInt(node.getNodeValue());

                    if(node.getNodeName().equals("Type")) thisType = Short.parseShort(node.getNodeValue());

                }

                thisDataObj = createObjectOfContent(thisData, thisType);

                _logger.debug("This name: {}", thisName);

                _logger.debug("This ID: {}", thisId);

                _logger.debug("This Type: {}", thisType);

                _logger.debug("This Data Object: {}", thisDataObj);

                _logger.debug("TibrvMsg.MSG = {}", TibrvMsg.MSG);

                if(thisType == TibrvMsg.MSG) {

                    TibrvMsg rvSubsection = new TibrvMsg();

                    rvSubsection = convertDocToRvMsg(currentNode);

                    rvSection.add(thisName, rvSubsection, thisType);

                } else {

                    rvSection.add(thisName, thisDataObj, thisType);

                }

            }

        }

    }

    return rvSection;

}

 

xmlMessage = testExec.getStateString("xmlMessage", "");

myDoc = convertStringToDocument(xmlMessage);

TibrvMsg responseMessage = new TibrvMsg();

responseMessage = convertDocToRvMsg(myDoc.getDocumentElement());

return responseMessage;

 Operation

The standard virtual service is deployed. The replay bridge virtual service is deployed. My “tibrvlisten” monitor is showing that the response message is as I expect.

It takes perhaps 100mS to respond, and my scripting is bound to have introduced memory leaks, so this isn't suitable for performance testing. My requirement is for 1:1 request:response. If your requirements differ, you want to select alternative components for parts of your half-bridge.

So there we have it. A half-bridge implementation for providing virtual services, using out-of-the-box facilities in DevTest and providing both generation and replay of virtual services.

The next stage is to test it with my real TIBCO Rendezvous implementation, to see if there are any tweaks I need to add. As fas as I can tell, though, I've supported everything I need to do.

Outcomes