I awoke with a start at 3am this morning, with a “Eureka” moment, and I had to get up and try it out!
This is a convergence between my home automation IoT interest and DevTest, where a couple of things have come together to add lots of value to both.
First, the home automation side:
I already have both Hive (for my heating and hot water) and Hue (for lights in my top-floor bedrooms). They are both Zigbee gateways, but neither of them provide the flexibility of general-purpose hubs. I wanted to investigate low-power sensors, and I would need to link them into the rest of my automated home, so I investigated whether any of the available smart hubs would integrate with Node-Red or MQTT. Nothing does, out-of-the-box, but there are a couple of options. Custom firmware for the Wink hub can include MQTT, but the Wink is difficult to get in the UK. There’s an open-source project to link SmartThings to MQTT, and Amazon had a good deal for a SmartThings starter kit, so I jumped at it.
I configured my new SmartThings hub, installed the smartthings-mqtt-bridge in pm2 alongside Node-Red, and I could now see the SmartThings sensor data when I subscribed to the correct topics.
SmartThings sensors are fairly well-priced, but I want to go completely overkill on my sensor readings. Browsing through the gearbest website, I found some great deals on Xiaomi Aqara sensors, so I purchased 10 temperature & humidity sensors, a water leak sensor, a motion detector and a magic cube. I connected all of these to SmartThings using user-provided data handlers, and added them to the smartthings-mqtt-gateway, so now I have Node-Red seeing all the new sensor data. Lovely!
Now I need to monitor MQTT, so I can see what topics are configured in case my automation is publishing to topics that I’m not subscribed to. Astonishingly, this functionality doesn’t exist in MQTT, so I’m stuck!
Now to DevTest:
I’ve been having conversations with MarcyNunns about the client-side JARs we need for ActiveMQ JMS support, and which we can ignore, because DevTest uses ActiveMQ internally and so we can’t simply use “apache-activemq-all.x.x.x.jar” file for communication. This prompted me to look at all possible ActiveMQ JAR files, and I see some files that I don’t understand, that seem to indicate some functionality for AMQP, for REST, for MQTT, for Stomp, and more. I wonder what those are for?
Fast forward to 3am this morning:
What if ActiveMQ is able to be a MQTT broker?
What if I’m able to view the topics using the ActiveMQ admin page?
What if ActiveMQ will somehow bridge between MQTT and JMS?
Could I integrate DevTest with this?
Would that allow me to test and virtualize MQTT services?
Does MQTT have enough structure to provide the payload, header properties, encoding information, and other requirements for scalable enterprise use (i.e.: is it mature enough) to let me create something useful in DevTest?
I had to try it out!
First thing – investigate ActiveMQ. I immediately fell down a rabbit-hole of ActiveMQ -> ServiceMix -> Mule -> WSO2 -> JBoss Fuse. Hmm … reading about them, we should be able to apply service virtualization to all of these technology stacks with little difficulty, so long as the mock endpoints are coded during development, but that is all outside the scope of my requirement. I don’t need to integrate with routing gateways, service catalogues, OSGi, Enterprise Integration Patterns, or anything like that right now (although I see immediately where DevTest adds value to all of those and how we could improve integration with them), so I back out until I've returned to ActiveMQ information.
According to the documentation, there’s a config file somewhere that lets ActiveMQ be a broker for lots of different protocols, and it apparently translates between them all invisibly! This is looking hopeful!
So I install ActiveMQ onto my monitoring server. I configure ActiveMQ for automatic start-up, and I find the config file. It looks like many of the transports are enabled out-of-the-box, and the MQTT connector is running on the default MQTT port, so I halt my Mosquitto broker and start ActiveMQ. My MQTT client processes don’t even notice the change – they just continue working!
I visit the ActiveMQ default monitoring page, with hope in my heart. All of the topics are there, and ActiveMQ has even added advisory topics for each one so it can be monitored by JMX. I’m impressed! This solves my home automation problem, giving me a more enterprise grade ESB. But what’s the cost? ActiveMQ uses the Apache licence, so there’s no licensing costs. Looking at my system monitoring, I see an extra 10% of RAM is used by my server, but CPU and disk space don’t seem to be affected. So ActiveMQ is heavier than Mosquitto, but I can cope with that overhead for the benefits I’m getting.
What about the DevTest side? I read a little about how ActiveMQ translates between MQTT and JMS. It looks like “/” separators are replaced by “.”, and “#” is replaced by “>”. It doesn’t look like $SYS/Broker topics are translated, but JMX is a better solution, so I’m not worried about that.
I configure a JMS topic in DevTest that happens to have the same name as one of the MQTT topics, and I create a quick subscriber in DevTest. I then publish a message on MQTT, and I immediately see DevTest reading the message! Marvellous! But hold on – what’s this? DevTest is seeing it as a binary message, but everything in my home automation is text strings, JSON objects, XML and arrays. I’ll need to investigate this, after I test virtual service functionality.
I configure a Node-Red to use a pair of topics to use client-side and a pair of topics to use server-side.
I build a JSON message and configure it to send to the client-side request topic.
I link the server-side request and server-side response topics.
I listen to the client-side response topic and log the output.
I configure DevTest to use a pair of proxy topics and a pair of live topics. In DevTest nomenclature, “proxy” is short-hand for topics that are usually client-side, and changeable by the team who need to perform service virtualization, and “live” is short-hand for topics that are usually server-side, and not changeable by the team who need to perform service virtualization.
I go through the Virtual Service generation wizard, selecting the options to record JMS virtual services.
I start the flow in Node-Red.
DevTest starts recording services immediately. I’m impressed!
During the recording, I look at the meta-data. MQTT isn’t sending any custom header properties. It looks like it doesn’t support advanced messaging facilities like this.
I look at the recorded data. It’s being shown as binary. I need to investigate this now.
I look again at the meta-data. DevTest is seeing the message as a javax.jms.BytesMessage, but I’m sending text from MQTT.
I go to Node-Red and change my message to a JSON object instead of a JSON string. DevTest sees it as the same sort of message.
I go to Node-Red and change my message to a piece of raw text. DevTest sees it as the same sort of message.
I go to Node-Red, but I can’t see any way of reporting what type of message is being sent.
I read the ActiveMQ documentation. It says that MQTT doesn’t have the capability to notify subscribers of message type, and messages can contain any type of data, so ActiveMQ always reports javax.jms.BytesMessage for safety, and leaves it to the subscriber to understand what is being sent and decode it.
Therefore, I will need to tell DevTest what it’s expecting, and deal with it accordingly.
In DevTest, we have the Scriptable Data Protocol Handler, so I take advantage of this facility to decode and normalize the messages.
Request-side scriptable DPH:
byte mqttMessage = lisa_vse_request.getBodyBytes();
String textMessage = new String(mqttMessage);
Response-side recording scriptable DPH:
byte mqttMessage = lisa_vse_response.getBodyBytes();
String textMessage = new String(mqttMessage);
Response-side replay scriptable DPH:
String textMessage = lisa_vse_response.getBodyText();
byte mqttMessage = textMessage.getBytes();
I complete recording and deploy the virtual service.
I go to Node-Red and start the flow. My virtual service responds correctly. DevTest is now perfectly supporting MQTT for service virtualization.
I go to DevTest workstation and send a JMS message to the request “live” topic. DevTest receives a response from Node-Red. DevTest is now perfectly supporting MQTT for testing.
So, simply by deploying ActiveMQ and making sure the transports are enabled, DevTest now has perfect functionality for testing and virtualizing MQTT. Assuming the other transports in ActiveMQ work in the same way, DevTest now has perfect functionality for testing and virtualizing all of them. For protocols that DevTest will work with in this manner, see the webpage https://activemq.apache.org/protocols.html