3. Data Acquisition and Manipulation
5 minute read
The United Manufacturing Hub provides several simulators, which deliver different data through various protocols such as MQTT, PackML or OPC UA. In this chapter we will take the MQTT simulated data and show you how to format it into the UMH data model using Node-RED. It is highly recommended to read through the documentation of the UMH data model before continuing with this chapter.
For a deep dive into industry 4.0 data models, check out this article on our learn page.
Creating Node-RED flow with simulated MQTT-Data
Access the Node-RED web interface through your local installation’s device overview by selecting the open button on the Node-RED popup. You can access popups by clicking on the corresponding microservice tile.
To access the simulated raw data via MQTT, you need to add an MQTT broker. From the left-hand column, drag an mqtt-in node, an mqtt-out node, and a debug node into your flow and connect the mqtt-in and to the debug node.
Double-click on the mqtt-in node and click on the pencil button on the right of the Server field. Use the service name united-manufacturing-hub-mqtt as Server (or the name located in the Management Console Useful Information-tab). Leave the port as autoconfigured and click on Add to save your changes.
To view all incoming messages from a specific topic, type
ia/#under Topic and click on Done.
In brokers like MQTT or Kafka, data is structured hierarchically in topics. These topics categorize data into specific groups for efficient management. If you subscribe to a topic, you receive every message that is published to it. In the UMH data-model, the topics are defined in this format:
- MQTT:
ia/<customerID>/<location>/<AssetID>/<message> - Kafka:
ia.<customerID>.<location>.<AssetID>.<message>
customerID represents the enterprise, location signifies the
factory from which the data originates, AssetID denotes the specific machine
generating the data, and the final segment message is the name of the
message or data.
To apply the changes, click on Deploy located at the top right of the screen. Once the changes have been deployed, you can view the debug information by clicking on Debug-Messages located under Deploy.
Messages sent to the raw topic are not stored in the UMH database and are
not processed by the UMH stack. You can use them to build your own data
processing pipelines like in this example. For additional information,
refer to the
data-model documentation
In this column, you can view all incoming messages and their respective topics. The incoming topics follow this format:
ia/raw/development/ioTSensors/. To keep it simple, this tutorial will be only using the temperature topic, but feel free to choose any topic you’d like. Copy the temperature topicia/raw/development/ioTSensors/Temperature, open the mqtt-in node, paste the copied topic in the Topic field, click on Done, and then press Deploy again to apply the changes.
To format the incoming message according to the UMH data-model, add a function node, an mqtt-out and an mqtt-in to your flow. Connect the nodes in the following order: mqtt-in → function → mqtt-out and the second flow: mqtt-in → debug.
Incoming messages are not following the UMH data-model. To contextualize and format them correctly, you can use the different message types. This is needed, to ensure that the data is correctly processed by the UMH stack and other consumers.
In the documentation, you can find the complete list of all available message types and the respective topics.
Open the function node and paste in the following:
msg.payload = { "timestamp_ms": Date.now(), "temperature": msg.payload } return msg;Now click on Done, open the mqtt-out node and paste
ia/factoryinsight/Aachen/testing/processValuein the Topic field and click on Done. To apply the changes, click on Deploy. Now the message is correctly formatted into the data-model, specifically into a processValue message.
We are creating a new object, an array, with two keys timestamp_ms and
temperature and their corresponding values Date.now() and
msg.payload. The last value, msg.payload, is referring to the incoming
message. Since it is only carrying a single value, one does not need to
further specify it.
We also created a new topic in the mqtt-out node, where the messages will be sent to. The topic ends with the key processValue which is used whenever a custom process value with unique name has been prepared. The value is numerical. You can learn more about our message structure here.
- Open the second mqtt-in node, connected to the debug node, and set
the topic to
ia/factoryinsight/Aachen/testing/processValue, to receive all the new messages. Make sure to select the created broker. Click on Deploy to save the changes. The flows should now look like this:
You should now see the converted message under Debug-messages. To clear any previous messages, click on the trash bin icon.
Congratulations, you have successfully converted the incoming message and exported it via MQTT. However, since we are currently only exporting the temperature without actually working with the data, let’s create a function that counts the critical temperature exceedances.
Drag another function-node into your page, open it and navigate to On Start.
Paste in the following code, which will only run on start:
flow.set("count", 0); flow.set("current", 0)Then click on On-Message and paste in the following and click done:
flow.set("current",msg.payload); if (flow.get("current") > 47) { flow.set("count", flow.get("count") + 1); msg.payload = { "timestamp_ms":Date.now(), "TemperatureWarning":flow.get("count") } return msg; }The pasted in code will work for each new message as shown in the diagram below.
Finally, connect the function-node like shown below and click on deploy.
If the incoming value of temperature is now greater than 47, you will see another message consisting of TemperatureWarning and a timestamp in debug-messages.
What’s next?
Continue with the tutorial in the Management Console. The next chapter will be about the use of Grafana to display the formatted data to proceed.