3. Data Acquisition and Manipulation

Learn how to connect various data sources to the UMH and format data into the UMH data model.

The United Manufacturing Hub excels in its ability to integrate diverse data sources and standardize data into a unified model, enabling seamless integration of existing data infrastructure for analysis and processing.

Currently, data sources can be connected to the UMH through Benthos for OPC UA and Node-RED for other types.

The UMH includes 3 pre-configured data simulators for testing connections:

Connect OPC UA Data Sources

OPC UA, often complex, can be streamlined using our Benthos-based OPC UA connector accessible from the Management Console.

Create a Connection with the Management Console

After logging into the Management Console and selecting your instance, navigate to the Connection Management tab, where you’ll find all your connections alongside their status.

Connection Management
Connection Management

Uninitialized Connections are established but not yet configured as data sources, while Initialized Connections are fully configured.

The health status reflects the UMH-data source connection, not data transmission status.

To add a new connection, click Add Connection. Currently, we only provide two type of connections:

  • OPC-UA Server: represents a connection to an OPC-UA server.
  • n/a: represents a generic asset (useful for connections we don’t support yet).

Enter the required server details, which include the unique name and address with the format ip:port. Optionally, you can also attach some notes to the connection, which can be useful for documentation purposes.

For testing with the OPC UA simulator, select the OPC-UA Server type and use the following address:

united-manufacturing-hub-opcuasimulator-service:46010

Connection Details
Connection Details

Test the connection, and if successful, click Add Connection to save and deploy it.

Initialize the Connection

Back at Connection Management, your new connection should be listed in the table, and surely you’ll notice that it’s health is reported as Not configured.

At this point, it’s worth discussing what initializing a connection means and why it’s important.

New connections are created in an “uninitialized” state, meaning they are not yet configured as data sources, hence the Not configured health status. So for them to be actually useful, they need to be initialized, which will fully configure them as data sources and create a new Benthos deployment for data publishing to the UMH Kafka broker.

Initialize the connection by pressing the “play” button under the Actions column.

Initialize Connection
Initialize Connection

Enter authentication details (use Anonymous for no authentication, as with the OPC UA simulator).

Specify OPC UA nodes to subscribe to in a yaml file, following the ISA95 standard:

  nodes:
    - opcuaID: ns=2;s=Pressure
      enterprise: pharma-genix
      site: aachen
      area: packaging
      line: packaging_1
      workcell: blister
      originID: PLC13
      tagName: machineState
      schema: _historian

Mandatory fields are opcuaID, enterprise, tagName and schema.

Learn more about Data Modeling in the Unified Namespace in the Learning Hub.

Review and confirm the nodes, then proceed with initialization. Successful initialization will be indicated by a green message.

The connection’s health status should now be marked as Healthy and display the current message rate. You can also check the tooltip for more details.

Connection Management
Connection Management

Connect MQTT Servers

There are a lot of options to connect an MQTT server to the UMH. For this guide, we’ll use Node-RED to connect to the MQTT simulator and format data into the UMH data model.

To access Node-RED’s web interface, navigate to:

http://<instance-ip-address>:1880/nodered

Replace <instance-ip-address> with your UMH instance’s IP. Ensure you’re on the same network for access.

Add the MQTT Connection

In Node-RED, find the mqtt-in node from the node palette and drag it into your flow. Double-click to configure and click the pencil button next to the Server field.

Enter your MQTT broker’s details:

  • Server: united-manufacturing-hub-mqtt
  • Port: 1883

For the purpose of this guide, we’ll use the UMH MQTT broker, even though the data coming from it is already bridged to Kafka by the MQTT Kafka Bridge. Since the simulated data is using the old Data Model, we’ll use Node-RED to convert it to the new Data Model.

Click Add to save.

Connect MQTT to Node-RED
Connect MQTT to Node-RED

Define the subscription topic. For example, ia/raw/development/ioTSensors/Temperature is used by the MQTT Simulator.

To test, link a debug node to the mqtt-in node and deploy. Open the debug pane by clicking on the bug icon on the top right of the screen to view messages from the broker.

MQTT Debug Connection
MQTT Debug Connection

Explore Unified Namespace for details on topic structuring.

Format Incoming Messages

Use a function node to format raw data. Connect it to the mqtt-in node and paste this script:

msg.payload = {
  timestamp_ms: Date.now(),
  temperature: msg.payload,
};
return msg;

Finalize with Done.

Then, connect a JSON node to the function node to parse the object into a string.

This function transforms the payload into the correct format for the UMH data model.

Send Formatted Data to Kafka

For this guide, we’ll send data to the UMH Kafka broker.

Ensure you have node-red-contrib-kafkajs installed. If not, see How to Get Missing Plugins in Node-RED.

Add a kafka-producer node, connecting it to the JSON node. Configure as follows:

  1. Open the configuration menu by double-click on the kafka-producer node. After that, click on the edit button.

    Node-RED Kafka Producer
    Node-RED Kafka Producer

  2. Change the fields of Brokers and Client ID as follows:

    • Brokers: united-manufacturing-hub-kafka:9092
    • Client ID: nodered

    Node-RED Kafka Broker Configuration
    Node-RED Kafka Broker Configuration

    Click on Update to save.

  3. Structure Kafka topics according to UMH data model, following the ISA95 standard:

    umh.v1.<enterprise>.<site>.<area>.<line>.<workcell>.<originID>.<schema>.<tagName>
    
    • umh.v1: obligatory versioning prefix
    • enterprise: The company’s name
    • site: The facility’s location
    • area: The specific production’s area
    • line: The production line
    • workcell: The workcell in the production line
    • originID: The data source ID
    • schema: The schema of your data
    • tagName: Arbitrary tags dependent context

    The enterprise and schema fields are required. To learn more about the UMH data-model, read the documentation.

    For example, if you want to structure a topic for the temperature in celsius from the PLC, which

    • is running in a factory of Pharma-Genix in Aachen.
    • is running in blister workcell in the packaging line 1 in the packaging area.
    • has the ID PLC13.

    and you want to use _historian schema, then the topic should look like

    msg.topic = umh.v1.pharma-genix.aachen.packaging.packaging_1.blister.PLC13._historian.temperatureCelsius
    

    Add this topic to the script in the function node, which created in Format Incoming Messages section.

    Node-RED Kafka Topic
    Node-RED Kafka Topic

    Alternatively, you can set the topic to the kafka-producer node directly.

  4. Click Done and deploy.

Optional: Add a debug node for output visualization.

Node-RED MQTT to Kafka
Node-RED MQTT to Kafka

Connect Kafka Data Sources

Kafka data sources can be integrated with UMH exclusively through Node-RED.

To access Node-RED’s web interface, navigate to:

http://<instance-ip-address>:1880/nodered

Replace <instance-ip-address> with your UMH instance’s IP, ensuring you’re on the same network for access.

Before proceeding, make sure the node-red-contrib-kafkajs plugin is installed. For installation guidance, see How to Get Missing Plugins in Node-RED.

Add the Kafka Connection

In Node-RED, locate the kafka-consumer node and drag it into your flow. Double-click to configure and click the pencil button beside the Server field.

If you have followed the guide, the kafka client should already be configured and automatically selected.

Enter your Kafka broker’s details:

  • Brokers: united-manufacturing-hub-kafka:9092
  • Client ID: nodered

Click Add to save.

Connect Kafka to Node-RED
Connect Kafka to Node-RED

Set the subscription topic. For demonstration, we’ll use the topic created earlier:

umh.v1.pharma-genix.aachen.packaging.packaging_1.blister.PLC13._historian.temperatureCelsius

Link a debug node to the kafka-consumer node, deploy, and observe messages in the debug pane.

Kafka Debug Connection
Kafka Debug Connection

For topic structuring guidelines, refer to Unified Namespace.

Format Incoming Messages

Since the data is already processed from the previous step, use a function node to convert the temperature from Celsius to Fahrenheit. Connect it to the kafka-consumer node and paste the following script:

const payloadObj = JSON.parse(msg.payload.value);
const celsius = payloadObj.temperature;
const fahrenheit = (celsius * 9) / 5 + 32;

msg.payload = {
  timestamp_ms: Date.now(),
  temperature: fahrenheit,
};

return msg;

Finalize with Done.

Then, connect a JSON node to the function node to parse the object into a string.

Send Formatted Data Back to Kafka

Now, we’ll route the transformed data back to the Kafka broker, in a different topic.

Add a kafka-producer node, connecting it to the JSON node. Use the same Kafka client as earlier, and the same topic for output:

umh.v1.pharma-genix.aachen.packaging.packaging_1.blister.PLC13._historian.temperatureFahrenheit

For more on UMH data modeling, consult the documentation.

Press Done and deploy.

Consider adding a debug node for visualizing output data.

Node-RED Kafka to Kafka
Node-RED Kafka to Kafka

What’s next

Next, we’ll dive into Data Visualization, where you’ll learn to create Grafana dashboards using your newly configured data sources. This next chapter will help you visualize and interpret your data effectively.