The Unified Namespace (UNS) within the United Manufacturing Hub is a vital module
facilitating the streamlined flow and management of data. It comprises various
microservices:
data-bridge:
Bridges data between MQTT and Kafka and between multiple Kafka instances, ensuring
efficient data transmission.
HiveMQ:
An MQTT broker crucial for receiving data from IoT devices on the shop floor.
Redpanda (Kafka):
Manages large-scale data processing and orchestrates communication between microservices.
Redpanda Console:
Offers a graphical interface for monitoring Kafka topics and messages.
The UNS serves as a pivotal point in the UMH architecture, ensuring data from shop
floor systems and sensors (gathered via the Data Connectivity module) is effectively
processed and relayed to the Historian and external Data Warehouses/Data Lakes
for storage and analysis.
graph LR
4["`**Data Warehouse/Data Lake**
Stores data for analysis, on-premise or in the cloud`"]
style 4 fill:#f4f4f4,stroke:#f4f4f4,color:#000000
64["`**Historian**
Stores events in a time-series database and provides visualization tools.`"]
style 64 fill:#aaaaaa,stroke:#47a0b5,color:#000000
85["`**Connectivity**
Includes tools and services for connecting various shop floor systems and sensors.`"]
style 85 fill:#aaaaaa,stroke:#47a0b5,color:#000000
subgraph 51 [Unified Namespace]
style 51 fill:#ffffff,stroke:#47a0b5,color:#47a0b5
52["`**Redpanda (Kafka)**
Handles large-scale data processing and communication between microservices.`"]
style 52 fill:#aaaaaa,stroke:#47a0b5,color:#000000
53["`**HiveMQ**
MQTT broker used for receiving data from IoT devices on the shop floor.`"]
style 53 fill:#aaaaaa,stroke:#47a0b5,color:#000000
54["`**Redpanda Console**
Provides a graphical view of topics and messages in Kafka.`"]
style 54 fill:#aaaaaa,stroke:#47a0b5,color:#000000
55["`**databridge**
Bridges messages between MQTT and Kafka as well as between Kafka and other Kafka instances.`"]
style 55 fill:#aaaaaa,stroke:#47a0b5,color:#000000
end
54-.->52
52<-.->55
55<-.->53
55-. Provides data .->4
52-. Stores data in a
predefined schema .->64
85-. Provides
contextualized data .->53
85-. Provides
contextualized data .->52
Data-bridge is a microservice specifically tailored to adhere to the
UNS
data model. It consumes topics from a message broker, translates them to
the proper format and publishes them to the other message broker.
How it works
Data-bridge connects to the source broker, that can be either Kafka or MQTT,
and subscribes to the topics specified in the configuration. It then processes
the messages, and publishes them to the destination broker, that can be either
Kafka or MQTT.
In the case where the destination broker is Kafka, messages from multiple topics
can be merged into a single topic, making use of the message key to identify
the source topic.
For example, subscribing to a topic using a wildcard, such as
umh.v1.acme.anytown..*, and a merge point of 4, will result in
messages from the topics umh.v1.acme.anytown.foo.bar,
umh.v1.acme.anytown.foo.baz, umh.v1.acme.anytown and umh.v1.acme.anytown.frob
being merged into a single topic, umh.v1.acme.anytown, with the message key
being the missing part of the topic name, in this case foo.bar, foo.baz, etc.
The value of the message is not changed, only the topic and key are modified.
Another important feature is that it is possible to configure multiple data
bridges, each with its own source and destination brokers, and each with its
own set of topics to subscribe to and merge point.
The brokers can be local or remote, and, in case of MQTT, they can be secured
using TLS.
What’s next
Read the Data Bridge reference
documentation to learn more about the technical details of the data-bridge microservice.
2 - Kafka Broker
The Kafka broker in the United Manufacturing Hub is RedPanda,
a Kafka-compatible event streaming platform. It’s used to store and process
messages, in order to stream real-time data between the microservices.
How it works
RedPanda is a distributed system that is made up of a cluster of brokers,
designed for maximum performance and reliability. It does not depend on external
systems like ZooKeeper, as it’s shipped as a single binary.
Read the Kafka Broker reference documentation
to learn more about the technical details of the Kafka broker microservice
3 - Kafka Console
Kafka-console uses Redpanda Console
to help you manage and debug your Kafka workloads effortlessy.
With it, you can explore your Kafka topics, view messages, list the active
consumers, and more.
How it works
You can access the Kafka console via its Service.
It’s automatically connected to the Kafka broker, so you can start using it
right away.
You can view the Kafka broker configuration in the Broker tab, and explore the
topics in the Topics tab.
What’s next
Read the Kafka Console reference documentation
to learn more about the technical details of the Kafka Console microservice.
4 - MQTT Broker
The MQTT broker in the United Manufacturing Hub is HiveMQ
and is customized to fit the needs of the stack. It’s a core component of
the stack and is used to communicate between the different microservices.
How it works
The MQTT broker is responsible for receiving MQTT messages from the
different microservices and forwarding them to the
MQTT Kafka bridge.
What’s next
Read the MQTT Broker reference documentation
to learn more about the technical details of the MQTT Broker microservice.