This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Core

This section contains the overview of the Core components of the United Manufacturing Hub.

The microservices in this section are part of the Core of the United Manufacturing Hub. They are mainly developed by the UMH team, with the addition of some third-party software. They are used to provide the core functionality of the platform.

1 - Cache

The technical documentation of the redis microservice, which is used as a cache for the other microservices.

The cache in the United Manufacturing Hub is Redis, a key-value store that is used as a cache for the other microservices.

How it works

Recently used data is stored in the cache to reduce the load on the database. All the microservices that need to access the database will first check if the data is available in the cache. If it is, it will be used, otherwise the microservice will query the database and store the result in the cache.

By default, Redis is configured to run in standalone mode, which means that it will only have one master node.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-redis-master
  • Service:
    • Internal ClusterIP:
      • Redis: united-manufacturing-hub-redis-master at port 6379
      • Headless: united-manufacturing-hub-redis-headless at port 6379
      • Metrics: united-manufacturing-hub-redis-metrics at port 6379
  • ConfigMap:
    • Configuration: united-manufacturing-hub-redis-configuration
    • Health: united-manufacturing-hub-redis-health
    • Scripts: united-manufacturing-hub-redis-scripts
  • Secret: redis-secret
  • PersistentVolumeClaim: redis-data-united-manufacturing-hub-redis-master-0

Configuration

You shouldn’t need to configure the cache manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the redis section of the Helm chart values file.

You can consult the Bitnami Redis chart for more information about the available configuration options.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
ALLOW_EMPTY_PASSWORDAllow empty passwordbooltrue, falsefalse
BITNAMI_DEBUGSpecify if debug values should be setbooltrue, falsefalse
REDIS_PASSWORDRedis passwordstringAnyRandom UUID
REDIS_PORTRedis port numberintAny6379
REDIS_REPLICATION_MODERedis replication modestringmaster, slavemaster
REDIS_TLS_ENABLEDEnable TLSbooltrue, falsefalse

2 - Database

The technical documentation of the database microservice, which stores the data of the application.

The database microservice is the central component of the United Manufacturing Hub and is based on TimescaleDB, an open-source relational database built for handling time-series data. TimescaleDB is designed to provide scalable and efficient storage, processing, and analysis of time-series data.

You can find more information on the datamodel of the database in the Data Model section, and read about the choice to use TimescaleDB in the blog article.

How it works

When deployed, the database microservice will create two databases, with the related usernames and passwords:

  • grafana: This database is used by Grafana to store the dashboards and other data.
  • factoryinsight: This database is the main database of the United Manufacturing Hub. It contains all the data that is collected by the microservices.

Then, it creates the tables based on the database schema.

If you want to learn more about how TimescaleDB works, you can read the TimescaleDB documentation.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-timescaledb
  • Service:
    • Internal ClusterIP for the replicas: united-manufacturing-hub-replica at port 5432
    • Internal ClusterIP for the config: united-manufacturing-hub-config at port 8008
    • External LoadBalancer: united-manufacturing-hub at port 5432
  • ConfigMap:
    • Patroni: united-manufacturing-hub-timescaledb-patroni
    • Post init: timescale-post-init
    • Postgres BackRest: united-manufacturing-hub-timescaledb-pgbackrest
    • Scripts: united-manufacturing-hub-timescaledb-scripts
  • Secret:
    • Certificate: united-manufacturing-hub-certificate
    • Patroni credentials: united-manufacturing-hub-credentials
    • Users passwords: timescale-post-init-pw
  • PersistentVolumeClaim:
    • Data: storage-volume-united-manufacturing-hub-timescaledb-0
    • WAL-E: wal-volume-united-manufacturing-hub-timescaledb-0

Configuration

There is only one parameter that usually needs to be changed: the password used to connect to the database. To do so, set the value of the db_password key in the _000_commonConfig.datastorage section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
BOOTSTRAP_FROM_BACKUPWhether to bootstrap the database from a backup or not.int0, 10
PATRONI_KUBERNETES_LABELSThe labels to use to find the pods of the StatefulSet.stringAny{app: united-manufacturing-hub-timescaledb, cluster-name: united-manufacturing-hub, release: united-manufacturing-hub}
PATRONI_KUBERNETES_NAMESPACEThe namespace in which the StatefulSet is deployed.stringAnyunited-manufacturing-hub
PATRONI_KUBERNETES_POD_IPThe IP address of the pod.stringAnyRandom IP
PATRONI_KUBERNETES_PORTSThe ports to use to connect to the pods.stringAny[{"name": "postgresql", "port": 5432}]
PATRONI_NAMEThe name of the pod.stringAnyunited-manufacturing-hub-timescaledb-0
PATRONI_POSTGRESQL_CONNECT_ADDRESSThe address to use to connect to the database.stringAny$(PATRONI_KUBERNETES_POD_IP):5432
PATRONI_POSTGRESQL_DATA_DIRThe directory where the database data is stored.stringAny/var/lib/postgresql/data
PATRONI_REPLICATION_PASSWORDThe password to use to connect to the database as a replica.stringAnyRandom 16 characters
PATRONI_REPLICATION_USERNAMEThe username to use to connect to the database as a replica.stringAnystandby
PATRONI_RESTAPI_CONNECT_ADDRESSThe address to use to connect to the REST API.stringAny$(PATRONI_KUBERNETES_POD_IP):8008
PATRONI_SCOPEThe name of the cluster.stringAnyunited-manufacturing-hub
PATRONI_SUPERUSER_PASSWORDThe password to use to connect to the database as the superuser.stringAnyRandom 16 characters
PATRONI_admin_OPTIONSThe options to use for the admin user.stringComma separated list of optionscreaterole,createdb
PATRONI_admin_PASSWORDThe password to use to connect to the database as the admin user.stringAnyRandom 16 characters
PGBACKREST_CONFIGThe path to the configuration file for Postgres BackRest.stringAny/etc/pgbackrest/pgbackrest.conf
PGDATAThe directory where the database data is stored.stringAny$(PATRONI_POSTGRESQL_DATA_DIR)
PGHOSTThe directory of the runnning databasestringAny/var/run/postgresql

3 - Factoryinsight

The technical documentation of the Factoryinsight microservice, which exposes a set of APIs to access the data from the database.

Factoryinsight is a microservice that provides a set of REST APIs to access the data from the database. It is particularly useful to calculate the Key Performance Indicators (KPIs) of the factories.

How it works

Factoryinsight exposes REST APIs to access the data from the database or calculate the KPIs. By default, it’s only accessible from the internal network of the cluster, but it can be configured to be accessible from the external network.

The APIs require authentication, that can be ehither a Basic Auth or a Bearer token. Both of these can be found in the Secret factoryinsight-secret.

API documentation

Kubernetes resources

  • Deployment: united-manufacturing-hub-factoryinsight-deployment
  • Service:
  • Secret: factoryinsight-secret

Configuration

You shouldn’t need to configure Factoryinsight manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the factoryinsight section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
CUSTOMER_NAME_{NUMBER}Specifies a user for the REST API. Multiple users can be setstringAny""
CUSTOMER_PASSWORD_{NUMBER}Specifies the password of the user for the REST APIstringAny""
DEBUG_ENABLE_FGTRACEEnables the use of the fgtrace library. Not reccomended for productionstringtrue, falsefalse
DRY_RUNIf enabled, data wont be stored in databasebooltrue, falsefalse
FACTORYINSIGHT_PASSWORDSpecifies the password for the admin user for the REST APIstringAnyRandom UUID
FACTORYINSIGHT_USERSpecifies the admin user for the REST APIstringAnyfactoryinsight
INSECURE_NO_AUTHIf enabled, no authentication is required for the REST API. Not reccomended for productionbooltrue, falsefalse
LOGGING_LEVELDefines which logging level is used, mostly relevant for developersstringPRODUCTION, DEVELOPMENTPRODUCTION
MICROSERVICE_NAMEName of the microservice. Used for tracingstringAnyunited-manufacturing-hub-factoryinsight
POSTGRES_DATABASESpecifies the database name to usestringAnyfactoryinsight
POSTGRES_HOSTSpecifies the database DNS name or IP addressstringAnyunited-manufacturing-hub
POSTGRES_PASSWORDSpecifies the database password to usestringAnychangeme
POSTGRES_PORTSpecifies the database portintValid port number5432
POSTGRES_USERSpecifies the database user to usestringAnyfactoryinsight
REDIS_PASSWORDPassword to access the redis sentinelstringAnyRandom UUID
REDIS_URIThe URI of the Redis instancestringAnyunited-manufacturing-hub-redis-headless:6379
SERIAL_NUMBERSerial number of the cluster. Used for tracingstringAnydefalut
VERSIONThe version of the API used. Each version also enables all the previous onesintAny2

4 - Grafana

The technical documentation of the grafana microservice, which is a web application that provides visualization and analytics capabilities.

The grafana microservice is a web application that provides visualization and analytics capabilities. Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

It has a rich ecosystem of plugins that allow you to extend its functionality beyond the core features.

How it works

Grafana is a web application that can be accessed through a web browser. It let’s you create dashboards that can be used to visualize data from the database.

Thanks to some custom datasource plugins, Grafana can use the various APIs of the United Manufacturing Hub to query the database and display useful information.

Kubernetes resources

  • Deployment: united-manufacturing-hub-grafana
  • Service:
    • External LoadBalancer: united-manufacturing-hub-grafana at port 8080
  • ConfigMap: united-manufacturing-hub-grafana
  • Secret: grafana-secret
  • PersistentVolumeClaim: united-manufacturing-hub-grafana

Configuration

Grafana is configured through its user interface. The default credentials are found in the grafana-secret Secret.

The Grafana installation that is provided by the United Manufacturing Hub is shipped with a set of preinstalled plugins:

  • ACE.SVG by Andrew Rodgers
  • Button Panel by CloudSpout LLC
  • Button Panel by UMH Systems Gmbh
  • Discrete by Natel Energy
  • Dynamic Text by Marcus Olsson
  • FlowCharting by agent
  • Pareto Chart by isaozler
  • Pie Chart (old) by Grafana Labs
  • Timepicker Buttons Panel by williamvenner
  • UMH Datasource by UMH Systems Gmbh
  • UMH Datasource v2 by UMH Systems Gmbh
  • Untimely by factry
  • Worldmap Panel by Grafana Labs

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
FACTORYINSIGHT_APIKEYThe API key to use to authenticate to the Factoryinsight APIstringAnyBase64 encoded string
FACTORYINSIGHT_BASEURLThe base URL of the Factoryinsight APIstringAnyunited-manufacturing-hub-factoryinsight-service
FACTORYINSIGHT_CUSTOMERIDThe customer ID to use to authenticate to the Factoryinsight APIstringAnyfactoryinsight
FACTORYINSIGHT_PASSWORDThe password to use to authenticate to the Factoryinsight APIstringAnyRandom UUID
GF_PATHS_DATAThe path where Grafana will store its datastringAny/var/lib/grafana/data
GF_PATHS_LOGSThe path where Grafana will store its logsstringAny/var/log/grafana
GF_PATHS_PLUGINSThe path where Grafana will store its pluginsstringAny/var/lib/grafana/plugins
GF_PATHS_PROVISIONINGThe path where Grafana will store its provisioning configurationstringAny/etc/grafana/provisioning
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINSList of plugin identifiers to allow loading even if they lack a valid signaturestringComma separated listumh-datasource,umh-factoryinput-panel,umh-v2-datasource
GF_SECURITY_ADMIN_PASSWORDThe password of the admin userstringAnyRandom UUID
GF_SECURITY_ADMIN_USERThe username of the admin userstringAnyadmin

5 - Kafka Bridge

The technical documentation of the kafka-bridge microservice, which acts as a communication bridge between two Kafka brokers.

Kafka-bridge is a microservice that connects two Kafka brokers and forwards messages between them. It is used to connect the local broker of the edge computer with the remote broker on the server.

How it works

This microservice has two ways of operation:

  • High Integrity: This mode is used for topics that are critical for the user. It is garanteed that no messages are lost. This is achieved by committing the message only after it has been successfully inserted into the database. Ususally all the topics are forwarded in this mode, except for processValue, processValueString and raw messages.
  • High Throughput: This mode is used for topics that are not critical for the user. They are forwarded as fast as possible, but it is possible that messages are lost, for example if the database struggles to keep up. Usually only the processValue, processValueString and raw messages are forwarded in this mode.

Kubernetes resources

  • Deployment: united-manufacturing-hub-kafkabridge
  • Secret:
    • Local broker: united-manufacturing-hub-kafkabridge-secrets-local
    • Remote broker: united-manufacturing-hub-kafkabridge-secrets-remote

Configuration

You can configure the kafka-bridge microservice by setting the following values in the _000_commonConfig.kafkaBridge section of the Helm chart values file.

  kafkaBridge:
    enabled: true
    remotebootstrapServer: ""
    topicmap:
      - bidirectional: false
        name: HighIntegrity
        send_direction: to_remote
        topic: ^ia\..+\..+\..+\.((addMaintenanceActivity)|(addOrder)|(addParentToChild)|(addProduct)|(addShift)|(count)|(deleteShiftByAssetIdAndBeginTimestamp)|(deleteShiftById)|(endOrder)|(modifyProducedPieces)|(modifyState)|(productTag)|(productTagString)|(recommendation)|(scrapCount)|(startOrder)|(state)|(uniqueProduct)|(scrapUniqueProduct))$
      - bidirectional: false
        name: HighThroughput
        send_direction: to_remote
        topic: ^ia\..+\..+\..+\.(processValue).*$

Topic Map schema

The topic map is a list of objects, each object represents a topic (or a set of topics) that should be forwarded. The following JSON schema describes the structure of the topic map:

{
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "array",
    "title": "Kafka Topic Map",
    "description": "This schema validates valid Kafka topic maps.",
    "default": [],
    "additionalItems": true,
    "items": {
        "$id": "#/items",
        "anyOf": [
            {
                "$id": "#/items/anyOf/0",
                "type": "object",
                "title": "Unidirectional Kafka Topic Map with send direction",
                "description": "This schema validates entries, that are unidirectional and have a send direction.",
                "default": {},
                "examples": [
                    {
                        "name": "HighIntegrity",
                        "topic": "^ia\\..+\\..+\\..+\\.(?!processValue).+$",
                        "bidirectional": false,
                        "send_direction": "to_remote"
                    }
                ],
                "required": [
                    "name",
                    "topic",
                    "bidirectional",
                    "send_direction"
                ],
                "properties": {
                    "name": {
                        "$id": "#/items/anyOf/0/properties/name",
                        "type": "string",
                        "title": "Entry Name",
                        "description": "Name of the map entry, only used for logging & tracing.",
                        "default": "",
                        "examples": [
                            "HighIntegrity"
                        ]
                    },
                    "topic": {
                        "$id": "#/items/anyOf/0/properties/topic",
                        "type": "string",
                        "title": "The topic to listen on",
                        "description": "The topic to listen on, this can be a regular expression.",
                        "default": "",
                        "examples": [
                            "^ia\\..+\\..+\\..+\\.(?!processValue).+$"
                        ]
                    },
                    "bidirectional": {
                        "$id": "#/items/anyOf/0/properties/bidirectional",
                        "type": "boolean",
                        "title": "Is the transfer bidirectional?",
                        "description": "When set to true, the bridge will consume and produce from both brokers",
                        "default": false,
                        "examples": [
                            false
                        ]
                    },
                    "send_direction": {
                        "$id": "#/items/anyOf/0/properties/send_direction",
                        "type": "string",
                        "title": "Send direction",
                        "description": "Can be either 'to_remote' or 'to_local'",
                        "default": "",
                        "examples": [
                            "to_remote",
                            "to_local"
                        ]
                    }
                },
                "additionalProperties": true
            },
            {
                "$id": "#/items/anyOf/1",
                "type": "object",
                "title": "Bi-directional Kafka Topic Map with send direction",
                "description": "This schema validates entries, that are bi-directional.",
                "default": {},
                "examples": [
                    {
                        "name": "HighIntegrity",
                        "topic": "^ia\\..+\\..+\\..+\\.(?!processValue).+$",
                        "bidirectional": true
                    }
                ],
                "required": [
                    "name",
                    "topic",
                    "bidirectional"
                ],
                "properties": {
                    "name": {
                        "$id": "#/items/anyOf/1/properties/name",
                        "type": "string",
                        "title": "Entry Name",
                        "description": "Name of the map entry, only used for logging & tracing.",
                        "default": "",
                        "examples": [
                            "HighIntegrity"
                        ]
                    },
                    "topic": {
                        "$id": "#/items/anyOf/1/properties/topic",
                        "type": "string",
                        "title": "The topic to listen on",
                        "description": "The topic to listen on, this can be a regular expression.",
                        "default": "",
                        "examples": [
                            "^ia\\..+\\..+\\..+\\.(?!processValue).+$"
                        ]
                    },
                    "bidirectional": {
                        "$id": "#/items/anyOf/1/properties/bidirectional",
                        "type": "boolean",
                        "title": "Is the transfer bidirectional?",
                        "description": "When set to true, the bridge will consume and produce from both brokers",
                        "default": false,
                        "examples": [
                            true
                        ]
                    }
                },
                "additionalProperties": true
            }
        ]
    },
    "examples": [
   {
      "name":"HighIntegrity",
      "topic":"^ia\\..+\\..+\\..+\\.(?!processValue).+$",
      "bidirectional":true
   },
   {
      "name":"HighThroughput",
      "topic":"^ia\\..+\\..+\\..+\\.(processValue).*$",
      "bidirectional":false,
      "send_direction":"to_remote"
   }
]
}

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
DEBUG_ENABLE_FGTRACEEnables the use of the fgtrace library, do not enable in productionstringtrue, falsefalse
KAFKA_GROUP_ID_SUFFIXIdentifier appended to the kafka group ID, usually a serial numberstringAnydefalut
KAFKA_SSL_KEY_PASSWORD_LOCALPassword for the SSL key pf the local brokerstringAny""
KAFKA_SSL_KEY_PASSWORD_REMOTEPassword for the SSL key of the remote brokerstringAny""
KAFKA_TOPIC_MAPA json map of the kafka topics should be forwardedJSONSee below{}
KAKFA_USE_SSLEnables the use of SSL for the kafka connectionstringtrue, falsefalse
LOCAL_KAFKA_BOOTSTRAP_SERVERURL of the local kafka broker, port is requiredstringAny valid URLunited-manufacturing-hub-kafka:9092
LOGGING_LEVELDefines which logging level is used, mostly relevant for developers.stringPRODUCTION, DEVELOPMENTPRODUCTION
MICROSERVICE_NAMEName of the microservice (used for tracing)stringAnyunited-manufacturing-hub-kafka-bridge
REMOTE_KAFKA_BOOTSTRAP_SERVERURL of the remote kafka brokerstringAny valid URL""
SERIAL_NUMBERSerial number of the cluster (used for tracing)stringAnydefalut

6 - Kafka Broker

The technical documentation of the kafka-broker microservice, which handles the communication between the microservices.

The Kafka broker in the United Manufacturing Hub is RedPanda, a Kafka-compatible event streaming platform. It’s used to store and process messages, in order to stream real-time data between the microservices.

How it works

RedPanda is a distributed system that is made up of a cluster of brokers, designed for maximum performance and reliability. It does not depend on external systems like ZooKeeper, as it’s shipped as a single binary.

Read more about RedPanda in the official documentation.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-kafka
  • Service:
    • Internal ClusterIP (headless): united-manufacturing-hub-kafka
    • External NodePort: united-manufacturing-hub-kafka-external at port 9094 for the Kafka API listener, port 9644 for the Admin API listener, port 8083 for the HTTP Proxy listener, and port 8081 for the Schema Registry listener.
  • ConfigMap: united-manufacturing-hub-kafka
  • Secret: united-manufacturing-hub-kafka-sts-lifecycle
  • PersistentVolumeClaim: datadir-united-manufacturing-hub-kafka-0

Configuration

You shouldn’t need to configure the Kafka broker manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the redpanda section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
HOST_IPThe IP address of the host machine.stringAnyRandom IP
POD_IPThe IP address of the pod.stringAnyRandom IP
SERVICE_NAMEThe name of the service.stringAnyunited-manufacturing-hub-kafka

7 - Kafka Console

The technical documentation of the kafka-console microservice, which provides a GUI to interact with the Kafka broker.

Kafka-console uses Redpanda Console to help you manage and debug your Kafka workloads effortlessy.

With it, you can explore your Kafka topics, view messages, list the active consumers, and more.

How it works

You can access the Kafka console via its Service.

It’s automatically connected to the Kafka broker, so you can start using it right away. You can view the Kafka broker configuration in the Broker tab, and explore the topics in the Topics tab.

Kubernetes resources

  • Deployment: united-manufacturing-hub-console
  • Service:
    • External LoadBalancer: united-manufacturing-hub-console at port 8090
  • ConfigMap: united-manufacturing-hub-console
  • Secret: united-manufacturing-hub-console

Configuration

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
LOGIN_JWTSECRETThe secret used to authenticate the communication to the backend.stringAnyRandom string

8 - Kafka to Postgresql

The technical documentation of the kafka-to-postgresql microservice, which consumes messages from a Kafka broker and writes them in a PostgreSQL database.

Kafka-to-postgresql is a microservice responsible for consuming kafka messages and inserting the payload into a Postgresql database. Take a look at the Datamodel to see how the data is structured.

This microservice requires that the Kafka Topic umh.v1.kafka.newTopic exits. This will happen automatically from version 0.9.12.

How it works

By default, kafka-to-postgresql sets up two Kafka consumers, one for the High Integrity topics and one for the High Throughput topics.

The graphic below shows the program flow of the microservice.

Kafka-to-postgres-flow
Kafka-to-postgres-flow

High integrity

The High integrity topics are forwarded to the database in a synchronous way. This means that the microservice will wait for the database to respond with a non error message before committing the message to the Kafka broker. This way, the message is garanteed to be inserted into the database, even though it might take a while.

Most of the topics are forwarded in this mode.

The picture below shows the program flow of the high integrity mode.

high-integrity-data-flow
high-integrity-data-flow

High throughput

The High throughput topics are forwarded to the database in an asynchronous way. This means that the microservice will not wait for the database to respond with a non error message before committing the message to the Kafka broker. This way, the message is not garanteed to be inserted into the database, but the microservice will try to insert the message into the database as soon as possible. This mode is used for the topics that are expected to have a high throughput.

The topics that are forwarded in this mode are processValue, processValueString and all the raw topics.

Kubernetes resources

  • Deployment: united-manufacturing-hub-kafkatopostgresql
  • Secret: united-manufacturing-hub-kafkatopostgresql-certificates

Configuration

You shouldn’t need to configure kafka-to-postgresql manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the kafkatopostgresql section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
DEBUG_ENABLE_FGTRACEEnables the use of the fgtrace library. Not reccomended for productionstringtrue, falsefalse
DRY_RUNIf set to true, the microservice will not write to the databasebooltrue, falsefalse
KAFKA_BOOTSTRAP_SERVERURL of the Kafka broker used, port is requiredstringAnyunited-manufacturing-hub-kafka:9092
KAFKA_SSL_KEY_PASSWORDKey password to decode the SSL private keystringAny""
LOGGING_LEVELDefines which logging level is used, mostly relevant for developersstringPRODUCTION, DEVELOPMENTPRODUCTION
MEMORY_REQUESTMemory request for the message cachestringAny50Mi
MICROSERVICE_NAMEName of the microservice (used for tracing)stringAnyunited-manufacturing-hub-kafkatopostgresql
POSTGRES_DATABASEThe name of the PostgreSQL databasestringAnyfactoryinsight
POSTGRES_HOSTHostname of the PostgreSQL databasestringAnyunited-manufacturing-hub
POSTGRES_PASSWORDThe password to use for PostgreSQL connectionsstringAnychangeme
POSTGRES_SSLMODEIf set to true, the PostgreSQL connection will use SSLstringAnyrequire
POSTGRES_USERThe username to use for PostgreSQL connectionsstringAnyfactoryinsight

9 - MQTT Bridge

The technical documentation of the mqtt-bridge microservice, which acts as a communication bridge between two MQTT brokers.

MQTT-bridge is a microservice that connects two MQTT brokers and forwards messages between them. It is used to connect the local broker of the edge computer with the remote broker on the server.

How it works

This microservice subscribes to topics on the local broker and publishes the messages to the remote broker, while also subscribing to topics on the remote broker and publishing the messages to the local broker.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-mqttbridge
  • Secret: united-manufacturing-hub-mqttbridge-secrets
  • PersistentVolumeClaim: united-manufacturing-hub-mqttbridge-claim

Configuration

You can configure the URL of the remote MQTT broker that MQTT-bridge should connect to by setting the value of the remoteBrokerUrl parameter in the _000_commonConfig.mqttBridge section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
BRIDGE_ONE_WAYWhether to enable one-way communication, from local to remotebooltrue, falsetrue
INSECURE_SKIP_VERIFY_LOCALSkip TLS certificate verification for the local brokerbooltrue, falsetrue
INSECURE_SKIP_VERIFY_REMOTESkip TLS certificate verification for the remote brokerbooltrue, falsetrue
LOCAL_BROKER_SSL_ENABLEDWhether to enable SSL for the local MQTT brokerbooltrue, falsetrue
LOCAL_BROKER_URLURL for the local MQTT brokerstringAnyssl://united-manufacturing-hub-mqtt:8883
LOCAL_CERTIFICATE_NAMESet to NO_CERT to allow non-encrypted MQTT access, or to USE_TLS to use TLS encryptionstringUSE_TLS, NO_CERTUSE_TLS
LOCAL_PUB_TOPICLocal MQTT topic to publish tostringAnyia
LOCAL_SUB_TOPICLocal MQTT topic to subscribe tostringAnyia/factoryinsight
MQTT_PASSWORDPassword for the MQTT brokerstringAnyINSECURE_INSECURE_INSECURE
REMOTE_BROKER_SSL_ENABLEDWhether to enable SSL for the remote MQTT brokerbooltrue, falsetrue
REMOTE_BROKER_URLURL for the local MQTT brokerstringAnyssl://united-manufacturing-hub-mqtt.united-manufacturing-hub:8883
REMOTE_CERTIFICATE_NAMESet to NO_CERT to allow non-encrypted MQTT access, or to USE_TLS to use TLS encryptionstringUSE_TLS, NO_CERTUSE_TLS
REMOTE_PUB_TOPICRemote MQTT topic to publish tostringAnyia/factoryinsight
REMOTE_SUB_TOPICRemote MQTT topic to subscribe tostringAnyia

10 - MQTT Broker

The technical documentation of the mqtt-broker microservice, which forwards MQTT messages between the other microservices.

The MQTT broker in the United Manufacturing Hub is HiveMQ and is customized to fit the needs of the stack. It’s a core component of the stack and is used to communicate between the different microservices.

How it works

The MQTT broker is responsible for receiving MQTT messages from the different microservices and forwarding them to the MQTT Kafka bridge.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-hivemqce
  • Service:
    • Internal ClusterIP:
      • HiveMQ local: united-manufacturing-hub-hivemq-local-service at port 1883 (MQTT) and 8883 (MQTT over TLS)
      • VerneMQ (for backwards compatibility): united-manufacturing-hub-vernemq at port 1883 (MQTT) and 8883 (MQTT over TLS)
      • VerneMQ local (for backwards compatibility): united-manufacturing-hub-vernemq-local-service at port 1883 (MQTT) and 8883 (MQTT over TLS)
    • External LoadBalancer: united-manufacturing-hub-mqtt at port 1883 (MQTT) and 8883 (MQTT over TLS)
  • ConfigMap:
    • Configuration: united-manufacturing-hub-hivemqce-hive
    • Credentials: united-manufacturing-hub-hivemqce-extension
  • Secret: united-manufacturing-hub-hivemqce-secret-keystore
  • PersistentVolumeClaim:
    • Data: united-manufacturing-hub-hivemqce-claim-data
    • Extensions: united-manufacturing-hub-hivemqce-claim-extensions

Configuration

Most of the configuration is done through the XML files in the ConfigMaps. The default configuration should be sufficient for most use cases.

The HiveMQ installation of the United Manufacturing Hub comes with these extensions:

If you want to add more extensions, or to change the configuration, visit the HiveMQ documentation.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
HIVEMQ_ALLOW_ALL_CLIENTSWhether to allow all clients to connect to the brokerbooltrue, falsetrue

11 - MQTT Kafka Bridge

The technical documentation of the mqtt-kafka-bridge microservice, which transfers messages from MQTT brokers to Kafka Brokers and vice versa.

Mqtt-kafka-bridge is a microservice that acts as a bridge between MQTT brokers and Kafka brokers, transfering messages from one to the other and vice versa.

This microservice requires that the Kafka Topic umh.v1.kafka.newTopic exits. This will happen automatically from version 0.9.12.

Since version 0.9.10, it allows all raw messages, even if their content is not in a valid JSON format.

How it works

Mqtt-kafka-bridge consumes topics from a message broker, translates them to the proper format and publishes them to the other message broker.

Kubernetes resources

  • Deployment: united-manufacturing-hub-mqttkafkabridge
  • Secret:
    • Kafka: united-manufacturing-hub-mqttkafkabridge-kafka-secrets
    • MQTT: united-manufacturing-hub-mqttkafkabridge-mqtt-secrets

Configuration

You shouldn’t need to configure mqtt-kafka-bridge manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the mqttkafkabridge section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
DEBUG_ENABLE_FGTRACEEnables the use of the fgtrace library. Not reccomended for productionstringtrue, falsefalse
INSECURE_SKIP_VERIFYSkip TLS certificate verificationbooltrue, falsetrue
KAFKA_BASE_TOPICThe Kafka base topicstringAnyia
KAFKA_BOOTSTRAP_SERVERURL of the Kafka broker used, port is requiredstringAnyunited-manufacturing-hub-kafka:9092
KAFKA_LISTEN_TOPICKafka topic to subscribe to. Accept regex valuesstringAny^ia.+
KAFKA_SENDER_THREADSNumber of threads used to send messages to KafkaintAny1
LOGGING_LEVELDefines which logging level is used, mostly relevant for developersstringPRODUCTION, DEVELOPMENTPRODUCTION
MESSAGE_LRU_SIZESize of the LRU cache used to store messages. This is used to prevent duplicate messages from being sent to Kafka.intAny100000
MICROSERVICE_NAMEName of the microservice (used for tracing)stringAnyunited-manufacturing-hub-mqttkafkabridge
MQTT_BROKER_URLThe MQTT broker URLstringAnyunited-manufacturing-hub-mqtt:1883
MQTT_CERTIFICATE_NAMESet to NO_CERT to allow non-encrypted MQTT access, or to USE_TLS to use TLS encryptionstringUSE_TLS, NO_CERTUSE_TLS
MQTT_PASSWORDPassword for the MQTT brokerstringAnyINSECURE_INSECURE_INSECURE
MQTT_SENDER_THREADSNumber of threads used to send messages to MQTTintAny1
MQTT_TOPICMQTT topic to subscribe to. Accept regex valuesstringAnyia/#
POD_NAMEName of the pod. Used for tracingstringAnyunited-manufacturing-hub-mqttkafkabridge-Random-ID
RAW_MESSSAGE_LRU_SIZESize of the LRU cache used to store raw messages. This is used to prevent duplicate messages from being sent to Kafka.intAny100000
SERIAL_NUMBERSerial number of the cluster (used for tracing)stringAnydefault

12 - Node-RED

The technical documentation of the nodered microservice, which wires together hardware devices, APIs and online services.

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the Node-RED library.

How it works

Node-RED is a JavaScript-based tool that can be used to create flows that interact with the other microservices in the United Manufacturing Hub or external services.

See our guides for Node-RED to learn more about how to use it.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-nodered
  • Service:
    • External LoadBalancer: united-manufacturing-hub-nodered-service at port 1880
  • ConfigMap:
    • Configuration: united-manufacturing-hub-nodered-config
    • Flows: united-manufacturing-hub-nodered-flows
  • Secret: united-manufacturing-hub-nodered-secrets
  • PersistentVolumeClaim: united-manufacturing-hub-nodered-claim

Configuration

You can enable the nodered microservice and decide if you want to use the default flows in the _000_commonConfig.dataprocessing.nodered section of the Helm chart values.

All the other values are set by default and you can find them in the Danger Zone section of the Helm chart values.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
NODE_RED_ENABLE_SAFE_MODEEnable safe mode, useful in case of broken flowsbooleantrue, falsefalse
TZThe timezone used by Node-REDstringAnyBerlin/Europe

13 - Sensorconnect

The technical documentation of the sensorconnect microservice, which reads data from sensors and sends them to the MQTT or Kafka broker.

Sensorconnect automatically detects ifm gateways connected to the network and reads data from the connected IO-Link sensors.

How it works

Sensorconnect continuosly scans the given IP range for gateways, making it effectively a plug-and-play solution. Once a gateway is found, it automatically download the IODD files for the connected sensors and starts reading the data at the configured interval. Then it processes the data and sends it to the MQTT or Kafka broker, to be consumed by other microservices.

If you want to learn more about how to use sensors in your asstes, check out the retrofitting section of the UMH Learn website.

IODD files

The IODD files are used to describe the sensors connected to the gateway. They contain information about the data type, the unit of measurement, the minimum and maximum values, etc. The IODD files are downloaded automatically from IODDFinder once a sensor is found, and are stored in a Persistent Volume. If downloading from internet is not possible, for example in a closed network, you can download the IODD files manually and store them in the folder specified by the IODD_FILE_PATH environment variable.

If no IODD file is found for a sensor, the data will not be processed, but sent to the broker as-is.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-sensorconnect
  • Secret:
    • Kafka: united-manufacturing-hub-sensorconnect-kafka-secrets
    • MQTT: united-manufacturing-hub-sensorconnect-mqtt-secrets
  • PersistentVolumeClaim: united-manufacturing-hub-sensorconnect-claim

Configuration

You can configure the IP range to scan for gateways, and which message broker to use, by setting the values of the parameters in the _000_commonConfig.datasources.sensorconnect section of the Helm chart values file.

The default values of the other parameters are usually good for most use cases, but you can change them in the Danger Zone section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
ADDITIONAL_SLEEP_TIME_PER_ACTIVE_PORT_MSAdditional sleep time between pollings for each active portfloatAny0.0
ADDITIONAL_SLOWDOWN_MAPJSON map of values, allows to slow down and speed up the polling time of specific sensorsJSONSee below[]
DEBUG_ENABLE_FGTRACEEnables the use of the fgtrace library. Not reccomended for productionstringtrue, falsefalse
DEVICE_FINDER_TIMEOUT_SECHTTP timeout in seconds for finding new devicesintAny1
DEVICE_FINDER_TIME_SECTime interval in seconds for finding new devicesintAny20
IODD_FILE_PATHFilesystem path where to store IODD filesstringAny valid Unix path/ioddfiles
IP_RANGEThe IP range to scan for new sensorstringAny valid IP in CIDR notation192.168.10.1/24
KAFKA_BOOTSTRAP_SERVERURL of the Kafka broker. Port is requiredstringAnyunited-manufacturing-hub-kafka:9092
KAFKA_SSL_KEY_PASSWORDThe encrypted password of the SSL key. If empty, no password is usedstringAny""
KAFKA_USE_SSLSet to true to use SSL encryption for the connection to the Kafka brokerstringtrue, falsefalse
LOGGING_LEVELDefines which logging level is used, mostly relevant for developersstringPRODUCTION, DEVELOPMENTPRODUCTION
LOWER_POLLING_TIME_MSTime in milliseconds to define the lower bound of time between sensor pollingintAny20
MAX_SENSOR_ERROR_COUNTAmount of errors before a sensor is temporarily disabledintAny50
MICROSERVICE_NAMEName of the microservice (used for tracing)stringAnyunited-manufacturing-hub-sensorconnect
MQTT_BROKER_URLURL of the MQTT broker. Port is requiredstringAnyunited-manufacturing-hub-mqtt:1883
MQTT_CERTIFICATE_NAMESet to NO_CERT to allow non-encrypted MQTT access, or to USE_TLS to use TLS encryptionstringUSE_TLS, NO_CERTUSE_TLS
MQTT_PASSWORDPassword for the MQTT brokerstringAnyINSECURE_INSECURE_INSECURE
POD_NAMEName of the pod (used for tracing)stringAnyunited-manufacturing-hub-sensorconnect-0
POLLING_SPEED_STEP_DOWN_MSTime in milliseconds subtracted from the polling interval after a successful pollingintAny1
POLLING_SPEED_STEP_UP_MSTime in milliseconds added to the polling interval after a failed pollingintAny20
SENSOR_INITIAL_POLLING_TIME_MSAmount of time in milliseconds before starting to request sensor data. Must be higher than LOWER_POLLING_TIME_MSintAny100
SUB_TWENTY_MSSet to 1 to allow LOWER_POLLING_TIME_MS of under 20 ms. This is not recommended as it might lead to the gateway becoming unresponsive until a manual rebootint0, 10
TESTIf enabled, the microservice will use a test IODD file from the filesystem to use with a mocked sensor. Only useful for development.stringtrue, falsefalse
TRANSMITTERIDSerial number of the cluster (used for tracing)stringAnydefault
UPPER_POLLING_TIME_MSTime in milliseconds to define the upper bound of time between sensor pollingintAny1000
USE_KAFKAIf enabled, uses Kafka as a message brokerstringtrue, falsetrue
USE_MQTTIf enabled, uses MQTT as a message brokerstringtrue, falsefalse

Slowdown map

The ADDITIONAL_SLOWDOWN_MAP environment variable allows you to slow down and speed up the polling time of specific sensors. It is a JSON array of values, with the following structure:

[
  {
    "serialnumber": "000200610104",
    "slowdown_ms": -10
  },
  {
    "url": "http://192.168.0.13",
    "slowdown_ms": 20
  },
  {
    "productcode": "AL13500",
    "slowdown_ms": 20.01
  }
]