This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

United Manufacturing Hub

Learn how to contribute to the United Manufacturing Hub.

1: Setup Local Environment
2: Coding Conventions
3: Automation Tools
4: Release Process

1 - Setup Local Environment

This document describes how to set up your local environment for contributing to the United Manufacturing Hub.

The following instructions describe how to set up your local environment for contributing to the United Manufacturing Hub.

You can use any text editor or IDE. However, we recommend using JetBrains GoLand.

Requirements

The following tools are required to contribute to the United Manufacturing Hub. Use the links to install the correct version for your operating system. We recommend using a package manager where possible (for Windows, we recommend using Chocolatey).

Git
Go version 1.19 or later
Docker version 20.10 or later
kubectl version 1.26 or later
Helm version 3.11 or later
k3d version 5.0 or later
GNU C Compiler version 12 or later. The gcc binaries must be in your PATH environment variable, and the go variable CGO_ENABLED must be set to 1. You can check this by running go env CGO_ENABLED in your terminal.

Other tools that are not required, but are recommended:

GNU make to easily run scripts
GNU awk is used by some make scripts
Python 3 is used by some make scripts

Fork the documentation repository

If you are not a member of the United Manufacturing Hub organization, you will need to fork the repository to your own GitHub account. This is done by clicking the Fork button in the top-right corner of the united-manufacturing-hub/united-manufacturing-hub repository page.

Clone the repository

Clone the repository to your local machine:

git clone https://github.com/<user>/united-manufacturing-hub.git
# or: git clone [email protected]:<user>/united-manufacturing-hub.git

Where <user> is your GitHub username, or united-manufacturing-hub if you are a member of the United Manufacturing Hub organization.

If you are not a member of the United Manufacturing Hub organization, you will need to add the upstream repository as a remote:

git remote add origin https://github.com/united-manufacturing-hub/united-manufacturing-hub.git
# or: git remote add upstream [email protected]:united-manufacturing-hub/united-manufacturing-hub.git

# Never push to upstream master
git remote set-url --push origin no_push

Install dependencies

Download the go dependencies:

make go-deps

Build the container images

These are the make targets to manage containers:

# Build the container images
make docker-build

# Push the container images
make docker-push

# Build and push the container images
make docker

You can pass the following variables to change the behavior of the make targets:

CTR_REPO: The container repository to push the images to. Defaults to ghcr.io/united-manufacturing-hub.
CTR_TAG: The tag to use for the container images. Defaults to latest.
CTR_IMG: Space-separated list of container images. Defaults to all the images in the deployment directory.

Run a cluster locally

To run a local cluster, run:

# Create a cluster that runs the latest version of the United Manufacturing Hub
make cluster-install

# Create a cluster that runs the local version of the United Manufacturing Hub
make cluster-install CHART=./deployment/helm/united-manufacturing-hub

You can pass the following variables to change the behavior of the make targets:

CLUSTER_NAME: The name of the cluster. Defaults to umh.
CHART: The Helm chart to use. Defaults to united-manufacturing-hub/united-manufacturing-hub.
VERSION: The version of the Helm chart to use. Default is empty, which means the latest version.
VALUES_FILE: The Helm values file to use. Default is empty, which means the default values.

Test

To run the unit tests, run:

make go-test-unit

To run e2e tests, run:

make helm-test-upgrade

# To run the upgrade test with data
make helm-test-upgrade-with-data

Other useful commands

# Display the help for the Makefile
make help

# Pass the PRINT_HELP=y flag to make to print the help for each target
make cluster-install PRINT_HELP=y

What’s next

Read about our coding conventions.

2 - Coding Conventions

This document outlines a collection of guidelines, style suggestions, and tips for writing code in the different programming languages used throughout the Kubernetes project.

Code conventions

Bash
- https://google.github.io/styleguide/shell.xml
- Ensure that build, release, test, and cluster-management scripts run on macOS
Go
- Go Code Review Comments
- Effective Go
- Know and avoid Go landmines
- Comment your code.
  - Go’s commenting conventions
  - If reviewers ask questions about why the code is the way it is, that’s a sign that comments might be helpful.
- Command-line flags should use dashes, not underscores
- Naming
  - Please consider package name when selecting an interface name, and avoid redundancy. For example, storage.Interface is better than storage.StorageInterface.
  - Do not use uppercase characters, underscores, or dashes in package names.
  - Please consider parent directory name when choosing a package name. For example, pkg/controllers/autoscaler/foo.go should say package autoscaler not package autoscalercontroller.
    - Unless there’s a good reason, the package foo line should match the name of the directory in which the .go file exists.
    - Importers can use a different name if they need to disambiguate.
  - Locks should be called lock and should never be embedded (always lock sync.Mutex). When multiple locks are present, give each lock a distinct name following Go conventions: stateLock, mapLock etc.

Testing conventions

All new packages and most new significant functionality must come with unit tests.
Significant features should come with integration and/or end-to-end.
Do not expect an asynchronous thing to happen immediately—do not wait for one second and expect a pod to be running. Wait and retry instead.

Directory and file conventions

Avoid package sprawl. Find an appropriate subdirectory for new packages.
- Libraries with no appropriate home belong in new package subdirectories of pkg/util.
Avoid general utility packages. Packages called “util” are suspect. Instead, derive a name that describes your desired function. For example, the utility functions dealing with waiting for operations are in the wait package and include functionality like Poll. The full name is wait.Poll.
All filenames should be lowercase.
Go source files and directories use underscores, not dashes.
- Package directories should generally avoid using separators as much as possible. When package names are multiple words, they usually should be in nested subdirectories.
Document directories and filenames should use dashes rather than underscores.
Go code for normal third-party dependencies is managed using go modules.

3 - Automation Tools

This section contains the description of the automation tools used in the United Manufacturing Hub project.

Automation tools are an essential part of the United Manufacturing Hub project. They automate the building and testing of the project’s code, ensuring that it remains of high quality and stays reliable.

We rely on GitHub Actions for running the pipelines, which are defined in the .github/workflows directory of the project’s repository.

Here’s a brief overview of each workflow:

Build Docker Images

This pipeline builds and pushes all the Docker images for the project, tagging them using the branch name or the git tag. This way there is always a tagged version for the latest release of the UMH, as well as specific version for each branch to use for testing.

It runs on push events only when relevant files have been changed, such as the Dockerfiles or the source code.

GitGuardian Scan

This pipeline scans the code for security vulnerabilities, such as exposed secrets.

It runs on both push and pull request events.

Test Deployment

Small deployment test

(deactivated for now as they were flaky. will be replaced in the future with E2E tests)

This pipeline group verifies that the current changes can be successfully installed and that data flows correctly. There are two pipelines: a “tiny” version with the minimum amount of services needed to run the stack, and a “full” version with as many services as possible.

Each pipeline has two jobs. The first job installs the stacks with the current changes, and the second job tries to upgrade from the latest stable version to the current changes.

A test is run in each workflow to verify that simulated data flows through MQTT, NodeRed, Kafka, and TimescaleDB. In the full version, an additional test for sensorconnect is run, using a mocked sensor to verify the data flow.

It runs on pull request events when the Helm configuration or the source code changes.

Full E2E test

On every push to main and staging, an E2E test is executed. More information about this can be found on Github

4 - Release Process

This page describes how to release a new version of the United Manufacturing Hub.

Releases are coordinated by the United Manufacturing Hub team. All the features and bug fixes due for a release are tracked in the internal project board.

Once all the features and bug fixes for a release are ready and merged into the staging branch, the release process can start.

Companion

This section is for internal use at UMH.

Testing

If a new version of the Companion is ready to be released, it must be tested before it can be published. The testing process is done in the staging environment.

The developer can push to the staging branch all the changes that needs to be tested, including the new version definition in the Updater and in the version.json file. They can then use the make docker_tag GIT_TAG=<semver-tag-to-be-released> command from the Companion directory to build and push the image. After that, from the staging environment, they can trigger the update process.

This process will not make the changes available to the user, but keep in mind that the tagged version could still be accidentally used. Once the testing is done, all the changes are pushed to main and the new release is published, the image will be overwritten with the correct one.

Preparing the Documentation

Begin by drafting new documentation within the /docs/whatsnew directory of the United Manufacturing Hub documentation repository. Your draft should comprehensively include:

The UMH version rolled out with this release.
The new Companion version.
Versions of any installed plugins, such as Benthos-UMH.

Initiate your document with an executive summary that encapsulates updates and changes across all platforms, including UMH and Companion.

Version Update Procedure

Navigate to the ManagementConsole repository and contribute a new .go file within the /updater/cmd/upgrades path. This file’s name must adhere to the semantic versioning convention of the update (e.g., 0.0.5.go).

This file should:

Implement the Version interface defined in upgrade_interface.go.
Include PreMigration and PostMigration functions. These functions should return another function that, when executed, returns nil unless specific migration tasks are necessary. This nested function structure allows for conditional execution of migration steps, as demonstrated in the PostMigration example below:
```
func (v *v0x0x5) PostMigration() func(version *semver.Version, clientset kubernetes.Interface) error {
    return func(version *semver.Version, clientset kubernetes.Interface) error {
        zap.S().Infof("Post-Migration 0.0.5")
        return nil
    }
}
```

Define GetImageVersion to return the Docker tag associated with the new version. For 0.5.0 this would look like:

func (v *v0x0x5) GetImageVersion() *semver.Version {
    return semver.New(0, 0, 5, "", "")
}

Specify any Kubernetes controllers (e.g., Statefulsets, Deployments) needing restart post-update in the GetPodControllers function. Usually you just need to restart the companion itself, so you can use:

func (v *v0x0x5) GetPodControllers() []types.KubernetesController {
    return []types.KubernetesController{
        {
            Name: constants.StatefulsetName,
            Type: types.Statefulset,
        },
    }
}

Validate that all kubernetes objects referenced here, are designed to restart after terminating their Pod. This is especially important for Jobs.

Inside the versions.go, ensure to add your version inside the buildVersionLinkedList function.

func buildVersionLinkedList() error {
	var err error
	builderOnce.Do(func() {
		zap.S().Infof("Building version list")
		start := v0x0x1{}
		versionLinkedList = &start
		/*
		    Other previous versions
		 */
		
		// Our new version
		err = addVersion(&v0x0x5{})
		if err != nil {
			zap.S().Warnf("Failed to add 0.0.5 to version list: %s", err)
			return
		}
		zap.S().Infof("Build version list")
	})
	return err
}

Update the version.json in the frontend/static/version directory with the new image tag and incorporate the changelog derived from your initial documentation draft.

{
  "companion": {
    "versions": [
      {
        "semver": "0.0.1",
        "changelog": {
          "full": ["INTERNAL TESTING 0.0.1"],
          "short": "Bugfixes"
        },
        "requiresManualIntervention": false
      },
       
       // Other previous versions        

       // Our new version 
      {
        "semver": "0.0.5",
        "changelog": {
          "full": ["See 0.0.4"],
          "short": "This version is the same as 0.0.5 and is used for upgrade testing"
        },
        "requiresManualIntervention": false
      }
    ]
  }
}

Finalizing the Release

To finalize:

Submit a PR to the documentation repository to transition the release notes from draft to final.
Initiate a PR from the staging to the main branch within the ManagementConsole repository, ensuring to reference the documentation PR.
Confirm the success of all test suites.
Merge the code changes and formalize the release on GitHub, labeling it with the semantic version (e.g., 0.0.5, excluding any preceding v).
Merge the documentation PR to publicize the new version within the official documentation.

Checklist

Draft documentation in /docs/whatsnew with version details and summary.
Add new .go file for version update in /updater/cmd/upgrades.
Implement Version interface and necessary migration functions.
Update version.json with new image tag and changelog.
Submit PR to finalize documentation.
Create and merge PR in ManagementConsole repository, referencing documentation PR.
Validate tests and merge code changes.
Release new GitHub version without the v prefix.
Merge documentation PR to publish new version details.

Helm Chart

Prerelease

The prerelease process is used to test the release before it is published. If bugs are found during the prerelease, they can be fixed and the release process can be restarted. Once the prerelease is finished, the release can be published.

Create a prerelease branch from staging:

git checkout staging
git pull
git checkout -b <next-version>-prerelease1

Update the version and appVersion fields in the Chart.yaml file to the next version:

version: <next-version>-prerelease1
appVersion: <next-version>-prerelease1

Validate that all external docker images are correctly overwritten. This is especially important if an external chart is updated. The easiest way to do this is to run helm template and check the output.
Navigate to the deployment/helm-repo directory and run the following commands:
```
helm package ../united-manufacturing-hub
helm repo index --url https://staging.united-manufacturing-hub.pages.dev --merge index.yaml .
```
Pay attantion to use - instead of . as a separator in <next-version>.

Commit and push the changes:

git add .
git commit -m "build: <next-version>-prerelease1"
git push origin <next-version>-prerelease1

Merge prerelease branch into staging

Test

All the new releases must be thoroughly tested before they can be published. This includes specific tests for the new features and bug fixes, as well as general tests for the whole stack.

General tests include, but are not limited to:

Deploy the stack with flatcar
Upgrade the stack from the previous version
Deploy the stack on Karbon 300 and test with real sensors

If any bugs are found during the testing phase, they must be fixed and pushed to the prerelease branch. Multiple prerelease versions can be created if necessary.

Release

Once all the tests have passed, the release can be published. Merge the prerelease branch into staging and create a new release branch.

Create a release branch from staging:

git checkout main
git pull
git checkout -b <next-version>

Update the version and appVersion fields in the Chart.yaml file to the next version:
```
version: <next-version>
appVersion: <next-version>
```

Navigate to the deployment/helm-repo directory and run the following commands:

helm package ../united-manufacturing-hub
helm repo index --url https://repo.umh.app --merge index.yaml .

Commit and push the changes, tagging the release:

git add .
git commit -m "build: <next-version>"
git tag <next-version>
git push origin <next-version> --tags

Merge the release branch into staging
Merge staging into main and create a new release from the tag on GitHub.