Search by Tags

TorizonCore Builder Tool “build” command

 
Applicable for

Article updated at 11 May 2021
Subscribe for this article updates

Introduction

The TorizonCore Builder Tool provides a series of commands that allow the quick and easy customization of a Toradex Easy Installer image for deployment and provisioning. Most commands allow for the customization of very specific items, meaning that the full customization of an image would commonly require the execution of multiple such commands. This gives a great deal of flexibility to the user but it also makes the utilization of the tool more difficult. To simplify its usage, the tool provides the build command that allows the full customization of an image as a single step.

This article complies to the Typographic Conventions for the Toradex Documentation.

Prerequisites

This article assumes that the tool has been installed as described in Installing TorizonCore Builder.

For a quick hands-on introduction to the tool, go over the User Guide. For more details about the build command and on the contents of the configuration file that controls its behavior, see the Detailed manual. They are both accessible through the tabs below.


General Procedure

In order to leverage the build command you will basically need to:

  • Create a configuration file with the desired settings (the tool can even help you with that);
  • Make sure all local artifacts possibly referenced by the configuration file are available;
  • Run the command.

Creating a Configuration File

The easiest and thus recommended way for creating an initial configuration file is by running:

$ torizoncore-builder build --create-template
The previous command will create a file named tcbuild.yaml containing all possible configuration parameters accepted by the command.

The configuration file is in a standard YAML format which should be recognized by most modern programming editors. Any other name can be chosen for the file, but tcbuild.yaml is the name that the build command will expect by default when running in build mode. Most parameters in the generated file will be commented out which can be seen by the # character at the beginning of each line. Lines having the characters >> are just explaining the contents of the file and should be left as-is or removed if desired.

The contents of the file are organized in a tree structure where the leaves are configurable properties. In order to set a certain property you need to make sure all its parent properties are also uncommented (by removing the # character). For example, below we have a short stretch of the generated configuration file:

# >> The customization section defines the modifications to be applied to get
# >> the desired output image.
# customization:
  # >> Splash screen:
  # splash-screen: custom-splash-screen.png

If you wanted to set a splash screen, all you would need to do is to uncomment (and edit) the splash-screen property and its parent properties, in this case the one named customization, thus producing:

# >> The customization section defines the modifications to be applied to get
# >> the desired output image.
customization:
  # >> Splash screen:
  splash-screen: files/my-splash-screen.png
Also, the file files/my-splash-screen.png should be available before running the build command.

Note: All paths to files or directories in the configuration file are relative to the working directory of the tool, which is set when the torizoncore-builder alias is created (see the prerequisites section). Moreover, since TorizonCore Builder runs in a container, no paths above the working directory will be accessible by the tool (e.g. ../my-splash-screen.png would not work).

In the following sections we provide some examples that show how easy it is to create a custom image by employing the build command.

Example: Minimal Configuration File

A minimal configuration file has to specify at least one input and one output. Below we assume that the input is a previously downloaded Toradex Easy Installer image that is stored in a directory called images relative to the working directory (set by the alias of the tool).

# Sample configuration file:
input:
  easy-installer:
    local: images/torizon-core-docker-colibri-imx6-Tezi_5.3.0-devel-202105+build.12.tar

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM

Warning: As a general rule, the input image to be customized MUST NOT contain bundled container images – when choosing an image to download, e.g. from Toradex Easy Installer - OS and Demo Images be sure to download an image without evaluation containers.

The directory structure of such a "project" would look like this:

.
├── images
│   └── torizon-core-docker-colibri-imx6-Tezi_5.3.0-devel-202105+build.12.tar
└── tcbuild.yaml

To run the tool one would simply execute:

$ torizoncore-builder build
Once the build process finishes, the custom Easy Installer image would be present in a directory called torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM.

Example: Fetching Remote Images

The input to the tool can be specified in different ways which are described in the Detailed Manual and they are independent of the customization being performed.

Warning: Notice that not all images allow the customization of every possible property that the "build" command provides; e.g. older images may not allow that kernel modules to be built from source and deployed with the image.

In the sample configuration file below, we directly download an image from Toradex's artifacts server:

# Sample configuration file specifying a remote:
input:
  easy-installer:
    remote: "https://artifacts.toradex.com:443/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/1/colibri-imx6/torizon-upstream/torizon-core-docker/oedeploy/torizon-core-docker-colibri-imx6-Tezi_5.1.0%2Bbuild.1.tar;sha256sum=5d11dc0b6be688f6a7d159280e9bca15e26bc58732724e3e1dd66b0d0a6ada08"

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-Tezi_5.1.0+build.1.CUSTOM
    name: "My customized image"
Notice the presence of the sha256sum parameter which can be used to ensure that the downloaded file has a certain SHA-256 checksum – the tool will stop if the actual file checksum does not match that value (the Detailed Manual gives more information about how a remote location can be specified).

This "project" would simply contain the tcbuild.yaml file and by running:

$ torizoncore-builder build
You would get a custom Easy Installer image in the directory torizon-core-docker-colibri-imx6-Tezi_5.1.0+build.1.CUSTOM. The only item actually customized in that image is its name which is shown by the Toradex Easy Installer tool in its UI.

Example: Simple Customization

Here we have a configuration file where multiple items are being customized:

# Sample configuration file:
input:
  easy-installer:
    toradex-feed:
      version: "5.3.0"
      release: monthly
      machine: colibri-imx6
      distro: torizon-upstream
      variant: torizon-core-docker
      build-number: "12"
      build-date: "202105"

customization:
  splash-screen: custom/splash-screen.png
  filesystem:
     - changes1/
     - changes2/
  kernel:
    arguments:
      - key1=val1
      - key2=val2

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-custom
    name: "My customized image"
    description: "My customized image (description)"
    licence: custom/licence.html
    release-notes: custom/release-notes.html
The input here was defined via the toradex-feed property which allows you to set the desired attributes of the image to be downloaded. Notice also that we are applying two groups of changes to the filesystem which are listed under the customization/filesystem property. These changes could have been extracted from a live device, for example, by utilizing the torizoncore-builder isolate command. We are also setting custom kernel arguments and requesting the use of specific licence and release-notes files.

The directory structure of such a "project" could be like this:

.
├── changes1
│   └── usr
│       └── etc
│           ├── test1.txt
│           └── timezone
├── changes2
│   └── usr
│       └── etc
│           └── test2.txt
├── custom
│   ├── licence.html
│   ├── release-notes.html
│   └── splash-screen.png
└── tcbuild.yaml

Again, by simply running:

$ torizoncore-builder build
You would get your custom Easy Installer image in the directory torizon-core-docker-colibri-imx6-custom.

Example: Device-tree Customization

In this sample configuration file we customize the device-tree to be deployed with an image:

# Sample configuration:
input:
  easy-installer:
    local: images/torizon-core-docker-colibri-imx6-Tezi_5.3.0-devel-202105+build.12.tar

customization:
  device-tree:
    include-dirs:
      - device-trees/include/
    custom: device-trees/dts-arm32/imx6dl-colibri-eval-v3.dts
    overlays:
      add:
        - device-trees/overlays/colibri-imx6_hdmi_overlay.dts
        - device-trees/overlays/colibri-imx6_atmel-mxt-connector_overlay.dts

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-custom-dt
Usually one would create a device-tree (DT) source file and supply it to the tool via the customization/device-tree/custom property. Another, usually simpler, option is to leverage the device-tree source files provided by Toradex along with DT overlays that override the relevant parts of the tree – this was the approach taken on the example. To get the source code of the Toradex supplied device-tree files (including overlays), one could use the torizoncore-builder dt checkout command. A possibly better solution if you use Git for version control would be to manage these external device-trees through the Git submodule facility.

The directory structure for this "project" would look like this:

.
├── device-trees
│   ├── dts-arm32
│   │   ├── imx6dl-colibri-eval-v3.dts
│   ├── include
│   │   └── dt-bindings
│   │       ├── ...
│   ├── overlays
│   │   ├── ...
│   │   ├── colibri-imx6_atmel-mxt-adapter_overlay.dts
│   │   ├── colibri-imx6_atmel-mxt-connector_overlay.dts
│   │   ├── colibri-imx6_hdmi_overlay.dts
│   │   ├── colibri-imx6_parallel-rgb-lvds_overlay.dts
│   │   ├── colibri-imx6_parallel-rgb_overlay.dts
│   │   ├── ...
│   │   └── Makefile
│   └── README.md
├── images
│   └── torizon-core-docker-colibri-imx6-Tezi_5.3.0-devel-202105+build.12.tar
└── tcbuild.yaml

The device-trees directory subtree might have been obtained via the torizoncore-builder dt checkout command, as mentioned.

Again, through the execution of:

$ torizoncore-builder build
You would get a custom Easy Installer image in the directory torizon-core-docker-colibri-imx6-custom-dt.

Example: Building Kernel Modules

The build command allows you to build and install kernel modules into the deployed OS image. This is demonstrated in this configuration file:

# Sample configuration: build virtual touchscreen into the image
input:
  easy-installer:
    local: images/torizon-core-docker-colibri-imx6-Tezi_5.3.0-devel-202105+build.12.tar

customization:
  kernel:
    modules:
      - source-dir: virtual_touchscreen/
        autoload: no

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-vt
In this case, by running:
$ torizoncore-builder build
The tool would build the kernel module from the sources located in directory virtual_touchscreen and create a custom Easy Installer image in the directory torizon-core-docker-colibri-imx6-vt.

For more general information about the subject, please refer to Building External Kernel Modules With Torizon.

Example: Pre-provisioning Container Images

The build command also simplifies the process of bundling container images into an installer image. A sample configuration file leveraging that feature is this:

# Sample configuration file:
input:
  easy-installer:
    remote: "https://artifacts.toradex.com:443/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/1/colibri-imx6/torizon-upstream/torizon-core-docker/oedeploy/torizon-core-docker-colibri-imx6-Tezi_5.1.0%2Bbuild.1.tar;sha256sum=5d11dc0b6be688f6a7d159280e9bca15e26bc58732724e3e1dd66b0d0a6ada08"

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-with-containers
    name: "My customized image with container images"
    bundle:
      compose-file: custom/docker-compose.yml
With such an input, the tool will fetch the container images referenced by the specified docker-compose.yml file and combine them with the installer image. If you don't want the tool to download the images every time the build command is run, you can download them beforehand via the torizoncore-builder bundle command which will fetch the images and store them into a directory which can then be supplied to the tool via the output/easy-installer/bundle/dir property (in place of the output/easy-installer/bundle/compose-file as in the example).

As before, by running:

$ torizoncore-builder build
You will get a custom Easy Installer image in the directory torizon-core-docker-colibri-imx6-with-containers.

Warning: Due to the way TorizonCore Builder works, a directory called certs will be created in the working directory when build is run with the compose-file property. This is a known issue that will be solved in a future version of the tool.

Example: Variable Substitution

A feature that can be especially useful in CI scripts is called variable substitution which allows the replacement of property values (or parts of them) based on parameters passed on the command-line of the tool. The following configuration file employs that feature:

# Sample configuration file:
input:
  easy-installer:
    remote: "https://artifacts.toradex.com:443/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/1/colibri-imx6/torizon-upstream/torizon-core-docker/oedeploy/torizon-core-docker-colibri-imx6-Tezi_5.1.0%2Bbuild.1.tar;sha256sum=5d11dc0b6be688f6a7d159280e9bca15e26bc58732724e3e1dd66b0d0a6ada08"

output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-with-containers
    name: "${IMAGE_NAME-Default image name}"
    description: "${IMAGE_DESC?Please provide a description for this image}"
    bundle:
      compose-file: custom/docker-compose.yml
The value of property output/easy-installer/name in the example is set to the value of variable IMAGE_NAME or to "Default image name" when the variable is not set. The property output/easy-installer/description depends on IMAGE_DESC which is required and if not set will cause the tool to exit showing the specified message. Running the build command by itself would fail like this:
$ torizoncore-builder build
Building image as per configuration file 'tcbuild.yaml'...
 
** Exiting due to user-defined error: Please provide a description for this image

In order to successfully build an image with the previous configuration file (using the default value for IMAGE_NAME), one could run:

$ torizoncore-builder build --set IMAGE_DESC="This is the required description"

For more information on the notation for replacing variables in the configuration file, please refer to the Detailed Manual.

Next Steps

We recommend that you take a look at the Detailed Manual tab to get a better understanding of the ideas behind the build command.

Overview

To simplify the process of generating custom images for deployment and provisioning, the TorizonCore Builder tool provides the “build” command, whose inputs and outputs are depicted below:


  • Inputs and outputs of the torizoncore-builder build command

    Inputs and outputs of the torizoncore-builder build command

The build process is fully controlled by a configuration file (having a default name of tcbuild.yaml) which informs the tool what installer image to take as input, what modifications are to be applied (along with the associated data) and what output to produce. Usually the input would be a Toradex Easy Installer image, while the output would be a new, customized image of the same kind.

The configuration file and the artifacts referenced by it are common filesystem objects such as files or directories which can be version controlled allowing for reproducible builds. The artifacts used to customize the output include a splash-screen image, source files for device-tree and device-tree overlays, source code for kernel modules, etc.

Configuration File Contents

The configuration file follows the YAML format and syntax. The tool expects it to be made up of three sections, represented by the top-level properties (keys): input, customization and output as outlined below:

# Configuration file outline:
input:
  # Input section items (required).
customization:
  # Customization section items.
output:
  # Output section items (required).
On the next items we describe each one of these properties. The descriptions are organized in the same form as they are expected in the file, for example: to provide a local file as input to the tool, you would set the property input/easy-installer/local so you would have to set property local inside easy-installer which is in the input section.

The expected types of the elements in the file are indicated within parenthesis: objects are elements having child properties that in YAML parlance make up a mapping; string, boolean, number, integer types are leaf elements; properties of type array of a certain type represent sequences in the YAML naming. We recommend the usage of double quote " characters around string values (even though they are not required in YAML) just to avoid them to be interpreted as numbers in some cases.

Note: Use double quote characters around strings.

Many of the properties in the configuration file accept paths to files. Relative paths (those not starting with a / character) are actually relative to the working directory of torizoncore-builder (which is usually set when the alias to invoke the tool is set) and must be underneath it. Absolute paths (which start with a / character) are also possible but they must be specified from the perspective of the container where torizoncore-builder will run. Due to the complications this might cause, we discourage the use of absolute paths in the configuration file.

Warning: Do not use absolute paths in the configuration file.

Input Section

The input section defines the source artifact for customization. The definition can be provided in two different ways (mutually exclusive):

  • easy-installer (object): With this property, the source for customization is a Toradex Easy Installer image, which, in turn, can be specified in three different ways (also mutually exclusive):
    • local (string): Specifies a local directory or tarball containing the image.
    • remote (string): Specifies an HTTP or HTTPS URL used to download the image. For integrity checks, the expected SHA-256 checksum of the file can be appended to the path part of the URL using a semicolon as separator, e.g.: http://artifactory.toradex.com/.../tezi-image.tar;sha256sum=ad523…da02?key=val1&key2=val2. Notice that the sha256sum parameter comes before the query parameters (if any). The name of the downloaded file will be determined from the last part of the path (the so called slug), if present. The parameter filename can be employed to force a specific name for the file, e.g.: http://artifactory.toradex.com/.../tezi-image.tar;filename=teziimage_new.zip?key=val1&key2=val2.
    • toradex-feed (object): Defines individual image attributes that will be used by the tool to download a standard image from the official feed server:
      • version (string): Software version in the form major.minor.patch (e.g. “5.2.0”).
      • release (string: nigthly/monthly/quarterly) Release type (to choose the feed used).
      • machine (string): One of the supported module types (ex: colibri-imx7, apalis-im6, etc.).
      • distro (string: torizon/torizon-rt/torizon-upstream/torizon-upstream-rt): Distribution type (at the moment we support regular kernel or PREEMPT_RT).
      • variant (string: torizon-core-docker/torizon-core-podman, optional): Variant of the torizon image.
      • build-number (string): Build number as a string, e.g. “252“.
      • build-date (string): Build date as a string, e.g. "20210408" – this property is not required when using a quarterly release.
  • ostree (object): not available yet.

Customization Section

The customization section determines the modifications to be applied to the input to produce the desired output. It has the following structure:

  • splash-screen (string, optional): Local file path of a PNG file used to generate the splash screen image.
  • filesystem (array of strings, optional): List of directories to be merged with the input – the corresponding directory trees will be applied on top of the ones present in the input image, in the listed order. These directory trees can be created manually or as a result of customizing a live device and running the torizoncore-builder isolate command.
  • device-tree (object, optional): This entry allows the configuration of a base device tree (describing the whole hardware setup of the target device) and of additional overlays (describing specific peripherals or configurations).
    • include-dirs (array of strings, optional): List of directories where to look for include files while compiling device tree source files (used both for full device trees and for overlays).
    • custom (string, optional): Path of a device tree source (dts) file that will be built and integrated with the image, replacing existing ones. Beware that when a custom device tree is set, all overlays possibly present in the base image are cleared.
    • overlays (object, optional):
      • clear (boolean, optional): Boolean indicating whether or not to ignore all the binary overlays (dtbo files) that may already be present in the base image. This property is useful only if no custom device tree was set. Default: false.
      • remove (array of strings, optional): List of specific binary overlays (only the base names of the dtbo files) to be ignored or not taken from the base image. This property is useful only if no custom device tree was set.
      • add (array of strings, optional): List of paths for device tree source files that will be compiled and added to the image. Those overlays will provide specific changes to the base device tree and will be applied in the order listed in the configuration file. The tool will test the overlay against the current device tree (DT) only if a DT has been set (via the custom property or predefined in the base image).
  • kernel (object, optional): This property provides access to the facilities provided by the torizoncore-builder kernel command.
    • arguments (array of strings, optional): List of custom arguments to be passed to the kernel of the customized image.
    • modules (array of objects): List of kernel modules to be built from source and deployed with the image.
      • source-dir (string): Path to directory containing the kernel module source code. For further details, please refer to Building External Kernel Modules With Torizon.
      • autoload (boolean, optional): Boolean indicating whether the module should be automatically loaded upon system boot.

Output Section

The output section provides the details about the output artifact(s) to be produced by the torizoncore-builder build command. The section has this structure:

  • ostree (object, optional): The configuration items that are specific to OSTree are kept under this property; they directly relate to the torizoncore-builder union command.
    • branch (string, optional): Name of the OSTree repository branch that will contain the commit with the changes applied by the tool.
    • commit-subject (string, optional): OSTree commit subject. Default: "TorizonCore Builder <timestamp>"
    • commit-body (string, optional): OSTree commit body message.
  • easy-installer (object, optional): The configuration items specific to the Toradex Easy Installer image output are under this property.
    • local (string): Name of a local directory for the generated image.
    • name (string, optional): Name of the image as shown by the installer.
    • description (string, optional): Short description of the image (displayed in the image list screen of the installer).
    • licence (string, optional): Path of an HTML file with the image licence (displayed before installation when running interactively).
    • release-notes (string, optional): Path of an HTML file with the image release notes (accessible from the installer UI).
    • bundle (object, optional) – first form: Specifies a Docker compose file to be processed. The container images referenced by the compose file will be fetched during build time and a modified version of the compose file referencing the fetched image IDs will be stored in the output image.
      • compose-file (string): Path of a docker-compose file that will be included in the final image along with the required container images (i.e. with the images referenced by it). This will be considered only if the image variant is torizon-core-docker.
      • platform (string, optional): Platform for fetching multi-platform container images. If not specified, the tool will select between “linux/arm/v7” or “linux/arm64“ depending on the architecture of the input image.
      • username (string, optional): Docker login username to be used if accessing a private registry is required.
      • password (string, optional): Docker login password to be used if accessing a private registry is required.
      • registry (string, optional): Registry host name (e.g. hub.docker.com); this will be used only if username is also set.
    • bundle (object, optional) – second form:
      • dir (string): Path to directory containing the container images bundle to be combined with the installer image. Such bundle can be obtained by executing the torizoncore-builder bundle command beforehand.

Variable Substitution

In order to simplify the usage of the configuration file in cases where some values need to be set differently on each invocation of the build command, we provide a feature commonly offered by command-line shells called variable substitution. If such feature is enabled, all occurrences of $VAR in string values are replaced by the corresponding value of variable named VAR, which should be provided as part of the torizoncore-builder build command-line invocation; the notation $$ can be used for inserting a single literal $ sign within strings in the configuration file. Additionally, the following notations are also allowed:

  • ${VAR}: Same as $VAR; when the variable is not set in command-line, torizoncore-builder will issue a warning and assume the variable value is the empty string.
  • ${VAR:-default}: If VAR is not set or empty, use default as its value.
  • ${VAR-default}: If VAR is not set, use default as its value.
  • ${VAR:?error}: If VAR is not set or empty, issue the error message error and exit the tool.
  • ${VAR?error}: If VAR is not set, issue the error message error and exit the tool.

In a configuration file, the variable substitution notation would be used like this:

# ...
      username: "${USERNAME?Please provide a username}"
      password: "${PASSWORD?Please provide a password}"
# ...
      commit-subject: "${SUBJECT-default commit subject}"

Command-line Interface

The build command has two different modes of operation. The first one allows the creation of a template configuration file that can then be modified to suit one's needs. In this case, usage is:

torizoncore-builder build [--file NAME] --create-template
If NAME is not passed, the tool assumes the default file name of tcbuild.yaml. It is possible to check how a template configuration file will be created before creating it. To do this, one would provide the tool with a file name like - which will cause it to dump the template file contents to standard output, e.g.
$ torizoncore-builder build --file - --create-template
The second mode of operation, when the switch --create-template is not passed, corresponds to the normal operation mode of the tool where it actually processes the configuration file to perform the customization. Usage:
torizoncore-builder build [--file NAME] [--set VAR=VALUE] [--no-subst] [--force]
Again, if NAME is not passed the tool assumes the default file name of tcbuild.yaml (taken as input in this case). Variables used in the configuration file can be set via the --set parameter (e.g. --set USERNAME="user1"). The variable substitution feature which is enabled by default can be disabled completely by the --no-subst switch. The --force switch tells the tool to automatically remove the output if it already exists (by default, build fails in that case).

Version Controlling

In order to have reproducible builds, one should version control the inputs to the build command – in general, this entails keeping the configuration file and each of the files referenced by it under version control. Moreover, if the configuration file employs variable substitution, the same values should be passed as arguments to build to ensure the same results. The input image does not necessarily need to be version controlled by the user because Toradex already takes care of that – so if you reference a certain quarterly or monthly release, you know that the exact referenced version will be available to you over the Internet (at least for a period of time).

Device-tree (DT) and device-tree overlay (DTO) source files are a kind of input to the build command that may require especial treatment. If you create your own DT or DTO source files then you would keep them under version control as usual. However, if you want to use DT and/or DTO source files supplied by Toradex, then the recommended approach is to set up a Git submodule referencing our device-trees repo – this way, one could more easily keep in sync with upstream changes which is commonly desired.