Search by Tags

TorizonCore Builder Tool

 

Article updated at 17 Sep 2020
Compare with Revision

Subscribe for this article updates

Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.

Torizon 5.0.0

Introduction

After using Toradex Easy Installer to flash a TorizonCore image on modules, a developer will usually change the system's configuration, set up new peripheral devices by applying device tree overlay, or design new Docker containers for the end application solution.

Now, after customizing the system, how to replicate the same on other modules? TorizonCore Builder allows the developer to pick those modifications and create another image out of it. TorizonCore builder is containerized and run on the host system.

This article complies to the Typographic Conventions for Torizon Documentation

TorizonCore Builder and OSTree

TorizonCore Builder interacts with the OSTree library to capture/apply changes made on a filesystem.

OSTree is a library, which official name is libostree, which handles updates for the entire filesystem tree (the whole Linux root filesystem). An essential aspect of OSTree is that it uses a “git-like” model for committing and downloading bootable filesystem trees. The library is a key technology used by the Torizon Remote Update System.

Workflow and Use Cases

The following diagram shows the expected workflow when working with TorizonCore Builder:


  • TorizonCore Builder Workflow

    TorizonCore Builder Workflow

Developers will typically use TorizonCore Builder in the following use cases:

  • Setup new hardware - When setting up new hardware, for example, adding a Display to the board, or changing some SoC's pin functionality, sometimes a developer will have to make changes on the Linux Device Tree. A Linux Device Tree (DT) is a data structure and language for describing hardware. Device Tree Overlays provide a way to modify the overall device tree without re-compiling the complete device tree. TorizonCore Builder can build a full device tree or applying device tree overlays, if needed, using the dt subcommand. Check the specific article of this use case for more details.

  • Capture changes in the configuration of a board- If the user has made any configuration changes, e.g., the /etc directory, the isolate command will get all those changes (modifications, addition, and deletion). It requires an ssh connection for the communication between the host PC and the target board.

  • Change the system's splash screen - If the user needs to change the splash screen, the splash command allows to do it.

  • Create a branch for OSTree repo - If the user has used isolate, splash, and/or dt command, the union command will be used to create a branch out of it for the OSTree repo.

  • Deploy changes to Ostree repo or to a board - If the user has used the union command, the deploy command will deploy the OSTree branch either on the target machine connected via ssh OR onto the OSTree file system (from unpacked Toradex Easy Installer image of Torizon) and create a new Toradex Easy Installer image of Torizon.

  • Create a new custom image - Combine is used to create a Toradex Easy Installer image of Torizon by combining the output of the bundle command with a Toradex Easy Installer Image of Torizon. That Toradex Easy Installer Image of Torizon can be either an output image from the deploy command or an unpacked base image.

  • Pre-provisioning Docker Containers onto a TorizonCore image - TorizonCore Builder can download user docker containers and combine it with an off-the-shelf TorizonCore image to generate a new, monolithic, image. This process is especially useful when preparing an image to be flashed on several units in production. Check the specific article of this use case for more details of how to use the bundle + combine subcommands to perform this task.

Installing TorizonCore Builder

We provide TorizonCore Builder as a Docker image. To use TorizonCore Builder, you need to install it on your host machine by creating an alias.

Note: The alias uses the $(PWD) command, which does expand at “aliasing time”, that means the user cannot change the directory afterward.

TorizonCore Builder tool can work either by providing the working directory (capable of carrying Linux filesystem metadata) or Docker volumes for intermediate data storage. So all commands are explained with both approaches.

Warning: Move to the working directory before executing alias commands, so the required one gets bind-mounted when the alias executes

With Docker volumes:

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Here, two volumes are being created and mounted at /storage and /deploy of container for internal usage, and the host's current directory is bind mounted at /workdir within a container.

With user provided working directory (capable of handling Linux filesystem metadata):

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

The user also needs a base Toradex Easy Installer image of Torizon, used by the TorizonCore Builder.

Download the image specific for your TorizonCore version on the Toradex Easy Installer page. If you are not sure which TorizonCore version you are using, on your board's terminal, send the following command:

# cat /etc/*-release

After downloading the image to your host PC, uncompress the image.

$ tar xf teziimage.tar

Warning: Make sure to use an empty working directory and navigate to it. This directory will be bind-mounted to the container and will be used to store the final image, and may also use to store temporary files.

TorizonCore builder provides a command to unpack the Toradex Easy Installer image of Torizon inside Docker volume on the host machine.

$ torizoncore-builder unpack --image-directory <Path to TorizonCore Toradex Easy Installer source image>

For example:

$ tar xf teziimage.tar
$ torizoncore-builder unpack --image-directory torizon-core-docker-apalis-imx6-Tezi_5.0.0-devel-20200924030222+build.0/
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Running tar command: cat '/storage/tezi/torizon-core-docker-apalis-imx6.ota.tar.zst' | zstd -dc | tar --xattrs --xattrs-include='*' -xhf - -C /storage/sysroot
Importing OSTree revision 54fd6637c9b992e30b7d81e9adf27c4df0e9617c4d94feaa8fb330ae79d170ee from local repository...
Unpacked OSTree from Toradex Easy Installer image:
Commit checksum: 54fd6637c9b992e30b7d81e9adf27c4df0e9617c4d94feaa8fb330ae79d170ee
TorizonCore Version: 5.0.0-devel-20200924030222+build.0

TorizonCore Builder Commands

TorizonCore Builder has the following base commands.

`--verbose`: enable formatted logs.

`--log-level`: set log level from options `debug`, `info`, `warning`,` error` and `critical`
**default:** info

`--log-file`: write logs to the specified file instead of the console.

`--bundle-directory`: a path to the directory to store the output of the `bundle` command

**default:** creates and use directory with name `bundle` in the working directory

`--storage-directory`: a path to a directory to use as intermediate storage for sub-commands.
**default:** Docker volume mounted at`/storage` Note: Do not set it if using Docker volumes.

However, the most common use case is by using it with the commands:


# torizoncore-builder <command> <command options>

In this session, we list the TorizonCore Builder's commands.

The Isolate Command

isolate get configuration changes(modifications/addition/deletion) from the target board. It creates an ssh session with the target board to get changes and store them within Docker volume on the host.

isolate command gets the configuration (/etc) changes (modifications, additions, and deletions) from the user target module. It ignores the following files/directories for any change.

'gshadow', 'machine-id', 'group', 'shadow', 'systemd/system/sysinit.target.wants/run-postinsts.service',
'ostree/remotes.d/toradex-nightly.conf', 'docker/key.json', '.updated', '.pwd.lock', 'group-',
'gshadow-', 'hostname', 'ssh/ssh\_host\_rsa\_key', 'ssh/ssh\_host\_rsa\_key.pub', 'ssh/ssh\_host\_ecdsa\_key',
'ssh/ssh\_host\_ecdsa\_key.pub','ssh/ssh\_host\_ed25519\_key','ssh/ssh\_host\_ed25519\_key.pub',
'ipk-postinsts', 'fw\_env.conf'

Usage

$ torizoncore-builder isolate --remote-ip <IP of target board> --remote-username <username on target board> --remote-password <password of user on target board>

Arguments

--remote-IP: IP of the target board

--remote-username: username of the target board, to be used for creating ssh session.

--remote-password: user password of the target board, to be used for creating ssh session

--changes-directory (optional): If docker volume is not to be used, directory for storing configuration changes.

default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--mdns-source: Use the given IP address as mDNS source. This is useful when multiple interfaces are used, and mDNS multicast requests are sent out the wrong network interface.

Example

With docker volume
$torizoncore-builder isolate --remote-ip <target board IP> --remote-username <username of target board > --remote-password <user password on target board> --mdns-source <Use the given IP address as mDNS source>
without Docker volume

To use without Docker volume, the directory path is needed (--changes-directory) to store changes inside the user-provided working directory.

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> isolate --remote-ip <target board IP> --remote-username <username of target board> --remote-password <user password on target board > --changes-directory <directory to store changes> --mdns-source <Use the given IP address as mDNS source>

The Union Command

union makes an OSTree branch(containing the commit for changes) for all changes provided by the user to be made in OSTree rootfs of unpacked base Torizon image. It commits changes made by isolate, splash, and dt commands to the OSTree repo i.e., unpacked Toradex Easy Installer image of Torizon. If changes are in a Docker volume, it runs with Docker volume, and if changes are in the working directory, the user needs to provide the same directory to the command.

Note: If the user has used the union command, the deploy command is necessary to deploy the OSTree branch either on the target machine connected via ssh OR onto the OSTree file system (from unpacked Toradex Easy Installer image of Torizon) and to create a new Toradex Easy Installer image of Torizon.

Usage

$ torizoncore-builder union --union-branch <name to be given to the branch containing the changes> --extra-changes-directory <additional path with user changes to be committed> --extra-changes-directory <optionally can be provided multiple times to provide multiple directories>

Arguments

----union-branch: branch name containing the changes committed to OSTree repo of unpacked OSTree filesystem

--changes-directory(optional): path (within the working directory) to changes made by isolate, splash, and dt commands. Each change can be provided with a single option, so use it multiple times to provide multiple directories. default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--extra-changes-directory(optional): additional path with user changes to be committed. Each change can be provided with a single option, so use it multiple times to provide multiple directories. default: None

--subject: OSTree commit subject default: TorizonCore Builder [timestamp]

--body: OSTree commit body message

Example

with Docker volume
$ torizoncore-builder union --union-branch <name to be given to the branch> --extra-changes-directory <path to directory containing changes to be committed> --extra-changes-directory <optionally can be provided multiple times to provide multiple directories containing changes> --subject <TorizonCore Builder [timestamp]> --body <OSTree commit body message>
without Docker volume

If any changes are made by isolate, splash, and dt commands within the working directory, the same directories are needed to be provided with --changes-directory option. Each change can be provided with a single option, so use it multiple times to provide multiple directories. If the user has made any other changes, it can be provided with --extra-changes-directory option. --extra-changes-directory is also needed to be used multiple times to provide multiple directories.

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> union --union-branch <name to be given to the branch> --changes-directory <path to directory containing configuration changes> --changes-directory <optionally can be provided multiple times to provide multiple directories containing changes> --extra-changes-directory <additional directories containing changes> --extra-changes-directory <optionally can be provided multiple times to provide multiple directories containing changes> --subject <TorizonCore Builder [timestamp]> --body <OSTree commit body message>

The Deploy Command

deploy takes a branch(containing the commit for changes) from the union command, make it default deployment. Deployment can be either set on an unpacked Toradex Easy Installer image of Torizon or directly on the target board after creating an ssh session.

Suppose deployment is to be set on the unpacked Toradex Easy Installer image of Torizon. In that case, it makes a directory (inside the bind-mounted working directory) with a user-provided name and stores the Torizon image in it as output.

If pre-installed container(s) are needed, the combine command is necessary to create a new Toradex Easy Installer image of Torizon with a bundled container(s) from the bundle command.

Usage

Note: Output directory is not needed to be created.

$ torizoncore-builder deploy --output-directory <name of directory to store Torizon image> <name or checksum of branch from Union command>

If the deployment is to be set on the target board, the user needs to provide hostname or IP, username, and password.

$ torizoncore-builder deploy --remote-host <IP or hostname(.local) of target board> --remote-username <username of target board> --remote-password <password of user on target board> <name or checksum of branch from Union command>

Arguments

--output-directory: directory to store Toradex Easy Installer image of Torizon with deployment set to branch from union command. The output directory should not be created by the user. --remote-host: remote target machine IP or hostname (.local) --remote-username: username of the remote target machine --remote-password: user password of the remote target machine --reboot: reboot remote target machine after deploying default: do not reboot after deployment --deploy-sysroot-directory (optional): directory to store the intermittent deployment sysroot (used internally by the tool). NOTE: OSTree needs to be able to write extended attributes in this directory. [better to be used for without Docker volume option] default: docker volume mounted at /deploy

Positional Argument

branch name/checksum to be deployed from union command default: active deployment in OSTree repo

Example

with Docker volume

To create an output image

$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizon with deployment set to provided reference> <name or checksum of a branch within unpacked OSTree repo (output from union command or any other branch)>

To set branch for deployment on the remote target machine.

$ torizoncore-builder deploy <name or checksum of branch within unpacked OSTree repo (output from union command or any other branch)> --remote-host <IP or hostname(.local) or remote target machine> --remote-username <user name of remote target machine> --remote-password <user password of remote target machine> --reboot <indicates reboot machine after deploying>
without Docker volumes

To create an output image:

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> deploy <name or checksum of the branch within unpacked OSTree repo (output from union command or any other branch)> --output-directory <directory to store Toradex Easy Installer image of Torizon image with deployment set to provided reference> --deploy-sysroot-directory <directory to store the intermittent deployment sysroot>

To set branch for deployment on a remote target machine

$ torizoncore-builder deploy <name or checksum of branch within unpacked OSTree repo (output from union command or any other branch)> --remote-host <IP or hostname(.local) or remote target machine> --remote-username <user name of remote target machine> --remote-password <password of remote target machine> --reboot <indicates reboot machine after deploying>

The Bundle Command

It creates a tar archive of bundle container(s) from provided docker-compose file and stores it in either a "bundle" named directory inside working directory or with user-provided name and path using

--bundle-directory

option of torizonbuilder. Docker-compose file ( docker-compose.yml) and tar archive is later combined with Toradex Easy Installer image of Torizon using combine command to pre-install containers in Toradex Easy Installer image of Torizon. Toradex EasyInstaller tool copies containers data to appropriate places in rootfs while flashing the image on the Target board.

Usage

$ torizoncore-builder bundle --file <path to docker-compose.yml> --platform <target board like linux/arm/v7 or linux/arm64> --host-workdir $(pwd)

Arguments

--compose-file: the path to the docker-compose file default: docker-compose.yml in working directory

--platform: specify platform to make sure fetching the correct container image when multi-platform container images are specified (e.g. Linux/arm/v7 or Linux/arm64). default: linux/arm/v7

--host-workdir: directory to save some internal information for DIND(Docker in Docker) and output container tar archive.

Examples

$ torizoncore-builder --bundle-directory <path to store docker bundle tar archive> --compose-file <pat to docker compose file> --platform <platform of container> --host-workdir <path to directory to store continaer bundle>

The Combine Command

combine command is used to create Toradex Easy Installer image of Torizon by combining user-provided Toradex Easy Installer image of Torizon (either base Torizon Image OR output (output image) from deploy command) with bundled Docker container(s) from bundle command.

Usage

$ torizoncore-builder combine --image-directory <path to Torizon image> --output-directory <Torizon image with pre-provisioned Docker container(s)>

Arguments

--image-directory: the path to Toradex Easy Installer image of Torizon to be combined with docker bundle

--output-directory: the path to combined Toradex Easy Installer image of Torizon with bundled containers

--image-name (optional) : image name to be used in Easy Installer image JSON

--image-description (optional): image description to be used in Easy Installer image json

--image-licence (optional): the path to license file which will be shown on image installation

--image-release-notes (optional): the path to release notes file which will be shown on image installation

Example
$ torizoncore-builder --bundle-directory <path to docker bundle tar archive> --image-directory <path to Tezi image to be combined with docker bundle> --output-directory <directory to store combined Tezi image> --image-name <name of image to be set in image json file> --image-description <image desription to be set in image json file> --image-licence <path to license file to be shown at time of installation of image> --image-release-notes <path to release notes file which will be shown on image installation>

The Unpack Command

unpack gets the base Toradex Easy Installer image of Torizon and unpacks the rootfs (OSTree filesystem) to Docker volume

Usage

$ torizoncore-builder unpack --image-directory <base Toradex Easy Installer image of Torizon>

Arguments

--image-directory : the path to base Toradex Easy Installer image of Torizon in the working directory

Example

Note: If there is already an unpacked OSTree filesystem, the user is prompted to delete it because currently, a single OSTree filesystem can be handled.

With docker volume
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in the working directory>
Without Docker Volume

Additional argument --storage-directory specified with alias torizoncore-builder, sets the path(capable of handling Linux filesystem metadata) to be used to store intermediate data. It will be used to store unpacked OSTree rootfs as a result of this command.

$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in working directory>

The Dt (Device Tree) Command

dt has subcommands to perform device-tree-related operations, as shown below. Refer to Device Tree Overlays on Torizon for details on usage.

$ torizoncore-builder dt --help
usage: torizoncore-builder dt [-h] {overlay,custom,checkout,list-overlays, list-devicetrees} ...

optional arguments:
-h, --help show this help message and exit

Commands::
{overlay,custom,checkout,list-overlays}
overlay Apply an overlay
custom Compile device tree
checkout Checkout a git branch from remote repository
list-overlays List overlays
list-devicetrees List devicetrees

dt checkout

The dt checkout subcommand allows to checkout Toradex maintained overlays, base device trees (dts files), device tree include files (.dtsi files), and required header files (.h files). It clones either the user-provided remote repository or default Toradex provided remote repository device-trees in the absence of user input. It uses the current working directory as default for storing the cloned remote repository.

Usage
$ torizoncore-builder dt checkout
Arguments

--repository (optional): URL of remote repository or path to a local repository default: https://github.com/toradex/device-trees

--branch (optional)`: branch to checkout default: gets the kernel version of the deployed image from unpacked Toradex Easy Installer image of Torizon and checkout the corresponding branch in default repository only.

Example
With docker volume
$ torizoncore-builder dt checkout
Without docker volume
$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> dt checkout --repository <repository URL> --branch <branch to checkout>

dt custom

dt custom subcommand compile the device tree source file. It uses device-trees/include directory to search for .dtsi or .h file(s). However, users can optionally provide --include-directory multiple times to provide multiple directories to search for .dtsi or .h file(s).

Usage
$ torizoncore-builder dt custom --devicetree-source <path to device tree source file> --include-dir <optionally provide path to directory containing dtsi and/or .h files> --include-dir <optionally can be used multiple times to provided multiple directories>
Arguments

--devicetree-source: Provides the device tree source file to be compiled.

--devicetree-out (optional): If it is specified, device tree binary is stored inside user-specified directory(should be empty) like <user dir>/usr/lib/modules/<kver>/devicetree. Output binary is always named devicetree. default: device tree binary is stored(replacing any old) in Docker volume and is also named devicetree

--include-dir (optional): It should point to directory containing .dtsi and .h files. It allows multiple declarations to provide multiple directories. default: It searches in the device-trees/includes directory of checked out repository.

Example
With docker volume
$ torizoncore-builder dt custom --devicetree-source <Path to device tree source file> --devicetree-out <Path to save output device tree binary named 'devicetree'> --include-dir <Path to directory containing dtsi and/or .h file(s)>
without docker volume
$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> dt custom --devicetree-source <path to device tree source file> --devicetree-out <Path to save output device tree binary named 'devicetree'> --include-dir <Path to directory containing dtsi and/or .h file(s)> --devicetree-out <Path to store device tree binary in `/usr/lib/modules/` with name 'devicetree' inside provided directory`>

dt list-overlays

dt list-overlays lists all compatible device tree overlay file(s) in a user-provided directory for user-provided device tree source files or device tree binary files. It searches for compatible overlays in the device-trees/overlays directory as default behavior, but the user can provide a directory to search for overlays using --overlays-dir argument.

Usage

Using a device tree source file

$ torizoncore-builder dt list-overlays --devicetree-source <path to device tree source file> --overlays-dir <optionally provide a path to a directory containing overlay files to be filtered out based on compatibility>

Using a device tree binary file

$ torizoncore-builder dt list-overlays --devicetree <path to device tree binary file> --overlays-dir <optionally provide a path to a directory containing overlay files to be filtered out based on compatibility>
Arguments

--devicetree-source: Path to device tree source file

--devicetree: Path to device tree binary file

--overlays-dir (optional): Path to the directory containing overlays to be filtered out default: It searches in device-trees/overlays directory of checked out repository.

Example
$ torizoncore-builder dt list-overlays --devicetree-source <Path to device tree source file> --overlays-dir <Path to source file(s) of device tree overlay(s)>

Or

$ torizoncore-builder dt list-overlays --devicetree <Path to device tree binary file> --overlays-dir <Path to source file(s) of device tree overlay(s)>

dt list-devicetrees

dt list-devicetrees lists all available device tree binaries inside unpacked Toradex Easy Installer image of Torizon.

Usage
$ torizoncore-builder dt list-devicetrees
Arguments

None

Example
With Docker Volume
$ torizoncore-builder dt list-devicetrees
without Docker Volume
$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> dt list-devicetrees

dt overlay

dt overlay compiles and apply device tree overlay(s) to the device tree binary. It searches for included .dtsi and .h files in device-trees/includes directory, but optionally --include-dir can be used multiple times to provide multiple directories to search for .dtsi and .h files. Overlay(s) is provided as a positional argument, i.e., provide them without specifying any argument.

If the user does not provide the --devicetree option, devicetree named device tree binary is searched in unpacked Toradex Easy Installer image of Torizon. Then, if available, overlay(s) is applied to it.

Suppose the user provides a device tree binary without specifying any path. In that case, it searches in the unpacked Toradex Easy Installer image of Torizon, and then, if available, overlay(s) are applied to it.

If the user provides a device tree binary with a path, it searches in that path, and the software will apply the overlays to it. If it is available in the user-provided path, the software searches inside unpacked Toradex Easy Installer image of Torizon. Then, if available, the software applies the overlay(s) to it.

Usage
$ torizoncore-builder dt overlay --include-dir <optionally provide path to directory containing dtsi and/or .h files> --include-dir <optionally can be used multiple times to provided multiple directories> <overlay file(s)>

Optionally, the user can provide a device tree binary that will receive the overlay.

$ torizoncore-builder dt overlay --devicetree <devcie tree name or path to it> --include-dir <optionally provide path to directory containing dtsi and/or .h files> --include-dir <optionally can be used multiple times to provided multiple directories> <overlay file(s)>
Arguments

--devicetree (optional): Name of device tree binary inside OSTree(unpacked Torizon rootfs) or a path to device tree binary in the working directory. If the device tree binary is not found in the working directory, it searches inside OSTree as default. default: devicetree binary file from OSTree.

--devicetree-out (optional): Path to overlaid device tree binary output named devicetree default: If the user does not specify it, Docker volume saves the output.

--include-dir (optional): Path to directory containing .dtsi and .h files. It allows multiple declarations to provide multiple directories/files. default: It searches in the device-trees/includes directory of checked out repository.

Positional Argument

Overlay(s) to be applied to the device tree binary.

Examples
With docker volume

Without a tree binary:

$ torizoncore-builder dt overlay --include-dir <Path to directory containing .dtsi and/or .h file(s)> --<provide overlay(s)>

With device tree binary:

$ torizoncore-builder dt overlay --device-tree <Name of device tree binary inside OSTree(unpacked Torizon rootfs) or Path to device tree binary> --include-dir <Path to directory containing .dtsi and/or .h file(s)> <provide overlay(s)>
without docker volume

Without the device tree binary:

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> dt overlay --include-dir <Path to directory containing .dtsi and/or .h file(s)> <provide overlay(s)> --devicetree-out <Path to store overlaid device tree in `/usr/lib/modules/` with name 'devicetree'`inside provided directory>

With the device tree binary:


$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> dt overlay --device-tree <Name of device tree binary inside OSTree(unpacked Torizon rootfs) or Path to device tree binary> --include-dir <Path to directory containing .dtsi and/or .h file(s)> <provide overlay(s)> --devicetree-out <Path to store overlaid devicetree in `/usr/lib/modules/` with name 'devicetree'`>

Example

Note: This section does not show how to use the dt (Device Tree) command. For dt command usage, refer to Device Tree Overlays on Torizon

Following is the usage of the tool with Docker volume

Creating alias

Note: alias uses $(PWD) which does expand at “aliasing time”, that means the user cannot change the directory afterwards.

$ alias torizoncore-builder='docker run -it --rm -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Unpack base Toradex Easy Installer image of Torizon

$ torizoncore-builder unpack --image-directory colibri-imx7
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Unpacked OSTree from oradex Easy Installer image:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117

Get configuration changes from the target board

$ torizoncore-builder isolate --remote-ip 192.168.1.146 --remote-username torizon --remote-password 1234
isolation command completed

Merge changes to base Toradex Easy Installer image of Torizon

$ torizoncore-builder union --union-branch=changes_committed
Commit d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 has been generated for changes and ready to be deployed.

Set branch to deploy at boot. The output image can be deployed on the target board.

$ torizoncore-builder deploy --output-directory deployed_image d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Running tar command: tar --xattrs --xattrs-include='*' -cf /workdir/deployed_image/torizon-core-docker-colibri-imx7.ota.tar.xz --xz -S -C /deploy -p .
Using unpacked Toradex Easy Installer image as the base:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117
Kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash ostree=/ostree/boot.1/torizon/d103ac800d2c9a67cce7ec68ad1accb836d766ff959895820deea195f16eaeed/0

Deploying commit ref: d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Pulling OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 from local repository...
Pulling done.
Deploying OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Deploying done.
Copy rootdirs such as /home from original deployment.
Packing rootfs...
Packing rootfs done.

If the branch is to be deployed on the target board

$ torizoncore-builder deploy d36d7f8b393e9226af780d16c0813eaa087fd35ff1251c86b5dc0d61a9ca29b3 --remote-host 192.168.1.111 --remote-username torizon --remote-password 1234
Pulling OSTree with ref d36d7f8b393e9226af780d16c0813eaa087fd35ff1251c86b5dc0d61a9ca29b3 (checksum d36d7f8b393e9226af780d16c0813eaa087fd35ff1251c86b5dc0d61a9ca29b3) from local archive repository...
Starting http server to serve OSTree.
Starting OSTree pull on the device...
Deploying new OSTree on the device...
Deploying successfully finished.
Please reboot the device to boot into the new deployment.

If required, create a tar archive of Docker container(s)

$ torizoncore-builder --bundle-directory bundled_containers bundle -f docker-compose.yml --platform linux/arm/v7 --host-workdir $(pwd)

Create an image with pre-provisioned docker containers. --image-directory can be either base image or output from deploy command.

$ mkdir preprov_cont_image
$ torizoncore-builder --bundle-directory bundled_containers combine --image-directory deployed_image --output-directory preprov_cont_image
Creating copy of TorizonCore source image.
Combining TorizonCore image with Docker Container bundle.
Successfully created a TorizonCore image with Docker Containers pre-provisioned in preprov_cont_image

Torizon 4.0.0

Introduction

Toradex provides Easy Installer to flash images on modules. Developer may make changes in configuration (/etc) of the system and/or work with containers for the end user solution. Now how to replicate the same on other modules? Toradex has developed a new tool, TorizonCore Builder, to let the developer pick those modifications/containers and create another image out of it, keeping the original Toradex Easy Installer image of Torizon as base. TorizonCore Builder is containerized and run on the host system.

This article complies to the Typographic Conventions for Torizon Documentation

Workflow


  • TorizonCore Builder Workflow

    TorizonCore Builder Workflow

Commands

TorizonCore Builder is provided in the form of Docker image and has following commands to create Toradex Easy Installer image of Torizon with required changes

  • Unpack

  • Bundle

  • Isolate

  • Union

  • Deploy

  • Combine

Operation

Firstly, user needs a base Toradex Easy Installer image of Torizon which will be used by TorizonCore Builder container to add user changes in it. User has two options, either to perform all tasks in a “working directory”, which must be a file system capable of carrying Linux file system metadata (such as Unix style file system permissions and extended attributes), by setting appropriate arguments OR let the commands use Docker volumes for intermediate storage to perform the operations on image.

  • The very first command to use is unpack, which unpacks the OSTree file system from user provided Toradex Easy Installer image of Torizon

  • If pre-installation of Docker container(s) onto a Torizon image with Docker engine is required, bundle command is to be used by providing a Docker compose file stating all image(s).

  • If user has made any changes in configuration (/etc), isolate command is to be used to get all those changes (modifications, addition and deletion). It requires ssh connection to the target board.

  • If user has used isolate command, union command is to be used to capture all changes in the form of a commit.

  • If user has used union command, deploy command is to be used to deploy the OSTree commit onto the OSTree file system and create a new Toradex Easy Installer image of Torizon.

  • Combine is used to create a Toradex Easy Installer image of Torizon by combining outputs from deploy command and bundle command i.e. combine user configuration changes and pre-provisioned Docker container(s).

Usage

Warning: Make sure to use an empty working directory and navigate to it. This will be bind mounted to the container this tool is running in and will be used to store the final image and may also use to store temporary files.

Note: Toradex Easy Installer image of Torizon can be downloaded from Toradex Easy Installer

Extract the downloaded image tar file

$ mkdir <directory to extract image into>
$ tar -xf <downloaded image.tar> -C <directory to extract image into>

Basic

For convenience, create alias first

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

TorizonCore Builder tool uses Docker volumes as persistent storage for intermediate outputs of sub-commands.

TorizonCore Builder tool provides different logging levels, which can be set by using --log-level option. There are five logging levels i.e. debug, info, warning, error and critical and default is set to info. Formatted logging can be enabled by using --verbose option.

Sub-commands of TorizonCore Builder tool can be used as follows

unpack

unpack gets the base Toradex Easy Installer image of Torizon and unpacks the rootfs (OSTree filesystem) to Docker volume

$ torizoncore-builder unpack --image-directory <base Toradex Easy Installer image of Torizon>

Isolate

isolate is used to get configuration changes(modifications/addition/deletion) from the Target board. It creates ssh session with Target board to get changes and store them within Docker volume on host.

$ torizoncore-builder isolate --remote-ip <ip of target board> --remote-username <username on target board> --remote-password <password of user on target board> 

Union

union is used to make an OStree commit of all changes in OSTree repo of unpacked base Torizon image.

$ torizoncore-builder union --union-branch <name to be given to the branch containing the changes commit>

Deploy

deploy is used to take branch from union command, make it default deployment, create the new Torizon image, make directory (inside bind mounted working directory ) with user provided name and store Torizon Image in it.

Note: Output directory is not needed to be created.

$ torizoncore-builder deploy --output-directory <name of directory to store Torizon image>

Bundle

bundle is used to bundle the Docker container(s), which is used by combine command to pre-provision container(s) bundle with Torizon Image.

User provides the docker-compose.yml, platform of target board (linux/arm/v7 or linux/arm64) and working directory of host to store data used for DIND (docker in docker) and bundle command stores the Docker container(s) tar archive in directory named “bundle” inside the working directory.

$ torizoncore-builder bundle --file <path to docker-compose.yml> --platform <target board like linux/arm/v7 or linux/arm64> --host-workdir $(pwd)

Combine

combine command is used to create Torizon image by combining user provided Torizon image (either base Torizon Image OR output of deploy;depending on what is needed) with bundled Docker container(s) from bundle command.

$ torizoncore-builder combine --image-directory <Path to Torizon image> --output-directory <Torizon image with pre-provisioned Docker container(s)>

Example

Creating alias

$ alias torizoncore-builder='docker run -it --rm -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Unpack base Toradex Easy Installer image of Torizon

$ torizoncore-builder unpack --image-directory colibri-imx7
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Unpacked OSTree from oradex Easy Installer image:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117

Get configuration changes from target board

$ torizoncore-builder isolate --remote-ip 192.168.1.146 --remote-username torizon --remote-password 1234 
isolation command completed

Merge changes to base Toradex Easy Installer image of Torizon

$ torizoncore-builder union --union-branch=changes_committed
Commit d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 has been generated for changes and ready to be deployed. 

Set branch to deploy at boot. Output image can be deployed on target board. Note: deployed_image is not needed to be created

$ torizoncore-builder deploy --output-directory deployed_image --ref=d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Running tar command: tar --xattrs --xattrs-include='*' -cf /workdir/deployed_image/torizon-core-docker-colibri-imx7.ota.tar.xz --xz -S -C /deploy -p .
Using unpacked Toradex Easy Installer image as base:
  Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
  TorizonCore Version: 4.0.0-devel-20200607+build.117
  Kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash ostree=/ostree/boot.1/torizon/d103ac800d2c9a67cce7ec68ad1accb836d766ff959895820deea195f16eaeed/0

Deploying commit ref: d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Pulling OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 from local repository...
Pulling done.
Deploying OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Deploying done.
Copy rootdirs such as /home from original deployment.
Packing rootfs...
Packing rootfs done.

If required, create tar archive of Docker container(s)

$ torizoncore-builder --bundle-directory bundled_containers  bundle -f docker-compose.yml --platform linux/arm/v7 --host-workdir $(pwd)

Create image with pre-provisioned docker containers. --image-directory can be either base image or output from deploy command.

$ mkdir preprov_cont_image
$ torizoncore-builder --bundle-directory bundled_containers combine --image-directory deployed_image --output-directory preprov_cont_image
Creating copy of TorizonCore source image.
Combining TorizonCore image with Docker Container bundle.
Successfully created a TorizonCore image with Docker Containers preprovisioned in preprov_cont_image

Advanced

User can use TorizonCore Builder tool by either providing the working directory (capable of carrying Linux filesystem metadata) or Docker volumes for intermediate data storage. So all commands are explained with both approaches. Commands can be used with mix of Docker volumes and working directory but that will complicate the usage. So it is recommended to use either working directory or Docker volumes for all the commands.

As it is provided in the form of Docker image, any of the following commands (based on working directory or Docker volumes) can be executed to create an alias to be easy to use.

Usage of all commands explained later are based on the following aliases.

With Docker volumes:

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Here, two volumes are being created and mounted at /storage and /deploy of container for internal usage and host current directory is bind mounted at /workdir within container.

With user provided working directory (capable of handling Linux filesystem metadata)

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

move to the working directory so the required one gets bind mounted when alias is executed

Base Command

Base command is now alias torizoncore-builder and has below optional options

--verbose: enable formatted logs

--log-level: set log level from options debug, info, warning, error and critical
default: info

--log-file: write logs to specified file instead of console

--bundle-directory: path to directory to store output of bundle command
default: creates and use directory with name bundle in working directory

--storage-directory: path to directory to use as intermediate storage for sub-commands.
default: Docker volume mounted at/storage
Note: do not set it if Docker volumes are to be used.

Unpack

Unpack command is used to unpack the ostree filesystem from user provided Toradex Easy Installer image of Torizon base image.

Arguments

--image-directory : path to base Toradex Easy Installer image of Torizon in the working directory

--sysroot-directory (optional): path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
With docker volume
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in the working directory>  
Without Docker Volume

Additional argument --sysroot-directory, specifying path to store unpacked ostree filesystem, is needed in case of user provided working directory (capable of handling Linux filesystem metadata) is to be used for storage of intermediate data.

$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in working directory> --sysroot-directory <path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked>

If there is already an unpacked ostree filesystem, user is prompted to delete it because currently a single ostree filesystem can be handled.

Isolate

Isolate command is used to get the configuration (/etc) changes (modifications, additions and deletions) from user target module. It ignores following files/directories for any change

'gshadow', 'machine-id', 'group', 'shadow', 'systemd/system/sysinit.target.wants/run-postinsts.service', 'ostree/remotes.d/toradex-nightly.conf', 'docker/key.json', '.updated', '.pwd.lock', 'group-', 'gshadow-', 'hostname', 'ssh/ssh_host_rsa_key', 'ssh/ssh_host_rsa_key.pub', 'ssh/ssh_host_ecdsa_key', 'ssh/ssh_host_ecdsa_key.pub','ssh/ssh_host_ed25519_key','ssh/ssh_host_ed25519_key.pub', 'ipk-postinsts', 'fw_env.conf'

Isolate command establishes a ssh session with the target board, gets all the changes and places them in either user provided directory or within docker volumes.

Arguments

--remote-ip: IP of target board

--remote-username: Username of target board to be used for creating ssh session

--remote-password: Password of target board to be used for creating ssh session

--diff-dir (optional): If docker volume is not to be used, directory for storing configuration changes
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
with docker volume
$torizoncore-builder  isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password>
without docker volume

Directory path is needed (--diff-dir) to store changes within user provided working directory

$ torizoncore-builder  isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password> --diff-dir <directory to store changes>

Union

It is to be run after isolate command to commit changes to the ostree filesystem. If changes are stored in docker volume, it is to be run with docker volume and if changes are stored in working directory, same directory is needed to be provided.

Arguments

----union-branch: branch name containing the changes committed to ostree repo of unpacked ostree filesystem

--diff-directory(optional): path (within working directory) to configuration changes picked by isolate command.
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--sysroot-directory(optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon (same as used in unpack command)
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
with docker volume
$ torizoncore-builder union --union-branch <name to be given to the branch>
without docker volume

isolate command is used to store changes in a directory within working directory. Same directory is needed here with --diff-directory option.

$ torizoncore-builder union --union-branch <name to be given to the branch> --diff-directory <path to directory containing configuration changes> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon>  

Deploy

It takes a branch from union command, sets it up for the boot, packs the rootfs and creates the Toradex Easy Installer image of Torizon ready to be deployed (if no pre-install container(s) are needed).

If pre-installed container(s) are needed, “combine command“ is used to create a new Toradex Easy Installer image of Torizon with bundled container(s) from bundle command.

Arguments

--output-directory: directory to store Toradex Easy Installer image of Torizon with deployment set to branch from union command. It is not needed to be created by the user.

--diff-ref: branch name/checksum to be deployed from union command
default: active deployment in ostree repo

--sysroot-directory (optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--deploy-sysroot-directory (optional): directory to store the intermittent deployment sysroot (used internally by tool).
default: docker volume mounted at /deploy

Example
with docker volume
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizon with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> 
without docker volumes
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizonm image with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon> --deploy-sysroot-directory <directory to store the intermittent deployment sysroot>

Bundle

It creates a tar archive of bundle container(s) from provided docker-compose file and stores it in ether "bundle" named directory inside working directory or with user provided name and path using

--bundle-directory

option of

torizonbuilder

. Docker-compose file and tar archive is later combined with Toradex Easy Installer image of Torizon using combine command to pre-install containers in Toradex Easy Installer image of Torizon. Toradex EasyInstaller tool copies containers data to appropriate places in rootfs during flashing the image on Target board.

Arguments

--compose-file: path to docker-compose file
default: docker-compose.yml in working directory

--platform: specify platform to make sure fetching the correct container image when multi-platform container images are specified (e.g. linux/arm/v7 or linux/arm64).
default: linux/arm/v7

--host-workdir: directory to save some internal information for DIND(Docker in Docker) and output container tar archive.

Examples
$ torizoncore-builder --bundle-directory <path to store docker bundle tar archive> --compose-file <pat to docker compose file> --platform <platform of container> --host-workdir <path to directory to store continaer bundle>

Combine

It is used to create a Toradex Easy Installer image of Torizon by combining tar archive with user provided Toradex Easy Installer image of Torizon. User provided Toradex Easy Installer image of Torizon can be either image created by deploy command or base Toradex Easy Installer image of Torizon as per need.

Arguments

--image-directory: path to Toradex Easy Installer image of Torizon to be combined with docker bundle

--output-directory: path to combined Toradex Easy Installer image of Torizon with bundled containers

--image-name (optional) : image name to be used in Easy Installer image json

--image-description (optional): image description to be used in Easy Installer image json

--image-licence (optional): path to license file which will be shown on image installation

--image-release-notes (optional): path to release notes file which will be shown on image installation

Example
$ torizoncore-builder --bundle-directory <path to docker bundle tar archive> --image-directory <path to Tezi image to be combined with docker bundle> --output-directory <directory to store combined Tezi image> --image-name <name of image to be set in image json file> --image-description <image desription to be set in image json file> --image-licence <path to license file to be shown at time of installation of image> --image-release-notes <path to release notes file which will be shown on image installation>