Build a Reference Image with Yocto Project/OpenEmbedded
Introduction
This hands-on document describes how to build our Reference Images for Yocto Project from scratch using OpenEmbedded-Core (OE-Core). For a conceptual article about the relationship between Toradex' Embedded Linux offerings and the Yocto Project, you must read Yocto Project. For building Torizon OS, additional instructions are provided on Build Torizon OS With Yocto Project.
OpenEmbedded (openembedded.org) is a build framework that creates kernel images, root filesystem images and installable packages from source code. It is used by the Yocto Project to build Embedded Linux images. For the sake of this document OpenEmbedded and Yocto Project are used interchangeably unless otherwise noted.
OpenEmbedded uses meta-information (called recipes) for downloading/compiling/deploying of software packages on a x86/x86_64 Linux build host for a target device. This meta-information is structured into so-called layers. Layers are directory trees with recipes that provide similar functionality, e.g. meta-kde which provides recipes to build packages for KDE workspace and applications.
Versions and Source-Code
Layers from git.toradex.com are used together with a number of other layers. The collection of Toradex and third-party layers used on a specific BSP release are provided in a repo manifest on toradex-manifest, and you will learn more about it later in this article.
There might be several concurrent versions available at a given point in time, learn more about this on the Toradex Embedded Linux Support Strategy. Versions of the Reference Images for Yocto Project and their correspondent distribution, OpenEmbedded codename, and Yocto Project releases are provided in the following table:
| Toradex Version | Status | Distribution | OpenEmbedded/Yocto Project Codename | Yocto Project Release |
|---|---|---|---|---|
| 2.1 | Outdated | Ångström v2013.06 | dylan | 1.4 |
| 2.2 | Outdated | Ångström v2013.12 | dora | 1.5 |
| 2.3 | Outdated | Ångström v2014.06 | daisy | 1.6 |
| 2.4 | Outdated | Ångström v2014.12 | dizzy | 1.7 |
| 2.5 | Outdated | Ångström v2015.06 | fido | 1.8 |
| 2.6 / 2.6.1 | Outdated | Ångström v2015.12 | jethro | 2.0 |
| 2.7 | Outdated | Ångström v2016.12 | morty | 2.2 |
| 2.8 | LTS | Ångström v2017.12 | rocko | 2.4 |
| 3.0 | LTS | Poky based | thud | 2.6 |
| 4.0.0 (deferred*) | Deferred | Poky based | zeus | 3.0 |
| 5.7 | LTS | Poky based | dunfell | 3.1 LTS |
| 6.0 | Latest | Poky based | kirkstone | 4.0 LTS |
| 7.0 | Latest | Poky based | scarthgap | 5.0 LTS |
* 4.0.0 is deferred, learn more on https://www.toradex.com/blog/torizon-yocto-project-long-term-support-alignment
A release matrix that summarizes versions of Yocto/OpenEmbedded, the Linux kernel and U-Boot, and release dates for Toradex Embedded Linux releases is provided in a separate article:
Release Notes
Concise, high-level release notes are published on the News section of toradex.com:
Comprehensive, detailed release notes are maintained on the following pages on the developer website:
Building a Reference Image with Yocto Project
Prerequisites
A Yocto Project build requires intensive computing power and specific software packages.
Computer for the Yocto Project Build
A powerful host machine is highly recommended to build a Yocto Project image. If the host system is a multi-core machine, you can configure the Yocto Project build system to decrease the time needed to build images significantly. There should be a minimum of 60 GBytes of free disk space. For some images, a 32-bit host with 4 GBytes of memory is enough, but to build applications such as the WebKit web engine, a 64-bit machine with at least 8 GBytes of RAM is recommended.
Operating System and Build Dependencies
Yocto Project Release 5.0 is officialy supported by the Yocto Project. The following Linux distributions are supported and you must use one of them on your development PC to build our BSPs:
You must install the required Yocto build dependencies on your host PC:
Make sure to follow only the steps on the Yocto Mega Manual to install the build dependencies. Do not go any further into the environment setup.
Virtual Machines, Containers and Cloud Builds
Toradex does not go on about doing a Toradex Reference Image build on a VM, Container, or a public cloud provider as AWS or Azure. Nevertheless, it is possible. Therefore if you feel like doing it, you may consider studying your options. Find some generic remarks for a simplified overview:
- VM: a Yocto Project build is resource-intensive, it may not be a good idea to use a VM, because a VM may introduce a performance penalty that will make the build take longer.
- Container: it has a negligible performance penalty. Pay attention to using bind-mounts or volumes to keep persistent data; otherwise, you may lose time on every build. Containers theoretically allow you to do a Yocto Project build from a Windows PC.
- Toradex supports a container environment for building Torizon OS with the Yocto Project, read the article Build Torizon OS With Yocto for details. You may use it for building Reference Images but do it at your own risk since building Reference Images is not officially supported.
- Cloud: the cloud has powerful machines that make it possible to run a full Yocto Project build in less than half an hour if you disregard setup and pre-requisites installation time. Similar to containers, you should plan how to store the output of a build. Also, don't forget to evaluate the cost of this approach!
Repo and Git
Since OpenEmbedded requires several git repositories to build our images, starting with V2.1 images, we use a utility called 'repo'. The repo manifest manages the various git repositories and their branches. (more on repo: http://code.google.com/p/git-repo/)
Install the repo bootstrap binary:
$ mkdir ~/bin
$ export PATH=~/bin:$PATH
$ curl https://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo
Repo uses Git. Make sure you have it installed and your user name and e-mail address configured. Below is an example for Debian-based systems:
$ sudo apt install git
$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com
Installation
This section has two major sub-sections you have to choose:
- First-time Configuration: if you are doing a build for the first time.
- Update an Existing Configuration: if you have a build already set up and want to update it.
First-time Configuration
Create a directory for your OE-Core setup to live in and clone the meta-information:
Please be sure to have Python installed on your machine. For the command "repo init" to work, the executable "python" must be available, "python" can be a symbolic link to "python3". In addition, remember to have the PATH variable exported on the terminal you're currently using.
Note that support for the Verdin AM62 v1.1 was not added until version 6.4.0-devel-202309. Therefore, older tags or the default.xml configuration prior to the release of BSP 6.4.0 will not generate a usable image.
$ mkdir ${HOME}/oe-core
$ cd ${HOME}/oe-core
$ repo init -u git://git.toradex.com/toradex-manifest.git -b scarthgap-7.x.y -m tdxref/default.xml
$ repo sync
Source the file export to setup the environment. On the first invocation, this also copies a sample configuration to build/conf/*.conf.
$ . export
Note: Sourcing export configures the shell environment for the current shell session. You must enter this command whenever you open a new shell session for use with OpenEmbedded.
Adjust the settings in the local.conf file
Adapt build/conf/local.conf to your needs.
The MACHINE variable specifies the target device for the image. Set this variable to the module type you are planning to build for.
Our BSP layers provide the following machines:
| Machine Name | Corresponding Toradex Module |
|---|---|
apalis-imx6 | Apalis iMX6 |
apalis-imx8 | Apalis iMX8 |
colibri-imx6 | Colibri iMX6 |
colibri-imx6ull | Colibri iMX6ULL 512MB (equipped with raw NAND flash) |
colibri-imx6ull-emmc | Colibri iMX6ULL 1GB (equipped with eMMC flash) |
colibri-imx7 | Colibri iMX7S 256MB and iMX7D 512MB (equipped with raw NAND flash) |
colibri-imx7-emmc | Colibri iMX7D 1GB (equipped with eMMC flash) |
colibri-imx8x | Colibri iMX8X V1.0C or newer |
verdin-imx8mm | Verdin iMX8M Mini |
verdin-imx8mp | Verdin iMX8M Plus |
verdin-am62 | Verdin AM62 |
aquila-am69 | Aquila AM69 |
E.g. set in local.conf
MACHINE ?= "colibri-imx6"
Note: You can explicitly override the MACHINE setting on the command line. To do that, set the variable MACHINE when calling the bitbake command (e.g. MACHINE=apalis-imx6 bitbake...).
The DISTRO variable specifies the target distribution for the image. Set this variable to the distro you are planning to build. To learn about the available options, see Distro.
Make sure to set the correct distro for iMX6/7 modules, as in BSP 6, only upstream distros are supported.
If you have lots of disk space and want to browse the unpacked sources, consider commenting out the INHERIT += "rm_work" line.
If you already have a download directory with sources from a previous OE setup, consider moving that directory into oe-core/build/download or change local.conf to point to your standard download location. DL_DIR.
If you want to build for a machine based on an NXP based SoM, some downloads require you to read and accept the NXP®/Freescale EULA available in layers/meta-freescale/EULA.
You have to state your acceptance by adding the following line to your local.conf file:
ACCEPT_FSL_EULA = "1"
If this line is missing, the relevant packages will produce an ERROR similar to:
ERROR: To use 'imx-vpu' you need to accept the NXP®/Freescale EULA at 'layers/meta-freescale/EULA'. Please read it and in case you accept it, write: ACCEPT_FSL_EULA = "1" in your local.conf.
When the setup finishes, continue with the Building chapter.
Update an Existing Configuration
If you need to update the installation, you first update the repo manifest to the version you want to work with and then sync all layers. Check the known issues section below before starting the update.
If you did local changes to the individual layers, merge conflicts may occur, which you will have to resolve in the respective layer.
Update to the HEAD Revision
You cannot update from BSP 4 or older to 5 or newer, because a different Git repository is used!
A list of available branches is available in the repository: https://git.toradex.com/cgit/toradex-manifest.git/refs/heads
$ repo init -u git://git.toradex.com/toradex-manifest.git -b scarthgap-7.x.y -m tdxref/default.xml
$ repo sync
Update to a Specific Version by Using its Tag
If you need to update to a specific version, a list of available tags is available in the manifest repository: https://git.toradex.com/cgit/toradex-manifest.git/refs/tags
As an example:
$ repo init -u git://git.toradex.com/toradex-manifest.git -b refs/tags/7.0.0 -m tdxref/default.xml
$ repo sync
Building the BSP from our Nightly Platform
This is for development and testing only, to validate the latest changes/bugfixes from Toradex at BSP 6. Do not use it for production.
In some cases, you may want to experience our latest additions to the BSP, so to try a fix or even an improvement.
That can be done with a repository set to use the tdxref/integration.xml manifest:
$ repo init -u git://git.toradex.com/toradex-manifest.git -b scarthgap-7.x.y -m tdxref/integration.xml
You can then sync it, to get the resources:
$ repo sync
In addition, make sure that the following parameter is present in your conf/local.conf file:
MACHINEOVERRIDES =. "use-head-next:"
After that, you can build your image using bitbake as instructed in Build an Image below.
Known Issues During repo Setup
If your repo command succeeds, you can directly proceed with the instructions in the chapter Building below. See additional known issues notes at the end of this page.
Repo Sync Issues
Due to many company firewalls/proxies only permitting HTTP traffic, our default repo manifest is using. However, the OpenEmbedded or Linaro repositories might show inconsistencies if fetched over HTTP, or their web servers might be under such heavy load. If you get any of the following repo fetch/sync issues:
error: Unable to find d29dee17d9a95f2ec6b641c234870f3ccbe24102 under http://git.openembedded.org/meta-openembedded
Cannot obtain needed blob d29dee17d9a95f2ec6b641c234870f3ccbe24102
or:
error: Unable to find 318f971bb4515a6dcbbb0b87f85f2bc58b6e314f under http://git.linaro.org/git-ro/openembedded/meta-linaro.git
Cannot obtain needed commit 318f971bb4515a6dcbbb0b87f85f2bc58b6e314f
or:
fatal: loose object 8807fd141bb73d6b405eff223b3b105613bc809a (stored in /home/user/oe-core/.repo/projects/layers/meta-openembedded.git/objects/88/07fd141bb73d6b405eff223b3b105613bc809a) is corrupt
error: http://git.openembedded.org/meta-openembedded did not send all necessary objects
First, try repo syncing again. It might only have been a connection glitch.
Then, if the error(s) persist, try using git's garbage collection on the failing repository before attempting to sync again, e.g. if the error indicates an issue in the meta-openembedded repository:
$ cd .repo/projects/layers/meta-openembedded.git
$ git gc
If the error(s) persist try removing your local git/repo cache before attempting to sync again, e.g. if the error indicates an issue in the meta-openembedded repository:
$ rm -rf .repo/projects/layers/meta-openembedded.git
$ rm -rf .repo/project-objects/meta-openembedded.git
Additionally, you can limit the number of downloads of git repositories in parallel using the additional parameter -jX. E.g. to download one repository at a time:
$ repo sync -j1
If your Internet connection permits you might also try switching those failing repositories to using the git protocol instead by modifying your .repo/manifest.xml as follows:
<remote fetch="git://git.openembedded.org" name="oe"/>
<remote fetch="git://git.linaro.org/openembedded" name="linaro"/>
Bitbake Build Issues
There are no bitbake build issues reported for the latest BSPs. You can consult the other versions of this article with the version selector to see issues for older releases.
Building
This section goes on about build steps.
Setup the Environment
Every time you open a new terminal, go to the OpenEmbedded directory and setup the environment:
$ . export
Directory Structure
OE-Core installation, configuration, and build will setup the following directory structure:
oe-core/
+-- build
¦ +-- conf
¦ +-- downloads
¦ +-- tmp
¦ +-- sstate-cache
¦ +-- deploy
¦ +-- images
¦ ¦ +-- <machine 1...n>
¦ +-- ipk
¦ +-- licenses
¦ +-- sdk
+-- layers
+-- meta-browser
+-- meta-freescale
(... other layers)
+-- openembedded-core
- layers: stores metadata, as machine and distro information, but also the recipes for building all components of an embedded Linux image.
- build: keeps build configuration, downloaded source codes, intermediate build output, generated packages (IPK, DEB, RPM), SDKs.
- conf: this is the only subdirectory that you must preserve under build. All other subdirectories are regenerated when you start a new build if they don't exist yet.
- deploy: the final images, ready to install (flash) into a computer on module. The packages that compose an image, in our case packaged as IPK, since we use the OPKG package manager. Licenses for all the software installed. The SDK, if you build one.
Openembedded uses several places for configurations.
- General files from e.g. bitbake folder.
- Distro (from
<layer-directory>/conf/distro/<distro-name>.conf) - Machine (from
<layer-directory>/conf/machine/<machine-name>.conf) - Local Build (from
<build-directory>/conf/local.conf)
Distro
A Linux distribution (often abbreviated as distro) is a software collection based upon the Linux kernel. It comprises the Linux Kernel, tools, libraries and additional software.
Openembedded / Yocto gets the distribution policy from a configuration file which is set by the DISTRO variable in local.conf.
This policy for example defines the libc variant, the init system, the ARM vs. THUMB instruction set, or the package manager used. Also, it can set distro features which the individual recipes can then evaluate to configure their build, e.g. if polkit is used/available or what display server protocol is available.
For all iMX8 based machines we use the downstream kernel. The mainline kernel does not (yet) contain the features needed for production use. For the iMX8M based modules, there is experimental compatibility with upstream distros.
upstream: marks that the kernel built will be close to mainline and that the userspace uses mainlinish/opensource userspace drivers (Mostly graphical components, e.g. Vivante closed source vs. etnaviv). Note that each <machine>.conf decides what that mainline kernel will be.
rt: Toradex also provides real-time kernel flavours for use in an OpenEmbedded-Core (Yocto Project) build, and related distros that use it. See the Real-Time Linux article with Real-Time Linux article, Yocto Recipe for more information.
The default distro is tdx-xwayland. You can select other distro by modifying the DISTRO variable on the oe-core/buildconf/local.conf file.
See the configuration files of the distro on the layers/meta-toradex-distro/conf/distro/ directory.
Build an Image
A image recipe list all packages which have to be built and installed into the final root file system. The build system will take care that any known dependent package will also be installed.
Toradex provides two reference image variants built on top of the distribution variants. Below we list the image variants and their description and some remarkable features. If you want to know exactly all the features, follow the links as they point to the relevant Yocto recipes.
| Image | Description |
|---|---|
| Torizon OS Reference Minimal Image torizon-minimal (Torizon OS BSP) | Torizon OS reference minimal image without a container engine: - Network Manager: NetworkManager - Init system: systemd - OTA Framework: OSTree & Uptane - All other Torizon OS features Refer to Torizon OS Reference Minimal Image Technical Overview and Build Torizon OS from Source With Yocto Project/OpenEmbedded |
| Reference Minimal Image tdx-reference-minimal-image | Minimal image without graphical interface that just boots - Network manager: connman - Init system: systemd - Base command-line packages packagegroup-base-tdx-cli included in packagegroup-tdx-cli.bb |
| Reference Multimedia Image tdx-reference-multimedia-image | Image for BSP verification with Qt and multimedia features - All that is included in the Reference Minimal Image - Graphics stack: Weston / Wayland + XWayland - Graphical User Interface framework: Qt - Camera and Video framework: V4L2 and Gstreamer - All command-line packages included in packagegroup-tdx-cli.bb - All graphical packages included in packagegroup-tdx-graphical.bb - All Qt5 packages included in packagegroup-tdx-qt5.bb |
You can use them as a base for your project. Don't use them as-is because they are not ready to be deployed in production (not hardened, etc.). The available image recipes are located in layers/meta-toradex-demos/recipes-images/images/, also in our Git repository - just make sure to select the correct branch!
After you choose the image from the table above, you can build it:
$ bitbake <image>
Bitbake automatically logs console output to timestamped files in build/tmp/log/cooker/$MACHINE/.
With OE-Core you need to be in the directory 'build' when you execute bitbake.
This will at first build take hours, download lots of stuff and fill 'build/tmp', so have at least 60 GByte free.
Check available demo image recipes in layers/meta-toradex-demos/recipes-images/images/
If you are executing an unattended build, -k tells bitbake to keep going with independent packages even after an error occurred:
$ bitbake -k <image>
Building with reduced RAM usage
Your computer may run out of RAM while compiling some packages (such as Qt).
To reduce RAM usage, limit the number of threads used by Make by setting the environment variable PARALLEL_MAKE to the desired number of threads.
Example using 6 Make threads:
$ PARALLEL_MAKE="-j 6" bitbake <image>
Reducing the number of threads used by Make may not be enough, with BitBake running compilation tasks in parallel.
To reduce the number of tasks running simultaneously, set the environment variable BB_NUMBER_THREADS. Setting BB_NUMBER_THREADS also limits the number of download and configuration threads, increasing the image build time.
Example using 4 Make threads and 6 BitBake threads:
$ PARALLEL_MAKE="-j 4" BB_NUMBER_THREADS="6" bitbake <image>
If a build fails due to lack of RAM, some files could be corrupted. Trying to build again may not solve this issue.
This is a common cause when errors are encountered building unmodified Toradex reference images.
You may try to delete the corrupted package sstate, but its safer to manually clear the all temporary files before another build attempt.
Bitbake Commands
See in this section other useful bitbake commands.
Build a Single Package
Build a single package and all the things it depends on.
$ bitbake samba
Build an SDK
Building an SDK for your image, pass the parameter -c populate_sdk (See also Linux SDKs)
$ bitbake <image> -c populate_sdk
Build the Linux kernel
There is a virtual recipe for building the Linux kernel for your machine regardless of its version:
$ bitbake virtual/kernel
Additional Logs
the number of messages to stdout is huge!
If you need to dig into what recipes and settings are used for, the following command outputs a bunch of environment settings and info about what is going on, use the parameter -e:
$ bitbake -e virtual/kernel
$ bitbake -e virtual/kernel | grep ^WORKDIR
Available Tasks for a Recipe
If you want to see all tasks which exist for a given recipe, use the parameter -c listtasks:
$ bitbake -c listtasks virtual/kernel
Build Artifacts
The output artifacts can be found here:
- For images, u-boot, uImage, rootfs, deployable tarball:
build/deploy/images/${MACHINE}/ - For SDKs:
build/deploy/sdk/ - For ipk packages:
build/deploy/ipk/<package-arch>/*.ipk
Install the Image on the Board
Get the deployable tarball image from the directory pointed in the previous section.
The image tarball, (e.g. Verdin-iMX8MP_Reference-Minimal-Image-Tezi_6.0.0-devel-20221021202153+build.0.tar) has the same structure as the ones Toradex provides pre-built for download.
To update your computer on module with a newly built image, follow the procedure documented in the Toradex Easy Installer article.
First Steps Into Your Custom Build
The Yocto Project can be quite defying and hard to use. In the following documentation, we cover some basic functionality that is likely you will have to go through during your development:
It does not replace the official Yocto Project documentation, though. At the end of this page, we provide links to it, and you will most likely need to study it to some degree.
Production Programming
Since BSP 5.2, we have introduced an End User License Agreement (EULA) to our Reference Images. It prevents you to use the autoinstall feature of the Toradex Easy Installer, as suggested in the article Production Programming for Linux.
You must remove the license line from the image.json file included in the deployable tarball image resulting from the OpenEmbedded build to allow the autoinstall. It also means that you accept the EULA.
Failing Builds
An OpenEmbedded build sometimes fails. Below some recommended steps to follow when a task fails:
- Reading through the error messages.
- Restarting bitbake without deleting anything and see if it happens again.
- Clean the task which failed and then restart building the image, e.g. if python-native fails
$ bitbake -c clean python-native
$ bitbake <image>
- Clean the task which failed, including deleting cached output and then restart building the image
$ bitbake -c cleansstate python-native
$ bitbake <image>
- Clean the task which failed, including deleting cached output and the downloaded files and then restart building the image
$ bitbake -c cleanall python-native
$ bitbake <image>
Check the layer which provides the recipe if a newer commit exists which addresses the problem.
If there are many issues with fetching sources, you could first try to download all needed sources.
$ bitbake <image> --runall=fetch
Check the legacy section at the end of this page to find out about failing builds on older BSP versions.
Additional Layers
Recipes for even more software are available in additional layers. E.g. the Java layer provides recipes related to Java. Most layers are registered at openembedded.org. They provide a web interface to find layers or individual recipes.
Note that just because a layer or a recipe exists, that does not mean that these recipes will compile/execute. The layers/recipe might have a version conflict with what we use or might have never been used with or intended for the Arm architecture.
Refer to the following sections to see examples of how to add a new layer to our setup.
Adding the Qt Layer
Please refer to How to set up Qt Creator to cross-compile for embedded Linux.
Adding the Java Layer
Please refer to Java Virtual Machine.
Working with the Package Manager
Starting with BSP 3.0, we dropped the Ångström Distribution in favour of a Poky based distribution. For BSP 3.0 and newer based images, you cannot install packages from an online feed, only packages you build. However, if you use Torizon OS, you can use a container that makes it possible to use Debian feeds.
Read-Only Rootfs
Our base BSP layers and reference images do not support a read-only filesystem because there are features that won't work with a read-only system. In other words, the features of the read/write filesystem are used in several parts of the system, such as the case of the serial number in the hostname of the module. If the rootfs is read-only there is no way to write that modification at runtime.
For more information, refer to How to implement a Read-Only Rootfs (Linux).
Further Reading
The Yocto Project Documentation, particularly the Yocto Project Reference Manual, provides a lot of information. Two books we can recommend are: