Build a Reference Image with Yocto Project/OpenEmbedded
Introduction
This hands-on article describes how to build our Reference Images for Yocto Project from scratch using OpenEmbedded-Core (OE-Core). For a conceptual article about the relationship between Toradex' Embedded Linux offerings and the Yocto Project, you must read Yocto Project. For building Torizon OS, additional instructions are provided on Build Torizon OS With Yocto Project.
OpenEmbedded (openembedded.org) is a build framework that creates kernel images, root filesystem images, and installable packages from source code. It is used by the Yocto Project to build Embedded Linux images. In this article, OpenEmbedded and Yocto Project are used interchangeably unless otherwise noted.
OpenEmbedded uses meta-information (called recipes) for downloading/compiling/deploying of software packages on a x86/x86_64 Linux build host for a target device. This meta‑information is structured into layers. Layers are directory trees with recipes that provide similar functionality, e.g. meta-kde which provides recipes to build packages for KDE workspace and applications.
For further information about the Yocto project and use-cases beyond the scope of our articles, check out the Yocto Project Documentation and the Yocto Project Reference Manual.
Prerequisites
- Following the article: Host Machine Setup for Building with Yocto
A Yocto Project build requires intensive computing power and specific software packages, which are specified below.
Computer for the Yocto Project Build
A powerful host machine is highly recommended to build a Yocto Project image. If the host system is a multi-core machine, you can configure the Yocto Project build system to decrease the time needed to build images significantly. We recommend around 120 GB of free disk space. For some images, a 32-bit host with 4 GBytes of memory is enough, but to build other images such as the tdx-reference-multimedia-image, a 64-bit machine with at least 16 GBytes of RAM is recommended.
Operating System and Build Dependencies
Yocto Project Release 5.0 is officially supported by the Yocto Project. The following Linux distributions are supported and you must use one of them on your development PC to build our BSPs:
You must install the required Yocto build dependencies on your host PC:
Follow only the steps in the Yocto Mega Manual to install the build dependencies. Do not go any further into the environment setup.
Virtual Machines, Containers and Cloud Builds
It is possible to build a Toradex Reference Image build on a Virtual Machine (VM), Container, or a public cloud provider as AWS or Azure. We provide here, without detailed instructions, a general comparison between each of these alternatives:
- VM: A VM may introduce a high performance penalty, and a Yocto Project build is resource-intensive, thus we do not advise to use a VM for building with Yocto.
- Container: Containers have a negligible performance penalty. Use bind-mounts or volumes to keep persistent data and reduce build time. Containers theoretically allow you to do a Yocto Project build from a Windows PC.
- Toradex supports a container environment for building Torizon OS with the Yocto Project, read the article Build Torizon OS With Yocto for details. You may use it to build Reference Images, but note that this is not officially supported.
- Cloud: the cloud has powerful machines that significantly reduce the build time for a Yocto Project. Similar to containers, you should plan how to store the output of a build. You must also evaluate the pricing of your cloud service provider.
Build a Reference Image
This section provides instructions on building a Toradex image with Yocto. The build process consists of the following steps:
- Setting up the shell environment you will use to build the image
- Editing your build configuration files
- Build the image with the
bitbaketool
The detailed instructions for following these steps are provided in the sections bellow.
Set Up the Environment Shell
Every time you open a new terminal, go to the OpenEmbedded directory and setup the environment:
$ . export
Edit the Build Configuration Files
The main configuration file a generic Yocto build is conf/local.conf. There are two main variables you need to modify for a basic build:
MACHINE: specifies the target device for the image. Set this variable to the module type you are planning to build for.DISTRO: specifies the target distribution for the image. Set this variable to the distro you are planning to build.
More information on the MACHINE and DISTRO variables are provided below:
The MACHINE Yocto variable
Our BSP layers provide the following machines:
| Machine Name | Corresponding Toradex Module |
|---|---|
apalis-imx6 | Apalis iMX6 |
apalis-imx8 | Apalis iMX8 |
colibri-imx6 | Colibri iMX6 |
colibri-imx6ull | Colibri iMX6ULL 512MB (equipped with raw NAND flash) |
colibri-imx6ull-emmc | Colibri iMX6ULL 1GB (equipped with eMMC flash) |
colibri-imx7 | Colibri iMX7S 256MB and iMX7D 512MB (equipped with raw NAND flash) |
colibri-imx7-emmc | Colibri iMX7D 1GB (equipped with eMMC flash) |
colibri-imx8x | Colibri iMX8X V1.0C or newer |
verdin-imx8mm | Verdin iMX8M Mini |
verdin-imx8mp | Verdin iMX8M Plus |
verdin-imx95 | Verdin iMX95 |
verdin-am62 | Verdin AM62 |
verdin-am62p | Verdin AM62P |
aquila-am69 | Aquila AM69 |
toradex-smarc-imx8mp | SMARC iMX8M Plus |
toradex-smarc-imx95 | SMARC iMX95 |
The DISTRO Yocto variable
A Linux distribution (often abbreviated as distro) is a software collection based upon the Linux kernel. It comprises the Linux Kernel, tools, libraries and additional software.
Openembedded / Yocto gets the distribution policy from a configuration file which is set by the DISTRO variable in local.conf.
This policy for example defines the libc variant, the init system, the ARM vs. THUMB instruction set, or the package manager used. Also, it can set distro features which the individual recipes can then evaluate to configure their build, e.g. if polkit is used/available or what display server protocol is available.
Toradex provides several distribution policy files based on the Poky one. We provide distro variants for different kernel configurations, summarized in the table below. Even though all distributions set the policy for the graphics stack, for example Weston/Wayland + XWayland, not all images include the graphics libraries: for instance, the Reference Minimal Image is a console-only image.
If you want to know exactly what is defined for each distro, follow the links as they point to the relevant configuration files.
| Distro | kernel base | kernel config |
|---|---|---|
| tdx-xwayland | Downstream kernel from SoC vendor (NXP) | Default (without the PREEMPT_RT patch) |
| tdx-xwayland-rt | Downstream kernel from SoC vendor (NXP) | Fully preemptive (real-time Linux PREEMPT_RT patch) |
| tdx-xwayland-upstream | Mainline kernel | Default (without the PREEMPT_RT patch) |
| tdx-xwayland-upstream-rt | Mainline kernel | Fully preemptive (real-time Linux PREEMPT_RT patch) |
For all iMX8 based machines we use the downstream kernel. The mainline kernel does not (yet) contain the features needed for production use. For the iMX8M based modules, there is experimental compatibility with upstream distros.
upstream: marks that the kernel built will be close to mainline and that the userspace uses mainlinish/opensource userspace drivers (Mostly graphical components, e.g. Vivante closed source vs. etnaviv). Note that each <machine>.conf decides what that mainline kernel will be.
rt: Toradex also provides real-time kernel flavours for use in an OpenEmbedded-Core (Yocto Project) build, and related distros that use it. See the Real-Time Linux article with Real-Time Linux article, Yocto Recipe for more information.
See the configuration files of the distro on the layers/meta-toradex-distro/conf/distro/ directory.
The default distro is tdx-xwayland, and it is already configured in the conf/local.conf file. You can set it to your distro of choice.
As an example, we will set the MACHINE to colibri-imx6. You should set it to your corresponding SoM.
MACHINE ?= "colibri-imx6"
If you want to build for a machine based on an NXP based SoM, some downloads require you to read and accept the NXP®/Freescale EULA available in layers/meta-freescale/EULA.
You have to state your acceptance by adding the following line to your local.conf file:
ACCEPT_FSL_EULA = "1"
If this line is missing, the relevant packages will produce an ERROR similar to:
ERROR: To use 'imx-vpu' you need to accept the NXP®/Freescale EULA at 'layers/meta-freescale/EULA'. Please read it and in case you accept it, write: ACCEPT_FSL_EULA = "1" in your local.conf.
You can explicitly override the MACHINE and DISTRO settings on the command line. To do that, set the variable MACHINE when calling the bitbake command (e.g. MACHINE=apalis-imx6 DISTRO=tdx-wayland-rt bitbake ...).
Make sure to set the correct distro for iMX6/7 modules, as in BSP 7, only upstream distros are supported.
Build the Image
The final step to build a reference image is to select which image you want to build. The image is the recipe which lists all packages which have to be built and installed into the final root file system. The build system will take care that any known dependent package will also be installed.
Toradex provides two reference image variants built on top of the distribution variants. Below, we list the image variants, their descriptions, and some notable features. To learn about all the features, follow the links, which point to the relevant Yocto recipes. These images are primarily for BSP evaluation, and we do not recommend using them as-is.
Additionally, we provide a Torizon OS Yocto Reference Minimal Image for OS evaluation. For information on building Torizon images with Yocto, refer to Torizon OS Yocto Reference Minimal Image Technical Overview and Build Torizon OS from Source With Yocto Project/OpenEmbedded.
| Image | Description |
|---|---|
| Reference Minimal Image tdx-reference-minimal-image | Minimal image without graphical interface that just boots - Network manager: connman - Init system: systemd - Base command-line packages packagegroup-base-tdx-cli included in packagegroup-tdx-cli.bb |
| Reference Multimedia Image tdx-reference-multimedia-image | Image for BSP verification with Qt and multimedia features - All that is included in the Reference Minimal Image - Graphics stack: Weston / Wayland + XWayland - Graphical User Interface framework: Qt - Camera and Video framework: V4L2 and Gstreamer - All command-line packages included in packagegroup-tdx-cli.bb - All graphical packages included in packagegroup-tdx-graphical.bb - All Qt5 packages included in packagegroup-tdx-qt5.bb |
You can use them as a base for your project, but keep in mind that these images are not production-ready and should not be used as-is in production (not hardened, etc.). The available image recipes are located in layers/meta-toradex-demos/recipes-images/images/. You can also find the image recipes in our Git repository, but make sure to download it from the same branch as your Yocto build repository.
After you choose the image from the table above, you can build it with the bitbake tool. Keep in mind that bitbake will not be available in your shell until you source the export file, as instructed in Set Up the Environment Shell.
- You need to be in the directory 'build' when you execute bitbake.
- The first build is expected to take a large amount of time (possibly several hours) and download several files. We recommend having around 120 GB of free disk space.
$ bitbake <image>
Bitbake automatically logs console output to timestamped files in build/tmp/log/cooker/$MACHINE/.
If you are executing an unattended build, -k tells bitbake to keep going with independent packages even after an error occurred:
$ bitbake -k <image>
If your build finishes without errors, you can deploy the resulting image to your board
Building with reduced RAM usage
Your computer may run out of RAM while compiling some packages (such as Qt).
To reduce RAM usage, set the environment variable PARALLEL_MAKE to the desired number of threads. This will limit the number of threads used by make.
$ PARALLEL_MAKE="-j 6" bitbake <image>
Bitbake also runs compilation tasks in parallel. To reduce the number of tasks running simultaneously, set the environment variable BB_NUMBER_THREADS. Setting BB_NUMBER_THREADS also limits the number of download and configuration threads, increasing the image build time.
$ PARALLEL_MAKE="-j 4" BB_NUMBER_THREADS="6" bitbake <image>
If a build fails due to lack of RAM, some files could be corrupted. Trying to build again may not solve this issue.
This is a common cause when errors are encountered building unmodified Toradex reference images.
You may try to delete the corrupted package state, but it’s safer to manually clear all temporary files before another build attempt.
Bitbake Commands
Below are some useful common bitbake commands and their descriptions.
| Image | Additional parameters | Description |
|---|---|---|
bitbake recipe | -- | Build a recipe and all its dependencies. A recipe can be an image to upload to your board, the linux kernel, u-boot, a software package, an SDK, etc |
bitbake image -c populate_sdk | -c populate_sdk | Build an SDK for your image (See also Linux SDKs). |
bitbake virtual/kernel | -- | Build only the Linux kernel for your machine |
bitbake -e recipe | -e | Output the environment variables and settings for your build. You can see all the settings with less or other text viewers, or use grep to see the value of a specific variable |
bitbake -c listtasks recipe | -c listtasks | List all tasks for a given recipe |
Bitbake Build Issues
There are no bitbake build issues reported for the latest BSPs. You can consult the other versions of this article with the version selector to see issues for older releases.
Deploy the Reference Image to Your Board
After the bitbake tasks have finished, the output image will be located at build/deploy/images/${MACHINE}/.
We recommend deploying the Tezi image generated in the build process (e.g. Verdin-iMX8MP_Reference-Minimal-Image-Tezi_6.0.0-devel-20221021202153+build.0.tar). You can deploy the image to your board using Toradex Easy Installer (Tezi).
Customize a Yocto Build
Directory Structure
OE-Core installation, configuration, and build will setup the following directory structure:
oe-core/
+-- build
¦ +-- conf
¦ +-- downloads
¦ +-- tmp
¦ +-- sstate-cache
¦ +-- deploy
¦ +-- images
¦ ¦ +-- <machine 1...n>
¦ +-- ipk
¦ +-- licenses
¦ +-- sdk
+-- layers
+-- meta-browser
+-- meta-freescale
(... other layers)
+-- openembedded-core
- layers: stores metadata, such as machine and distro information, but also the recipes for building all components of an embedded Linux image.
- build: keeps build configuration, downloaded source codes, intermediate build output, generated packages (IPK, DEB, RPM), SDKs.
- conf: this is the only subdirectory that you must preserve under build. All other subdirectories are regenerated when you start a new build if they don't exist yet.
- deploy: the final images, ready to install (flash) into a computer on module. The packages that compose an image, in our case packaged as IPK, since we use the OPKG package manager. Licenses for all the software installed. The SDK, if you build one.
Openembedded uses several places for configurations.
- General files from e.g. bitbake folder.
- Distro (from
<layer-directory>/conf/distro/<distro-name>.conf) - Machine (from
<layer-directory>/conf/machine/<machine-name>.conf) - Local Build (from
<build-directory>/conf/local.conf)
Build Artifacts
The output artifacts can be found here:
- For images, u-boot, uImage, rootfs, deployable tarball:
build/deploy/images/${MACHINE}/ - For SDKs:
build/deploy/sdk/ - For ipk packages:
build/deploy/ipk/<package-arch>/*.ipk
First Steps Into Your Custom Build
The Yocto Project can be quite challenging and hard to use. In the following documentation, we cover some basic functionality that is likely you will have to go through during your development:
It does not replace the official Yocto Project documentation, though. At the end of this page, we provide links to it, and you will most likely need to study it to some degree.
Production Programming
Since BSP 5.2, we have introduced an End User License Agreement (EULA) to our Reference Images. It prevents you from using the autoinstall feature of the Toradex Easy Installer, as suggested in the article Production Programming for Linux.
You must remove the license line from the image.json file included in the deployable tarball image resulting from the OpenEmbedded build to allow the autoinstall. It also means that you accept the EULA.
Failing Builds
An OpenEmbedded build sometimes fails. Below some recommended steps to follow when a task fails:
- Reading through the error messages.
- Restarting bitbake without deleting anything and see if it happens again.
- Clean the task which failed and then restart building the image, e.g. if python-native fails
$ bitbake -c clean python-native
$ bitbake <image>
- Clean the task which failed, including deleting cached output and then restart building the image
$ bitbake -c cleansstate python-native
$ bitbake <image>
- Clean the task which failed, including deleting cached output and the downloaded files and then restart building the image
$ bitbake -c cleanall python-native
$ bitbake <image>
-
Check the layer which provides the recipe if a newer commit exists which addresses the problem.
-
If there are many issues with fetching sources, you could first try to download all needed sources.
$ bitbake <image> --runall=fetch
Check the legacy section at the end of this page to find out about failing builds on older BSP versions.
Additional Layers
Recipes for even more software are available in additional layers. E.g. the Java layer provides recipes related to Java. Most layers are registered at openembedded.org. They provide a web interface to find layers or individual recipes.
Note that just because a layer or a recipe exists, that does not mean that these recipes will compile/execute. The layers/recipe might have a version conflict with what we use or might have never been used with or intended for the Arm architecture.
Refer to the following sections to see examples of how to add a new layer to our setup.
Adding the Qt Layer
Please refer to How to set up Qt Creator to cross-compile for embedded Linux.
Adding the Java Layer
Please refer to Java Virtual Machine.
Working with the Package Manager
Starting with BSP 3.0, we dropped the Ångström Distribution in favour of a Poky based distribution. For BSP 3.0 and newer based images, you cannot install packages from an online feed, only packages you build. However, if you use Torizon OS, you can use a container that makes it possible to use Debian feeds.
Read-Only Rootfs
Our base BSP layers and reference images do not support a read-only filesystem because there are features that won't work with a read-only system. In other words, the features of the read/write filesystem are used in several parts of the system, such as the case of the serial number in the hostname of the module. If the rootfs is read-only there is no way to write that modification at runtime.
For more information, refer to How to implement a Read-Only Rootfs (Linux).