OSTree
Introduction
OSTree is a library - actually libostree - that handles updates for filesystem trees, that is, the entire Linux root filesystem. It applies updates atomically, therefore guaranteeing the system integrity in case of a power-cut or dropped internet connectivity for instance. Different from the A/B partition mechanism of updates, OSTree works in a git-like model, providing file updates and thus saving storage space as well as connection bandwidth and allowing more than 2 software revisions on the device at the same time. In addition, some directories such as /var and /etc and handled as special cases, providing persistent configuration and user data to be preserved between updates.
OSTree is a key technology used by Torizon Remote and Offline Updates, which is based on Aktualizr. If Aktualizr is used, OSTree is managed by it and should not be manipulated directly. This article shows how to use OSTree directly from command-line, useful for development or when the Torizon Cloud/Aktualizr is not being used.
Intended Audience
If you just want to use the Torizon Remote and Offline Updates, it will abstract the OSTree for you, therefore you can skip this article. If you want to learn more, or if you plan to only use the OSTree capabilities from Torizon OS, but you don't plan to use Torizon Cloud, then this article may be useful for you.
This article complies with the Typographic Conventions for Torizon Documentation.
Prerequisites
- A Computer on Module with Torizon installed.
- (optional) The setup and output of a Torizon OS build.
Direct OSTree Update
This article shows how-to perform a Torizon OS update using the ostree command directly in a shell on the device.
Whenever Torizon OS is built using Yocto/OpenEmbedded's bitbake, a OSTree repository is automatically initialized and a new OSTree commit is created. For every following build, a new commit is created where only changed files are added to the repository. Inside the OSTree repository, the builds are organized in branches (or REFs). REFs follow a naming convention consisting of the Torizon OS major version number, machine name, distribution name, image name and release type, e.g.:
4/colibri-imx6/torizon/torizon-core-docker/nightly
REFs with a major version 0 are builds from sources taken from the master branch, typically meant for Torizon OS development only.
Toradex uploads the OSTree repository of Torizon OS builds produced by the Toradex CI infrastructure to make them available publicly https://feeds.toradex.com/ostree/.
This article shows how to update from both sources:
- Toradex Torizon OS OSTree repository
- Local OSTree repository from a local OpenEmbedded/bitbake build
OSTree allows to switch between different sources transparently. E.g. it is possible to deploy a OSTree from a local build on a Torizon OS installation made from the official builds.
All device commands must be executed as root, so you have to prefix the following command lines with "sudo" to execute them as root.
Update From Toradex Torizon OS OSTree repository
This chapter shows how to update from a Torizon OS installation to a newer build obtained from the Toradex provided Torizon OS OSTree.
On the board, add the Toradex OSTree repository as a new remote using ostree remote, similar to how one would add a remote in a git repo:
# ostree remote add --no-gpg-verify torizon https://feeds.toradex.com/ostree/
Currently Toradex does not sign the vanilla Torizon OS OSTree commits (hence the --no-gpg-verify is required). In the Torizon Cloud the Torizon OS commits are cryptographic signed by Uptane/aktualizr.
The ostree admin commands allow to manage the OSTree on the host system. The command ostree admin status shows details about the currently active deployments (root filesystems).
# ostree admin status
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
This shows an installation of a monthly build on a Colibri iMX6. At this point, there is only a single deployment active, the one from the original installation.
The ostree admin upgrade command can be used to upgrade to the latest monthly release. This command downloads the latest commit of the origin branch, and deploys that in one step. At the time of writing (July 16th) the monthly branch is not available on the OSTree repository. Future monthly releases are expected to be available.
Switch To a Different OSTree Branch
To switch to a different branch, e.g. to nightly build or a new major release, the branch name needs to be explicitly stated. The ostree remote refs command allows to get a list of branches available on a remote OSTree repository:
# ostree remote refs torizon
...
torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
...
Use OSTree pull to download the lastest version of a particular branch:
# ostree pull torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
32 metadata, 53 content objects fetched; 51978 KiB transferred in 21 seconds
To see more details about the fetched OSTree commits, ostree log can be used:
# ostree log torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
commit 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e
ContentChecksum: fc8b18d816f877671d52193221f888d9b651732f1644b08827bdc9bdec61d996
Date: 2020-07-15 19:43:31 +0000
Version: 4.0.0-devel-20200715+build.166
4.0.0-devel-20200715+build.166
meta-toradex-torizon HEAD:75acc4a3acfd2104e47769b2a7e038f2503d57dc
meta-toradex-distro HEAD:276c966a04bbe4d19866becb3b8fc147d1079049
meta-toradex-bsp-common HEAD:419eba81a51fb8f8e5a4eef9f23f1734c7044c78
meta-oe HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-networking HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-filesystems HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-python HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-perl HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-virtualization HEAD:514ce5f6b240600f06b9956737be1e33900bacac
meta-updater HEAD:12e0cd9e18afadfdd90d81fb947950b4620901fc
meta-toradex-nxp HEAD:acbfe83716e6a045d9290d932eec29a9afb468f1
meta-freescale HEAD:f535ddcd5cd3da846a34b1721ceda3c8b18b37a0
meta-freescale-3rdparty HEAD:dbcc686f52c3c84db8cb86aa8973a4e373651b98
meta-yocto-bsp HEAD:2d2dc20359ca75203f2194415b53731ad8ff0066
meta-poky HEAD:2d2dc20359ca75203f2194415b53731ad8ff0066
meta-security HEAD:440c37f0b623ccc0aa0328613908608d6362adda
meta-lmp-base HEAD:067487d1d4a823237eb6e7bbdebf9b023b58affa
meta HEAD:ee95a399285abbde84e0148ca957b59d65bcad0a
<< History beyond this commit not fetched >>
To see which files have been changed, ostree diff can be used. To show the complete diff of the current running commit and the target REF, you have to specify the commit from the ostree admin status command explicitly (leave away the part after the dot).
# ostree diff 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
M /usr/package.manifest
M /usr/bin/chardetect3
...
The command ostree admin deploy allows to create a new deployment using the given branch which will be used upon next boot:
# ostree admin deploy torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
Copying /etc changes: 4 modified, 1 removed, 7 added
Transaction complete; bootconfig swap: yes; deployment count change: 1
OSTree shows that there is a switch to a new root filesystem tree pending:
# ostree admin status
torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0 (pending)
Version: 4.0.0-devel-20200715+build.166
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
On the next boot, the system will boot using the new filesystem tree.
# reboot
After reboot you can view your current and previous deployment (still available for rollback):
# ostree admin status
* torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0
Version: 4.0.0-devel-20200715+build.166
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0 (rollback)
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
OSTree Update From a Toradex Easy Installer image
A Toradex Easy Installer image internally contains an OSTree repository which can be served via HTTP with TorizonCore Builder. For this, unpack the image with images unpack and make it available on the network with ostree serve, that is:
$ torizoncore-builder images unpack path/to/image
$ torizoncore-builder ostree serve
On the board, add the local OSTree repository using ostree remote, similar to how one would add to a Git repo:
# ostree remote add --no-gpg-verify local-build \
http://<your PC host name/ip address>:8080/
You then can list the available branches and deploy a new tree using the same commands documented above (using the remote name local-build).
OSTree Update From a Local Torizon OS Build
If you are building Torizon OS image locally you may provide an ostree repo directly from your PC.
Just move to the ostree_repo folder of your build directory (located in the OpenEmbedded build directory under deploy/images/$MACHINE/ostree_repo) and start an HTTP server using one of the following options:
Using TorizonCore Builder's ostree serve command (with the --ostree-repo-directory switch to serve an external OSTree repository):
$ torizoncore-builder ostree serve --ostree-repo-directory ./
Using Python's built-in basic HTTP server:
$ python -m SimpleHTTPServer 8080 # Python 2 (some systems use 'python2' as the executable)
$ python -m http.server 8080 # Python 3 (some systems use 'python3' as the executable)
In this way your PC will be sharing the image you just built on port 8080.
On the board, add the local OSTree repository using ostree remote, similar to how one would add to a Git repo:
# ostree remote add --no-gpg-verify local-build \
http://<your PC host name/ip address>:8080/
You then can list the available branches and deploy a new tree using the same commands documented above (using the remote name local-build).
OSTree Update from a USB flash drive
If you don't have network connectivity on target board an update can also be deployed using a USB flash drive or SD card.
If you are building Torizon OS image locally the OSTree repository will be in deploy/images/$MACHINE/ostree_repo. The repository is in archive mode and hence can be copied with a regular copy command:
$ cp -r deploy/images/$MACHINE/ostree_repo path/to/flash-drive
You can also create a new OSTree repository and fetch an update from the Toradex Torizon OS OSTree repository:
$ cd path/to/flash-drive
$ ostree init --repo=ostree_repo --mode=archive
$ ostree remote add --repo=ostree_repo --no-gpg-verify torizon https://feeds.toradex.com/ostree/
$ ostree pull --repo=ostree_repo torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
$ ostree refs --repo=ostree_repo --create=4/colibri-imx6/torizon/torizon-core-docker/nightly torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
It seems that the ostree pull-local command used on the target cannot handle REFs with a colon, hence a new remote independent reference needs to be created.
On the target, list the refs available on the given repository:
# ostree refs --repo=path/to/flash-drive
torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
4/colibri-imx6/torizon/torizon-core-docker/nightly
Pull the commit from the given branch name from the USB flash drive to the target board:
# ostree pull-local /home/root/pendrive/ostree_repo/ 4/colibri-imx6/torizon/torizon-core-docker/nightly
Create a new deployment from the given branch name:
# ostree admin deploy 4/colibri-imx6/torizon/torizon-core-docker/nightly
OSTree shows that there is a switch to a new root filesystem tree pending:
# ostree admin status
torizon 1d98e25015a390556e7ab93c9b94e01e8632d30d800b36bff1ddd54fa9fc7cf4.0 (pending)
origin refspec: apalis-imx6/torizon/torizon-core-docker
* torizon d0d53487d305358ad50c37b1b2a94fc78eaea5e770749a33f5607573e2447660.0
origin refspec: d0d53487d305358ad50c37b1b2a94fc78eaea5e770749a33f5607573e2447660
On the next boot, the system will boot using the new filesystem tree.
# reboot
OSTree Rollback
OSTree allows to rollback to the last root filesystem in case of a bad update. A rollback can be manually triggered by setting the U-Boot environment variable rollback:
> env set rollback 1
The boot script will load the kernel/device tree and initramfs from the rollback deployment. This is shown in the boot message as well as visible when looking at ostree admin status
...
Rollback enabled. Booting previously deployed version.
...
# ostree admin status
torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0 (pending)
Version: 4.0.0-devel-20200715+build.166
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
In this state a new (hopefully working) OSTree can be deployed.
Automatic Rollback
Torizon OS relies on U-Boot boot count support to provide automatic rollback in case a bad update has been deployed. Any update that causes issues is expected to reboot. For that Torizon OS configures the kernel to automatically reboot in case of a kernel panic, and instructs systemd to reboot in case a user defined critical application (by default docker.service) does not start. If the system detects more reboots than what is defined in U-Boot's bootlimit environment variable, the rollback mechanism will be triggered. With aktualizr the U-Boot boot count is automatically enabled after an upgrade. When manually deploying an update using OSTree (ostree admin deploy), the boot count support needs to be enabled manually to enable automatic rollback support:
- For emmc based modules
# boot_partition=$(readlink /dev/emmc-boot0)
# echo 0 > /sys/block/${boot_partition}/force_ro
# fw_setenv upgrade_available 1
# fw_setenv bootcount 0
# echo 1 > /sys/block/${boot_partition}/force_ro
- For MTD based modules
# fw_setenv upgrade_available 1
# fw_setenv bootcount 0
After the reboot, the output of the ostree admin status can be used to see if the upgrade succeeded.
After an unsuccessful update, the system stays in "rollback" mode. A new update need to be deployed and the rollback variable needs to be cleared:
# fw_setenv rollback 0
After a successful update, the boot count should be disabled again (since regular flash is used as boot counter, this prevents unnecessary wear on the flash used for the U-Boot environment)
# fw_setenv upgrade_available 0
OSTree Deployments
By default, OSTree will retain only the last two deployments, the current one and the last one to roolback in case of a bad update. If you want to retain more than two deployments, the --retain parameter can be used when creating a new deployment:
# ostree admin deploy --retain torizon:5/$MACHINE/torizon-rt/torizon-core-podman/nightly
Copying /etc changes: 4 modified, 2 removed, 9 added
Transaction complete; bootconfig swap: yes; deployment count change: 1
Now OSTree will show three deployments:
# ostree admin status
torizon eb07b368ff1e41350835af66252b736e3f73f4f063d70d599208b6776a0e4a1e.0 (pending)
Version: 5.1.0-devel-20201122+build.128
origin refspec: torizon:5/apalis-imx8/torizon-rt/torizon-core-podman/nightly
* torizon 4ebb32723cdd786c172ec7fd2be213bf49faf13bbec4b885b248978c822325e9.0
Version: 5.1.0-devel-20201122+build.128
origin refspec: torizon:5/apalis-imx8/torizon/torizon-core-podman/nightly
torizon 5bda7e34fcbda221e728e9b7ad79349ba88671e166b683bb7f3399d43fd2aff0.0 (rollback)
Version: 5.1.0-devel-20201122+build.128
origin refspec: torizon:5/apalis-imx8/torizon-rt/torizon-core-docker/nightly
When deploying a new filesystem with OSTree, some directories will be handled in a special way:
/etc: when OSTree creates a deployment, it performs a 3-way merge using the old default configuration, the active system's/etc, and the new default configuration. In the final filesystem tree for a deployment,/etcis a regular writable directory that can be used to store system configuration./var: this directory is not touched by OSTree, and can be used to store logs, databases and any other data consumed or generated by applications./home: this directory is also not touched by OSTree and can be used to store any user-related files. It is actually a link to/var/rootdirs/home.