Search by Tags

Torizon OTA Technical Overview

 

Article updated at 12 May 2021
Subscribe for this article updates

Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.



Remember that you can always refer to the Torizon Documentation, there you can find a lot of relevant articles that might help you in the application development.

Torizon 5.0.0

Introduction

TorizonCore is built with OSTree and Aktualizr-Torizon, the former is a shared library and suite of command-line tools that combines a "git-like" model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration. The latter is a fork of Aktualizr, a "daemon-like" open-source implementation of the Uptane SOTA standard that secures updates from end-to-end.

OSTree and Aktualizr-Torizon are complementary and together they form the foundation for OTA (over-the-air) update capabilities on the device.

On the server-side, Toradex is working on a cloud-based hosted option as well as an on-premise option to provide a complete OTA solution that works with TorizonCore.

This article complies to the Typographic Conventions for Torizon Documentation.

OSTree

OSTree has its own article, please refer to OSTree for a brief overview and a demonstration of how to use it.

Uptane

Uptane is a de facto automotive SOTA standard, held by a non-profit consortium named Uptane Alliance under the IEEE/ISTO Federation. Its focus is to enable secure software updates over-the-air resiliently. It relies on multiple servers to provide security by validating data before a download starts and ensuring that even an offline attack that compromises a single server would still not be enough to compromise the system security. Uptane is an enhancement to the TUF (The Update Framework) security framework, which is currently a very widely used framework to secure software and package updates on computers and smartphones. The motivations to expand the TUF framework are described in detail in the Uptane Design page and a favorable explanation of TUF is in its docs page Understand the Notary service architecture.

Aktualizr-Torizon

Aktualizr-Torizon is TorizonCore's client implementation of Uptane (forked from Aktualizr's default client). It is written in C++ and its responsibility is to communicate with a Uptane compatible server. It verifies if new downloads are available, installs those updates on the system, and reports status to the server while guaranteeing the integrity and confidentiality of OTA updates. Aktualizr handles Docker image updates seamlessly by using Docker Compose YAML files.

How to Use Aktualizr-Torizon

Aktualizr - Modifying the Settings of Torizon OTA Client is a dedicated article that covers the practical aspects, including its usage.

Update rollbacks

There can be cases where the system may fail to boot or the boot process is considered unsuccessful either due to kernel panic or failure to start any critical user-space application. These issues can be handled by developers during development, but it becomes a nightmare if the solution is deployed and such an issue occurs due to any bad update.

TorizonCore and Aktualizr-Torizon are fully capable to recover from bad updates by doing the following:

  1. Identifying unsuccessful updates and rebooting the device when it occurs.
  2. Rolling back to the previous operating system version after 3 unsuccessful boots.

Identifying unsuccessful updates

In TorizonCore, the Linux kernel is configured to panic and reboot in case of freezes or crashes. This helps to recover from bad kernel updates.

At the user-space level, systemd hardware watchdog integration is enabled by default in TorizonCore. That means systemd will regularly ping the watchdog hardware, and if systemd (or the kernel) hangs, this ping will not happen anymore and the hardware will automatically reboot the device. This helps to recover from bad updates when the kernel or the initialization daemon (systemd) is not able to run.

Lastly, TorizonCore will consider a successful boot if the boot-complete systemd target is successfully executed. This is because the main operating system services required for proper operation, including the Docker daemon, are inside boot-complete.target. And if boot-complete.target fails during an update, TorizonCore will automatically reboot. This helps TorizonCore to recover from bad updates when critical processes from the base operating system don't run as expected.

A TorizonCore user may also define his own "rules" to validate an update using the Greenboot framework. Greenboot (Generic Health Check Framework) is a Fedora project that helps manage systemd services health, and TorizonCore uses Greenboot as a framework to make update checks more flexible and manageable for the user. With Greenboot, you can define a shell script that can do additional checks in the system and force a reboot if needed. For more information about how to use Greenboot, have a look at Update Checks and Rollbacks.

Rolling back to the previous operating system version

As mentioned above, TorizonCore will automatically roll back after 3 unsuccessful reboots. And the automatic rollback feature relies on Aktualizr-Torizon’s rollback support and U-Boot's bootcount feature.

TorizonCore uses Aktualizr-Torizon with rollback_mode set to uboot_masked. This enables Aktualizr-Torizon’s U-Boot bootcount integration:

  1. After an update, Aktualizr-Torizon enables boot counting by setting U-Boot's environment variables upgrade_available to 1 and bootlimit to 3.
  2. In case of a bad update, the system will reboot and U-Boot will increment bootcount environment variable. After three times (when bootcount is greater than bootlimit), the system will roll back to the previously installed OS version.
  3. In case of a good update, Aktualizr-Torizon is normally started and U-Boot environment variables upgrade_available and bootcount are set back to 0.

TorizonCore’s OTA allows it to roll back to the last installed update thanks to its OSTree based root file system. It also allows to keep multiple deployments (kernel/initramfs/device-tree and the rootfs) on a system and have them bootable. The initial (factory) image has only a single deployment available and is assumed to be a working deployment (no rollback can be done at this point). After the first update has been rolled out, there will be two deployments on the system at all times. If a new deployment fails, the system will automatically roll back to the previous deployment.

Note: When installing an update without Aktualizr-Torizon (e.g. using ostree admin directly) automatic rollback will not work. To use automatic rollback in a pure OSTree system, those steps need to be executed manually as described in Ostree!

Webinars

Beyond Development: Torizon

Meeting the Challenge of OTA for Embedded Linux Systems