Search by Tags

Torizon OTA (Over the Air Updates)

 

Article updated at 26 Mar 2021
Compare with Revision




Subscribe for this article updates

Introduction

Torizon OTA is the recommended OTA method for Torizon, our easy-to-use embedded Linux platform.

This article is split into major sections covering the following topics:

  • How to get started: a simple and concise introduction. It will help you have a smooth hands-on experience.
  • Technical overview: an optional read with technical details for those who are interested in learning more.
  • Documentation: Links to important articles and documents to use Torizon OTA.

Prerequisites

How to Get Started With Torizon OTA

It is very simple to get started, you just need to create an account and login to our hosted dashboard:

Get Started With Torizon OTA

It is a good experience to just explore the dashboard, but you can of course learn more with documentation:

Technical Overview

TorizonCore is built with OSTree and Aktualizr, the former is a shared library and suite of command line tools that combines a "git-like" model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration. The latter is a "daemon-like" open-source implementation of the Uptane SOTA standard that secures updates from end-to-end.

OSTree and Aktualizr are complementary and together they form the foundation for OTA (over-the-air) update capabilities on the device.

The device portion of the Torizon OTA reuses what Linux microPlatform and meta-updater are providing. You can find more about the OTA strategy on the foundries.io Blog.

On the server side, Toradex is working on a cloud-based hosted option as well as an on-premise option to provide a complete OTA solution that works with Torizon-Core. This is currently a work in progress, subscribe to developer website updates to keep track of the progress. Meanwhile, you can Update Your Device Using HERE OTA Connect for early testing.

This article complies to the Typographic Conventions for Torizon Documentation.

OSTree

OSTree has its own article, please refer to OSTree for a brief overview and a demonstration of how to use it.

Uptane

Uptane is a de facto automotive SOTA standard, held by a non-profit consortium named Uptane Alliance under the IEEE/ISTO Federation. Its focus is to enable secure software updates over-the-air resiliently. It relies on multiple servers to provide security by validating data before a download starts and ensuring that even an offline attack that compromises a single server would still not be enough to compromise the system security. Uptane is an enhancement to the TUF (The Update Framework) security framework, which is currently a very widely used framework to secure software and package updates on computers and smartphones. The motivations to expand the TUF framework is described in detail in the Uptane Design page and a favorable explanation of TUF is in its docs page Understand the Notary service architecture.

Aktualizr

Aktualizr is the client implementation of Uptane. It is written in C++ and its responsibility is to communicate with a Uptane compatible server. It verifies if new downloads are available, install those updates on the system and reports status to the server, while guaranteeing the integrity and confidentiality of OTA updates. Aktualizr handles Docker image updates seamlessly by using Docker Compose yml files.

How to Use Aktualizr

Aktualizr is a dedicated article that covers the practical aspects, including usage.

Rollback

There can be cases where the system may fail to boot or the boot process is considered unsuccessful either due to kernel panic or failure to start any critical user space application. These issues can be handled by developers during development, but it becomes a nightmare if the solution is deployed and such an issue occurs due to any bad update. This issue can be avoided if the following mechanisms are present:

  • kernel reboots on panic or hung task
  • the system is able to detect a failure of any critical user space operation
  • capability to configure the bootloader to boot a usable image in case of any failure

Rollback and bootcount support of aktualizr

The automatic rollback feature relies on aktualizr’s rollback support as well. TorizonCore uses aktualizr with rollback_mode set to uboot_masked (see aktualizr client configuration options). This enables aktualizr’s U-Boot bootcount integration: After an update aktualizr enables boot counting by setting the upgrade_available U-Boot environment variable to 1. In the error case, when the system reboots due to kernel/service failure, aktualizr won’t get started. After three tries U-Boot will rollback the system by setting the U-Boot environment rollback to 1. If the system has been rolled back, aktualizr does not mark the boot as successful (the system stays in rollback mode). If the system has been booted successfully, the upgrade_available and bootcount is set back to 0 to let the boot loader know that boot counting is no longer necessary.

Note: When installing an update without aktualizr (e.g. using ostree admin directly) automatic rollback will not work. To use automatic rollback in a pure OSTree system, those steps need to be executed manually as described in Ostree!

Implementation

The implementation makes use of the following:

  • OSTree
  • U-Boot’s bootcount support
  • Automatic reboot on kernel panic and service failure
  • aktualizr’s U-Boot bootcount integration.

  • Torizon Update Workflow

    Torizon Update Workflow - Click to expand

TorizonCore’s OTA allows to rollback to the last installed update thanks to its OSTree based root file system. It also allows to keep multiple deployments (kernel/initramfs/device-tree and the rootfs) on a system and have them bootable. The initial (factory) image has only a single deployment available and is assumed to be a working deployment (no rollback can be done at this point). After the first update has been rolled out, there are two deployments on the system at all times. If a new deployment fails, the system will automatically roll back to the previous deployment.

U-Boot bootcount

The U-Boot bootcount feature provides a simple boot counter and alternative boot command. The alternative boot command stored in altbootcmd and will boot into the previous OSTree deployment. If boot counting is enabled and the boot counter exceeds a predefined limit, the alternative boot command is executed. The boot limit is defined by the bootlimit environment variable by default set to 3. The boot count is held in the U-Boot environment on the on-module eMMC/raw NAND flash. To avoid too much wear, boot counting is only enabled after an update, controlled by the upgrade_available environment variable.

Automatic Reboot on Failure

The current implementation relies on software to reboot in case of a serious failure. The TorizonCore kernel has the CONFIG_PANIC_TIMEOUT configuration option enabled which reboots the system automatically on a kernel panic situation. In user space, the default configuration assumes the docker.service to be the crucial service. TorizonCore uses systemd’s FailureAction to tell systemd to reboot the system if starting the service fails.

Note: This setup relies on Linux' and systemd’s corporation in the failure case. Since these two software projects are fairly well tested and the features relied upon are fairly small, the risk of reboot failing is fairly small.

Boot Assessment - systemd

systemd offers automatic boot assessment for UEFI based systems. Since the Toradex module does not use UEFI the systemd boot assessment does not directly apply to Toradex modules. However, we reuse some aspects of the automatic boot assessment. In particular, the boot-complete.target is used as the synchronization point for services which are required to consider the system boot to be successful and the service which marks the system boot as successful. By default, we order the docker.service before the boot-complete.target, and the aktualizr.service after the boot-complete.target.

Torizon OTA Documentation

An overview of relevant articles is presented in this tab. You can scroll through contents or click on your topic of interest below:

First Steps and Essentials, Torizon Tools and TorizonCore Customization

We use icons to help you identify the content in an article:

Visual Studio Visual Studio Code Command Line Conceptual Starting Point Docker Concepts Samples Repository

First Steps and Essentials

Technical information about Torizon OTA implementation.

Article Brief Description
Over-The-Air Updates An overview of over-the-air update technologies for Toradex modules
Torizon OTA An overview of the Torizon OTA, the OTA solution integrated to Torizon
Torizon OTA Web Interface An overview of the web interface of Torizon OTA and its features
Signing and pushing TorizonCore images to Torizon OTA A step-by-step tutorial on how to build and push to Torizon OTA a custom TorizonCore image for OTA Deployment
Using Private Registries With Torizon OTA How to deploy private registry credentials to your devices
OSTree An overview of a core technology used in the Torizon OTA
Aktualizr Practical steps for using Aktualizr, a core technology used in the Torizon OTA

Torizon Tools and TorizonCore Customization

The articles below show information about Torizon Tools and TorizonCore Customization to prepare an image for OTA deployment.

Article Brief Description
Developer Tools Container for TorizonCore A container with developer tools for TorizonCore
TorizonCore Builder Tool - Customization for Production Programming and Torizon OTA Create a custom TorizonCore image that you can use in production programming with Toradex Easy Installer, or push your changes to Torizon OTA
Device Tree Overlays on Torizon How to modify the device tree without having to re-compile it
Device Tree Customization Examples Some examples of Device Tree Customization
Pin Multiplexing - Changing Pin Functionalities in the Linux Device Tree How to create a dts file to change the pin muxing configuration
Building External Kernel Modules With Torizon How to deploy an external kernel module using TorizonCore Builder with an example
Customizing Kernel Arguments in Torizon How to use TorizonCore Builder to customize kernel arguments in Torizon
Setting up Displays with Torizon How to set up displays with Torizon
Splash Screen Customization on TorizonCore Learn how to change the splash screen using initramfs
Touch Screen Calibration (Torizon) How to use weston-touch-calibrator to calibrate the touch
Capturing Changes in the Configuration of a Board on TorizonCore Use TorizonCore Builder to save and reproduce board customization
Pre-provisioning Docker Containers onto a TorizonCore image How to preinstall a Docker Container onto a TorizonCore image using your PC
How to Store Docker Data on an External Storage Device (USB/SD Card) How to storage Docker data on an external device like an SD Card or a USB stick
Persistent Journald Logging How switch from in-RAM to persistent Journald logging
Using Private Registries With Torizon OTA How to deploy private registry credentials to your devices