Aktualizr - Torizon Update Client Reference Documentation
Introduction
Torizon Remote Updates is the recommended method for Over-the-Air updates using Torizon, our easy-to-use embedded Linux platform. It also supports Secure Offline Updates.
Torizon OS is built with OSTree and Aktualizr. OSTree and Aktualizr are complementary and together they form the foundation for OTA (over-the-air) update capabilities on the device. To learn more about the technical aspects of Torizon Remote and Offline Updates, see the Torizon Updates Technical Overview article.
If you just want to use the Torizon Remote Updates with default settings, the pre-installed software on a Torizon OS image abstracts the settings of Aktualizr for you, therefore you can skip this article.
However, if you need to modify some common configurations of the client-side of Torizon Remote Updates, such as how to block updates, update the server polling time, disable the automatic reboot, among other, or if you just want to learn more about Aktualizr, then this article may be useful for you.
This article will be also useful if you plan to only use the Aktualizr capabilities from Torizon OS, but you don't plan to use the Torizon Remote Updates service.
Prepare for Production
After changing the setting in the SoM, we recommend that you use the TorizonCore Builder Tool to apply customization options to a Torizon OS image for production.
This article complies with the Typographic Conventions for Torizon Documentation.
Prerequisites
- A Computer on Module with Torizon installed.
- Learn more about Aktualizr, by reading the Torizon Remote Updates Technical Overview article.
Before Starting: Stopping and Restarting the Torizon Remote Updates Client Service
You will need to execute the commands as root. You can log in as root or (better) use the sudo command when logged in with a regular user to do it.
Stopping Aktualizr
Before to start changing the Torizon Remote Updates client settings, make sure that Aktualizr is stopped on the board so it won't interfere while creating configuration files:
# systemctl stop aktualizr
Starting Aktualizr
After applying the changes described below, you need to start Aktualizr with the commands below:
# systemctl start aktualizr
You can use journald to view the logs and monitor the operation of the update client:
# journalctl -f -u aktualizr*
Disabling and Enabling Aktualizr
If you don't want Aktualizr to start on every reboot, disable it. If you want to revert it, enable it again:
# systemctl disable aktualizr
# systemctl enable aktualizr
Restarting Aktualizr
If you need to restart Aktualizr use the command below:
# systemctl restart aktualizr
Modifying the Settings of Torizon Remote Updates Client
In this section, we will show how to make common adjustments to Aktualizr settings.
Aktualizr is configured via .toml configuration files. It searches the following directories for files with a .toml extension:
/usr/lib/sota/conf.d//etc/sota/conf.d/
It then processes all the .toml files it finds in alphabetical order. If a config option is specified in multiple files, the last entry overrules any previous entries. Torizon OS includes the following config files by default in /usr/lib/sota/conf.d/:
20-sota-device-cred.toml30-rollback.toml40-hardware-id.toml50-docker-compose.toml60-polling-interval.toml
If you want to override config options in those files, you should create a config file in /etc/sota/conf.d/ to add or modify config options.
Once you have changed the configuration options to the way that suits you, you can apply the changes to a custom Torizon OS image that you can use for production programming, following the instructions in Capture Changes in the Configuration of a Board on Torizon OS.
Aktualizr Configurations and Provisioning Process
You can add provisioning data to a Torizon OS image:
- The shared-data (used both with "offline" and "online" provisioning modes) allows the device to validate updates (by populating the initial Uptane root metadata).
- The online-data (added with the "online" mode provisioning) allows the device to register itself to the platform which is required for it to take updates from the platform.
You may still need to configure Aktualizr to get the desired operation mode, even with your device imbued with provisioning data.
- Online updates are enabled by default on Aktualizr and don't strictly require any extra configuration.
- Offline-updates are disabled by default and would normally need some configuration besides being enabled. Check the steps in the Offline Updates article. A device having only the shared-data will not try to register itself to the platform, but that does not prevent it from accepting offline-updates; it will accept them if Aktualizr is configured to do so.
The registration of the device to the platform is done by the auto-provisioning service. In that context, we overload the meaning of the "provisioning" word to mean the actual registration in the Torizon Cloud.
Configuring for Offline Updates
Aktualizr's standard configuration is to monitor update requests coming through the Remote Updates method, because of this an Offline Update request will not work out-of-the-box.
To reconfigure a device for using Secure Offline updates, you must create or modify a configuration file at /etc/sota/conf.d/. If the folder doesn't exist, you can create it with the following command:
# sudo mkdir -p /etc/sota/conf.d/
Use the following configuration in the file:
# cat /etc/sota/conf.d/99-offline-updates.toml
[uptane]
enable_offline_updates = true
enable_online_updates = false
offline_updates_source = /media/usb/update
This will enable offline updates but disable online updates. For Torizon OS versions before 6.6.0-devel-202402 (monthly), the two update methods were mutually exclusive, i.e. only one method could be enabled at a time, as done above. If you wish to enable both methods, see the next section - Enabling both Online and Offline Updates.
In the sample configuration file, offline_updates_source indicates the folder that aktualizr will monitor for offline updates. In the example, /media/usb it's the path where the relevant USB drive is auto-mounted and update is the name of the actual folder inside that drive containing the update artifacts (i.e. the Lockbox).
You also have to restart the aktualizr-torizon client for configuration changes to take place.
Enabling both Online and Offline Updates
Starting with Torizon OS version 6.6.0-devel-202402 (monthly), it is possible to enable online and offline Updates at the same time. This is achieved simply by setting both booleans enable_offline_updates and enable_online_updates to true in the configuration file shown before. In that case, users will be able perform updates via both methods and in whatever order they like. It is also worth mentioning that the offline method will have a higher priority than the other such that, when triggered, it may even interrupt an ongoing online update.
Enabling Online & Offline Updates simultaneously requires Torizon OS 6.6.0-devel-202402 or newer.
Configuring the Polling Frequency
In Torizon OS, Aktualizr is configured to poll the server for new updates every 5 minutes. To modify this behavior, we can change the uptane.polling_sec option. For example, to change the polling frequency to 1 hour, you could create a configuration file in /etc/sota/conf.d/ called 61-custom-polling-interval.toml:
# cat <<EOF | sudo tee /etc/sota/conf.d/61-custom-polling-interval.toml
[uptane]
polling_sec = 3600
EOF
After applying the changes, don't forget to Restart Aktualizr.
Configuring the Hardware ID
When Toradex publishes our Torizon OS images to the OTA server, their metadata includes a list of which hardware IDs they are compatible with. This ensures that you don't accidentally send an image built for a colibri-imx7 to an apalis-imx8. This hardware compatibility is enforced by the update client: if your device reports that it is an apalis-imx8, it will refuse any update images that don't list apalis-imx8 as a compatible hardware ID.
Normally, it's best not to modify the hardware ID your device reports. If you give it a custom hardware ID, it will no longer accept stock Torizon OS images. But sometimes, that's exactly what you want: for example, if you have several different SKUs that need slightly different OS images, or if you have products with the same SoM but a different carrier board or peripherals. In those cases, you might want to make sure that each SKU or variant has its own hardware ID, and then publish a customized Torizon OS build for each one; that way you can make sure that nobody can accidentally create an OTA update that sends incompatible OS images.
To achieve that, you need to do two things:
- Modify the Aktualizr config on your device image to report a custom hardware ID
- Upload the customized image to Torizon Remote Updates along with metadata indicating it's compatible with your custom hardware ID
Configuring the Hardware ID on Torizon OS
To configure the hardware ID in the Aktualizr config, you need to set the provision.primary_ecu_hardware_id config value. For example, if you wanted your device to be registered with the hardware ID apalis-imx8-sku001, you could create a configuration file in /etc/sota/conf.d/ called 41-custom-hardware-id.toml:
# cat <<EOF | sudo tee /etc/sota/conf.d/41-custom-hardware-id.toml
[provision]
primary_ecu_hardware_id = apalis-imx8-sku001
EOF
Because this is a change to provisioning configuration, you will need to re-provision the device if you want to change it on a live device. It's highly recommended to apply the changes to a custom Torizon OS image that you can use for production programming, following the instructions in Capture Changes in the Configuration of a Board on Torizon OS. That way, your devices will use their custom hardware ID as soon as they are programmed.
Configuring Custom Hardware IDs for Torizon Remote Updates Images
When sending a custom image to the OTA server, you can specify a list of hardware IDs it is compatible with. Using the push command of the TorizonCore Builder Tool, you can specify one or more --hardwareid arguments to override the image's compatible hardware IDs.
$ torizoncore-builder push my-branch --credentials credentials.zip --package-name my-custom-image --hardwareid apalis-imx8-sku0001
Each image can have several hardware IDs assigned to them. To add multiple hardware IDs to a custom image, use the --hardwareid argument once for each hardware ID.
$ torizoncore-builder push my-branch --credentials credentials.zip --package-name my-custom-image --hardwareid apalis-imx8-sku0001 --hardwareid apalis-imx8-sku0003 --hardwareid apalis-imx8-sku007-beta
Remember, Torizon Remote Updates will only permit you to install an image on a device if the hardware IDs match.
Configuring the loglevel
Aktualizr produces logs at different levels of detail depending on the loglevel setting:
- trace (0) logs everything, including the content of every HTTP request
- debug (1) has extra information useful for debug purposes, but is not as verbose as trace
- info (2) is the default, and logs basic information about normal and exceptional Aktualizr operations
- warning (3) only logs warnings and other exceptional information
- error (4) only logs errors
- fatal (5) only logs fatal errors
If you are troubleshooting an issue with the OTA client, it may be helpful to increase the loglevel. To do this, we can change the logger.loglevel option. For example, to enable trace logging, you could create a configuration file in /etc/sota/conf.d/ called 99-custom-loglevel.toml:
# cat <<EOF | sudo tee /etc/sota/conf.d/99-custom-loglevel.toml
[logger]
loglevel = 0
EOF
More detailed loglevels, especially trace, can write a large volume of log data if left on for a long time. We recommend leaving the loglevel at the default except when specifically needed for troubleshooting.
After applying the changes, don't forget to Restart Aktualizr.
Allowing and Blocking Updates
You can lock Aktualizr to avoid receiving new updates on the client.
Every time before applying an update, Aktualizr attempts to acquire a lock on /run/lock/aktualizr-lock using flock.
To help control when updates are applied, you can have a custom code in your application(s) that acquires and releases this lock (see flock (2) man page for more details).
If you are using the command line, it is possible to apply an advisory lock on a open file using the command flock. For example, the command below will apply an advisory lock on /run/lock/aktualizr-lock for 30 seconds:
# sudo flock --verbose -x /run/lock/aktualizr-lock -c "sleep 30"
Enabling and Disabling Automatic Reboot
By default, Torizon OS is configured to automatically reboot the device after a successful update of the operating system. If you want to disable this feature, run the commands below:
# systemctl stop ostree-pending-reboot.path
# systemctl disable ostree-pending-reboot.path
In case you disabled the automatic reboot for system updates and want to re-enable it, run the commands below:
# systemctl enable ostree-pending-reboot.path
# systemctl start ostree-pending-reboot.path
After applying the changes, don't forget to Restart Aktualizr.
Changing the Default Reboot Command
By default, Aktualizr will use /sbin/reboot to reboot the device after a successful update (if enabled). To modify the command used to reboot the device, we can change the bootloader.reboot_command option. For example, you could create a configuration file in /etc/sota/conf.d/ called 21-custom-reboot.toml pointing to a custom reboot script:
# cat <<EOF | sudo tee /etc/sota/conf.d/21-custom-reboot.toml
[bootloader]
reboot_command = "/my-custom-reboot-command"
EOF
Advanced Aktualizr Config Options
Aktualizr has many more config options available. For more details, please consult the project's official documentation.
Setting the Device ID and Name During Provisioning
When you provision a device with Torizon Remote Updates, it will be assigned a Device ID and a Device Name. Both Device ID and Device Name must be unique in your account. By default, Device ID is derived from the SoM model and serial number, and Device Name is generated from a random wordlist.
If you want to provide those values yourself, you can add them to the provisioning command by adding the -d option for Device ID and the -n option for Device Name. Your provisioning command will normally be of the form docker run -v /:/som_sysroot --network=host --privileged -it torizon/torizon-provisioner:0.0.11 provision-device -t ${token} && sudo systemctl restart aktualizr fluent-bit. To set the custom Device ID and Device Name, you would change that to docker run -v /:/som_sysroot --network=host --privileged -it torizon/torizon-provisioner:0.0.11 provision-device -t ${token} -d "my-device-id" -n "My Product 0053" && sudo systemctl restart aktualizr fluent-bit. Note that the added parameters need to come before the &&.
Configuring Secondaries
regular users of Torizon Remote Updates don't need to configure secondaries, they are already configured by default.
Secondary is a concept in Uptane-compatible OTA systems that make it possible to update not only the main operating system but also other firmware and devices connected to it. Torizon OS uses secondaries to update containers via docker-compose files. Secondaries are also used in Subsystem Updates and Bootloader Updates
If you want to override the default configuration - located at paths under /usr, which is ready-only - take them as starting point and create custom configuration files under /etc/sota/conf.d/.
Aktualizr first reads the configuration files from /usr/lib/sota/conf.d/ and loads it. If any custom configuration files are found in /etc/sota/conf.d/, the respective configurations are overridden.
The secondary configuration .toml file points to the secondary definition .json file.
To get the content from the default files, execute the commands below.
Secondary configuration .toml file (copy it from /usr/lib/sota/conf.d and edit under /etc/sota/conf.d/ to define the path for your secondary definition .json file):
# cp /usr/lib/sota/conf.d/50-secondaries.toml /etc/sota/conf.d/50-secondaries.toml
# vi /etc/sota/conf.d/50-secondaries.toml
[uptane]
secondary_config_file = "/etc/sota/secondaries.json"
Secondary definition .json file (copy it from /usr/lib/sota and edit under /etc/sota/):
# cp /usr/lib/sota/secondaries.json /etc/sota/secondaries.json
# cat /etc/sota/secondaries.json
{
"docker-compose": [
{
"partial_verifying": false,
"ecu_hardware_id": "docker-compose",
"full_client_dir": "/var/sota/storage/docker-compose",
"ecu_private_key": "sec.private",
"ecu_public_key": "sec.public",
"firmware_path": "/var/sota/storage/docker-compose/docker-compose.yml",
"target_name_path": "/var/sota/storage/docker-compose/target_name",
"metadata_path": "/var/sota/storage/docker-compose/metadata"
}
],
"torizon-generic": [
{
"partial_verifying": false,
"ecu_hardware_id": "verdin-imx8mp-bootloader",
"full_client_dir": "/var/sota/storage/bootloader",
"ecu_private_key": "sec.private",
"ecu_public_key": "sec.public",
"firmware_path": "/var/sota/storage/bootloader/u-boot.img",
"target_name_path": "/var/sota/storage/bootloader/target_name",
"metadata_path": "/var/sota/storage/bootloader/metadata",
"action_handler_path": "/usr/bin/bl_actions.sh"
}
]
}
Edit to add your custom configuration. For example, add a new secondary for a subsystem update - maintaining the other definitions:
{
"docker-compose": [
{
"partial_verifying": false,
"ecu_hardware_id": "docker-compose",
"full_client_dir": "/var/sota/storage/docker-compose",
"ecu_private_key": "sec.private",
"ecu_public_key": "sec.public",
"firmware_path": "/var/sota/storage/docker-compose/docker-compose.yml",
"target_name_path": "/var/sota/storage/docker-compose/target_name",
"metadata_path": "/var/sota/storage/docker-compose/metadata"
}
],
"torizon-generic": [
{
"partial_verifying": false,
"ecu_hardware_id": "verdin-imx8mp-bootloader",
"full_client_dir": "/var/sota/storage/bootloader",
"ecu_private_key": "sec.private",
"ecu_public_key": "sec.public",
"firmware_path": "/var/sota/storage/bootloader/u-boot.img",
"target_name_path": "/var/sota/storage/bootloader/target_name",
"metadata_path": "/var/sota/storage/bootloader/metadata",
"action_handler_path": "/usr/bin/bl_actions.sh"
},
{
"partial_verifying": false,
"ecu_hardware_id": "my-secondary",
"full_client_dir": "/var/sota/storage/my-secondary",
"ecu_private_key": "sec.private",
"ecu_public_key": "sec.public",
"firmware_path": "/var/sota/storage/my-secondary/my-payload.file",
"target_name_path": "/var/sota/storage/my-secondary/target_name",
"metadata_path": "/var/sota/storage/my-secondary/metadata",
"action_handler_path": "/usr/bin/my_secondary_actions.sh"
}
]
}
After making changes, don't forget to Restart Aktualizr.
Configuring Options for Low-Bandwidth Connections
These options are only available in Torizon OS 6.5.0 or later.
Torizon Remote Updates can present some limitations when dealing with very constrained low-bandwidth connections. These limitations usually manifest as either timeouts or various connection/download errors.
There are a couple of options that can be tweaked to improve update stability in these situations. These options affect the download behavior of OS updates, specifically with how OSTree performs downloads.
To set these options, you must pass environment variables to Aktualizr. Therefore, you it is needed to modify the systemd service that launches Aktualizr.
When editing the systemd service, the environment variables that can be set to control these options are:
[Service]
...
Environment="OSTREE_CURL_TIMEOUT=<timeout_setting_in_s>"
Environment="OSTREE_CURLM_MAX_TOTAL_CONN=<max_number_of_connections>"
...
OSTREE_CURL_TIMEOUT: This sets the timeout, in seconds, for curl requests performed by OSTree. This timeout is per individual curl request and not for the entire update itself. By default this is about 13 minutes. Try increasing this if you find your connection is slow enough to trigger the default timeout.OSTREE_CURLM_MAX_TOTAL_CONN: This sets the max amount of parallel connections that can be used for downloads with OSTree. By default this will automatically optimize to use the max amount of parallel downloads possible. Decreasing the number here will increase the overall time it takes for your OS update to complete. However, less parallel downloads can improve download stability. Since more downloads occuring in parallel increase the chance of a connection error when working with a poor, or low-bandwidth connection.
After applying changes to the systemd service, don't forget to Restart Aktualizr.
Subsystem Updates Resources
The following section brings information about environment variables that Aktualizr provides for the Action handler script or program execution.
Environment Variables
Environment variables passed to the action-handler upon every action:
| Variable | Description |
|---|---|
| SECONDARY_INTERFACE_MAJOR | Major version of the interface; this will be increased upon breaking changes. Initial major is 1.0. |
| SECONDARY_INTERFACE_MINOR | Minor version of the interface; this will be increased upon non-breaking changes. Initial minor is 0. |
| SECONDARY_HARDWARE_ID I/F version: 1.0 | Hardware ID is the identification of the target subsystem for the update. |
| SECONDARY_ECU_SERIAL I/F version: 1.0 | Serial number of the target subsystem being updated; this is a unique number identifying a system component that could be a target for an update. |
Environment variables for the get-firmware-info action:
| Variable | Description |
|---|---|
| SECONDARY_FWINFO_DATA I/F version: 1.0 | JSON string with all available data for the action (Reserved for future use). |
| SECONDARY_FIRMWARE_PATH I/F version: 1.0 | Full-path to the payload file. Upon the “get-firmware-info” action, the file may not exist if there was no previous successful installation. Your get-firmware-info script part can use this information to compare the last provided payload with the currently installed version. |
Environment variables for the install action:
| Variable | Description |
|---|---|
| SECONDARY_INSTALL_DATA I/F version: 1.0 | JSON string with available information for the action (Reserved for future use). |
| SECONDARY_UPDATE_TYPE I/F version: 1.0 | Type of update being performed: online or offline. |
| SECONDARY_FIRMWARE_PATH I/F version: 1.0 | Full-path to the payload file to be installed. Upon the “install” action, this path would point to a new payload file that will replace the previous one (if installation is successful). |
| SECONDARY_FIRMWARE_SHA256 I/F version: 1.0 | The SHA-256 checksum of the firmware file to be installed. |
| SECONDARY_CUSTOM_METADATA I/F version: 1.0 | JSON string containing the merged custom metadata from the director and image Uptane repositories associated with the target (package) being installed. |
Environment variables for the complete-install action:
| Variable | Description |
|---|---|
| SECONDARY_CMPLINSTALL_DATA I/F version: 1.0 | JSON string with all available data for the action (Reserved for future use). |
| SECONDARY_FIRMWARE_PATH I/F version: 1.0 | Full-path to the firmware file to be installed. Upon the “complete-install” action, this path would point to a new firmware file that will replace the previous one (if installation is successful). |
| SECONDARY_FIRMWARE_SHA256 I/F version: 1.0 | The SHA-256 checksum of the firmware file to be installed. |
| SECONDARY_CUSTOM_METADATA I/F version: 1.0 | JSON string containing the merged custom metadata from the director and image Uptane repositories associated with the target (package) being installed. |
Exit Codes
The following exit codes will be received from the action-handler execution:
| Exit Code | Description |
|---|---|
| 0 (Action handled) | Indicate that the action was handled by the handler which in turn should also output a JSON string (into its standard output) with information specific to the executed action. |
| 64 (Request normal processing) | Indicate that the action was not handled by the handler but that the Aktualizr update client can move on by performing its normal (non-error) processing for that action if possible. |
| 65 (Request error processing) | Indicate that the action was not handled by the handler and that the Aktualizr update client should consider the situation as a failure. |
| 66 (Abort processing) | Reserved for future use |
Update Checks and Rollbacks
Greenboot (Generic Health Check Framework) is a Fedora project that helps manage systemd services health. Torizon OS uses Greenboot as a framework to make update checks and rollbacks more flexible and manageable by the user.
By default, Torizon OS will consider a successful boot if the boot-complete systemd target is successfully executed. This is because the main operating system services required for proper operation, including the Docker daemon, are inside boot-complete.target. And if boot-complete.target fails during an update, Torizon OS will automatically reboot, and after three tries, it will rollback to the previous operating system version.
In case you want to add additional checks to confirm a successful update, you can add shell scripts to /etc/greenboot/check/required.d/. As a convention, the script name should start with two numbers and finish with .sh (Example: 01_check_system.sh). Scripts in /etc/greenboot/check/required.d/ will be executed as part of the boot health checks.
If the scripts in /etc/greenboot/check/required.d/ are successfully executed, and the boot-complete.target is successfully started, the system will enter in the GREEN state. In this case, the greenboot-task-runner service will be triggered, and user-defined scripts inside /etc/greenboot/green.d/ will be executed. These scripts can be used to execute post-install operations during a successful update, but be aware that they will run every time the operating system boots.
Now, if one of the scripts in /etc/greenboot/check/required.d/ fail (exit code is not 0), the boot-complete.target will also fail, and the system will enter in the RED state. In this case, the redboot-task-runner service will be triggered, and user-defined scripts inside /etc/greenboot/red.d/ will be executed. These scripts can be used to execute post-install operations during a failed update, but be aware that they will run every time the operating system boots (when boot-complete.target fails). After redboot-task-runner service finishes execution, the redboot-auto-reboot service is triggered, and this service will run a script that will reboot the system (in case an update is in progress). After 3 reboots, the system will rollback to the previous operating system version.
To confirm if a boot was successful (GREEN status) or if it failed (RED status), you can check the status of the greenboot-status systemd service:
$ systemctl status greenboot-status
* greenboot-status.service - greenboot MotD Generator
Loaded: loaded (/usr/lib/systemd/system/greenboot-status.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2021-04-26 12:50:04 UTC; 16min ago
Process: 809 ExecStart=/usr/libexec/greenboot/greenboot-status (code=exited, status=0/SUCCESS)
Main PID: 809 (code=exited, status=0/SUCCESS)
Apr 26 12:50:04 apalis-imx6-05039068 systemd[1]: Starting greenboot MotD Generator...
Apr 26 12:50:04 apalis-imx6-05039068 greenboot-status[814]: Boot Status is GREEN - Health Check SUCCESS
Apr 26 12:50:04 apalis-imx6-05039068 systemd[1]: Started greenboot MotD Generator.
Alternatively, you can list the content of the /run/boot-status file:
$ cat /run/boot-status
Boot Status is GREEN - Health Check SUCCESS
For more information about the Greenboot framework, please see the project's website.
Troubleshooting
To see in more detail what Aktualizr is doing, you can stop the systemd service and start Aktualizr manually, e.g. with an increased loglevel for debugging:
# systemctl stop aktualizr
# aktualizr-torizon --loglevel 1
# systemctl stop aktualizr
# aktualizr --loglevel 1
To provision the device again, make sure to stop Aktualizr, remove the device and start Aktualizr again:
# rm /var/sota/sql.db
# rm -rf /var/sota/storage/
Common Issues
If you see the following message:
response http code: 400
response: "An error occurred: Missing entity: Ecu"
could not put manifest
Make sure to properly delete the storage of the secondary (see above).