Search by Tags

Aktualizr - Modifying the Settings of Torizon OTA Client

 

Article updated at 26 Apr 2021
Compare with Revision




Subscribe for this article updates

Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.



Remember that you can always refer to the Torizon Documentation, there you can find a lot of relevant articles that might help you in the application development.

Torizon 5.3.0

Introduction

Torizon OTA is the recommended method for Over-the-Air updates using Torizon, our easy-to-use embedded Linux platform.

TorizonCore is built with OSTree and Aktualizr. OSTree and Aktualizr are complementary and together they form the foundation for OTA (over-the-air) update capabilities on the device. To learn more about the technical aspects of Torizon OTA, see the Torizon OTA (Over the Air Updates) article.

If you just want to use the Torizon OTA with default settings, the pre-installed software on a TorizonCore image abstracts the settings of Aktualizr for you, therefore you can skip this article.

However, if you need to modify some common configurations of the client-side of Torizon OTA, such as how to block updates, update the server polling time, disable the automatic reboot, among other, or if you just want to learn more about Aktualizr, then this article may be useful for you.

This article will be also useful if you plan to only use the Aktualizr capabilities from TorizonCore, but you don't plan to use the Torizon OTA service.

Prepare for Production

After changing the setting in the SoM, we recommend that you use the TorizonCore Builder Tool to apply customization options to a TorizonCore image for production.

This article complies to the Typographic Conventions for Torizon Documentation.

Prerequisites

Before Starting: Stopping and Restarting the Torizon OTA Client Service

Attention: You will need to execute the commands as root. You can log in as root or (better) use the sudo command when logged in with a regular user to do it.

Stopping Aktualizr

Before to start changing the Torizon OTA client settings, make sure that Aktualizr is stopped on the board so it won't interfere while creating configuration files:

# systemctl stop aktualizr

Starting Aktualizr

After applying the changes described below, you need to re-start Aktualizr with the commands below:

# systemctl start aktualizr

You can use journald to view the logs and monitor the operation of the update client:

# journalctl -f -u aktualizr*

Disabling and Enabling Aktualizr

If you don't want Aktualizr to start on every reboot, disable it. If you want to revert it, enable it again:

# systemctl disable aktualizr
# systemctl enable aktualizr

Modifying the Settings of Torizon OTA Client

In this section, we will show how to make common adjustments on Aktualizr settings.

Allowing and Blocking Updates

You can lock Aktualizr to avoid receiving new updates on the client.

Every time before applying an update, Aktualizr attempts to acquire a lock on /run/lock/aktualizr-lock using flock.

To help control when updates are applied, you can have a custom code in your application(s) that acquire and release this lock (see flock (2) man page for more details).

If you are using the command line, it is possible to apply an advisory lock on a open file using the command flock. For example, the command below will apply an advisory lock on /run/lock/aktualizr-lock for 30 seconds:

# sudo flock --verbose -x /run/lock/aktualizr-lock -c "sleep 30"

After applying the changes, don't forget to Restart Aktualizr.

Configuring the Polling Frequency

In TorizonCore, Aktualizr is configured to poll the server for new updates every 5 minutes. To modify this behavior, we can change the polling_sec option. For example, create the configuration file below to change the polling frequency to 60 seconds:

# cp /usr/lib/sota/conf.d/60-polling-interval.toml /etc/sota/conf.d/
# cat /etc/sota/conf.d/60-polling-interval.toml 
[uptane]
polling_sec = 60

After applying the changes, don't forget to Restart Aktualizr.

Configuring the Serial Number

Optionally, you can configure the serial number of your device, that will help differentiate it from other devices in the OTA server Web interface. For that, run the commands below:

# cp /usr/lib/sota/conf.d/40-hardware-id.toml /etc/sota/conf.d/
# echo primary_ecu_serial = $(tr -d '\0' </proc/device-tree/serial-number) >> /etc/sota/conf.d/40-hardware-id.toml

After applying the changes, don't forget to Restart Aktualizr.

Enabling and Disabling Automatic Reboot

By default, TorizonCore is configured to automatically reboot the device after a successful update of the operating system. If you want to disable this feature, run the commands below:

# systemctl stop ostree-pending-reboot.path
# systemctl disable ostree-pending-reboot.path

In case you disabled the automatic reboot for system updates and want to re-enable it, run the commands below:

# systemctl enable ostree-pending-reboot.path
# systemctl start ostree-pending-reboot.path

After applying the changes, don't forget to Restart Aktualizr.

Changing the Default Reboot Command

By default, Aktualizr will use /sbin/reboot to reboot the device after a successful update (if enabled). To modify the command used to reboot the device, we can change the reboot_command option. For example, create the configuration file below to change the command to reboot the device:

# cat /etc/sota/conf.d/20-bootloader.toml
[bootloader]
reboot_command = "/my-custom-reboot-command"

Note: For more information about Aktualizr configuration options, please see the project's official documentation

After applying the changes, don't forget to Restart Aktualizr.

Provisioning and Configuring a New Device

The first step to start receiving updates from an Uptane-compatible OTA server is to provision the device, importing credentials and generating certificates that will be used to communicate with the server.

The Torizon OTA Web Interface automatically provides the commands to registers, download, and installs credentials for you when provisioning a new device. Therefore, in most of the cases, a developer is not required to proceed with the instructions of this section.


  • Command to provision a new device provided by the Torizon OTA Web Interface

    Command to provision a new device provided by the Torizon OTA Web Interface

However, if you want to learn more about the provisioning process, we will show how to execute the process manually. These instructions will be also useful if you plan to only use the Aktualizr capabilities from TorizonCore, but you don't plan to use the Torizon OTA service. Note that you may need to read the documentation of the Uptane-compatible OTA server you are using, such as Torizon OTA.

If your OTA server provides a credentials.zip file, you can use the aktualizr-cert-provider tool to import the security information from the file and provision the device:

Attention: You should not copy this file to the device because it contains your private keys, during device setup you may access it from a USB thumbdrive or SD card. If you have to copy it, ensure that you delete the file after you executed the command below.

# aktualizr-cert-provider -c <path to removable media>/credentials.zip -l / -u -r

Attention: Note the device id in the first line of output of aktualizr-cert-provider command. Your device will probably show as this id in the OTA server Web interface.

After applying the changes, don't forget to Restart Aktualizr.

Configuring Secondaries (Application Container Updates)

Note: regular users of Torizon OTA don't need to configure secondaries, they are already configured by default.

Secondary is a concept in Uptane-compatible OTA systems that make it possible to update not only the main operating system but also other firmware and devices connected to it. TorizonCore uses secondaries to update containers via docker-compose files.

If you want to override the default configuration, take it as an example and create custom configuration files under /etc/sota/. To get the content from the default files, execute the commands below:

# cat /usr/lib/sota/conf.d/50-docker-compose.toml 
[uptane]
secondary_config_file = "/usr/lib/sota/docker-compose.json"

# cat /usr/lib/sota/docker-compose.json
{
    "docker-compose": [
        {
            "partial_verifying": false,
            "ecu_hardware_id": "docker-compose",
            "full_client_dir": "/var/sota/storage/docker-compose",
            "ecu_private_key": "sec.private",
            "ecu_public_key": "sec.public",
            "firmware_path": "/var/sota/storage/docker-compose/docker-compose.yml",
            "target_name_path": "/var/sota/storage/docker-compose/target_name",
            "metadata_path": "/var/sota/storage/docker-compose/metadata"
        }
    ]
}

After applying changes, don't forget to Restart Aktualizr.

Update Checks and Rollbacks

Greenboot (Generic Health Check Framework) is a Fedora project that helps manage systemd services health. TorizonCore uses Greenboot as a framework to make update checks and rollbacks more flexible and manageable by the user.

By default, TorizonCore will consider a successful boot if the boot-complete systemd target is successfully executed. This is because the main operating system services required for proper operation, including the Docker daemon, are inside boot-complete.target. And if boot-complete.target fails during an update, TorizonCore will automatically reboot, and after three tries, it will rollback to the previous operating system version.

In case you want to add additional checks to confirm a successful update, you can add shell scripts to /etc/greenboot/check/required.d/. As a convention, the script name should start with two numbers and finish with .sh (Example: 01_check_system.sh). Scripts in /etc/greenboot/check/required.d/ will be executed as part of the boot health checks.

If the scripts in /etc/greenboot/check/required.d/ are successfully executed, and the boot-complete.target is successfully started, the system will enter in the GREEN state. In this case, the greenboot-task-runner service will be triggered, and user-defined scripts inside /etc/greenboot/green.d/ will be executed. These scripts can be used to execute post-install operations during a successful update, but be aware that they will run every time the operating system boots.

Now, if one of the scripts in /etc/greenboot/check/required.d/ fail (exit code is not 0), the boot-complete.target will also fail, and the system will enter in the RED state. In this case, the redboot-task-runner service will be triggered, and user-defined scripts inside /etc/greenboot/red.d/ will be executed. These scripts can be used to execute post-install operations during a failed update, but be aware that they will run every time the operating system boots (when boot-complete.target fails). After redboot-task-runner service finishes execution, the redboot-auto-reboot service is triggered, and this service will run a script that will reboot the system (in case an update is in progress). After 3 reboots, the system will rollback to the previous operating system version.

To confirm if a boot was successful (GREEN status) or if it failed (RED status), you can check the status of the greenboot-status systemd service:

$ systemctl status greenboot-status
* greenboot-status.service - greenboot MotD Generator
     Loaded: loaded (/usr/lib/systemd/system/greenboot-status.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2021-04-26 12:50:04 UTC; 16min ago
    Process: 809 ExecStart=/usr/libexec/greenboot/greenboot-status (code=exited, status=0/SUCCESS)
   Main PID: 809 (code=exited, status=0/SUCCESS)

Apr 26 12:50:04 apalis-imx6-05039068 systemd[1]: Starting greenboot MotD Generator...
Apr 26 12:50:04 apalis-imx6-05039068 greenboot-status[814]: Boot Status is GREEN - Health Check SUCCESS
Apr 26 12:50:04 apalis-imx6-05039068 systemd[1]: Started greenboot MotD Generator.

Alternatively, you can list the content of the /run/boot-status file:

$ cat /run/boot-status
Boot Status is GREEN - Health Check SUCCESS

For more information about the Greenboot framework, please see the project's website.

Troubleshooting

To see in more detail what Aktualizr is doing, you can stop the systemd service and start Aktualizr manually, e.g. with an increased loglevel for debugging:

# systemctl stop aktualizr
# aktualizr-torizon --loglevel 1

To provision the device again, make sure to stop Aktualizr, remove the device and start Aktualizr again:

# rm /var/sota/sql.db
# rm -rf /var/sota/storage/

Common Issues

If you see the following message:

response http code: 400
response: "An error occurred: Missing entity: Ecu"
could not put manifest

Make sure to properly delete the storage of the secondary (see above).

Torizon 5.0.0

Introduction

Torizon OTA is the recommended method for Over-the-Air updates using Torizon, our easy-to-use embedded Linux platform.

TorizonCore is built with OSTree and Aktualizr. OSTree and Aktualizr are complementary and together they form the foundation for OTA (over-the-air) update capabilities on the device. To learn more about the technical aspects of Torizon OTA, see the Torizon OTA (Over the Air Updates) article.

If you just want to use the Torizon OTA with default settings, the pre-installed software on a TorizonCore image abstracts the settings of Aktualizr for you, therefore you can skip this article.

However, if you need to modify some common configurations of the client-side of Torizon OTA, such as how to block updates, update the server polling time, disable the automatic reboot, among other, or if you just want to learn more about Aktualizr, then this article may be useful for you.

This article will be also useful if you plan to only use the Aktualizr capabilities from TorizonCore, but you don't plan to use the Torizon OTA service.

Prepare for Production

After changing the setting in the SoM, we recommend that you use the TorizonCore Builder Tool to apply customization options to a TorizonCore image for production.

This article complies to the Typographic Conventions for Torizon Documentation.

Prerequisites

Before Starting: Stopping and Restarting the Torizon OTA Client Service

Attention: You will need to execute the commands as root. You can log in as root or (better) use the sudo command when logged in with a regular user to do it.

Stopping Aktualizr

Before to start changing the Torizon OTA client settings, make sure that Aktualizr is stopped on the board so it won't interfere while creating configuration files:

# systemctl stop aktualizr

Starting Aktualizr

After applying the changes described below, you need to re-start Aktualizr with the commands below:

# systemctl start aktualizr

You can use journald to view the logs and monitor the operation of the update client:

# journalctl -f -u aktualizr*

Disabling and Enabling Aktualizr

If you don't want Aktualizr to start on every reboot, disable it. If you want to revert it, enable it again:

# systemctl disable aktualizr
# systemctl enable aktualizr

Modifying the Settings of Torizon OTA Client

In this section, we will show how to make common adjustments on Aktualizr settings.

Allowing and Blocking Updates

You can lock Aktualizr to avoid receiving new updates on the client.

Every time before applying an update, Aktualizr attempts to acquire a lock on /run/lock/aktualizr-lock using flock.

To help control when updates are applied, you can have a custom code in your application(s) that acquire and release this lock (see flock (2) man page for more details).

If you are using the command line, it is possible to apply an advisory lock on a open file using the command flock. For example, the command below will apply an advisory lock on /run/lock/aktualizr-lock for 30 seconds:

# sudo flock --verbose -x /run/lock/aktualizr-lock -c "sleep 30"

After applying the changes, don't forget to Restart Aktualizr.

Configuring the Polling Frequency

In TorizonCore, Aktualizr is configured to poll the server for new updates every 5 minutes. To modify this behavior, we can change the polling_sec option. For example, create the configuration file below to change the polling frequency to 60 seconds:

# cp /usr/lib/sota/conf.d/60-polling-interval.toml /etc/sota/conf.d/
# cat /etc/sota/conf.d/60-polling-interval.toml 
[uptane]
polling_sec = 60

After applying the changes, don't forget to Restart Aktualizr.

Configuring the Serial Number

Optionally, you can configure the serial number of your device, that will help differentiate it from other devices in the OTA server Web interface. For that, run the commands below:

# cp /usr/lib/sota/conf.d/40-hardware-id.toml /etc/sota/conf.d/
# echo primary_ecu_serial = $(tr -d '\0' </proc/device-tree/serial-number) >> /etc/sota/conf.d/40-hardware-id.toml

After applying the changes, don't forget to Restart Aktualizr.

Enabling and Disabling Automatic Reboot

By default, TorizonCore is configured to automatically reboot the device after a successful update of the operating system. If you want to disable this feature, run the commands below:

# systemctl stop ostree-pending-reboot.path
# systemctl disable ostree-pending-reboot.path

In case you disabled the automatic reboot for system updates and want to re-enable it, run the commands below:

# systemctl enable ostree-pending-reboot.path
# systemctl start ostree-pending-reboot.path

After applying the changes, don't forget to Restart Aktualizr.

Changing the Default Reboot Command

By default, Aktualizr will use /sbin/reboot to reboot the device after a successful update (if enabled). To modify the command used to reboot the device, we can change the reboot_command option. For example, create the configuration file below to change the command to reboot the device:

# cat /etc/sota/conf.d/20-bootloader.toml
[bootloader]
reboot_command = "/my-custom-reboot-command"

Note: For more information about Aktualizr configuration options, please see the project's official documentation

After applying the changes, don't forget to Restart Aktualizr.

Provisioning and Configuring a New Device

The first step to start receiving updates from an Uptane-compatible OTA server is to provision the device, importing credentials and generating certificates that will be used to communicate with the server.

The Torizon OTA Web Interface automatically provides the commands to registers, download, and installs credentials for you when provisioning a new device. Therefore, in most of the cases, a developer is not required to proceed with the instructions of this section.


  • Command to provision a new device provided by the Torizon OTA Web Interface

    Command to provision a new device provided by the Torizon OTA Web Interface

However, if you want to learn more about the provisioning process, we will show how to execute the process manually. These instructions will be also useful if you plan to only use the Aktualizr capabilities from TorizonCore, but you don't plan to use the Torizon OTA service. Note that you may need to read the documentation of the Uptane-compatible OTA server you are using, such as Torizon OTA.

If your OTA server provides a credentials.zip file, you can use the aktualizr-cert-provider tool to import the security information from the file and provision the device:

Attention: You should not copy this file to the device because it contains your private keys, during device setup you may access it from a USB thumbdrive or SD card. If you have to copy it, ensure that you delete the file after you executed the command below.

# aktualizr-cert-provider -c <path to removable media>/credentials.zip -l / -u -r

Attention: Note the device id in the first line of output of aktualizr-cert-provider command. Your device will probably show as this id in the OTA server Web interface.

After applying the changes, don't forget to Restart Aktualizr.

Configuring Secondaries (Application Container Updates)

Note: regular users of Torizon OTA don't need to configure secondaries, they are already configured by default.

Secondary is a concept in Uptane-compatible OTA systems that make it possible to update not only the main operating system but also other firmware and devices connected to it. TorizonCore uses secondaries to update containers via docker-compose files.

If you want to override the default configuration, take it as example and create custom configuration files under /etc/sota/. To get the content from the default files, execute the commands below:

# cat /usr/lib/sota/conf.d/50-docker-compose.toml 
[uptane]
secondary_config_file = "/usr/lib/sota/docker-compose.json"

# cat /usr/lib/sota/docker-compose.json
{
    "docker-compose": [
        {
            "partial_verifying": false,
            "ecu_hardware_id": "docker-compose",
            "full_client_dir": "/var/sota/storage/docker-compose",
            "ecu_private_key": "sec.private",
            "ecu_public_key": "sec.public",
            "firmware_path": "/var/sota/storage/docker-compose/docker-compose.yml",
            "target_name_path": "/var/sota/storage/docker-compose/target_name",
            "metadata_path": "/var/sota/storage/docker-compose/metadata"
        }
    ]
}

After applying changes, don't forget to Restart Aktualizr.

Troubleshooting

To see in more detail what Aktualizr is doing, you can stop the systemd service and start Aktualizr manually, e.g. with an increased loglevel for debugging:

# systemctl stop aktualizr
# aktualizr --loglevel 1

To provision the device again, make sure to stop Aktualizr, remove the device and start Aktualizr again:

# rm /var/sota/sql.db
# rm -rf /var/sota/storage/

Common issues

If you see the following message:

response http code: 400
response: "An error occurred: Missing entity: Ecu"
could not put manifest

Make sure to properly delete the storage of the secondary (see above).