OSTree

Introduction

OSTree is a library, technically called libostree, that manages multiple bootable filesystem trees within a single partition. Versions are delivered and maintained atomically, therefore guaranteeing the system integrity in case of a power-cut or dropped internet connectivity. When used for over-the-air updates, it has benefits over the traditional dual partition approach; OSTree updates are provided at the file-level, saving storage and connection bandwidth. Having multiple software revisions on the device at the same time allows for a known-good rollback candidate when deploying updates over-the-air. Additionally, directories such as /var and /etc are handled as special cases, providing persistent configuration and preservation of user data.

OSTree is a key technology used by Torizon Remote and Offline Updates, based on the Aktualizr update client. Users of Torizon shouldn't need to directly interact with OSTree.

To customize a Torizon OS image, use TorizonCore Builder instead of modifying OSTree directly. TorizonCore Builder builds the changes into a proper OSTree deployment to stay compatible with rollback.

warning

Invoking OSTree commands directly on a Torizon device may almost certainly break things. Only do that on debug devices and at the specific direction of Toradex support. To ensure that the OSTree system is reset to properly function with the Torizon Remote and Offline Updates system, reflash the entire device using the Toradex Easy Installer after debugging is completed.

Description

OSTree can store multiple bootable filesystem versions in a single partition. This works in a manner very similar to how git stores multiple revisions of a source base in a single repository. All files that are needed for any of the bootable versions are stored in a repository. This is a folder structure where the files are stored, called objects at this point, indexed by their checksum so that the repository object for a specific file can be easily located. When all objects have been added to the repository, OSTree creates a "deployment" for the specific collection of objects that make up a specific bootable version of the filesystem. This deployment (which is different from, but related to, a "deployment" in the Torizon cloud platform) is a directory of hard links to the objects in the repository. Code running in both the bootloader and the initial ramdisk can select, at boot time, the appropriate version.

Torizon uses OSTree to provide a "known-good" rollback version during updates. This allows systems to have a fully automated rollback candidate. While OSTree can store an arbitrary number of versions, Torizon prunes out older versions to keep the OSTree repository limited in size.

When deploying a new filesystem with OSTree, some directories are handled in a special way:

• /usr: this directory holds the OS itself and is mounted read-only. Even the root user can't write to it without explicitly unlocking the deployment. This is what prevents accidental or partial modification of the OS.
• /etc: when OSTree creates a deployment, it performs a 3-way merge using the old default configuration, the active system's /etc, and the new default configuration. This ensures that changes made locally on the device persist across updates. In the final filesystem tree for a deployment, /etc is a regular writable directory that can be used to store system configuration.
• /var: this directory isn't touched by OSTree, and can be used to store logs, databases, and any other data consumed or generated by applications.
• /home: this directory also isn't touched by OSTree and can be used to store any user-related files. This is actually a symbolic link to /var/rootdirs/home.

Boot Flow

Adding OSTree to the boot process introduces a single, well-defined extra step rather than a new or opaque boot flow:

1. The bootloader loads the kernel and the initial ramdisk, initramfs, as it would on any Linux system
2. The initramfs reads the ostree= kernel argument to determine which deployment is active
3. The initramfs performs a pivot_root into that deployment's filesystem tree
4. The system continues booting normally from there

Benefits

Using OSTree for updates provides the following benefits:

• Automatic Rollback: Torizon OS relies on U-Boot boot count support to provide automatic rollback in case a bad update has been deployed. Any update that causes issues is expected to reboot and trigger this rollback. Issues such as a Kernel panic automatically trigger a reboot, and if the Kernel itself crashes, the watchdog timer forces one. Additionally, Torizon users can install custom post-install checks that can also trigger this process. Because the previous deployment stays on disk untouched, it's always available as the automatic fallback target.

• Atomic Updates: OSTree deployments are atomic units, meaning that they're either completely installed, or not installed at all. A new deployment is fully staged on disk before the bootloader is ever pointed at it, so a power loss or dropped connection during an update can't leave the device in a half-updated state. Traditional desktop-style updates using package managers such as apt are generally not atomic, and in those cases, it can be difficult to know the exact set of packages installed on any device based on a single version number. OSTree solves this issue by using a cryptographically unique commit hash for each version, and content is indexed by checksum, so corruption or tampering is detected rather than silently deployed.

• Immutable Updates: Related to atomic updates, OSTree revisions, as used in the Torizon OS, are immutable: /usr is mounted read-only, so devices running the same OSTree commit hash have the exact same set of OS files.

• Space reduction: Compared to block-based (that is, dual-partition) update mechanisms, OSTree is more space efficient. You don't need a separate, inactive partition to store new updates, so you can use a single partition instead. Because different OSTree versions share identical objects through hard links, this gives you the same fail-safe redundancy as A/B partitioning without paying for two full copies of the OS on disk.

• Bandwidth reduction: When a new OSTree version is deployed, only new and modified objects need to be downloaded. All objects that are identical to the ones in the current version are simply reused with a filesystem hard link. Static delta support can shrink update payloads even further for larger jumps between versions.

• Downtime reduction: When deploying a new update, compared to block-based updates, downloading only the new or modified objects can result in a significant speedup. This results in faster updates and less downtime for users. Runtime performance isn't affected either: an OSTree-managed root filesystem is a standard ext4 filesystem, not an overlay, so reads and writes perform the same as on a non-OSTree system.

Common Misconceptions

OSTree is often assumed to be slow, fragile, or overly complex, based on outdated or generic information. The following points clarify some of the most common misconceptions about its use in Torizon OS:

• "OSTree is slow": The root filesystem is standard ext4, not an overlay, so there's no runtime performance penalty. Any perceived slowness comes from build or image-generation time, not from running the device.
• "OSTree is unreliable and prone to data corruption": The opposite is true: atomic deployments and checksummed content are specifically what protect against corruption from power loss or interrupted updates.
• "OSTree is too complex for simple changes": The TorizonCore Builder can be used to make customizations into an image without needing to directly interact with OSTree. Application data belongs in /var or /home, not in the OS partition, regardless of the update mechanism.
• "OSTree changes the boot flow into a black box": The usage of OSTree just adds a single, well-defined extra step to the boot process. The bootloader still loads the kernel and initramfs, and the system continues booting normally after that. For more information, refer to the Boot flow section.
• "OSTree can't provide A/B-style redundancy": The usage of hard links allows OSTree to provide the same fail-safe redundancy as A/B partitioning without paying for two full copies of the OS on disk. For more information, refer to the Benefits section.

For a deeper technical walkthrough and benchmark data behind each of these points, refer to The 5 Myths of OSTree blog post.