Is this page helpful?

Root Filesystem Protection on Torizon OS

Introduction

This article is part of the documentation about Secure Boot on Torizon OS providing details on the root filesystem protection. Notice that the present document uses terminology introduced in the main article.

Underlying technologies

The root filesystem protection in Torizon OS is built on composefs – a modern technology integrated into OSTree. Both topics are extensive but well-documented. We encourage interested readers to explore the provided links for more details on each.

Integration between OSTree & composefs

With composefs integrated into OSTree, the /etc directory becomes volatile, i.e. any changes made at runtime are lost on a system reboot. This is the main user-visible difference which affects the workflow of users customizing the OS and it may also impact the design decisions for a product.

The following sections will examine the impacts due to the volatile /etc directory. When reading on, keep in mind that whenever we refer to a volatile file or directory, what we mean is that changes to the file or directory are volatile: if a change is made at runtime, it will be lost as soon as the device reboots. Similarly, when we refer to a file as persistent, we mean that the file can be modified, and those modifications will still persist after the system reboots.

info

For a more in-depth explanation of the differences introduced by the integration of composefs into OSTree, please refer to the section Technical details.

Dealing with a volatile /etc directory

Overview

No universal approach exists for dealing with a volatile /etc directory. Each user will have their own product requirement. Because of that, when Torizon OS is built with the root filesystem protection enabled, we take the secure-by-default approach: all artifacts inside /etc are volatile. For each file that needs to allow persistent changes, the user must make modifications to the system to allow that file to be persistent.

This section serves primarily as a tutorial where we'll examine an example where runtime changes have been made to files inside the /etc directory, and we'll decide how to handle each one of the files. Beware that this is just an example and so are the decisions taken; different products may have requirements leading to different decisions.

Before we start, it's worth reiterating that we're talking about runtime changes to the directory. Those changes are normally made by system services or libraries with the goal of keeping some information across reboots, most often configuration information. For example, if a user of the system connects to a new Wifi network using NetworkManager, a config file will be created in /etc/NetworkManager/system-connections/ saving the connection settings, and that config file is normally supposed to persist across reboots.

Here are the steps we will follow in the example:

1. Identify the modifications made to the /etc directory.
2. Determine which files need to be persistent.
3. Decide on a strategy for how the files will be made persistent.
4. Make the necessary system changes to ensure persistence, and integrate those changes into a custom image.

Identifying modifications to the /etc directory

The recommended way for identifying the modifications to the /etc directory is actually running the desired image on a device. The image should already include all expected customizations for the final version, except possibly those related to the /etc persistence which we are introducing here. However, the procedure described also supports an incremental approach.

Assuming one already has an image available, which is supposed to have been built with the torizon-signed class (as per Build Secure Boot Image), they would prepare the device according to these steps:

1. Install the image; the installation should be from scratch which normally means doing it via Toradex Easy Installer.
2. Boot the device with the new image; beware that the first boot after installation will take longer due to the filesystem preparation for use with composefs.
3. Log into the system as usual.
4. Set the boot arguments to disable composefs:

   $ sudo fw_setenv tdxargs "cfs.enabled=false"
5. Reboot the system:

   $ sudo reboot

The system will then boot without the rootfs protection. Some warning messages will show up on the serial console regarding the changes in boot arguments, which is expected. Notice that such changes are only allowed when the device is in the "open" state (in terms of secure-boot).

At this point one can log into the system and exercise any operations that might cause changes to the /etc directory. Those operations could even include rebooting the device multiple times if needed. In most cases though, the changes to /etc will occur on the first boot and will remain there due to us disabling composefs.

Now we can use an administrative command provided by ostree to determine the changes that have occurred, as shown:

$ sudo ostree admin config-diff
M    shadow
M    mtab
M    machine-id
D    systemd/system/sysinit.target.wants/run-postinsts.service
D    rpm-postinsts/100-torizon-users
A    ssh/ssh_host_rsa_key
A    ssh/ssh_host_rsa_key.pub
A    ssh/ssh_host_ecdsa_key
A    ssh/ssh_host_ecdsa_key.pub
A    ssh/ssh_host_ed25519_key
A    ssh/ssh_host_ed25519_key.pub
A    .pwd.lock
A    locale.conf
A    vconsole.conf
A    .updated
A    shadow-
A    .passwd_changed
A    hostname

The first character indicates if the file was modified (M), added (A) or deleted (D) when compared to the version deployed with the OS image (i.e. the pristine version under /usr/etc).

caution

At the time of this writing, the ostree admin config-diff command does not support detecting changes to the /etc directory when composefs is enabled. If one boots without disabling it, they would see no output when running that command - this situation should be fixed in the future.

Determining which files need to be persistent

At this point we have a list of modified, added and deleted files in /etc:

• Modified:
  machine-id, mtab, shadow
• Added:
  hostname, shadow-, .pwd.lock, .passwd_changed, .updated, locale.conf, vconsole.conf, ssh/ssh_host_*_key, ssh/ssh_host_*_key.pub
• Deleted:
  systemd/system/sysinit.target.wants/run-postinsts.service, rpm-postinsts/100-torizon-users

The basic idea is to go over the list and determine whether each file needs to be persistent to meet our requirements. This work might involve looking up documentation, source code or metadata (Yocto). Another possibility which is encouraged is to return to the design and explore ways to eliminate the need for the file persistence. In fact, from the security standpoint it is always preferable to keep system changes volatile, with only essential modifications made persistent.

tip

As a general principle, we aim to minimize file persistence, striving to make the system as stateless as possible.

Following with the example, let's go over each one of the files and take our decisions:

File	Reasoning	Decision
machine-id	The systemd documentation explains that this file is used to decide whether a boot is the first one; its contents is used by system components requiring a unique identity for the system; it also says that on stateless systems (which we are aiming for) this ID would be generated automatically during early-boot.	Keep volatile
mtab	This is just a symlink to /proc/self/mounts generated at runtime by systemd-tmpfiles. This could be considered as a false positive caused by the dynamic generation of the symlink. One way to prevent the false positive is to change the design so that the symlink already exists with the correct target in the OSTree commit. Anyway, keeping it volatile makes no harm.	Keep volatile
shadow	The shadow file stores encrypted user passwords. For simplicity, we will assume the device will accept logins using predefined users and passwords, allowing for a volatile file. A change in design would be advisable: to only allow remote logins through key-based authentication. If local logins are required, they could be set up to use challenge-response authentication. If none of these were acceptable, the shadow file would need to be made persistent which unfortunately is both not straightforward to achieve and undesirable for security reasons.	Keep volatile
hostname	On Torizon OS, this file is generated by the set-hostname service based on static information for a given device (machine name and device serial number); thus it is acceptable to regenerate it on every boot.	Keep volatile
shadow-	This is a just backup file created when shadow is modified; we don't need to persist it.	Keep volatile
.pwd.lock	This is a lock file used to prevent multiple processes from simultaneously modifying /etc/passwd; there is no need to persist it.	Keep volatile
.passwd_changed	Looking up this filename on the whole root filesystem we can find a single occurrence inside /usr/etc/rpm-postinsts/100-torizon-users which is a script that causes the expiration of the torizon user password; on systems that have already been customized, this script should not even exist and, if it does, this is an indication that some customization is missing; so let us ignore this one.	Ignore
.updated	Checking the contents of this text file it says it was generated by systemd-update-done.service; looking up the service documentation it mentions the file is used by systemd to implement the condition ConditionNeedsUpdate= for triggering post-update steps; out of the box, Torizon OS does not have any important services needing post-update execution, so we'll consider there is no need to persist this file.	Keep volatile
locale.conf, vconsole.conf	A lookup on the root filesystem shows these files are copies (performed at boot time by systemd-tmpfiles) of default templates so it would be fine to leave them as not persistent.	Keep volatile
ssh_host_*_key, ssh_host_*_key.pub	These are files holding keys that uniquely identify the device as a host on the SSH protocol; they are produced the first time the device boots after image installation. If this device is going to accept SSH connections then it is best to keep the files across reboots otherwise clients connecting to it might detect the change in keys as a security issue. So, our decision is to persist these files.	Make persistent
run-postinsts.service, 100-torizon-users	These two files implement the password expiration for the torizon user and on a system properly customized they should not exist. Notice that the files were deleted (as detected by ostree admin config-diff) which is an additional complication. If that happened in practice, the recommendation would be to change the design to add rather than delete files inside /etc. For example, the service could be modified to create a file (on a persist location) after it executes successfully once and use the presence of that file to prevent its own execution on future system boots. Here, we assume these deletions will not be present.	Ignore

Additional tips:

• Before deciding to persist a configuration file, ensure it does not allow setting paths for executable artifacts. For example, suppose the above list included a configuration file that defined the path to a program executed upon a specific event. If that file were made persistent, an attacker could modify it to execute a malicious program stored in a persistent location outside of the composefs protection. This would allow the attacker to introduce long-standing changes to the system, compromising the Secure Boot Chain of Trust.
• Executable scripts should never be made persistent.
• Making whole directories persistent should be avoided because one would need to ensure that none of the artifacts in the directory would ever be considered as configuration files or executable programs from the perspective of the program that parses the directory. For example, one could be tempted to make the /etc/ssh directory persistent because that directory stores the host keys but the same directory can store a configuration file parsed by the SSH server which an attacker could modify.

danger

Artifacts that enable the direct or indirect execution of code should never be made persistent, as they could become an attack vector for compromising the Secure Boot Chain of Trust.

Deciding on a strategy for how the files will be made persistent

In fact, there is no way to make a file in /etc persistent. So, if one wants a file currently in that directory to become persistent they can:

• Change the system configuration in order to read or write that file to a persistent location (e.g. somewhere under /var); this is the preferable and most reliable solution.
• Replace that file by a symlink pointing to the actual file which in turn would be stored on a persistent location; however, keep in mind this technique may not work depending on how the libraries/programs access the file (specially how the file is written/updated).

Regardless of the selected approach, users may also need to set the initial contents of the file in the persistent location, for which they can set up a configuration file for systemd-tmpfiles. This program is invoked by systemd-tmpfiles-setup.service at relatively early stages of the boot process making it fit for many use cases. Overall, by following one of the two approaches, it should be possible to handle most of the files under /etc. However, if a customer comes across a situation where none of them work, we encourage them to get in touch with Toradex via our support channels.

Now, continuing with our example, after our analysis we ended up with a small set of files that we want to persist, namely:

• ssh/ssh_host_*_key, ssh/ssh_host_*_key.pub: These files are created and read from locations defined by the configuration file /etc/ssh/sshd_config. The most straightforward and reliable solution is to change the configuration so that the host keys are stored in a persistent location.
  • Checking the sshd_config documentation one can determine the desired result can be achieved by setting the HostKey keywords, like this:

    # sample stretch of a /etc/ssh/sshd_config storing key files under /var/rootdirs/etc/ssh/
    HostKey /var/rootdirs/etc/ssh/ssh_host_rsa_key
    HostKey /var/rootdirs/etc/ssh/ssh_host_ecdsa_key
    HostKey /var/rootdirs/etc/ssh/ssh_host_ed25519_key
  • We must also make sure the path defined above exists in the persistent location, which can be achieve by a systemd-tmpfiles configuration like this (matching the above example):

    # sample /etc/tmpfiles.d/persist-sshd-keys.conf (or /usr/lib/tmpfiles.d/persist-sshd-keys.conf)
    d /var/rootdirs/etc 0755 root root -
    d /var/rootdirs/etc/ssh/ 0755 root root -

Making the system changes and integrating them into the image (Yocto)

Here we outline what needs to be done in terms of Yocto to achieve the persistence of selected files as decided on the previous section. The implementation details are left to the developer who is assumed to be familiar with Yocto:

• ssh/ssh_host_*_key, ssh/ssh_host_*_key.pub:
  • Create an append file for the recipe responsible for installing file /etc/ssh/sshd_config.
  • In the append file, patch sshd_config to set the paths for the various keys to the desired persistent (as per previous section).
  • As part of the recipe, also install a configuration file for systemd-tmpfiles to cause the creation of the required persistent directories at runtime (as per previous section).

Other impacts due to composefs

Besides the /etc directory being volatile, there are a few other differences visible at runtime when composefs is active:

• The device's first boot after installation typically takes a few minutes. This is due to a process required for enabling fsverity on the full OSTree repository. During that process, a progress bar will be output on the serial console:

  Enabling fsverity on the ostree repository - this may take a few minutes.
  
  Progress: [==================================================] (11801/11801)
  
  Enabling fsverity took 163 seconds.

• Following the filesytem preparation, the use composefs does not introduce any noticeable boot delays. Updating the device to a new rootfs does not cause the fsverity enablement task to run again.

• Given that composefs relies on fsverity for protection, the overhead for file reads is generally considered negligible.

Considerations regarding system updates

• It is important to keep in mind that once a file is moved to a persistent location, that file is no longer managed by OSTree. If one needs the file to be updated upon a system update, then extra actions will be required.
• When the root filesystem protection is active, OSTree's usual mechanism of doing a 3-way merge on the contents of /etc is no longer meaningful, since /etc is not allowed to be modified from the version in the deployed rootfs. A system update causes the /etc directory to be "reset" to the (new) pristine version in the newly deployed OSTree commit, except for the files explicitly made persistent by storing them outside of /etc.

danger

Upgrading devices in the field to start using root filesystem integrity protection (i.e., moving from BCoT to ECoT) requires a special procedure; documentation on the topic is under way.

Known issues

• Handling files whose location is normally fixed, such as /etc/password and /etc/shadow, presents unique challenges. The main difficulties arise from two key factors:
  • Many libraries and utilities have these file locations hardcoded in their implementations.
  • Since these files represent critical system databases, their updates is normally implemented in a atomic fashion by creating a new file and renaming it on top of the original one. This makes the symlink technique ineffective (i.e. the symlink gets replaced by a regular file during those updates).

Technical details

For the interested reader, here we provide a more detailed technical description of the changes introduced by the integration of composefs into OSTree.

On-disk representation of deployments

On an OSTree system without composefs, what we'll call plain OSTree, a deployment is represented by a simple directory tree on disk where the files are hard links to a central repository of files – this repository makes up a content addressed store (CAS) as the files are referenced by their unique cryptographic hashes.

With the introduction of composefs, the OSTree deployment is now represented by a single file on disk – the so called composefs image – holding the directory tree structure of the deployment along with related metadata, but not including each file's data. The contents of each file is still kept in the same OSTree CAS while the image keeps just references to them. Besides holding the directory structure, the image contains other pieces of metadata such as the expected fsverity hashes of the referenced files.

Integrity and authenticity guarantees

In Torizon OS, the system is set up to ensure the measured file hashes match those inside the composefs image, which guarantees that any changes to the individual files in the OSTree CAS would be detected at runtime (when the file is accessed). Moreover, the composefs image itself is also protected by fsverity, which ensures that no change can be made to the directory structure or any metadata of the deployment. Finally, the expected fsverity hash of the image is signed and both the hash and signature are stored as OSTree commit metadata which are checked at runtime with a public key stored in the ramdisk (which is inside a signed FIT image along with the kernel). This effectively extends the Chain of Trust, initially covering the bootloader, kernel and ramdisk, into the root filesystem.

Management of the /etc directory

With plain OSTree in Torizon OS:

The root filesystem is read-only (though immutability is not enforced) but some directories are made writable to allow the storage of persistent runtime data. In particular, two important directories are writable: /var which contains data completely outside the control of OSTree and /etc which contains critical system-wide configuration files and scripts managed by OSTree. The way OSTree handles the latter is quite special in that:

• The directory is populated by making copies rather than hard links to files in the OSTree CAS which allows them to be modified locally without corrupting the CAS.
• During system updates, the files in the directory undergo a 3-way merge where each file is updated to the new version only if it was not modified locally. To determine if a given file was modified, OSTree compares it with a pristine version which is always available inside the /usr/etc directory of a deployment. More detail about how OSTree handles deployments is available in the OSTree documentation.

When OSTree is used with composefs enabled in Torizon OS:

The root filesystem is read-only and that condition is enforced at runtime by composefs. As with plain OSTree, the /var directory is made writable to allow the storage of data outside the control of OSTree. However, unlike the plain case, the /etc directory is mounted as an overlay. The overlay is configured in such a way that its base contents come from /usr/etc (the pristine version of the whole directory) and any changes are kept in memory only. This leads to two important differences when compared with plain OSTree:

• There is no 3-way merge of the /etc directory (on system updates).
• The contents of /etc are volatile: although the directory is writable and changes can be made at runtime, they will be discarded upon reboot, resetting the /etc directory to its pristine version.

These behavioral changes are crucial for security. The point of protecting the integrity of the root filesystem is to provide extra protection against attackers who manage to gain unauthorized access to the system, escalate privileges, and execute arbitrary code. With a read-only, integrity-protected filesystem, anything such an attacker can do would be wiped away upon reboot, and so the attack can be mitigated by issuing a software update that patches the initial bug or vulnerability.

If we were to combine a read-only, integrity-protected root filesystem with a writable, persistent /etc directory, it would defeat that purpose: an attacker would have many vectors for persistence, like adding a new systemd service that ran a malware downloader on every reboot, or modifying a service configuration that allowed code execution from non-protected locations. This would effectively be a backdoor compromising the Secure Boot Chain of Trust.

However, it can also be impractical to have a completely read-only /etc directory. There are libraries and utilities that expect to be able to write to /etc, and many of them have the locations in /etc hard-coded. That is why we deal with this scenario by making /etc volatile: it doesn't break the assumptions of existing software, but it also maintains the necessary security guarantees for a persistent Secure Boot Chain of Trust.

Still, it is often the case that some persistent changes to /etc are necessary for the needs of the product. User-configurable or environment-specific settings are two common examples. With an integrity-protected rootfs and a volatile /etc, developers must selectively modify the system configuration to ensure persistence across reboot of only the specific files and configurations they need. The practical aspects of this were explained on section Dealing with a volatile /etc directory.