In this article, you can learn more about real-time solutions on Linux. Most of the focus beyond the introduction is on the soft real-time PREEMPT_RT patch.
There are multiple options for using Linux in a real-time, deterministic, low-latency application.
There are advantages & disadvantages to all of these approaches. Toradex provides official support for Linux on:
Paid support for these alternative solutions may be available from companies in the Toradex partner network. Additionally, information about the real-time capabilities of Linux itself, including the PREEMPT_RT patch set, is provided below.
The BSP Layers and Reference Images for Yocto Project and TorizonCore distributions with the PREEMPT_RT patch, as well as their pre-built images available on our Toradex Easy Installer online feeds or for offline installation, haven't been fully validated by Toradex as of February, 2021. We have only run smoke tests and cyclictest
on TorizonCore to make sure that the system boots properly and the PREEMPT_RT patches are working under stressful conditions. You can find the tests, including the stress conditions, on our GitHub repository rt-validation.
Any tuning, additional kernel configuration, and validation of the RT performance for your own use case must be done at your discretion. The contents provided in this article are guidelines to help you and it doesn't mean that any specific use case or features are supported or validated by Toradex at the moment. If you have questions or want to share your use case with us, so we can have a more accurate understanding of customers' needs, please contact us.
Get an overview of different options and get see some test results.
Linux has been built as a general purpose operating system and therefore is, by design, not well optimized for real-time operations. However, since desktop applications such as audio or video playback require a certain real-time capability, the standard kernel today provides rather impressive (soft) real-time capabilities. The main knob to improve the kernels latency is the Preemption Model (available in the submenu Kernel Features of the kernel configuration). A modern vanilla Linux kernel offers three preemption options:
The PREEMPT_RT kernel (see below) adds additional options, notably:
You can use /proc/config.gz to access the kernel configuration of the running kernel:
root@vybridvf61-v12:~# zcat /proc/config.gz | grep PREEMPT # CONFIGPREEMPTNONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIGPREEMPT is not set
To provide real-time capabilities to a process or thread the priority and scheduler options need to be set. From within an application the standard Linux system call sched_setscheduler can be used. For a code example refer to the Hello World example on the Real-Time Linux Wiki at kernel.org. The utility chrt (part of util-linux) can be used to configure a process' priority and scheduler after start directly from the shell.
The command ps can be used to verify a process'/thread's current priority and scheduler settings:
root@vybridvf61-v12:~# ps -eLfc | grep cyclictest root 3825 3816 3825 2 TS 19 21:47 pts/0 00:00:00 cyclictest -n -p 60 root 3825 3816 3826 2 FF 100 21:47 pts/0 00:00:00 cyclictest -n -p 60 root 3828 439 3828 1 TS 19 21:47 ttyLP0 00:00:00 grep cyclictest
One of the contributors to latency in the Voluntary Preemption and Preemptible Kernel models are console messages written to a serial console. Serial console messages may be minimized by using the kernel argument quiet. To suppress messages after boot, use the following line to only enable emergency messages:
echo 1 > /proc/sys/kernel/printk
The configuration file /etc/sysctl.conf provides a means of configuring the log level applied automatically after boot:
kernel.printk = 1 4 1 7
Disabling memory overcommiting to avoid latency generated by the Out-of-Memory Killer is a worthwhile option too:
echo 2 > /proc/sys/vm/overcommit_memory
Using the configuration file /etc/sysctl.conf:
vm.overcommit_memory = 2
See also: http://linuxrealtime.org/index.php/Basic_Linux_from_a_Real-Time_Perspective#Memory_Overcommit
General information regarding PREEMPT_RT may be found at the Real-Time Linux Wiki at kernel.org. The PREEMPT_RT kernel patches can be downloaded directly from kernel.org.
Note: The mainline PREEMPT_RT patches are specifically written for the vanilla mainline Linux kernel. The Toradex BSPs may provide drivers which are not part of the vanilla Linux kernel and therefore do not get patched by the PREEMPT_RT patch set.
To apply these patches to the kernel, follow the directions to Build U-Boot and Linux Kernel from Source Code. After cloning the appropriate kernel branch, the real-time patches may be applied. Be sure to use the patches for the nearest kernel version.
Note: In most cases the patches will fail to apply cleanly. As such, it may be necessary to fix patch errors by manually resolving discrepancies between the patches and the source code.
After applying the patches and executing "make {defconfig target}", the kernel config must be modified to enable CONFIG_PREEMPT_RT. "make menuconfig" or "make nconfig" can be used to make these edits to the kernel configuration. Also ensure that the high resolution timer is enabled. Also note that some configurations and drivers may adversely affect real-time performance. For more general information about patch application and kernel configuration, see the "How-To" at the Real-Time Linux Wiki at linuxfoundation.org.
Toradex provides a real-time kernel recipe for the i.MX6 & i.MX7 for use in an OpenEmbedded-Core (Yocto Project) build. The linux-toradex-rt recipe may be found in the meta-toradex-nxp layer.
To select the real-time kernel as the preferred kernel in a build, set the preferred provider variable for virtual/kernel in an appropriate configuration file (e.g. conf/local.conf
):
PREFERRED_PROVIDER_virtual/kernel = "linux-toradex-rt"
Starting with the 3.0 BSP Toradex has own distro in meta-toradex-distro layer. To use real-time kernel, set the distro for i.MX8 in configuration file (e.g. conf/local.conf
):
DISTRO_mx8 = "tdx-xwayland-rt"
Note: It may be advantageous to disable display/graphics related drivers - namely FB, DRM & Vivante drivers. Power management or CPU governing may also present issues.
The following tests have been made using the cyclictest utility (part of rt-tests package). Cyclictest was run for 30 minutes to measure system response latency. Ideally, these tests should be run for a much longer period of time (24 hr vs 30 min).
These tests were performed on the Apalis iMX6Q 1GB SoM using the 4.9.84 kernel from Toradex patched with PREEMPT_RT and graphics drivers disabled (as described above). During these tests, a set of stress scripts were executed to randomly exercise various interrupts while fully loading all CPU cores. Then cyclictest was run with 4 threads (1 for each core). Inspection revealed that the 4 threads were evenly distributed across the 4 cores, but ideally this should be enforced by the scheduler.
Below are the test results for the three different preemption configurations - all with Vivante-related configs disabled on the console-only rootfs while running the stress scripts described above.
i.MX6 4.9.84 linux-toradex Voluntary Kernel Preemption (CONFIG_PREEMPT_VOLUNTARY=y):
# uname -a
Linux apalis-imx6 4.9.84-2.8.2+gb2a7f2f #1 SMP Mon Apr 2 16:25:31 PDT 2018 armv7l GNU/Linux
# cyclictest --smp -m -p 80 -D60m
policy: fifo: loadavg: 19.32 14.73 7.24 19/314 11152
T: 0 ( 852) P:80 I:1000 C: 386868 Min: 9 Act: 815 Avg: 381 Max: 180105333
T: 1 ( 853) P:80 I:1500 C: 283180 Min: 10 Act: 19 Avg: 96 Max: 7908
T: 2 ( 856) P:80 I:2000 C: 212767 Min: 10 Act: 44 Avg: 130 Max: 8056
T: 3 ( 857) P:80 I:2500 C: 170268 Min: 11 Act: 68 Avg: 88 Max: 6200
i.MX6 4.9.84 linux-toradex Preemptible Kernel (CONFIG_PREEMPT=y):
# uname -a
Linux apalis-imx6 4.9.84-2.8.2+gb2a7f2f #1 SMP Mon Apr 2 17:47:14 PDT 2018 armv7l GNU/Linux
# cyclictest --smp -m -p 80 -D60m
policy: fifo: loadavg: 23.27 19.79 19.62 10/308 23113
T: 0 (30542) P:80 I:1000 C:1257523 Min: 11 Act: 28 Avg: 50 Max: 1286
T: 1 (30543) P:80 I:1500 C: 838377 Min: 12 Act: 46 Avg: 35 Max: 658
T: 2 (30544) P:80 I:2000 C: 628765 Min: 11 Act: 41 Avg: 39 Max: 875
T: 3 (30545) P:80 I:2500 C: 503002 Min: 12 Act: 29 Avg: 40 Max: 924
i.MX6 4.9.84 linux-toradex-rt Fully Preemptible Kernel (CONFIG_FULL_RT=y):
# uname -a
Linux apalis-imx6 4.9.84-rt62+gb2a7f2f #1 SMP PREEMPT RT Mon Apr 2 14:24:06 PDT 2018 armv7l GNU/Linux
# cyclictest --smp -m -p 80 -D60m
policy: fifo: loadavg: 15.72 18.24 12.28 12/289 17867
T: 0 ( 1113) P:80 I:1000 C: 853649 Min: 10 Act: 22 Avg: 30 Max: 163
T: 1 ( 1115) P:80 I:1500 C: 569078 Min: 10 Act: 20 Avg: 28 Max: 125
T: 2 ( 1116) P:80 I:2000 C: 426792 Min: 10 Act: 24 Avg: 30 Max: 131
T: 3 ( 1118) P:80 I:2500 C: 341421 Min: 11 Act: 24 Avg: 29 Max: 128
Note that the worst case latency on the kernel with real-time preemption has a latency of <200us.
The following tests were performed on the Colibri iMX7D 512MB SoM locked at 792MHz using the 4.9.84 kernel from Toradex.
i.MX7 4.9.84 linux-toradex Voluntary Kernel Preemption (CONFIG_PREEMPT_VOLUNTARY=y):
# uname -a
Linux colibri-imx7 4.9.84-2.8.2+gb2a7f2f #1 SMP Fri Mar 30 13:55:10 PDT 2018 armv7l GNU/Linux
# cyclictest --smp -m -p 80 -D60m
policy: fifo: loadavg: 8.45 9.60 9.96 9/177 1685
T: 0 ( 670) P:80 I:1000 C:3598421 Min: 8 Act: 15 Avg: 51 Max: 1740
T: 1 ( 671) P:80 I:1500 C:2399989 Min: 8 Act: 34 Avg: 46 Max: 1684
i.MX7 4.9.84 linux-toradex-rt Fully Preemptible Kernel (CONFIG_FULL_RT=y):
# uname -a
Linux colibri-imx7 4.9.84-rt62+gb2a7f2f #1 SMP PREEMPT RT Mon Apr 2 15:40:10 PDT 2018 armv7l GNU/Linux
# cyclictest --smp -m -p 80 -D60m
policy: fifo: loadavg: 10.79 8.72 4.81 10/209 10398
T: 0 ( 642) P:80 I:1000 C: 552900 Min: 10 Act: 23 Avg: 29 Max: 122
T: 1 ( 643) P:80 I:1500 C: 368584 Min: 11 Act: 28 Avg: 26 Max: 110
The following tests were performed on the Colibri VF61 256MB SoM using the 4.4.5 kernel from Toradex patched with PREEMPT_RT.
Vybrid 4.4.5 linux-toradex Voluntary Kernel Preemption:
~# uname -a
Linux vybridvf61-v12 4.4.5-v2.6b1+g88c66ff #1 Wed Mar 30 11:57:03 PDT 2016 armv7l GNU/Linux
~# cyclictest -n -p 80 -t -D30m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.32 0.19 0.15 1/102 951
T: 0 ( 580) P:80 I:1000 C: 554804 Min: 14 Act: 26 Avg: 32 Max: 11425
Vybrid 4.4.5 linux-toradex Preemptible Kernel:
~# cyclictest -n -p 80 -t -D30m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.01 0.05 0.07 1/100 646
T: 0 ( 451) P:80 I:1000 C:1799978 Min: 15 Act: 29 Avg: 29 Max: 5368
Vybrid 4.4.5 linux-toradex Fully Preemptible Kernel (RT):
~# uname -a
Linux vybridvf61-v12 4.4.5-rt12-00145-g88c66ff-dirty #1010 PREEMPT RT Fri Apr 1 16:08:10 PDT 2016 armv7l GNU/Linux
~# cyclictest -n -p 80 -t -D30m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 4.07 4.11 3.51 4/103 3044
T: 0 ( 575) P:80 I:1000 C:1799993 Min: 20 Act: 41 Avg: 38 Max: 96