Search by Tags

Real-Time Linux

 

Compare with Revision




Subscribe for this article updates

Real-time Linux Solutions

There are multiple options for using Linux in a real-time, deterministic, low-latency application. One option is to use Linux in-conjunction with a hypervisor or co-kernel. In these cases, the hypervisor or co-kernel has priority over the Linux kernel and is where the real-time tasks are performed. Another similar option is to use a heterogeneous asymmetric multi-core system where Linux and a different deterministic kernel run independently on separate processor cores. Finally, the popular strategy of improving the real-time capability of Linux by making it more preemptive may be used. This is most commonly achieved by applying a patch set known as PREEMPT_RT to the kernel.

There are advantages & disadvantages to all of these approaches. The only option mentioned above that Toradex provides official support for is the use of Linux with our System-on-Modules featuring a heterogeneous asymmetric multi-core architecture allowing an RTOS such as FreeRTOS to execute in parallel with Linux on separate processor cores. However, paid support for these alternative solutions may be available from companies in the Toradex partner network. Additionally, information about the real-time capabilities of Linux itself, including the PREEMPT_RT patch set, is provided below. The PREEMPT_RT patched kernels referenced by this article are provided as-is.

Real-time Linux Webinar

Get an overview of differnt options and get see some test results.

  • Real-Time Patch, Real-Time Linux
  • Xenomai
  • Heterogeneous multicore processing with i.MX 7

Linux Real-time Capabilities

Linux has been built as a general purpose operating system and therefore is, by design, not well optimized for real-time operations. However, since desktop applications such as audio or video playback require a certain real-time capability, the standard kernel today provides rather impressive (soft) real-time capabilities. The main knob to improve the kernels latency is the Preemption Model (available in the submenu Kernel Features of the kernel configuration). A modern vanilla Linux kernel offers three preemption options:

  • CONFIG_PREEMPT_NONE: No Forced Preemption (Server)
  • CONFIG_PREEMPT_VOLUNTARY: Voluntary Kernel Preemption (Desktop)
  • CONFIG_PREEMPT: Preemptible Kernel (Low-Latency Desktop)

The PREEMPT_RT kernel (see below) adds additional options, notably:

  • PREEMPT_RT_FULL: Fully Preemptible Kernel (RT)

You can use /proc/config.gz to access the kernel configuration of the running kernel:

root@vybridvf61-v12:~# zcat /proc/config.gz | grep PREEMPT
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set

Scheduler and Priorities

To provide real-time capabilities to a process or thread the priority and scheduler options need to be set. From within an application the standard Linux system call sched_setscheduler can be used. For a code example refer to the Hello World example on the Real-Time Linux Wiki at kernel.org. The utility chrt (part of util-linux) can be used to configure a process' priority and scheduler after start directly from the shell.

The command ps can be used to verify a process'/thread's current priority and scheduler settings:

root@vybridvf61-v12:~# ps -eLfc | grep cyclictest
root      3825  3816  3825    2 TS   19 21:47 pts/0    00:00:00 cyclictest -n -p 60
root      3825  3816  3826    2 FF  100 21:47 pts/0    00:00:00 cyclictest -n -p 60
root      3828   439  3828    1 TS   19 21:47 ttyLP0   00:00:00 grep cyclictest

Real-time Optimization

One of the contributors to latency in the Voluntary Preemption and Preemptible Kernel models are console messages written to a serial console. Serial console messages may be minimized by using the kernel argument quiet. To suppress messages after boot, use the following line to only enable emergency messages:

echo 1 > /proc/sys/kernel/printk

The configuration file /etc/sysctl.conf provides a means of configuring the log level applied automatically after boot:

kernel.printk = 1 4 1 7

Disabling memory overcommiting to avoid latency generated by the Out-of-Memory Killer is a worthwhile option too:

echo 2 > /proc/sys/vm/overcommit_memory

Using the configuration file /etc/sysctl.conf:

vm.overcommit_memory = 2

See also: http://linuxrealtime.org/index.php/Basic_Linux_from_a_Real-Time_Perspective#Memory_Overcommit

PREEMPT_RT Patch Set

General information regarding PREEMPT_RT may be found at the Real-Time Linux Wiki at kernel.org. The PREEMPT_RT kernel patches can be downloaded directly from kernel.org.

Note: The mainline PREEMPT_RT patches are specifically written for the vanilla mainline Linux kernel. The Toradex BSPs may provide drivers which are not part of the vanilla Linux kernel and therefore do not get patched by the PREEMPT_RT patch set.

To apply these patches to the kernel, follow the directions to Build U-Boot and Linux Kernel from Source Code. After cloning the appropriate kernel branch, the real-time patches may be applied. Be sure to use the patches for the nearest kernel version.

Note: In most cases the patches will fail to apply cleanly. As such, it may be necessary to fix patch errors by manually resolving discrepancies between the patches and the source code.

After applying the patches and executing "make {defconfig target}", the kernel config must be modified to enable CONFIG_PREEMPT_RT. "make menuconfig" or "make nconfig" can be used to make these edits to the kernel configuration. Also ensure that the high resolution timer is enabled. Also note that some configurations and drivers may adversely affect real-time performance. For more general information about patch application and kernel configuration, see the "How-To" at the Real-Time Linux Wiki at kernel.org.

OE-Core (Yocto Project) Kernel Recipe

i.MX6 & i.MX7 based modules

Toradex provides a real-time kernel recipe for the i.MX6 & i.MX7 for use in an OpenEmbedded-Core (Yocto Project) build. The linux-toradex-rt recipe may be found in the meta-toradex layer.

To select the real-time kernel as the preferred kernel in a build, set the preferred provider variable for virtual/kernel in an appropriate configuration file (e.g. conf/local.conf):

PREFERRED_PROVIDER_virtual/kernel = "linux-toradex-rt"

Note: It may be advantageous to disable display/graphics related drivers - namely FB, DRM & Vivante drivers. Power management or CPU governing may also present issues.

Installation via Package Manager

i.MX6 & i.MX7 based modules

Toradex provides a compiled 3.14.28 i.MX6 kernel in its package feeds. This kernel is for evaluation use only and is not officially supported by Toradex. To install this kernel from the package feeds, update the module to the latest V2.5 beta Linux image version, boot the module into Linux and follow these steps.

Download additional repo entries to the package manager:

wget –P /etc/opkg/ http://feeds.toradex.com/extra-kernel/2.5/conf/all/extra-kernel.conf

Update the package manager's package cache:

opkg update

Download and execute an installation script to install and configure the real-time kernel:

cd
wget http://feeds.toradex.com/extra-kernel/2.5/conf/colibri-imx6/install-rt-kernel.sh
./install-rt-kernel.sh

Reset the system:

reboot

The module should then boot the real-time kernel.

Test Results

The following tests have been made using the cyclictest utility (part of rt-tests package). Cyclictest was run for 30 minutes to measure system response latency. Ideally, these tests should be run for a much longer period of time (24 hr vs 30 min).

i.MX6 based modules

These tests were performed on the Apalis iMX6Q 1GB SoM using the 3.14.28 kernel from Toradex patched with PREEMPT_RT and display/graphics drivers disabled (as described above). CPU frequency was set to 800MHz. During these tests, a set of stress scripts were executed to randomly exercise various interrupts while fully loading all CPU cores. Then cyclictest was run with 4 threads (1 for each core). Inspection revealed that the 4 threads were evenly distributed across the 4 cores, but ideally this should be enforced by the scheduler.

“Scheduling while atomic” bugs are logged by the kernel - so even with display/graphics removed, some improperly patched parts of the kernel remain. Below are the test results for the three different preemption configurations - all with FB/DRM/Vivante configs disabled on the console-only rootfs while running the stress scripts described above.

iMX6 3.14.28 linux-toradex Voluntary Kernel Preemption:

# uname -a
Linux apalis-imx6 3.14.28-01660-g3718b8f-dirty #3 SMP Wed Sep 2 15:29:02 PDT 2015 armv7l GNU/Linux
# cyclictest -n -p 80 -t -D30m
policy: fifo: loadavg: 4.31 4.29 3.71 5/114 6286

T: 0 ( 1845) P:80 I:1000 C:1799993 Min:     10 Act:   19 Avg:   44 Max:   23350
T: 1 ( 1846) P:80 I:1500 C:1199995 Min:     10 Act:   35 Avg:   37 Max:   13460
T: 2 ( 1847) P:80 I:2000 C: 899996 Min:     10 Act:   35 Avg:   41 Max:   13631
T: 3 ( 1848) P:80 I:2500 C: 719997 Min:     11 Act:   38 Avg:   46 Max:   10856

iMX6 3.14.28 linux-toradex Preemptible Kernel:

# uname -a
Linux apalis-imx6 3.14.28 #3 SMP PREEMPT Wed Sep 2 11:31:11 PDT 2015 armv7l GNU/Linux
# cyclictest -n -p 80 -t -D30m
policy: fifo: loadavg: 4.98 5.07 4.37 5/115 11803           

T: 0 ( 2133) P:80 I:1000 C:1799998 Min:     11 Act:   29 Avg:   35 Max:     821
T: 1 ( 2134) P:80 I:1500 C:1199997 Min:     11 Act:   35 Avg:   40 Max:     597
T: 2 ( 2135) P:80 I:2000 C: 899998 Min:     12 Act:   37 Avg:   44 Max:     886
T: 3 ( 2136) P:80 I:2500 C: 719998 Min:     12 Act:   60 Avg:   42 Max:     721

iMX6 3.14.28 linux-toradex Fully Preemptible Kernel (RT):

# uname -a
Linux apalis-imx6 3.14.28-rt25 #2 SMP PREEMPT RT Tue Sep 1 15:50:05 PDT 2015 armv7l GNU/Linux
# cyclictest -n -p 80 -t -D30m
policy: fifo: loadavg: 4.62 4.52 3.98 4/163 27123    

T: 0 (12907) P:80 I:1000 C:1799985 Min:     10 Act:   24 Avg:   26 Max:     166
T: 1 (12908) P:80 I:1500 C:1199994 Min:      9 Act:   27 Avg:   27 Max:     187
T: 2 (12909) P:80 I:2000 C: 899996 Min:     11 Act:   21 Avg:   25 Max:     135
T: 3 (12910) P:80 I:2500 C: 719996 Min:     10 Act:   30 Avg:   28 Max:     129

Note that worst case latency on the kernel with voluntary preemption is >20ms while the kernel with real-time preemption has worst case latency of <200us. Perhaps if the “scheduling while atomic” bugs are addressed, the preempt-rt kernel could possibly achieve even better results.

i.MX7 based modules

The following tests were performed on the Colibri iMX7D 512MB SoM locked at 792MHz using the 3.14.52 kernel from Toradex patched with PREEMPT_RT.

I.MX7 3.14.52 linux-toradex-rt Fully Preemptible Kernel (RT)

# uname -a
Linux colibri-imx7 3.14.52-rt64-v2.6b2-rt+g9ace52b #1 SMP PREEMPT RT Wed Apr 6 10:14:04 PDT 2016 armv7l GNU/Linux

# cyclictest --smp -m -p 80 -D60m
policy: fifo: loadavg: 11.58 9.86 9.95 6/181 29725

T: 0 (  775) P:80 I:1000 C:3599992 Min:      9 Act:   20 Avg:   21 Max:     102
T: 1 (  776) P:80 I:1500 C:2399976 Min:      9 Act:   27 Avg:   26 Max:      97

Vybrid based modules

The following tests were performed on the Colibri VF61 256MB SoM using the 4.4.5 kernel from Toradex patched with PREEMPT_RT.

Vybrid 4.4.5 linux-toradex Voluntary Kernel Preemption:

~# uname -a
Linux vybridvf61-v12 4.4.5-v2.6b1+g88c66ff #1 Wed Mar 30 11:57:03 PDT 2016 armv7l GNU/Linux
~# cyclictest -n -p 80 -t -D30m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.32 0.19 0.15 1/102 951           

T: 0 (  580) P:80 I:1000 C: 554804 Min:     14 Act:   26 Avg:   32 Max:   11425

Vybrid 4.4.5 linux-toradex Preemptible Kernel:

~# cyclictest -n -p 80 -t -D30m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.01 0.05 0.07 1/100 646          

T: 0 (  451) P:80 I:1000 C:1799978 Min:     15 Act:   29 Avg:   29 Max:    5368

Vybrid 4.4.5 linux-toradex Fully Preemptible Kernel (RT):

~# uname -a
Linux vybridvf61-v12 4.4.5-rt12-00145-g88c66ff-dirty #1010 PREEMPT RT Fri Apr 1 16:08:10 PDT 2016 armv7l GNU/Linux
~# cyclictest -n -p 80 -t -D30m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 4.07 4.11 3.51 4/103 3044           

T: 0 (  575) P:80 I:1000 C:1799993 Min:     20 Act:   41 Avg:   38 Max:      96