PREEMPT_RT transforms the Linux kernel into a real-time operating system, ensuring precise timing for critical tasks. Fully integrated into Linux 6.12 (released September 20, 2024), it enables real-time capabilities for architectures like x86, ARM64, and RISC-V. Here’s what you need to know:

• Key Features:
  • Replaces traditional spinlocks with preemptible sleeping locks.
  • Converts interrupt handlers into threads, making them schedulable and preemptible.
  • Implements priority inheritance to address priority inversion issues.
  • Makes RCU (Read-Copy-Update) operations fully preemptible.
• Applications:
  • Used in industries like automotive, robotics, telecommunications, and medical devices where precise timing is critical.
  • Powers systems like industrial automation tools, flight controls, and hosting environments requiring low-latency virtual private servers.
• Setup:
  • Enable CONFIG_PREEMPT_RT in the kernel configuration.
  • Fine-tune settings like CONFIG_NO_HZ_FULL and CONFIG_RCU_BOOST for optimal performance.
  • Use tools like cyclictest to measure latency and validate performance.

PREEMPT_RT prioritizes timing over throughput, making Linux suitable for applications where deadlines are non-negotiable. It’s a game-changer for industries demanding deterministic performance.

PREEMPT_RT Explained: Build & Optimize Linux Kernel for Real-Time and Ultra Low Latency

sbb-itb-59e1987

PREEMPT_RT Core Features

PREEMPT_RT focuses on four primary features designed to reduce non-preemptible code and improve control over task scheduling. Here’s a closer look at each.

Full Kernel Preemption

One of the biggest updates is transforming standard spinlocks (spinlock_t) and reader-writer locks (rwlock_t) into mutex-based sleeping spinlocks. Traditional spinlocks can cause delays because they disable preemption, forcing tasks into busy-waiting. PREEMPT_RT changes this by introducing locks that allow tasks to sleep and be preempted, even when holding a resource.

This means high-priority tasks can interrupt lower-priority ones, even if the lower-priority tasks are holding locks. However, for certain critical operations – like the scheduler or hardware entry points – PREEMPT_RT retains raw_spinlock_t, which behaves like the original non-preemptible locks. As Paul McKenney, a Distinguished Engineer, puts it:

"The key point of the PREEMPT_RT patch is to minimize the amount of kernel code that is non-preemptible, while also minimizing the amount of code that must be changed in order to provide this added preemptibility."

Next, let’s see how interrupt handling benefits from this approach.

Threaded Interrupts

PREEMPT_RT shifts most hardware interrupt handlers from the "hard IRQ" context into kernel threads running in process context. This adjustment allows interrupt handlers to be prioritized, preempted, or even blocked.

In a standard Linux kernel, a long-running interrupt handler can lead to unbounded latency because it halts all other execution. Threaded interrupts solve this issue. By default, these interrupt threads run with a SCHED_FIFO priority of 50, but administrators can tweak their priorities using tools like chrt. For instance, you could prioritize a network card’s interrupt for industrial control while lowering the priority for disk I/O. Since these threads use sleeping spinlocks instead of raw spinlocks, they avoid the need to disable hardware interrupts while holding a lock.

Now, let’s explore how PREEMPT_RT handles priority-related challenges.

Priority Inheritance and Rtmutex

Priority inversion is a major issue where a high-priority task gets stuck waiting for a resource held by a low-priority task, while a medium-priority task (which doesn’t need the resource) preempts the low-priority one. PREEMPT_RT addresses this with priority inheritance, temporarily boosting the low-priority task’s priority to match the highest-priority task waiting for the resource.

The rtmutex primitive is the tool that makes this possible. It ensures that when a conflict arises, the low-priority task gets a priority boost to finish its critical section without interference from medium-priority tasks. If the boosted task gets blocked on another lock, the priority boost cascades down the dependency chain. As the Linux kernel documentation explains:

"Priority inheritance allows well-designed applications to use userspace locks in critical parts of an high priority thread, without losing determinism."

To achieve this, rtmutex uses a flag and a priority-ordered tree to manage waiting tasks, keeping overhead low on supported architectures.

Finally, let’s look at how PREEMPT_RT improves RCU operations.

Preemptible RCU (Read-Copy-Update)

Read-Copy-Update (RCU) is a synchronization technique used extensively in the Linux kernel. In standard kernels, RCU read sections are non-preemptible, which can lead to unpredictable delays. PREEMPT_RT changes this by making RCU read sections fully preemptible, ensuring that real-time task deadlines are not compromised. This adjustment is a cornerstone of achieving the predictable behavior required in real-time systems.

How to Configure and Use PREEMPT_RT

PREEMPT_RT is fully integrated into mainline Linux kernels, which means external patches are no longer necessary. However, applying the latest patch queue is still a good idea for better architecture support and improved graphics. Once the kernel is ready, you’ll need to tweak its settings to take full advantage of its capabilities.

Kernel Configuration Settings

To enable a fully preemptible kernel, turn on CONFIG_PREEMPT_RT. In newer kernels, this setting is located under "General Setup", but you may need to enable CONFIG_EXPERT first to make it visible in the configuration menu.

For production dedicated servers and other high-performance environments, additional options can further optimize performance:

• CONFIG_NO_HZ_FULL: Reduces scheduling-clock interrupts on busy CPUs, which helps minimize jitter.
• CONFIG_RCU_BOOST: Prevents preempted RCU readers from delaying grace periods.
• CONFIG_RCU_NOCB_CPU: Offloads RCU callback handling to specific CPUs, reducing interference with real-time tasks.

It’s also critical to disable debugging options that can introduce high latency. Turn off settings like CONFIG_DEBUG_LOCKDEP, CONFIG_DEBUG_PREEMPT, CONFIG_DEBUG_OBJECTS, and CONFIG_SLUB_DEBUG. While these tools are useful for development, they can significantly hurt latency targets. As kernelconfig.io explains:

"This option turns the kernel into a real-time kernel by replacing various locking primitives (spinlocks, rwlocks, etc.) with preemptible priority-inheritance aware variants."

Once your kernel is built and booted, confirm that PREEMPT_RT is active by running cat /sys/kernel/realtime. A return value of 1 indicates success. You can also verify by checking for "PREEMPT_RT" in the output of uname -a.

Debugging and Performance Optimization

Fine-tuning the kernel is essential for achieving optimal performance with real-time workloads. One key area to address is the real-time throttling mechanism, which reserves 50ms per second for non-real-time tasks by default. If your workload is purely real-time, you can disable this mechanism by writing -1 to /proc/sys/kernel/sched_rt_runtime_us. As Jan Altenberg, Senior Open Source Consultant at OSADL, points out:

"A ‘runaway’ real-time task can starve the system. As a protection mechanism the runtime of real-time tasks can be limited by setting a value in microseconds in /proc/sys/kernel/sched_rt_runtime_us."

For enhanced determinism, isolate specific CPU cores using parameters like isolcpus=2,3, rcu_nocbs=2,3, nohz_full=2,3, and set irqaffinity=0. This reserves those cores exclusively for real-time tasks.

To manage interrupt threads, use the chrt tool. These threads typically run with a default SCHED_FIFO priority of 50, but you can adjust them to avoid conflicts with your application. For example, to set a network card’s IRQ thread to priority 98, use the command: chrt -p -f 98 <PID>.

After completing the configuration, it’s crucial to test and validate latency performance. Tools like cyclictest can measure latency (e.g., cyclictest -S -m -p98 -i250), while rtla (Real-time Linux Analysis) helps identify and analyze latency spikes. These tools ensure your setup meets the demands of real-time applications.

PREEMPT_RT Applications and Benefits

Real-Time Application Use Cases

PREEMPT_RT shines in systems where timing precision is non-negotiable. In aerospace, it ensures navigation and flight controls operate without microsecond delays, which could otherwise jeopardize safety. Telecommunications companies count on it to manage data routing in real time, ensuring smooth voice and video calls by eliminating jitter and latency spikes.

In the automotive industry, it’s a cornerstone for vehicle control systems, with major players like Continental Automotive supporting its adoption. In industrial automation, PREEMPT_RT powers Programmable Logic Controllers (PLCs) and SCADA systems, often working alongside protocols like EtherCAT to maintain safety and efficiency on factory floors. Robotics also heavily relies on this technology for precise actuator control and instant sensor feedback, enabling robots to respond immediately to environmental changes.

Perhaps the most critical applications are in medical devices, where precision is paramount. From patient monitors to surgical robots, PREEMPT_RT provides the deterministic performance needed to ensure life-critical systems function flawlessly.

Benefits for Hosting Environments

The advantages of PREEMPT_RT extend into hosting environments, where real-time responsiveness is just as crucial. For VPS and dedicated servers, it minimizes latency by allowing the scheduler to prioritize high-priority tasks over less critical ones. This ensures consistent response times, which directly impact user experience and service reliability.

The threaded interrupt model prevents "interrupt storms" from overwhelming systems during heavy I/O operations. With rt_mutex, priority inheritance ensures that low-priority background tasks don’t block critical hosting services. High-resolution timers enable microsecond-level scheduling accuracy, reducing jitter in virtualized setups. Hosting providers, like Serverion, offer custom kernel builds with PREEMPT_RT, giving administrators the flexibility to fine-tune configurations for specific workloads. By using parameters like isolcpus and irqaffinity, providers can dedicate CPU cores to performance-critical tasks while keeping routine system operations isolated.

Conclusion

This guide has delved into how PREEMPT_RT turns Linux into a real-time operating system by giving the scheduler near-complete control. Since its inclusion in mainline kernels starting with Linux 6.12 (September 2024), it has eliminated the need for external patches across architectures like x86, ARM64, and RISC-V.

The concept is straightforward: reduce non-preemptible code as much as possible. By converting spinlocks into sleeping locks and running interrupt handlers as threads, high-priority tasks can preempt nearly all kernel activities. Features like priority inheritance prevent low-priority tasks from delaying critical operations, while preemptible RCU ensures even read-side critical sections don’t cause significant delays. As Sebastian Siewior, the PREEMPT_RT maintainer, aptly states:

"All control to the scheduler."

This technical leap offers practical benefits. For instance, Serverion uses custom PREEMPT_RT kernel builds to fine-tune CPU isolation and optimize real-time workloads, ensuring stable response times even under intense I/O pressure.

What sets real-time systems apart isn’t just speed – it’s predictability. PREEMPT_RT minimizes jitter, ensuring tasks execute precisely when needed. This is crucial for applications like industrial automation, telecommunications, and performance-critical hosting services. It brings the kind of deterministic behavior that standard kernels simply can’t provide.

With its integration into the mainline kernel and backing from enterprise distributions like Ubuntu Pro (since February 2023), PREEMPT_RT has become more accessible for hosting providers and system administrators. It delivers the reliable, low-latency performance required for today’s most demanding and time-sensitive environments.

FAQs

Do I need PREEMPT_RT or is standard Linux enough?

The standard Linux kernel, when combined with the PREEMPT_RT patch, gains real-time capabilities. This makes it a strong choice for applications where low latency and consistent response times are critical. On its own, however, the mainline Linux kernel might fall short of meeting strict real-time demands.

Will PREEMPT_RT reduce overall throughput on my system?

To achieve real-time performance, PREEMPT_RT focuses on reducing non-preemptible kernel code, which helps improve system responsiveness. This approach may lead to a slight decrease in overall throughput, but it ensures consistent and predictable behavior – critical for time-sensitive applications. This trade-off is intentional and central to meeting the demands of real-time systems.

What latency target is realistic with PREEMPT_RT?

With PREEMPT_RT, achieving latency in the millisecond range is a realistic goal for both desktop and embedded systems. However, the actual performance heavily depends on how well the system is configured and tuned to meet specific requirements. Proper setup is key to ensuring the desired level of responsiveness.

Related Blog Posts

• Real-Time Anomaly Detection for AI Workloads
• Latency Testing in Failover Systems: Key Metrics
• Ultimate Guide to Real-Time Cloud Storage Monitoring
• How to Optimize Web Servers for Low Latency