I’ve wondered for years why SCHED_IDLE tasks are allowed to preempt other tasks. Nobody seems to know, and I’ve not had the time to dig in and understand the Linux kernel scheduler better. But now I finally have my answer.

The problem

I want to be able to run a CPU-heavy task in the background, without it stealing any CPU, any at all, from other processes.

If you read various documentation, you’d be forgiven if this is as simple as schedtool -D -e ./my-background-thing.

SCHED_FIFO can be used only with static priorities higher than 0,
which means that when a SCHED_FIFO thread becomes runnable, it
will always immediately preempt any currently running SCHED_OTHER,
SCHED_BATCH, or SCHED_IDLE thread.

(and similar statements, e.g. from sched.7)

But if you try that, you’ll find that it’ll actually preempt other processes, taking little but nonzero CPU. There’s a test program, if you want to try it out.

I literally want it to take zero CPU. And not just against SCHED_FIFO/RR, but everything.

The existing settings

There appear to be two places that minmax guarantees can show up.

Settings with default values:

$ sudo sysctl -a | grep sched_rt_
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000

If CONFIG_RT_GROUP_SCHED is enabled in the kernel, then this setting is the maximum limit how much CPU the real-time (RT) classes (FIFO and RR) can take, if there are other runnable processes. If CONFIG_RT_GROUP_SCHED is not enabled, then it’s a more hazy tunable, with no preemption guarantees.

You can set kernel.sched_rt_runtime_us to 1000000, to remove this limit. (though again, it’s not actually a limit unless CONFIG_RT_GROUP_SCHED is enabled)

Actually, technically -1 removes the limit, and 1000000 just sets it to allow taking all of it. I’m not sure what the difference between unconstrained and “constrained to 100%” is. The behavior seems to be the same.

But that’s not the only limit, so even with CONFIG_RT_GROUP_SCHED, the problem is not fixed.

So that’s the maximum that a RT process can claim during contention. But it’s not the minimum that other process can take.

The non-RT processes are scheduled by the “fair scheduler”, and it gives a minimum of 50ms (50 million nanoseconds) per second that deadlined processes get to make progress in (I think I got the terminology right on that):

$ sudo grep . /sys/kernel/debug/sched/fair_server/cpu0/{runtime,period}
/sys/kernel/debug/sched/fair_server/cpu0/runtime:50000000
/sys/kernel/debug/sched/fair_server/cpu0/period:1000000000

If you set /sys/kernel/debug/sched/fair_server/cpu0/runtime to 0, then realtime processes will never be interrupted by lower priority tasks.

Note that if your kernel is in lockdown mode (from a SecureBoot), you can neither read nor write this setting.

But doing this can make things get wonky. My wifi broke while I was testing this until I killed the test program. Presumably from my test preventing even interrupt handling (or supporting kernel threads) from completing. So if running realtime classes without limits, you may want to ensure those CPUs have no IRQs assigned to them.

Maybe changing PREEMPT_RT in the kernel would fix that aspect, but I’ve not tried it.

But this is just for realtime processes. This doesn’t prevent SCHED_IDLE from preempting the default scheduling class, SCHED_OTHER. And this is the main thing I want to do.

SCHED_OTHER vs SCHED_IDLE

The scheduler weighting for SCHED_IDLE is not 0. It’s 3. The default weight is 1024. This means the supposedly IDLE tasks will always make progress. Another word for “make progress” is “selfishly steal CPU resources that could go to higher priority things”.

This is a compile time constant, and you cannot change it at runtime. It’s not even as simple as changing the constant and recompiling, because the math doesn’t work (think divide by zero).

You can also change weights between cgroups, but again not to zero:

$ grep . /sys/fs/cgroup/*/*/cpu.weight
/sys/fs/cgroup/system.slice/accounts-daemon.service/cpu.weight:100
/sys/fs/cgroup/system.slice/avahi-daemon.service/cpu.weight:100
/sys/fs/cgroup/system.slice/bluetooth.service/cpu.weight:100
[…]

If my goal is not to let the realtime classes dominate, but to prevent IDLE from taking even a pinky toe, I’ve actually not solved anything at all.

Solution: patch the kernel

I’m not a kernel programmer. But I know a robot who is. After having it confirm my understanding of the various knobs mentioned, I just said Make a patched kernel that allows 0 weighting for SCHED_IDLE against SCHED_OTHER.

It went brrr for a while, and even wrote scripts to spin up a VM in qemu to test with, and finally produced a patch that adds a scheduler feature that ensures IDLE only gets scheduled if there’s nothing non-IDLE running.

While I have done some minimal kernel coding manually, I’m not competent to evaluate the scheduler changes. But it does work. When you enable the feature, the IDLE thread instantly stops being scheduled.

echo IDLE_ZERO_WEIGHT | sudo tee /sys/kernel/debug/sched/features

This is a patch on top of kernel 6.12.111, with scripts and stuff in this repo.

A patched kernel tree in this branch, and the exact patch is this file.

Mission accomplished?

Well, in terms of understanding why it was broken, yes.

But ideally this scheduler feature should be upstreamed. I need to understand it fully before I’d ask someone else to take a look.

But if anybody reading this wants to take that on, go for it!

Maybe one day our SCHED_IDLE processes will finally live up to their promise.

Resources