Scaling Governors Can Have Bugs

All started on 27th April when my colleague Andi found a tweak that would make requests towards our Matrix API a little bit faster for matrices with only a few dozens of locations. He just had to find the specific number of locations when it made sense to use it in production.

But something was wrong. The numbers didn’t make sense. A method that took 1ms suddenly took longer when it was used in the context of a web server (Jetty or Tomcat). The overhead was significant and it was not the gzip compression or http overhead. And even worse: sometimes this strange overhead was not constant but increased when the original execution time of the method increased. This means our real matrix code would be roughly 2 times slower for certain situations.

We were able to exclude a hardware problem and our first thoughts were that it had something to do with the type of the physical RAM but this was only an unlucky coincidence as we found other servers with a different RAM vendor that showed the same problem.

Andi posted this problem on the jetty mailing list and created a new GraphHopper issue. Unfortunately none of the ideas and suggestions brought us further at that moment.

At some point we realized that a Debian 11 system behaved differently than a Debian 10 system that was upgraded and a few days later Andi noticed that installing one specific package was causing this difference. Which finally brought us to the conclusion that it must have something to do with the “scaling governor” of the AMD CPU.

What is a scaling governor?

Scaling governors are power schemes determining the desired frequency for the CPU. Some request a constant frequency, others implement algorithms to dynamically adjust according to the system load.

Arch Linux Wiki

For more details we recommend reading the kernel documentation.

And indeed, as soon as we set this governor to “performance” everything was fine:

echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

And the reason we saw this strange overhead was that the default governor “schedutil” has a bug for AMD CPUs. And the difference we saw can be explained because Debian 11 has a different default for AMD CPUs than Debian 10. We cannot yet explain why we also saw similar performance issues for other governors like “powersave”, but at least for recent kernels >= 5.15 this bug is stated to be fixed for schedutil.

The story would end here, but while our findings two GraphHopper contributors tested the benchmark code on their servers and it turned out that even Intel machines showed a similar problem and we were able to confirm this on some of our machines. And it seems that in these cases a recent kernel 5.15 does not fix the problem. We didn’t find a bug that could explain this for Intel machines, but it was also fixed when the governor was set to “performance”.

If you have more information about this matter – please post it into the mentioned issue 🙂