The Curious Case of a Missing CPUID Flag

Recently I was referred to a bug in one of our SLE kernels that manifested in the following warning being printed in dmesg, on a virtualized guest:

Subsequently it was reported that the same warning manifests on the host machine, where the guest VM was run. What those warnings actually mean is that the Speculative Return Stack Overflow (SRSO) mitigation might not be complete. The largest implication is for userspace tasks since “Safe RET” mitigation protects the kernel, but userspace tasks also require the application of the respective microcode, which extends the functionality of IBPB. Whether this concerns a particular user should be judged on a case-by-case basis, based on the threat scenario that people/organizations want handled. 

My first reaction was to consider that the microcode had indeed not been applied to the affected machine. It was an Epyc 7xx3 (aka an AMD Milan-based Zen 3 core), and based on Amd’s own security bulletin, those cores require a microcode update. However, after looking at the provided logs, it became apparent that the correct microcode is indeed being applied:

However, one thing that struck me as odd was the timestamps for each particular event. Based on them, the mitigation-related code seems to run way before the microcode actually gets loaded, at least based on the logs themselves. Or does it?

A slight detour into the microcode-loading world

In order to fully understand why the above assumption was false (i.e., microcode was indeed loaded at the correct time), one has to know how it’s actually loaded. Essentially, when a modern x86 CPU, irrespective of the manufacturer (Intel or AMD), boots, it has some baked-in version of the microcode or the firmware of the CPU. It dictates how instructions are being executed and their semantics. This level of indirection allows vendors to issue updated versions of the microcode if bugs are found in the way the CPU executes certain instructions. CPUs provide an interface that allows external parties to load microcode binaries provided by the respective vendor. The best time to load a microcode is as early as possible as this reduces the window in which a bug could manifest. This means that the microcode can either be loaded very early by the BIOS of the system or, alternatively, early in the boot process of the operating system.

Since Suse is a Linux distribution vendor, I’ll focus on the OS-based loading mechanism. In order to load the microcode as early as possible, Linux incorporates the so-called “early loader” that is tasked with loading a provided microcode update in initramfs. This early loader runs during kernel initialization and, more specifically, during CPU bring-up. During this phase, the CPU is still being configured, and since no kernel services can run, logging also doesn’t work naturally. This means that the microcode is loaded way earlier than we get a printout of it in dmesg.

Armed with this knowledge now, it became apparent that as far as the microcode is related, we were in the clear. Then I went on to compare our Suse-specific code with the upstream version, and it became apparent that there were differences in the logic behind printing the warning on the host. Once those were fixed, the warning was no longer printed on the host, but on the guest, it still continued. What’s more, Suse’s QA team also verified that the guest issue persisted when an upstream kernel was being run on the host.

At this point, we had fixed one of the two issues. In part two of this blog, I will explain the steps we took to isolate the issue and uncover the true cause of the guest not recognizing that the firmware was updated.