Live Patching and Immutable System

SUSE products offer two interesting technologies, which can be seen as conflicting with each other: Kernel Live Patching, which allows patching kernel without the need to reboot, and Immutable System, which helps prevent accidental changes to the system and only allows activating them via reboot.

These two technologies were initially introduced in different products; however, with SUSE Linux Enterprise Micro (SLE Micro), we have a product which combines both of them.

This blog post tries to explain how these technologies can be used together and how users can benefit from combining them.

Understanding the Immutable System

Many people, who get in contact with the immutable system, get the wrong understanding about what can and cannot be done to the system.

First, let’s take a closer look at what we mean when talking about SLE Micro being an immutable system. The immutability is implemented on a filesystem level. Large parts of the filesystem are mounted read-only (the read-only snapshot, using the btrfs terminology); those include the directories holding system libraries, binaries, or kernel and initial ramdisk. Obviously, in order to make the system usable, other parts have to stay writable (e.g. /var or /home directories), similarly for /etc holding configuration files. You might wonder: can we live patch an immutable system when large parts of the filesystem are read-only?

No matter what happens at the run-time, the read-only part of the filesystem assures that when booting the same snapshot at any later point of time, you get the system in the same state. You get the same kernel, the same modules loaded from the initial ram-disk, the same system daemons etc.

However, the matter of immutable system does not prevent you from regularly working with the system; you may start or stop more system services; you can load more kernel modules (e.g. drivers for hardware you connect); all those can have impact on the stability of the running system yet do not impact the currently running btrfs snapshot. When rebooting to the same btrfs snapshot, you will get the system at the same state as with previous reboot.

You can learn more about transactional update in the SLE MIcro documentation at https://documentation.suse.com/sle-micro/6.0/html/Micro-transactional-updates/transactional-updates.html

How is the kernel live patching working

The kernel live patches are distributed as regular RPM packages via product update channels from SUSE CDN, either directly or via local proxy (RMT) or SUSE Manager. The RPM contains a kernel module which includes the changes that need to be done in the kernel; when the RPM is installed, the respective kernel module gets loaded to the kernel, which makes sure that the vulnerability or bug that should be fixed by the live patch is active immediately.

How are updates applied to the system?

For applying updates to an immutable system SUSE developed transactional update. The principle of transactional update is very simple; it follows several steps:

1. Create a read-write copy of the running snapshot; thanks to the btrfs filesystem features, it does not really involve copying files (and wasting a lot of disk space), the filesystem will make sure that only files which get changed in the read-write snapshot get duplicated
2. Chroot to the read-write snapshot
3. Perform any changes you need to do; this may involve (but is not limited to) updating existing or installing new packages, or enabling some core features of the system (e.g. kdump)
4. Exit the chroot and make the snapshot read-only
5. If everything went successfully, make the snapshot the default one to be booted and, when convenient, reboot the system

The important are the changes which can be made to the system in step 3 above. While it cannot change the binaries in the running snapshot (as it is guarded by no way to escape chroot), otherwise it can do pretty much anything without limits, as if running in the read-only snapshot outside chroot. Among other things, it can load kernel modules.

As explained above, loading a kernel module does not conflict with the immutability of the system. Whenever a user decides to boot any previous snapshot, he gets exactly the same he did before; the same kernel, the same initrd, the same binaries, without any trace of any additional kernel module getting loaded.

So, how does it work together

Live patching an immutable system can be seen as something controversial or conflicting. Because of that, kernel live patching is not enabled on SLE Micro by default, but has to be enabled explicitly.

The main trick is that loading a kernel module can be done any time, independently of the root filesystem being read-only or read-write, as long as you can get the new kernel module in – which you can do, even without executing transactional-update, via downloading it from internet or via connecting a USB stick, as long as the kernel module is compatible with the running kernel and signed properly. However, just loading a kernel module is not enough; it would not be loaded again after reboot.

When applying a kernel live patch via transactional update, the respective live patch gets installed into the snapshot as a regular RPM. In order to become active after reboot, the initrd gets re-created including the live patch (as kernel module) to be loaded on next boot. This assures that you get a patched system after reboot.

The last step is to actually immediately apply the patch – which is done via loading the kernel module.

Would anything go wrong and the system would not behave correctly with the live patch applied, either after reboot or instantly the operator can load the snapshot which was active when applying the live patch. This snapshot does not have the live patch RPM installed, nor is the initrd including the respective kernel module, therefore you can get back to the system in the same status it was before live patching.

Conclusion

This blog post clarified how kernel live patching can play well together even with immutable systems. Applying a live patch, does not even need to start a new read-write snapshot, as loading a new kernel module can be done any time; the critical feature – possibility to revert the system to previously known working state, is not compromised when applying kernel live patches.