KVM live migration fails between SLES11 SP3 and SP4

SUSE Linux Enterprise Server 11 Service Pack 3 (SLES 11 SP3)
SUSE Linux Enterprise Server 11 Service Pack 4 (SLES 11 SP4)
Kernel-based Virtual Machine (KVM)
Migration
x86 Architecture

Live KVM migration fails using qemu-kvm.

Error: "qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed"

When live migrating a KVM virtual machine, or restoring a saved image, the specifics of the new instance of the virtual machine must carefully match the old instance. This includes the memory layout of the guest.

Due to an unnoticed linker change, when building the SLE 11 SP4 kvm package, the size of the virtio-net pxe rom image (/usr/share/qemu-kvm/pxe.virtio.rom) changed from the expected <=64K size to > 64K, which impacts the runtime memory layout of the guest, preventing a clean migration or save/restore in some instances (see below).

SLE 11 SP1, SLE 12 GA, and SLE 12 SP1 x86 all have a consistent pxe-virtio.rom sizes, but the SLE 11 SP4 kvm package has been found to have included an incompatible sized pxe-virtio.rom file as described above. Unfortunately it would be extremely difficult to resolve this inconsistency for all supported migration configurations in a transparent way, due to the stringent requirements of the live migration and save/restore process.

It was decided that the best solution was to simply correct the size of the pxe-virtio.rom image in a SLE 11 SP4 package update, allowing for proper forward migration from SLE 11 SP3 to SLE 11 SP4, as well as from SLE 11 SP4 to SLE 12 versioned hosts. This change, which occurred as of the kvm-1.4.2-35.1 package update, unfortunately then creates a migration incompatibility from pre-fixed SLE 11 SP4 hosts to post-fixed SLE 11 SP4 hosts, when the virtio-net device is assigned to the guest, and the option rom is enabled. (The option rom for a network device is enabled by default).

Mitigation:

The following are ways to avoid being impacted by this issue.

If at all possible do not do a live migration (or restore) of a guest using the virti-net interface from a KVM host with the pre kvm-1.4.2-35.1 package update to one with that update (or newer) applied. If you can off-line migrate your guests from the pre-patched KVM host to a patched KVM host, there will be no problem.

If your workflow is such that you require a live migration (or restore) from a pre-patched SLE 11 KVM host to a post-patched one (for example, due to an emergency migration need) you could temporarily replace the kvm-1.4.2-35.1 package /usr/share/qemu-kvm/pxe.virtio.rom file with the file from the old package (eg the one on the source host of the migration), and then restore the file as soon as the guest is migrated. You'll then want to shut down and restart the migrated guest as soon as reasonable to ensure it can continue to be migratable. Rebooting the guest is not sufficient, in this proposed workaround.

Note that if the pxe rom is not enabled for the virtio-net device, the issue will not occur. If no network booting through this interface is needed, this is also a way to avoid this issue. This is done at the libvirt level by specifying <rom bar='off'/> as an element for the virtio network device in the guest xml config. At the QEMU command line level, it is done by specifying romfile= or rombar=0 to the -device virtio-net-pci command line option.

The symptom of this incompatibility is that the live migration (or restore) will fail with the following message being produced by the QEMU executable:

qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed

This message is written to stderr by the QEMU program. When using libvirt to control KVM, it will be recorded in the per guest log file of on the KVM host of the target of the migration / restore. (eg: /var/log/libvirt/qemu/$vmname.log).

Note that it is possible for some other guest ram migration incompatibility not related to the above mentioned issue to generate this message, but it is quite unlikely.