Clock drifts in KVM virtual machines

This document (7008698) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 1

Situation

At at customer site it has been observed that the clock is drifting on some of the guests / virtual machines (VM's) on a SUSE Linux Enterprise 11 Service Pack 1 (SLES11SP1) hosts with KVM hypervisor.
Kernel version on hosts : 2.6.32.27-0.2
Kernel version on guests : 2.6.32.23-0.3
The host machines are busy and each hosting around four VM's.
In the problematic guests the time is correct just after reboot, but then start drifting as days go by. They are all running with the clock source being kvm_clock ad per the default configuration.

Researching the issue revealed that the VM's suffering the most from this problem are the ones that do not use a lot of CPU are the ones having the largest time shift.

Resolution

There are known issues with time skew/drift/jumping in KVM guests.
While the community is working on resolving these, the following workarounds can stabilize the clock.

Implement CPU pinning - also known as defining CPU affinity
If possible, pin the virtual CPU(s) of the VM to one physical CPU socket.
Use the output of

# /bin/lscpu

to get an overview of available CPUs and then /proc/cpuinfo contains the details about which CPU-IDs are on what physical processor.

There are three ways to configure CPU pinning/affinity.

In virt-manager edit the VM details and configure "Initial pinning" in the CPU pinning section of Processor properties.
Here a range (e.g. 4-7) will ensure all four VCPU's of the guest run on these host CPU's.
Define it in the XML domain definition file.
To achieve the same configuration as above, change the <vcpu> tag to look like this :
<vcpu cpuset='4-7'>4</vcpu>

Use the virsh tool. This requires a command for each VCPU. E.g.

# virsh vcpupin <domain-name/id> 0 4,5,6,7
# virsh vcpupin <domain-name/id> 1 4,5,6,7
# virsh vcpupin <domain-name/id> 2 4,5,6,7
# virsh vcpupin <domain-name/id> 3 4,5,6,7

Refer to either the man page for virsh or the command "virsh help vcpupin" for details. virsh vcpupin also has the capability to pin any VCPU to a specific CPU instead of a list like above, to completely prevent VCPUs from floating around.

Prevent VM from idling too long
Create some sort of simple "keep alive" script that runs in the VM and uses a few CPU cycles on a regular basis. It could also be a service running on a (remote) machine and establishes regular ssh sessions to VMs and perform some minor task to achieve the same.

Additional Information

To verify that kvm_clock is correctly configured in the VM :
The command

# dmesg | grep kvm-clock

should return messages like
kvm-clock: cpu 0, msr 0:1941601, boot clock
kvm-clock: cpu 0, msr 0:6213601, primary cpu clock
kvm-clock: cpu 1, msr 0:6293601, secondary cpu clock
kvm-clock: cpu 2, msr 0:6313601, secondary cpu clock
kvm-clock: cpu 3, msr 0:6393601, secondary cpu clock
Switching to clocksource kvm-clock

Furthermore this can be confirmed with

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource

which should return
kvm-clock

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.