Recommended base configuration for XEN Host

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10

 

XEN Dom0 becomes unresponsive when shutting down a domU virtual machine (VM)

VM may fail to load

Random, system-wide performance issues

Random host or vm failures and hangs

Looking for the best configuration guidelines for setting up a XEN server

What to check before troubleshooting a XEN Host

To avoid random, unexplained performance issues and failures in a XEN virtualization environment the following configuration items should always be checked;  Correspondingly, before troubleshooting ANY XEN Host or VM issue verify the XEN Host configuration as follows:
 

• "Lock down" the amount of memory assigned to dom0.

1. Set the amount of memory Dom0 is given at boot time
2. Disable ballooning

• Do not restrict dom0 to a single VCPU
• Only use Logical Volume Manager (LVM) on Dom0 or DomU, but not both

Restricting the amount of memory assigned to dom0 at boot time:

First determine a reasonable amount of memory for dom0. This amount will vary depending on your hardware and configuration.  Generally speaking, 2048M is used.   Then use an editor to change the hypervisor line in /boot/grub/menu.lst to have "dom0_mem=X", where X is the memory that will be assigned to dom0. In the example below 2048m (2G) was determined to be the amount:

title Xen -- SUSE Linux Enterprise Server 10 SP1 - 2.6.16.54-0.2.5
root (hd0,1)
kernel /xen.gz dom0_mem=2048m
module /vmlinuz-2.6.16.54-0.2.5-xen root=/dev/disk/by-id/scsi-3600508e00000000015153e9beb87870e-part1 vga=0x314 resume=/dev/sda3 splash=0 showopts
module /initrd-2.6.16.54-0.2.5-xen

Disable ballooning:

For SLES10 & SLES11: Change dom0-min-mem in /etc/xen/xend-config.sxp to match the value specified in the dom0_mem= boot option
Additionally for SLES11: Change (enable-dom0-ballooning yes) in /etc/xen/xend-config.sxp to (enable-dom0-ballooning no).

Ensure that dom0 has more than 1 processor.

This is the default (as long as the server has more than 1 cpu). However, you can verify this by using `xm list` and checking the VCPUs column for Domain-0 (example below):

Name ID Mem VCPUs State Time(s)
Domain-0 0 4096 16 r----- 478.2
sles-10-sp1 55000 1 27.4

If for some reason dom0 has been restricted to a single vcpu change it back by editing /etc/xen/xend-config.sxp. In that file there is a line that normally reads "(dom0-cpus 0)". If dom0 is coming up with only 1 vcpu then the line will read (dom0-cpus 1). Changing it back to the default will restore the desired behavior.

Only use LVM on Dom0 or DomU, but not both

Testing has shown that a DomU should never have LVM enabled if the disk the DomU resides on is managed with LVM by the Dom0.    It is recommended that you only use LVM on Dom0 or DomU, but not both.

After making these changes, reboot the server.  REBOOT is required!

By default, a XEN installation will allow dom0 access to all physical memory installed in the system. XEN will then attempt to free (and reclaim) memory as needed for domU's...a process called "ballooning".

Testing has shown that this process may cause dom0 to become unresponsive for periods of several minutes during the shutdown/creation/migration of a domU if the domU has been assigned a memory amount > 512M or dom0 has been restricted to a single VCPU.

When using LVM on both the Dom0 and a DomU, I/O performance suffers.

A single processor is insufficient for Dom0.

Varying from any of configuration points mentioned in this document will cause a wide variety of issues including unexplained behavior, performance issues and random failures. 

Interchangable terms:

VM = Virtual Machine
DomU = VM
Dom0 = Host

Frequently asked questions:

1- What is "ballooning" and how/why does it work against us here?

"Ballooning" is the process by which the amount of memory assigned to a domain is changed (up or down). In the case of this issue, domain-0 automatically "balloons" to free up memory domU needs to reclaim for shutdown.

2- Is there a formula for computing memory allocation. If I understand correctly, the total assignment to dom0 + hypervisor + vms must be < or = total physical memory in the machine. Is this correct?

Correct.

The amount of memory needed by the hypervisor, domain0, and each domU varies based on machine specifications, type of VM, etc. Therefore, there are no fixed values that can be assigned to any of those variables. However, 'xm info' can provide a view into the current state of the system. There are 5 fields in the list that provide information about the memory status:

total_memory : 8190 # Total RAM available to the system
free_memory : 177 # Free RAM (not in use by hypervisor, dom0, or any domU)
max_free_memory : 5610 # Maximum value possible for"free_memory" after ballooning dom0
max_para_memory : 5606 # Maximum amount of memory that could be assigned to a new pv guest
max_hvm_memory : 5584 # Maximum amount of memory that could be assigned to a new fv guest

3- Is there a limit regarding how much memory can be assigned to dom0? i.e. In a machine with 16G of RAM will "dom0_mem=12G" work as long as no more than 4G is allocated to vms and the hypervisor?

The only maximum for dom0 is the amount of memory in the physical system. The bigger concern should be the minimum that can be allocated to dom0. Ideally, find a value that is small enough to maximize the amount of memory that can be assigned to the guests, yet large enough that dom0 can handle all of the required I/O, management tasks, etc. Again, this value varies based on numerous factors, so experimentation is required in every environment. Once the appropriate value has been determined, the best thing to do is to set the initial/maximum memory assigned to dom0 using the "dom0_mem=" boot option and then prevent dom0 from ballooning below that value by changing the value of dom0-min-mem in /etc/xen/xend-config.sxp. This, in effect, disables the ballooning process.

4- Is there a formula for computing how much memory is required by the hypervisor (how much to leave "unassigned" for the hypervisor's needs)? Is this a static value?

This is not a static value. At very least there is going to be variation based on the hardware. For example, the hypervisor needs more memory to track the RAM on a 128 GB machine than it does on a machine with only 1 GB of memory. However, it is possible to get a good idea of how much is required by booting XEN in the default configuration (all available memory assigned to dom0). In this configuration, the total system memory minus the amount of memory assigned to dom0 equals the amount required by the hypervisor.

5-  But shouldn't the OS get the memory back for use with other applications when the VM is shut down?

In a single 'desktop' environment memory ballooning does make sense.  In a production server environment running mission critical systems there are better ways to address performance and availability.  Note that although XEN can be installed on Suse Linux Enterprise Desktop it is for technical preview only;  It is NOT officially supported.