Engineer notes: Xen in 2019 and beyond

Updated: 11th October 2019

Introduction

We have been managing Xen based virtual infrastructure since 2006, so like to think we know it quite well. Not only have we been hosting thousands of virtual machines on it for well over a decade, but we have also been providing consultancy to other companies. This includes dealing with a variety of different hardware, and building our own Xen hypervisor and Kernel packages where needed.

Although RedHat Enteprise Linux moved from Xen to Linux-KVM in RHEL6 and major cloud hosting companies like AWS have migrated from Xen to Linux-KVM, we are commited to Xen. It is a stable, mature virtualization technology with active development, and recent changes mean Xen provides equal or better performance and security than alternatives.

What is Xen?

Xen is a Type-1 hypervisor and runs directly on hardware, controlling CPU, Memory and Interrupts only. Hardware drivers e.g. Disk and Network, are offloaded to virtual machines. The host operating system or control domain is referred to as Domain-0 (or Dom0 for short), and virtual guests are referred to as Domain-U's (or DomU's for short).

Core Features

Virtualization Mode 1: Paravirtualization (PV)

Xen Paravirtualization is a technology which allows slightly modified Linux operating systems to be virtualized on commodity hardware with very good performance, and does not require the processor in the host machine to support virtualization extensions.

This was the key selling point of Xen back in the day, as it performed significantly better than the alternatives. The only downside was that guest operating systems must include the required drivers, which communicate with drivers running on Dom0.

Today these drivers are included in the Linux upstream kernel, and the kernels in all modern Linux distributions.

  • For networking: Dom0 (Netback driver) --> DomU (Netfront driver)
  • For disk: Dom0 (Blkback driver) --> DomU (Blkfront driver)

PyGrub

PyGrub is a python implementation of the grub bootloader for paravirtualized guests. Prior to PyGrub, kernels had to exist on Dom0 and passed into DomU memory during the boot process. PyGrub reads the DomU filesystem and parses the grub.conf configuration file (Supporting both Legacy Grub 0.97 and Grub 2.x formats), then retrieves the selected (or default) kernel and ramdisk and places them into the DomU's memory so it can boot.

PyGrub works well and will continue to be supported for the forseeable future, however it is not the most secure option and does not support all available grub configuration options.

PvGrub

PvGrub is a port of legacy Grub 0.97 for Paravirtualization, and runs inside the DomU in memory underneath MiniOS which we described earlier. Like PyGrub it reads the guest filesystem, parses the grub.conf configuration file and loads the appropriate kernel and ramdisk image into memory.

It provides greater security than PyGrub, since the process from start to finish occurs within the DomU. For example if a malicious kernel or device driver was created gain access to memory or escalated privileges, you would only have control of your own DomU not the control domain Dom0. The drawback of PvGrub however is that it only supports Legacy Grub 0.97 configuration files.

PvGrub2

In 2013, PvGrub code was merged with the upstream grub2 code instead of being maintained seperately. This means it is now possible to compile Grub2 using the latest upstream code specifically for Xen. This is a full version of Grub2 for Xen paravirtualized guests, however note that it is not compatible with legacy grub configuration files.

For some reason adoption of PvGrub2 has been slow, and the required kernel image is not included with every distribution, including the CentOS virt-sig or Xen4CentOS packages for CentOS 6 and CentoS 7. For this reason many hosting providers do not appear to use it, however its the most secure and flexible option for modern guest operating systems..

The good news is that we build PvGrub2 binaries you can use, which include a few modifications to improve compatibility across different Linux distributions. We can also provide PvGrub2 compatible OS Templates, but the key thing is PVGrub2 should be able to handle any unmodified grub2 configuration files!

Virtualization Mode 2: Hardware Virtual Machine (HVM)

When CPUs started to include virtualization extensions, Xen HVM was introduced which relies on the CPU features for CPU and Memory virtualization, then uses QEMU for the remaining hardware emulation e.g. Motherboard, Disk Controllers etc. This provides a fully virtualized system, which means you can run any operating system with no special drivers.

HVM provides better CPU and Memory performance than Paravirtualization, however Disk, Network and Interrupts all rely on QEMU for software emulation which is significantly slower.

To help solve this problem, PV Drivers were created which enabled a HVM virtual machine to use Xen Paravirtualization technology for Disk and Network IO which improved performance, however this left Interrupts/Timers still being emulated in software under QEMU.

MiniOS

MiniOS is a small purpose-built operating system specifically to run on the Xen hypervisor. It's primarily used in the context of security as you will see later, and runs a single process.

Stubdomains

One problem with using QEMU for device emulation which affects both Xen HVM and Linux-KVM, is the fact a privileged process has to run on the control domain. Therefore it is vulnerable to security issues which may lead to access to memory or elevated privileges on the control domain, and there have been issues like this in the past.

Xen has a feature to mitigate this called stubdomains, whereby each HVM virtual machine has a small additional virtual machine running alongside it called a stubdomain. If a guest name is vm123 then the subdomain will be launched as vm123-dm. This stubdomain is running a purpose-built operating system (MiniOS) and QEMU, thereby greatly reducing the attack surface. If QEMU is exploited then the attacker will only have access to the stubdomain, not the control domain.

Performance of HVM virtual machines with stubdomains has shown to be equal to, or in some cases better than without stubdomains. Remember subdomains require memory, so this will need to be taken into account when planning how many virtual machines you place on each server.

Stubdomains are enabled by appending "device_model_stubdomain_override = 1" into the guest configuration file

On SolusVM, stubdomains can be added to guest configuration files automatically by creating the file /usr/local/solusvm/data/xen-stubdomain on each Xen slave

Virtualization Mode 3 (Hybrid): HVM + PV / PV-on-HVM / or PVHVM

This is hybrid of HVM + Paravirtualization and in most workloads provides better performance than Paravirtualization. At the time of writing, this should be the default for all new virtual machines.

To enable this mode, you simply need create a HVM DomU which has "xen_platform_pci = 1" specified in the configuration file. On SolusVM installations, you can do this by enabling the toggle as shown in the below screenshot:

The guest operating system must also include the required PV Drivers, which are included in the Linux upstream kernel since 2.6.39. For Microsoft Windows drivers are available on the Xen.org website HERE

Virtualization Mode 4 (Hybrid): PVH

Introduced in Xen 4.4, PVH provides the best performance and security possible by combining the advantages of Paravirtualization technology and hardware acceleration with HVM. It's first implementation had some issues, however since Xen 4.10 PVHv2 has been included which is significantly improved. Work is underway to make Dom0 run as PVH, and this will later become a default.

To use PVH you need to be running at least Xen 4.10, preferably 4.12 or newer and the Linux guest operating system must be on Kernel 4.11 or newer. It's also currently limited to direct kernel boot from Dom0, and is not supported by SolusVM at this time.

So... Should I use PV, HVM, PVHVM or PVH?

Performance

For the best performance PVH is the obvious winner as it brings together the best parts from PV and HVM. However it's still very new, and it assumes your Xen version, toolset and guest operating system supports it.

In general you should use PVHVM for all new virtual machines, or if you really don't want to install from an ISO use PV. As we said before, in most cases PVHVM will outperform PV and in some benchmarks this can be upto 2.5x. Your milage of course, may vary.

There is no reason to use regular HVM, but be wary of enabling PVHVM/PV-on-HVM mode on existing virtual machines as devices and drivers will change.

Security

Since Xen has multiple virtualization options, each has a different attack surface. Overall however, PV and HVM (All combinations) are about equal in terms of security if the following best-practices are followed:

  • For Xen PV either pass kernels from the host or use PvGrub/PvGrub2
  • For Xen HVM, enable subdomains for all your virtual guests

Questions?

Any technical information provided on this page is accurate to the best of our knowledge at the time it was published. It may however contain instructions which are not suitable for your server, or are no longer the best option.

Remember technology moves fast, a good engineer learns something new every day!

If you are a customer and have any questions about how the information on this page may apply to your servers, please get in touch and we will be happy to help.

Thanks for reading.