Meet Gunyah - Qualcomm’s open-source, lightweight hypervisor for battery-constrained devices

 Gunyah™ is a high performance and scalable Type-1 hypervisor built for demanding battery-powered, real-time, safety critical systems, we’ve developed for Qualcomm Technologies chipsets. The team developing a Linux driver for it takes active inputs from kernel maintainers and the community. With Gunyah, we can use one hypervisor across use cases as varied as automotive, mobile broadband, IoT and wearables.

This post covers the work in the Linux kernel to add support for Gunyah, with plans to upstream our patches. It also covers our work on extending support for Gunyah beyond the CrosVM virtual machine manager (VMM) to QEMU, the generic, open-source machine emulator and virtualizer. The Linux driver the team is developing will allow applications such as QEMU to interact with the hypervisor for VM management. You’ll see how you can test-launch a VM on Gunyah yourself.

(This is a summary of my presentation “Gunyah Accelerator for Qemu” at Linaro Connect. See below for details and links.)

Project 1: Ongoing development of Gunyah

Gunyah is an open-source, Type-1 hypervisor from Qualcomm, built with a strong focus on security and performance on battery-constrained devices. It has been designed as a lightweight, versatile microkernel with just enough features to manage virtual machines.

As shown in the block diagram below, the architecture supports multiple OSes, has a small attack surface and includes a resource manager (RM):

The RM is a virtual machine (VM) that runs in execution level 1 (EL1), working tightly with the core Gunyah hypervisor in EL2. The RM implements some of the policies that are required to manage VMs, like the heavy lifting of creating and launching a virtual machine. It’s an integrated component of the hypervisor in a different execution level.

Protected and unprotected VMs

Protected VMs that host some confidential/sensitive workloads have been the primary focus of Gunyah from the start. We didn’t want Android on mobiles to handle sensitive tasks like face authentication or verification of mobile banking credentials. So, in that phase of an app, control of the display, touchscreen or camera can be transferred to a protected VM, where the sensitive information is validated. After that, the rest of the application can run in Android.

Gunyah supports two main types of VM:

• Protected, in which the VM’s memory is protected from the host.
• Unprotected, in which the VM’s memory can be completely accessed by the host.

Additionally, there are two types of protected VMs:

1. Qualcomm Trusted VM (QTVM) – These are VMs whose main execution image is signed using Qualcomm tools. The hypervisor ensures that the VM’s image loaded in memory is verified by Trustzone, Qualcomm’s secure software, before allowing the VM to start. Besides a kernel and maybe a ramdisk, the VM’s execution image contains a devicetree (DT), which can specify devices that the VM needs to access. This static DT implies that some of the VM resources (such as amount of memory and number of VCPUs) are static in nature.
2. Google VMs – These are VMs as defined by the Android Virtualization Framework (AVF). The hypervisor does not enforce VM image authentication in this case. Instead, authentication is performed by PVM firmware, a software blob that runs in the VM before the VM kernel takes over. The DT used by the VM in this case is dynamic; i.e., generated at runtime.

Gunyah features

• A protected VM’s memory is private to it and cannot be accessed by its host. That is achieved by the hypervisor using the page table-based memory protection feature of a CPU. A second-stage page table associated with the host OS is modified by the hypervisor so that the host loses access to the VM’s memory.

Of course, in some cases it’s desirable to share memory between a protected VM and its host. But as of today, Gunyah is not exposing any API for the protected VM to share part of its private memory with the host. Instead, the host can assign some of its memory for the VM to use in either shared or private mode, which needs to happen before the VM can begin execution. As an example, the host could, as part of the VM creation process, assign 500 MB of its memory for the VM’s private use and 100 MB of its memory for shared use.

Protection from malicious devices is achieved using the system memory management unit (SMMU). For instance, a device assigned to the host cannot use direct memory access (DMA) to access the protected VM’s memory. Second-stage page tables in the SMMU associated with the device will be set up so that the device will see a fault when accessing the protected VM’s memory.

• The generic interrupt controller (GIC) is virtualized at the layer of the hypervisor, EL2.
• Both proxy and native scheduling schemes for VCPUs are supported. In the proxy scheme, the host scheduler handles VCPU scheduling intelligently. The host scheduler may have advanced algorithms that can improve both the performance and power consumption of a VM. In the native scheme, the hypervisor scheduler directly handles VCPU scheduling, which could be desirable for some critical VMs.
• The first gigabyte of the VM’s intermediate physical address space is available as memory-mapped I/O (MMIO) regions that can be trapped and emulated.
• Protected VM memory is sanitized after a warm reset of the system. A warm reset could be triggered for various reasons, for example, when there is a deadlock in the host OS. On some platforms, after a warm reset, memory contents are preserved, which could include data of a protected VM that was active at the time of reset. To prevent the host OS from accessing such data as it boots up again, Gunyah arranges for the VM’s memory to be sanitized by Trustzone software, which runs early after a warm reset.
• Gunyah provides for pre-host VMs, which can be launched very early during the boot sequence (or even before the host OS starts booting) by UEFI firmware. That is useful for VMs that need to be functional very early during boot and cannot wait until the host has come up.
• To allow VMs to communicate with one another, the hypervisor supports shared memory, doorbell and message queue.
• Gunyah now supports demand paging. Before, if you wanted to launch a VM with, say, 500 MB, all 500 MB had to be allocated and pinned before Gunyah would start executing the VM. But if the VM needed only 100 MB to run well, then memory was wasted. So, we've added support for demand paging in the hypervisor for better memory management. Now, when the VM accesses a valid page that has not yet been allocated, a fault goes out to the host, which allocates a page of varying size that the hypervisor maps into the VM.
• Device passthrough allows some devices to be assigned and directly accessed by the VM. We’re working on optimizing Gunyah for automotive, IoT and mobile use cases.

Project 2: The Linux Gunyah driver

The Gunyah driver that we are actively developing is close to being accepted by the upstream Linux kernel community. As shown in the diagram below, the driver is designed to enable VMMs like CrosVM and QEMU to manage VMs on the Gunyah hypervisor:

The driver runs at EL1 and provides an interface between VMMs and the hypervisor. The driver provides UAPIs for standard tasks like running a VCPU, registering eventfds and creating VMs and VCPUs. Additionally:

• There's an explicit ioctl to start the VM. We took that path instead of folding it into the API for running a VCPU.
• Another ioctl tells the driver to share and lend memory to a VM. The only difference between share and lend is that lent memory is protected, which makes the lend API useful for allocating private memory of protected VMs. Note that the v17 upstream driver we have published does not currently support the lend API. However, the version of driver included in Android Common Kernel¹ supports the lend API.
• An ioctl is provided for VMMs to specify where the devicetree is located in VM’s memory. Before the VM is allowed to start, the VM’s devicetree needs to be validated by the resource manager.
• There is currently no provision to set and get VCPU registers on the fly. Instead, the hypervisor sets the initial register context of the boot VCPU.

We envision future changes to the Linux driver to add support for different VM types, including QTVM and Google AVF VMs.

Project 3: Gunyah Accelerator for QEMU

We’re working on changes to enable QEMU to support Gunyah as an accelerator option. We expect the changes will be merged into the QEMU code base once our Linux kernel driver is accepted upstream.

A typical invocation of QEMU to launch a VM on Gunyah could look like this:

Some salient points about the proposed changes to QEMU:

• Only AArch64 is supported as a target.
• The argument –accel gunyah specifies that the Gunyah accelerator is to be used.
• Both protected and unprotected types of VM are supported.
• We’ve tested VM bringup on both a Qualcomm SoC and the QEMU virtual platform running the open-source version of Gunyah. Instructions have been provided in the patch on how to test VM bringup on a QEMU virtual platform using that open-source version.
• We’ve introduced some changes for the Arm virt machine platform.
  • Support for protected VMs (guest), indicated on the command line as follows:

swiotlb-size indicates the size of the shared memory region that QEMU needs to allocate and share with VM. That region is also marked reserved in the devicetree and associated with the restricted DMA pool functionality of the Linux kernel.

• • We have noticed subsequently that there are others working on adding similar support for the virt machine platform. We shall review and join efforts with them as necessary.
  • Support for accelerator-specific devicetree customization. For example, Gunyah needs to see additional attributes in a devicetree, like the scheduling scheme required for the VM, and virtual devices that the hypervisor needs to create to support injection of an interrupt (doorbell).
• Only proxy-scheduled VMs are supported. Future changes could be published to add support for hypervisor-scheduled VM.
• The diagram below illustrates how QEMU can inject an interrupt into the VM. Since the GIC is handled by the hypervisor itself, we create doorbells as virtual devices and bind them to various IRQ lines. QEMU registers an eventfd for each interrupt it wishes to inject. Whenever it writes into the eventfd, the Gunyah Linux kernel driver rings the corresponding doorbell to activate its IRQ.

Your turn

You can follow the instructions published in the patch series and test-launch a VM on the Gunyah hypervisor.

Don’t have access to Qualcomm SoCs running Gunyah? Not to worry – you can still use QEMU to emulate a generic ARM machine running the Gunyah hypervisor, then use a separate QEMU binary with support for the Gunyah accelerator to launch a VM in it.

For more details, watch my presentation “Gunyah Accelerator for Qemu” at Linaro Connect 2024 or refer to the patches I have published on QEMU mailing lists.

We now have a dedicated open-source channel on Developer Discord – join the community of like-minded developers to connect, get support and exchange the ideas.