Skip to content

Hypervisor Architectures Compared

23 min4561 words2 diagrams

Source: Victor Bona's Obsidian Compendium snapshot, Knowledge base/Virtualization/17 Hypervisor Architectures Compared.md.

Purpose: Compare the production hypervisors and VMMs you will be measured against (where they put device models, what language they use, what they expose to the guest, what they boot and how fast) so you can deliberately copy the right decisions into your own Rust VMM.

Hypervisor Architectures Compared

Related notes: 01 Virtualization Mental Model and History, 07 Device Emulation PIO MMIO and the Bus Model, 08 Virtio Devices Virtqueues and vhost, 11 Device Passthrough VFIO IOMMU and Nested Virtualization, 13 Security Isolation and Attack Surface, 18 Rust VMM Ecosystem and Project Templates, 09 cgroups Namespaces Containers and Runtime Isolation.

Every hypervisor answers the same five questions differently:

  1. Which component runs in the most privileged mode, and how big is it?
  2. Where do device models run (hypervisor, kernel, privileged VM, per-VM process, per-device process)?
  3. What language are the guest-reachable parts written in?
  4. What virtual platform does the guest see (legacy PC, minimal microVM, paravirtual-only)?
  5. Who manages lifecycle (API, CLI, libvirt, a cloud control plane)?

Once you can answer those for each system below, the rest of the comparison falls out.

The placement axis: where device models live

The single most important architectural decision is where device emulation runs, because device models parse guest-controlled data and are therefore the main attack surface.

Rendering diagram...

Moving right reduces blast radius per bug and increases IPC cost and engineering complexity. KVM-based VMMs sit in the right half by construction: KVM itself contains only the CPU, MMU and a few interrupt-controller and timer models; everything else is in user space.

Summary table

SystemHypervisor coreDevice model placementLanguageGuest arches (host = guest)Typical boot pathPrimary use
QEMU (with KVM)KVM (or TCG, HVF, WHPX, Xen, NVMM)Per-VM process, optional vhost and vhost-user offloadC (some Rust experiments in progress)Very broad: x86, ARM, RISC-V, POWER, s390x, MIPS, and more (TCG emulates any)SeaBIOS, OVMF/edk2, direct kernel, many machine typesGeneral purpose, libvirt, OpenStack, desktop
FirecrackerKVMPer-VM process (VMM thread plus vCPU threads), jailedRustx86-64 (Intel, AMD), aarch64 (AWS Graviton)Direct kernel boot (ELF vmlinux on x86, PE Image on aarch64), no firmwareServerless (Lambda, Fargate), multi-tenant sandboxes
Cloud HypervisorKVM or MSHVPer-VM process, device threads, vhost-user offloadRustx86-64, aarch64; riscv64 experimentalRust Hypervisor Firmware (PVH), edk2 (CLOUDHV.fd, CLOUDHV_EFI.fd), direct kernel (PVH or bzImage)Cloud workloads, Kata, Windows guests
crosvmKVM (also Gunyah, GenieZone, HAXM, WHPX backends in tree)Per-device processes sandboxed with minijailRustx86-64, aarch64, riscv64 directories in treeDirect kernel, BIOS option, Android pVM firmwareChromeOS (Crostini), Android Virtualization Framework
XenXen (type 1 microkernel-ish hypervisor)dom0 or driver domains (QEMU for HVM emulation, PV backends in dom0 kernel)Cx86-64, ARM64 (ARM32), RISC-V port in progressPV, HVM (firmware), PVHCloud (historic AWS), embedded/automotive, Qubes OS
Hyper-VHyper-V hypervisor (type 1)Root partition (VSPs), worker process per VMC/C++ (Microsoft is adding Rust, OpenVMM/OpenHCL)x86-64, ARM64UEFI (Gen 2), BIOS (Gen 1)Windows Server, Azure, WSL2
bhyvevmm.ko in FreeBSD kernelPer-VM user process (bhyve)Camd64; arm64 and riscv added in FreeBSD 15.0bhyveload, UEFI (edk2), u-boot on arm64/riscvFreeBSD hosting, TrueNAS, SmartOS/illumos port
VMware ESXivmkernel (type 1)Per-VM VMX user world plus vmkernel device layerC/C++ (proprietary)x86-64, ARM64 (ESXi-Arm fling, limited)BIOS or EFIEnterprise data center
Apple Hypervisor.frameworkmacOS kernel owns VT-x / EL2Entirely in the client app (or Virtualization.framework)C/Swift API, clients in any languageHost-native: x86-64 on Intel Macs, ARM64 on Apple siliconWhatever the client implementsDocker Desktop, UTM, Lima, Tart

Treat the guest-arch column as "host architecture equals guest architecture with hardware acceleration". Only QEMU's TCG offers cross-architecture emulation.

QEMU

QEMU is the reference implementation of "everything". Architecture:

  • One process per VM. A main loop thread (glib-based AIO context), one thread per vCPU, optional IOThreads for block and virtio, migration thread.
  • Accelerators: KVM on Linux, HVF on macOS, WHPX on Windows, Xen, NVMM on NetBSD, and TCG (its own dynamic binary translator) everywhere.
  • QOM (QEMU Object Model): typed object hierarchy with properties; every device is a QOM object composable from the command line.
  • Device breadth: hundreds of devices across dozens of machine types (pc, q35, microvm, virt on ARM and RISC-V, pseries, s390-ccw-virtio, ...).
  • Management: QMP (JSON over a socket), HMP for humans, usually driven by libvirt.
  • Offload: vhost-net (kernel), vhost-user (external process), vhost-vdpa.

Attack surface: large. Even if a given VM exposes few devices, the binary contains them, and QEMU historically had a steady stream of device-model CVEs (VENOM in the floppy controller, 2015, is the canonical example). Mitigations: build-time device trimming, the microvm machine type, seccomp sandbox (-sandbox on), running under SELinux sVirt or AppArmor via libvirt, and offloading devices to vhost-user.

What to copy:

  • The virt machine types on ARM64 and RISC-V are the de facto reference platform for "what Linux expects": memory at a fixed base, GIC or AIA/PLIC, PL011 or 16550 UART, virtio-mmio or PCIe ECAM, generated FDT. Read QEMU's hw/arm/virt.c and hw/riscv/virt.c when designing your FDT.
  • QMP's model of an async event stream plus synchronous commands.
  • Use QEMU as your differential oracle: when your guest misbehaves, boot the same kernel on QEMU with -machine microvm or virt and compare.

What not to copy: the breadth. Every device you add is code reachable by a hostile guest.

Firecracker

Firecracker (AWS, open sourced 2018) is the canonical minimal Rust VMM.

  • Process model: one Firecracker process per microVM, launched by the jailer, which sets up a chroot, cgroups, namespaces (including a network namespace), drops privileges to an unprivileged uid/gid, then execs Firecracker.
  • Threads: an API server thread (REST over a Unix socket), a VMM thread running the event loop with all device emulation, and one thread per vCPU. Each thread type gets its own seccomp filter (compiled from JSON by seccompiler).
  • Devices: virtio-net, virtio-block, virtio-vsock, virtio-balloon, virtio-rng (entropy), virtio-pmem, a serial console, an i8042 controller used only for guest reset on x86, an RTC (PL031 on aarch64), and a guest-facing metadata service (MMDS). Transport historically virtio-mmio; virtio-pci hotplug is listed as a developer preview in the current README.
  • Rate limiters: token buckets for bandwidth and ops/s per block and net device.
  • Boot: direct kernel boot only, no BIOS or UEFI. x86 uses an uncompressed ELF vmlinux (PVH entry is supported); aarch64 loads a PE-format Image and passes an FDT.
  • Snapshots: full and diff snapshots (guest memory file plus serialized VMM state), restore from a memory file with lazy loading (UFFD backend option).
  • Supported hosts: x86-64 Intel and AMD, aarch64 on AWS Graviton; tested host kernels listed in the README (5.10, 6.1 lines).

Published targets from SPECIFICATION.md (on specific EC2 metal instances, minimal kernel and rootfs, serial console disabled):

MetricTarget
Boot time (InstanceStart to guest /sbin/init)At most 125 ms
VMM memory overhead (1 vCPU, 128 MiB guest)At most 5 MiB
Startup to API socket available8 CPU ms (typical wall clock about 12 ms)
Guest CPU performanceGreater than 95 percent of bare metal

What to copy:

  • The thread model and per-thread seccomp filters. Small, auditable, effective.
  • The jailer as a separate binary. Your VMM should not need root; the launcher does the privileged setup and then the VMM runs with nothing.
  • Direct kernel boot as the default fast path.
  • Rate limiters as a first-class device property for multi-tenant fairness.
  • A small, versioned REST API with explicit pre-boot vs post-boot actions.
  • A specification document with measurable targets and tests that enforce them.

What it chooses not to do: no PCI passthrough, no GPU, no live migration (snapshot and restore instead), no Windows guests, no firmware. Those omissions are the product.

Cloud Hypervisor

Cloud Hypervisor (started by Intel, now a Linux Foundation project) aims for modern cloud workloads with a minimal but broader device set.

  • Runs on KVM or the Microsoft Hypervisor (MSHV), through a hypervisor crate abstraction. Minimum host kernel stated as 6.8 in the current README.
  • Arches: x86-64 and aarch64 as primary; riscv64 explicitly experimental.
  • Guests: 64-bit Linux and Windows 10 / Windows Server 2019.
  • Devices: virtio-net, virtio-block, virtio-pmem, virtio-fs, virtio-vsock, virtio-rng, virtio-console, virtio-iommu, virtio-mem, virtio-balloon, VFIO passthrough, vDPA, vhost-user for net/block/fs. PCI (virtio-pci) is the main transport, with hotplug of vCPUs, memory, VFIO and virtio devices.
  • Firmware: Rust Hypervisor Firmware (a lightweight PVH firmware), or edk2 builds (CLOUDHV.fd for x86-64, CLOUDHV_EFI.fd for aarch64). Direct kernel boot needs PVH support or bzImage on x86-64.
  • Management: REST API over a Unix socket plus ch-remote CLI; live migration and snapshot/restore.
  • Security: seccomp per thread, Landlock support, memory-safe Rust.
  • Built on rust-vmm crates and shares code with other VMMs.

What to copy:

  • The hypervisor abstraction crate if you ever want MSHV or another backend. Keep KVM types out of device code.
  • PCI as the main transport if you want hotplug, MSI-X and passthrough. virtio-mmio is simpler but does not scale to many devices or hotplug as cleanly.
  • vhost-user as the escape hatch for heavyweight backends (virtiofsd, SPDK, DPDK).
  • Its CI approach: integration tests that boot real guests across firmware and kernel combinations.

crosvm

crosvm (Google, for ChromeOS and Android) is the most security-paranoid of the Rust VMMs.

  • Each virtual device runs by default in its own process, sandboxed with minijail (namespaces, seccomp policy per device). The main process proxies MMIO/PIO accesses to device processes over sockets. A compromised device process sees only its own resources.
  • Hypervisor backends in hypervisor/src: KVM, Gunyah, GenieZone, Halla, HAXM, WHPX (Windows). Architecture directories: x86_64, aarch64, riscv64.
  • Devices: virtio-gpu (with virgl/gfxstream), virtio-wayland, virtio-snd, virtio-input, virtio-fs, vhost-user frontends and backends, VFIO, and much more than Firecracker because it serves desktop Linux apps on ChromeOS.
  • Used by the Android Virtualization Framework (AVF) with pKVM for protected VMs, where the host cannot read guest memory.
  • Has its own crates (base, cros_async, vm_memory, kernel_loader, devices) rather than depending wholly on rust-vmm; it was one of the two founding projects of rust-vmm.

What to copy:

  • The idea that a device process boundary is cheap insurance for complex devices (GPU, USB, sound). If you add a device that parses complex formats, consider running it out of process via vhost-user.
  • A base crate that hides OS differences (crosvm runs on Windows hosts too).
  • Its fuzzing setup for device models.

Xen

Xen is a true type 1 hypervisor with a small core and policy pushed into a privileged guest.

  • Xen itself handles scheduling (credit2, RTDS, ARINC653, null), memory (p2m tables), event channels, grant tables, and hypercalls.
  • dom0 is a Linux (or NetBSD/FreeBSD) domain with hardware access and the toolstack (xl, libxl). Driver domains can isolate specific hardware.
  • Guest types: PV (modified kernel, no hardware assist needed, now mainly for legacy), HVM (hardware assist plus device emulation by a QEMU instance in dom0 or a stub domain), PVH (hardware assist, PV-style boot and PV devices, no emulated platform). PVH is the modern recommended mode on x86.
  • Split drivers: blkfront/blkback, netfront/netback over shared-memory rings using grant tables.
  • ARM64: Xen uses EL2 directly with no PV/HVM split; guests are hardware-assisted with PV devices and a device tree. Popular in automotive and embedded (safety partitioning, dom0less boot).
  • RISC-V port: in progress upstream; treat as not production ready.

What to copy:

  • PVH as a boot protocol: hardware-assisted guest, entered in 32-bit protected mode with paging off, start_info pointer in ebx. Linux and Cloud Hypervisor's firmware support it, and it avoids the bzImage real-mode setup path. See 09 Booting Guests Firmware and the Linux Boot Protocol.
  • Grant tables as a mental model for explicit, revocable memory sharing; virtio relies on implicit full guest memory access, which is simpler but broader.
  • Disaggregation: device backends in less-privileged domains.

Microsoft Hyper-V

  • Type 1 hypervisor with a root partition (Windows) that owns devices and runs the virtualization stack (VMMS service, per-VM worker process vmwp.exe).
  • Child partitions use enlightened drivers: Virtualization Service Clients (VSCs) talk over VMBus (shared-memory ring channel) to Virtualization Service Providers (VSPs) in the root. Linux has upstream VMBus drivers (hv_vmbus, hv_netvsc, hv_storvsc).
  • Hypervisor Top Level Functional Specification (TLFS) documents hypercalls, synthetic MSRs, SynIC (synthetic interrupt controller), reference TSC page. KVM can expose Hyper-V enlightenments to Windows guests (KVM_CAP_HYPERV_*) because Windows runs faster with them.
  • MSHV: Linux can run as the root partition on the Microsoft hypervisor and expose /dev/mshv, which Cloud Hypervisor uses.
  • OpenVMM and OpenHCL: Microsoft's open-source Rust VMM and paravisor (running inside a confidential VM's VTL2). Worth reading as a third large Rust VMM codebase.
  • Arches: x86-64 and ARM64.

What to copy: the Hyper-V enlightenments list when you want Windows guests on KVM; the paravisor idea for confidential computing.

bhyve

  • FreeBSD's hypervisor: vmm.ko kernel module (like KVM) plus a bhyve user-space process per VM (like your VMM), configured by bhyvectl and bhyveload.
  • Minimal legacy: no BIOS emulation originally (bhyveload loads FreeBSD kernels directly); UEFI via edk2 for other guests.
  • Devices: virtio-net, virtio-blk, virtio-9p, NVMe, AHCI, e1000, xHCI tablet, framebuffer (VNC), passthrough.
  • Arches: amd64 long-standing. FreeBSD 15.0 release notes add arm64 and riscv support for vmm and bhyve, with u-boot ports as the boot loaders.
  • Ported to illumos (SmartOS, OmniOS) where it became the main hypervisor.

What to copy: the clean kernel/user split mirrors KVM and shows the model is OS-independent. Its NVMe emulation is a readable reference.

VMware ESXi

  • vmkernel is a purpose-built type 1 kernel with its own scheduler, memory manager (transparent page sharing, ballooning, compression), storage stack (VMFS, vSAN) and network stack (vSwitch).
  • Each VM has a VMX user-world process for device emulation plus a VMM component per vCPU in the vmkernel context.
  • Historically used binary translation on x86; now hardware-assisted only.
  • Devices: emulated (e1000, LSI Logic) and paravirtual (vmxnet3, PVSCSI).
  • Closed source; studied through papers and public documentation.

What to copy: ballooning and memory overcommit ideas (virtio-balloon descends from this), and the notion that a paravirtual NIC (vmxnet3) beats emulating an Intel NIC.

Apple Hypervisor.framework and Virtualization.framework

  • Hypervisor.framework (available since macOS 10.10 on Intel, and on Apple silicon since macOS 11) is a low-level API: create a VM, map memory, create vCPUs, run, handle exits. It is architecturally like /dev/kvm with no device models at all. On Apple silicon the kernel owns EL2; the API exposes ARM64 system registers and a vGIC (in recent macOS versions).
  • Virtualization.framework is the high-level API built on it, with virtio devices (block, net, console, fs, vsock, GPU), Linux direct boot and macOS guests.
  • Clients: QEMU (HVF accelerator), UTM, Lima, Tart, Docker Desktop, Podman machine.

What to copy: if you want your Rust VMM to run on macOS, put your vCPU and memory API behind a trait with KVM and HVF backends, as QEMU and crosvm (with WHPX) do. Note that an M-series Mac only runs ARM64 guests with acceleration.

Kata Containers and gVisor (for contrast)

These are sandboxes for containers, not hypervisors, but they consume or replace hypervisors:

Kata ContainersgVisor
ModelEach pod runs inside a lightweight VM with its own guest kernel; an agent inside the VM runs containers.A user-space kernel (Sentry, Go) implements the Linux syscall ABI; the app never talks to the host kernel directly.
HypervisorPluggable: QEMU, Cloud Hypervisor, Firecracker, Dragonball (built-in Rust VMM in the Kata runtime).Platforms: systrap (default since mid-2023, seccomp SECCOMP_RET_TRAP), KVM (Sentry acts as guest kernel and VMM), ptrace (deprecated).
File sharingvirtio-fs (virtiofsd), or block devicesGofer process or directfs
Interface exposed to the workloadA full Linux kernel inside a VMSentry's Linux ABI implementation
StrengthStrong hardware boundary, full kernel compatibilityNo hardware virtualization needed, small host syscall surface
CostGuest kernel memory, boot time, double page cacheSyscall overhead, incomplete ABI coverage

Lesson for your VMM: Kata's pluggable hypervisor interface is a real customer. If your VMM can boot a kernel, attach virtio-fs, virtio-vsock and a block device and expose an API, it can be a Kata backend.

Attack surface comparison

SystemGuest-reachable code in privileged contextGuest-reachable code in host user spaceMemory safety of device modelsDefault sandbox
QEMU/KVMKVM (CPU, MMU, irqchip, vhost-net if used)Large device set in one processCseccomp (opt-in), SELinux/AppArmor via libvirt
FirecrackerKVMSmall device set in one processRustjailer (chroot, namespaces, cgroups, uid drop) plus per-thread seccomp
Cloud HypervisorKVM or MSHVMedium device set, plus vhost-user backendsRustper-thread seccomp, Landlock
crosvmKVM (or others)Per-device processesRustminijail per device
XenXen hypervisorQEMU in dom0 or stubdom (HVM), PV backends in dom0 kernelCStub domains, driver domains, XSM/FLASK
Hyper-VHyper-V hypervisorVSPs in root partition kernel, worker processC/C++Root partition is highly privileged
ESXivmkernelVMX user worldC/C++VMX sandboxing (proprietary)

KVM itself is common to most of these: a bug in KVM's instruction emulator or MMU affects QEMU, Firecracker, Cloud Hypervisor and crosvm alike. Your VMM cannot fix KVM, but it can avoid enabling KVM features it does not need (for example, not creating the in-kernel PIT if you do not need it, not exposing nested virtualization).

Boot time and footprint

Indicative ranges for a minimal Linux guest to init on modern hardware (project-published or commonly reported; always measure your own):

VMM and boot pathOrder of magnitude to guest initNotes
Firecracker, direct bootAbout 100 to 150 msSpec target at most 125 ms to /sbin/init
Cloud Hypervisor, direct boot (PVH)Comparable to Firecracker, around 100 msFirmware boot adds time
QEMU microvm machine, direct bootLow hundreds of msMuch faster than q35 with firmware
QEMU q35 with OVMFSecondsFirmware init, PCI enumeration
Your first VMMDominated by kernel decompression and initUse an uncompressed or PVH kernel, disable unneeded drivers, keep serial output small

Boot time is mostly guest kernel time once the VMM is lean. VMM-side levers: avoid firmware, pre-fault or lazily fault memory deliberately, keep ACPI/FDT generation cheap, and do not probe devices you do not expose.

Supported architectures in detail

Systemx86-64ARM64RISC-V
QEMUFull (KVM, TCG)Full (KVM, HVF, TCG); virt machineFull TCG; KVM on RISC-V hosts with H extension; virt machine
FirecrackerYes (Intel, AMD)Yes (Graviton 2 and later)No
Cloud HypervisorYesYesExperimental, limited
crosvmYesYesriscv64 arch crate in tree (maturity unclear, check upstream)
XenYesYesPort in progress
Hyper-VYesYesNo
bhyveYesAdded in FreeBSD 15.0Added in FreeBSD 15.0
ESXiYesLimited (Arm fling)No
Apple Hypervisor.frameworkIntel MacsApple siliconNo

For RISC-V specifically, QEMU's virt board plus KVM is the most mature reference today, and kvm-ioctls supports riscv64 in its examples (set_one_reg with RISC-V register IDs). See 16 RISC-V Virtualization and the Hypervisor Extension.

Process and thread models side by side

VMMProcesses per VMThreadsWhere a PIO/MMIO exit is handledWhere queue notifications are handled
QEMU1 (plus optional vhost-user daemons, virtiofsd)Main loop, one per vCPU, optional IOThreads, migration, worker poolsvCPU thread, under the Big QEMU Lock (BQL) unless the device is lock-freeMain loop or an IOThread via ioeventfd
Firecracker1 (launched by jailer)API, VMM (event loop), one per vCPUvCPU thread, through the MMIO or PIO busVMM thread via ioeventfd and event-manager
Cloud Hypervisor1 (plus optional vhost-user backends)API/HTTP, VMM control loop, one per vCPU, one or more per virtio devicevCPU threadPer-device threads with their own epoll loop
crosvm1 main plus 1 per sandboxed devicevCPU threads in main, device worker threads in each device processvCPU thread forwards over a socket to the device processDevice process
Xen HVMHypervisor plus a QEMU (device model) per domain in dom0 or a stub domainXen schedules vCPUs; QEMU threads in dom0Xen forwards ioreq to QEMU via shared page and event channelPV backends in dom0 kernel (blkback, netback)

The common thread: in every KVM-based design the vCPU thread itself handles synchronous register accesses, and asynchronous work goes elsewhere. Your VMM should follow the same split.

Memory management features

FeatureQEMUFirecrackerCloud HypervisorcrosvmNotes for your VMM
Hugepage backingYes (memory-backend-file / memfd with hugetlb)Yes (hugetlbfs option)YesYesBig win for TLB reach; complicates dirty tracking granularity and snapshot lazy loading
virtio-balloonYesYes (with statistics)YesYesCooperative; a hostile guest can ignore it
virtio-memYesRecent memory hotplug support listed in READMEYesCheck upstreamFiner-grained, guest-driven hot(un)plug
Shared file-backed memoryYesYes (snapshot restore maps memory file)YesYesNeeded for vhost-user (backend must map guest memory)
Userfaultfd lazy restorePostcopy migrationSnapshot restore via UFFD handlerNot the main pathCheck upstreamFast restore, page fault latency during warm-up
KSM / page sharingHost-side madvise(MADV_MERGEABLE) optionNot by defaultOptionOptionSide-channel risk across tenants; usually off for multi-tenant

State: snapshots and migration

VMMSnapshot/restoreLive migrationState format concerns
QEMUYes (savevm, external snapshots)Yes: pre-copy, post-copy, multifd, RDMAVMState descriptions per device with versioning and subsections
FirecrackerYes, full and diff, designed for fast clone-and-restoreNo (snapshot-based workflows instead)Versioned state with explicit compatibility rules per release
Cloud HypervisorYesYes, pre-copy over TCP or Unix socketserde-based, Snapshottable and Migratable traits per component
crosvmYes (snapshot support used by Android)LimitedPer-device serialization
XenSave/restoreYeslibxc stream format
Hyper-V / ESXiYesYes (Live Migration, vMotion)Proprietary

Design lesson: give every device a snapshot() and restore() from day one, even if the first implementation returns "unsupported". See 12 Snapshots Live Migration and Dirty Page Tracking.

Interrupt controller and timer choices

VMMx86-64ARM64RISC-V
QEMU/KVMIn-kernel LAPIC, IOAPIC, PIC (or split irqchip with user-space IOAPIC), kvmclockIn-kernel vGICv2/v3 with ITS, arch timerIn-kernel AIA (APLIC, IMSIC) or user-space PLIC/APLIC, Sstc timer or SBI
FirecrackerIn-kernel irqchip, PIT, kvmclockIn-kernel vGICv2/v3, arch timerNot supported
Cloud HypervisorSplit model: in-kernel LAPIC, user-space IOAPIC device; MSI for PCIvGICv3 with ITS for MSIExperimental (AIA)
crosvmIn-kernel or split irqchip optionsvGICAIA in tree (check maturity)

The split irqchip (KVM_CAP_SPLIT_IRQCHIP on x86) keeps the LAPIC in the kernel for speed and moves the IOAPIC and PIC to user space, shrinking in-kernel attack surface. It is a reasonable second step after you get the full in-kernel irqchip working. See 06 Interrupts Timers and Interrupt Controller Virtualization.

Management interfaces

VMMAPIConfiguration modelWho usually drives it
QEMUQMP (JSON over socket), HMPCommand-line device graphlibvirt, OpenStack Nova, Proxmox, Kubevirt
FirecrackerREST over Unix socket (OpenAPI spec), or JSON config file at startPre-boot resource configuration, then InstanceStartCustom control planes (Lambda, Fargate), firecracker-containerd, Kata
Cloud HypervisorREST over Unix socket, ch-remote, D-Bus optionCLI or API, hotplug at runtimeKata, libvirt driver (in progress), custom
crosvmCLI plus control socket (crosvm_control)CLIChromeOS concierge, Android virtualizationservice
Xenlibxl / xl, XenStoreDomain config filesXAPI (XenServer), Qubes

Copy Firecracker's discipline: an OpenAPI-described API, explicit pre-boot vs runtime actions, and refusal (not silent ignore) of unsupported requests.

What to copy for your own VMM

Rendering diagram...

A concrete decision table:

DecisionRecommended defaultCopied fromWhen to deviate
Device transportvirtio-mmio first, virtio-pci laterFirecracker, then Cloud HypervisorNeed hotplug, MSI-X, VFIO, or more than a handful of devices: go PCI
BootDirect kernel (bzImage or PVH on x86, Image + DTB on ARM64 and RISC-V)Firecracker, Cloud HypervisorNeed arbitrary OS images or Windows: add edk2
ThreadingOne thread per vCPU, one event-loop thread for devices, one API threadFirecrackerHigh I/O: per-device threads or vhost-user (Cloud Hypervisor)
InterruptsIn-kernel irqchip (x86 KVM_CREATE_IRQCHIP, ARM vGICv3 device, RISC-V AIA device) with irqfdAll KVM VMMsResearch or split irqchip for x2APIC experiments
SandboxSeparate jailer binary, seccomp per thread, no rootFirecrackerComplex devices: separate processes (crosvm)
APISmall REST or JSON-RPC over a Unix socketFirecracker, Cloud HypervisorLibrary embedding: a Rust API only
StateSerializable device state from day oneFirecracker snapshots, Cloud Hypervisor migrationNever; retrofitting is painful
Hypervisor abstractionThin trait, KVM only at firstCloud Hypervisor hypervisor crateDo not over-abstract before you have a second backend

Common pitfalls

  • Copying QEMU's breadth into a security-sensitive VMM. Each device must justify its existence.
  • Copying Firecracker's minimalism into a general-purpose VMM and then bolting on PCI, firmware and hotplug in ways the original architecture did not plan for.
  • Treating "written in Rust" as the whole security story. Firecracker's isolation comes as much from the jailer and seccomp as from the language; unsafe around guest memory and ioctls remains.
  • Assuming a VMM supports an architecture because it compiles there. Check the project's tested-hosts list (Firecracker lists specific instance types and kernels).
  • Building a hypervisor abstraction trait before having two backends; it will encode KVM's shape anyway.
  • Comparing boot times measured differently (to API ready, to kernel entry, to init, to application ready). Always state the endpoint.
  • Forgetting that KVM is shared attack surface; enabling KVM capabilities you do not need (nested, PIT, Hyper-V enlightenments) widens it.

Review checklist

  • For each system, I can name where device models run and what sandbox protects them.
  • I can explain why Firecracker uses one process with per-thread seccomp while crosvm uses one process per device.
  • I can explain Xen's PV, HVM and PVH modes and why PVH matters for direct boot.
  • I can explain Hyper-V's VMBus/VSP/VSC model and why KVM implements Hyper-V enlightenments.
  • I can list which systems support x86-64, ARM64 and RISC-V hosts, and at what maturity.
  • I can state Firecracker's published boot and memory targets and their measurement conditions.
  • I can contrast Kata Containers and gVisor and explain what a VMM must provide to be a Kata backend.
  • I have a written decision table for my VMM's transport, boot, threading, interrupts, sandbox, API and state.

Primary Sources

contactvictor.bona@hotmail.com
victor-bona · terminal

type help for commands · esc to close