DEV Community

hon6
hon6

Posted on AI-assisted

Making an Android VM on macOS give CPU and memory back to the host

An Android emulator that only runs well is half done. If it keeps a core busy while sitting on the home screen, or holds on to gigabytes it no longer uses, people notice on a laptop very quickly. Here are the changes APK PlayStudio, an Android game emulator for Apple Silicon Macs coming soon to the Mac App Store, made to QEMU with Hypervisor.framework (HVF) to fix both.

Idle CPU: 65% on an empty home screen

Sampling the QEMU process with macOS sample (the guest has no root, so simpleperf is not an option) showed two causes.

1. WFI that never sleeps

When an ARM vCPU has nothing to do it executes WFI and waits for an interrupt. Under HVF that traps to QEMU, which decides how long the vCPU thread sleeps. The stock rule was: if the next timer fires in less than 2 ms, don't sleep, just re-enter the guest. Android's scheduler tick and timers are dense enough that this was almost always true, so vCPU threads were effectively busy-waiting.

The fix lowers the threshold to 200 µs (tunable via APS_WFI_MIN_SLEEP_US) and marks vCPU threads QOS_CLASS_USER_INTERACTIVE so they are scheduled promptly after waking.

Result: idle home screen 65% → 9% of a core, a 60 fps benchmark 56% → 29%, with no change in frame pacing.

2. A userspace interrupt controller behind a global lock

With QEMU's userspace GIC, every interrupt ack, EOI, inter-processor interrupt and timer tick exits to QEMU and takes the big QEMU lock. In a busy game scene there were about 3,800 IPIs per second, while the main loop held that same lock during rendering. Each vCPU spent around 27% of its time waiting.

On macOS 15 and later, Hypervisor.framework has an in-kernel GICv3 (hv_gic_create). Using it:

-accel hvf,kernel-irqchip=on -machine virt,gic-version=3
Enter fullscreen mode Exit fullscreen mode

GICv3, the virtual timer and WFI are handled in the kernel (MSI goes through GICv2m). This is a trimmed port of upstream QEMU 11.1's hvf-arm-gicv3, without migration or snapshot support. Older macOS falls back to the previous path.

Result: a busy in-game scene went from about 150% to 50% CPU; idle vCPU threads sit at 3–5% each.

Memory: returning what the guest no longer uses

macOS MADV_DONTNEED doesn't free anything

On macOS, MADV_DONTNEED only marks pages inactive; the process footprint stays the same. Switching QEMU's discard path to MADV_FREE_REUSABLE actually releases them, and the footprint grows back honestly when the guest touches the pages again.

Android leaves nothing free to report

Free page reporting only returns pages the guest considers free, but Android turns idle memory into file cache, so there is little to report. Instead of a permanent balloon (guest 4K pages vs host 16K pages makes partial returns useless), the emulator:

  • reads /proc/meminfo every 20 seconds;
  • when cold cache (Inactive(file)) exceeds about 280 MB, inflates the balloon in a short pulse (free memory plus up to 512 MB, keeping 300 MB headroom) and deflates it 1–2 seconds later;
  • the pages freed by that pulse are returned to the host in 2 MB chunks by free page reporting.

Measured on a 4 GB instance running a game: physical footprint peaked at 7.4 GB and settled at 5.2 GB; three pulses each reclaimed about 450–570 MB, with no visible effect on frame times.

Summary

Change Scenario Before After
WFI threshold 2 ms → 200 µs, high-QoS vCPUs Idle home screen 65% 9%
In-kernel GICv3 (macOS 15+) Busy game scene ~150% ~50%
MADV_FREE_REUSABLE + balloon pulses 4 GB instance in game 7.4 GB peak 5.2 GB steady

Numbers are from an M5 Max and will vary by machine and game. More at apkplaystudio.betac.cn.

Top comments (0)