Discussion on: Running MacOS on Windows 10 with WSL2, KVM and QEMU

[Nicole Stevens]

love this :) many thanks for sharing.
Wish I had something running AMD to help in the hunt for speedup tweaks. Will contact a couple of local user groups in the case anyone has something going spare.

[MichaelF]

I was hoping to join in the optimization testing (3950X w/64GB RAM to play with) but so far I haven't made it past the first reboot when installing. It always stalls at:
"Initialized /dev/rdisk4 as a 10 MB case-insensitive HFS Plus volume".

@pkostelnik I'm using your wsl and basic files... can you share what you changed with the custom kernel?

[Nicole Stevens]

Might be able to help there @pkostelnik didn't use a custom kernel, I think that's what is meant in the first comment, no special build

[Pawel Kostelnik]

it does work with and without the Kernel. I build mine along the tutorial from: Hayden Barns but simply changed from KVM for Intel to KVM for AMD and build it. (not as module).

But it took a couple of hours (i belive) to install the MacOS in the box and also the boot takes about 15min and yes you are not really able to work with but for me it's quite the first step as an engineering sample not even a PoC that it will work. ;-)

After some checks, seems like kvm is installed but not working ... trying now with a custom kernel and KVM_AMD as module!
Edit:
@nicole and @ciparis it seems like KVM (not as Module) is enabled in the system but didn't used. That`s why the catalina is sooo slow.

[Pawel Kostelnik]

just installed over night the insider 20211.1000 now it's a bit faster but still slow:

my ubuntu Ubuntu 20.04.1 LTS linux kernel:

[Pawel Kostelnik]

Ok, next step/s tweaks. Reached up to maybe 30% speedup of the machine:

1. Compiled the actual Kernel for KVM (not Module but including both systems Intel and AMD).
2. set up dbus on debian from: DamionGans on Github. After setting up the dbus you need to Terminate the WSL2 distro with: wsl.exe -t distroname on CDM or PowerShell. In my case Ubuntu as distroname.
3. after next start the dbus incl systemctl will become available.
4. its still hell slow ;-)

[MichaelF]

Mine is FINALLY working!

I ended up bailing on macOS-Simple-KVM (I could just never get past the first reboot). I tried everything -- new kernels, various options, nothing worked.

I found another, more complicated approach using OpenCore instead of Clover that ended up working out. Piling on extra RAM helped greatly with performance, as did using as many AMD processor flags as I could for KVM options. I'm pretty happy with where it ended up (using Catalina) but might try Mojave just to compare performance.

Thanks for this guide, Nicole! This post led me down a rabbit hole but it was a good one.

[Nicole Stevens]

Great news! Many thanks for the update. That's interesting, am still trying to get my hands on an AMD machine to have a go at this.

[Nicole Stevens]

@pkostelnik yeah dbus is a great shout, same IPC etc, did you check the performance gain after the full kernel compile or with both together?

[Pawel Kostelnik]

@nicole Not really because there is only a slightly speedup between this changes maybe 10-15%

[Pawel Kostelnik]

@ciparis would you mind to share your configuration please.

[MichaelF]

Here's the guide I followed:

github.com/kholia/OSX-KVM

I used the 2 networking lines from macOS-Simple-KVM rather than the method he listed, which did not work in WSL2. I've added a few more processor flags to my launch shell script (and changed the name for ease of typing) :

.wslconf borrowed and modified from yours (I'm pretty sure some of those AMD settings are non-functional) --note that the number of processors is inherited from the host and does not need listing here. When building my own kernel I had to use the 104 one -- the available 128 option failed to mount my C drive after building. I probably didn't need to go that route at all, since the current Insider dev build (20211.1005) has nested KVM support out of the box.

[wsl2]
nestedVirtualization=true
kernel=C:\\Users\\myUserPath\\bzImage
kernelCommandLine=amd_iommu=on iommu=pt vfio-pci.ids=1002:66af,1002:ab20 kvm.ignore_msrs=1 kvm-amd.nested=1 kvm-amd.ept=1 kvm-amd.emulate_invalid_guest_state=0 kvm-amd.enable_shadow_vmcs=1 kvm-amd.enable_apicv=1
memory=24GB
localhostForwarding=true

Launch script from this tutorial with my additions to the flags (I think svm is critical for AMD) and adjustments to RAM/Proc

############################################################################
# NOTE: Tweak the "MY_OPTIONS" line in case you are having booting problems!
############################################################################

MY_OPTIONS="+svm,+hypervisor,+msr,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+xsave,+xsaveopt,check"

# This script works for Big Sur, Catalina, Mojave, and High Sierra. Tested with
# macOS 10.15.6, macOS 10.14.6, and macOS 10.13.6

ALLOCATED_RAM="17166" # MiB
CPU_SOCKETS="1"
CPU_CORES="4"
CPU_THREADS="8"

REPO_PATH="./"
OVMF_DIR="."

It's possible (likely?) that I could trace my original installation-reboot issue by comparing the launch file with the one from macOS-simple-kvm.

That list of processor flags provide a big performance gain, but I'm not done yet. My next step is to place all of the AMD processor flags into the options line and then systematically remove the ones that cause errors -- this is similar to the process used for the huge long list of Intel flags at the bottom of the Accelerated KVM Guests post, and my performance with the graphical Ubuntu shell he walks through the installation of at the bottom is fantastic. Granted that's has a much lighter overhead then macOS.

[Pawel Kostelnik]

@ciparis i'm sure there is also a huge difference with the vCPU which he uses for QEMU.
I also use the switch -deamonize to free the WSL console for htop and other monitoring.

[MichaelF]

Here's the list of every AMD CPU flag (from /proc/cpuinfo) that is accepted without error by QEMU:

"+fpu,+vme,+de,+pse,+tsc,+msr,+pae,+mce,+cx8,+apic,+sep,+mtrr,+pge,+mca,+cmov,+pat,+pse36,+clflush,+mmx,+fxsr,+sse,+sse2,+ht,+syscall,+nx,+mmxext,+fxsr_opt,+pdpe1gb,+rdtscp,+lm,+pni,+pclmulqdq,+ssse3,+fma,+cx16,+sse4_1,+sse4_2,+movbe,+popcnt,+aes,+xsave,+avx,+f16c,+rdrand,+hypervisor,+lahf_lm,+cmp_legacy,+svm,+abm,+sse4a,+misalignsse,+3dnowprefetch,+osvw,+topoext,+ssbd,+ibpb,+stibp,+fsgsbase,+bmi1,+avx2,+smep,+bmi2,+rdseed,+adx,+smap,+clflushopt,+clwb,+xsaveopt,+xsavec,+xgetbv1,+xsaves,+clzero,+xsaveerptr,+arat,+npt,+nrip_save,+tsc_scale,+vmcb_clean,+flushbyasid,+decodeassists,+pfthreshold,+umip,+rdpid,check"

Performance remains good; I should probably benchmark between this list and the previous one.

[MichaelF]

@pkostelnik What version of macOS are you using?

I just tried Mojave and it ran much slower than Catalina (the opposite of what I expected). For whatever reason, Catalina is faster on my Ryzen setup. Still laggy, but usable.

[Pawel Kostelnik]

@ciparis trying actually both catalina (which started with clover) and also mojave, but trying now to change to opencore.