I wrote an X11 server in Swift so Linux apps on my Mac get their own Dock icons

MSL is my Mac Subsystem for Linux: type msl <instance> in any terminal and you're in a Linux shell, like WSL. That part is the actual point of the project.

This post is about the part that isn't the point. WSL has WSLg, so I decided MSL should have something similar. That turned into mslgd, an X11 server written from scratch in Swift that puts Linux windows on the Mac as real NSWindows. It's arguably where most of my effort went, it's still labelled experimental, and it taught me more about X11 than I ever planned to know.

The goal: Linux Chromium, with "Chromium" in the Mac menu bar and its own Dock tile. Not "XQuartz", and not "MSL".

Why not just use XQuartz?

The first version did. Phase 1 was literally "X11 over vsock to XQuartz", and it worked: the windows appeared.

But under XQuartz every Linux app is a window that belongs to XQuartz. You get one Dock tile for all of them, one icon, a menu bar that says XQuartz whatever app you're in, and Mission Control stacks them all together. I wanted each window to belong to its app. As far as I could tell, the only way to get there was to be the X server.

The pipe

Before any X11 is interpreted, the bytes have to get out of the VM. That part is deliberately dumb.

Linux app (say, GIMP)
   │  DISPLAY=:0  →  /tmp/.X11-unix/X0
   ▼
x11tunnel          (C, in the guest)  says who the app is, then relays raw bytes
   │  vsock, port 5002
   ▼
mslhd              (Mac, the background service that owns the VM)
   │  reads the app's name, hands the live socket off with SCM_RIGHTS
   ▼
~/Library/Caches/MSL/AppShims/GIMP   (one host process per app)
      its own X server state, NSWindows, Dock tile, menu bar

x11tunnel.c never parses a single X request. It listens where a Linux app expects an X server, and for each client it dials a fresh vsock connection to the Mac and pumps bytes both ways. One X client means one vsock connection, never multiplexed, because X11 expects one full-duplex stream per client.

Everything else happens on the Mac side, in Swift.

Step one: speaking the protocol

X11 is old and very precise. The connection setup reply is the most failure-sensitive part, because every byte has a fixed position in the spec, and one miscounted field means every client fails before sending a single real request. The server also has to reply in whatever byte order the client chose in its first byte (B or l), so every multi-byte write goes through one small byte-order-aware writer.

Getting things "nearly right" doesn't work with X11. Early on I had PolyArc on opcode 67. 67 is actually PolyRectangle, so arc requests fell through unhandled, and any real rectangle request would have been parsed as arc data, at 12 bytes per entry instead of 8. A pixel-comparison test caught it, which is a theme you'll see again.

The Dock problem, and why every app gets its own process

At first, every Linux window lived inside mslhd. That gave me exactly one Dock tile, labelled "mslhd", whose icon changed to whichever Linux app was in front. Apps visibly replaced each other as focus moved.

The reason is simple and unfixable from the inside: macOS gives one Dock tile per NSApplication, and there's no public API for a second one. So per-app tiles mean per-app processes.

Then there's the name. An unbundled executable is named in the Dock, in ⌘-Tab and in NSRunningApplication.localizedName() by its filename. I checked that with a probe binary before building anything on top of it. So there's one host binary, mslgui. MSL copies it to AppShims/Krita, starts that copy, and macOS calls it "Krita" everywhere. There's no .app bundle and no Info.plist.

Grouping is by app, not by connection. Krita opens five X11 connections and several windows, and they all need to land in one tile, so hosts are keyed by name and reused.

Who is this app, anyway?

To route a connection to the right process, I need to know which app it is before serving it. My first attempt was to read ahead in the X11 stream until the client sets WM_CLASS, then route.

That deadlocks, and it's obvious in hindsight. X11 setup is a request and a response: the client sends its setup, then waits. Nothing else arrives until the server answers, and answering means already being the server for that connection. I watched galculator sit there connected while not one byte showed up on the host.

So the guest says who it is instead, and it knows for free. The peer of a unix socket is a real local process, so SO_PEERCRED gives the pid, and /proc/<pid>/cmdline gives the name. x11tunnel sends one small frame before relaying:

'M' 'S' 'L' 'A'  <uint8 length>  <name bytes>

An X11 stream always starts with B or l, never M, so one byte is enough to tell an announcement from a client that didn't send one. Any bytes read while finding out get replayed into the real connection.

Handing over a live socket

mslhd has to keep owning the vsock device, because it's the only thing Virtualization.framework gives a handle to. A file descriptor is per-process, so "send this connection to the Krita process" means asking the kernel to install the descriptor in another process. That's SCM_RIGHTS over a unix socket, sent along with the bytes already consumed.

The fun part: the CMSG_* macros are C macros, so Swift can't see them, and I had to rebuild them by hand. Darwin aligns control messages to 4 bytes, not pointer size. Get that wrong and the kernel silently ignores the control message: the handoff "works", and the descriptor never arrives.

What splitting cost

Splitting one X server into many processes broke things that X11 quietly assumes:

• Resource IDs collided. Each X client gets a range of IDs based on its client index. With one server those were unique. With one server per app, every host started counting from the same base, so two apps' windows had identical IDs. The snapshot test harness names files <windowID>.png, so four running apps overwrote each other's snapshots and seven tests failed comparing one program against another's reference. Fixed by giving each host its own slot in the ID space.
• Copy and paste between apps died. X11 selections are peer-to-peer. The server stores nothing except who owns the selection, and the owner is asked for the data on demand. Two apps in two processes can't see each other at all. The fix was to make NSPasteboard the broker, which also gives copy and paste with real Mac apps for free. It only mirrors CLIPBOARD, not PRIMARY (select-to-copy), because mirroring PRIMARY would overwrite your Mac clipboard every time you dragged across some Linux text. And because AppKit has no clipboard change notification, it polls changeCount, like every clipboard manager on macOS does.

There's no window manager

On Linux, the window manager decides where a new window goes. Toolkits usually ask for (0, 0), meaning "no preference". mslgd has no window manager, so two galculator windows landed at exactly the same spot, one hidden perfectly behind the other. They were both rendering correctly the whole time, which made it look like a rendering bug when it wasn't. The fix is what twm and mwm were already doing in the '90s: cascade new top-level windows.

The bug that made apps vanish

This was MSL's longest-running bug. GIMP (and other GTK apps) would just disappear: no crash report, no signal, no core dump, no log line. The window was simply gone mid-frame.

Every tool I had came up empty. mslgd's logs looked fine, gdb saw nothing, and xcb_connection_has_error() returned 0 every time, because nothing was ever wrong with the socket.

What finally cracked it was xspy, an LD_PRELOAD shim. It replaces GDK's I/O error handler, which turns out to call _exit(1), with one that dumps state and a backtrace, and it logs every syscall on the X socket. At the moment of death, the app had written 672 MB and read 27 MB, with short writes all the way through. Across 31 recorded deaths, every single one was waiting on a reply, and GetInputFocus was the most common unanswered request.

What was happening: mslgd read a request and handled it on the same thread. Any slow handler (a big image, drawing, anything waiting on the main thread) stopped that thread from reading. A GTK app repainting hard filled its send buffer, started getting short writes, and the next time it needed a reply it couldn't flush its queued requests to ask for one. Xlib treats a flush that can't complete as a fatal I/O error, and GDK's handler calls _exit(1). Gone.

The fix is boring, which is how I know it's right: a reader thread that does nothing but pull bytes off the socket, feeding a queue that a separate dispatcher works through in order. The tunnel had a matching problem. It used blocking writes, so while it was busy pushing bytes into the Mac it wasn't reading the replies coming back, and the two sides could jam each other. It's non-blocking now, with a 256 KB buffer instead of 8 KB (a single full repaint of a 1140×740 window is about 3.4 MB).

Side note: I tried letting a local vision model drive GIMP to reproduce the bug. llama3.2-vision wouldn't load at all, and qwen2.5vl answered "10 10" and "100 100" (round numbers unrelated to the screenshot) at 57 seconds per call. A boring deterministic script that churned layers and undo/redo found it much faster.

Making input feel like a Mac

• Scrolling. Real Xorg with libinput reports both XInput 2.1 smooth-scroll values and legacy wheel clicks, and toolkits each pick one: GTK3 uses both, Qt and Chromium read the smooth values, and old apps only see buttons. mslgd does the same, treating the mouse wheel and the trackpad as two devices. The old code truncated each wheel event with Int(dy), so slow wheel ticks under one line did nothing at all.
• The Option key. On a Mac, Option is both the shortcut modifier and the key that types é and ™. Linux splits those across Alt and AltGr, so you get to choose: left Alt and right AltGr (the default), both Alt, or both type special characters.
• Keyboard layouts. The keymap is built from your live Mac layout and switches when you switch it.
• Mac shortcuts (experimental). ⌘C, ⌘V and friends get translated into what Linux apps expect. Some Mac commands have no single Linux key. ⌘⌫ ("delete to start of line") becomes Shift+Home, then BackSpace. Terminals are a special case, because Ctrl+C in a terminal means something very different.
• The close button asks first. Clicking the red button sends WM_DELETE_WINDOW, so unsaved work still gets its "Save changes?" dialog.

Pixels

By default, one X pixel is one Mac point, which is sharp on Retina (each X pixel is exactly 2×2 physical pixels) and is what every app has been tested at. But Linux UIs are roomier than macOS: GNOME's body text is 11pt at 96 dpi, about 14.7 pixels, against the Mac's 13 points. So Linux apps looked right and read big. Native Scaling draws them at 0.85 and reports a correspondingly bigger screen, so apps lay themselves out for the room they really have. Retina rendering goes the other way: half a point per X pixel, so every X pixel is one physical pixel. It's four times the pixels crossing the X11 connection, so it costs speed.

All of mslgd's riskier bits live behind flags, and each one says what it costs.

Small things matter too. Dock icons come from the app's _NET_WM_ICON, which stores straight alpha while Core Graphics wants premultiplied alpha. Get that backwards and nothing crashes, but every icon gets a faint bright halo around its edges. Also, picking the first icon in the list usually lands on the 16×16 one, which looks like garbage as a Dock tile.

OpenGL, and the real GPU

mslgd originally advertised no GLX, so glxinfo, glxgears, Blender and most CAD refused to start. Krita uses EGL, so it was fine, which is why this went unnoticed for so long.

The trick that made it manageable: mslgd renders no OpenGL itself. It implements the setup half of GLX (framebuffer configs, contexts, drawables). Mesa in the guest does the actual rendering, then presents the finished frame through ordinary PutImage. With plain Mesa that's CPU rendering (llvmpipe, OpenGL 4.5).

For the GPU, the guest's Mesa already ships a driver called virpipe that sends Gallium commands over a socket instead of to a GPU device. On the Mac end, virglrenderer (the same library QEMU and UTM use) turns them into Apple's OpenGL on Metal:

Linux app → Mesa (GALLIUM_DRIVER=virpipe) → socket
   → [Mac] virgl_test_server → virglrenderer → OpenGL 4.1 → Metal → GPU

The guest needs no changes at all. In the first proof of concept, es2gears ran at 215 FPS on an M2, rendered by the real GPU. The catch is that Apple's OpenGL stops at 4.1. Blender wants 4.3, so it stays on the CPU path, and that's why the GPU path is an opt-in flag that you can also switch per app.

Testing an X server when you're one person

Yes, there are a lot of tests. I'd never have got this semi-viable by clicking around alone. There are three layers:

Reference images. Small C, Cairo, GTK and Qt programs draw known scenes. The harness snapshots each window and compares it, pixel by pixel, against a reference image.

Not pretty, but every pixel is checked.

xhell, the torture suite. It's 166 tests in Python that speak X11 byte for byte, with no Xlib, so they can send what a real library never would: truncated requests, stale IDs, length lies, 5,000 sibling windows, a 100k-rectangle flood, and replays of real GTK, Qt and Tk sessions. The worst result isn't CRASH, it's DOWN: the test finished, but a fresh connection afterwards couldn't complete a round trip, meaning the whole app host died with it.

It earned its keep straight away. Request handlers read fields with a reader that returns 0 past the end of the body. So a 4-byte DestroyWindow with no body became "destroy window 0", which walked the root window's bookkeeping and recursed until the stack overflowed. One malformed request from any client killed the whole host. Now every core request is checked for minimum length and valid IDs first, and gets the same X errors a real server would give.

Hunting kits. For bugs that only show up after hours of real use, like the vanishing apps, there are scripts that boot an instance, drive a real app, notice a death, grab gdb output plus mslgd's recent wire traffic, and restart.

Where it is now

• About 22,700 lines of Swift across 46 files, plus the 436-line C tunnel. Yes, X11Connection.swift on its own is 12,337 lines. I know.
• No Xlib and no X.Org code underneath. X.Org and XQuartz were reading material, not dependencies.
• It handles the core protocol, RENDER (glyphs, gradients, trapezoids, transforms), SHAPE, RANDR, BIG-REQUESTS, XInput 2 and XKB, GLX, the clipboard bridge, and Linux app menus in the Mac menu bar (experimental, via dbusmenu).
• GTK 3 and 4, Qt, GIMP, Krita, LibreOffice, Chromium, galculator and xeyes all run as Mac windows.
• X11 only. Wayland-only apps won't find a display, so set GDK_BACKEND=x11 or QT_QPA_PLATFORM=xcb.
• Still experimental. Some apps repaint oddly on the GPU path, some draw their menus twice when the menu bar flag is on, and very old toolkits run but look plain.

mslgd is the one part of MSL under GPLv3 (the rest is MIT). It took a lot of effort and a lot of testing hours, so it should stay free: use it, change it, ship it, and share your changes.

MilkmanAbi / MSL

MSL; WSL feel for macOS. Integration as a priority, seamlessness as a philosophy

MSL

Mac Subsystem for Linux
Real Linux on your Mac. A shell in about a second, your files on both sides,
and Linux apps that open as Mac windows with Dock tiles of their own

Getting started  ·  MSL Files  ·  mslgd  ·  The guide  ·  The MSL family

MSL runs real Linux distributions on your Mac — an actual kernel, an actual userland, on Apple's own Virtualization framework. Nothing's emulated, and there's no VM window to babysit. Type msl and you're in. Your home folder's already there. Linux apps show up in the Dock, in Spotlight, in ⌘-Tab, like they always lived there.

It's been over a year getting here, and most of that time was spent on problems I didn't know existed until I hit them. It's free, open source, and still held together by a fair amount of stubbornness. No like genuinely, around 4 months in planning…

View on GitHub

If you've ever written a display server, or tried to, I'd genuinely love to hear what bit you. And if you're the person who asks "why not just use XQuartz", now you know. (◞‸◟)