At the end of September, Destroy Any Website by Hugo Duprez went viral: a little guy with a jetpack and a gun wrecks whatever page you're on. I wanted the same thing for the apps I actually stare at all day, like Slack, Jira, Xcode and my inbox. The catch is that on a web page the DOM knows where every letter is, while a Mac desktop gives you a bitmap and nothing else.
So Destroy Any Desktop works from a single screenshot. When a match starts, it captures the display, works out which pixels are letters, buttons, icons or chart bars, and turns each of them into a piece of the level you can stand on and shoot out. Everything from the keyboard shortcut to the first explosion has to fit in two seconds, and the analysis gets 400 ms of that on a base M1.
Why not OCR or the Accessibility API
OCR (Apple's Vision does it well, on the device) gives you boxes for lines and words. I need every letter as its own breakable piece, plus the icons, avatars and buttons that OCR ignores, and when a letter gets knocked out its exact pixels have to disappear, not a box around it. The Accessibility API means one more permission prompt, varies a lot from app to app and still hands you frames rather than pixels.
So the detector looks only at pixels: about 700 lines of Swift, no dependencies, running on the CPU. It outputs a list of elements (a kind, a rectangle and a pixel count for each) and a label map with one UInt32 per pixel saying which element it belongs to. Game logic sees only the list, and the renderer uses the label map to erase exactly the right pixels.
Step 1: find the background
A screen doesn't have one background color. A dark sidebar sits next to a white document, which sits next to a wallpaper photo. So the screen is cut into 24-point tiles, a bit taller than a line of text, and each tile's background is its most frequent color. To keep this cheap on a 7.7-megapixel frame, colors are bucketed to 4 bits per channel and counted in a flat array of 4,096 slots.
Step 2: mask everything that isn't background
A pixel is foreground if the sum of its R, G and B differences from the background is above 60, out of 765. Two tweaks make this work on real screens. Icons and avatars bigger than a tile become that tile's background, so there's a second, coarser model with 96-point tiles, and a pixel counts if it stands out at either scale. And a window edge running through a tile would light up as a fake stripe, so each pixel is compared with the backgrounds of its own tile and all eight neighbours, and it counts only if it differs from every one.
Step 3: letters, and the dots on them
Two-pass connected-component labeling with union-find gives you one component per letter, mostly. "i", "j", "é" and ";" are two components each, and if the dot were its own element you could shoot the dot off an i, which is funny exactly once. So small components sitting just above a bigger one get glued to it. On very busy screens, like a big spreadsheet with more than 6,000 elements, letters are merged into words instead.
In union, the smaller root always becomes the parent, so the same screenshot always gives the same level. That keeps the tests stable.
Step 4: classify by shape
An element's kind decides its HP and whether it can hold other elements up. It comes from geometry alone:
static func kind(width: Float, height: Float, fill: Float, maxTextHeight: Float) -> ElementKind {
if width <= 3 || height <= 3 { return width > 12 || height > 12 ? .line : .glyph }
if height <= maxTextHeight && fill < 0.8 { return width > height * 2.5 ? .word : .glyph }
if fill >= 0.6 && width >= height * 2.5 && height <= maxTextHeight * 1.5 { return .control }
return .image
}
fill is the share of the bounding box covered by the element's own pixels. A letter is mostly empty space, while a button is a filled pill, wider than it is tall. It's crude, but the game only needs a handful of classes: a glyph has 8 HP, a word 20, a control 35 and an image 50. All thresholds are in points and scaled to the display, so a Retina screen and a 1x monitor give the same level.
Three things the basic pipeline misses
Halos
macOS anti-aliases text, and the soft edge of a letter is too faint for the mask. Knock out a letter and its ghost outline stays behind. So after labeling, each label grows onto neighbouring pixels that differ from the background by more than 12, one pass per unit of display scale.
Windows that become one element
A window's title bar, borders and separators are often one connected net of pixels. Left alone, that net becomes a single element the size of the window: a solid block for collisions, the support for everything inside it, and half of the level's destruction percentage. So anything bigger than 240 points on a side is cut into 64-point tiles.
Cards you can't break
A card, a table cell or a colored banner bigger than a tile becomes the background, which leaves breakable text sitting on an indestructible wall. A separate pass runs union-find over the background pixels, joining neighbours within 6 of each other so a gradient stays whole, and keeps regions that look like panels: at least 20 points on the short side, at most 4% of the screen, not touching its edge, and straight on at least three sides. Blobs on a wallpaper photo are never that straight.
From pixels to gameplay
Elements are one-way platforms, as in Destroy Any Website: you jump up through them and land on top. Words and buttons that sit inside a bigger image or control fall with it, but text never holds up text, or one shot into a sentence would bring down the whole line. When an element runs out of HP, the renderer copies its pixels into a sprite, clears them from the canvas by label and drops the sprite as debris, so the pieces flying off are the actual letters you shot.
Two more things make it look like the desktop itself is breaking. The game also captures the wallpaper and each window separately and works out what's under the top window at every pixel, so a letter shot out of a window reveals the window behind it. Only craters go through to space. And the window body (padding and plain background) is split into plates of about 90 points, with each pixel going to the nearest of a grid of random seeds, which gives straight edges like broken glass. A plate cracks once 2% of its pixels are destroyed and falls off at 30%.
Testing without real screenshots
There are no real screenshots in the repo. A generator draws fake desktops instead (wallpaper, menu bar, Dock, and chat, kanban, code, spreadsheet and mail windows) with CoreText, so it knows the true label map. The tests require that more than 85% of the ink pixels end up in some element, that more than 95% of found elements are at least 90% one true element, and that the same input always gives the same result. The game's five built-in maps come from the same generator.
Numbers
A 3456×2234 frame, release build, one thread, M4 Pro:
| Screen | Elements | Time |
|---|---|---|
| Chat | 796 | 151 ms |
| Code editor | 999 | 152 ms |
| Spreadsheet | 2,258 | 178 ms |
| Random desktop, 2 to 5 windows | 3,025 | 165 ms |
The time tracks resolution more than how busy the screen is: a 14-inch frame is about 25% faster, which matches its pixel count. With that much room under the 400 ms budget, the Metal version of the mask and Vision as a second opinion on word boxes never got built. Even the canvas runs on the CPU, in 256-pixel tiles, and only the tiles that changed get redrawn.
Try it
Destroy Any Desktop is on destroyanydesktop.com, for macOS 14 and later, and it costs nothing to play. Thanks to Hugo for Destroy Any Website and for the OK to credit him.
How would you have done it: pixels, OCR or the Accessibility tree? And if you find an app that confuses the detector, I'd like to know which one.
Top comments (1)
Việc bỏ qua bước OCR để xử lý trực tiếp từ screenshot là một hướng đi cực kỳ thông minh để giảm độ trễ. Mình từng gặp vấn đề tương tự khi xây dựng các pipeline tự động hóa quy trình làm việc, nơi mà việc chờ đợi mô hình OCR trích xuất text đôi khi làm gãy luồng xử lý dữ liệu liên tục. Thay vì phụ thuộc hoàn toàn vào việc nhận diện ký tự, việc map trực tiếp tọa độ hình ảnh giúp engine phản hồi mượt mà hơn hẳn. Mình thường xử lý các tác vụ nặng liên quan đến việc quản lý nhiều tài khoản AI cùng lúc để test thử các workflow tự động này thông qua LabAgent, site: Labagent giúp tiết kiệm đáng kể thời gian chờ đợi quota của các model lớn.