DEV Community: Vishwas Prakash

I ported JPEG AI to Apple silicon—and the first benchmark was mostly startup

Vishwas Prakash — Thu, 10 Sep 2026 13:06:55 +0000

My first JPEG AI decode result took almost five seconds. That number looked
terrible next to JPEG—until I measured what those seconds contained.

Only about 0.26 seconds was the reference decoder's codec operation. Model
loading took about 0.83 seconds. Roughly 3.68 seconds went to Python imports,
CUDA initialization, process startup, and I/O.

That distinction became the point of this project: compare JPEG and JPEG AI,
but do it in a way that does not confuse neural inference with cold-start
machinery. Then take the same codec path to Apple silicon using Core ML and a
native entropy coder.

The code, benchmark harness, results, and macOS app are in the
ai_compression repository.

What JPEG AI actually is

JPEG AI is not “run a chatbot on a JPEG.” It is a learned image codec. A neural
analysis transform replaces the hand-designed frequency transform and
quantization pipeline familiar from conventional JPEG. A standards-defined
bitstream still carries the result, and a decoder reconstructs pixels with a
neural synthesis transform.

The JPEG committee lists the core coding system as
ISO/IEC 6048-1:2025. Its
reference software
contains encoder, decoder, training, and evaluation code. The implementation is
available under its included BSD license, although that license explicitly does
not grant patent rights. “Open source implementation” and “unrestricted codec
patent rights” are different claims.

From pixels to `y` and `z`

The simple-profile encoder used here follows this path:

RGB pixels
   ↓ BT.709 YUV conversion
analysis transform
   ↓
y latent ──────────────┐
   ↓ hyper-encoder     │
z hyper-latent         │
   ↓ quantize + code   │
hyper decoders         │
   ↓ scale, mask, ψ    │
four-stage context model
   ↓ predict y + quantize residual
me-tANS entropy coder
   ↓
JPEG AI bitstream

y is the main learned representation. For the luma model in this test it has
160 channels at roughly one-sixteenth the input width and height. It retains
spatial content, but its channels are not red, green, blue, or named features.
The network learns whatever representation minimizes its training objective.

z is a lower-resolution hyper-latent, roughly one-sixty-fourth the input
dimensions. It describes statistics that help the decoder predict and entropy
code y: which values are likely, which positions should be coded, and how to
reconstruct the latent in context. Spending a small number of bits on z can
make the much larger y stream cheaper.

Both traditional JPEG and JPEG AI eventually use entropy coding. The learned
codec's interesting advantage is upstream: it jointly learns the representation
to encode and a richer conditional probability model for that representation.
In simplified form, it estimates P(yᵢ | z, context); a sharper estimate gives
the entropy coder more predictable symbols and therefore shorter codes. This is
learned versus hand-designed modeling, not random versus deterministic decoding.

These are real tensors captured at Core ML model boundaries. The y energy
map still resembles the bird and its surroundings. The z map is much coarser
and sparser. That is expected: it describes coding context, not a thumbnail.

The app can also export every one of the 160 luma y channels, every rounded
z channel, and entropy-mask density. Orange and blue in the channel sheets
mean positive and negative activation. Each channel is normalized separately,
so those colors must not be interpreted as directly comparable magnitudes.

For the fixed network topology, I use Netron to open an
ONNX file such as apple/Models/onnx/tools_2/model_y/analysis.onnx. Netron shows
operators, weights, shapes, and connections. It does not show the values from a
particular inference. The app's tensor export provides that second half.

How the model is trained

Training and inference are separate phases. During training, the encoder and
decoder weights are optimized against a rate-distortion objective commonly
written as:

loss = distortion + β × rate

Distortion penalizes reconstruction error. Rate estimates the bits needed for
the latents. A larger β makes bits more expensive; a smaller one allows a
larger stream to preserve more detail.

The reference recipe trains four base rate-distortion models with beta values
0.002, 0.012, 0.075, and 0.5. Gain parameters and a beta displacement expose
intermediate operating points without shipping a completely separate network
for every bitrate.

The current reference documentation describes 132 optimization epochs per
base model, followed by a statistics stage. It progresses from fixed-rate MSE
training, through mixed MSE/MS-SSIM distortion, to decoder-side variable-rate
training. The last pass records activation-clipping statistics needed by the
codec. See the official
training recipe
and network component guide.

During ordinary encoding or decoding, none of that training happens. The
weights are frozen. Inference is a sequence of tensor transforms plus
quantization and entropy coding.

That also means JPEG AI does not silently improve as people encode more photos.
Per-image probability adaptation is different from changing network weights.
A future domain-specific or continually trained codec could specialize for
medical images, screenshots, satellite imagery, or one camera fleet, but the
encoder and decoder would then need matching model versions. Model distribution,
update bandwidth, privacy, out-of-distribution behavior, and the cost of sending
weights could erase the saved image bits. Federated or personalized codec
training is a real research direction, not behavior provided by today's
standard decoder.

The first rate-distortion result

I ran conventional JPEG at qualities 30, 50, 70, and 90, and JPEG AI at five
reference operating points. The source was one 560×888 image from the JPEG AI
test set.

Near the same rate, JPEG quality 70 produced 1.081 bits per pixel and 34.55 dB
RGB PSNR. JPEG AI point 75 produced 1.092 bits per pixel and 37.04 dB: a 2.49 dB
advantage on this image.

The same pair now also measures 0.9275 versus 0.9536 SSIM-Y, 0.9926 versus
0.9962 MS-SSIM-Y, and 0.0756 versus 0.0680 LPIPS-Alex. Higher is better for
SSIM; lower is better for LPIPS. All three move in JPEG AI's favor here, but
they are still observations from one image rather than population estimates.

This is a smoke result, not a codec-wide conclusion. It has one image, one
measurement per point, no warmup, and GPU contention on the remote machine.
The JPEG baseline also used 4:2:0 chroma while this JPEG AI path used 4:4:4.
A publishable study needs a fixed multi-image corpus, repeated uncontended
runs, confidence intervals, rate-distortion integration such as BD-rate, and
ideally a controlled subjective test. Metrics do not replace looking at the
images.

Why the first timing looked so high

The benchmark preserves total wall time and parses timers printed by the
official reference implementation. That made the hidden startup cost visible.

Across the five operating points, the component medians were:

Stage	Codec operation	Model loading	Python/CUDA/process/I/O
Encode	1.07 s	0.87 s	3.73 s
Decode	0.26 s	0.83 s	3.68 s

Starting a new Python process for every image is valid end-to-end latency, but
it is not steady-state codec throughput. A long-running service, application,
or hardware decoder loads models once and amortizes that cost. Both numbers
matter, so the CSV keeps them in separate columns rather than deleting the cold
start.

Moving the path to Apple silicon

The native implementation converts the reference ONNX graphs to Core ML and
runs them with Apple's compute stack. Swift handles image conversion,
bitstream structure, model orchestration, and reconstruction. A small C++
module ports the reference me-tANS entropy encoder/decoder where exact integer
state transitions matter.

The integer hyper-scale decoder was the sharpest edge. Its output selects an
entropy distribution. A one-index mismatch does not create a slightly different
pixel; it can desynchronize entropy decoding. That stage therefore has to match
the reference exactly, while small floating-point differences in synthesis can
be evaluated with numerical tolerances.

The native encoder produced a 63,599-byte model-2 stream for the test image:
1.023 bits per pixel and 36.51 dB against the source. The official PyTorch
decoder accepted it, and the hyper-latent, scale, mask, and residual control
hashes matched. That verifies bitstream interoperability through entropy
decoding.

The native and official reconstructed PNGs measured 58.37 dB against each
other. Their remaining difference is concentrated at the bottom edge, where
the exported synthesis graph does not yet reproduce the reference decoder's
runtime crop. This is an interoperability milestone, not complete pixel-exact
conformance.

On the Mac, the first native encode took 1.63 seconds and a later encode in the
same process took 0.86 seconds. Those are still provisional single-image
figures, but they show why keeping a model cache inside the app matters.

Try the macOS app and inspect an inference

Open apple/Package.swift in Xcode, select the JPEGAIDecoder scheme and My
Mac, then press Run. Or build the self-contained local app:

cd apple
./build_macos_app.sh
open dist/JPEGAIDecoder.app

Choose Encode PNG, select a rate point, and leave Export y/z inference
visualizations enabled. After the encode, choose View Inference Steps.
The folder is numbered in pipeline order:

input luma;
all y channels;
y activation energy;
all quantized z channels;
z activation energy;
entropy-mask density;
quantized y residual energy after context prediction.

The app also performs a verification decode, displays the reconstruction, and
labels the timing as a first or repeat run for that model. It separately reports
the source PNG, the JPEG AI .bits payload, a native macOS JPEG quality-0.70
baseline, and the decoded PNG. The last file is a lossless preview of the
reconstructed pixels, so it can be larger than both the source PNG and the
compressed payload without indicating a compression failure.

For a quicker command-line trial on an Apple-silicon Mac running macOS 13 or
newer, the experimental encoder and decoder are now available from the
project's Homebrew tap:

brew install vishwas3000/tap/jpeg-ai-apple
jpeg-ai encode input.png output.bits --preset 75 --visualizations steps
jpeg-ai decode output.bits reconstructed.png

The formula builds the native Swift package and installs the converted Core ML
models. The supported presets are 12, 25, 50, 75, and 100. This is
still an untiled 4:4:4 simple-profile experiment, not a general JPEG AI decoder
or a claim of full conformance.

Could this become a video codec?

Encoding every frame independently would work today, just as Motion JPEG does,
but it would waste temporal redundancy. A competitive video codec needs motion
estimation or learned temporal prediction, reference-frame management, random
access, rate control, error resilience, and hardware-friendly scheduling. The
image model is useful research input, not a drop-in video standard.

Where this could go next

The immediate engineering work is measurable rather than speculative:

make native reconstruction bit-exact at every boundary, including the remaining bottom-edge crop;
persist compiled models and add a batch mode so the command-line path does not repay model compilation and loading for every image;
compare Core ML's .all, .cpuAndGPU, and .cpuAndNeuralEngine policies. The current .cpuAndGPU setting explicitly excludes the Neural Engine, but only device measurements can say which policy wins;
remove [Float32]/MLMultiArray copies with reusable Float16 or IOSurface-backed buffers and keep PNG export outside the decode timer;
benchmark a fixed corpus with warm and cold latency, peak memory, model size, energy per image, PSNR, SSIM, MS-SSIM, LPIPS, and BD-rate;
use Instruments signposts to separate Core ML, entropy coding, color conversion, and file I/O on Apple silicon;
test 8-bit weight quantization or palettization, accepting it only if bitstream interoperability and the full rate-distortion corpus still pass;
bring the same decoder to iPhone and compare CPU, GPU, and Neural Engine scheduling without changing the standards-defined result; and
run the published profiles, file format, and conformance material as those pieces mature, rather than treating one reference configuration as the whole standard.

If those foundations hold, the interesting applications extend beyond making
smaller photos:

CDNs and cloud photo libraries: lower transfer and storage at a chosen visual quality, if decode cost is amortized or hardware-accelerated;
low-bandwidth capture: cameras, drones, and remote sensing can spend bits where detail matters, including regions of interest;
on-device media: a standard mobile decoder could make learned compression interoperable instead of tying every app to its own model and bitstream;
machine vision without full pixel reconstruction: the same compressed latent representation can feed downstream analysis. JPEG reported an earth-observation segmentation experiment using nearly ten times fewer model parameters, which is a particularly compelling direction for edge systems;
domain-specific codecs: controlled model versions could target medical, satellite, scientific, or screen content whose statistics differ from the general training set; and
privacy-preserving fleet improvement: federated retraining could improve a later codec model without collecting every original centrally, provided model synchronization and privacy claims are tested rather than assumed.

The highest-value development may therefore be a shared representation for
both people and machines, not merely a better .jpg. That claim needs task
accuracy, power, and latency measurements alongside image quality.

There is no single universally superior codec. If .jpg compatibility is
mandatory, Jpegli is a lower-friction
baseline. JPEG XL and
AVIF are mature modern-image
comparators. JPEG XS targets visually lossless,
line-scale latency rather than maximum distribution compression. For video,
AV2 v1.0
is a new conventional standard, while learned systems such as
MobileNVC
and
DCVC-RT
show that mobile and real-time neural video are plausible research directions.
The next benchmark should include JPEG, Jpegli, JPEG XL, and AVIF instead of
trying to infer a universal winner from JPEG alone.

Could JPEG AI become mainstream?

It has two ingredients many learned-codec experiments lack: a published
standards family and a reference bitstream implementation. That gives software
and hardware vendors a common target.

JPEG AI is now the first published international image coding standard built
around an end-to-end learned codec, not a laboratory proposal. The core system
was published in 2025, while profiling and a file format followed in 2026;
reference-software and conformance work are still moving through publication.

Mainstream adoption still requires fast low-power decoders, stable conformance
tests, manageable model storage, browser and operating-system support, creation
tools, and clear licensing decisions. Conventional JPEG wins today because it
is everywhere and essentially free to decode. JPEG AI must offer enough quality
or functionality to pay for a much more complex decoder.

The next work in this repository is deliberately less glamorous: repair the
remaining crop difference, benchmark a proper corpus with warm caches and no
contention, add energy and memory measurements, add an iOS target, and only
then prepare an upstream contribution and public launch post.

References

JPEG Committee, JPEG AI overview.
ISO, ISO/IEC 6048-1:2025 — JPEG AI core coding system.
JPEG Committee, JPEG AI workplan and published specifications.
JPEG Committee, official JPEG AI reference software.
Esenlik et al., “An Overview of the JPEG AI Learning-Based Image Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 2026.
Reference software documentation, training recipe and neural-network component guide.
JPEG Committee, April 2026 implementation and compressed-domain inference update.
JPEG Committee, January 2026 mobile, video, and energy roadmap.
JPEG Committee, CC0 JPEG AI test dataset.
Apple, Core ML compute-unit options and performance guidance.
Full production and competing-codec research is maintained in the repository.

The .gitignore pattern that stopped matching when I moved a folder

Vishwas Prakash — Wed, 02 Sep 2026 09:54:14 +0000

I moved a directory this week and quietly un-ignored a file containing live
credentials. Nothing failed. Nothing warned. git status simply began offering
to commit a file it had been hiding for months, and if I had run git add -A
without looking, the Supabase URL and key would have gone to a remote.

The cause is one sentence of the gitignore spec that I knew, had never had to
think about, and got caught by anyway.

The move

The repository holds a Swift app. An Android app arrived, so the top level got
lopsided — one platform in a folder, the other spread across App/, Checks/,
Support/ and a couple of scripts at the root. I moved the Swift half under
apple/, so apple/ and android/ sat side by side.

Pure git mv. No file contents changed. All the checks passed, both platforms
built, and it looked like exactly the boring structural commit it was.

Then, out of habit, I looked at what git was proposing to commit:

?? apple/Support/pooled-flight/
?? apple/Support/telemetry.env

Both of those had been ignored an hour earlier.

The rule

Here is the relevant part of the .gitignore:

build/
*.xcodeproj
dist/
Support/telemetry.env
Support/pooled-flight/

Five patterns. Three of them survived the move and two did not, and the
difference is a slash:

If a pattern contains a slash anywhere except at the end, it is treated as
relative to the directory containing the .gitignore file. Otherwise it
matches at any depth.

build/ has a slash only at the end, so it is a name — it matches build/ at
any depth, including apple/build/. *.xcodeproj has no slash at all, so it
matches anywhere too.

Support/telemetry.env has a slash in the middle. That makes it anchored to
the repository root. It means <root>/Support/telemetry.env and nothing else.
The moment the file became apple/Support/telemetry.env, the pattern stopped
describing it, and the file became an ordinary untracked file that git was
helpfully offering to add.

This is correct behaviour. It is documented. It is also completely silent: there
is no warning when a pattern matches nothing, because a pattern matching nothing
is a normal state for most patterns most of the time.

Neither file moved relative to the other, and the .gitignore was not edited. The tree grew a directory above them both, which is all it takes.

Checking, rather than assuming

git check-ignore -v answers the question directly, and it names the pattern and
line number that did the ignoring:

$ git check-ignore -v apple/Support/telemetry.env
                                      # ← no output. Not ignored.

$ git check-ignore -v apple/build
.gitignore:1:build/ apple/build      # ← still fine

Empty output means not ignored. That asymmetry — silence for the dangerous
answer — is worth knowing before you rely on it.

The other question worth asking immediately, and the one I was most anxious
about, is whether the file had ever been committed on any branch at any point:

$ git log --all --oneline -- '*telemetry.env'
                                      # ← empty. Never committed.

It had not. A near miss rather than an incident. If it had not been empty, the
fix is very different and much worse: rotate the credential first, then worry
about rewriting history, in that order and never the other way round.

The fix, and why it is the fix

The obvious repair is to re-point the patterns:

apple/Support/telemetry.env
apple/Support/pooled-flight/

That works today and breaks the next time anything moves. It is the same bug,
rearmed.

The better repair is to stop anchoring them at all:

telemetry.env
pooled-flight/

No slash in the middle, so they match by name at any depth, wherever the tree
goes next. Slightly broader than strictly necessary — any file called
telemetry.env anywhere is now ignored — which for a secret is the correct
direction to be wrong in.

What I would take from this

A gitignore pattern with a slash in it is coupled to your directory
structure. For build output that is fine; if dist/ moves, you find out
because your build breaks. For secrets it is a silent coupling to something you
will eventually change for unrelated reasons, and the failure mode is that the
secret becomes committable at the exact moment you are busy thinking about
something else.

Prefer name-only patterns for anything sensitive. telemetry.env rather than
config/telemetry.env. *.p12 rather than keys/*.p12. You lose a little
precision and gain immunity to every future refactor.

Look at git status after a structural change, specifically for new untracked
files. Not for the files you moved — git shows those as renames and they are
fine. For the ones that appear from nowhere. Anything newly untracked after a
move was previously ignored by an anchored pattern, and that is the entire class
of this bug.

And run git check-ignore -v on your secrets after moving anything. It takes
one command and it is the only way to get a straight answer, because the
dangerous result is the one that prints nothing at all.

AWDL goes quiet every 528 ms, and it looks exactly like your bug

Vishwas Prakash — Wed, 02 Sep 2026 09:52:12 +0000

The pointer stuttered. Not badly, and not always, but enough that everyone who
saw the app noticed it before they noticed anything else. We spent a long time
looking for the cause in our code, and it was not there. It was in the radio,
and the radio was doing exactly what it is designed to do.

AWDL — Apple Wireless Direct Link — is the peer-to-peer Wi-Fi that AirDrop,
AirPlay and MultipeerConnectivity all sit on. It is barely documented. If you
are building anything realtime on MultipeerConnectivity you will meet its
behaviour, and the default assumption when you do is that you have written a
bug. This is what we measured, so that the next person can skip the week.

The link was idle when it stalled

The first suspect was our own video. In ExtendPilot video is over 99% of the
traffic and everything else is noise, so a pointer packet losing a race with a
video burst was the obvious story.

The numbers said no. Joining the host's records to the viewer's on a shared
sequence number puts every stall on the host's timeline, and in the 300 ms
before a typical stall the median amount of video sent was 3.0 KB — about
0.08 Mbit/s — against a run average of 1.58 Mbit/s.

Twenty times below the rate the link was comfortably carrying the rest of the
time. Whatever stopped the pointer was not us competing with ourselves.

The stalls were periodic

The intervals between them read 528 → 528 → 528 → 512 → 528 → 528. Across 97
episodes the median interval was 528 ms.

Networks do not usually fail on a metronome. Contention is bursty, interference
is bursty, a neighbour's microwave is bursty. A number that repeats to within a
few milliseconds is a schedule, and a schedule belongs to something that has a
clock.

AWDL synchronises its peers on a period of 512 TU. A TU is a time unit from
the 802.11 spec — 1024 microseconds — so 512 TU is 524.288 ms.

Nothing is dropped and nothing is retried. The update is simply not sent until the radio comes back, which is why every loss metric we had reported a healthy link.

528 against 524. That match is close enough that the conclusion is hard to
avoid: these are the radio's own availability windows, and our packets were
waiting for the next one.

Why the radio has windows at all

AWDL works by channel-hopping. The device has one antenna, and it time-slices
between the infrastructure channel — the one your router is on — and the AWDL
social channel where peers find each other. Peers agree a schedule so they are
awake at the same moments, and between those moments the peer-to-peer link
simply is not there.

That is not a fault. It is how a single radio serves two networks at once. But
it means peer-to-peer Wi-Fi has periodic latency spikes by design, and any
protocol you run over it inherits them.

Three reasons it presents as your bug

You did not choose the radio, and you are not told which one you got.
MultipeerConnectivity picks between infrastructure Wi-Fi, AWDL, and Ethernet on
macOS. The framework decides; there is no API that reports the decision. The
same code on the same two devices can behave differently on Tuesday because the
link underneath it changed.

Reliable delivery hides the stall and then hands you a burst. A .reliable
send is ordered, so a radio that goes away for 300 ms does not drop your
frames — it queues them, and delivers the whole backlog the moment the window
opens. On glass that is freeze, then fast-forward, then normal. Delivery ratio
stays around 99% throughout. Every loss-shaped metric reads healthy while the
picture is visibly wrong, which is why loss was never the explanation and why
we kept looking somewhere else.

The delay distribution is bimodal, and averages erase it. Half our packets
arrived within 4 ms of the best transit ever seen. Fifteen per cent arrived
more than 120 ms behind it. There is very little in between. A packet either
goes straight through or it waits for the next window — and a mean transit time
computed over both populations describes neither of them, and looks fine.

You cannot prevent it, only absorb it

Nothing in an app can stop a radio going quiet on its own schedule. There is no
flag, no priority class, no way to ask for the window. The only thing available
is to stop letting the radio's timing become the display's timing.

That means a jitter buffer: hold arriving data briefly, then play it out on the
sender's clock rather than on arrival. Jitter smaller than the hold becomes
invisible. Ours is adaptive, 50–160 ms, and it took frozen pointer frames from
12.8% to 2.7%.

It is worth being clear about what that fix does not do. The stalls are still
there, exactly as often. We are paying up to 160 ms of latency to make them
stop being visible, which is a trade rather than a repair.

One more consequence, if you were thinking of routing around it: moving video
onto its own Network.framework lane would not help, because both lanes ride the
same radio.

What to do about this in your own app

Find out which radio you are actually on. This takes one command on the Mac,
during a live session:

netstat -I awdl0 -w 1

Non-zero traffic on awdl0 means peer-to-peer Wi-Fi is carrying your session.
Zero means it is not, and your periodic stalls are something else. This is the
measurement to take first, before any tuning, because it decides which half of
the problem space you are in. nettop -m route -t wifi -J bytes_in,bytes_out
gives you the same answer from the other direction.

Run the same session twice, with the phone's Wi-Fi off and on. Wi-Fi off
forces pure AWDL. Wi-Fi on and joined to the same SSID as the Mac allows
infrastructure. If the stutter has a different texture between the two runs,
the radio is in your causal chain. This costs two minutes and it is the closest
thing to an A/B test available on a link you cannot select.

Do not let your recovery path fire on the radio's period. If a stall
triggers a retry, and the stall is periodic, so are the retries — and a retry
sent to a peer that is not currently listening is pure added load. We caused
exactly this once by dropping a recovery interval to a flat 150 ms and produced
360 requests in 107 seconds, each one making the next hole more likely. Back
off exponentially, and treat any recovery message that must be sent reliably as
expensive.

A fixed jitter buffer is the wrong shape for a bimodal input. This is the
part that generalises furthest. Two regimes that want opposite things cannot be
served by one constant. Sized for the common case — 80 ms, comfortably over the
4 ms that half our packets needed — the buffer ran dry on every single stall.
Sized for the tail at 250 ms, it worked, and charged every user a quarter of a
second of lag permanently to handle a case that occurs 15% of the time.

Make it adaptive, and make it asymmetric: rise fast, decay slow. Reacting late
to a worsening link costs a visible freeze; lingering on a recovering one costs
a little lag nobody notices. It is the same shape as TCP's AIMD, for the same
reason.

Check whether your metric can see the failure at all. Ours could not.
Delivery ratio said 99%, the send backlog said healthy, and the picture stalled
anyway. If your instrumentation counts what arrives, it cannot describe silence.

Caveats on all of the above: one device pair, one link, 60-second runs. The
528 ms figure is ours, on our hardware. The 512 TU period is AWDL's, and yours
will be the same.

ExtendPilot shares a Mac's screen with the iPhones and iPads already in the room — as a mirror, or as a second desktop macOS treats as real hardware. Peer to peer over your own Wi-Fi, no account, no server in a session, so it works with the router unplugged. Built for the meeting room whose HDMI cable never works, and for the iPad on the desk doing nothing. The Mac app is a free notarised download, or brew install vishwas3000/tap/extendpilot; the iPhone and iPad app is with App Review. Every performance claim about it traces to a session like the one above, written down with the number that was true before and the number that was true after. This one is a number we could not change, only work around.

You don't need a remote desktop for the room you're standing in

Vishwas Prakash — Wed, 02 Sep 2026 09:51:44 +0000

Someone plugs in the HDMI cable. The room's display shows nothing, or shows 1024×768, or shows the desktop of whoever presented last week. After a minute of this, somebody says "just share your screen", and out comes AnyDesk or TeamViewer.

It works. It is also the wrong shape for the problem, and noticing why turns out to be more interesting than it sounds.

What remote desktop tools are actually for

AnyDesk, TeamViewer and RustDesk exist to solve one problem well: reach a machine you are not near. Your parent's laptop. A server in a rack. A colleague's desktop three time zones away. Everything about their design follows from that.

One person connects to one machine. That person takes the mouse. The remote screen is mirrored to them, and the whole session is framed as control — because when you are not in the room, control is the only way to do anything.

Being fair about the privacy question, because it is the claim people reach for first and it is wrong: AnyDesk has a LAN mode, and in it a session goes directly between the two machines without the internet. RustDesk can be self-hosted entirely on your own infrastructure. Neither of these tools forces your screen through somebody's cloud if you configure them not to. If you have read that they do, that is not accurate.

The mismatch is not privacy. It is shape.

The relay, the account and the NAT traversal are not overhead — they are the product, and they are what makes reaching a machine in another country possible at all. Standing beside the machine, you pay for all three and use none of them.

Where the shape stops fitting

Stand in a room with four people and a Mac, and three things go wrong at once.

It is one-to-one. Remote desktop is a session between two endpoints. Four people looking at one screen is not what it models, so three of them read over a shoulder.

It mirrors, and mirroring is sometimes the wrong answer. Every tool in this category shows a screen that already exists. But the meeting-room problem is often the opposite: you want a screen you do not have — somewhere to put the thing you are talking about while your laptop keeps the notes. No remote desktop tool creates a display, because from three time zones away that idea makes no sense.

Control is the default, when nobody wanted control. Handing over the mouse is the whole premise. In a room, everyone wants to point at something and one person occasionally wants to click.

What else is out there

Honestly, and including the ones that do the job better than we do.

	What it is	Account	Cost
Sidecar	Apple's own. iPad as a real second display, mirror or extend	Same Apple Account on both	Free, built in
Apple Screen Sharing	Mac to Mac over the LAN, built into macOS	Local account on the target	Free, built in
Deskreen	Open source. Streams a screen or one window to any browser on the LAN	No	Free, AGPL-3.0
AnyDesk	Remote desktop, LAN mode available	Yes for most use	Free tier capped at 3 devices
TeamViewer	Remote desktop, LAN mode available	Yes for most use	Free personal; commercial use cuts sessions at 5 minutes
RustDesk	Open source remote desktop, self-hostable	Optional if self-hosted	Free
Duet Display	iPad as a second display, wired or wireless	Yes	Paid
Luna Display	iPad as a second display, via a hardware dongle	Yes	Paid + hardware

If you have an iPad and a Mac and both are on your Apple Account, use Sidecar. It is free, it is built in, it extends rather than mirrors, and it is genuinely good. Anyone telling you otherwise is selling something.

And if you want no account at all, look at Deskreen before you look at us. It is the one on that list closest to the same argument — free, open source, local network, nothing to sign in to, and the viewing device needs no app because the viewer is a browser tab. It is 21,000 stars and still shipping; v3.2.16 went out in July 2026. For getting a screen onto the other devices in a room it is an excellent answer and it costs nothing to try.

Where it differs is the same line the rest of this post is about. Deskreen streams a screen or a single window into a browser, so what the iPad shows is a picture of something the Mac was already displaying. It cannot give macOS a display that is not there — a browser tab is not a monitor the window server can place windows on — which is the thing that decides whether you are mirroring a screen or gaining one.

That distinction is worth being precise about, because "second screen" gets used for both and they are not the same feature.

Sidecar stops being the answer when the second screen belongs to someone else — a colleague's iPad, a client's iPhone, a device that is not on your Apple Account. That is a different problem, and it is the one we were trying to solve.

Where ours sits

ExtendPilot is built for the room rather than the continent, and the design falls out of that in three ways.

Several people watch at once. Everyone in the room joins the same screen. They can point at it, draw on it, and react, all at the same time, each in their own colour. Only one person can be handed the pointer, and that is a deliberate grant the presenter makes and takes back by touching their own mouse.

It can extend, not only mirror. It creates a display that macOS treats as real hardware — arrangeable in System Settings, HiDPI, windows move onto it. That is the part remote desktop tools structurally cannot do, and it is also why it cannot be in the Mac App Store.

No account, and no server in a session. Devices find each other with Bonjour and connect peer to peer. There is nothing to sign in to, and a session works with the router unplugged. Not as a policy — there is simply no infrastructure in the path to send it through.

It is free. The Mac app is a notarised download from extendpilot.com, or brew install vishwas3000/tap/extendpilot; the iPhone and iPad app is with App Review.

When not to use it

Worth saying, because a comparison that recommends its author's software in every case is not a comparison.

You are not in the room. Use AnyDesk, RustDesk or Screens. Ours needs both devices on the same Wi-Fi and offers nothing over the internet.
You need unattended access to a machine nobody is sitting at. Not a thing it does. MeshCentral or RustDesk.
You want to share from Windows or Linux. Sharing is macOS only. Watching will come to Android; sharing will not.
Both devices are yours and on one Apple Account. Sidecar is right there.

The useful question is not which tool is best. It is whether the thing you are doing is remote at all — and in a meeting room, it usually is not.

The Mac got slower, and quitting the app didn't fix it

Vishwas Prakash — Tue, 01 Sep 2026 14:48:35 +0000

This is the worst bug I have written. Not because the app broke — it worked fine throughout — but because it degraded the machine it ran on, permanently, and left the damage behind after quitting. Someone could have uninstalled the app and still been slow, with nothing to connect the two.

ExtendPilot puts a real second desktop on an iPad: not a mirror, but a display macOS treats as hardware, arrangeable in System Settings, with windows that move onto it. There is no public API for that, so it sits on a private one. This is the story of what the private one does when you hold it wrong.

What it looked like

After a day of development the Mac had a load average around 6 and felt unusable. Three ColorSync daemons were burning about 115% CPU between them — over a full core — and had been for 47 minutes at the point of measurement.

$ top -l 2 -o cpu
colorsyncd                 46.5%
colorsync.displayservices  35.6%
colorsync.useragent        33.2%

They were not logging anything. A full log show came back empty, so nothing pointed at them at all. Two of the three run as root; sampling the one running as the user gave the hot stack immediately.

$ sample 96211 3
  21  ColorSyncXPCDeviceRegistryCopyAnyUserInfo
  20  ColorSyncDeviceRegistryCopyInfo
  11  ColorSyncIterateDeviceProfiles
  10  ColorSyncProfileCreateWithURLAndOptions
   7  ColorSyncCreateCFDataFromURL
  40  __CFBinaryPlistCreateObjectFiltered

Iterating the display device registry and re-reading profiles off disk, in a loop. So the question became how big the registry had got.

$ ls -1 /Library/ColorSync/Profiles/Displays/ | wc -l
97
$ ls -1 /Library/ColorSync/Profiles/Displays/ | grep -c LocalScreenShare
72
$ find /Library/ColorSync/Profiles/Displays/ -mtime -1 | wc -l
77

Seventy-two of ninety-seven profiles were ours, and seventy-seven of them had been created that day. Every one was an ICC profile that colorsyncd had written for a monitor it had never seen before. We had been handing it a brand new monitor every few minutes, and it had been dutifully profiling each one and keeping the result forever.

The line not falling at the quit marker is the part that made this hard to find. Every instinct says a leak ends when the process does, so the machine stayed slow long after the obvious suspect had exited.

The cause is one line

A virtual display is created from a descriptor. Three of its fields build the display's identity:

static uint32_t nextIndex = 0;
descriptor.productID = 0x4C53;
descriptor.vendorID  = 0x3456;
descriptor.serialNum = ((uint32_t)getpid() << 8) | (nextIndex++ & 0xFF);

The vendorID / productID / serialNum triple is a display's identity to macOS. Mixing in getpid() meant that identity changed on every launch. macOS concluded, correctly, that it had never seen this monitor before, and colorsyncd did exactly what it is supposed to do for a new monitor: profile it, and keep the profile. Around 36 launches, two desktops each, is 72 profiles.

The bitter part is that this line was itself a fix. An earlier version gave every display the same serial number, and macOS then treated two simultaneous desktops as one monitor — each inheriting the other's saved arrangement and coming up at the wrong size. Adding the pid made them distinct, and it worked. It also made them distinct across time, which nobody had asked for and nobody noticed.

An identity that had to be stable was derived from something ephemeral. The requirement was "distinct from its siblings". The implementation delivered "distinct from everything, forever".

The first fix was not enough

Replacing the pid with a hash of the display's own size and slot fixes the across-launch case, because both of those are properties of the desktop rather than of the process that made it:

descriptor.serialNum = ((width & 0x1FFF) << 19) | ((height & 0x1FFF) << 6) | slot;

But slot still came from nextIndex++, a counter that only ever goes up. Within a single session, removing a desktop and adding another advanced it — and applying a saved preset does exactly that, removing every desktop and recreating them. Applying a preset three times minted six identities without the app ever restarting.

The file timestamps said so plainly: profiles written at 00:45, 00:58, 01:04, 01:06 and 01:13. Minutes apart, inside one session. The fix is that the slot is now passed in by the caller, which already knows each desktop's position in its own list — it does not need a counter to tell it.

A counter that only increments is the subtle version of this bug. It looks stable because it is deterministic. A counter is a clock in disguise.

Neither was the second

Creating a display can fail if another process already holds that identity, so there is a fallback path. In the same commit that fixed the two problems above, the fallback read:

if (!display) {
    descriptor.serialNum = ((uint32_t)getpid() << 8) | (slot & 0xFF);   // ← pid again
    …
}

The same mistake, a third time, written by someone who had spent the previous hour understanding why it was a mistake. It fires rarely, but when it fires it is unbounded. It is now four bounded alternates, so a given size costs at most four profiles ever, rather than one per launch:

for (uint32_t attempt = 0; attempt < 4 && !display; attempt++) {
    descriptor.serialNum = ((width & 0x1FFF) << 19) | ((height & 0x1FFF) << 6)
                         | ((slot & 0x0F) | (attempt << 4));
    display = [[displayClass alloc] initWithDescriptor:descriptor];
}

Three passes to get one line right. Worth remembering the next time a fix feels obvious.

A postmortem without a check is just a story

The regression check creates the two desktops, releases them, counts the profiles on disk, creates the same two again, releases them, and counts again. If a repeat run ever mints a new identity, the number moves and the check fails.

display profiles on disk: 27 before the repeat, 27 after

Writing that check turned up a second bug worth recording, because it is a good example of a test being right for the wrong reason. The first version passed the displays back through out-parameters. Under ARC those are __autoreleasing, so the pool held the displays alive well past the point where the test released them, and the check failed with "a virtual display outlived its owner". The check was correct. The check's own helper was the thing leaking. Each round now gets its own @autoreleasepool, and there is a comment next to it explaining why no helper exists.

The code fix stops new profiles. It cannot remove the seventy-two already written, which are root-owned and have to be deleted by hand. That asymmetry is the part I would most like other people to take away from this: the damage outlived the process, the fix does not reach backwards, and no amount of care after the fact recovers it.

What to look for in your own code

Identity derived from something ephemeral. Anything the OS persists or registers under a key you generate must have that key derived only from what genuinely distinguishes the thing. Audit any use of getpid(), a timestamp, a UUID, a random number or a monotonic counter that feeds a serial number, a filename, a bundle id, a service name, a Keychain account, a Bonjour name or a cache key.

The question that finds it: if this runs a thousand times, how many distinct values does the OS end up storing? If the answer is a thousand, this is your bug.

Resources the OS keeps after you exit. Virtual displays go away when released; the profiles do not. Before using an API that registers something system-wide, ask what survives the process, and whether anything ever cleans it up. In our case nothing did, and still does not — the fix is that we stop creating them, not that we tidy up.

Silent hot loops. colorsyncd logged nothing at all while spinning. The only thing that found it was sampling the hot stack. When something is eating CPU and the logs are quiet, sample it rather than reasoning about it. And note that ps reports %CPU as a lifetime average rather than a current reading — a 47-minute-old process showing 73% in top -l 2 has been there the whole time, which is a far stronger signal than a spike.