The pointer stuttered. Not badly, and not always, but enough that everyone who
saw the app noticed it before they noticed anything else. We spent a long time
looking for the cause in our code, and it was not there. It was in the radio,
and the radio was doing exactly what it is designed to do.
AWDL — Apple Wireless Direct Link — is the peer-to-peer Wi-Fi that AirDrop,
AirPlay and MultipeerConnectivity all sit on. It is barely documented. If you
are building anything realtime on MultipeerConnectivity you will meet its
behaviour, and the default assumption when you do is that you have written a
bug. This is what we measured, so that the next person can skip the week.
The link was idle when it stalled
The first suspect was our own video. In ExtendPilot video is over 99% of the
traffic and everything else is noise, so a pointer packet losing a race with a
video burst was the obvious story.
The numbers said no. Joining the host's records to the viewer's on a shared
sequence number puts every stall on the host's timeline, and in the 300 ms
before a typical stall the median amount of video sent was 3.0 KB — about
0.08 Mbit/s — against a run average of 1.58 Mbit/s.
Twenty times below the rate the link was comfortably carrying the rest of the
time. Whatever stopped the pointer was not us competing with ourselves.
The stalls were periodic
The intervals between them read 528 → 528 → 528 → 512 → 528 → 528. Across 97
episodes the median interval was 528 ms.
Networks do not usually fail on a metronome. Contention is bursty, interference
is bursty, a neighbour's microwave is bursty. A number that repeats to within a
few milliseconds is a schedule, and a schedule belongs to something that has a
clock.
AWDL synchronises its peers on a period of 512 TU. A TU is a time unit from
the 802.11 spec — 1024 microseconds — so 512 TU is 524.288 ms.
Nothing is dropped and nothing is retried. The update is simply not sent until the radio comes back, which is why every loss metric we had reported a healthy link.
528 against 524. That match is close enough that the conclusion is hard to
avoid: these are the radio's own availability windows, and our packets were
waiting for the next one.
Why the radio has windows at all
AWDL works by channel-hopping. The device has one antenna, and it time-slices
between the infrastructure channel — the one your router is on — and the AWDL
social channel where peers find each other. Peers agree a schedule so they are
awake at the same moments, and between those moments the peer-to-peer link
simply is not there.
That is not a fault. It is how a single radio serves two networks at once. But
it means peer-to-peer Wi-Fi has periodic latency spikes by design, and any
protocol you run over it inherits them.
Three reasons it presents as your bug
You did not choose the radio, and you are not told which one you got.
MultipeerConnectivity picks between infrastructure Wi-Fi, AWDL, and Ethernet on
macOS. The framework decides; there is no API that reports the decision. The
same code on the same two devices can behave differently on Tuesday because the
link underneath it changed.
Reliable delivery hides the stall and then hands you a burst. A .reliable
send is ordered, so a radio that goes away for 300 ms does not drop your
frames — it queues them, and delivers the whole backlog the moment the window
opens. On glass that is freeze, then fast-forward, then normal. Delivery ratio
stays around 99% throughout. Every loss-shaped metric reads healthy while the
picture is visibly wrong, which is why loss was never the explanation and why
we kept looking somewhere else.
The delay distribution is bimodal, and averages erase it. Half our packets
arrived within 4 ms of the best transit ever seen. Fifteen per cent arrived
more than 120 ms behind it. There is very little in between. A packet either
goes straight through or it waits for the next window — and a mean transit time
computed over both populations describes neither of them, and looks fine.
You cannot prevent it, only absorb it
Nothing in an app can stop a radio going quiet on its own schedule. There is no
flag, no priority class, no way to ask for the window. The only thing available
is to stop letting the radio's timing become the display's timing.
That means a jitter buffer: hold arriving data briefly, then play it out on the
sender's clock rather than on arrival. Jitter smaller than the hold becomes
invisible. Ours is adaptive, 50–160 ms, and it took frozen pointer frames from
12.8% to 2.7%.
It is worth being clear about what that fix does not do. The stalls are still
there, exactly as often. We are paying up to 160 ms of latency to make them
stop being visible, which is a trade rather than a repair.
One more consequence, if you were thinking of routing around it: moving video
onto its own Network.framework lane would not help, because both lanes ride the
same radio.
What to do about this in your own app
Find out which radio you are actually on. This takes one command on the Mac,
during a live session:
netstat -I awdl0 -w 1
Non-zero traffic on awdl0 means peer-to-peer Wi-Fi is carrying your session.
Zero means it is not, and your periodic stalls are something else. This is the
measurement to take first, before any tuning, because it decides which half of
the problem space you are in. nettop -m route -t wifi -J bytes_in,bytes_out
gives you the same answer from the other direction.
Run the same session twice, with the phone's Wi-Fi off and on. Wi-Fi off
forces pure AWDL. Wi-Fi on and joined to the same SSID as the Mac allows
infrastructure. If the stutter has a different texture between the two runs,
the radio is in your causal chain. This costs two minutes and it is the closest
thing to an A/B test available on a link you cannot select.
Do not let your recovery path fire on the radio's period. If a stall
triggers a retry, and the stall is periodic, so are the retries — and a retry
sent to a peer that is not currently listening is pure added load. We caused
exactly this once by dropping a recovery interval to a flat 150 ms and produced
360 requests in 107 seconds, each one making the next hole more likely. Back
off exponentially, and treat any recovery message that must be sent reliably as
expensive.
A fixed jitter buffer is the wrong shape for a bimodal input. This is the
part that generalises furthest. Two regimes that want opposite things cannot be
served by one constant. Sized for the common case — 80 ms, comfortably over the
4 ms that half our packets needed — the buffer ran dry on every single stall.
Sized for the tail at 250 ms, it worked, and charged every user a quarter of a
second of lag permanently to handle a case that occurs 15% of the time.
Make it adaptive, and make it asymmetric: rise fast, decay slow. Reacting late
to a worsening link costs a visible freeze; lingering on a recovering one costs
a little lag nobody notices. It is the same shape as TCP's AIMD, for the same
reason.
Check whether your metric can see the failure at all. Ours could not.
Delivery ratio said 99%, the send backlog said healthy, and the picture stalled
anyway. If your instrumentation counts what arrives, it cannot describe silence.
Caveats on all of the above: one device pair, one link, 60-second runs. The
528 ms figure is ours, on our hardware. The 512 TU period is AWDL's, and yours
will be the same.
ExtendPilot shares a Mac's screen with the iPhones and iPads already in the room — as a mirror, or as a second desktop macOS treats as real hardware. Peer to peer over your own Wi-Fi, no account, no server in a session, so it works with the router unplugged. Built for the meeting room whose HDMI cable never works, and for the iPad on the desk doing nothing. The Mac app is a free notarised download, or brew install vishwas3000/tap/extendpilot; the iPhone and iPad app is with App Review. Every performance claim about it traces to a session like the one above, written down with the number that was true before and the number that was true after. This one is a number we could not change, only work around.
Top comments (0)