DEV Community: xbill

The Desktop Looked Right: 2,093 Parse Errors a Boot, and 10.9 Seconds That Weren't Doing Anything

xbill — Tue, 08 Sep 2026 18:05:25 +0000

This article walks through one afternoon of Debian system management, in which the machine turned out to have been reporting two of its own defects continuously, for as long as it had been configured, to nobody.

Debian tells you almost everything. It just does not tell you on screen. The kernel ring buffer, the systemd journal, systemd-analyze, dpkg, apt-cache policy and /sys between them describe the state of the machine in far more detail than any settings panel — and a desktop can look completely correct while those sources are recording a failure a few hundred times a minute.

What that gap creates is a feedback loop with a missing half. The machine emits the evidence; nothing reads it. The two defects below had both been present since the day the setup script was written, both were sitting in plain text in the journal, and neither was discoverable from the desktop:

The GTK theme was discarded at parse time, 161 rules per application launch, 2,093 log lines in the boot I measured, while reporting success and looking approximately right.
10.9 seconds of a 39.3-second boot were spent waiting for something nothing was waiting for.

Both were introduced by flex, the script in this repository whose job is to configure the machine:

https://github.com/xbill9/chromeos-boot

The Machine

One laptop, measured on 2026-09-08:


Hardware	Lenovo Yoga 9i, Intel CometLake-H, NVIDIA GTX 1650 Ti Mobile
OS	Debian 13 (trixie)
Kernel	`7.1.8+deb13-amd64` (from `trixie-backports`)
Desktop	GNOME 48, Wayland, gdm3
GTK	`libgtk-4-1` 4.18.6, `libadwaita-1-0` 1.7.6

That GTK version is the whole article, and nothing on the desktop ever mentions it.

At This Point You Should Have

A Debian desktop you have customised with something — a script, a dotfile repo, a settings panel, an afternoon of gsettings — and that appears to work
sudo, or membership of adm / systemd-journal. Check this first, for reasons in Step 0
systemd-analyze, journalctl, dpkg, apt-cache — all base install
Somewhere durable to write findings that is not your shell scrollback

None of the diagnosis below changes anything. Steps 0 through 3 are reads.

Step 0 — Find Out Whether You Can Read the Evidence At All

This step exists because the investigation nearly started with a wrong answer.

The first journalctl run reported four errors this boot, all of them GNOME session scopes failing to start, and that looked like the whole picture. It was not the whole picture. It was the picture available to an unprivileged user:

id

uid=1000(xbill) gid=1000(xbill) groups=1000(xbill),24(cdrom),25(floppy),
27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),101(netdev),
102(scanner),106(bluetooth),108(lpadmin),989(docker),992(render)

No adm, no systemd-journal. Without either, journalctl silently narrows to your own user journal rather than refusing — you get output, it is correctly formatted, and it is a small fraction of what happened. The kernel buffer is closed off separately:

dmesg | tail -3

dmesg: read kernel buffer failed: Operation not permitted

The difference between the two views on this machine:

View	`err` and above, this boot
Unprivileged `journalctl`	4
`sudo journalctl`	17

Four against seventeen, and nothing in the first view indicates the other thirteen exist. A loop that reads a filtered log converges on a wrong answer confidently, which is worse than not reading it. Add yourself to adm and log back in, or accept that everything downstream needs sudo — but decide it deliberately, because the failure mode is silence, not an error.

Step 1 — Count Before You Read

err and above is where people look, and on this machine it was almost entirely a red herring. The volume was one priority down:

sudo journalctl -b -p err     --no-pager -q | wc -l
sudo journalctl -b -p warning --no-pager -q | wc -l

17
1032

A thousand warnings is too many to read, so collapse them into shapes first. Normalising the numbers and hex out of each message turns a thousand lines into a histogram of distinct problems:

sudo journalctl -b -p warning --no-pager -q -o cat \
  | sed 's/[0-9a-f]\{8,\}/HEX/g; s/[0-9]\+/N/g' \
  | sort | uniq -c | sort -rn | head -5

    795 Theme parser error: libadwaita.css:N:N-N: Unknown @ rule
     27 blacklist: Duplicate blacklisted hash bin:HEX
     24 Can't update stage views actor unnamed [StBin] is on because it needs an allocation.
     10 Theme parser error: libadwaita-tweaks.css:N:N-N: Unknown @ rule
      7 Unable to get default source

805 of the 1,032 warnings on this system were one defect. It had never appeared on screen, in any settings panel, or in the output of the script that caused it — which had reported, and still reports, success.

Step 2 — Attribute the Message Before You Believe the Field

The obvious next question is which program is producing them, and the obvious way to ask is the journal's own _COMM field. That gives a wrong answer, and it is worth showing because the same trap applies to any log the agent will read.

  1288  cat
   322  gnome-control-c
   161  gnome-software
   161  xdg-desktop-por
   161  mutter-x11-fram

cat is not a GTK application and does not parse stylesheets. Pulling one of those 1,288 records apart explains it:

_COMM=cat  _PID=11105  _EXE=/usr/bin/cat
MESSAGE=(chrome:11097): Gtk-WARNING **: 09:20:36.757: Theme parser error: ...

The webapp launchers this repository installs pipe their stderr, so journald attributes the line to the process that wrote it rather than the process that produced it. The real emitter is in the message payload, in the (name:pid) prefix that GLib puts there.

The structured field describes the plumbing; the unstructured payload describes the event. Grouping by _COMM credits 1,288 parse errors to cat and hides Chrome entirely. Parse the prefix instead:

sudo journalctl -b -q -o cat | grep "Theme parser error: libadwaita" \
  | sed -E 's/^\(([a-zA-Z0-9_.-]+):[0-9]+\).*/\1/' | sort | uniq -c | sort -rn

Corrected, the emitters are gnome-control-center, gnome-software, xdg-desktop-portal-gtk, mutter-x11-frames and Chrome — every GTK4 process on the system, each one failing identically, every time it starts.

What the Theme Declared, and What GTK Served

161 distinct parse failures per process: 159 in libadwaita.css, 2 in libadwaita-tweaks.css. At warning and above each one appeared exactly five times — 159 × 5 = 795, 2 × 5 = 10 — one set per GTK4 process running at boot. Across all priorities, including the Chrome instances started later from the shelf, 2,093 lines in one boot.

The message names the construct:

Theme parser error: libadwaita.css:94:1-7: Unknown @ rule

Columns 1-7 of line 94:

@media (prefers-color-scheme: dark) { @define-color window_bg_color #222226; ... }

@media. GTK's CSS parser is not a browser's, and GTK 4.18 does not implement @media. It reports the rule as unknown and discards the entire block.

Which block is discarded matters more than the noise. flex installs adw-gtk3 from the upstream release tarball, pinned:

grep ADW_VERSION flex

ADW_VERSION=${ADW_VERSION:-v6.5}

Upstream's own release notes, read in order, are unambiguous about what that pin means:

Release	Upstream's note	`@media` in the GTK4 CSS
v5.7	"New release for GNOME 48 and libadwaita 1.7"	0
v6.3	last release before the change	0
v6.4	*"The GTK4 theme now requires GTK 4.20 or later."*	3
v6.5	"Release for GNOME 50."	159

The machine runs GNOME 48, GTK 4.18.6 and libadwaita 1.7.6. The pin was v6.5 — GNOME 50, GTK 4.20+. There is a release named for this exact platform, in upstream's own words, and it is two major versions below the pin.

git log says the pin was never wrong later; it was wrong from the start:

git log --oneline -L58,58:flex

12a0ddc Add flex: Debian desktop customization to mimic ChromeOS Flex
+ADW_VERSION=${ADW_VERSION:-v6.5}

The newest release at the time, taken as the right one because it was newest, never checked against the desktop it had to run on.

Why Nothing Looked Wrong

A theme silently dropping 159 rule blocks should be visible, and it was not. Two reasons, and the second is the expensive one.

First, libadwaita applications largely do not use the GTK theme at all — they carry their own stylesheet and follow color-scheme directly. Most of the desktop stayed dark because most of the desktop was never asking the theme.

Second, and much worse:

T=~/.local/share/themes
for f in $(cd $T/adw-gtk3/gtk-4.0 && find . -type f | sort); do
  cmp -s "$T/adw-gtk3/gtk-4.0/$f" "$T/adw-gtk3-dark/gtk-4.0/$f" \
    && echo "identical: $f" || echo "DIFFERS:   $f"
done

identical: ./assets/bullet-symbolic.svg
identical: ./assets/check-symbolic.svg
identical: ./assets/dash-symbolic.svg
identical: ./assets/devel-symbolic.svg
identical: ./gtk.css
identical: ./gtk-dark.css
identical: ./libadwaita.css
identical: ./libadwaita-tweaks.css

At v6.5, every one of the eight files under gtk-4.0/ is byte-identical between adw-gtk3 and adw-gtk3-dark. From v6.4 upstream stopped shipping two GTK4 stylesheets. There is one file, and dark is selected at runtime by the @media query — the query GTK 4.18 throws away.

So the dark theme's own file defines the light value, unconditionally, and the dark value only inside the block that never executes:

adw-gtk3-dark/gtk-4.0/libadwaita.css:62:
    @define-color window_bg_color #fafafb;      <- light, applied
inside @media (prefers-color-scheme: dark):
    @define-color window_bg_color #222226;      <- dark, discarded

46 of the 154 @define-color rules in that file are inside @media blocks. All 46 were being dropped. And because the two GTK4 trees are identical, set-mode.sh — the helper this repository installs specifically to flip light and dark — could not change a GTK4 application's palette at all. It flipped a setting between two names that pointed at the same bytes.

That is the shape of the whole problem: the setting declared a dark theme, and the filesystem served a light one. Nothing in the declaring layer knows what the serving layer did with it, and the only place the two are compared is the log.

The Fix, and the Second Bug It Exposed

Moving the pin to v5.7 is one line. Applying it was not, and the reason is a bug the first bug was hiding.

bash flex theme

==> theme: adw-gtk3 v5.7 into /home/xbill/.local/share/themes
    already installed -- skipping

The stage's idempotence guard tested that the theme directories existed, not which version was in them. That is fine while the pin is right and fatal once it is wrong: correcting the pin was a no-op, because the evidence of the bad install was also the thing that suppressed the repair. Every re-run confirmed the wrong theme.

Idempotence has to mean "on the pinned version", not "something is there". The guard now records what it installed and compares:

if [ -d "$THEMES/adw-gtk3-dark" ] && [ -d "$THEMES/adw-gtk3" ] \
   && [ "`cat "$THEMES/.adw-gtk3-version" 2>/dev/null`" = "$ADW_VERSION" ] ; then
  step "already installed at $ADW_VERSION -- skipping"
  return
fi

It also removes the old tree before extracting rather than unpacking over it. The layout changed between the two releases — v6.3 added libadwaita.css and libadwaita-tweaks.css to gtk-4.0, and v5.7 has neither — so extracting v5.7 on top of v6.5 would have left both files orphaned on disk, imported by nothing and still found by anything globbing the directory.

Re-run, and the two themes are two themes again:

adw-gtk3/gtk-4.0/gtk.css:      @define-color window_bg_color #fafafb;
adw-gtk3-dark/gtk-4.0/gtk.css: @define-color window_bg_color #222226;

sudo journalctl -b -o cat -q | grep -c "Theme parser error"

2,093 to 0, and warning-and-above for the whole system fell from 1,032 to 151. ✅

The Same Loop, Three More Times

One defect found this way could be luck. The value is in the loop, so it is worth running it against three other subsystems — each with its own evidence source, and each teaching something different about how to read one.

Drivers: the error you must refuse to act on

The kernel log's firmware section looks alarming and mostly is not:

sudo journalctl -b -k -q -o cat | grep -i iwlwifi

iwlwifi 0000:00:14.3: Detected Intel(R) Wi-Fi 6 AX201 160MHz
iwlwifi 0000:00:14.3: firmware: failed to load iwl-debug-yoyo.bin (-2)
iwlwifi 0000:00:14.3: firmware: failed to load iwl-debug-yoyo.bin (-2)
iwlwifi 0000:00:14.3: firmware: failed to load iwl-debug-yoyo.bin (-2)
iwlwifi 0000:00:14.3: loaded firmware version 77.2753b721.0 QuZ-a0-hr-b0-77.ucode

Three failures, then success. iwl-debug-yoyo.bin is an optional debug blob that Debian does not ship and the driver does not need; the device is working. An agent that treats "failed" as actionable installs packages to fix a working radio. The correction is cheap and general: read the lines after the error before acting on it, and prefer the state of the device to the wording of the log.

The genuinely interesting driver finding was on the other side, and it is a non-event worth confirming rather than assuming:

sudo dkms status

nvidia/610.57.04, 6.12.107+deb13-amd64, x86_64: installed
nvidia/610.57.04, 7.1.8+deb13-amd64, x86_64: installed

This machine runs a backports kernel with an out-of-tree NVIDIA module, which is the configuration most likely to break on an upgrade. DKMS had rebuilt against both kernels, and both header packages are present — so the next kernel will not silently land without a GPU. The taint word says exactly what is loaded and why:

cat /proc/sys/kernel/tainted     # 12289 -> bits 0, 12, 13
mokutil --sb-state               # SecureBoot disabled

Proprietary, out-of-tree, unsigned — all three expected here, and all three would be worth an alarm on a machine where Secure Boot was on.

Packages: what apt declares against what apt will serve

apt will happily tell you a package is available without telling you it will never choose it:

apt-cache policy | grep -A1 backports | head -4

 100 https://deb.debian.org/debian trixie-backports/main amd64 Packages
     release o=Debian Backports,a=stable-backports,n=trixie-backports

Priority 100 against 500 for everything else. Backports is enabled and will never be selected automatically — which is correct, and is also why the running kernel had to be asked for by name. Six third-party origins are configured on this box (Docker, Chrome, cloud-sdk, CUDA, Claude Desktop, and backports), and apt-cache policy is the only place that relationship is written down.

The rest of the package state was one line of debris and one pending upgrade:

rc  linux-image-6.12.94+deb13-amd64   (removed, config files remain)
containerd.io  2.3.4 -> 2.3.5  (upgradable)

Neither is a problem. Both are the kind of thing that is invisible until something asks.

Tuning: the wait that nothing was waiting for

systemd-analyze is the least-used excellent tool on a Debian desktop, because it does not report which unit is slow — it reports which unit is in the way:

systemd-analyze critical-chain

graphical.target @10.626s
└─power-profiles-daemon.service @10.547s +78ms
  └─multi-user.target @10.545s
    └─docker.service @9.507s +1.037s
      └─network-online.target @9.505s
        └─NetworkManager-wait-online.service @1.730s +7.774s

7.774 seconds, on the path to the desktop appearing, spent blocking until the network was routable — because docker.service asks for network-online.target and NetworkManager-wait-online is what satisfies it. Docker does not need a routable address at start; it needs one when a container asks. The dependency is a default, not a decision.

And one phase up, the loader was carrying a 5-second GRUB menu nobody reads — on a machine whose own repository already ships boot-splash.sh to remove it, documented as installed-but-never-run.

Both changed, one reboot, measured:

phase	before	after	delta
firmware	16.064s	14.448s	−1.616s	not touched
loader	8.014s	2.776s	−5.238s	`GRUB_TIMEOUT` 5 → 0
kernel	4.617s	4.978s	+0.361s	not touched
userspace	10.626s	4.941s	−5.685s	`wait-online` disabled
total	39.324s	27.144s	−12.180s

The attributable figure is −10.9 s, not −12.180 s, and the table says why. Firmware and kernel are phases nothing in this change touches, and they moved by −1.255 s between the two runs on their own. That is the run-to-run noise of the measurement, and it is the reason the two columns cannot simply be subtracted: a change measured against the previous boot is measured against time as well. One boot either side is a weak design, and the honest claim is a bound — around eleven seconds, of which 5.238 s and 5.685 s land in exactly the two phases that were altered.

network-online.target is still reached, incidentally. It is reached at @1.720 s instead of @9.505 s, because nothing is now blocking on it. Docker started clean.

And the loop immediately found the next thing

systemd-analyze blame | head -3

3.303s nvidia-persistenced.service
2.689s plymouth-quit-wait.service
2.208s thermald.service

nvidia-persistenced is now 3.303 s of a 4.941 s userspace boot and sits on the critical chain where wait-online used to. That is not a disappointment; it is the loop working. Removing the largest cost promotes the next one into view, and it was invisible while a 7.8-second wait was in front of it. ✅

Where the Agent Actually Fits

Nothing above required a capability Debian was missing. Every number came from journalctl, systemd-analyze, dpkg, dkms, apt-cache and cmp — all of them installed, all of them documented, none of them new. The evidence had been there, in the same files, since the machine was configured.

What was missing was the other half of the loop: something willing to read a thousand warnings, at the moment they stop being free to ignore.

That is a specific job, and it is worth being precise about which parts of it are the agent's:

Reading at volume without triage fatigue. 1,032 warnings is where a human starts sampling. The 805-line defect was in the histogram's first row, and the histogram is trivial to build — the reason nobody had built it is that nobody had a reason to.
Correlating across sources that share no vocabulary. The finding needed a journal message, a CSS file, a dpkg -l version, four GitHub release notes and a git log -L on one line of a shell script. Each is unremarkable alone.
Acting, then re-reading. The fix is only confirmed by the same query that found it returning 0 — and by warning+ dropping 1,032 → 151, which is a different check than "the desktop still looks fine".

And the parts that are not:

Deciding what is worth fixing. The iwlwifi line is a failure the correct action is to ignore. An agent that treats every error as work will find plenty of it.
Anything that touches the boot chain. boot-splash.sh is in this repository precisely because it edits GRUB, and this repository deliberately never runs it for you. Reading it first was a human decision; so was accepting that Docker would now start without waiting for the network.
Believing the log's own metadata. _COMM said cat 1,288 times.

The loop that produced everything above is four steps and does not need tooling: read the evidence, form a claim about which layer disagrees with which, change one thing, re-read the same source. The last step is the one that distinguishes it from a guess, and it is the step a settings panel cannot offer, because a settings panel only ever shows you what was declared.

Cheat Sheet

Nothing here writes:

# 0. can you actually see the system's evidence, or only your own?
id | grep -o 'adm\|systemd-journal' || echo "unprivileged view -- use sudo"

# 1. volume first, at warning, not just err
sudo journalctl -b -p err     --no-pager -q | wc -l
sudo journalctl -b -p warning --no-pager -q | wc -l

# 2. collapse a thousand lines into distinct problems
sudo journalctl -b -p warning --no-pager -q -o cat \
  | sed 's/[0-9a-f]\{8,\}/HEX/g; s/[0-9]\+/N/g' \
  | sort | uniq -c | sort -rn | head -20

# 3. attribute by payload, not by _COMM
sudo journalctl -b -q -o cat | grep "<pattern>" \
  | sed -E 's/^\(([a-zA-Z0-9_.-]+):[0-9]+\).*/\1/' | sort | uniq -c | sort -rn

# 4. what is in the way of the desktop -- not what is slow
systemd-analyze
systemd-analyze critical-chain
systemd-analyze blame | head -10

# 5. out-of-tree modules against every installed kernel
sudo dkms status ; cat /proc/sys/kernel/tainted ; mokutil --sb-state

# 6. what apt will actually serve you, priorities included
apt-cache policy ; apt list --upgradable 2>/dev/null ; dpkg -l | grep '^rc'

# 7. re-read the exact query that found it -- this is the step that confirms

Summary

The goal of this article was to find out what a correctly-behaving Debian desktop was failing to mention, and whether reading its own logs systematically would repay the effort. The key to the solution was treating the configuration layer and the serving layer as two separate claims that have to be compared — what gsettings declared against what GTK parsed, what a unit file requested against what the boot actually waited for — because every defect found was invisible in the declaring layer and plainly stated in the serving one. The measured results were:

2,093 CSS parse errors per boot went to 0. The theme pin was two major versions above what the desktop's GTK could parse, from the first commit that introduced it.
System-wide warning-and-above fell 1,032 → 151. err-and-above stayed at 17, correctly — none of those were theme-related.
46 of 154 @define-color rules were being discarded, and all eight gtk-4.0 files were byte-identical between the light and dark themes, so the mode switcher could not change a GTK4 application at all.
Boot went 39.324 s → 27.144 s, of which −10.9 s is attributable — 5.238 s of GRUB timeout and 5.685 s of NetworkManager-wait-online — with the untouched phases drifting −1.255 s as noise.
network-online.target is now reached at @1.720 s rather than @9.505 s, and Docker starts clean without it being waited on.
A second bug was hidden by the first: the theme stage's idempotence guard tested only that files existed, so correcting the pin was a no-op until the guard was made version-aware.

Scope: one laptop, one operating system, measured 2026-09-08 on Debian 13 with GNOME 48, GTK 4.18.6 and kernel 7.1.8+deb13-amd64 from backports. The boot figures are n = 1 on each side — a single boot before and a single boot after — which is why the attributable claim is stated as a bound rather than a measurement, and the ±1.255 s drift in the two untouched phases is the only estimate of noise available from that design; repeated boots on each configuration are queued and would replace it. The 2,093 → 0 result is not subject to the same weakness, being a count of a deterministic parse failure rather than a timing. Whether any application's appearance visibly changed as a result of the theme fix was not measured: the five confirmed emitters prove the file was parsed, not that a user could see the difference, and libadwaita applications never consulted the theme either way. nvidia-persistenced at 3.303 s is now the largest item on the critical chain and has not been investigated.

The strategy of reading the system's own evidence at volume, attributing it by payload rather than metadata, changing one thing, and re-running the query that found it was validated with an incremental step by step approach.

Disclosure: I ran this with Claude Code as the diagnostic agent — it swept the journal, built the histograms, traced the pin to the upstream release notes, wrote the fix and re-measured after the reboot, while I steered and made the calls about what to touch. The commands, the counts and both defects are real and archived in the repository.

The Cable Buys Headroom: 91% of a USB 2.0 Bus, 3.6% of a Thunderbolt One

xbill — Tue, 08 Sep 2026 16:09:41 +0000

This article walks through diagnosing a USB tether that would not negotiate above USB 2.0 speed, on a phone that advertises 10 Gb/s. The hunt took four cables and three physical ports across two days.

USB link speed is autonegotiated between the host, the cable and the device, and the result is the slowest mode all three can manage. The cable is a full participant in that negotiation, a USB 2.0 cable has no way to vote for anything above 480 Mbps, and nothing anywhere reports that it happened. The tether comes up, DHCP works, traffic flows, and the link is capped at a fifth of what both ends were offering.

What the right cable buys is not speed. It is headroom, and the difference is not subtle: the best USB 2.0 pass in this log used 91% of what that bus can usably carry, while the same link on a SuperSpeed cable uses 3.6%.

Every command below is one you can run against your own tether, and every number comes from a record in the repository:

https://github.com/xbill9/tether

The Symptom

The repository holds 45 measurement records across 14 handsets from four vendors, each one a full pass — three sequential single-stream transfers, a four-stream parallel test, and a ping — against a fixed endpoint and a fixed 8 MB transfer size.

41 of those 45 records enumerated at 480 Mbps. Fourteen handsets, three drivers, two carriers, three physical receptacles, and for two days not one USB 3.0 negotiation anywhere.

The most conspicuous offender was an iPhone 17 Pro, whose BOS descriptor is not shy about what it can do:

bcdUSB               2.10
SuperSpeed USB Device Capability:
  wSpeedsSupported   0x000e     -> Full, High, SuperSpeed (5Gbps)
SuperSpeedPlus USB Device Capability:
  bmSublinkSpeedAttr[0]  10Gb/s Symmetric RX SuperSpeedPlus
  bmSublinkSpeedAttr[1]  10Gb/s Symmetric TX SuperSpeedPlus

A device declaring 10 Gb/s, enumerating at 480 Mbps. That is a factor of about twenty-one, and it is the kind of gap that makes you stop measuring throughput and start reading sysfs.

The Bus Rate Is Negotiated, and Every Link in the Chain Votes

USB link speed is not a property of the phone, and it is not a property of the laptop. It is negotiated at every attach between three parties — host controller, cable, device — and the trained rate is the fastest mode all three can do. One USB 2.0 component anywhere in that chain and the entire link comes up at 480 Mbps.

The cable is a real participant in that vote, not a passive wire. SuperSpeed rides on two extra differential pairs that a USB 2.0 cable does not physically contain — it carries D+/D- and nothing else. Put one between a 10 Gb/s phone and a 10 Gb/s port and the two ends will negotiate down to High Speed and work perfectly.

And nothing tells you. No error, no dmesg warning, no degraded-mode indicator, no line anywhere saying a cable capped you. The only evidence is a number in sysfs that you have to go and read, next to a second number saying what the device had offered:

Reading	Where	What it is
`speed`	`/sys/bus/usb/devices/<dev>/speed`	what the three parties agreed on
BOS descriptor	`lsusb -v -d <vid>:<pid>`	what the device offered

The gap between those two numbers is the cable's vote. Steps 1 and 2 below are those two readings, and everything after them is narrowing down which participant cast the low one.

At This Point You Should Have

A phone tethered over USB and holding the default route — check this first, or you will measure Wi-Fi
lsusb from usbutils, and ip from iproute2
Read access to /sys/bus/usb/devices and /sys/class/net, which needs no root
Every other route-capable interface down. If Wi-Fi is up, curl will succeed over it and the result will look like a tethering measurement

None of the diagnosis below transfers a single byte over the cellular link. That matters here: a full measurement pass in this repository costs roughly 56 MB of metered data, and everything in Steps 1 through 4 is free.

Step 1 — Read the Bus Speed First

This is the one reading that invalidates everything downstream if you skip it. A charge-only or USB 2.0 cable silently caps the bus at 480 Mbps and nothing anywhere reports an error.

# find the tether interface and its USB device
for i in /sys/class/net/*/device/driver; do
  ifn=$(basename $(dirname $(dirname $i)))
  drv=$(basename $(readlink -f $i))
  case $drv in cdc_ncm|rndis_host|ipheth) echo "$ifn  $drv";; esac
done

enxb65575abcda3  ipheth

Then walk from the interface back to the USB device and read its trained speed:

dev=$(basename $(readlink -f /sys/class/net/enxb65575abcda3/device/..))
cat /sys/bus/usb/devices/$dev/speed

480 is USB 2.0, 5000 is USB 3.0 Gen 1, 10000 is Gen 2. Alongside it, /sys/class/net/<if>/speed gives the negotiated link rate — though note that ipheth returns EINVAL there and reports nothing, so on an iPhone this field stays blank in every record.

Step 2 — Ask the Device What It Is Capable Of

A 480 reading only means something once you know what the device brought to the negotiation.

lsusb -v -d 05ac:12a8 2>/dev/null | grep -A3 "SuperSpeed"

The descriptor block at the top of this article is that output. The device capability is the control. Compare the iPhone 16e measured eight minutes before the fastest pass in this log: it advertises no SuperSpeed capability at all, so its 480 is its own ceiling and no cable on earth will move it.

That single distinction separates "something in the chain voted 480" from "the phone itself voted 480", and it is free to read. Where the device offers SuperSpeed and the link trained at High Speed, one of the other two participants is responsible — and there are only two of them.

Step 3 — Find Every Port, Not Just the Pretty Ones

This step exists because the investigation got it wrong, in print, twice.

The obvious move on a modern laptop is to enumerate the Type-C connectors, since the kernel binds each one to its USB ports at boot:

dmesg | grep "typec port"

typec port0: bound usb3-port1 (ops connector_ops [usbcore])
typec port0: bound usb4-port1 (ops connector_ops [usbcore])
typec port1: bound usb3-port2 (ops connector_ops [usbcore])
typec port1: bound usb2-port1 (ops connector_ops [usbcore])

Read against lspci, that is a complete map of the machine's USB-C:

Type-C connector	USB 2.0 half	SuperSpeed half	Controller
`port0`	`3-1`	`4-1` (10 Gb/s)	`00:14.0` Alder Lake PCH
`port1`	`3-2`	`2-1` (20 Gb/s)	`00:0d.0` Thunderbolt 4

That table is correct. The conclusion drawn from it — "there is no untried port on this machine" — was wrong, and it cost about an hour.

/sys/class/typec enumerates Type-C connectors only. A Type-A receptacle has no Power Delivery and therefore no connector object, so it cannot appear there. The machine had a third external receptacle the map could not see.

Take the port inventory from the hotplug entries in the port list instead:

for p in /sys/bus/usb/devices/usb*/*/usb*-port*; do
  [ "$(cat $p/connect_type 2>/dev/null)" = "hotplug" ] || continue
  peer="-"; [ -L "$p/peer" ] && peer=$(basename "$(readlink -f "$p/peer")")
  echo "$(basename $p)  peer=$peer  location=$(cat $p/location)"
done | sort

usb3-port3  peer=usb4-port2  location=0x80000301
usb4-port2  peer=usb3-port3  location=0x80000301

usb3-port3 is the USB-A receptacle. Its peer is a 10 Gb/s port on another hub, and the shared location is the kernel saying these two entries are two halves of one physical hole in the side of the laptop.

That peer relationship is the single most useful line in this whole investigation, for a reason that only becomes clear in a moment.

Step 4 — Ask the Cable Directly, and Watch It Refuse

An electronically marked USB-C cable declares its own data capability over Power Delivery, and Linux exposes that as a portN-cable device with an identity/ directory of PD VDOs. The USB-C specification requires an e-marker on a passive cable supporting more than USB 2.0, and does not require one on a USB 2.0 cable. So a missing e-marker would be real evidence.

ls /sys/class/typec/
cat /sys/class/typec/port0-partner/usb_power_delivery_revision

port0  port0-partner  port1  usb_power_delivery_revision
0.0

There is no cable node, and that is not evidence about the cable. The stack is not surfacing VDO data at all: the connected partner — the phone itself — has an empty identity/ directory and reports PD revision 0.0. A stack that will not show the phone's PD identity was never going to show the cable's.

Worth knowing precisely because it looks like it should work. On a host that does report cable identity this is the whole answer in one cat. Here it is silence, not a finding, and treating it as a finding would have convicted the cable on no evidence.

The Hunt

With the cable unreadable, the only remaining instrument is substitution. Four cables, three receptacles, over one afternoon:

Pass	Cable	Receptacle	Bus
1	C-to-C, unmarked	`3-2` (Thunderbolt 4 connector)	480
1b	same cable	`3-1` (the other Type-C)	480
2	C-to-C, second cable	`3-1`	480
3	C-to-C, third cable	`3-1`	480
4	A-to-C, unmarked	`3-3` (USB-A)	480

Zero USB errors and zero drops in every pass. The host has two SuperSpeed root hubs, both enumerated, both idle, and neither had ever seen a device.

The tempting reading is that four negatives is mounting evidence against the host. It is not. None of those four cables was known to carry SuperSpeed pairs. Unmarked C-to-C cables — the kind that ship in the box with a phone or a charger — are overwhelmingly USB 2.0 by construction. Four of them all being USB 2.0 is not a coincidence in need of explanation; it is the expected outcome.

Four unknowns is close to one result repeated, not four independent tests.

The Trap in the Middle

Here is the part of the investigation most worth stealing, because it has nothing to do with USB.

Throughput was measured on every one of those passes, and it moved a great deal:

	12:40 cable 1	13:15 cable 2	13:19 cable 2, settled	13:26 cable 3	14:07 cable 4
Single (Mbps)	96 / 103 / 103	56 / 36 / 21	64 / 57 / 59	55 / 25 / 68	76 / 58 / 61
4-stream aggregate	178	73	85	101	130
RTT avg	38.0 ms	103.6 ms	54.2 ms	45.9 ms	52.1 ms
RTT mdev	6.7 ms	110.3 ms	30.9 ms	9.3 ms	12.1 ms

Aggregate climbed 73 → 85 → 101 → 130 across four consecutive cable swaps. Every new cable looked better than the one before it.

None of that is a cable effect. The carrier was recovering from a dip across the afternoon, and each swap happened to coincide with a step of that recovery. Time is the variable that moved; the cable is the one that did not. The USB errors were zero throughout, and a cable degrading throughput by 60% would show up in those counters.

Two rules came out of this and both are now in the repository's methodology:

Record three single-stream runs, never one. One phone in this log produced 15, 44 and 116 Mbps on three consecutive identical transfers. Any single number is a lie about the link.
A/B a hardware change against time, not against the previous hour. If you cannot interleave, do not attribute.

The Cable That Worked

The next day, a new USB-A to USB-C cable — around $15, bought specifically to be a known SuperSpeed cable rather than another unknown.

480  ->  5000

The iPhone trained SuperSpeed and moved to a different hub in sysfs, appearing at 4-2 rather than the 3-x of every previous pass. That looks like a port change and it is not:

usb4-port2  peer=usb3-port3  location=0x80000301

4-2 is usb4-port2, whose peer is usb3-port3 — the same USB-A receptacle tested at 480 the day before, same location code. The device moved hubs only because it started training SuperSpeed, which is exactly what a successful 5 Gb/s negotiation looks like on this host. The receptacle is a control here. The cable is the only variable that moved.

A Pixel 9a on the same new cable, same receptacle, same DHCP lease, confirmed it from the other side:

	2026-09-05, old cable	2026-09-06, $15 cable
Bus speed	480 Mbps	5000 Mbps
Negotiated link	425 Mbps	3750 Mbps

Two handsets, one cable swap, and the question that had eaten an afternoon was answered. The unmarked cables were USB 2.0 cables. ✅

What the Cable Unlocked: Headroom

The bus reading is the headline. Headroom is what it actually buys, and it is the number that says whether any of this was worth $15.

The repository's diagnostic constant for the realistic bulk throughput of a 480 Mbps USB 2.0 bus, after protocol overhead, is 300 Mbps. Measure every pass in the log as a share of the bus it ran on and the picture is stark:

Pass	Bus	4-stream aggregate	Share of the bus
🥉 Best USB 2.0 pass in the log	480	273 Mbps	56.9% of raw — and 91% of usable
🥈 Anker USB-A to USB-C, BBR	5000	365.185 Mbps	7.3%
🥈 Same receptacle, CUBIC	5000	370.695 Mbps	7.4%
🥇 Thunderbolt cable	10000	361.330 Mbps	3.6%
🥇 Thunderbolt, BBR	10000	317.250 Mbps	3.2%

91% against 3.6% is the whole argument, and the two denominators are worth naming so the comparison is read correctly. The 91% is 273 Mbps against the usable 300 Mbps of a 480 Mbps bus; the 3.6% is 361.330 against the raw 10000, because this repository has no measured usable ceiling for a SuperSpeed bus and inventing one would be worse than the mismatch. Compared raw to raw it is 56.9% against 3.6% — a factor of sixteen rather than twenty-five, and the same conclusion either way.

Share of the bus is arithmetic throughout — each record's 4-stream aggregate over its own bus_speed_mbps, so 273/480 = 56.9% and 365.185/5000 = 7.3%. The 91% figure is 273 against the 300 Mbps usable ceiling rather than the raw 480.

On USB 2.0 the best pass in 45 records was at 91% of what the bus can usably carry. There was nothing left. Every good pass in the 480 era was pressed up against a ceiling, and the ones that were not were being held down by the carrier instead.

On SuperSpeed the same link uses about 7% of the bus. The cable did not make anything faster by itself; it moved the ceiling from just above the traffic to fourteen times above the traffic, so the tether is now free to take whatever the radio gives it on any given evening — and this link swings by more than 4x on its own within minutes.

Four of those measurements also clear the 300 Mbps line outright:

Measurement	Result	Above 300 Mbps?
Best 480-era aggregate, 45 records	273 Mbps	no — 91% of usable
4-stream aggregate, USB-C receptacle	370.695 Mbps	yes
4-stream aggregate, Anker cable under BBR	365.185 Mbps	yes
4-stream aggregate, Thunderbolt	361.330 Mbps	yes
Single 32 MB transfer, USB-C receptacle	351.1 Mbps	yes

Those numbers were not reachable on a USB 2.0 bus. Not "would have been slower" — not reachable. Every one of them needed a cable that could vote for SuperSpeed. ✅

Why the Aggregate Is the Test

The four-stream number is doing the work in both tables above, and it is worth saying why rather than treating it as a second opinion on the single-stream figure.

Single-stream against four-stream aggregate is the one comparison that separates "the carrier is slow" from "something on this machine is collapsing." If four flows together go much faster than one, the wide-area path has headroom and the problem is local. If they do not, there is no headroom to find.

The Anker cable's BBR pass is a clean worked example. Single-stream ran 134.930 / 113.576 / 123.229 Mbps — a 1.19x spread — against a four-stream aggregate of 365.185 Mbps, 2.95x the single-stream mean.

Read those two numbers together and the link describes itself. Four flows found nearly three times what one flow did, so the wide-area path plainly had that much to give and nothing local was collapsing: a 1.19x spread with zero errors and zero drops leaves no radio loss for congestion control to misread. The ceiling in that reading is not on this machine, which is exactly the state you want to be in — the tether is out of the way and the only thing left is the radio.

The reverse reading is the one that catches people. Where the aggregate lands on top of the single-stream figure, four flows found nothing extra, and no amount of tuning on the host will produce headroom that is not there.

Read the aggregate before you attribute anything to hardware. It is the reading that tells you whether the answer is even on your side of the link.

What the Link Reached

Once the bus was out of the way, three passes on this handset across two evenings, on two cables and two host controllers:

	🥇 Thunderbolt, CUBIC	🥈 USB-C receptacle, CUBIC	🥉 Anker cable, BBR
Bus speed	10000 Mbps	5000 Mbps	5000 Mbps
Single stream (Mbps)	151.036 / 119.931 / 147.540	134.856 / 143.749 / 129.250	134.930 / 113.576 / 123.229
4-stream aggregate	361.330	370.695	365.185
RTT avg	27.268 ms	29.601 ms	36.715 ms
RTT mdev	2.272 ms	7.171 ms	5.597 ms

Three aggregates within 3% of each other — 361.330, 370.695 and 365.185 — across two different cables, two host controllers, two congestion control algorithms and a bus that differs by 2x. None of those variables moved the number. The link settles around 365 Mbps and the hardware underneath it has stopped mattering, which is what having headroom looks like.

The RTT came with it: 27.268 ms average with mdev of 2.272 ms on the Thunderbolt pass, the flattest jitter anywhere in the log.

The BBR column is worth its place for a second reason. Every early SuperSpeed pass ran cubic where the 480-era records ran bbr, so bus and congestion control had moved together and neither could be credited. Matched on the same port and bus, BBR gives 365.185 against CUBIC's 370.695 — within a few percent, with mdev slightly lower. The bus is what these figures rest on.

Buy Headroom Once, and Stop

Note the third column of that table: moving from the Thunderbolt port back to a plain USB-C receptacle halved the bus and cost nothing — 370.695 Mbps against 361.330. At 361 Mbps the phone is using about 3.6% of a 10000 Mbps bus, by arithmetic, against 7.4% of a 5000 Mbps one. Both are miles of headroom. Doubling an already-idle ceiling buys exactly nothing, and Gen 2 is not worth chasing on a tether.

So the rule is narrow and cheap:

✅ One known-good cable, then stop. Four unmarked cables cast four identical 480 votes. One rated cable produced 5000 on two different handsets, on the same receptacle, with nothing else changed.
✅ Read the bus before you tune anything. cat /sys/bus/usb/devices/<dev>/speed is free, and a 480 reading on a SuperSpeed-capable phone caps you at roughly 300 Mbps of real throughput no matter what else you fix.
✅ Do not buy ceiling you cannot reach. 5000 and 10000 measured the same on this link, repeatedly. Headroom is binary in practice: you either have enough or you do not.
❌ Do not trust the connector to tell you. Two of the four failing cables were USB-C on both ends, in a Thunderbolt 4 port, on a phone advertising 10 Gb/s. Every visible part of that chain looked right.

One limit on all of this, and it is being closed by measurement rather than argument. The 300 Mbps figure is the repository's diagnostic constant, so the claim above is a bound argument rather than a controlled A/B — no pass has yet put the fast link back onto a USB 2.0 cable to measure what it would cost. That controlled re-test is queued, and it is the number this article most wants.

One More Trap: the Transfer Size

While measuring the fast link, the single-stream figure sat near 140 Mbps against a four-stream aggregate of 361 — the classic shape of per-flow carrier shaping, and that is what it was written up as.

It was wrong, and the correction is worth the paragraph.

A single 32 MB transfer on the same connection ran 332.3 Mbps — essentially the whole four-stream aggregate, from one flow. So the gap was about the fixed 8 MB transfer size, not the link. Except the first attempt to prove that was also uncontrolled: a fitted model predicted 267 Mbps at 20 MB and the measurement came back at 73.0 Mbps, because this link can collapse by 4.8x on its own between one transfer and the next.

The design that settled it was six transfers alternating 8 and 32 MB, the whole run spanning 3.89 seconds so that link variation could not fall differently on the two sizes:

#	size	duration	throughput
1	8 MB	0.418 s	153.3 Mbps
2	32 MB	0.722 s	354.6 Mbps
3	8 MB	0.438 s	146.1 Mbps
4	32 MB	0.701 s	365.1 Mbps
5	8 MB	0.362 s	176.9 Mbps
6	32 MB	0.699 s	366.3 Mbps

8 MB mean 158.75 Mbps, 32 MB mean 361.96 Mbps, a ratio of 2.28x. Every 32 MB run beat the 8 MB run beside it, and the two sets do not overlap at all — the fastest 8 MB run, 176.9, is below the slowest 32 MB run, 354.6.

The mechanism is still open. A fixed-cost fit to those durations gives an asymptote of 637 Mbps and a startup cost of 0.305 s, which is about 10 round trips at this link's 29.6 ms, where slow-start to the ~503 KB bandwidth-delay product should need roughly 6. Something beyond textbook slow-start is in there, and six transfers cannot say what.

Same failure mode as the cable, one layer up. A measurement instrument was producing a number that described the instrument rather than the link, and the only way out was interleaving the two conditions inside a window too short for anything else to change.

Cheat Sheet

Both of these are in the repository and run directly, no arguments needed:

bin/tether-bus-check      # everything below, plus a verdict. No cellular data
bin/tether-report         # the full diagnostic rubric. No cellular data
bin/tether-report --measure   # the standard pass. ~56 MB of metered data
bin/tether-interleave     # A/B two transfer sizes against a moving link

tether-bus-check reads both sides of the negotiation and says which participant cast the low vote. tether-interleave is the design from the section above — alternating sizes back to back so link drift falls on both equally — and it prints the cost before it spends it.

The individual readings, if you would rather run them by hand:

# 1. which interface is the tether, and what driver
for i in /sys/class/net/*/device/driver; do
  ifn=$(basename $(dirname $(dirname $i)))
  case $(basename $(readlink -f $i)) in
    cdc_ncm|rndis_host|ipheth) echo "$ifn $(basename $(readlink -f $i))";; esac
done

# 2. is it holding the default route (or are you about to measure Wi-Fi)
ip route show default

# 3. THE READING THAT MATTERS - 480 / 5000 / 10000
dev=$(basename $(readlink -f /sys/class/net/<if>/device/..))
cat /sys/bus/usb/devices/$dev/speed

# 4. could the device do better? if no SuperSpeed capability, stop here
lsusb -v -d <vid>:<pid> 2>/dev/null | grep -A3 SuperSpeed

# 5. every external port, including the ones /sys/class/typec cannot see
for p in /sys/bus/usb/devices/usb*/*/usb*-port*; do
  [ "$(cat $p/connect_type 2>/dev/null)" = hotplug ] || continue
  peer="-"; [ -L "$p/peer" ] && peer=$(basename "$(readlink -f "$p/peer")")
  echo "$(basename $p) peer=$peer location=$(cat $p/location)"
done | sort

# 6. errors and drops - non-zero points at cable or power, not config
ip -s link show <if>

Steps 1 through 6 cost no cellular data. Only run a transfer once they have told you what you are looking at.

Summary

The goal of this article was to find out why a phone advertising 10 Gb/s would not negotiate above USB 2.0 speed on any cable or port, and what fixing it was worth. The key to the solution was treating the link rate as a three-way negotiation and reading both sides of it — what the link agreed on against what the device offered — then reading the port peer map rather than the Type-C connector list, so that a receptacle could be held constant while only the cable changed. The measured results were:

A $15 cable took the bus from 480 to 5000 Mbps on two handsets, on the same physical receptacle, confirming that four earlier unmarked cables voted 480.
The Pixel 9a's negotiated link went 425 to 3750 Mbps, 8.8x, with the receptacle and the DHCP lease held constant.
Headroom went from 91% consumed to about 7% consumed. The best USB 2.0 pass in 45 records used 91% of what that bus can usably carry; the best SuperSpeed passes use 7.3% and 7.4% of theirs.
Four aggregates cleared 300 Mbps — 370.695, 365.185 and 361.330, plus a 351.1 Mbps single 32 MB transfer — none of them reachable on a USB 2.0 bus.
5000 and 10000 Mbps of bus measured the same — 370.695 against 361.330 — so once the ceiling clears the traffic, more ceiling buys nothing.
Four cables and three ports produced an apparent 73 → 130 Mbps improvement that was entirely the carrier recovering, and zero USB errors throughout said so.

Scope: one host, two handsets on AT&T, one physical location, measured 2026-09-05 and 2026-09-06, against a fixed endpoint and a fixed 8 MB transfer size across 45 records. The first SuperSpeed passes moved congestion control alongside the bus, running cubic where the 2026-09-05 records ran bbr; that confound has since been closed by matched BBR passes on 2026-09-08 — 365.185 Mbps aggregate against the CUBIC pass's 370.695 on the same port and bus — so the bus, not the congestion control, is what the SuperSpeed figures rest on. The 300 Mbps USB 2.0 bulk ceiling is the repository's own diagnostic constant rather than a measurement on this host, so the claim that the fast figures were unreachable on the old cable is a bound argument; no control run has yet put the fast link back onto a USB 2.0 cable, and that re-test is queued. carrier.network is unobtainable on iOS, so carrier conditions across passes cannot be read directly and are inferred from RTT.

The strategy for using a fixed rubric for USB tethering diagnosis was validated with an incremental step by step approach.

Generosity Is a Default Setting

xbill — Sat, 05 Sep 2026 22:04:33 +0000

International Day of Charity falls on September 5, and the most generous thing in most software is not a donation button. It is a default.

Defaults decide what happens to everyone who never opens the settings, which is almost everyone. A default that quietly spends a resource its user is short of is a small unkindness repeated at scale. A default that does not is a gift nobody has to ask for.

Here is one worth changing. For a lot of people the phone is not a second connection — it is the connection, and tethering is how the laptop gets online without a second line, a second bill, or a router that has to be bought before anything works. That tether is often far slower than the radio it sits on, the reason is usually on the laptop rather than the carrier, and the fix is a default setting that costs nothing to change.

This is a tool that finds out which problem you have without spending your data to do it, and the survey that taught it what to look for.

Repository: https://github.com/xbill9/tether

What I Built

tether-report reads every host-observable fact about an attached USB tether, applies a diagnostic rubric, and prints a markdown report telling you what is wrong and what to change.

By default it measures nothing and costs no cellular data.

That default is the whole point. The standard measurement pass in this repo is three single-stream transfers, a four-stream parallel test and a ping — roughly 56 MB of metered data. On an unlimited home plan that is nothing. On a prepaid plan bought by the gigabyte, spending 56 MB to discover that your problem is a one-line setting is a bad trade, and it is exactly the trade most speed-test tools make you take before they tell you anything.

So the tool splits the two. Everything the kernel already knows — the driver, the bus speed, the negotiated link rate, the MTU ceiling, the NCM aggregation buffers, the congestion control algorithm, the interface error counters — is free to read, and most real problems are visible in it. --measure is opt-in, and it prints what it will cost before it spends it.

The rubric it applies is the same one the survey was written against:

Shape	Diagnosis	Where the fix is
🥇 Single-stream spread wide, RTT flat	Congestion control collapsing on radio loss	Your laptop. Free
Single ≈ 4-stream aggregate	Genuinely WAN-limited	Nowhere. Nothing to tune
Aggregate near 300 Mbps on a 480 Mbps bus	USB 2.0 ceiling	Cable or port
RTT `mdev` high, average unchanged	Bufferbloat	Depends what you run
Spread flat, aggregate far above single	Per-flow limit or shaping	Carrier, probably

The first row is the one worth having, because it is the only one where the answer is both free and on your side of the link.

Demo

A Pixel 9a on Google Fi, attached over USB-C. This is a real run, captured while writing this article, and it spent no cellular data at all:

$ python3 bin/tether-report

It opens with what the link actually is:


Interface	`enxce1d58e89c0f`
Driver	`cdc_ncm`
IPv4	`10.244.144.215/24`
MTU / max	1500 / 1500
Negotiated link	425 Mbps
Holds default route	yes

Then the bus, which is where the ceiling usually is:


Device	`3-2` 18d1:4eec - Pixel 9a
Bus speed	480 Mbps (USB 2.0)
BOS capability	SuperSpeed
Type-C connector	`port1`

And then the part that matters — the rubric's verdict on all of it:

BLOCKER: Another route-capable link is up

wlo1 is up alongside the tether. If the tether drops mid-run, curl still succeeds and the result looks like a tethering measurement. Take these down first.

Problem: Device can do better than 480 Mbps

The device advertises SuperSpeed in its BOS descriptor but enumerated at 480 Mbps. Connector port1 has an enumerated SuperSpeed half, so the host is declared capable and the cable is the leading suspect — an unmarked C-to-C cable is very often USB 2.0.

Note: NCM aggregation buffers

rx_max=16384, tx_max=16384. Compare these against the device-advertised dwNtbInMaxSize/dwNtbOutMaxSize: if they already match, the common "raise these to 32768" advice is a no-op and changing them will do nothing.

Note: No measurements taken

Run again with --measure for the throughput and latency half of the rubric. That costs roughly 56 MB of metered cellular data.

Four useful things about this link, and the price was zero.

The blocker is the one I want to point at. Wi-Fi was still up, so any transfer run at that moment could have gone out over Wi-Fi and come back looking like a tethering result. That is not a hypothetical — it is the most common way a tethering measurement quietly becomes a measurement of something else, and a tool that charged you 56 MB before mentioning it would have charged you for a number that meant nothing.

The last note is the design in one line. It did not measure. It told me what measuring would cost and let me decide.

One difference from the records in this repo: the tether was enabled here with adb shell svc usb setFunctions ncm rather than the phone's own tethering toggle, and it enumerated as 18d1:4eec where the recorded passes show 18d1:4eeb. Same driver and same interface, slightly different USB function composition.

Code

Everything is in one public repository: https://github.com/xbill9/tether

Path	What it is
`bin/tether-report`	The diagnostic. One file, standard library only
`README.md`	The format spec, field reference and the diagnostic rubric
`tests/`	40 measurement records, one markdown file each
`INDEX.md`	One row per test
`TEMPLATE.md`	The skeleton a new record is copied from

tether-report has no dependencies beyond the Python standard library and the curl, ping and ip binaries. That is deliberate: a tool for people on expensive connections should not begin by downloading a dependency tree.

git clone https://github.com/xbill9/tether
cd tether
python3 bin/tether-report

No install step, no virtualenv, no package manager. If it runs, it runs.

How I Built It

The measurements came first, the tool second

The tool is a rubric with a reader attached, and the rubric came out of taking the same measurement forty times and noticing which distinctions actually separated one failure from another.

Three rules turned out to matter, and all three exist because ignoring them produced a wrong answer first:

Record three runs, never one. The Pixel 9a produced 15, 44 and 116 Mbps on three consecutive identical transfers. Any one of those numbers, reported alone, is a lie about the link. The spread is the finding.

Always run the parallel test too. Single-stream against four-stream aggregate is the one comparison that separates "the carrier is slow" from "congestion control is collapsing on this machine". If four streams together go much faster than one, the WAN has headroom and the problem is local. That distinction drove every fix worth making.

Record the congestion control algorithm. It is the largest single lever available, and a record without it cannot be interpreted.

The finding

Same phone, same cable, same session. Only the host TCP settings changed:

net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_mtu_probing = 1

	Single-stream runs (Mbps)	Spread	RTT avg	RTT mdev
CUBIC, as shipped	15 / 44 / 116	7.73x	29.6 ms	4.1 ms
BBR	125 / 153 / 106	1.44x	29.7 ms	5.8 ms

The worst run went from 15 Mbps to 106 Mbps. Average round-trip time did not move — 29.6 to 29.7 ms — which is what rules out the boring explanation that BBR simply bought throughput by filling a queue.

The mechanism is that CUBIC treats packet loss as a congestion signal. A radio drops packets for reasons that have nothing to do with congestion, CUBIC halves its window anyway, and a single flow spends its life recovering. BBR models the path instead of reacting to loss, so random drops stop being interpreted as a full pipe.

The part that generalises, and the part that does not

This is a mechanism argument, not a second measurement, and it should be read as one. CUBIC's failure is triggered by random loss. Weak signal produces more random loss. So the worse the radio, the more this setting should be worth — and the people on the worst radios are the ones for whom a free fix matters most.

I have not measured that. Every record in this survey was taken on Google Fi or AT&T, in one location, on networks whose 120 individual transfers ran from 5.3 to 163.0 Mbps with a median of 69.8. Nothing here was collected anywhere under-served, and I am not going to claim otherwise. What I can say is that the mechanism does not depend on geography, that the fix costs nothing to try, and that tether-report will tell you for free whether your link has the shape the fix addresses.

What the survey actually showed

Across 40 records and 14 handsets:

Every single record enumerated at 480 Mbps. Fourteen handsets, three drivers, and not one USB 3.0 negotiation anywhere in 40 records. The bus is a more consistent ceiling than the radio. On the one phone chased hardest — an iPhone 17 Pro that advertises SuperSpeedPlus — 480 survived four cables and three physical ports, and that hunt is still open in the repo. No cable tried was verifiably wired for SuperSpeed, so those four results may be closer to one result repeated.
The driver is not the story. cdc_ncm, rndis_host and ipheth all appear at both ends of the results.
The radio label predicts nothing. Two Galaxy S24 units differing only in radio — 5G against LTE — and the LTE unit won on every measure, including 14.4 ms of round-trip time.
Re-running the same phone moves the answer more than changing phones does. Which is the finding that makes a single-run speed test close to worthless.

Scope and limits

Forty records, fourteen handsets, two carriers (Google Fi and AT&T), one host, one physical location, all on 2026-09-05. The endpoint and transfer size are fixed by the README so records stay comparable. The CUBIC-against-BBR comparison rests on two CUBIC records against 38 BBR ones, so it should be read as a strong result on one phone plus a consistent second case, not as a survey-wide average. Radio type is unrecorded on several records because a reading taken after a pass does not establish what was in use during it, and those fields are left blank rather than guessed. No measurement in this repository has ever been edited; a re-test after a configuration change is always a new file.

Three defaults, and who pays for them

The theme this weekend is generosity, and the argument of this piece is that defaults are where most of it either happens or does not. Three of them here:

The diagnostic's default spends nothing. Reading the kernel is free, so the free half does the diagnosing and --measure is opt-in and prices itself before it runs. The alternative default — measure first, explain later — bills the user in the exact currency they came in short of.
The fix is a default someone else already chose for you. CUBIC ships as the kernel's congestion control, it is the wrong choice on a lossy radio, and changing it costs no hardware, no plan change and no subscription. Every phone in this survey was carrying that cost silently until it was measured.
The method is published, not just the conclusion. The rubric, the record format, all 40 records and the reasoning behind them are in the repository, against a fixed endpoint and transfer size so a stranger's numbers compare to these. A conclusion you cannot check is a thing you have to take on trust; a method is a thing you can own.

The most useful thing I can give someone whose connection is their phone is not my numbers. It is a way to get their own, at no cost, and a fix that was already paid for before they arrived.

Prize Categories

None. This entry does not use Snowflake, Solana, ElevenLabs or Google AI, so it is submitted to the overall category only.

Summary

The goal of this article was to make a slow USB tether diagnosable without spending metered data to diagnose it. The key to the solution was separating what the kernel already knows from what has to be paid for in bytes, and putting the entire rubric on the free side. The measured results were:

Worst single-stream run moved from 15 Mbps to 106 Mbps on one free setting
Single-stream spread collapsed from 7.73x to 1.44x
Round-trip time was unchanged at 29.6 against 29.7 ms, ruling out queue depth
40 records, 14 handsets, 0 that negotiated USB 3.0

Measured on one host across two US carriers in a single location on 2026-09-05, three single-stream runs plus a four-stream parallel pass per record, against a fixed endpoint and transfer size; the congestion-control comparison rests on two CUBIC records against 38 BBR ones.

The strategy for using a fixed rubric for tethering diagnosis was validated with an incremental step by step approach.

AWS Has Two Iceberg REST Catalogs: What Each One Actually Serves

xbill — Fri, 04 Sep 2026 18:29:18 +0000

This article provides a step by step comparison of the two Apache Iceberg REST catalog implementations AWS ships — AWS Glue and Amazon S3 Tables — measuring what each one actually serves. A Python probe harness issues one identical request suite to both and stores the raw response as evidence.

https://github.com/xbill9/lakehouse-iceberg-2026

AWS ships two Iceberg REST catalogs. Both are managed, both are SigV4-signed, and both implement the same published specification. If you are choosing between them, or writing a client that has to work against either, the interesting question is where they diverge.

They score identically and behave differently in thirteen places.

All results below were measured on 2026-09-03.

What Is the Iceberg REST Catalog?

An Iceberg table is a directory of Parquet files plus a chain of JSON metadata files recording which files belong to the table right now. Something has to hold the pointer to the current metadata file, and make commits atomic by swapping it. That is the catalog.

The REST catalog is one HTTP API for that job, so an engine needs one driver rather than one per catalog. The specification lives in the Iceberg repository as open-api/rest-catalog-open-api.yaml and defines 35 operations.

$ curl -sL https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml -o irc.yaml
$ grep -cE '^    (get|post|delete|head|put):' irc.yaml
35

This harness probes 25 of those 35, or 71%.

At This Point You Should Have

An AWS account, and credentials with enough privilege to create a Glue database, an S3 bucket and an S3 Tables table bucket
Python 3.13 with requests, pyiceberg and botocore
botocore available for SigV4 signing

$ aws sts get-caller-identity --query Arn --output text
arn:aws:iam::AWS_ACCOUNT_ID:root
$ python3 -c "
import sys, pyiceberg, requests, botocore
print('python    ', sys.version.split()[0])
print('pyiceberg ', pyiceberg.__version__)
print('requests  ', requests.__version__)
print('botocore  ', botocore.__version__)"
python     3.13.13
pyiceberg  0.12.0
requests   2.34.2
botocore   1.43.34

Everything below was measured with account root, so no result here is a permissions artefact.

Where Are the Two Endpoints?

They are different services with different signing names.

	AWS Glue	Amazon S3 Tables
Endpoint	`https://glue.us-east-1.amazonaws.com/iceberg`	`https://s3tables.us-east-1.amazonaws.com/iceberg`
SigV4 signing name	`glue`	`s3tables`
Warehouse	the account id	the table bucket ARN

$ # GET /v1/config against Glue
{"defaults":{"header.Content-Type":"application/x-amz-json-1.1","rest.sigv4-enabled":"true",
 "rest-table-scan-enabled":"true","prefix":"AWS_ACCOUNT_ID","rest.signing-region":"us-east-1",
 "rest.signing-name":"glue", ...}}

Signing Is Where the First Hour Goes

SigV4 signs the exact query string, so the URL you sign must be the URL you send. Building the URL and then letting an HTTP client re-encode the parameters separately produces a different canonical string:

$ # signing one URL, sending another
HTTP 403
{"message":"The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method."}

Build it once and send it whole. urlencode's default quote_plus is also wrong here, because SigV4 wants %20 rather than +. With both fixed, the same request reaches the service and returns a real answer:

$ # same probe, correct canonical query string
HTTP 404
{"error":{"code":404,"message":"The specified bucket does not exist.","type":"no_such_bucket"}}

A 403 signature error and a 404 no-such-bucket look equally like failure in a log. Only one of them is about the catalog.

Bringing Up AWS Glue

$ aws s3api create-bucket --bucket $BUCKET --region us-east-1
$ aws glue create-database --region us-east-1 \
    --database-input "{\"Name\":\"probe_ns\",\"LocationUri\":\"s3://$BUCKET/probe_ns/\"}"
$ aws glue get-databases --region us-east-1 --query 'DatabaseList[].Name' --output text
probe_ns

Glue rejects createTable without an explicit table location:

$ # createTable with no location field
InvalidInputException: Location information cannot be null while creating an iceberg table

Every other catalog in this suite infers it from the warehouse. Glue does not.

Bringing Up Amazon S3 Tables

$ ARN=$(aws s3tables create-table-bucket --name iceberg-probe \
    --region us-east-1 --query arn --output text)
$ echo $ARN
arn:aws:s3tables:us-east-1:AWS_ACCOUNT_ID:bucket/iceberg-probe
$ aws s3tables create-namespace --table-bucket-arn "$ARN" \
    --namespace probe_ns --region us-east-1
{"tableBucketARN": "arn:aws:s3tables:us-east-1:AWS_ACCOUNT_ID:bucket/iceberg-probe",
 "namespace": ["probe_ns"]}

Two constraints surface only when you hit them. Namespace names reject uppercase, which a timestamped scratch namespace will contain:

$ # createNamespace named irc_probe_20260903T1626
HTTP 400
The specified namespace name isn't valid. Specify a different namespace name, and
then try again.

And createTable requires stage-create in the body:

$ # createTable without it
HTTP 400  stage-create is a required field and cannot be null

Seeding a table also failed against the managed bucket with pyiceberg's default writer:

$ # appending with the default PyArrow FileIO
AWS Error [code 134] during CreateMultipartUpload operation: The authorization
mechanism you have provided is not supported. Please use Signature Version 4.

Switching to pyiceberg.io.fsspec.FsspecFileIO worked. Note the bucket name in that error is not the one you created — S3 Tables stores data in a managed bucket of its own.

The Prefixes Do Not Look Alike

Every client reads the routing prefix from /v1/config and puts it in every later URL. The two services return very different shapes:

Catalog	Prefix as returned
Glue	`catalogs/AWS_ACCOUNT_ID`
S3 Tables	`arn%3Aaws%3As3tables%3Aus-east-1%3A...%3Abucket%2Ficeberg-probe`

Glue returns two path segments. S3 Tables returns a percent-encoded ARN. A client that assumes one segment, or that re-encodes what it was handed, produces URLs neither service routes.

The Scores Are Identical

	🥈 AWS Glue	🥈 Amazon S3 Tables
Read probes served	9/15	9/15
Write probes served	10/17	10/17
Not tested	1	1
`loadTable` fields present	26/30	27/30
Endpoints declared in `/v1/config`	none	none

Read and write surfaces are scored separately rather than summed, and probes whose prerequisite failed are excluded rather than counted as failures.

That table is the least interesting thing in this article. The two implementations arrive at the same totals by different routes.

Neither One Tells You What It Supports

The specification lets a server advertise its own surface:

endpoints: A list of endpoints that the server supports.

— rest-catalog-open-api.yaml, CatalogConfig

Five of the seven catalogs in the wider comparison publish that array. Neither AWS catalog does:

$ python3 -c "
import json
for c in ('aws-glue','aws-s3tables'):
    d = json.load(open('evidence/%s.json' % c))
    print('%-14s declares %s endpoints' % (c, len(d.get('declared_endpoints') or []) or 'no'))
"
aws-glue       declares no endpoints
aws-s3tables   declares no endpoints

The field is optional, so this is not a specification violation. It does mean capability discovery is unavailable on both, and a client has no way to learn what either serves short of probing it — which is what this harness does.

Thirteen Probes Behave Differently

Of 33 probes, 20 return the same verdict on both and 13 do not.

Probe	AWS Glue	Amazon S3 Tables
`list_views`	❌ 406 not supported	❌ 404 unknown operation
`create_view`	❌ 406 not supported	❌ 404 unknown operation
`load_view`	❌ 406 not supported	❌ 404 unknown operation
`replace_view`	❌ 406 not supported	❌ 404 unknown operation
`rename_view`	❌ 406 not supported	❌ 404 unknown operation
`drop_view`	❌ 406 not supported	❌ 404 unknown operation
`rename_table`	❌ 406 not supported	✅ 204
`update_namespace_props`	✅ 200	❌ 404 unknown operation
`plan_table_scan`	⚠️ 200 with exception	❌ 404 unknown operation
`report_metrics`	⚠️ 200 with exception	❌ 400 not supported
`commit_transaction`	⚠️ 200 with exception	❌ 404 unknown operation
`drop_table_purge`	❌ 400 purge forbidden	✅ 204
`drop_table`	✅ 204	❌ 400 purge required

Neither implements views, but they refuse differently, and one of them renames tables while the other does not.

Glue Names the Operation It Is Refusing

Glue's refusals are specific, and they name the endpoint:

$ # POST .../views against Glue
HTTP 406  CreateView endpoint is not supported for Glue Catalog.
$ # POST .../tables/rename against Glue
HTTP 406  RenameTable endpoint is not supported for Glue Catalog.

S3 Tables returns a bare XML document with no operation name and no JSON error body:

$ # POST .../views against S3 Tables
HTTP 404  <UnknownOperationException/>

Both mean the same thing to a user and not to a program. Glue's 406 with a named operation is machine-readable enough to log usefully; the bare exception is not.

The exception to S3 Tables' silence is report_metrics, which is the one place it says what it means:

$ # POST .../tables/{table}/metrics against S3 Tables
HTTP 400  ReportMetrics is currently not supported.

Glue Answers Three Unrouted Operations With HTTP 200

This is the finding a client author should care about most.

$ # POST .../tables/{table}/plan against Glue
HTTP 200
{"Output": {"__type": "com.amazon.coral.service#UnknownOperationException"}, "Version": "1.0"}

The same body comes back from report_metrics and commit_transaction. The mechanism is the AWS protocol layer answering an operation its front door does not route, rather than Glue returning a broken success for an implemented endpoint.

The consequence does not depend on the mechanism. Code that branches on the status code sees three endpoints that work, and only code that parses the body finds out otherwise. The harness gives these their own verdict rather than scoring them OK.

Glue's /v1/config also advertises rest-table-scan-enabled: true, and scan planning is one of the three.

The Drop Requirements Are Opposite

Glue refuses a purge drop. S3 Tables refuses a plain one.

$ # DELETE .../tables/{table}?purgeRequested=true against Glue
HTTP 400  PurgeRequested cannot be true for Glue iceberg tables.

$ # DELETE .../tables/{table} against S3 Tables
HTTP 400  DropTable operation failed. S3 Tables only supports dropping tables with
purge enabled.

The two products have different storage models, so this is defensible rather than a defect — Glue points at a bucket you own, and S3 Tables owns the storage it drops. It is still the sharpest example of why "AWS supports the Iceberg REST catalog" is not a sentence a client can act on.

Neither Supports Multi-Level Namespaces

Both reject the parent query parameter on listNamespaces, and both say so plainly:

$ # GET .../namespaces?parent=probe_ns against Glue
HTTP 400  Glue dataCatalog does not support multipart namespace.
$ # the same against S3 Tables
HTTP 400  Multipart namespaces are not supported.

Where They Agree

Twenty of the 33 probes return the same verdict, and the agreement is the core of the specification. Both serve config, namespace listing and loading, table listing and loading, loadTable with full snapshot history, and both accept all five updateTable actions probed — set-properties, remove-properties, add-schema, set-current-schema and upgrade-format-version.

Both also return nearly identical loadTable documents: 26 of 30 checked specification field paths on Glue, 27 on S3 Tables. The single extra on S3 Tables is a config block. Whatever separates these two, it is not the fidelity of the metadata they return.

Summary

The goal of this article was to measure what AWS's two Iceberg REST catalog implementations actually serve, rather than what "supports the REST catalog" implies. The key to the solution was issuing one identical request suite to both and storing the raw response for every probe. The comparison results were:

Identical totals, thirteen behavioural differences. Both serve 9 of 15 read probes and 10 of 17 write probes, and 20 of 33 probes agree.
Neither publishes an endpoints declaration, so capability discovery is unavailable on both and a client must probe.
Glue answers three unrouted operations with HTTP 200 carrying an UnknownOperationException, one of which is the scan planning its own config advertises as enabled.
Neither implements views, and they refuse differently — Glue with a 406 naming the operation, S3 Tables with a bare <UnknownOperationException/>.
The drop requirements are opposite. Glue forbids purgeRequested=true, S3 Tables requires it.
rename_table works on S3 Tables and not on Glue.

Scope: both catalogs probed once in us-east-1 on 2026-09-03 with account root, against one table shape seeded through pyiceberg 0.12.0, covering 25 of the specification's 35 operations, 5 of its 25 update actions and 1 of its 8 table requirements; neither service exposes a version, so no result here can be tied to a release; and the field tier records that a value is present, never that it is correct.

The strategy for comparing two managed catalogs against one specification was validated with an incremental step by step approach.

Seven Iceberg REST Catalogs: What They Declare, and What They Serve

xbill — Fri, 04 Sep 2026 18:29:02 +0000

This article provides a step by step comparison of seven Apache Iceberg REST catalog implementations, measuring what each one actually serves against the published specification. A Python probe harness is built to issue one identical request suite to every catalog and store the raw response as evidence.

https://github.com/xbill9/lakehouse-iceberg-2026

Every vendor with a lakehouse now ships an Iceberg REST catalog, and every one of them says it implements the same specification. That claim is checkable. The specification also asks a server to publish a machine-readable list of the endpoints it supports, which makes a second claim checkable: does a catalog agree with itself?

All results below were measured on 2026-09-03.

Nothing here is about the REST catalog being broken. It works — every catalog served the core read path.

What Do You Need to Reproduce This?

Seven catalogs means seven accounts, and three of mine were trials that will have expired by the time most people read this. Read this section before investing an afternoon.

Catalog	What you need	What I used
Apache Polaris	Docker	local container, free
Google BigLake	GCP project, Lakehouse API enabled	owned project
AWS Glue	AWS account	owned account
AWS S3 Tables	AWS account	owned account
Databricks Unity	Premium workspace	14-day trial
Snowflake Horizon	any Snowflake account	30-day Enterprise trial
Microsoft OneLake	Fabric licence on a work account	60-day Fabric trial capacity

Three of those need saying plainly.

Databricks Free Edition will not work. Enabling external data access is an account-level action, and Free Edition documents no access to the account console.

A personal Microsoft account cannot hold a Fabric licence. The Fabric API returns UserNotLicensed and no configuration changes it. A work or school account in an Entra tenant can, and the free 60-day trial capacity is enough — the paid F-SKU the Azure portal steers you toward is not needed.

Snowflake Open Catalog is closed to new signups. Snowflake's documentation directs new customers to Horizon, which is what this measures.

Being on a trial tier is also a genuine confound in the results, not only an inconvenience. The limitations section returns to it.

What Is Apache Iceberg?

Iceberg is a table format. A table is a directory of Parquet files plus a chain of JSON metadata files recording which of those files belong to the table right now, what the schema is, how it is partitioned, and what it looked like at every previous commit. That history is why engines can time-travel and why two writers can commit without corrupting each other.

Something has to hold the pointer to the current metadata file. That something is the catalog. It answers one question — for table X, where is the metadata right now — and it makes commits atomic by swapping that pointer.

What Is the REST Catalog?

For years each engine brought its own catalog: Hive Metastore, a Glue client, a JDBC catalog, a filesystem convention. Every engine needed a driver for every catalog.

The Iceberg REST catalog replaces that with one HTTP API. A client speaks HTTP to a URL and the vendor implements the endpoints behind it. The specification lives in the Iceberg repository as open-api/rest-catalog-open-api.yaml.

$ curl -sL -o irc.yaml https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml
$ python3 - <<'EOF'
import re
lines = open('irc.yaml').read().split('\n')
inpaths = False; cur = None; ops = []
for l in lines:
    if re.match(r'^paths:', l): inpaths = True; continue
    if inpaths and re.match(r'^\S', l): break
    m = re.match(r'^  (/\S*):\s*$', l)
    if m: cur = m.group(1); continue
    m2 = re.match(r'^    (get|put|post|delete|head|patch):\s*$', l)
    if m2 and cur: ops.append('%s %s' % (m2.group(1).upper(), cur))
print("spec operations:", len(ops))
EOF
spec operations: 35

Thirty-five operations: listing namespaces, loading a table, committing an update, creating a view, planning a scan.

What Does a Catalog Say About Itself?

The endpoint every client calls first is GET /v1/config. It returns the routing prefix for every later request, and it may return one more thing:

endpoints: A list of endpoints that the server supports. The format of each endpoint must be "<HTTP verb> <resource path from OpenAPI REST spec>".

— rest-catalog-open-api.yaml, CatalogConfig

That field is the reason this article exists. A catalog publishes a machine-readable list of what it supports, and nothing stops a client from trusting it.

Throughout, declared means named in that array and served means returned a 2xx to the probe. Where a catalog declares an endpoint and does not serve it, I call that an overclaim, on the strength of the spec's own word supports.

What This Suite Does Not Do

Apache ships a REST Compatibility Kit, RESTCompatibilityKitSuite, which tests a server's behaviour against the Java reference implementation. It is the right tool for "is this catalog correct". It does not compare a server's declaration against what that server serves, which is the axis here. The two are complementary.

This suite also never checks whether a returned value is right. It checks whether a field is present.

The Harness

Three tiers of evidence, from one identical request suite:

Endpoint tier — does the operation exist, and what status comes back
Field tier — 30 specification field paths checked against each loadTable
Declaration tier — the endpoints array cross-checked against behaviour

$ python3 -c "
import sys; sys.path.insert(0,'.')
from probe import spec
sigs = {p.signature() for p in spec.PROBES + spec.WRITE_PROBES}
print('probes:', len(spec.PROBES) + len(spec.WRITE_PROBES))
print('distinct endpoint signatures:', len(sigs))
"
probes: 33
distinct endpoint signatures: 25

Twenty-five of the specification's 35 operations, or 71%.

Two design rules matter for reading the results. Raw request and response are stored for every probe, so a verdict is re-derivable without re-running against a vendor. And a probe whose prerequisite failed is marked not-tested rather than failed — if a catalog refuses to create a namespace, the probes that needed one prove nothing about the endpoints they target.

At This Point You Should Have

Docker, for the control catalog
Python 3.13 with requests, pyiceberg and botocore
An account on whichever vendors you intend to probe
The maximum privilege available on each — every result below was gathered with roles/owner, AWS account root, ACCOUNTADMIN, Databricks account admin, Fabric workspace Admin, and the Polaris root principal

$ python3 -c "
import sys, pyiceberg, pyarrow, requests, botocore
print('python     ', sys.version.split()[0])
print('pyiceberg  ', pyiceberg.__version__)
print('pyarrow    ', pyarrow.__version__)
print('requests   ', requests.__version__)
print('botocore   ', botocore.__version__)
"
python      3.13.13
pyiceberg   0.12.0
pyarrow     24.0.0
requests    2.34.2
botocore    1.43.34

Bringing Up Apache Polaris, the Control

Start with the control, not the clouds. A red cell in a permissively configured reference implementation is almost always your bug, not a specification gap.

$ docker run -d --name polaris -p 8181:8181 -p 8182:8182 \
    --user "$(id -u):$(id -g)" \
    -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro \
    -v "$WH:$WH" \
    -e HADOOP_USER_NAME="$(id -un)" \
    -e POLARIS_BOOTSTRAP_CREDENTIALS=POLARIS,root,s3cr3t \
    -e JAVA_OPTS_APPEND="-Dpolaris.features.\"ALLOW_INSECURE_STORAGE_TYPES\"=true \
       -Dpolaris.features.\"SUPPORTED_CATALOG_STORAGE_TYPES\"=[\"FILE\"] \
       -Dpolaris.readiness.ignore-severe-issues=true \
       -Dpolaris.features.\"DROP_WITH_PURGE_ENABLED\"=true" \
    apache/polaris:latest
$ curl -sf http://localhost:8182/q/health/ready
{"status": "UP", "checks": [...]}

Four things there are not obvious, and each one cost a failed attempt.

FILE storage is refused by default, and enabling it needs both ALLOW_INSECURE_STORAGE_TYPES and SUPPORTED_CATALOG_STORAGE_TYPES. Enabling it then escalates the production-readiness check from warning to fatal:

$ docker logs polaris 2>&1 | grep -A1 'Severe production'
Caused by: java.lang.IllegalStateException: Severe production readiness issues
detected, startup aborted!

So polaris.readiness.ignore-severe-issues is required as well. Third, the container writes table metadata while the client writes data files, so both need the same warehouse path — hence the bind mount at an identical absolute path and --user so files are owned by you.

Fourth, and the one that wastes an hour: with --user, Hadoop's UserGroupInformation cannot resolve the uid, and the failure surfaces as a 503 that reads like a storage error.

$ # symptom before mounting /etc/passwd
RuntimeIOException: Failed to get file system for path: file:/.../metadata/00000-....json

Mounting /etc/passwd read-only fixes it. DROP_WITH_PURGE_ENABLED is on deliberately: with it off, Polaris refuses both a purge drop and dropView, which would put artificial red cells in the control.

Then the catalog and its grants:

$ TOK=$(curl -s -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \
    -d grant_type=client_credentials -d client_id=root -d client_secret=s3cr3t \
    -d scope=PRINCIPAL_ROLE:ALL | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
$ curl -s -o /dev/null -w "create catalog: HTTP %{http_code}\n" \
    -X POST http://localhost:8181/api/management/v1/catalogs \
    -H "Authorization: Bearer $TOK" -H "Content-Type: application/json" \
    -d "{\"catalog\":{\"name\":\"quickstart_catalog\",\"type\":\"INTERNAL\", ... }}"
create catalog: HTTP 201

Bringing Up Google BigLake

The endpoint is https://biglake.googleapis.com/iceberg/v1/restcatalog. Note v1, not v1beta — both resolve and return identical config, but v1 is what the documentation gives.

$ gcloud services enable biglake.googleapis.com --project=$PROJECT
Operation "operations/acat...." finished successfully.
$ gcloud storage buckets create gs://$PROJECT-iceberg-probe \
    --project=$PROJECT --location=us-central1 --uniform-bucket-level-access
Creating gs://$PROJECT-iceberg-probe/...

Two things are required beyond the URL. The API is enabled under the name biglake.googleapis.com, but the error calls it the Lakehouse API:

$ curl -s "https://biglake.googleapis.com/iceberg/v1/restcatalog/v1/config?warehouse=gs://$BUCKET" \
    -H "Authorization: Bearer $(gcloud auth print-access-token)"
{"error": {"code": 403, "message": "Lakehouse API has not been used in project
... before or it is disabled."}}

And every request needs an x-goog-user-project header. With both in place:

$ curl -s "https://biglake.googleapis.com/iceberg/v1/restcatalog/v1/config?warehouse=gs://$BUCKET" \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "x-goog-user-project: $PROJECT"
{"overrides":{"prefix":"projects/GCP_PROJECT_NUMBER/catalogs/GCP_PROJECT-iceberg-probe",
 "catalog_credential_mode":"CREDENTIAL_MODE_END_USER"},"defaults":{...},"endpoints":[...]}

The warehouse is gs://BUCKET for a single-bucket catalog or bl://projects/PROJECT/catalogs/CATALOG for a multi-bucket one — not bq://, which addresses the separate BigQuery federation endpoint.

Seeding writes data files to GCS with application-default credentials, which are separate from the gcloud user token. A invalid_rapt error means gcloud auth application-default login.

Bringing Up AWS Glue

The endpoint is https://glue.us-east-1.amazonaws.com/iceberg, SigV4-signed with signing name glue, and the warehouse is the account id.

$ aws s3api create-bucket --bucket $BUCKET --region us-east-1
$ aws glue create-database --region us-east-1 \
    --database-input "{\"Name\":\"probe_ns\",\"LocationUri\":\"s3://$BUCKET/probe_ns/\"}"
$ aws glue get-databases --region us-east-1 --query 'DatabaseList[].Name' --output text
probe_ns

Glue rejects createTable without an explicit table location, which every other catalog here infers:

$ # createTable with no location
BadRequestException: Location information cannot be null while creating an iceberg table

Bringing Up AWS S3 Tables

The endpoint is https://s3tables.us-east-1.amazonaws.com/iceberg, SigV4 with signing name s3tables, and the warehouse is the table bucket ARN.

$ ARN=$(aws s3tables create-table-bucket --name iceberg-probe \
    --region us-east-1 --query arn --output text)
$ echo $ARN
arn:aws:s3tables:us-east-1:AWS_ACCOUNT_ID:bucket/iceberg-probe
$ aws s3tables create-namespace --table-bucket-arn "$ARN" \
    --namespace probe_ns --region us-east-1
{"tableBucketARN": "...", "namespace": ["probe_ns"]}

Two constraints do not appear until you hit them. Namespace names reject uppercase, so a scratch namespace stamped irc_probe_20260903T1626 fails on the T:

$ # createNamespace with an uppercase character
The specified namespace name isn't valid. Specify a different namespace name.

And createTable requires stage-create in the body. S3 Tables' managed bucket also rejected pyiceberg's default PyArrow writer:

$ # seeding with the default FileIO
AWS Error [code 134] during CreateMultipartUpload operation: The authorization
mechanism you have provided is not supported. Please use Signature Version 4.

Switching to pyiceberg.io.fsspec.FsspecFileIO worked.

Bringing Up Databricks Unity

The endpoint is https://<workspace>.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest. Three prerequisites, in order.

External data access is off by default, per metastore:

$ curl -s -H "Authorization: Bearer $TOKEN" "$HOST/api/2.1/unity-catalog/metastore_summary" \
    | python3 -c "import sys,json;print(json.load(sys.stdin)['external_access_enabled'])"
False
$ curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \
    "$HOST/api/2.1/unity-catalog/metastores/$METASTORE_ID" \
    -d '{"external_access_enabled": true}' | python3 -c "import sys,json;print(json.load(sys.stdin)['external_access_enabled'])"
True

Second, the privilege, which must be granted on the catalog:

$ # on a schema in the default workspace catalog
GRANT EXTERNAL USE SCHEMA ON SCHEMA workspace.probe_ns TO `you@example.com`
[INVALID_PARAMETER_VALUE.PRIVILEGE_NOT_APPLICABLE_TO_ENTITY] Privilege EXTERNAL
USE SCHEMA is not applicable to this entity [workspace.probe_ns:SCHEMA/SCHEMA_DB_STORAGE]

$ GRANT EXTERNAL USE SCHEMA ON CATALOG workspace TO `you@example.com`
SUCCEEDED

Third, and the one worth knowing: the Iceberg endpoint requires a token with the all-apis scope. A PAT scoped to unity-catalog plus sql — sufficient for Unity's own REST API and for the SQL statement API — is rejected:

$ curl -s -H "Authorization: Bearer $SCOPED_TOKEN" \
    "$HOST/api/2.1/unity-catalog/iceberg-rest/v1/config?warehouse=workspace"
{"error_code":403,"message":"Provided access token does not have required scopes: all-apis"}

Probing the open-standard surface needs a broader credential than the vendor's own APIs do.

Unity also rejects two things during seeding — identifier_field_ids returns Table with identifier columns is not allowed, and managed Iceberg tables reject write.delete.mode with MANAGED_ICEBERG_OPERATION_NOT_SUPPORTED. Finally, its vended credentials explicitly deny s3:PutObject on Databricks-managed storage, so an external engine can create a table through the REST catalog but cannot write data files into it:

$ # pyiceberg appending through Unity's vended credentials
AWS Error ACCESS_DENIED during CreateMultipartUpload operation: User: ... is not
authorized to perform: s3:PutObject ... with an explicit deny in a resource-based policy

That table was seeded through Databricks SQL instead.

Bringing Up Snowflake Horizon

The endpoint is https://<account>.snowflakecomputing.com/polaris/api/catalog — the same path as Open Catalog, because Apache Polaris is embedded in Horizon. The warehouse is a Snowflake database name, not a warehouse.

Authentication is key-pair, so no account password is handled:

$ openssl genrsa -out sf_key.pem 2048
$ openssl rsa -in sf_key.pem -pubout -out sf_key.pub
writing RSA key

ALTER USER MYUSER SET RSA_PUBLIC_KEY='MIIBIjANBgkq...';
Statement executed successfully.

The token exchange differs from stock Polaris in two ways: the scope is session:role:<role> rather than PRINCIPAL_ROLE:ALL, and the client_credentials grant carries a signed JWT in client_secret with no client_id.

$ curl -s -X POST "$HOST/polaris/api/catalog/v1/oauth/tokens" \
    --data-urlencode 'grant_type=client_credentials' \
    --data-urlencode 'scope=session:role:ACCOUNTADMIN' \
    --data-urlencode "client_secret=$JWT" | head -c 60
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...

Snowflake-managed Iceberg tables need an external volume, which needs an IAM role whose trust policy names values you only get after creating the volume:

CREATE OR REPLACE EXTERNAL VOLUME SF_ICEBERG_PROBE
  STORAGE_LOCATIONS = ((NAME='us-east-1-s3' STORAGE_PROVIDER='S3'
    STORAGE_BASE_URL='s3://mybucket/snowflake/'
    STORAGE_AWS_ROLE_ARN='arn:aws:iam::ACCOUNT:role/snowflake-iceberg-probe'))
  ALLOW_WRITES = TRUE;
SF_ICEBERG_PROBE successfully created.

DESC EXTERNAL VOLUME SF_ICEBERG_PROBE;
STORAGE_AWS_IAM_USER_ARN: arn:aws:iam::...:user/...
STORAGE_AWS_EXTERNAL_ID:  ..._SFCRole=...

Update the role's trust policy with those two values, then verify:

SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME('SF_ICEBERG_PROBE');
{"success": true, "writeResult": "PASSED", "readResult": "PASSED",
 "listResult": "PASSED", "deleteResult": "PASSED",
 "awsRoleArnValidationResult": "PASSED"}

Two more. Snowflake rejects TIMESTAMP_TZ(9) for Iceberg tables and wants TIMESTAMP_LTZ(6). And creating a table through the REST catalog requires the database to carry a default external volume, or createTable returns 403 and four downstream probes fail for reasons unrelated to the endpoints they target:

ALTER DATABASE PROBE_DB SET EXTERNAL_VOLUME = 'SF_ICEBERG_PROBE';
Statement executed successfully.

Bringing Up Microsoft OneLake

The endpoint is https://onelake.table.fabric.microsoft.com/iceberg. Warehouse and prefix are both <workspaceId>/<dataItemId>, and the bearer token audience is https://storage.azure.com.

The gate here is licensing, not configuration:

$ FT=$(az account get-access-token --resource https://api.fabric.microsoft.com --query accessToken -o tsv)
$ curl -s -H "Authorization: Bearer $FT" https://api.fabric.microsoft.com/v1/workspaces
{"requestId":"...","errorCode":"UserNotLicensed","message":"User is not licensed","isRetriable":false}

That is a personal Microsoft account, and no configuration changes it. On a work account, with the free 60-day trial capacity activated:

$ curl -s -H "Authorization: Bearer $FT" https://api.fabric.microsoft.com/v1/capacities \
    | python3 -c "
import sys,json
for c in json.load(sys.stdin)['value']:
    print(c['displayName'], c['sku'], c['state'], c['region'])"
Trial-...  FTL4  Active  East US 2

Then a workspace on that capacity, and a lakehouse in it:

$ curl -s -X POST -H "Authorization: Bearer $FT" "$API/workspaces" \
    -d '{"displayName":"iceberg-probe-ws"}' | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])"
FABRIC_WORKSPACE_ID
$ curl -s -o /dev/null -w "assign capacity: HTTP %{http_code}\n" -X POST \
    -H "Authorization: Bearer $FT" "$API/workspaces/$WS/assignToCapacity" -d "{\"capacityId\":\"$CAP\"}"
assign capacity: HTTP 202
$ curl -s -X POST -H "Authorization: Bearer $FT" "$API/workspaces/$WS/lakehouses" \
    -d '{"displayName":"probe_lh"}' | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])"
FABRIC_LAKEHOUSE_ID

The table is loaded from CSV through the Fabric load-table API and stored as Delta. OneLake exposes it through the Iceberg endpoint by virtualisation — its properties carry XTABLE_METADATA with "sourceTableFormat":"DELTA".

Vendor Summary

Read and write surfaces are scored separately and deliberately not summed. A catalog that is read-only by design scores zero on writes, and folding that into one number makes a deliberate design read as a broken implementation.

Catalog	Read probes	Write probes	Not tested	Declares
🥇 Apache Polaris 1.7.0	15/16	16/17	0	36
🥈 Databricks Unity	12/14	10/17	2	18
🥉 Snowflake Horizon	12/14	7/14	5	23
Google BigLake	11/14	10/14	5	15
Microsoft OneLake	11/15	0/14	4	13
AWS Glue	9/15	10/17	1	none
AWS S3 Tables	9/15	10/17	1	none

Denominators differ because probes that could not be tested are excluded rather than counted as failures. The not-tested column makes each one reconstructable.

Two of seven publish no endpoints list at all, so a client cannot discover their capabilities without probing. That is not a specification violation — the field is optional — but it means capability discovery is unavailable on both AWS catalogs.

OneLake's read surface is mid-pack rather than last. Its zero on writes is its documented design.

Why the Prefix Matters

Each catalog's routing prefix, as its own /v1/config returns it:

Catalog	Prefix
Polaris	`quickstart_catalog`
Glue	`catalogs/AWS_ACCOUNT_ID`
S3 Tables	`arn%3Aaws%3As3tables%3A...%3Abucket%2Ficeberg-probe`
Unity	`catalogs/workspace`
BigLake	`projects/GCP_PROJECT_NUMBER/catalogs/GCP_PROJECT-iceberg-probe`
OneLake	`<workspaceId>/<lakehouseId>`
Horizon	`PROBE_DB`

A prefix is not a single path segment. Google returns four, Unity and OneLake return two, and S3 Tables returns a percent-encoded ARN. A client that treats the prefix as one segment, or that re-encodes it, produces URLs the server does not route. I made both mistakes while building this.

Views Are Declared Widely and Implemented Once

The specification has six view operations. Probing only listViews cannot distinguish "views are unimplemented" from "list is unimplemented", so all six are probed.

Catalog	createView, loadView, viewExists, replaceView, renameView, dropView
✅ Polaris	all six served
❌ Glue	406 — `ListViews endpoint is not supported for Glue Catalog`
❌ S3 Tables	404 `UnknownOperationException`
❌ Unity	404 `ENDPOINT_NOT_FOUND`
❌ BigLake	404 `Method not found`
❌ OneLake	404 and 405
❌ Horizon	403 `Authorization failed`, as ACCOUNTADMIN

Horizon declares seven view endpoints and serves none of them. Both Horizon and Unity were re-tested with a real native view present in the namespace, in case the failures were an empty-namespace artefact:

$ # a native view exists in both, created through each vendor's own SQL
$ curl -s -H "Authorization: Bearer $SFT" "$SF/v1/PROBE_DB/namespaces/PROBE_NS/views"
{"error":{"message":"Authorization failed","type":"ForbiddenException","code":403}}

$ curl -s -H "Authorization: Bearer $DBT" "$DBX/v1/catalogs/workspace/namespaces/probe_ns/views"
{"error_code":"ENDPOINT_NOT_FOUND","message":"No API found for 'GET .../views'"}

Neither is a fixture problem, and ACCOUNTADMIN is the highest role in a Snowflake account.

Glue Answers Unrouted Operations With HTTP 200

Three probes against Glue return 200 OK carrying an exception:

$ # POST .../tables/{table}/metrics against Glue
HTTP 200
{"Output": {"__type": "com.amazon.coral.service#UnknownOperationException"}, "Version": "1.0"}

The mechanism is the AWS protocol layer answering an operation its front door does not route, rather than Glue returning a broken success for an implemented endpoint. The consequence for a client is the same either way: code that checks status codes sees three endpoints that work. The harness scores these with their own verdict rather than as OK.

Glue's config also advertises rest-table-scan-enabled: true, and scan planning is one of the three.

Eleven Overclaims Across Four Vendors

Endpoints named in a catalog's own endpoints array that did not serve:

Catalog	Overclaimed
Microsoft OneLake	`create_namespace`, `create_table`, `commit_table`, `rename_table`
Snowflake Horizon	`list_views`, `create_view`, `update_namespace_props`, `commit_transaction`
Databricks Unity	`plan_table_scan`, `update_namespace_props`
Google BigLake	`load_credentials`
Apache Polaris	none
AWS Glue, AWS S3 Tables	not applicable — declare nothing

Only the open-source control declares honestly. Two catalogs cannot be judged on this axis at all.

Two of these deserve their exact wording. Unity's update_namespace_props requires a field the specification does not define:

$ # POST .../namespaces/{namespace}/properties against Unity
HTTP 400
Malformed request: INVALID_PARAMETER_VALUE: Etag token version is missing

The specification's UpdateProperties request has no etag field, so a conformant client cannot call this endpoint at all.

And OneLake's documentation and its declaration are wrong in opposite directions. Microsoft documents the endpoint as read-only, which matches the behaviour — every write returns Requested Api is not found — but /v1/config declares four write endpoints anyway. Meanwhile Microsoft documents that the parent query parameter on listNamespaces is not supported, and measured, it works:

$ curl -s -H "Authorization: Bearer $ST" "$OL/v1/$PREFIX/namespaces?parent=dbo"
{"namespaces":[],"next-page-token":null}
$ curl -s -H "Authorization: Bearer $ST" "$OL/v1/$PREFIX/namespaces?parent=zzz"
{"error":{"message":"The given namespace does not exist","type":"NoSuchNamespaceException"...

An empty list for a namespace with no children, and a typed error for one that does not exist. That is the parameter working.

The Commit Path Has Depth the Endpoint List Does Not Show

POST .../tables/{table} is one endpoint carrying 25 distinct update actions. Five are probed separately. Polaris, Glue, S3 Tables, Unity and BigLake accept all five. Horizon accepts two:

Action	Horizon
✅ set-properties	200
✅ add-schema with set-current-schema:-1	200
❌ remove-properties	409 or 500
❌ set-current-schema, standalone	409 or 500
❌ upgrade-format-version	400

The last is an explicit refusal:

$ # upgrade-format-version to 2 against Horizon
HTTP 400
Upgrading the Iceberg format version of an existing table is not allowed for
Horizon accounts

The two showing two status codes are showing observed nondeterminism. Across back-to-back sweeps both alternate between 409 CONFLICT and 500 SERVER_ERROR on identical input:

$ # two consecutive full sweeps, diffed
snowflake-horizon commit_remove_properties:  CONFLICT     -> SERVER_ERROR
snowflake-horizon commit_set_current_schema: SERVER_ERROR -> CONFLICT
differences across two sweeps: 2

The 500 carries UNEXPECTED_ERROR_SIGNALED ... Indeterminate result during conflict resolution check. Both fail either way, so the scores are stable and the status code is not. Two probes on the same conflict-resolution path behaving this way makes it a property of that path rather than a single flaky request. Every other cell in every catalog was stable across six sweeps.

What Does Not Differ

The suite also checks 30 field paths in each loadTable response — schemas, column IDs, partition specs, sort orders, snapshot history, refs, statistics. Twenty-four of the thirty are identical across all seven catalogs.

Whatever separates these implementations, it is not the fidelity of loadTable. Of the six rows that do differ, one is caused by the fixture rather than the catalog, two reflect storage and credential-vending configuration, and two are the difference between rendering "no statistics" as an empty list and omitting the key.

This is a null result and it is worth stating plainly.

Smaller: Purge Semantics Are Per-Catalog

Dropping a table is not uniform. Polaris refuses purgeRequested=true unless DROP_WITH_PURGE_ENABLED is set, Glue refuses it outright, S3 Tables refuses a plain drop, and BigLake, Unity and Horizon accept the purge form.

$ # DELETE .../tables/{table}?purgeRequested=true against Glue
HTTP 400  PurgeRequested cannot be true for Glue iceberg tables.
$ # DELETE .../tables/{table} against S3 Tables
HTTP 400  DropTable operation failed. S3 Tables only supports dropping tables with purge enabled.

This is worth knowing when writing a client, but it is the weakest result here: these are separate products with different storage models, and a client configures a catalog once rather than swapping between them. I mention it because it is measured, not because it is important.

Summary

The goal of this article was to measure what seven Iceberg REST catalog implementations actually serve, rather than what their documentation claims. The key to the solution was probing the endpoints array each catalog publishes about itself and comparing it against behaviour on the same request suite. The conformance results were:

Eleven overclaims across four vendors. Endpoints a catalog names in its own /v1/config and then does not serve. Only the open-source control, Apache Polaris, declares honestly.
Two of seven publish no declaration at all. Glue and S3 Tables leave a client no way to discover their surface short of probing it.
Views are effectively a Polaris-only feature. Six operations, six managed catalogs, zero implementations.
"Supports the REST catalog" is true and not very informative. All seven served the core read path. The divergence is in the write surface, in the update actions inside one endpoint, and in operational semantics.
loadTable fidelity is not a differentiator. Twenty-four of thirty checked field paths are identical everywhere.

Scope: seven catalogs, each probed once in a single region on 2026-09-03 against one table shape, covering 25 of the specification's 35 operations, 5 of 25 update actions and 1 of 8 table requirements; Polaris pinned to 1.7.0 locally with non-default permissive flags, BigLake in us-central1, Glue and S3 Tables in us-east-1, Unity in us-west-2, Horizon on AWS us-east-1 and OneLake in East US 2; Unity, Horizon and OneLake on trial accounts, which is a genuine confound because one Horizon refusal is worded "not allowed for Horizon accounts"; the six managed catalogs expose no version, so no result can be tied to a release; fixtures are identical on five of the seven, with the other two differing because those catalogs refuse part of the seed; and the field tier checks that a value is present, never that it is correct.

The strategy for using a single request suite across seven vendor catalogs was validated with an incremental step by step approach.

ChromeOS Lookalikes, Two Ways: One With Drivers, One Without

xbill — Thu, 03 Sep 2026 15:18:58 +0000

This article covers two scripts that live in the same repository for the same underlying reason — getting a Linux box to behave like a Chromebook — and that end up doing almost nothing alike, because they solve that problem for two different machines.

https://github.com/xbill9/chromeos-boot

Two Scripts, One Theme

chromeos-boot is not one tool with two modes. It is two standalone scripts that happen to share a repo and a name:

stage — brings a fresh ChromeOS Linux (Crostini) container up from nothing, when everything else it needs lives in a private Cloud Storage bucket.
flex — turns a stock Debian desktop into a ChromeOS Flex lookalike: shelf, web apps, keybindings, wallpaper. No bucket, no gcloud, nothing private.

stage gets a machine to where it can read the bucket. flex makes a machine look like the thing it is imitating — and that word "imitating" only applies to one of them. stage runs inside an actual Crostini container, on actual ChromeOS; it has nothing to fake. flex runs on a desktop that gave up ChromeOS entirely, so everything about the look is manufactured. Two ChromeOS lookalikes, and only one of them is actually pretending.

Why Two, Not One

The reason there's a machine running flex at all is that Crostini's Linux environment cannot load the NVIDIA driver, and that's a kernel problem before it's a permissions one.

Crostini's container doesn't run on the ChromeOS kernel directly. Enabling Linux spins up a lightweight VM (Termina) with its own guest kernel, one that Google builds, signs, and ships as part of the OS image — and your Debian container runs inside that VM, sharing its kernel rather than bringing one of its own. The NVIDIA driver isn't a userspace package; the installer's whole job is to compile nvidia.ko against the exact kernel that's running and load it with insmod. Termina's kernel has no headers shipped for that build, no writable path to drop a module into /lib/modules that survives the next ChromeOS update, and no interest in accepting an unsigned out-of-tree module on a kernel that's otherwise locked down by verified boot.

Even past the module problem, there's no card to attach it to. Graphics inside Crostini go through virtio-gpu — the VM sees a paravirtualized display device good enough for compositing and OpenGL/Vulkan passthrough, not the raw PCI device a discrete GPU actually is. The NVIDIA driver wants to bind directly to that PCI device. Crostini never hands it one.

So the fix was to stop going through ChromeOS for this at all: put Debian directly on the laptop's metal, where apt install nvidia-driver is building against a kernel you actually control, against a GPU that's actually visible on the PCI bus. That solves the driver problem and creates the one flex exists to solve — a stock Debian/GNOME desktop looks and behaves nothing like ChromeOS, and none of what came free inside Crostini comes free here. flex puts the shelf, the Google web apps, the keybindings, and the wallpaper back by hand.

stage and flex are answers to two different questions, not two takes on one. stage is what you run because you're still inside ChromeOS. flex is what you run because the kernel made you leave.

stage: Staging a Fresh Crostini Container

At this point you should have a brand-new Crostini container — the default state after enabling Linux on a Chromebook — and nothing else. No git, no gcloud, nothing installed.

That is a chicken-and-egg problem: reading the private bucket needs gcloud, and a bare container has no gcloud. stage is the one piece that has to be fetchable without credentials, so it is the only thing in this workflow that lives outside the bucket, pasted in or pulled straight from GitHub:

bash <(curl -sSL https://raw.githubusercontent.com/xbill9/chromeos-boot/main/stage)

Process substitution rather than a pipe is deliberate. curl ... | bash hands the script to bash on stdin, which is the same stdin gcloud auth login needs to read answers from; the login then fails or silently eats the rest of the script. bash <(curl ...) passes it as a file descriptor and leaves stdin attached to the terminal.

What it does, in order:

Installs the Google Cloud CLI from the tarball into $HOME — no sudo, no apt, no keyring.
Logs in, opening a browser tab. Crostini hands that URL to the ChromeOS browser, which is already signed in, so this is usually two clicks.
Copies nnn out of the bucket into ~/bin and runs it, which fetches everything else — scripts, dotfiles.
Replaces the tarball with the apt-managed google-cloud-cli in /usr/bin, then deletes ~/google-cloud-sdk.

Step 1's tarball is not the copy meant to stick around — it exists only to read the bucket once, before apt is reachable. Step 4 swaps it for the package apt will keep current, and is non-fatal: without sudo, or with apt unreachable, stage warns and leaves the tarball in place rather than failing the run.

Four commands finish the job, the last two coming from the .bashrc that step 3 just fetched:

bash <(curl -sSL https://raw.githubusercontent.com/xbill9/chromeos-boot/main/stage)
exec bash -l
bootstrap
bootstrap code

stage never asks what kind of hardware it is running on, because it doesn't need to know. Everything it touches — the bucket, gcloud, the dotfiles — is the same regardless of the laptop underneath. That is the shape of a script solving a credentials problem, not a hardware one.

flex: Skinning a Stock Debian Desktop

At this point you should have Debian 13 (trixie) with GNOME 48, installed on real hardware — not a container. flex is written from, and matches, the customization actually run on a Yoga 9 with an NVIDIA GPU.

$ bash flex -l
pkgs
theme
icons
shelf
webapps
appgrid
look
keys
helpers
wallpaper

Ten stages, run in that order by default, or individually to repair one without redoing the rest:

$ bash flex -h

Turn a stock Debian 13 (trixie) / GNOME 48 desktop into a ChromeOS Flex
lookalike.

Companion to `stage`.  `stage` gets a machine to where it can read the
private bucket; `flex` makes the machine look like the thing it is imitating:
a bottom shelf, the Google web apps as first-class icons, ChromeOS
keybindings, a pruned app grid, and a matching pair of wallpapers.

usage: bash flex [--light|--dark]        everything (dark unless told otherwise)
       bash flex -l                      list the stages
       bash flex <stage> [<stage>...]    run only those
       bash flex revert                  undo it

Nothing here needs root except pkgs (apt, non-fatal without sudo), and everything it creates lives under $HOME.

The Ten Stages, Briefly

pkgs — Roboto, gnome-tweaks, and Chrome if no Chromium-family browser is already there. The only stage that touches apt.
theme — adw-gtk3, light and dark, from the upstream tarball (trixie ships no package for it), so GTK3 apps match the libadwaita GTK4 ones.
icons — Papirus and Papirus-Dark, user-level. ~60MB, the slower of the two heavyweight stages.
shelf — installs dash-to-panel in place of the packaged dash-to-dock, because only dash-to-panel merges the taskbar and system tray into one bar — the ChromeOS shelf is one bar, not two. Bottom, 56px, 75% opacity, Google-Blue running-app dots, Alt+1–9 for the first nine pinned apps. The per-monitor layout is resolved through Mutter's DisplayConfig at run time rather than a hardcoded panel ID, so the same script works on a laptop panel and an external monitor without editing.
webapps — the nine Google apps (Gmail, Calendar, Drive, Docs, Sheets, Keep, Photos, Maps, YouTube) as windowless --app= launchers, each with its own shelf icon fetched from gstatic with a favicon-service fallback.
appgrid — hides the apps ChromeOS doesn't have (LibreOffice, xterm, Disk Utility, and 30-odd others) by shadowing each system .desktop file with a copy carrying NoDisplay=true. A copy, not a stub, so MIME associations and "Open with" keep working. Nothing is uninstalled.
look — Roboto as the UI/document/titlebar font, blue accent, no hot corners, time-only clock, one workspace, shelf favorites pinned.
keys — Caps Lock becomes Super, the ChromeOS Launcher key. Alt handles window minimize/maximize/tile; Alt+Tab cycles windows rather than app groups (which move to Super+Tab); Ctrl+F5 is overview; Ctrl+Shift+F5 is screenshot; Ctrl+Shift+Q signs out; Ctrl+Alt+T opens a terminal.
helpers — installs set-mode.sh, gen_wallpaper.py, and boot-splash.sh under ~/.local/share/chromeos-flex/.
wallpaper — renders both wallpaper variants with a hand-rolled PNG encoder (no PIL, no numpy on a stock desktop) — a diagonal gradient, four Google-palette blobs with a quadratic falloff, and an ordered dither to stop a 2560px gradient banding.

Wayland can't restart gnome-shell in place, so the shelf, the GTK theme, and the app grid only take effect after a full log out and back in. The wallpaper and the keybindings are live immediately.

Flipping Light and Dark Without Rerunning Everything

The look is roughly fifteen gsettings calls that only make sense moving together — color scheme, GTK theme, icon theme, both wallpaper keys, the lock screen, and eight dash-to-panel colors. helpers installs set-mode.sh once so that switching modes afterward doesn't mean rerunning the whole script:

$ ~/.local/share/chromeos-flex/set-mode.sh light
switched to light
$ ~/.local/share/chromeos-flex/set-mode.sh
current: 'prefer-light'
  gtk-theme  'adw-gtk3'
  icon-theme 'Papirus'
  wallpaper  'file:///home/xbill/.local/share/backgrounds/chromeos-element-light.png'

boot-splash.sh: Installed, Not Run

helpers writes one more file: boot-splash.sh, which edits /etc/default/grub to kill the boot menu and hand off to the Plymouth splash trixie already ships. flex drops it on disk and stops — it's the only piece of either script that touches the boot chain, it needs root, and GRUB is not something to hand to a stage list without reading it first.

It's not really a speed fix. On the machine it was written for, boot ran about 44 seconds, and roughly 30 of that was firmware and the boot loader before GRUB even started — killing the GRUB timeout only claws back the last 5 seconds or so. What it actually buys is continuity: no text menu flashing past on the way to the desktop, same as an actual Chromebook.

Where They're Alike

Different problems, same habits. Both scripts:


Stages	Broken into named, independently runnable steps (`bootstrap <stage>` / `bash flex <stage>`)
Idempotent	Re-running is how a half-finished run gets repaired, not something to avoid
Listable	`bootstrap -l` / `bash flex -l` print the stage names before you commit to any of them
Self-documenting	Usage lives in a comment block at the top of the file, read back out with `-h`/`--help`
Fail soft	A missing prerequisite warns and skips rather than aborting the rest of the run
No secrets on the command line	`stage`'s bucket name isn't sensitive — IAM gates the objects, not the name — and `flex` needs no credentials at all

Where They Differ

	`stage`	`flex`
Target	A Crostini container	A bare-metal Debian desktop
Needs	`gcloud`, an authorized Google account, a private bucket	Nothing — no bucket, no login, no network dependency beyond the packages it fetches
Privilege	Runs entirely unprivileged; the apt-managed `gcloud` swap is the one non-fatal exception	Only `pkgs` touches `sudo`/apt; the boot splash needs root but is never auto-run
Scope	Bootstraps a dev environment: language runtimes, Docker, cloud and agent CLIs	Cosmetic and GNOME-settings only: theme, shelf, keybindings, wallpaper
State model	Additive — there is no `stage revert`	Explicit `bash flex revert`, which resets touched settings to GNOME defaults rather than restoring whatever was there before
Hardware awareness	None — the same script regardless of what's underneath	Written for, and against, one specific machine's GPU and monitor, generalized via Mutter's `DisplayConfig` rather than hardcoded
Why it exists	Reading credentials before credentials exist	Recreating what real ChromeOS gave up when the GPU drove the OS choice

Summary

The goal here was to get a real NVIDIA driver stack under a ChromeOS-shaped workflow, without giving up the parts of ChromeOS that make a laptop pleasant to use day to day. The key to the solution was accepting that this is two separate problems, not one script with a hardware flag: stage solves getting credentials onto a machine that has none, and flex solves making a desktop look like ChromeOS after leaving ChromeOS entirely — for the same underlying reason, on two different machines, sharing nothing but a repo and a theme.

The strategy for making Linux look like ChromeOS — once inside a sandboxed container, once on the bare metal underneath it — was validated with an incremental, stage-by-stage approach on both sides.

Three Gemma 4 Deployments on One T4G for Under $3: What the Runtime Changes, and What It Doesn't

xbill — Tue, 01 Sep 2026 18:25:50 +0000

This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable.

https://github.com/xbill9/gemma4-dev

The whole exercise cost under three dollars, and that is the part worth keeping. Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts. It also bought five wrong claims, each caught by measuring instead of reasoning. On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.

What is this project trying to Do?

Three rigs in this monorepo serve google/gemma-4-E2B-it on an AWS G5g instance. One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers. The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.

For months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison.

Prerequisites

An AWS account with G-family quota in us-east-1. Each g5g.2xlarge is 8 vCPU, so 16 vCPU of spot quota runs two at once.
A subnet, a security group opening TCP 8000, and an instance profile carrying AmazonSSMManagedInstanceCore plus read on the Hugging Face token secret.
A Hugging Face token in Secrets Manager. It is fetched at boot into a root-only EnvironmentFile and never appears in user data.
boto3 and the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.

AWS EC2 G5g


Instance	`g5g.2xlarge` — 8 vCPU, 16 GiB host
Host CPU	AWS Graviton2, aarch64
GPU	1x NVIDIA T4G, Turing, SM 7.5
GPU memory	15,360 MiB per `nvidia-smi`; AWS lists 16,384 nominal

G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.

Gemma 4 E2B

google/gemma-4-E2B-it is the reference instruction-tuned release. It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.

The dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.

The Three Runtimes

runtime	engine	how it serves
vLLM	v0.27.2rc0, built from source for `sm_75`	continuous batching, paged KV, prefix caching
JAX	this project's own port	hand-written KV ring with a bucket ladder
PyTorch	`AutoModelForCausalLM` + transformers	`past_key_values`, one request at a time

Turing has no bfloat16 datapath, so all three run float16. It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.

Check the Quotas

check_g5g_quotas

| Quota | vCPUs |
| Running On-Demand G and VT instances (vCPU) | 16 |
| All G and VT Spot Instance Requests (vCPU) | 16 |

`g5g.2xlarge` needs 8 vCPUs.

That is the constraint behind every launch below: two rigs in parallel, and no more.

The Sweep Could Not See vLLM

The sweep script read its throughput figure straight out of the response body:

"decode_tps": usage.get("decode_tokens_per_second", 0.0),

usage.decode_tokens_per_second is a field our own servers invent. vLLM does not emit it, and neither does anything else, so the harness could not be pointed at the vLLM rig at all. The three-way comparison had never actually been run.

Re-running a rig does not fix that. Only a common statistic does.

One Statistic, Three Servers

Every OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.

python3 sweep.py --help | grep -A2 decode-source

  --decode-source {auto,usage,stream,both}
                        where the decode figure comes from; see the module
                        docstring

The stream path uses vllm bench serve's exact TPOT definition, (latency - ttft) / (output_len - 1), so a number from this harness is directly comparable to that tool's published figures. auto probes the endpoint once and picks both where the server emits its own gauge, stream where it does not.

Is the Calibration Transferable?

No, and that is worth a measurement rather than an assumption. Running both measures each rig's offset between the two statistics.

rig	server gauge	client stream	stream/gauge
JAX	12.962	12.687	0.9799
PyTorch	10.814	10.243	0.9543

Two percent against 4.6 percent, on the same day and the same instance shape. Borrowing one rig's ratio to convert the other's number would inject a 2.6 percent error into a comparison whose smallest interesting gap is 24 percent. The cross-rig table below is therefore built from stream throughout.

A Gauge Rounded to One Decimal

The JAX server emitted its decode gauge with one decimal place:

f'tpu_jax_decode_tokens_per_second{{model="{MODEL_ID}"}} {METRICS["last_tokens_per_second"]:.1f}',

At about 13 tok/s, one decimal is 0.78 percent resolution. Every sweep that rig had produced showed all three repeats of a cell as byte-identical: 12.8, 12.8, 12.8. That is not reproducibility, it is the measurement floor. The rig had been used to argue about two percent effects it could not resolve.

Two characters fixed it. The first run afterwards reads 12.962, where before it would have said 13.0.

Launch the Instance

Capacity for the whole G5g family was exhausted across all four availability zones several times, so the launcher cycles them with a sixty second backoff.

[12:51:45] round 5 us-east-1c: ❌ AWS InsufficientInstanceCapacity
[12:52:47] us-east-1a: ✅ Launching `i-02e79988a6cbeecbf` (g5g.2xlarge, spot, 1x T4G) in `us-east-1`.

The walkthrough from here follows the PyTorch rig on i-02e79988a6cbeecbf; the other two run the same steps against their own instances. All three landed in us-east-1a within hours of each other. Note that AWS names the other zones as available in every one of those errors — that text describes on-demand capacity and says nothing about spot.

Watch the Install

Cloud-init installs the runtime and then backgrounds itself, so the progress tool reports cloud-init's own state as well as the install log. A dead bootstrap and a slow one must not render identically.

get_install_progress i-02e79988a6cbeecbf

INSTALL COMPLETE
--- cloud-init ---
status: done
errors: []

This is a wheel install, not a build. Across the three timed boots the install finishes a median 113.55 s after launch, against the hours the vLLM rig needs for a from-source build.

Verify the GPU

A config flag being accepted proves nothing, so the probe runs a real matmul on the device.

verify_gpu_arch i-02e79988a6cbeecbf

NVIDIA T4G, 7.5, 15360 MiB
torch: 2.12.0+cu132
arch_list: ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
capability: (7, 5)
compute_dtype: float16
fp16 matmul ok: True

✅ torch reached the GPU and a real fp16 matmul executed.

The DLAMI's torch carries sm_75. Upstream PyPI aarch64 wheels do not, so a pip install torch on this box would serve on CPU without saying so.

Deploy the Server

The payload is the rig's own source, shipped over SSM as a gzipped tarball because user data caps at 16 KiB.

deploy_torch_server i-02e79988a6cbeecbf

✅ Deployed 3 files (16 KiB base64) to `i-02e79988a6cbeecbf`.

Payload root: `/home/xbill/gemma4-dev/gpu-pytorch-g5g-2b`
Build id: `060a572aeb55` — verify_model_health checks the running server reports this.

Verify the Installation

A non-empty reply is not evidence of health. One sibling was once measured answering ': ok: ok: ok…', so the check reads the server's own degenerate-response counter either side of its probe, and compares the served build id against the local payload.

verify_model_health i-02e79988a6cbeecbf

✅ health=200 tokens=5 reply='ok'

- Degenerate (server's own verdict on the full text): **no**
- Build id served: `060a572aeb55`
- Build id matches the local payload (`060a572aeb55`).

Run the Sweep

The same command runs against all three rigs. Only the endpoint changes.

python3 sweep.py --base http://<ip>:8000/v1 --out benchmarks/runs/<run> \
  --contexts 64,512,1024,2048,3072,3800 --outputs 32,128 --repeats 3 \
  --decode-source both

decode-source: both -> both
ctx~512 out=32: in=633 out=32 decode=10.96 tok/s  e2e=9.83 tok/s (warmup 11.08)  stream/usage=0.9616
ctx~2048 out=128: in=2501 out=89 decode=10.66 tok/s  e2e=9.16 tok/s (warmup 10.61)  stream/usage=0.9540
ctx~3800 out=32: FAILED HTTP Error 400 {"detail":"prompt is 4630 tokens and the context
  bound is 4096, leaving no room to decode. Start the server with a larger --seq."}

Cells that cannot exist on the hardware are recorded as infeasible rather than dropped. An absent cell is indistinguishable from an untried one, which is how a sweep overstates its own coverage.

Decode at Concurrency One

	runtime	decode tok/s	% of ceiling	vs PyTorch	cells
🥇	vLLM v0.27.2rc0	32.53	53.0%	3.18x	12/12
🥈	JAX	12.69	20.7%	1.24x	10/12
🥉	PyTorch + transformers	10.24	16.7%	1.00x	10/12

How Close Is That to the Hardware?

The ceiling is arithmetic, not a measurement. E2B streams 4.514 GB of weights per decode step against a measured 277 GB/s, giving 16.30 ms per step and 61.4 tok/s.

The PLE table is excluded from that figure because it is a gather and never a matmul. Quantising it from 9.257 GB to 5.752 GB moved decode by 0.00 tok/s across three cells, which is what confirms it never streams.

All three runtimes sit far below the ceiling, so none of them is bandwidth-bound at batch one. The PyTorch profile shows why: about 5,650 kernel launches per step at one to three microseconds each, on a chip whose launch overhead is five to ten.

The Number I Did Not Expect

Time to first token was the result of the run, right up until it was checked.

input tok	vLLM	JAX	PyTorch
92	103 ms	225 ms	164 ms
1,259	118 ms	1,615 ms	657 ms
3,746	178 ms	5,352 ms	2,339 ms

A 30x advantage, far larger than the 3.2x on decode. It is also impossible. Prefill at 3,746 tokens is roughly 14 TFLOP against a T4G's realistic 20 to 30 TFLOP/s, which is 460 ms at best. vLLM's row says 178 ms.

It did not. vLLM ships enable_prefix_caching=True, and its own metrics say so:

grep -E "^vllm:prefix_cache_(queries|hits)_total" metrics.prom

vllm:prefix_cache_queries_total{engine="0"} 102898.0
vllm:prefix_cache_hits_total{engine="0"}     97440.0

A 94.7 percent hit rate. vLLM genuinely prefilled 5.3 percent of the tokens it was sent, because the harness reused one prompt for a cell's warm-up and all three repeats. Neither sibling has a prefix cache, so both paid full prefill every time.

The fix places a nonce first in the prompt, since a shared prefix is exactly what the cache keys on and a trailing nonce would not have defeated it. That property is now a unit test.

What the Prefill Data Does Support

Strip the contaminated column and a real result remains. Neither of the other two runtimes caches prefixes, and both saw identical prompts.

	TTFT slope	at 3,746 tokens
JAX	1.403 ms/token	5,352 ms
PyTorch	0.595 ms/token	2,339 ms

JAX prefills 2.4x slower than PyTorch, consistently across all five shared context lengths. On an interactive workload with real context that dominates user-visible latency, and it runs in the opposite direction to the 1.24x decode advantage the same rig enjoys.

Boot Time Reverses the Ranking

Nine cold boots, three per runtime, plus nine warm reloads. The start line is the moment run_instances returns an id, because capacity wait measures AWS rather than the rig.

	runtime	cold boot	spread	warm reload	cold/warm
🥇	PyTorch	195.2 s	11.8%	24.5 s	8.0x
🥈	JAX	242.2 s	11.5%	74.1 s	3.3x
🥉	vLLM	1417.8 s	12.6%	264.3 s	5.4x

vLLM takes 23m 38s to serve, from a prebuilt AMI that downloads nothing. PyTorch installs its runtime from wheels and pulls the 9.54 GiB checkpoint over the network, and is still 7.3x faster.

Boot variance is 11.5, 11.8 and 12.6 percent — too consistent across three different runtimes to be a property of any of them. This family's decode noise floor is 1.7 percent, measured by running an identical build on two hosts, so boot is about seven times noisier and a single boot measurement is close to worthless.

Is Health 200 the Same as Ready?

Not on every runtime, which is why the harness records two stop lines.

runtime	first completion, cold	warm
vLLM	0.5 s	0.2 s
PyTorch	1.0 s	0.7 s
JAX	22.9 s	9.2 s

JAX returns health 200 and then compiles XLA per shape bucket on the first real request. Quoting health alone understates its time to serving by 22 seconds, and the cost does not vanish when warm. vLLM is the mirror image: slowest to boot, fastest first token, because graph capture is paid before the port binds.

What Does a Code Change Cost?

runtime	to change serving code
PyTorch	ship 3 files over SSM, restart — 25 s
JAX	same mechanism, plus 9.2 s of compile — 83 s
vLLM	no deploy path exists: rebuild from source, ~67 min, and reapply an out-of-tree Turing patch

vLLM's 264 s warm figure is a systemctl restart, not a code change, so it flatters the comparison. vLLM wins decode 3.2x and loses the iteration loop by 3 to 100x.

Five Theories About 546 Seconds

vLLM's cold boot is dominated by weight loading: 468 to 561 seconds across four measurements. Explaining it took five attempts, four of which were wrong.

g5g.2xlarge needs no swapfile and buys that time back. The rig's own documentation. Falsified by the three-boot campaign at 23m 38s.
A bigger host will not fix it. Retracted the same day — no large host had been measured.
A larger host would very plausibly fix it. Falsified by one g5g.4xlarge boot: available RAM 11.19 to 26.49 GiB, weight loading 546 to 468 s, and total boot 4.7 percent lower, inside the noise band.
It is the loader; vLLM's log says auto-prefetch is disabled on EXT4. Falsified by a within-box A/B — 76.13 s as shipped against 75.12 s with --safetensors-load-strategy=prefetch, which is 1.3 percent and therefore nothing.

What that last run did find is the useful part.

	weight load	n
cold boot, fresh instance	468-561 s	4
warm restart, same box	32-76 s	3

Same volume, same filesystem, same engine, differing only in whether the blocks had been read once. 9.54 GiB in 468 s is about 21 MiB/s, which is absurd for gp3 steady state and ordinary for first-touch reads against a snapshot-backed volume.

Theory five is EBS lazily hydrating the volume from the AMI snapshot, and it is written down as untested. Given the strike rate it does not get promoted by reasoning.

Why the First Campaign Was Thrown Away

The first boot campaign was discarded and re-run. Two independent instances had reported 214.4 s and 125.1 s, identical to the tenth. That is not consistency. It is a five second poll quantising two similar boots onto the same tick.

The data was not wrong; the campaign log shows 215 s and 216 s of wall clock. It was unusably coarse. Health polling went to half a second, and the next pair of boots came in at 216.90 s and 193.86 s — an 11.9 percent spread the old harness could not see.

And Price/Performance?

	$/hr
`g5g.2xlarge` on-demand	$0.556
spot, measured across four AZs	$0.3813 - $0.4416

The whole exercise — 19 instances, 4.06 hours of g5g.2xlarge plus 0.38 of g5g.4xlarge, a serving sweep, three cross-rig runs, nine boots and three A/B restarts — came to under $3. That is arithmetic rather than a bill: AWS drops terminated instances after an hour, so the derivation in cost_derivation.md bounds it at $1.84 all-spot and $2.68 all-on-demand.

That 26 to 46 percent premium is the entire spot versus on-demand decision on this hardware, which is to say there is not one. Try spot, fall back, keep working; the automatic fallback cost about $0.24 across a nine-boot campaign.

Cheap hardware is what made the method possible, not merely affordable. Discarding a completed campaign over a poll-interval bug cost twenty minutes and pennies. Where a run is expensive, the same discovery argues for shipping the numbers with a caveat instead.

AWS Services Used

service	what it does here
EC2	the `g5g.2xlarge` instances, spot and on-demand
Systems Manager	every remote command; there is no inbound SSH rule and no private key
Secrets Manager	the Hugging Face token, fetched at boot into a root-only `EnvironmentFile`
IAM	one instance profile, `AmazonSSMManagedInstanceCore` plus read on that one secret
EBS	gp3 root volumes, and the AMI snapshot behind the vLLM boot mystery

Clean Up

Every instance is terminated as soon as its artifacts are captured. There is no built image to lose, only a pip install and a model cache.

terminate_g5g_instance i-02e79988a6cbeecbf

🗑️ Terminating `i-02e79988a6cbeecbf`. Relaunch costs a pip install, not a build.

aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[?starts_with(InstanceType,`g5g`)].InstanceId' \
  --output text | grep . || echo "🟢 none running"

🟢 none running

What This Does Not Cover

Everything above is concurrency one, which makes it a latency comparison rather than a serving one. Continuous batching is vLLM's whole value proposition and it is untested here.

Only vLLM can serve concurrently at all today. The PyTorch server holds an asyncio.Lock with the comment "one GPU, one process -> serialize requests", and the JAX rig has no batching machinery whatever. The engine-level batch sweep suggests what is on the table — batch eight reaches 84.16 tok/s for an extra 0.258 GB, with per-step time growing two percent across an eight-fold batch — but that number never leaves the engine.

Three further gaps: the JAX leg ran its shipped quantised configuration against two dense runtimes; no output-quality axis was measured at all, on a comparison where one runtime uses a deliberately lossy LM head; and every TTFT figure predates the prompt-uniqueness fix, so only the JAX versus PyTorch half of that table is sound.

Summary

The goal of this article was to compare three inference runtimes on identical silicon without the harness being a variable. The key to the solution was a single client-side statistic that every OpenAI-compatible server can produce. The measured results were:

Decode at concurrency one: vLLM 32.53, JAX 12.69, PyTorch 10.24 tok/s — 53, 21 and 17 percent of a 61.4 tok/s bandwidth ceiling, so none is bandwidth-bound.
The ranking reverses on lifecycle. Cold boot 195.2 s for PyTorch against 1417.8 s for vLLM; a code change costs 25 s against a from-source rebuild.
JAX prefills 2.4x slower than PyTorch, 1.403 against 0.595 ms/token.
A 94.7 percent prefix-cache hit rate turned a 30x TTFT result into a harness artifact.
Calibration offsets are per-rig, 0.9799 and 0.9543, and are not transferable.
Boot variance is about 12 percent on this platform regardless of runtime.
The whole exercise cost under $3 in total. That is what made discarding a finished campaign over a poll-interval bug a twenty minute decision rather than an argument.

Scope: the decode and boot numbers are one g5g.2xlarge per runtime in us-east-1a on 2026-08-31, three repeats per cell and three repeats per boot, mixed spot and on-demand with the market recorded per run. Three things differed between the legs and are named where they matter: vLLM ran max_model_len 16384 against 4096 for the other two; the JAX leg ran its shipped quantised configuration against two dense runtimes; and vLLM booted from a prebuilt AMI carrying its model cache while the other two installed from wheels and downloaded the checkpoint, which is the point of the boot comparison rather than a flaw in it. Two runs sit outside that envelope and say so in the text: the RAM test was a single g5g.4xlarge, and the prefetch A/B ran 2026-09-01 on on-demand after a spot reclamation killed the first attempt. Decode is unaffected by the prefix-cache issue, which changes prefill only.

The strategy for using MCP for multi-runtime comparison was validated with a incremental step by step approach.

Three Gemma 4 Deployments on One T4G for Under $3: What the Runtime Changes, and What It Doesn't

xbill — Tue, 01 Sep 2026 18:25:48 +0000

https://github.com/xbill9/gemma4-dev

What is this project trying to Do?

For months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison.

Prerequisites

An AWS account with G-family quota in us-east-1. Each g5g.2xlarge is 8 vCPU, so 16 vCPU of spot quota runs two at once.
A subnet, a security group opening TCP 8000, and an instance profile carrying AmazonSSMManagedInstanceCore plus read on the Hugging Face token secret.
A Hugging Face token in Secrets Manager. It is fetched at boot into a root-only EnvironmentFile and never appears in user data.
boto3 and the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.

AWS EC2 G5g


Instance	`g5g.2xlarge` — 8 vCPU, 16 GiB host
Host CPU	AWS Graviton2, aarch64
GPU	1x NVIDIA T4G, Turing, SM 7.5
GPU memory	15,360 MiB per `nvidia-smi`; AWS lists 16,384 nominal

G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.

Gemma 4 E2B

The Three Runtimes

runtime	engine	how it serves
vLLM	v0.27.2rc0, built from source for `sm_75`	continuous batching, paged KV, prefix caching
JAX	this project's own port	hand-written KV ring with a bucket ladder
PyTorch	`AutoModelForCausalLM` + transformers	`past_key_values`, one request at a time

Turing has no bfloat16 datapath, so all three run float16. It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.

Check the Quotas

check_g5g_quotas

| Quota | vCPUs |
| Running On-Demand G and VT instances (vCPU) | 16 |
| All G and VT Spot Instance Requests (vCPU) | 16 |

`g5g.2xlarge` needs 8 vCPUs.

That is the constraint behind every launch below: two rigs in parallel, and no more.

The Sweep Could Not See vLLM

The sweep script read its throughput figure straight out of the response body:

"decode_tps": usage.get("decode_tokens_per_second", 0.0),

Re-running a rig does not fix that. Only a common statistic does.

One Statistic, Three Servers

Every OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.

python3 sweep.py --help | grep -A2 decode-source

  --decode-source {auto,usage,stream,both}
                        where the decode figure comes from; see the module
                        docstring

Is the Calibration Transferable?

No, and that is worth a measurement rather than an assumption. Running both measures each rig's offset between the two statistics.

rig	server gauge	client stream	stream/gauge
JAX	12.962	12.687	0.9799
PyTorch	10.814	10.243	0.9543

A Gauge Rounded to One Decimal

The JAX server emitted its decode gauge with one decimal place:

f'tpu_jax_decode_tokens_per_second{{model="{MODEL_ID}"}} {METRICS["last_tokens_per_second"]:.1f}',

Two characters fixed it. The first run afterwards reads 12.962, where before it would have said 13.0.

Launch the Instance

Capacity for the whole G5g family was exhausted across all four availability zones several times, so the launcher cycles them with a sixty second backoff.

[12:51:45] round 5 us-east-1c: ❌ AWS InsufficientInstanceCapacity
[12:52:47] us-east-1a: ✅ Launching `i-02e79988a6cbeecbf` (g5g.2xlarge, spot, 1x T4G) in `us-east-1`.

Watch the Install

get_install_progress i-02e79988a6cbeecbf

INSTALL COMPLETE
--- cloud-init ---
status: done
errors: []

This is a wheel install, not a build. Across the three timed boots the install finishes a median 113.55 s after launch, against the hours the vLLM rig needs for a from-source build.

Verify the GPU

A config flag being accepted proves nothing, so the probe runs a real matmul on the device.

verify_gpu_arch i-02e79988a6cbeecbf

NVIDIA T4G, 7.5, 15360 MiB
torch: 2.12.0+cu132
arch_list: ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
capability: (7, 5)
compute_dtype: float16
fp16 matmul ok: True

✅ torch reached the GPU and a real fp16 matmul executed.

The DLAMI's torch carries sm_75. Upstream PyPI aarch64 wheels do not, so a pip install torch on this box would serve on CPU without saying so.

Deploy the Server

The payload is the rig's own source, shipped over SSM as a gzipped tarball because user data caps at 16 KiB.

deploy_torch_server i-02e79988a6cbeecbf

✅ Deployed 3 files (16 KiB base64) to `i-02e79988a6cbeecbf`.

Payload root: `/home/xbill/gemma4-dev/gpu-pytorch-g5g-2b`
Build id: `060a572aeb55` — verify_model_health checks the running server reports this.

Verify the Installation

verify_model_health i-02e79988a6cbeecbf

✅ health=200 tokens=5 reply='ok'

- Degenerate (server's own verdict on the full text): **no**
- Build id served: `060a572aeb55`
- Build id matches the local payload (`060a572aeb55`).

Run the Sweep

The same command runs against all three rigs. Only the endpoint changes.

python3 sweep.py --base http://<ip>:8000/v1 --out benchmarks/runs/<run> \
  --contexts 64,512,1024,2048,3072,3800 --outputs 32,128 --repeats 3 \
  --decode-source both

decode-source: both -> both
ctx~512 out=32: in=633 out=32 decode=10.96 tok/s  e2e=9.83 tok/s (warmup 11.08)  stream/usage=0.9616
ctx~2048 out=128: in=2501 out=89 decode=10.66 tok/s  e2e=9.16 tok/s (warmup 10.61)  stream/usage=0.9540
ctx~3800 out=32: FAILED HTTP Error 400 {"detail":"prompt is 4630 tokens and the context
  bound is 4096, leaving no room to decode. Start the server with a larger --seq."}

Cells that cannot exist on the hardware are recorded as infeasible rather than dropped. An absent cell is indistinguishable from an untried one, which is how a sweep overstates its own coverage.

Decode at Concurrency One

	runtime	decode tok/s	% of ceiling	vs PyTorch	cells
🥇	vLLM v0.27.2rc0	32.53	53.0%	3.18x	12/12
🥈	JAX	12.69	20.7%	1.24x	10/12
🥉	PyTorch + transformers	10.24	16.7%	1.00x	10/12

How Close Is That to the Hardware?

The ceiling is arithmetic, not a measurement. E2B streams 4.514 GB of weights per decode step against a measured 277 GB/s, giving 16.30 ms per step and 61.4 tok/s.

The Number I Did Not Expect

Time to first token was the result of the run, right up until it was checked.

input tok	vLLM	JAX	PyTorch
92	103 ms	225 ms	164 ms
1,259	118 ms	1,615 ms	657 ms
3,746	178 ms	5,352 ms	2,339 ms

It did not. vLLM ships enable_prefix_caching=True, and its own metrics say so:

grep -E "^vllm:prefix_cache_(queries|hits)_total" metrics.prom

vllm:prefix_cache_queries_total{engine="0"} 102898.0
vllm:prefix_cache_hits_total{engine="0"}     97440.0

The fix places a nonce first in the prompt, since a shared prefix is exactly what the cache keys on and a trailing nonce would not have defeated it. That property is now a unit test.

What the Prefill Data Does Support

Strip the contaminated column and a real result remains. Neither of the other two runtimes caches prefixes, and both saw identical prompts.

	TTFT slope	at 3,746 tokens
JAX	1.403 ms/token	5,352 ms
PyTorch	0.595 ms/token	2,339 ms

Boot Time Reverses the Ranking

Nine cold boots, three per runtime, plus nine warm reloads. The start line is the moment run_instances returns an id, because capacity wait measures AWS rather than the rig.

	runtime	cold boot	spread	warm reload	cold/warm
🥇	PyTorch	195.2 s	11.8%	24.5 s	8.0x
🥈	JAX	242.2 s	11.5%	74.1 s	3.3x
🥉	vLLM	1417.8 s	12.6%	264.3 s	5.4x

vLLM takes 23m 38s to serve, from a prebuilt AMI that downloads nothing. PyTorch installs its runtime from wheels and pulls the 9.54 GiB checkpoint over the network, and is still 7.3x faster.

Is Health 200 the Same as Ready?

Not on every runtime, which is why the harness records two stop lines.

runtime	first completion, cold	warm
vLLM	0.5 s	0.2 s
PyTorch	1.0 s	0.7 s
JAX	22.9 s	9.2 s

What Does a Code Change Cost?

runtime	to change serving code
PyTorch	ship 3 files over SSM, restart — 25 s
JAX	same mechanism, plus 9.2 s of compile — 83 s
vLLM	no deploy path exists: rebuild from source, ~67 min, and reapply an out-of-tree Turing patch

vLLM's 264 s warm figure is a systemctl restart, not a code change, so it flatters the comparison. vLLM wins decode 3.2x and loses the iteration loop by 3 to 100x.

Five Theories About 546 Seconds

vLLM's cold boot is dominated by weight loading: 468 to 561 seconds across four measurements. Explaining it took five attempts, four of which were wrong.

g5g.2xlarge needs no swapfile and buys that time back. The rig's own documentation. Falsified by the three-boot campaign at 23m 38s.
A bigger host will not fix it. Retracted the same day — no large host had been measured.
A larger host would very plausibly fix it. Falsified by one g5g.4xlarge boot: available RAM 11.19 to 26.49 GiB, weight loading 546 to 468 s, and total boot 4.7 percent lower, inside the noise band.
It is the loader; vLLM's log says auto-prefetch is disabled on EXT4. Falsified by a within-box A/B — 76.13 s as shipped against 75.12 s with --safetensors-load-strategy=prefetch, which is 1.3 percent and therefore nothing.

What that last run did find is the useful part.

	weight load	n
cold boot, fresh instance	468-561 s	4
warm restart, same box	32-76 s	3

Theory five is EBS lazily hydrating the volume from the AMI snapshot, and it is written down as untested. Given the strike rate it does not get promoted by reasoning.

Why the First Campaign Was Thrown Away

And Price/Performance?

	$/hr
`g5g.2xlarge` on-demand	$0.556
spot, measured across four AZs	$0.3813 - $0.4416

AWS Services Used

service	what it does here
EC2	the `g5g.2xlarge` instances, spot and on-demand
Systems Manager	every remote command; there is no inbound SSH rule and no private key
Secrets Manager	the Hugging Face token, fetched at boot into a root-only `EnvironmentFile`
IAM	one instance profile, `AmazonSSMManagedInstanceCore` plus read on that one secret
EBS	gp3 root volumes, and the AMI snapshot behind the vLLM boot mystery

Clean Up

Every instance is terminated as soon as its artifacts are captured. There is no built image to lose, only a pip install and a model cache.

terminate_g5g_instance i-02e79988a6cbeecbf

🗑️ Terminating `i-02e79988a6cbeecbf`. Relaunch costs a pip install, not a build.

aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[?starts_with(InstanceType,`g5g`)].InstanceId' \
  --output text | grep . || echo "🟢 none running"

🟢 none running

What This Does Not Cover

Everything above is concurrency one, which makes it a latency comparison rather than a serving one. Continuous batching is vLLM's whole value proposition and it is untested here.

Summary

Decode at concurrency one: vLLM 32.53, JAX 12.69, PyTorch 10.24 tok/s — 53, 21 and 17 percent of a 61.4 tok/s bandwidth ceiling, so none is bandwidth-bound.
The ranking reverses on lifecycle. Cold boot 195.2 s for PyTorch against 1417.8 s for vLLM; a code change costs 25 s against a from-source rebuild.
JAX prefills 2.4x slower than PyTorch, 1.403 against 0.595 ms/token.
A 94.7 percent prefix-cache hit rate turned a 30x TTFT result into a harness artifact.
Calibration offsets are per-rig, 0.9799 and 0.9543, and are not transferable.
Boot variance is about 12 percent on this platform regardless of runtime.
The whole exercise cost under $3 in total. That is what made discarding a finished campaign over a poll-interval bug a twenty minute decision rather than an argument.

The strategy for using MCP for multi-runtime comparison was validated with a incremental step by step approach.

Streamline Publishing with a Claude Code Skill

xbill — Tue, 01 Sep 2026 14:37:19 +0000

TL;DR: publishing-kit packages the whole publishing lifecycle as a Claude Code skill. Write one markdown file, and it builds the dev.to, AWS Builder Center, Medium and LinkedIn versions, checks them, and posts the ones that have an API. This article, its cover and all four of its artifacts were produced by the thing the article is about — dogfooding all the way down. More on that at the end.

Publishing one technical article to four places involves a surprising amount of ceremony: making a cover at whatever size each destination wants, rendering tables to images because Medium's importer eats them, stripping emoji for AWS, checking that every number in the piece came from a real run, getting a long markdown file into a browser editor that has no API, remembering which organization the article routes to, and writing the announcement post afterwards — by which point you have four slightly different files and no idea which one is current.

I packaged all of that into publishing-kit — a Claude Code skill and a set of small scripts — so you can just ask Claude to publish.

What it does

The skill teaches Claude the publishing lifecycle; the scripts do the parts a language model should not be doing by hand.

Build the artifacts: one source file becomes the dev.to markdown, the Builder Center version with emoji stripped and the AWS disclaimer appended, Medium HTML with every table rendered to a PNG, and a LinkedIn post. make-builder.py, make-medium.py, make-linkedin.py.
Make the cover once: make-cover.py --flow --sizes devto,builder draws the pipeline as an illustration and renders it at every geometry the destinations demand, names the file by a hash of its own bytes, and reports which type sizes survive a 320px feed card — which is the size a cover is actually met at, and where a diagram with small labels turns to mush.
Trace every number: check-facts.py pulls the prices, measurements and versions out of your prose and reports which ones appear in no evidence file. It cannot tell you a figure is true. It tells you which ones you are asserting from memory.
Pre-flight: preflight.py --live runs the lot and exits non-zero — cover committed and matching HEAD, geometry right, front matter complete, no hard-wrapped paragraphs, and every published URL fetched and compared byte for byte against your disk.
Post where there is an API: publish-devto.py --create takes the front matter as the payload, so title, tags and cover ride along with the body, and --org-slug routes it to a community channel. No browser.
Drive the editors that have none: AWS Builder Center and Medium are Chrome work, and the skill knows the route — payload through window.name, a checksum on both sides, an emptiness assertion before the paste, and a landmark count after it.

It also encodes the hard-won details you would otherwise learn the day after publishing: that dev.to renders markdown with hard breaks on, so a source wrapped at 95 columns arrives with a stray break in every paragraph; that dev.to does not host your cover but proxies it at 2.381:1, so a 1376x768 cover loses 95px off the top and bottom; that Medium drops data: URI images on paste, so the self-contained build arrives with no pictures at all; and that LinkedIn's Posts API cannot create a draft, because PUBLISHED is the only state it accepts on creation.

Install

The fastest path is the plugin marketplace:

/plugin marketplace add xbill9/publishing-kit
/plugin install publishing@publishing-kit

Prefer the classic route? Clone it and symlink the skill:

git clone https://github.com/xbill9/publishing-kit
ln -s "$PWD/publishing-kit/skills/publishing" ~/.claude/skills/publishing

You will need Python 3 with Pillow for the cover and table rendering, a dev.to API key in ~/.devto.key, and a public repo to hold the article directory — both the cover and the Medium images are fetched by URL when the page renders, so unpushed means broken.

Mostly you will not run these scripts yourself — Claude does, and SKILL.md tells it where they are. When you do want one by hand, the paths differ by install method, so ask rather than remember: skill-footprint.py --where prints the skill directory. From a clone it is skills/publishing; from a marketplace install it sits under a version-numbered cache path, which means no invocation you write down survives an upgrade.

That version numbering matters if you plan to hack on the skill itself: installing takes a snapshot, and claude plugin update compares version strings, so edits under the same version never reach your session. Symlink while you are iterating.

What it looks like in practice

Once installed, you talk to Claude Code like this:

"Write this benchmark up and publish it to dev.to under aws-builders"

Claude writes the source article, generates the cover at both geometries, traces the numbers against your run artifacts, runs the pre-flight, tells you what failed, and — once it passes — posts it as a draft and hands you the link. Then:

"Now make the Medium and Builder Center versions"

It derives them, renders the tables to images, opens Chrome, and fills the editors. It stops at draft on every destination and hands back links. Publishing is your keystroke, not its.

Debugging a publish that went sideways

This is the half that surprised me most, and where most of the skill's value ended up.

The page does not look like the file. Fetch what the destination actually serves rather than reasoning about what you pushed:

python3 ../../skills/publishing/scripts/check-links.py article.md

devto-publishing-kit.md  (branch URLs)
  ok    cover.77acc7c4.jpg: HTTP 200, bytes match disk
  ok    img/cover.77acc7c4.jpg: HTTP 200, bytes match disk
  ok    img/devto-publishing-kit-table-1.png: HTTP 200, bytes match disk

A FAIL there reads "HTTP 200 but the served bytes differ from disk", which is usually an image regenerated after its commit. Every other check in the kit reasons about local state — is the file there, is it tracked — and each of those can pass while the published URL serves something else.

The paragraphs look ragged. check-article.py reports hard-wrapped paragraphs with the line number of the first one, and publish-devto.py unwraps on the way out so your repo copy stays readable at 95 columns and the published page does not.

An image is missing on Medium. You pasted -embed.html. Paste -hosted.html, which references real URLs Medium re-hosts, and commit the images first.

A number in the article has no artifact behind it. check-facts.py will name it. Every untraced claim is one of three things: measured but never archived, arithmetic that should be labelled as arithmetic, or asserted from memory — and it is always the third one that turns out to be wrong.

Under the hood

One SKILL.md, a reference file for each destination's quirks, and a script per job. The skill decides when each one runs; the scripts are independently runnable and print what they did.

The bit worth stealing for your own skills is references/house-style.md. Voice, section order, the opener and closing formulas live in that one file, and nothing in SKILL.md or the scripts depends on it. Swap it and the kit writes as somebody else.

skill-footprint.py measures the skill's own size and token cost, and emits the cost table and cover footer for articles like this one — because a figure that changes on every commit does not belong in prose you have to remember to update.

Dogfooding: this article, and its cover 🐕🍖

Every artifact here came out of the kit. The cover was rendered by make-cover.py, the Builder Center version was derived by make-builder.py, the Medium build by make-medium.py, and the dev.to draft posted by publish-devto.py.

The two numbers on the cover are real, and both were found by using the kit on itself:

	measured
dev.to, hard-wrapped paragraphs	47 of 62 in a published article carried a stray break
Medium, images pasted as data URIs	0 of 4 survived; from real URLs, 4 of 4

The first one had been happening to my articles for months. The second cost a full re-do the first time it happened. Neither is in the docs of either destination, and both are now checks.

The run also found five faults in the kit itself, which is the point of eating your own cooking: make-medium.py had another project's repo hardcoded as its default image base; check-article.py passed a cover that was tracked but regenerated; make-cover.py lost a --tile value beginning with a hyphen to argparse; check-facts.py read 127.0.0.1 as a version number; and unwrap() skipped block quotes, so this article's own TL;DR posted as five separate lines — while the check that should have caught it skipped block quotes too, and agreed.

Links

Repo: github.com/xbill9/publishing-kit (Apache-2.0)
The skill: skills/publishing/SKILL.md
This article's directory, with every artifact and the evidence behind every number: articles/publishing-kit-skill/
Claude Code: claude.com/claude-code

Issues and PRs welcome. This is a third-party community project, not affiliated with dev.to, Medium, AWS or LinkedIn — and the destination behaviours described here were measured on 2026-08-31, so check them again before you trust them.

Streamline Publishing with a Claude Code Skill

xbill — Tue, 01 Sep 2026 14:37:08 +0000

TL;DR: publishing-kit packages the whole publishing lifecycle as a Claude Code skill. Write one markdown file, and it builds the dev.to, AWS Builder Center, Medium and LinkedIn versions, checks them, and posts the ones that have an API. This article, its cover and all four of its artifacts were produced by the thing the article is about — dogfooding all the way down. More on that at the end.

I packaged all of that into publishing-kit — a Claude Code skill and a set of small scripts — so you can just ask Claude to publish.

What it does

The skill teaches Claude the publishing lifecycle; the scripts do the parts a language model should not be doing by hand.

Build the artifacts: one source file becomes the dev.to markdown, the Builder Center version with emoji stripped and the AWS disclaimer appended, Medium HTML with every table rendered to a PNG, and a LinkedIn post. make-builder.py, make-medium.py, make-linkedin.py.
Make the cover once: make-cover.py --flow --sizes devto,builder draws the pipeline as an illustration and renders it at every geometry the destinations demand, names the file by a hash of its own bytes, and reports which type sizes survive a 320px feed card — which is the size a cover is actually met at, and where a diagram with small labels turns to mush.
Trace every number: check-facts.py pulls the prices, measurements and versions out of your prose and reports which ones appear in no evidence file. It cannot tell you a figure is true. It tells you which ones you are asserting from memory.
Pre-flight: preflight.py --live runs the lot and exits non-zero — cover committed and matching HEAD, geometry right, front matter complete, no hard-wrapped paragraphs, and every published URL fetched and compared byte for byte against your disk.
Post where there is an API: publish-devto.py --create takes the front matter as the payload, so title, tags and cover ride along with the body, and --org-slug routes it to a community channel. No browser.
Drive the editors that have none: AWS Builder Center and Medium are Chrome work, and the skill knows the route — payload through window.name, a checksum on both sides, an emptiness assertion before the paste, and a landmark count after it.

Install

The fastest path is the plugin marketplace:

/plugin marketplace add xbill9/publishing-kit
/plugin install publishing@publishing-kit

Prefer the classic route? Clone it and symlink the skill:

git clone https://github.com/xbill9/publishing-kit
ln -s "$PWD/publishing-kit/skills/publishing" ~/.claude/skills/publishing

What it looks like in practice

Once installed, you talk to Claude Code like this:

"Write this benchmark up and publish it to dev.to under aws-builders"

"Now make the Medium and Builder Center versions"

It derives them, renders the tables to images, opens Chrome, and fills the editors. It stops at draft on every destination and hands back links. Publishing is your keystroke, not its.

Debugging a publish that went sideways

This is the half that surprised me most, and where most of the skill's value ended up.

The page does not look like the file. Fetch what the destination actually serves rather than reasoning about what you pushed:

python3 ../../skills/publishing/scripts/check-links.py article.md

devto-publishing-kit.md  (branch URLs)
  ok    cover.77acc7c4.jpg: HTTP 200, bytes match disk
  ok    img/cover.77acc7c4.jpg: HTTP 200, bytes match disk
  ok    img/devto-publishing-kit-table-1.png: HTTP 200, bytes match disk

An image is missing on Medium. You pasted -embed.html. Paste -hosted.html, which references real URLs Medium re-hosts, and commit the images first.

Under the hood

One SKILL.md, a reference file for each destination's quirks, and a script per job. The skill decides when each one runs; the scripts are independently runnable and print what they did.

Dogfooding: this article, and its cover 🐕🍖

The two numbers on the cover are real, and both were found by using the kit on itself:

	measured
dev.to, hard-wrapped paragraphs	47 of 62 in a published article carried a stray break
Medium, images pasted as data URIs	0 of 4 survived; from real URLs, 4 of 4

The first one had been happening to my articles for months. The second cost a full re-do the first time it happened. Neither is in the docs of either destination, and both are now checks.

g5g vs g6 for LLM Serving: the Same Code, and 3.7x the Throughput

xbill — Mon, 31 Aug 2026 01:40:10 +0000

This article compares two AWS GPU instance families for serving a small language
model, using a payload that is byte-identical on both. The older family loses
87% of decode to dtype conversion, and nothing in any log, metric or health
check says so.

The code is here:

https://github.com/xbill9/gemma4-dev

The Two Instances

g5g.2xlarge pairs a Graviton2 (aarch64) host with an NVIDIA T4G — Turing,
SM 7.5. g6.2xlarge is x86_64 with an NVIDIA L4 — Ada, SM 8.9. Both were run
on spot.

	`g5g.2xlarge`	`g6.2xlarge`
GPU	NVIDIA T4G — Turing, SM 7.5	NVIDIA L4 — Ada, SM 8.9
GPU memory	15,360 MiB	23,034 MiB
Host	Graviton2, aarch64	x86_64, `us-east-1d`
Purchase model	spot	spot
Run cited	`2026-08-28-full-run-cached-g5g`	`2026-08-28-first-serve-g6`

The workload is google/gemma-4-E2B-it, the dense reference checkpoint, served
through a hand-written pure-JAX port — no PyTorch, no vLLM, no torch_xla.

The payload is byte-identical on both instances: build id 51bc52c9e2e9,
config ple4 + int8_lm_head, and tpu_jax_weight_bytes reading 6,155,450,950
on each. Only the chip and its host differ.

At This Point You Should Have

Spot capacity for g5g.2xlarge and g6.2xlarge
A Hugging Face token with access to google/gemma-4-E2B-it
A Deep Learning base AMI matching the host architecture — the aarch64 and x86_64 images are not interchangeable
No CUDA toolkit and no Rust toolchain; jax[cuda13] supplies CUDA as wheels

Install Is a pip Install

There is no build step on either instance. jax[cuda13] ships wheels carrying
CUDA, including aarch64 wheels for the Graviton2 host.

Install: 117 s, with the cache restore included

XLA's persistent compilation cache is pushed to S3 and restored on boot. On the
g5g rig it restores 805 files / 12 MB in 6 seconds onto a fresh instance from
a box that had already been terminated.

Warm Up Before You Measure Anything

max_new_tokens is a static_argnames entry, so (bucket, max_tokens) is the
compiled shape. The first request off a fresh engine pays XLA compilation.

On the g5g that first request took 18.06 s against 4.50 s warm — a 4.0x
whole-request ratio, from 2026-08-21-cuda13-py314-g5g. A harness that skips
warm-up misreports the instance by a factor of four.

Note that the 56x figure quoted from the earlier first-serve baseline is TTFT
specifically, which is a different measurement and not interchangeable with the
whole-request ratio.

The Throughput Sweep

64 output tokens, concurrency 1, 3 repeats per cell, median reported. "Gauge" is
the engine's steady-state decode counter; "end-to-end" is wall time over the whole
request including prefill.

Input tokens	🥈 g5g gauge	g5g end-to-end	🥇 g6 gauge	g6 end-to-end
41	12.9 tok/s	12.43 tok/s	48.5 tok/s	46.23 tok/s
521	13.0 tok/s	11.28 tok/s	48.4 tok/s	42.87 tok/s
2,057	12.9 tok/s	8.22 tok/s	48.3 tok/s	34.57 tok/s
3,593	—	—	48.3 tok/s	27.55 tok/s

3.7x on decode, for the same code and the same weights.

Read the Gauge, Not End-To-End

Decode moves 0.8% across a 50x context range on the g5g and 0.4% on the g6.
End-to-end falls hard on both — 12.43 to 8.22, and 46.23 to 27.55.

That fall is prefill being linear in the padded bucket, not decode degrading. They
are two different claims, and conflating them makes a benchmark a lie.

A cost proportional to the weights rather than the context produces exactly
this shape, which is why the KV cache is not what sets decode speed on either
instance.

Usable context on the g5g is MAX_MODEL_LEN=4096, and that is the honest number:
4,105 prompt tokens serve, 5,120 fails on a prefill transient.

Where the g5g's Decode Actually Went

Profiling with xprof, 20 decode steps with the service stopped:

	🥈 g5g / T4G (SM 7.5)	🥇 g6 / L4 (SM 8.9)
dtype conversion	54.1%	0.0%
fp32 `gemvx`	32.8%	absent
Tensor Core	0.0%	0.0%
Total kernel time	1,466.0 ms	362.8 ms
Decode, gauge	12.9 tok/s	48.4 tok/s
Peak HBM bandwidth	298.083 GiB/s	279.441 GiB/s
Share of bandwidth roofline	26%	~100%

87% of decode on the g5g is not math. It is dtype conversion plus an fp32
gemvx path. The instance runs at 26% of its own memory-bandwidth roofline; the
g6 runs at roughly all of it.

Why This Is Invisible

A wrong compute dtype does not raise. It emulates. bfloat16 on a pre-Ampere
GPU does not fail — XLA routes it through fp32 and decode quietly disappears into
conversion.

Turing has neither bf16 nor fp8. Its only real 16-bit datapath is float16. So the
port reads the live compute capability off the device rather than trusting a config
file:

COMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16

The server states its decision on the first line it emits, so a misconfigured
instance is one grep away rather than a mystery in the throughput:

INFO ports.gemma4.jax_e_model: jax_e_model device policy: platform=gpu
compute_capability=8.9 compute_dtype=bfloat16 pallas_interpret=False

The Checkpoint Was Not the Problem

The obvious hypothesis was bf16 weights on a chip with no bf16 datapath, so the
checkpoint was converted to float16 host-side and re-run. Parameter dtypes read
{'float16': 541, 'uint8': 1, 'int8': 1} — and conversion stayed at 54.0%.

Storage dtype was never the problem. The fp32 gemvx line is the tell: XLA was
round-tripping through fp32 regardless of what the file on disk said. Only a card
whose compute dtype matches its storage dtype removes it, which is what the g6
shows.

The measurement reproduces: the same profile on a different instance, a different
AMI and a restored cache landed at 1466.0 ms against 1467.1 ms.

A Health Check Will Not Catch This

The g5g serves correctly the entire time. It returns HTTP 200, valid completions,
and a healthy /health. It is simply doing four times more work than it needs to.

The related trap in this engine is a padding-eviction bug in the KV ring cache,
whose failure mode is a token loop returning a clean 200 with
status: "success" and output like The The The The. Nothing in the logs or the
metrics is red. Only a degeneracy check on the response body catches it, which the
server now runs on every request.

On this stack, HTTP 200 is not evidence of anything.

Teardown

Both instances are spot and are terminated after collection. The XLA cache is
pushed to S3 first, which is what makes the 6-second restore on the next fresh
instance possible.

Summary

The goal of this article was to compare two AWS GPU instance families for serving a
small language model with a payload held byte-identical across both. The key to the
solution was profiling decode rather than trusting throughput alone. The measured
results were:

3.7x decode throughput on g6.2xlarge over g5g.2xlarge — 48.4 against 12.9 tok/s — for the same code and the same weights.
87% of decode on the g5g is dtype conversion and an fp32 path, and 0.0% of it on the g6.
The g5g runs at 26% of its memory-bandwidth roofline; the g6 at roughly 100%.
Nothing surfaces this. The g5g serves valid completions with a healthy endpoint throughout.
Tensor Core utilization is 0.0% on both instances and is not yet explained.

Scope: two spot instances, one in us-east-1d, each measured once with 3 repeats
per sweep cell and medians reported. The two differ in host architecture (aarch64
against x86_64) and base image as well as in GPU, so this is not a single-variable
experiment; the payload is byte-identical across them — build 51bc52c9e2e9, the
same config and the same 6,155,450,950 bytes of weights — which is the basis for
attributing the difference to the chip. The g5g profile was reproduced on a second
instance at 1466.0 ms against 1467.1 ms; the g6 profile was measured once. Price
and price-per-token were not measured and are not claimed here.

The strategy for using MCP for Gemma 4 serving across AWS GPU instance families was
validated with an incremental step by step approach.

Gemma 4 in Pure JAX: What Changes Between Turing and Ada, and What Doesn't

xbill — Mon, 31 Aug 2026 01:40:09 +0000

This article is a measurement report on running a hand-written Gemma 4 port in
pure JAX across two NVIDIA GPUs a generation apart, and on the two places the
"it's just JAX" abstraction leaks. One of those leaks costs 87% of decode and
nothing in the logs is red.

The code is here:

https://github.com/xbill9/gemma4-dev

What This Article Measures

One port, one build, one checkpoint, two cards. Everything below comes from two
archived runs, named so you can check them.

	G5g	G6
Chip	NVIDIA T4G — Turing, SM 7.5, 15,360 MiB	NVIDIA L4 — Ada, SM 8.9, 23,034 MiB
Host	`g5g.2xlarge` spot — Graviton2, aarch64	`g6.2xlarge` spot — x86_64, `us-east-1d`
Checkpoint	`google/gemma-4-E2B-it`, dense reference	`google/gemma-4-E2B-it`, dense reference
Compute dtype	`float16` (device-chosen)	`bfloat16` (device-chosen)
Stack	jax 0.11.1, CUDA from pip	jax 0.11.1, Python 3.14
Run cited	`2026-08-28-full-run-cached-g5g`	`2026-08-28-first-serve-g6`

Build id 51bc52c9e2e9 on both, config ple4 + int8_lm_head, and
tpu_jax_weight_bytes reads 6,155,450,950 on both cards — the same integer.
Only the chip and its host differ.

At This Point You Should Have

An AWS account with spot capacity for g5g.2xlarge and g6.2xlarge
A Hugging Face token with access to google/gemma-4-E2B-it
Python 3.13 or newer on the instance, system-wide, with no virtualenv
No CUDA toolkit, no Rust toolchain, and no compiler — none of them are needed

Why Pure JAX At All

The port lives in ports/gemma4/ and is driven by a generation loop behind an
OpenAI-compatible server. No PyTorch, no vLLM, no torch_xla.

The premise under test is that the same source runs on both cards with nothing
changed but a config file. It mostly holds. The interesting part is where it does
not.

Gemma 4 E2B Is Not a Stock Transformer

Any port has to carry four irregularities, and none of them are optional.

Two attention geometries. Sliding layers use head_dim=256, global layers use 512. Most inference stacks assume one head dimension per model.
8:1 MQA, so the KV budget is nothing like the parameter count suggests.
A KV-share map that collapses 35 layers onto 15 caches.
A 512-slot sliding ring, plus per-layer embeddings held in a 4.70 GB table that is quantized to 4 bits on load.

The Geometry Is What Breaks Other Stacks

That first irregularity is the expensive one. On the vLLM path the heterogeneous
head dims force the Triton attention backend:

Gemma4 model has heterogeneous head dimensions
(sliding=256, global=512); falling back to the Triton attention backend

On a Turing GPU that backend then asks for shared memory the hardware does not
have:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 147456, Hardware limit: 65536

JAX never enters that conversation. Attention is ordinary XLA rather than a
hand-tiled kernel, so there is no per-block shared-memory ceiling in the attention
path at all. The irregular geometry that is a special case everywhere else is just
array shapes here.

Leak One: The dtype Policy Has To Read the Device

This is the single most expensive lesson in the repository.

A wrong compute dtype does not raise. It emulates. bfloat16 on a pre-Ampere
GPU does not fail — XLA routes it through fp32 and most of decode disappears into
conversion. Nothing in the logs is red.

So the port does not take the dtype from a config file. It reads the live compute
capability off the device:

COMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16

On the SM 8.9 Ada card that resolves to bfloat16. On the SM 7.5 Turing card it
resolves to float16 — Turing's only real 16-bit datapath, since it has neither
bf16 nor fp8.

The Process States What It Decided

The first line the server emits is the policy, so a misconfiguration is one grep
away rather than a mystery in the throughput:

INFO ports.gemma4.jax_e_model: jax_e_model device policy: platform=gpu
compute_capability=8.9 compute_dtype=bfloat16 pallas_interpret=False

pallas_interpret=False matters just as much. It is the difference between
serving and silently running a simulator.

Leak Two: Pallas Is Not Portable as a Memory Model

Here is the part that does not port, and it is not a bug. It is a real hardware
difference wearing a portable API.

The fused W4A16 kernel is written in Pallas, and it was tiled for a device with
16 MB of scratchpad per core. At this model's shapes the tiles want 550 KiB to
1.1 MiB per block.

On a GPU, Pallas lowers through Triton, and those tiles become shared memory.
Turing gives you 64 KiB per block. Ada raises the ceiling, but nowhere near a
megabyte.

So the fast path runs on neither card. The engine computes the requirement at
startup and refuses with the arithmetic attached, rather than dying as a cryptic
OutOfResources at the first token:

check_w4a16_fits_scoped_memory()

The practical consequence is that both GPU rigs serve the dense reference
checkpoint at 16-bit. Pallas is portable as an API and not portable as a
memory model. That boundary is worth knowing before planning a port around a
fused kernel.

The Bug That Returns 200 OK

A padding-eviction bug in the KV ring cache cost a week, and it is the kind only
Gemma 4's geometry produces.

The invariant is that a cache index is an absolute real position, and padding
never occupies an index a real position uses. A port that right-pads into the
512-slot ring violates it, and the failure mode is not a crash and not a NaN. It is
a token loop — a clean HTTP 200, status: "success", and output like
The The The The.

Nothing in the logs is red. Nothing in the metrics is red. The only thing that
catches it is a degeneracy check on the output itself, which the server now runs on
every response.

The scariest bugs in this project all returned success.

Install: No Build Step

jax[cuda13] supplies CUDA as wheels, so the install needs no CUDA toolkit, no
Rust, and no compiler on the box.

Install: 117 s, with the cache restore included

XLA's persistent compilation cache ports as-is. On the T4G rig it restores 805
files / 12 MB in 6 seconds onto a fresh instance, from a box that had already
been terminated.

Warm Up At the Shape You Measure

max_new_tokens is a static_argnames entry, so (bucket, max_tokens) is the
compiled shape on every backend. A harness that does not warm up misreports the
rig badly.

On the T4G the first request off a fresh engine took 18.06 s against 4.50 s
warm — a 4.0x whole-request ratio, measured in
2026-08-21-cuda13-py314-g5g.

That run also notes something worth repeating: the 56x figure from the first-serve
baseline is TTFT specifically, not the same measurement as the whole-request
ratio. They are not interchangeable.

The Sweep

64 output tokens, concurrency 1, 3 repeats per cell, median. "Decode, gauge" is the
engine's steady-state counter. "End-to-end" is wall time over the whole request,
prefill included.

Input tokens	T4G gauge	T4G end-to-end	L4 gauge	L4 end-to-end
41	12.9 tok/s	12.43 tok/s	48.5 tok/s	46.23 tok/s
521	13.0 tok/s	11.28 tok/s	48.4 tok/s	42.87 tok/s
2,057	12.9 tok/s	8.22 tok/s	48.3 tok/s	34.57 tok/s
3,593	—	—	48.3 tok/s	27.55 tok/s

Decode Is Flat; End-To-End Is Not

Decode moves 0.8% across a 50x context range on the T4G and 0.4% on the L4. End-to-end
falls hard on both.

That fall is prefill being linear in the padded bucket, not decode degrading. They
are two different claims, and conflating them makes a benchmark a lie. Quote the
gauge.

A cost proportional to the weights rather than the context produces exactly this
shape, which is why KV is not what sets decode speed on either card — despite
Gemma 4's whole KV story.

On context specifically: MAX_MODEL_LEN=4096 is the honest number on the T4G.
4,105 prompt tokens serve; 5,120 fails on a prefill transient.

The Profile: 87% of Decode Was Not Math

Profiling decode with xprof on the Turing card, 20 decode steps with the service
stopped:

	🥉 T4G (SM 7.5)	🥇 L4 (SM 8.9)
dtype conversion	54.1%	0.0%
fp32 `gemvx`	32.8%	absent
Tensor Core	0.0%	0.0%
Total kernel time	1,466.0 ms	362.8 ms
Decode, gauge	12.9 tok/s	48.4 tok/s
Peak HBM bandwidth	298.083 GiB/s	279.441 GiB/s
Share of bandwidth roofline	26%	~100%

1,466 ms of kernels across 108 distinct kernels on a Tensor Core GPU, without one
Tensor Core firing. More than half of decode went to converting numbers between
formats before any math happened.

The Obvious Explanation Was Wrong

The obvious hypothesis was bf16 weights being converted on a chip with no bf16
datapath. So the checkpoint was converted to float16 host-side and re-run.
Parameter dtypes read {'float16': 541, 'uint8': 1, 'int8': 1} — and conversion
stayed at 54.0%.

The measurement itself is solid. The same profile on a different instance, a
different AMI and a restored cache landed at 1466.0 ms against 1467.1 ms. 1.1 ms
apart on 1467.

It Was the Datapath, Not the Checkpoint

The Ada card resolves it. Converting the stored weights changed nothing because
storage dtype was never the problem: Turing has no native bf16, and the fp32
gemvx line is the tell — XLA was round-tripping through fp32 regardless of what
the file on disk said.

Give it a card where storage and compute dtype actually match, and the 54%
conversion and the 32.8% fp32 path vanish together. An 87% tax gone, for 3.7x
the throughput, and a rig sitting at its bandwidth roofline instead of 26% of it.

The /health endpoint on the L4 reports weights=bfloat16 activations=bfloat16 kv_cache=bfloat16 pre_ampere=false — storage dtype and compute dtype matching for
the first time on this engine.

What Survived, and Is Still Unexplained

Tensor Core utilization is 0.0% on the Ada card too — 100 distinct kernels,
362.8 ms of them, and not one Tensor Core firing.

Removing the dtype pressure made the machine roughly four times faster without
making it touch the hardware it was sold for. That is the open question now, and it
is a better one than the question this started with.

Teardown

Both rigs run on spot capacity and are terminated after collection. The XLA cache
is pushed to S3 before teardown, which is what makes the 6-second restore on a
fresh instance possible.

Summary

The goal of this article was to find out which parts of "it's just JAX" survive a
move between GPU generations. The key to the solution was reading the compute dtype
off the live device rather than a config file. The measured results were:

The model code ported untouched. All four Gemma 4 irregularities, both attention geometries, the KV-share map and the ring — identical source on both cards.
The compilation cache and static-shape discipline ported untouched, including an 805-file restore in 6 seconds onto a fresh instance.
The fused Pallas kernel ported to neither card, because it was written against a 16 MB scratchpad and GPU shared memory is 64 KiB per block on Turing.
The dtype tax was 87% of decode on Turing and 0.0% on Ada, worth 3.7x throughput, with nothing red in any log.
Tensor Core utilization is 0.0% on both cards and remains unexplained.

Scope: two spot instances, one in us-east-1d, each measured once with 3 repeats
per sweep cell and medians reported. The two boxes differ in host architecture
(aarch64 against x86_64) and base image as well as in GPU, so this is not a
single-variable experiment; the payload is byte-identical across them — same build
51bc52c9e2e9, same config, same 6,155,450,950 bytes of weights — which is the
basis for attributing the difference to the chip. The Turing profile was reproduced
on a second instance at 1466.0 ms against 1467.1 ms; the Ada profile was measured
once.

The strategy for using MCP for Gemma 4 serving across GPU generations was validated
with an incremental step by step approach.