Our browser-based Python playground runs CPython via Pyodide — no backend, no container, everything client-side. That means every first-time visitor downloads about 5.55 MB of runtime before they can execute a single line.
For most people that's a few seconds. For the rest, it was a silent failure.
Fixing it took three attempts, because the first two were built on measurements that turned out to be wrong. This post is about those wrong measurements — they're the part that generalizes past our particular stack.
Sizing the problem, after distrusting the dashboard
Over 14 days we had 626 unresolved Sentry events. The dashboard sorted them into a dozen-odd issues, which is exactly the view that makes you start at the top and work down.
Instead we regrouped every issue by actual root cause:
| Family | Events | Share |
|---|---|---|
| A. Runtime fetch failed | ~334 | 53% |
| B. Cascading errors from A | 100 | 16% |
| C. Empty message / fatal with no info | 79 | 13% |
| D. Third-party injection & abort noise | 66 | 11% |
| E. Storage / filesystem robustness | 47 | 7% |
A and B are the same bug. One problem was producing 69% of our error volume, and the default issue list didn't show it that way — it showed five separate-looking issues in family A alone.
Two more things fell out of the grouping:
- 135 of 137 samples came from a single locale of a single tool page. Not spread across the site. One page.
- OS split: Windows 114 / Mac 12 / Android 10.
That Windows number is the one that stuck with us. It's wildly out of line with a normal desktop/mobile mix, and it's the kind of skew you only see if you group by root cause first and then look at the dimensions. Sorted by issue count, it's invisible.
Trap #1: the error title was lying
Our largest issue was titled:
undefined is not an object (evaluating 'str.length')
That reads like a null-safety bug. Someone forgot a guard, right?
We sampled 100 events from that issue and counted the actual metadata.value:
| Real message | Count |
|---|---|
PYODIDE_INIT_TIMEOUT |
89 |
| importScripts load failure | 4 |
str.length |
4 |
Cannot read properties of undefined ('length') |
2 |
| other | 1 |
89% of the events had nothing to do with the title. The title was just whichever error happened to arrive first and name the group.
The cause was our own code:
fingerprint: ['tool-runtime', slug, operation, stage]
An explicit fingerprint overrides Sentry's default grouping. We had told it to group by where the error happened, not what happened — so timeouts, network failures, and genuine type errors all collapsed into one issue wearing the name of the first arrival.
Takeaway: the moment you set an explicit fingerprint, the issue title stops being evidence. Before prioritizing off a dashboard, sample the events:
GET /api/0/issues/{id}/events/?full=true
and count the real distribution of metadata.value. We nearly spent a sprint chasing a str.length bug that was 4% of the volume.
The fix: add an error_kind field and put that in the fingerprint. The single mega-issue immediately split into distinct groups for init-timeout vs. runtime-error.
Trap #2: navigator.connection.downlink is a placeholder before it's a measurement
With the real problem identified — slow links timing out — the obvious move was to stop eagerly preloading the runtime for people on slow connections.
First version gated on bandwidth:
if (navigator.connection.downlink >= 5) preloadRuntime()
On a normal office connection, Chrome reported:
downlink: 1.7
rtt: 250
Constant. Unchanged over 8 seconds. On a connection doing far better than 1.7 Mbps.
That pair — 1.7 / 250 — is the default 4g placeholder Chrome returns when it has no throughput history to draw on. A brand-new profile on its first visit gets the placeholder, not a measurement. And a first-time visitor is exactly the person who has to download the whole 5.55 MB.
So the gate had it precisely backwards: it excluded first-time visitors on fast connections from an optimization designed for them. It turned 8 existing e2e tests red immediately.
Worth being fair to the API: the downlink values arriving from real users in production were meaningful — 3g clustered at 0.15–1.55, 2g at 0.05–0.25, rtt ranging from 50 ms on 4g to 2700 ms on slow-2g. The field isn't garbage. It's untrustworthy specifically on a cold profile, which is the case you most need it for.
Trap #3: effectiveType oscillates, and sampling it once is a coin flip
Second version switched to the coarser signal:
if (!['slow-2g', '2g', '3g'].includes(navigator.connection.effectiveType)) {
preloadRuntime()
}
Deployed. Production e2e: 7/7 green. Ship it.
Then we ran the same suite three more times and none of those runs preloaded.
Instrumenting an identical page load:
at load: effectiveType = 3g
+3 seconds: effectiveType = 4g
The full sequence oscillates 4g → 3g → 4g over the first few seconds of page life as Chrome revises its estimate.
Our gate ran inside a single requestIdleCallback. One sample. If that callback fired during the 3g dip, the visitor silently lost preloading for the whole session — on a perfectly fast connection.
The 7/7 green run wasn't a passing test. It was a sampling accident. A single green run against a value that moves tells you nothing. We only caught it by re-running the unchanged suite three more times.
The shipped version:
- skips preload when
effectiveType ∈ {slow-2g, 2g, 3g} -
subscribes to
connection.changeand re-evaluates, instead of sampling once - additionally consults a persisted "this device actually timed out before" flag, cleared once a manual run succeeds
What actually shipped
No new self-hosted assets — outbound bytes only went down:
| Area | Change |
|---|---|
| Init budget | Two-stage: 20s quiet period before bulk download (that phase is only ~246 KB), then a bandwidth-derived 60–120s once large files start, plus a 12s no-byte-progress kill |
| Observability | Wrapped fetch inside the Worker, turning a previously silent 5.5 MB into byte-level progress that feeds both the heartbeat and the progress bar |
| Caching | Write to Cache Storage after successful init; repeat visits download nothing |
For reference, the transfer breakdown (brotli, via jsDelivr):
| File | Raw | Compressed |
|---|---|---|
pyodide.asm.wasm |
10.09 MB | 2.99 MB |
python_stdlib.zip |
2.34 MB | 2.31 MB |
pyodide.asm.js |
1.23 MB | 0.24 MB |
The stdlib zip is the interesting row: 2.34 MB compresses to 2.31 MB. It's already-compressed data, so brotli buys you essentially nothing. If you're budgeting a Pyodide load, don't assume one compression ratio across the whole payload.
The retry that makes things worse
One change is worth calling out separately, because the instinct runs the other way.
Our original fallback logic was the standard one: on timeout, switch to a backup CDN and retry. That's correct for unreachable source — DNS failure, 403, regional block.
It is actively harmful for slow link. If you've already been streaming the 5.5 MB for ninety seconds and you switch sources, you don't resume — you download 5.5 MB again from zero, on the same slow connection that couldn't finish it the first time. The retry makes the user's situation strictly worse and doubles your egress.
So the rule became: once bulk transfer has begun, a timeout never switches sources. Source-switching is only allowed during the discovery phase, before large files start moving. Past that point a timeout is diagnosed as a bandwidth problem and handled by extending the budget, not by starting over.
Generalizes cleanly: retry is for the source is broken, not for the pipe is narrow. Retrying a narrow pipe just re-runs the thing that was already failing.
Bonus trap: the error path had its own error
This one didn't cause the outage, but it made the outage unreadable — and it explains a good chunk of Family C ("empty message / fatal with no info") above.
We render a tooltip per init stage:
getStageTooltip(stage) // → t(`runtime.stage.${stage}`)
The default branch interpolated the stage name straight into an i18n key. When we added new stages during this very fix, the new names had no translation entry — so the lookup threw MISSING_MESSAGE. And because that call happens during render, the throw landed inside the error boundary that was trying to display the original runtime error.
Result: the real error was replaced by a translation error, in the exact scenario where you most need the real error. Users saw a blank fatal; we saw events with no message.
Fix: the default branch became a whitelist lookup, returning a safe fallback for any unknown stage rather than constructing a key that might not exist.
The general shape: anything on the error-display path must not be able to fail. If your error boundary, your logger, or your fallback UI can throw, then your worst incidents are also your least observable ones. Audit that path for dynamic key construction, non-null assertions, and anything that assumes data that a failure state might not have.
Two bugs the e2e suite caught that no amount of reading would have
1. The heartbeat killed slow links. We added a 12-second "no progress = dead" check. But pyodide.asm.js loads via a synchronous importScripts, which blocks the Worker thread — it cannot emit progress even while working perfectly. On a slow link that phase exceeds 12s, so the watchdog killed healthy loads. Fix: the stall detector only arms after bulk download begins.
2. Cache backfill on the hot path caused backpressure. Writing to Cache Storage with clone() during init dragged initialization from 8 seconds to 2 minutes. The existing e2e suite went from 1.2 minutes to 7.3 minutes with 8/11 failing.
The second one is worth dwelling on, because the symptom — "everything is slow now" — pointed nowhere in particular. We found it by re-running the identical suite against origin/master to establish a baseline. The comparison localized it immediately.
When a change makes things globally slower, a baseline run beats reading the diff. Diff-reading biases you toward the code you think is hot; a baseline tells you how much slower, which narrows the candidates far faster.
Fix: move the backfill to after init succeeds.
Results
- Production e2e: 8/8 passing
- Auto-preload: 3/3 stable across consecutive runs — the metric that actually mattered, given trap #3
- Stage sequence:
loading_script → script_loaded → downloading_runtime → initialized → ready - Cache Storage populated; zero large-file requests on reload
- Sentry: the former mega-issue now splits by
error_kindinto distinct init-timeout and runtime-error groups
One testing note that saved a lot of noise: to verify the new error_kind tag actually reached Sentry, we used Playwright to intercept *.sentry.io/**/envelope/, read the payload, then abort() the request. You can assert on exact tags without writing test events into production Sentry.
The pattern
The three traps are the same shape: a signal that looks like a measurement but is actually a default, an alias, or a snapshot of something moving.
- The Sentry title was an alias for a grouping we defined ourselves
-
downlinkwas a default standing in for a measurement -
effectiveTypewas a snapshot of a value changing underneath us
And the tell was the same all three times: things agreed too readily. The dashboard handed us one tidy issue. The bandwidth number never moved across 8 seconds. The test suite went green on the first run. In each case the friction we should have hit was missing — and that absence was the signal we kept failing to read.
The cheap habit that would have caught all three: ask what the value looks like when the system has nothing to report. A default placeholder, a first-arrival label, an early estimate. Then check whether you can tell that state apart from a real reading. In all three cases here, we couldn't — and that's the bug, before any of the specific numbers matter.
If you're building on the Network Information API specifically: treat it as a hint you subscribe to, never a gate you sample once — and remember it has no useful history on exactly the first visit you're trying to optimize.
You can poke at the thing this was all about — Python Playground — browser-only, no signup. First load pulls the runtime; reload after that should be instant, which is the entire point of the work above.
Curious how others are handling Pyodide cold starts — particularly if you've found a preload signal more reliable than effectiveType.
Top comments (0)