DEV Community: szp2005

ONNX Runtime Web fixes its thread count once per page, so our timeout retry had to start a new Worker

szp2005 — Tue, 27 Oct 2026 02:35:20 +0000

Our background remover runs a BiRefNet model in the browser with ONNX Runtime Web 1.27 on the WASM backend. Multithreaded WASM is the fast path when the page can have it. Sometimes it can't: the worker script gets blocked somewhere on the network, or session creation just hangs. The obvious fix is to catch the timeout and call InferenceSession.create again with one thread.

In ORT-web that retry goes nowhere, and the reason is easy to miss.

Why the same-page retry is a dead end

The WASM backend is a page-level singleton. The thread count has to be settled before the first create in that context, and once the runtime is up, setting ort.env.wasm.numThreads again doesn't give later sessions a different thread pool. So "try multithreaded, fall back to single-threaded" can't happen inside one JavaScript context. You either decide before the first create, or you throw the context away.

We put that at the top of the module that makes the decision, so nobody "fixes" it later (comment translated from Chinese):

/**
 * ORT's WASM backend is a page-level singleton. Thread count must be
 * decided before the first create, so this never returns a
 * "retry single-threaded on the same page" plan.
 */
export function decideWasmSessionInit({ configuredThreads, storedProbeResult }) {
  if (configuredThreads <= 1 || storedProbeResult === "unavailable") {
    return { mode: "single-thread" };
  }
  return { mode: "multi-thread" };
}

Deciding before the first create

The thread count comes from one expression:

const configuredThreads = crossOriginIsolated
  ? Math.min(8, navigator.hardwareConcurrency || 4)
  : 1;

If the page isn't cross-origin isolated (no COOP/COEP headers), you get one thread no matter how many cores the machine has. That's the first thing I'd check when someone says ORT-web feels slow. If it is isolated, we cap at 8 and assume 4 when hardwareConcurrency is missing.

Isolation alone isn't enough, because ORT's threads are workers and a worker script can still fail to load. So before the first create we start a tiny probe worker whose entire source is postMessage("ok") and give it 2.5 seconds. An "ok" means multithreaded. An error or a timeout means single-threaded. Either result goes into sessionStorage, so a tab probes once. If sessionStorage throws, as it can in restricted private modes, we still use this run's probe result; it just isn't remembered.

When create hangs anyway

The probe can pass and session creation can still stall. We race create against a 20-second timer. Normal WASM init takes seconds, so hitting that limit usually means a worker request got stuck in the network layer. ORT has no way to abort a create, so if the abandoned one finishes late, we release that session right away instead of leaking it.

Recovery is where the singleton rule bites. The matting code runs in its own dedicated Worker. On a timeout, that worker reports a WasmSessionInitTimeoutError. The client sets a forceSingleThread flag and terminates the Worker, and its heap and threads go with it. The next request gets a fresh Worker that skips the probe and runs single-threaded. The page doesn't reload, so whatever the user uploaded or edited stays where it was.

What it costs

The request that timed out still fails. The worker client doesn't retry it on its own, because retrying in place would mean reusing the stuck runtime, and the only other option is reloading the page and losing the user's state.

Single-threaded is slower. I haven't measured by how much on real devices, so I'm not going to put a number on it.

One more thing to watch for. The same module still exports a second recovery helper that writes the single-thread flag, reloads the page, and keeps a guard so it only self-heals once. Nothing imports it. Read the module alone and you'd conclude we reload on timeout. We don't. To know which recovery path is live, look at what the caller imports, not at what the helper file offers.

If you run models with ORT-web, this is what I'd take from it:

When performance looks wrong, check crossOriginIsolated before anything else.
Decide the thread count once, before the first create.
After an init timeout, move to a new Worker or a new page, with a guard against loops. Don't rebuild in place.

The cutout tool this comes from: https://koutuxia.com/?utm_source=devto

Disclosure: I build Koutuxia.

WebGPU ran our in-browser cutout 6x faster, and drew the wrong mask

szp2005 — Tue, 20 Oct 2026 02:34:10 +0000

Our background remover runs a BiRefNet model inside the visitor's browser with ONNX Runtime Web. For a while the free tier used WebGPU when it was there and fell back to WASM when it wasn't. That fallback rests on an assumption most of us make without thinking: same model, same answer, one backend is just quicker.

On our model, with ORT-web 1.27, the assumption was false. WebGPU was fast and wrong, and none of our benchmarks noticed, because every one of them ran on CPU. We had never measured the path most visitors actually got.

How it showed up

I was checking a second matting pass in a real browser when the trimap's unknown region came out at about 95% of the image. Offline, the same image gave roughly 11%.

So I took the same model bytes, the same input and the same page, ran it three ways, and logged the logits and where the alpha values landed:

Weights + backend	Lowest logit	alpha <= 0.05	Soft band	Inference
int8 + WASM	-20.70	76.0%	0.8%	16.8 s
int8 + WebGPU	-5.34	2.2%	75.1%	2.8 s
fp32 + WebGPU	-20.71	76.0%	0.8%	3.7 s

The negative logits got squashed. Positive ones were mostly fine, so the subject still looked right, but the background lost its confidence and about three quarters of the frame slid into the semi-transparent band. On screen that passes for a "softer edge" until you score it against ground truth. Setting graphOptimizationLevel to basic or disabled didn't save it (disabled was worse). fp32 on WebGPU matched WASM, which pointed at the int8 DequantizeLinear path in that backend, not at WebGPU as such.

The finished cutouts confirmed it. We ran seven opaque objects from the AIM-500 set through the full production pipeline in a real browser, and WASM beat WebGPU on every one, with a median SAD of 1.10 against 6.15. We turned WebGPU off that day, September 13.

Getting the speed back without it

Part of WASM's slowness was the same int8 weights. WASM has no integer acceleration, so dequantizing is extra work on every layer. Switching to fp16 weights brought 1024-input inference on the same image from 16.8 s down to 6.95 s, around 8 s end to end, which is roughly what WebGPU users had been seeing. The fp16 output matched fp32 closely; mean alpha difference was on the order of 1e-5.

Getting a working fp16 file took a custom pass. The converter left one branch in fp32, ORT refused to create a session with mixed-type nodes, and we had to insert a Cast on every node that took both precisions until none were left.

Then we tried that fp16 model on WebGPU, hoping to turn it back on. It failed the opposite way: the whole output collapsed to background, so the cutout was empty. For this model on this ORT-web version, fp32 was the only WebGPU configuration we found that computed correct numbers. USE_WEBGPU is still false in production.

The check we run now

When you change backends, compare the output distribution, not the timing. The raw logits go through a sigmoid to become alpha, so a few counters tell you whether two backends agree:

// logits: Float32Array straight from session.run()
function maskStats(logits) {
  let min = Infinity, max = -Infinity, bg = 0, soft = 0;
  for (const x of logits) {
    if (x < min) min = x;
    if (x > max) max = x;
    const a = 1 / (1 + Math.exp(-x));
    if (a <= 0.05) bg++;
    else if (a < 0.95) soft++;
  }
  return { min, max, bg: bg / logits.length, soft: soft / logits.length };
}

If the soft band jumps from around 1% to around 70%, the backend is computing a different function. It isn't being gentler with edges. Keep an fp32 build around as the referee. And create the session with exactly one provider, executionProviders: ["webgpu"], so a silent fallback can't report the wrong backend and send you chasing a ghost.

What this costs

Going WASM-only gives up WebGPU's best case. In our telemetry from before the shutdown, warm WebGPU runs on 8 to 11 core desktops had a p50 near 1.6 s, though that was only three samples. Cold first cutouts were another story, since session creation alone could eat several seconds. The current default tier (WASM, 1024, fp16) isn't in the telemetry summary yet, so I can't show you today's real-device numbers for it.

I also want to be clear about scope. This is one quantized model on ORT-web 1.27. It is not proof that WebGPU can't run quantized models; it's a reason to check value for value before you trust a speedup.

The full real-device table, split by backend, device type and CPU cores, is on our own page: https://koutuxia.com/en/guide/webgpu-vs-wasm-background-removal?utm_source=devto

Disclosure: I build Koutuxia.

Apple's on-device model called our iPhone 17 "not eligible", so we tested everything except the model

szp2005 — Thu, 15 Oct 2026 02:31:57 +0000

The first time I asked the on-device model whether it could run on a real phone, SystemLanguageModel.default.availability answered .unavailable(.deviceNotEligible). The device was an iPhone 17 on iOS 26.6, which is new enough for Apple Intelligence. It was also bought in mainland China, where Apple Intelligence isn't offered, and as far as I can tell that is the whole reason.

That was a problem. It was the night before I submitted an AI menu for CapScribe, my caption app: keyword emphasis, title suggestions and emoji, all generated on device. The one phone I own will never run it. The simulator can't run it either.

So the honest version of this post is: here is how we shipped a Foundation Models feature we could not run, and what we tested instead.

The gate has more than two states

availability isn't a Bool. When it's unavailable you get a reason, and the reasons deserve different copy:

switch SystemLanguageModel.default.availability {
case .available:
    return .available
case .unavailable(let reason):
    switch reason {
    case .deviceNotEligible:
        return .unavailable("Device not eligible for Apple Intelligence")
    case .appleIntelligenceNotEnabled:
        return .unavailable("Apple Intelligence not enabled in Settings")
    case .modelNotReady:
        return .unavailable("Model is downloading or not ready")
    @unknown default:
        return .unavailable("Model unavailable")
    }
}

Only one of those is permanent. "Not enabled" is a trip to Settings. "Not ready" usually means the model is still downloading and will sort itself out. Collapse them into a single "AI unavailable" string and you tell the user with a fixable problem that they're stuck, while the user with an ineligible phone learns nothing.

My first instinct was to hide the menu whenever the model isn't there. I didn't. A missing button looks like a bug to anyone who read the release notes, and a hidden menu can't explain itself. What shipped: the menu always appears, the three actions are disabled, and a disabled row at the bottom says which case you're in.

A seam so the fallback can be tested

The service takes an optional model behind a one-method protocol:

public protocol AssistantModel: Sendable {
    func respond(to prompt: String) async throws -> String
}

If a model is injected, the service sends it the prompt and decodes the JSON it returns. If not, it checks availability and either opens a LanguageModelSession with a @Generable output type or throws .unavailable(reason). The real path is short on purpose. The less code that only runs on hardware I don't have, the better.

Unit tests inject a mock that records the prompt. That's where the input cap gets checked: transcripts are cut to 3,000 characters before they reach the model, and a test feeds a much longer string and asserts nothing past the cap made it into the prompt.

UI tests can't inject anything, so there's a DEBUG-only launch argument:

#if DEBUG
public static func makeFakeIfNeeded() -> OnDeviceAssistant? {
    guard ProcessInfo.processInfo.arguments.contains("-uitest-fake-assistant") else { return nil }
    return OnDeviceAssistant(model: FakeAssistantModel())
}
#endif

The fake returns fixed JSON: a couple of keywords, three titles, one 🎬. With the flag set, a UI test runs all three actions in the editor. It checks that keyword emphasis gets switched on in the style panel, that three title rows appear, and that the emoji really lands at the end of the caption text. Without the flag, a second UI test opens the same menu in the simulator, where Foundation Models reports unavailable, and asserts the actions are disabled and the reason row exists and isn't empty.

That second test is the one I care about most. On my own phone the unavailable path is the only path, and for plenty of users it will be too.

What this doesn't cover

The real model. Prompt quality, latency, whether @Guide(.count(3)) really yields three titles every time on a real device: none of that has run on hardware we own. The App Review notes spell out the device requirement, and real-model verification sits on our roadmap as blocked until we get an eligible phone.

I went back and forth on shipping like that. What tipped it was proportion. The code that only runs on eligible hardware is a handful of lines handed straight to Apple's framework. The code that decides what everyone else sees runs on every build.

If you're adding Foundation Models to an app, two things are worth doing on day one. Give each unavailability reason its own message, and put a protocol between your feature and LanguageModelSession, so the fallback path gets tested in the simulator, where it's the default anyway.

I build CapScribe, an on-device caption app for iOS. The menu in question is in the editor: https://apps.apple.com/app/id6801041856?utm_source=devto

One speed number lied about our in-browser cutout. We published the distribution instead.

szp2005 — Tue, 13 Oct 2026 02:39:06 +0000

Marketing loves a single millisecond. Real browser-side matting does not.

We run BiRefNet locally in the visitor's browser so pixels never leave the device. That choice makes latency a property of their machine: CPU class, WASM thread count, cold versus warm model cache. Quoting one lab laptop as "the" speed was the wrong default.

The wrong default

Early pages said something like "about two seconds." That figure came from one desktop class on one model tier. Readers with fewer cores, cold caches, or mobile devices saw something else and assumed the product was broken. A single hero number hides the spread.

We also tried enabling WebGPU as the "fast path." On ORT-web with our quantized weights, WebGPU did not merely run slower. It computed wrong masks. So USE_WEBGPU stays false. The live path is WASM-only, fixed square input at the current tier, lite fp16 gzip around seventy-eight megabytes. Correctness first.

What we ship instead

We opened a public benchmarks page built from opted-in telemetry. Last ninety days hold about five hundred samples. Each row is a phase (download, cache hit, session create, first inference), backend, device class, and CPU bucket.

Rules we follow on that page:

Show p50 and p95, not a marketing minimum.
Buckets with too few samples stay listed but greyed so absence of data is visible, not silently dropped.
JSON and CSV are downloadable so anyone can recompute.

One solid slice from the older mid-resolution quantized model (not the current larger fp16 tier): first inference on desktop WASM in the common eight-to-eleven-core bucket, with dozens of samples, sits near a two-second median. The same phase's p95 stretches past ten seconds.

A nearby twelve-plus-core bucket shows a higher median around three and a half seconds. Cache hits land in tens of milliseconds; cold downloads sit in the multi-second range. Session create for those classes is roughly two seconds at the median.

Those rows are historical mid-tier samples. The live product now serves the larger fp16 WASM build. We keep the older distribution public so the method stays honest while newer tiers accumulate past the display floor.

Cost of this approach

Publishing a distribution costs more words than a badge. It also costs the discipline not to cherry-pick. We accept grey cells and model-version labels instead of inventing a WebGPU "about one-point-seven seconds" story that our code does not run.

The upside is boring and useful: when someone asks "how fast is it?", we can point at buckets that match their machine class, and they can download the same numbers we use.

Live tables, greying rules, and downloads: https://koutuxia.com/en/benchmarks?utm_source=devto. I built this tool (Koutuxia); the benchmarks page is our own telemetry, not a third-party lab.

Fixing one typo in a caption used to throw away the word timings for the whole line

szp2005 — Thu, 08 Oct 2026 02:35:38 +0000

CapScribe burns captions into video on the iPhone, and one of its caption styles highlights each word as it is spoken. That only works because the on-device transcriber hands back a start time and a duration for every word. Then someone fixes a typo, and you have to decide what happens to those timings.

What it looked like

Until the 2.4 release, the answer was: they disappear. The edit handler set the segment's words to nil and moved on.

The renderer handles that case carefully. A segment with no word timings falls back to a static caption instead of inventing timings, and the editor shows a note saying so. That is honest, but from the user's side it is baffling. You correct "quik" to "quick", and the highlight on that line just stops. Fix a few typos across a video and a few lines go static.

Why not something simpler

Re-running recognition every time someone edits one line is a lot of machinery for a typo. Spreading the segment's duration evenly over the new words is cheap, but it throws away good data. In a four-word line where one word changed, three of the timings are still exactly right, and anything that recomputes them can only make them worse.

So the rule became: words the user didn't touch keep their timing byte for byte, and only the edited stretch gets new numbers.

What we do now

WordTimingRealigner is a pure function. It takes the old words, the old text, the new text and the segment bounds, and returns new word timings or nil.

Step one is a word-level longest common subsequence between old and new tokens, compared case-insensitively, with punctuation dropped during tokenization. Matched words become anchors and are copied across unchanged, confidence included. That alone means editing "hello world" into "Hello, world!" returns the original array, and the unit test asserts equality on the whole thing.

Step two fills the gaps. Each run of unmatched new tokens sits between two anchors, so it gets the window from the end of the previous anchor to the start of the next one, split by character count:

let totalChars = gapTokens.reduce(0) { $0 + max(1, $1.count) }
for (idx, tok) in gapTokens.enumerated() {
    let dur = idx == gapTokens.count - 1
        ? winEnd - curr   // last token absorbs rounding
        : winDuration * (Double(max(1, tok.count)) / Double(totalChars))
    output[gapStart + idx] = WordTiming(text: tok, start: curr, duration: dur, confidence: nil)
    curr += dur
}

Generated words get confidence: nil, so nothing downstream mistakes a guess for recognizer output. The last token in a gap takes whatever time is left, so floating-point drift can't push it into its neighbour.

Two details took longer than the algorithm itself.

The LCS table is built from the suffix end, which lets the backtrack walk forward. When a word repeats ("the cat saw the dog"), it takes the earliest match that still keeps the subsequence maximal, so the same edit always produces the same anchors.

Chinese has no spaces, and the stored words might be whole words or single characters. Before matching, the function checks whether the stored words agree with enumerateSubstrings(byWords) or with a one-token-per-character split, and tokenizes the new text the same way. If neither agrees, it doesn't guess.

Where it gives up

It returns nil, and that segment falls back to a static caption, when:

nothing matches at all (the line was rewritten),
the stored words don't validate against the old text, overlap each other, or fall outside the segment,
an insertion lands between two anchors that touch with zero gap, so there is no window to put the new word in.

The last case is a real limitation. If the recognizer reported two words back to back and you insert a word between them, that line still loses its highlight. Borrowing time from the neighbours would fix it, but it breaks the rule that untouched words never move, and I would rather show a static line than shift a correct timestamp.

The output is checked again before it is returned: right count, every duration positive, monotonic, inside the segment. A property test runs 50 seeded random edits (replace, insert, delete, punctuation) and asserts those invariants whenever the result isn't nil.

The cost

Splitting by characters is a guess. A long word does not take proportionally longer to say than a short one, so the highlight on a changed word can land visibly early or late. For a typo fix inside one word, the guess is boxed in by the anchors on both sides, so the error can't spread past that word's old window plus any silence around it. Bigger rewrites get less precise, and a full rewrite gets nothing, which is the point.

The table is O(M×N) per edit. That is fine for a caption line and not something I would run over a whole transcript.

I build CapScribe, an iOS app that transcribes and burns in captions on the device: CapScribe on the App Store

Our IP score knows this is iCloud Private Relay and still rates it like a paid VPN

szp2005 — Wed, 30 Sep 2026 02:08:02 +0000

Paste 104.28.28.5 into our IP checker and you get 65 out of 100, amber, with a relay tag sitting next to vpn and proxy. Paste a Mullvad WireGuard exit in Atlanta and you get the same score. One is an Apple egress carrying ordinary Safari traffic from people who flipped a switch in their iCloud settings. The other is a node on a VPN people pay for.

It looks like a bug. We looked at it as one, and then decided to keep it.

What the vendors actually return

Our score is a weighted average across a set of reputation sources, plus hard floors. The floor that matters here: if enough dedicated sources vote proxy or VPN, the score can't drop below 65. "Dedicated" means everyone except ip-api, whose flags are coarse booleans. For an address that already sits in a datacenter, "enough" means two votes, because a lone proxy flag on cloud space is usually one vendor painting a whole range.

On the Apple address, three sources voted: Scamalytics, proxycheck and vpnapi.io. The odd part is what two of them said next to the vote. Scamalytics returned is_apple_icloud_private_relay: true and flagged the IP as a VPN in the same response. vpnapi.io returned relay: true and vpn: true together. So they know exactly what this address is, and they still call it a VPN.

You can check it yourself, no key needed:

curl -s 'https://ipok.io/api/ip?ip=104.28.28.5' \
  | jq '{risk, signals, evidence, floors: .riskBreakdown.floors}'

The weighted average of the individual source scores comes out around 30. The floor is what drags it up to 65.

What we tried

The first fix was the obvious one. We already detect relay: Apple publishes its egress ranges and we load them into our offline list, and IPHub and vpnapi.io both expose a relay field. So exempt relay IPs from the VPN floor and move on.

We shipped half of that. Relay now exempts an IP from the hosting floor of 35, which exists because datacenter traffic gets treated with more suspicion even when nothing specific is wrong with the address. That floor was plainly wrong for Private Relay. The exit is in Cloudflare's address space, but there's a normal person behind every connection. Our own offline list also stopped casting a VPN vote on those ranges.

Dropping the 65 was a different question. The score isn't a verdict on whether the person behind the IP is up to something. It's a forecast of how other services will treat the address. Any site that asks one of those same vendors gets "VPN" back, relay or not. If we removed the floor, we'd be overruling three vendors whose verdicts other sites may well be reading. The score would look friendlier and predict worse.

What we do now

The decision sits in the code, right next to the check, so nobody "fixes" it later:

const isRelay = any("relay");
// Relay only exempts the hosting floor, NOT the 65 proxyVpn floor.
// The score describes how the outside world treats this IP,
// and many services really do block iCloud Private Relay.
if (any("hosting") && !hasTrustedWhitelist && !isRelay)
  floors.push({ key: "hosting", floor: 35, bucket: "contextual" });

(The comment is translated; the original is in Chinese.)

The relay tag in the result is the explanation, and the methodology page has a paragraph saying the same thing in plain language.

This has a cost. The headline number can't tell "you turned on an Apple privacy feature" apart from "you're on a paid VPN". If that difference matters to you, you have to look for relay in signals, or open the per-source raw fields and read Scamalytics' relay boolean yourself. We accepted that. The alternative is a score that disagrees with the vendors doing the actual blocking, which would make it wrong about the one thing it is supposed to predict.

If you run any kind of reputation scoring, it's worth deciding which of the two your number is: a judgment about the user, or a prediction of how others will judge them. We went with the prediction, and Private Relay is where that choice feels worst to explain to a user.

I built ipok, the checker used in the examples above.

Fable 5.1 costs twice as much as Opus 5, until your cache gets big enough

szp2005 — Mon, 28 Sep 2026 02:10:53 +0000

Claude Fable 5.1 shipped on September 1 with the same $10 input / $50 output per million tokens as Fable 5. One number moved: cache reads dropped from $1 to $0.25 per million.

On the price sheet it still costs twice what Opus 5 does ($5 / $25). But look at the cache-read column and the order flips. Fable 5.1 charges $0.25 there, Opus 5 charges $0.50. So which one is cheaper depends on how much of each request is served from cache, and I wanted the actual crossover instead of a vibe.

A request bill has three parts

Cached tokens times the cache-read price, fresh input times the input price, output times the output price. Fable 5.1 only changed the first term, so the saving over Fable 5 is exactly as big as that term's share of your bill.

Two workloads I priced out from the table on my site:

Per request	Fable 5	Fable 5.1	Opus 5
Agent: 100K cached, 2K new input, 1K output	$0.170	$0.095	$0.085
Chat: 5K cached, 1K new input, 800 output	$0.055	$0.051	$0.028

In the agent loop, cache reads are about 60% of the Fable 5 bill, so the price cut takes 44% off each request. That lines up with the "around 45% for heavy agent use" figure in the launch coverage.

The chat case barely moves. Output dominates, and Opus 5 stays roughly half the price. Switching there is just paying more.

The crossover

Fable 5.1 pays double for fresh input and output, and half for cache reads. Set the two bills equal:

0.25 * R = 5 * U + 25 * O
R = 20 * U + 100 * O

R is cached tokens per request, U is fresh input, O is output. Plug in the agent numbers above and the line sits at 140K cached tokens. Both models cost $0.105 a request there.

Past that point the gap keeps widening. At 200K cached, Fable 5.1 is about 11% cheaper. Double the cache again and it's more than a quarter cheaper.

Output length is what pushes the line out. Every extra thousand output tokens needs another hundred thousand cached tokens before Fable 5.1 catches up.

The part I got wrong the first time

My first pass stopped there and the conclusion was "long-context agents should move to Fable 5.1." Then I remembered that tokens have to get into the cache before they can be read from it.

Anthropic bills a 5-minute cache write at 1.25x the input price for Opus 5. My price table has no separate write price for Fable 5.1. If it follows the same 1.25x rule, writing a 200K prefix costs $1.25 more on Fable 5.1 than on Opus 5. At 200K cached, Fable 5.1 saves $0.015 per request. That's more than 80 requests on the same prefix before the write premium is paid back.

So if your prefix expires every five minutes, or your agent keeps rewriting its context, you may never reach the crossover at all. Measure how many requests actually reuse each prefix before you switch.

How I'd decide

Pull three averages from your logs: cached tokens, fresh input and output per request. Then check how many requests reuse the same prefix.

Below 20U + 100O cached tokens: stay on Opus 5.
Above it, but prefixes get rebuilt every few dozen requests: the write premium eats the difference. Stay on Opus 5.
Above it, with one long-lived prefix (a coding agent in a long session, Q&A over one big fixed document): Fable 5.1 wins per request, and it scores 66 vs 63 on the quality index I track.
Already on Fable 5: switch. Nothing got more expensive.

I built llmabacus.com, the LLM API price table these numbers come from. The full write-up, in Chinese, is at https://www.llmabacus.com/articles/fable-5-1-cache-read-vs-opus-5

CoreText said 59 points, the Chinese caption needed 71, and the export "succeeded" blank

szp2005 — Fri, 25 Sep 2026 02:12:44 +0000

The export finished. The file played and the size looked right. There was not a single caption on screen.

That's what my iOS caption-burning app did the day I tried to make Chinese App Store screenshots. Every Chinese caption came out blank. English was fine, which is why nobody had noticed. No API returned an error, and CGContext.makeImage() happily returned a non-nil image. It was fully transparent.

It was two separate bugs with the same symptom, and both only show up if you look at pixels.

How the captions get into the video

Captions are drawn with CoreText into a CGImage, put on a CALayer, and composited over the video with AVVideoCompositionCoreAnimationTool. Each caption layer starts at opacity = 0 and gets a discrete keyframe animation that switches it on and off at the segment's start and end times. Both steps can fail silently.

Bug 1: CoreText measured Chinese text too short

To size the bitmap, the old code asked CoreText how tall the text would be:

let size = CTFramesetterSuggestFrameSizeWithConstraints(
    framesetter, CFRangeMake(0, 0), nil,
    CGSize(width: width, height: .greatestFiniteMagnitude), &fitRange)
return ceil(size.height) + 2

For a 57.6pt Chinese caption (no spaces anywhere), it suggested 59 points. Actually laying out the same string needed 71. We built a CTFrame with a 61pt path, and CTFrameGetLines returned zero lines. Zero lines means nothing gets drawn, and every API involved treats the transparent result as a successful render.

We'd been bitten by this class of bug before. boundingRect once measured 68.0 where CoreText wanted 68.4, and half a point was enough to drop the only line. Our takeaway then was "use the CoreText API". It should have been "measure with the exact code path you draw with".

The current version lays the text out for real inside a path too tall to constrain anything, then sums the actual line metrics:

let probePath = CGPath(rect: CGRect(x: 0, y: 0, width: width, height: 100_000), transform: nil)
let frame = CTFramesetterCreateFrame(framesetter, CFRangeMake(0, 0), probePath, nil)
for line in CTFrameGetLines(frame) as? [CTLine] ?? [] {
    var ascent: CGFloat = 0, descent: CGFloat = 0, leading: CGFloat = 0
    CTLineGetTypographicBounds(line, &ascent, &descent, &leading)
    total += ascent + descent + leading
}

I didn't trust that alone. A second step builds the frame at the measured height and checks that CTFrameGetVisibleStringRange(frame).length covers the whole string. If not, it adds height and retries, up to five times. It's cheap, and it's the only code in the pipeline that can see this failure.

Bug 2: a caption covering the whole clip never appears

This one showed up on a 4-second screenshot clip with one caption running from 0 to the end. Our timing code turned that into keyTimes = [0, 1], values = [1]. That's a legal shape for a discrete animation (one more key time than values), but a one-value discrete animation changes nothing, and the layer sat at opacity 0 for the whole export.

Same sentence, measured both ways: spanning the whole clip, 0 caption pixels. Moved to 0.5s through 3.5s, 14,100.

The fix is boring on purpose. When the track has one value, skip the animation:

if track.values.count == 1 {
    return track.values[0] > 0 ? .always : .never
}

and .always just sets container.opacity = 1.

Counting pixels instead of watching videos

Neither bug throws or logs. A test asserting "export returned a URL" passes on both. Watching output by hand doesn't scale across 26 caption presets.

The simulator was no help either. CoreAnimation offline rendering crashes there in IOSurfaceCreate, and a round trip on a real device took about five minutes. So we added a thin shim (PlatformFont, PlatformColor) that lets the production ExportService.swift compile unchanged on macOS, and wrote a command-line harness, Tests/LocalHarness/burnin-main.swift, that links the real source files rather than a copy.

The harness:

Synthesizes a solid-color 6-second source video with a 90° rotation transform.
Runs the real ExportService.renderBurnIn on it.
Grabs frames at exact timestamps with AVAssetImageGenerator, zero tolerance on both sides.
Samples the background color near the top of the frame and counts pixels in the caption band that differ from it.

When a caption should be visible, it expects more than 500 foreign pixels. In a gap, exactly zero, which also catches captions lingering past their end time.

Both bugs are now permanent sections. Section 5 renders three space-free Chinese strings, one long enough to wrap. Section 6 renders a caption from 0 to 6 seconds and checks frames at 0.5s, 3.0s and 5.5s. Any ❌ exits non-zero.

What this costs

"Pixels that aren't background" tells you something was drawn. It can't tell you it's the right text, font or position. We have a few narrower checks (color-specific pixel counts for custom styles, alpha on rounded corners), but the core test is deliberately dumb, and because the source is a solid color it won't notice problems that only appear over busy footage.

I'll take that. In a caption app, the failure I worry about most is a successful export with nothing in it, and this is the check that catches it.

I built this for CapScribe, an iOS app that transcribes on device with Apple's SpeechAnalyzer and burns captions into the video file. grep -rn URLSession Sources/ in the repo returns nothing. https://apps.apple.com/app/id6801041856

One vendor was deciding our residential proxy verdicts, and I couldn't grade it against itself

szp2005 — Wed, 23 Sep 2026 02:07:54 +0000

My IP reputation checker has a rule that moves scores more than the weighted average ever does. When enough sources say "this address is a proxy or VPN", the score gets a hard floor of 65, which puts it in the caution band. How many sources count as "enough" depends on the address type:

const need = isDatacenter ? 2 : 1; // dedicated votes, ip-api excluded
const realProxy = dedicatedVotes >= need || !!knownAnonAsn;
if (realProxy) floors.push({ key: "proxyVpn", floor: 65 });

Datacenter ranges need two independent vendors, because a couple of feeds like to tag whole cloud blocks as proxies. Residential addresses need one, on the theory that residential proxies are rare but real when you do see them. I picked both numbers in August and never measured either.

What the logs said

Last week I pulled the per-lookup vote signatures we already keep. They record which source said what and the final verdict, with no IPs, ASNs or countries. Over a 15-day window the residential one-vote row was the uncomfortable one: 1,256 verdicts pushed into the caution band by a single vote, and in 96% of them that vote came from the same vendor.

So "one dedicated vote" really meant one vendor deciding, alone, for people on home connections. Nothing in the pipeline pushed back, because almost nothing else was voting at the time. One source sat behind a quota gate, another was dormant, and our own offline list only votes when an address is on it.

The datacenter side looked fine. The two-vote rule was absorbing that vendor's habit of flagging entire cloud ranges, which is what it was written for.

The yardstick I almost used

The obvious check is to compare the single vote against our final verdict. On a residential address with one vote, the final verdict is that vote, so the comparison returns 100% and tells you nothing.

Calibration needs a field the thing being graded can't write to. We had just wired in two more vendors, IPHub and vpnapi.io, which are unrelated to each other and to the one under test. The check became: on lookups where only that vendor said "proxy", what did the other two say?

for (const row of rows) {
  if (state(row, SUBJECT) !== ANON) continue;       // only its solo proxy calls
  const votes = YARDSTICKS.map((s) => state(row, s));
  if (votes.includes(ANON)) t.confirmed += row.n;
  else if (votes.includes(CLEAN)) t.refuted += row.n;
  else t.absent += row.n;                            // no opinion: not in the denominator
}
const rate = t.confirmed / (t.confirmed + t.refuted);

I care most about the absent bucket. If both yardsticks time out, or only report Tor or abuse history, they said nothing about proxies. Counting those rows as refuted would drag the rate down for no reason.

The decision rule is coarse on purpose. For residential, 70% confirmation or better keeps one vote, under 40% tightens it to two, and anything in between holds the current value. The datacenter rule can only get stricter, from two votes to three.

Letting a script touch a scoring rule

A weekly job rewriting a threshold that decides user-facing verdicts is a risk of its own, so three guardrails live in code. Each threshold has hard bounds (residential 1 or 2, datacenter 2 or 3). Nothing moves until confirmed plus refuted reaches 500 samples. One run can shift a threshold by one step at most.

My first version still got it wrong. The weekly build wrote the new JSON and deployed it, then the next ordinary push redeployed whatever file sat in the repo. Production would have flipped between the two, and nobody could have said which threshold was live on a given day.

Now the job only proposes. It writes the result to a temp file, and if a threshold would move, CI opens an issue with the full JSON and the evidence. Production reads the committed file and nothing else. I either paste the proposal into the repo or close the issue.

Where it stands, and what it costs

The first run changed nothing. The 500-sample gate held both thresholds, because the two yardstick vendors went live the same day the job did. The published calibration file still shows zero samples and the August values, and until data piles up, residential proxy verdicts remain one vendor's call.

The yardstick has a blind spot I can't measure yet. "Confirmed" means two other commercial feeds agreed. It doesn't mean anyone saw a real proxy, and if all three vendors buy from the same upstream list, agreement costs nothing.

What I actually got out of this week is a smaller thing than a better threshold: I know which number in the scoring code carries the most weight, and I know it had never been checked against anything.

I built ipok.io. The current calibration file is public at ipok.io/api/data/calibration if you want to watch the rate fill in.

Our 3 a.m. batch job was running at Beijing's afternoon peak rate

szp2005 — Mon, 21 Sep 2026 02:20:23 +0000

DeepSeek's API charges two different prices for the same tokens. Which one you pay depends on what time it is in Beijing when the request lands.

That is the whole bug. It took me an embarrassingly long time to find, because nothing in our code was wrong.

The rule

Beijing time, Monday through Friday, 09:00-12:00 and 14:00-18:00 are peak hours. Everything else is off-peak: nights, weekends, Chinese public holidays. Off-peak is exactly half of peak.

For V4 Pro, per million tokens, in CNY:

	Off-peak	Peak
Input, cache miss	¥4.5	¥9
Input, cache hit	¥0.15	¥0.30
Output	¥13.5	¥27

Peak is seven hours a day on weekdays, about a fifth of the week. So most of the time you are already in the cheap band by accident, which is why nobody notices the rule until they land in the other fifth.

Where it goes wrong

Our reindex job ran at 3 a.m. Eastern. That felt like the safe, nobody-is-awake slot.

3 a.m. EDT is 07:00 UTC. That is 15:00 in Beijing, the dead center of the second peak window.

Converted out of Beijing time, the two peak blocks land here:

Zone	Block 1	Block 2
Beijing (UTC+8)	09:00-12:00	14:00-18:00
UTC	01:00-04:00	06:00-10:00
US Eastern (EDT)	21:00-24:00 prev day	02:00-06:00

Read the last row again if you schedule on a US clock. The classic overnight maintenance window, roughly 2 a.m. to 6 a.m. Eastern, is peak from end to end.

Ordinary East Coast business hours, 6 a.m. through 9 p.m., are all off-peak. So the cheap slot is the middle of your working day and the expensive one is the middle of your night.

There is a day-boundary version of the same trap. Beijing Monday morning is Sunday evening Eastern, so a "start of week" full recompute kicked off Sunday night bills at peak from its first request.

The fix

One line at the top of the crontab:

TZ=Asia/Shanghai
0 12 * * 1-5  /usr/local/bin/reindex

Write the schedule in the timezone of whoever is billing you. For systemd the equivalent is an explicit timezone on the timer:

[Timer]
OnCalendar=Mon-Fri 12:00
Timezone=Asia/Shanghai

Do not hardcode the converted UTC times instead. US daylight saving shifts twice a year, so a hand-converted "02:00 UTC" quietly becomes wrong every spring and autumn. Pinning the timezone makes the conversion someone else's problem.

Two things I had to fix after the cron itself.

Long jobs straddle the boundary. A job starting at 11:00 Beijing and running two hours pays off-peak for one hour and peak for the next. Starting it an hour later puts the whole run in the cheap band.

Retry queues quietly move work into peak. Failures pile up overnight and get drained when someone comes in and restarts the consumer, which is exactly the expensive window. That cost never surfaces in a dashboard grouped by model, because the model did not change.

Caching matters more than scheduling

Moving a job out of peak saves half. Caching is the bigger multiplier: an off-peak cache hit is ¥0.15 per million input tokens against ¥9 for a peak cache miss, a spread of 60x once both axes stack.

The two multiply, so you do not have to choose. But if you only have time for one, stabilize your prompt prefix first and move the cron second.

This only applies where the vendor actually prices by time of day. Most APIs charge the same around the clock, so check before you build scheduling logic around it.

Prices here are from DeepSeek's official pricing page, checked 21 September 2026. The longer Chinese version, with the full timezone table, is on my own price-comparison site: https://www.llmabacus.com/articles/off-peak-pricing-timezone-trap

I built that site because price tables go stale without ever throwing an error.

Cache API drops the fragment, so every chunk I stored was the same entry

szp2005 — Fri, 18 Sep 2026 02:13:44 +0000

Our highest-quality model tier never once worked in production. From the day it shipped, every attempt died with the same line:

ERROR_CODE: 7, Failed to load model because protobuf parsing failed

I read that as an out-of-memory symptom and went hunting for memory. The error had already ruled that out and I wasn't listening.

Why memory was a believable story

The tier is genuinely big: 388 MB gzipped, 445 MB after decompression. We had been burned once already by holding the compressed buffer and the decompressed result at the same time, which peaked near 830 MB and killed the tab every time. So when a second failure turned up on the same tier, "too big for the device" fit the shape of what I thought I knew. A previous bug that looked similar makes the new one feel explained before you've looked at it.

What should have stopped me was the clock. Loading and running this thing takes tens of seconds. The failures came back in two or three. A job dying far faster than it could possibly finish is not running out of anything, because it never got started.

What was actually happening

We cache the model in chunks of 32 MB, then write a small manifest last, so that seeing a manifest means the chunks are all there. The keys looked like this:

const chunkKey = (cacheKey, part) => `${cacheKey}#${part}`;
// key#0, key#1, ... key#meta

The Cache API strips the URL fragment. It does it on put and on match, so key#0, key#7 and key#meta are one entry, not three. Every chunk write overwrote the one before it. The manifest goes in last, so what survived was a JSON blob of under thirty bytes.

Then the top of our load path does a plain lookup with no fragment at all:

const cached = await cache.match(cacheKey);

Which hits, because there was only ever one entry. So we handed ONNX Runtime a manifest, and it told us, accurately, that those bytes were not a protobuf.

Every layer here behaved as documented and nothing threw. The writes reported success and the read reported a hit. The parser was the first thing in the whole chain with an opinion about what the bytes actually contained.

The fix, and the gate behind it

Path segments instead of fragments:

const chunkKey = (cacheKey, part) => `${cacheKey}/part/${part}`;

That fixes this bug. It doesn't fix the class of bug, because the cache can still hand you truncated or evicted bytes through no fault of your keys. So there's a second check before anything reaches the parser:

const looksLikeOnnx = (bytes) =>
  bytes.byteLength > 1024 * 1024 && bytes[0] === 0x08;

ONNX is protobuf, and its first field is ir_version, a field-1 varint, so the first byte has to be 0x08. Add a size floor and a tiny manifest can never masquerade as a model again. When the check fails we delete the entry and treat it as a miss, which costs a re-download. I'll take the re-download over feeding questionable bytes to a parser and then reading its error message as gospel about my hardware.

If you cache anything in chunks, it's worth checking your own code for the same shape. Keep # out of cache keys entirely. After writing, read back and count the entries you expect, since overwrites are silent and successful. And validate a magic byte plus a size floor before cached bytes reach a parser.

The part I keep thinking about isn't the fragment rule. That one is in the spec and I could have looked it up any time. It's that I spent weeks treating a symptom as a resource problem when the duration of the failure ruled that out on the first day. A fast failure and a slow failure tell you different things about where the work stopped, and I only knew how to read one of them.

I build koutuxia, a background remover that runs the model in your browser, which is how I met this one.

A /24 where every address is flagged is weaker evidence, not stronger

szp2005 — Wed, 16 Sep 2026 02:10:01 +0000

A few months ago I wrote here about merging eight IP reputation feeds into one score, and I ended by asking what people do with the /24 neighbor signal. Nobody had a clean answer, so I built it. Then it lied to me, and the way it lied was more interesting than the feature.

The idea seems sound. If the address you're checking sits in a /24 where most of the other addresses are known VPN exits or sit on a spam blocklist, that range is probably being handled as a range by whoever is filtering. Signup forms and fraud rules do CIDR bans constantly. So the neighbor picture is real context even when the address itself looks clean.

I implemented it as a ratio: scan all 256 addresses in the /24 against a local offline list, count how many carry a flag, raise a floor on the score when that share crosses a threshold. Below a quarter it does nothing, past a quarter the floor is 40, past half it's 50. I capped it there on purpose, because a bad neighborhood is not a bad address.

Where it broke

The first version scored 1.19.0.5 and reported 256 of 256 neighbors flagged, verdict toxic. Maximum contamination, perfect signal.

Except there are no 256 neighbors. Spamhaus DROP lists 1.19.0.0/16 as one hijacked network, in one line. My scan expanded that line into thousands of addresses and then dutifully reported that every one of them was independently suspicious. The address already carried its own abuse flag from that same list entry. So one fact got counted as a direct hit on the address, and then again as 256 corroborating witnesses.

That isn't evidence stacking, it's one witness wearing 256 hats.

The tell is that the signal runs backwards. A /24 where 60 addresses are flagged by 60 separate entries really is 60 independent observations. A /24 where all 256 are flagged is almost always one observation about the whole range, and the fuller the coverage, the likelier that's what happened.

There was a second, dumber version of the same mistake. I had host-level evidence (an address appearing on three independent blocklists, or answering as an open proxy) counted into the neighbor numerator. That's evidence about one machine misbehaving, dragged in to convict the people next door. It also made the UI argue with itself: the score up top said high risk while the map below drew that address's own cell bright green.

The fix and what it costs

The check turned out to be one function. Is this /24 fully covered by a single entry in the list?

/** Is [a,b] fully contained in ONE range? A /24 painted by one wide CIDR is a
 *  range-level label, not 256 independent bad neighbors. */
function fullyCovered(ranges: [number, number][], a: number, b: number): boolean {
  let lo = 0, hi = ranges.length - 1;
  while (lo <= hi) {
    const mid = (lo + hi) >> 1;
    const [s, e] = ranges[mid];
    if (b < s) hi = mid - 1;
    else if (a > e) lo = mid + 1;
    else return s <= a && e >= b; // the matched entry swallows the whole block
  }
  return false;
}

const net = ipInt - (ipInt & 255);
const blanket = fullyCovered(VPN, net, net + 255) || fullyCovered(ABUSE, net, net + 255);

When blanket is true the neighbor floor is skipped. The address keeps whatever flag it earned on its own and gets nothing extra for the company it keeps. Host-level evidence moved into its own counter that the neighbor ratio never reads.

You can watch both halves from outside without logging in:

curl -s 'https://ipok.io/api/segment?ip=1.19.0.5' | jq .summary
# flagged 256 of 256, verdict "toxic"

curl -s 'https://ipok.io/api/ip?ip=1.19.0.5' | jq '.riskBreakdown.floors'
# only the address's own abuser floor. No neighbor floor at all.

curl -s 'https://ipok.io/api/ip?ip=89.149.52.9' | jq '.riskBreakdown.floors'
# block floor 40, bucket "contextual". 116 flagged, from many separate entries

The cost is that fullyCovered trusts the list to arrive merged and sorted. If a source emits a /24 as two adjacent halves, the block reads as non-blanket and the double count comes back for that range. Merging at build time covers the sources I pull, but it's an assumption rather than a guarantee, and I'd rather say that than pretend the rule is airtight.

The other thing I didn't solve is recency. A neighbor flagged last week and one flagged last spring count the same. Weighting by age is the obvious next move, I don't have a decay curve I can defend, so the flat count stays.

If you run range reputation anywhere, I'd like to know whether you dedupe at ingest instead. Handling it at scoring time still feels like the wrong layer to me.

I build ipok.io, a free IP reputation checker that shows this breakdown instead of one number.