DEV Community: TBDS

Filling a region by writing four bytes: region-id PNGs and an SkSL lookup shader

TBDS — Sun, 09 Aug 2026 09:11:38 +0000

A paint-by-number canvas has an awkward shape as a rendering problem. The image is large. The number of independently colorable regions is large. Every tap recolors exactly one of them, and the result has to appear within a frame while a pinch-zoom gesture may be running.

The obvious implementations all fail in a specific, instructive way. What actually works is to stop treating "fill a region" as an image operation at all: the pixels never change. Only a tiny lookup table changes.

Why the obvious approaches fail

Flood fill on tap. Scan-line fill from the touch point across a full-resolution image is tens of milliseconds of JS work on a large canvas, on the JS thread, while the gesture is live. Worse, it has to be redone on every reload unless you also persist the filled bitmap.

One vector path per region. Cleanest conceptually — SVG paths, hit-test, set fill. It collapses on two fronts: the path data for a detailed illustration is far heavier than a compressed PNG, and hit-testing many complex paths per tap is not cheap either.

One texture per region, or a full-canvas repaint. Either you allocate an absurd amount of GPU memory, or you rebuild a large texture per tap and upload it. Uploading a multi-megabyte texture on every tap is exactly the thing you cannot do sixty times a minute.

The common mistake in all three: they treat the region geometry as mutable. It is not. Geometry is fixed at authoring time. The only thing that changes when the user taps is which color a region maps to — a value with a few bits of entropy.

The representation

Region geometry is baked once into a PNG where the pixel channels encode a region id rather than a color:

R = id >> 8   (high byte)
G = id & 255  (low byte)
B = 255       marks an outline pixel
A = 255
id = 0        outline / background

so id = (R << 8) | G. Decoding on the CPU is a single linear pass:

export function decodeIdMap(pixels: Uint8Array, w: number, h: number): Uint16Array {
  const n = w * h;
  const ids = new Uint16Array(n);
  for (let i = 0, p = 0; i < n; i++, p += 4) {
    // outline pixels (B=255) encode id 0 even if R/G carry values
    if (pixels[p + 2] === 255) { ids[i] = 0; continue; }
    ids[i] = (pixels[p] << 8) | pixels[p + 1];
  }
  return ids;
}

Fill state is one byte per region — colorIndex + 1, with 0 meaning unfilled:

export function applyFill(state: Uint8Array, regionId: number, colorIndex: number): boolean {
  if (regionId < 1 || regionId >= state.length) return false;
  if (state[regionId] !== 0) return false;
  state[regionId] = colorIndex + 1;
  return true;
}

And the bridge between the two is a palette LUT: an (regionCount + 1) × 1 RGBA texture where texel id holds the current display color of region id.

So filling a region is: write one byte of state, rebuild a one-pixel-tall texture, upload it. The large id texture is never touched, never re-scanned, never re-uploaded. That is the whole trick.

The shader

The renderer is a single Skia RuntimeEffect fed two image shaders — the id texture and the LUT:

uniform shader idTex;
uniform shader lutTex;
uniform float showEdge;
uniform float hlMode;
uniform float time;

half4 main(float2 xy) {
  half4 c = idTex.eval(xy);
  if (c.b > 0.5) {                       // outline marker
    return showEdge > 0.5 ? half4(0.11, 0.11, 0.12, 1.0) : half4(1.0, 1.0, 1.0, 1.0);
  }
  float id = floor(c.r * 255.0 + 0.5) * 256.0 + floor(c.g * 255.0 + 0.5);
  if (id < 0.5) return half4(1.0, 1.0, 1.0, 1.0);
  half4 L = lutTex.eval(float2(id + 0.5, 0.5));
  float flag = floor(L.a * 255.0 + 0.5);
  if (flag > 253.5) {                    // 254 = unfilled base, 255 = filled
    return half4(L.rgb, 1.0);
  }
  // flag == 253: unfilled region belonging to the active color index
  if (hlMode < 0.5) {
    return half4(0.80, 0.80, 0.83, 1.0);
  } else if (hlMode < 1.5) {
    float cs = 22.0;
    float ch = mod(floor(xy.x / cs) + floor(xy.y / cs), 2.0);
    half3 col = ch < 0.5 ? half3(0.87, 0.87, 0.90) : half3(0.73, 0.73, 0.78);
    return half4(col, 1.0);
  }
  float pulse = 0.5 + 0.5 * sin(time * 4.0);
  half3 tint = mix(half3(0.90, 0.90, 0.92), L.rgb, 0.30 * pulse);
  return half4(tint, 1.0);
}

Note the LUT's alpha channel is not alpha. It is a three-state tag: filled, unfilled, unfilled-and-belongs-to-the-currently-selected-color. Highlighting the regions the user should be tapping next is therefore also just a LUT rebuild — no second texture, no second pass, no per-region draw calls. The CPU side that produces the tag:

export const LUT_FLAG_FILLED = 255;
export const LUT_FLAG_UNFILLED = 254;
export const LUT_FLAG_HIGHLIGHT = 253;

buildRenderLut walks ids once and writes RGB plus one of those three flags per texel. That function is pure — no React, no Skia, no storage — so the entire visual state machine is unit-testable without a renderer.

Hit testing, with intent correction

The CPU keeps the decoded Uint16Array, so a tap is an array index. But a raw index is a bad user experience: fingers are wide, regions are small, and a one-pixel miss lands on a neighbor. So the pick does a histogram inside a radius and prefers eligible regions:

for (let y = y0; y <= y1; y++) {
  const dy = y - cy;
  const row = y * w;
  for (let x = x0; x <= x1; x++) {
    const dx = x - cx;
    if (dx * dx + dy * dy > r2) continue;
    const id = ids[row + x];
    if (id === 0) continue;
    counts.set(id, (counts.get(id) ?? 0) + 1);
  }
}

The caller decides what "eligible" means, and it is stricter than "unfilled":

const eligible = (id: number) => {
  if (fsIsFilled(st.fills, id)) return true;  // already filled = not selectable
  const r = st.meta!.regions[id - 1];
  return !r || r.colorIndex !== active;       // wrong color number = not selectable
};
const radius = Math.max(1, Math.round(PICK_RADIUS_BASE / (s / (base.value || 1))));

Radius shrinks as the user zooms in, so correction is generous at fit scale and precise when they are working on detail. Tapping the wrong region simply returns false from the store and produces no haptic — a miss costs nothing.

Costs and boundaries

The color-space trap is the one that will cost you a day. The id texture must be uploaded with no color space attached. If any sRGB conversion touches it, the R and G channels — which are not colors, they are an integer — get rewritten, and ids decode to garbage. In this codebase the id PNG is decoded, read back as RGBA_8888 / Unpremul, and reconstructed with Skia.Image.MakeImage from raw bytes for exactly this reason.

Sampling must be nearest, mipmaps off. Linear filtering interpolates between adjacent ids and invents region ids that do not exist — typically producing a thin halo of a random unrelated region along every boundary. There is no way to detect this from the shader; you just get wrong colors at edges.

Byte order is a contract, and contracts get violated. An early asset generator wrote R = low, G = high; the shipping pipeline is the opposite. Nothing crashes when they disagree — ids simply map to unrelated regions and the picture fills in wrong. If you adopt this technique, write the byte order down in one place, and have the asset generator and the shader both quote it.

Hard ceiling of 65,535 regions. Two bytes. The B channel is spent on the outline flag, so there is no third byte to borrow without another convention.

Memory. The decoded id map is a Uint16Array of width × height — two bytes per pixel, resident for as long as the page is open, plus the id texture itself on the GPU. This buys O(1) hit testing but it is not free, and it is the main reason page teardown has to actively drop state.

Antialiasing is not available. Region ids cannot be blended, so boundaries are hard. The design absorbs this by baking a dark outline into the artwork (the B = 255 path in the shader) which visually hides the aliasing. On an illustration without outlines this technique would look noticeably crunchy.

LUT rebuild is O(regions), not O(1). Each tap rebuilds the whole LUT and allocates a new SkImage. It is small and fast, but it is a per-tap allocation, and at very high region counts you would want a partial texture update instead.

Debug hooks must be __DEV__-gated. The canvas here reads a command file to script fills and zoom levels for screenshot automation. In Release __DEV__ is false and the block compiles out — which is correct, and also means any test relying on it silently does nothing against a release build.

This is the rendering path used by Numbrush.

Chalk that looks like chalk: depositing particles instead of stroking paths

TBDS — Sun, 09 Aug 2026 09:06:22 +0000

Every drawing app starts the same way. You collect touch points, build a path, stroke it with a round cap and a width driven by pressure, and ship. It works. It also looks like plastic, and no amount of tuning fixes that, because the failure is structural rather than cosmetic.

This is a write-up of the rendering engine behind a chalkboard app, and specifically of three decisions that turned out to be load-bearing:

A stroke is a sequence of stamps deposited at fixed spacing, not a path.
The document stores input points, never pixels.
Undo is deterministic replay driven by a seeded RNG.

Each one buys something. Each one also has a bill, and the bill is the interesting part.

Why stroking a path fails

The obvious approach to "make it look like chalk" is: stroke the path, then add noise. Multiply by a grain texture, jitter the width, lower the alpha at high speed.

It never lands, for a reason that is easy to state once you see it. A stroked path is a continuous region with a single alpha value per pixel. Real chalk is not a region — it is a population of particles that either landed or did not. The visual signature of chalk (the broken edges, the dry-brush gaps when you move fast, the way repeated passes reinforce into a solid line) comes from discrete deposition events on a rough surface. If you start from a continuous region and multiply noise on top, you get a continuous region with noise on top. The eye reads it instantly as texture-over-shape.

So the model is inverted. Instead of "draw the shape, then roughen it," walk the path and drop individual grain stamps.

The baker

The whole thing lives in one function that turns a line segment between two input samples into a list of GPU instances:

static func bake(_ s: Stroke, from p0: InkPoint, to p1: InkPoint,
                 rng: inout SeededRNG, into out: inout [StampInstance],
                 scale: Float = 1) {
    let b = s.brush
    let dx = p1.x - p0.x, dy = p1.y - p0.y
    let dist = sqrt(dx * dx + dy * dy)

    let speed = min(1, dist / max(0.001, s.size * 2.9))
    let step = max(s.size * b.spacing, s.size * 0.02, 0.00005)
    let n = max(1, min(4000, Int((dist / step).rounded())))

Note that speed is normalized by pen diameter, not by an absolute point distance. This is not aesthetic pedantry; it is the fix for a real bug. The original code used a hardcoded 26pt as full speed. Once the canvas supports deep zoom, a fast gesture at 800× covers a tiny distance in canvas space, speed pins to zero, and the dry-brush effect silently disappears at high magnification. Scaling by pen diameter — which is itself already divided by zoom — makes the entire speed semantics zoom-invariant.

Per stamp, three families of parameters are applied: pressure, speed, and stylus tilt.

    let elong      = 1 + tilt * b.tiltElongation
    let broaden    = 1 + tilt * b.tiltBroaden
    let tintByTilt = 1 + tilt * (b.tiltAlpha - 1)

    let size  = s.size * broaden
              * (1 - speed * b.speedThin)
              * (0.72 + pr * 0.42)
    let alpha = b.alpha * tintByTilt
              * (1 - speed * b.speedFade)
              * (0.55 + pr * 0.65)

elong becomes a non-uniform scale on the stamp quad, rotated to the stylus azimuth — that is the flat-side stroke. Everything a "brush" is, in this engine, is a struct of numbers: spacing, jitter, alpha, speedFade, speedThin, tiltElongation, tiltBroaden, tiltAlpha, tooth, toothSoftness. Chalk, pencil and ink are the same code with different constants. Adding a material means adding numbers, not code paths.

Deposition happens in the fragment shader

Jitter alone still is not chalk. The missing piece: chalk dust only sticks where the pressure beats the surface roughness. So the stamp texture supplies coverage and a second, canvas-anchored texture supplies tooth:

fragment float4 stamp_fragment(StampOut in, ...)
{
    float mask = atlas.sample(smp, in.uv, in.stamp).r;
    float cov  = mask * in.color.a;                 // coverage 0..1

    float tooth = grain.sample(grainSmp, in.canvasUV).r;

    float deposit = smoothstep(1.0 - cov, 1.0 - cov + bu.softness, tooth);
    float a = mix(cov, deposit, bu.tooth);

    return float4(in.color.rgb * a, a);
}

The threshold moves down as coverage rises. Press hard and even the valleys of the board take dust, so the line goes solid; move fast and coverage drops, so only the peaks catch anything — which is dry brush. You never author the dry-brush effect; it emerges from the same expression that produces a solid line.

The single most important line is in the vertex stage:

o.canvasUV = pt * bu.grainScale;

pt is the canvas position of this fragment, not a stamp-local coordinate. If grain UVs were stamp-local, every stamp would sample a different random patch, and overlapping stamps would average toward flat gray. Anchoring grain to canvas space means repeated passes hit the same peaks and valleys and reinforce each other. That single substitution is the difference between "grainy smear" and "chalk."

The eraser reuses the identical stamping pipeline with a blend of (zero, oneMinusSrcAlpha). The consequence is free and correct: erased edges are dusty, not razor-cut.

Store points, not pixels

The document model is almost embarrassingly small:

struct Stroke: Codable {
    let brushKind: Material.BrushKind
    let color: SIMD4<Float>
    let size: Float           // diameter, in points
    let erasing: Bool
    let seed: UInt32
    var points: [InkPoint]
    var layer: Int = 0
}

That decision pays three times.

Zoom stays sharp. Strokes are stored in canvas coordinates. Zooming does not magnify a bitmap; it re-bakes the same points at a larger scale. Grain stays crisp at any magnification because it is regenerated, not stretched.

Layers are free. Ordering is a stable sort:

let sorted = list.enumerated()
    .sorted { ($0.element.layer, $0.offset) < ($1.element.layer, $1.offset) }
    .map(\.element)

A bitmap architecture needs one full-screen texture per layer. Here a layer costs one Int per stroke. The sort must be stable — Swift's sort is not — hence the enumeration offset as tiebreaker; without it, strokes within a layer reshuffle their overlap on every replay.

Undo is replay. No snapshots, no history of textures:

func undo() {
    if let backup = clearedBackup, strokes.isEmpty {
        strokes = backup; clearedBackup = nil; needsReplay = true; return
    }
    guard !strokes.isEmpty else { return }
    clearedBackup = nil
    redoStack.append(strokes.removeLast())
    needsReplay = true
}

For that to be pixel-identical, randomness must be reproducible. Each stroke carries a seed and replay reconstructs the same generator:

static func bakeAll(_ s: Stroke, into out: inout [StampInstance], scale: Float = 1) {
    guard s.points.count > 1 else { return }
    var rng = SeededRNG(seed: s.seed)
    for i in 1..<s.points.count {
        bake(s, from: s.points[i - 1], to: s.points[i], rng: &rng, into: &out, scale: scale)
    }
}

Rendering itself is incremental: the canvas texture is persistent with loadAction = .load, and only undo / clear / page change trigger a full replay. Redrawing every stroke per frame is fine on a whiteboard demo and dies under 120 Hz coalesced touches.

The failure modes — the part worth your time

RNG desync is the nastiest bug in the design. The baker skips stamps that are too small or too faint. The skip must happen after the RNG has been consumed:

let idx   = UInt32(rng.nextInt(StampAtlas.count))
let jx    = rng.signed() * s.size * b.jitter
let jy    = rng.signed() * s.size * b.jitter
let grain = rng.next() * .pi * 2

guard size * scale > 0.4, alpha > 0.004 else { continue }

Move that guard three lines up and the code is still correct-looking, still renders fine live — and every replay after an undo shifts every grain in the drawing. It reproduces only after an undo, which is exactly when nobody is watching for it. Any edit to this function needs the RNG call count and order re-verified first.

Screen-scale constants inside canvas-scale math. 0.4 above is a screen threshold ("a stamp under half a point is not worth drawing"), but size is in canvas units. At high zoom a stroke's canvas size is a fraction of a point, so without the * scale conversion every single stamp gets skipped: nothing renders, and the stroke is still persisted with that tiny size, so it stays invisible when you zoom back out. Silent data loss with no error anywhere. The same class of bug bit the minimum spacing, the jitter-filter distance, and the brush-size floor. If you build a deep-zoom canvas, audit every constant for which space it lives in.

A one-encoder buffer aliasing trap. An earlier version memcpy'd each stroke's instances to buffer offset 0 before its draw call. The copies happen on the CPU at encode time; the GPU executes after commit. So every draw in that encoder read the last batch written. Live drawing goes through the incremental path and looked perfect; only replay produced garbage, so the bug hid for a long time. The fix is one upload for the whole batch, with per-span offsets.

Empty is not failure. The upload helper returns early on an empty instance array — and it must return success, because "clear the board" is exactly the zero-stamp case. Treating it as failure skips the loadAction = .clear pass and the clear button appears to do nothing.

Real costs. CPU baking is O(total stamps), and a full replay re-bakes everything; a dense drawing has a visible hitch on undo. There is a hard cap of 4000 stamps per segment to stop a pathological segment from exploding. Overdraw is high by construction — every stamp is a blended quad. Editing an old stroke's geometry is not supported, because the model is append-only history. And the aspects that matter most (dry brush, tilt, finger-vs-pencil) cannot be validated in the simulator at all; they need a real device and a stylus, which makes the CI story weak.

One API landmine. altitudeAngle returns 0 for finger touches, and 0 means "stylus lying flat." Without a type == .pencil check, every finger stroke renders as an extremely elongated smear.

This engine powers Flying Chalk.

On-device speech recognition on iOS: the honest boundaries

TBDS — Sun, 09 Aug 2026 09:01:01 +0000

SFSpeechRecognizer gives you a flag called requiresOnDeviceRecognition. Set it
to true and the audio is transcribed by a model that already lives on the phone.
Set it to false and the audio is streamed to Apple's speech service and the text
comes back.

That flag is the source of nearly every misleading privacy claim in the transcription
category, because it is a request, not a guarantee, and because the failure mode
when the request cannot be honoured is an empty transcript rather than an error you
can show a user.

This is a write-up of what the API actually does, what a real implementation ends up
looking like, and what you cannot truthfully promise.

The problem

You want to transcribe a recording. The requirements are ordinary: it should work
offline where possible, it should work on files longer than a minute, it should
produce word-level timestamps so you can seek to a phrase, and it should not
silently hand the user an empty string.

Why the obvious approach fails

"Just set `requiresOnDeviceRecognition = true` and we're private"

Three things break this.

Supported is not installed. supportsOnDeviceRecognition tells you whether a
recognizer can run locally. The on-device model for a language arrives when the
system fetches it, which in practice means after that language has been in use and
the phone has had time on Wi-Fi. A freshly reset phone can report "supported" and
still have nothing local to run.

Coverage varies by language. Apple ships on-device recognition for a subset of
the languages the framework supports overall. Widely used ones have local models on
current hardware; smaller languages and some regional variants are server-only. The
list changes between iOS releases and no app controls it.

Local models are less tolerant. Quiet, distant, accented or noisy audio that the
server model handles will sometimes come back empty from the local one — not an
error, just nothing.

So a strict on-device-only implementation is honest and frequently useless. Users do
not report "the on-device model lacks coverage for your dialect". They report "the
app is broken".

"Just hand the whole file to the recognizer"

SFSpeechURLRecognitionRequest on a long file is where you learn about the
framework's practical limits. Long recognitions become unreliable: the task can run
long, drop most of the audio, or never deliver an isFinal result at all. There is
no documented ceiling you can code against, and the failure is not an exception.

"The completion handler will tell me when it's done"

It will, until it doesn't. A recognition task that hangs simply never calls back.
Without your own timeout, one bad chunk hangs the entire transcription forever.

What actually works

1. Probe capability, but treat it as a hint

AsyncFunction("isAvailableAsync") { (localeIdentifier: String?) -> Bool in
  guard let recognizer = makeRecognizer(localeIdentifier) else { return false }
  if #available(iOS 13.0, *) { return recognizer.supportsOnDeviceRecognition }
  return false
}

And build the recognizer defensively, because locale tags do not always match the
way callers expect (zh-Hans-CN is a classic):

private func makeRecognizer(_ localeIdentifier: String?) -> SFSpeechRecognizer? {
  // Try the requested language; fall back to the device default rather than
  // returning nil for a tag that simply didn't match.
  if let localeIdentifier, !localeIdentifier.isEmpty,
     let r = SFSpeechRecognizer(locale: Locale(identifier: localeIdentifier)) {
    return r
  }
  return SFSpeechRecognizer()
}

2. Chunk the audio, with overlap

Short recognitions are reliable; long ones are not. So the file is exported into
~50-second segments with a few seconds of overlap so a sentence straddling a
boundary is not lost, and each segment is recognised independently. Results
accumulate, and progress is reported per chunk so a long file shows movement:

private func transcribeStreaming(url: URL, recognizer: SFSpeechRecognizer,
                                 onDevice: Bool, recId: String) async throws
                                 -> (String, [[String: Any]]) {
  let asset = AVURLAsset(url: url)
  let total = CMTimeGetSeconds(asset.duration)
  guard total.isFinite && total > 0 else { return ("", []) }

  let chunk = 50.0
  let overlap = 4.0                 // so a sentence cut at a boundary isn't lost
  let step = max(1.0, chunk - overlap)
  var allSegs: [[String: Any]] = []
  var acceptedUntil = -1.0          // global end time already accepted, for dedupe
  var start = 0.0

  while start < total - 0.05 {
    let end = min(start + chunk, total)
    let segURL = try await exportSegment(asset: asset, start: start, end: end)
    defer { try? FileManager.default.removeItem(at: segURL) }

    let (piece, segs) = try await recognizeChunk(url: segURL, recognizer: recognizer,
                                                 onDevice: onDevice)
    if segs.isEmpty {
      // recognised text but no word timings (some recognizers): only accept it if
      // it advances past what we already have, so the overlap can't duplicate
      if !piece.isEmpty && end > acceptedUntil {
        allSegs.append(["s": piece, "t": max(start, acceptedUntil < 0 ? start : acceptedUntil), "d": 0.0])
        acceptedUntil = end
      }
    } else {
      for s in segs {
        let gStart = s.1 + start
        let gEnd = gStart + s.2
        if gStart >= acceptedUntil - 0.15 {   // drop words already taken from the overlap
          allSegs.append(["s": s.0, "t": gStart, "d": s.2])
          acceptedUntil = max(acceptedUntil, gEnd)
        }
      }
    }
    let progress = min(1.0, end / total)
    let final = end >= total - 0.05
    self.sendEvent("onTranscription", ["id": recId, "text": joinSegments(allSegs),
                                       "isFinal": final, "progress": progress])
    if final { break }
    start += step
  }
  return (joinSegments(allSegs), allSegs)
}

Overlap buys reliability and costs deduplication. The dedupe is by global end time
with a small tolerance, using the word-level timestamps the framework gives you in
bestTranscription.segments.

Chunking also means each segment must be exported to a temporary file, which costs
CPU and disk churn on top of the recognition itself.

3. Attempt on-device first, fall back explicitly

This is the design decision that has to be documented rather than buried:

// Recognise one chunk: local first (offline / privacy), and if the local pass is
// empty or throws, retry against the server recognizer (when online).
// Throwing = a real failure; an empty string = that chunk genuinely had no speech.
private func recognizeChunk(url: URL, recognizer: SFSpeechRecognizer, onDevice: Bool)
    async throws -> (String, [(String, Double, Double)]) {
  if onDevice {
    if let r = try? await recognizeOnce(url: url, recognizer: recognizer, forceOnDevice: true),
       !r.0.isEmpty {
      return r
    }
    // local empty/failed -> server fallback (more accurate online; offline it just
    // returns empty again)
    if let r = try? await recognizeOnce(url: url, recognizer: recognizer, forceOnDevice: false) {
      return r
    }
    return ("", [])
  }
  if let r = try? await recognizeOnce(url: url, recognizer: recognizer, forceOnDevice: false) {
    return r
  }
  return ("", [])
}

The on-device path is requested first, every time. When it returns nothing or errors,
the same audio is retried without the on-device flag — which means that audio
goes to Apple's speech service if the device is online. That is a real, deliberate
trade: the alternative is handing the user an empty transcript.

What you must not do is call this "fully on-device". It is not. It is on-device
preferred, with a first-party server fallback. Apple's server path is not a
third-party company, but "first-party" is not the same as "local", and those two get
conflated constantly in app descriptions.

The honest user-facing test is Airplane Mode: with no network, the fallback has
nowhere to go, so either the local model handles it or you get nothing — and either
way nothing was transmitted.

4. Own your timeout

private func recognizeOnce(url: URL, recognizer: SFSpeechRecognizer,
                           forceOnDevice: Bool) async throws
                           -> (String, [(String, Double, Double)]) {
  try await withCheckedThrowingContinuation { continuation in
    let request = SFSpeechURLRecognitionRequest(url: url)
    request.shouldReportPartialResults = false
    request.requiresOnDeviceRecognition = forceOnDevice
    if #available(iOS 16.0, *) { request.addsPunctuation = true }

    var didResume = false
    var task: SFSpeechRecognitionTask?
    let lock = NSLock()
    func finish(_ r: Result<(String, [(String, Double, Double)]), Error>) {
      lock.lock(); defer { lock.unlock() }
      guard !didResume else { return }
      didResume = true
      continuation.resume(with: r)
    }
    // If a chunk never reports isFinal (the server fallback occasionally stalls),
    // fail it after 60s so the layer above can fall back or skip, instead of the
    // whole transcription hanging forever.
    DispatchQueue.global().asyncAfter(deadline: .now() + 60) {
      lock.lock(); let done = didResume; lock.unlock()
      if !done {
        task?.cancel()
        finish(.failure(LocalTranscriberException("Recognition timed out.")))
      }
    }
    task = recognizer.recognitionTask(with: request) { result, error in
      if let result, result.isFinal {
        let segs = result.bestTranscription.segments.map { ($0.substring, $0.timestamp, $0.duration) }
        finish(.success((result.bestTranscription.formattedString, segs)))
        return
      }
      if let error { finish(.failure(error)); return }
    }
  }
}

Two details that are easy to get wrong: a continuation must be resumed exactly once,
so the resume is guarded by a flag under a lock (the timeout and the callback race);
and the real error is thrown rather than swallowed, so the caller — not this function
— decides whether to fall back.

5. Recording is a separate reliability problem

Transcription quality is downstream of capture, and capture has its own silent
failure: an interruption. A phone call takes the microphone. If you do nothing, you
lose the rest of the session.

// Interrupted (incoming call / another app / headphones unplugged):
// .began  -> pause, keep the file (don't finalize), wait to resume
// .ended  -> reactivate the session and record() into the SAME file;
//            if resuming fails, finalize so the captured part isn't lost
if type == .began {
  if let recorder = self.recorder {
    recorder.pause()
    self.stopVadTimer()
    self.interruptedRecording = true
    self.sendEvent("onRecordingPaused", [:])
  }
  return
}

guard self.interruptedRecording, let recorder = self.recorder else { return }
self.interruptedRecording = false
try? AVAudioSession.sharedInstance().setActive(true)
if recorder.record() {
  self.sendEvent("onRecordingResumed", [:])
} else {
  recorder.updateMeters()
  let duration = recorder.currentTime
  let url = recorder.url
  recorder.stop()
  self.recorder = nil
  self.endInterruptionObserving()
  self.sendEvent("onRecordingInterrupted", ["uri": url.lastPathComponent, "duration": duration])
}

The recording format itself is mono AAC in .m4a, with sample rate and encoder
quality as the only user-facing knobs:

let settings: [String: Any] = [
  AVFormatIDKey: Int(kAudioFormatMPEG4AAC),
  AVSampleRateKey: sampleRate,          // 22050 / 44100 / 48000
  AVNumberOfChannelsKey: 1,
  AVEncoderAudioQualityKey: encQuality.rawValue
]

Worth stating plainly because it is widely believed otherwise: a higher sample
rate does not make iOS use the on-device model. That choice depends on language
coverage and whether the local model is installed, not on how the audio was encoded.
Recording at 48kHz buys you nothing in recogniser selection.

Costs and boundaries

Being precise about what this design does and does not give you:

It is not "audio never leaves the device." On-device recognition is requested first. When it is unavailable or returns nothing, the implementation can fall back to Apple's server-based recognition, and on that path the audio itself is what gets sent — not a summary, not just text. Airplane Mode is the only way a user can be certain a particular transcription stayed local.
You can choose strictness instead, and some apps should: request on-device only and report failure. The cost is empty transcripts for every language and every clip the local model cannot handle. That is a product decision, not a technical one, and it should be stated either way.
Empty is ambiguous. An empty chunk means "no speech here" or "recognition failed quietly". The code above distinguishes them by convention (throw = failure, empty = genuinely silent), but the framework does not hand you that distinction.
Accuracy is unmeasured. Both the local and the server model are guessing under uncertainty; they differ in size, not in kind. Do not publish accuracy percentages you have not measured.
Chunking has costs: temporary file exports, overlap deduplication logic, and word timings that are only as good as what each chunk returned. Some recognizers return text with no segments at all, which is why the accumulator has a branch for it.
Speaker labels and summaries are not in this box. Diarization is not something the on-device path provides, and any summarisation means calling a model somewhere — so the text leaves even if the audio did not.
The capability check is per-locale and per-device, and it changes with iOS releases and with what the phone has downloaded. Cache it at your peril.

This is the transcription pipeline used by Voice Studio, an iOS recorder that
attempts the on-device path first and documents the fallback rather than claiming it
does not exist.

One function that makes canonical, hreflang and sitemap unable to disagree

TBDS — Sun, 09 Aug 2026 08:55:51 +0000

Multilingual static sites break in a boring, expensive way: the canonical tag says one URL, the hreflang alternates say a slightly different one, and the sitemap says a third. Google resolves the contradiction by ignoring your signals and picking whichever version it likes. You do not get an error. You get flat traffic and a "Duplicate, Google chose a different canonical" line in Search Console months later.

This is a write-up of a static generator — one file, no dependencies, no client JavaScript — built around two constraints:

Every URL the site emits comes out of a single function.
A page that does not clear a content-length threshold is not allowed to be indexable, and the build enforces it.

Why the usual setup fails

The usual setup is not careless. It is that URL construction is distributed. The canonical tag is written in a layout template. The hreflang block is generated in an i18n helper. The sitemap is emitted by a plugin. The nav links are written by hand. Four independent pieces of code each build "the URL for page X in language Y."

Each will be correct in isolation and they will still drift, because the interesting cases are the edges: trailing slash or not, does the default language live at / or /en/, is x-default a separate entity, does a page that only exists in some languages still emit alternates for all of them. Every one of those questions has to be answered identically in four places, and nothing checks that they were.

Client-side i18n makes it worse. If the language switcher swaps strings at runtime, then for a crawler every translation except the one baked into the HTML does not exist. You have one page pretending to be many.

What actually works: one exit point

The generator has exactly one URL constructor and one path constructor:

const ORIGIN = 'https://kibo.douhouse.com';

/** The single source of truth for every URL the site emits. */
const urlFor = (lang, slug) => `${ORIGIN}${pathFor(lang, slug)}`;

/** Root-relative form of the same URL, for links and assets inside a page:
 *  a page at /ja/ must not resolve ./styles.css against its own directory. */
const pathFor = (lang, slug) => {
  const base = lang === DEFAULT_LANG ? '/' : `/${lang}/`;
  return slug ? `${base}${slug}/` : base;
};

Twelve lines, and they answer every edge case once. Default language at root. Trailing slash always — chosen because that is the form the hosting platform serves with a 200 and redirects to; a canonical pointing at a URL that 301s is a self-inflicted wound.

Crucially, write() uses the same pathFor to decide where the file goes on disk:

const write = (lang, slug, html) => {
  const dir = path.join(DIST, pathFor(lang, slug));
  fs.mkdirSync(dir, { recursive: true });
  fs.writeFileSync(path.join(dir, 'index.html'), html);
};

So the URL structure and the directory structure cannot diverge either. There is no mapping layer to get wrong.

Head generation is also single-source

One head builder serves every page type. It takes langs — the set of languages this exact page exists in — and derives everything from it:

function headFor({ lang, slug, title, desc, langs, jsonld, noindex }) {
  const m = LANG_META[lang];
  const self = urlFor(lang, slug);
  const xdefault = langs.includes(DEFAULT_LANG) ? DEFAULT_LANG : langs[0];

  const alternates = langs.map(
    (l) => `  <link rel="alternate" hreflang="${LANG_META[l].html}" href="${urlFor(l, slug)}">`
  ).join('\n');

  return `  <meta charset="utf-8">
  ...
${noindex ? '  <meta name="robots" content="noindex,follow">\n' : ''}  <link rel="canonical" href="${self}">
${alternates}
  <link rel="alternate" hreflang="x-default" href="${urlFor(xdefault, slug)}">
  ...
  <meta property="og:url" content="${self}">`;
}

Three properties fall out for free:

Canonical and og:url are literally the same variable. They cannot disagree.
hreflang is mutually referencing by construction. Every language of a page renders alternates from the same langs array, so page A links B and B links A automatically. The most common hreflang error — non-reciprocal annotation — is unrepresentable.
A page that exists in only some languages emits alternates only for those. Because langs is computed per page: GUIDE_LANGS.filter((l) => GUIDES[l]?.[slug]).

The sitemap uses urlFor too, so it is the same string a fourth time:

const entry = (lang, slug, langs, priority) => {
  const alts = langs.map(
    (l) => `    <xhtml:link rel="alternate" hreflang="${LANG_META[l].html}" href="${urlFor(l, slug)}"/>`
  ).join('\n');
  const xd = langs.includes(DEFAULT_LANG) ? DEFAULT_LANG : langs[0];
  return `  <url>
    <loc>${urlFor(lang, slug)}</loc>
${alts}
    <xhtml:link rel="alternate" hreflang="x-default" href="${urlFor(xd, slug)}"/>

The thin-content gate

The second constraint is harder, because it is a rule about content, and content is written by humans in a hurry. A localized long-tail page that ends up as a heading plus two sentences is worse than not shipping it: it drags the whole directory's quality signal down.

So the build measures every page it just rendered and refuses to make short ones indexable.

function bodyWords(html, lang) {
  const main = html.match(/<main[^>]*>([\s\S]*?)<\/main>/i);
  if (!main) return 0;
  const text = main[1]
    .replace(/<(script|style)[\s\S]*?<\/\1>/gi, ' ')
    .replace(/<[^>]+>/g, ' ')
    .replace(/&[a-z]+;|&#\d+;/gi, ' ')
    .replace(/\s+/g, ' ')
    .trim();
  if (/^(ja|ko|zh)/.test(lang)) {
    const cjk = (text.match(/[぀-ヿ㐀-䶿一-鿿가-힯]/g) || []).length;
    return Math.round(cjk / 1.8);
  }
  return text ? text.split(' ').length : 0;
}

Two details matter here. It counts <main> only, so boilerplate nav, footer and CTA cannot inflate a thin page into passing. And CJK has no spaces — splitting on whitespace would count an entire Japanese page as one gigantic "word" — so codepoints are converted to a rough English-word equivalent instead.

The gate itself is not a single pass, and this is the part I did not expect when I started:

function runGate() {
  const all = [];
  for (const lang of GUIDE_LANGS) {
    for (const slug of GUIDE_SLUGS) if (GUIDES[lang]?.[slug]) all.push([lang, slug]);
  }

  let indexable = new Set(all.map(([l, s]) => `${l}/${s}`));
  let counts = new Map();

  for (let pass = 0; pass < 10; pass++) {
    const next = new Set();
    counts = new Map();
    for (const [lang, slug] of all) {
      const key = `${lang}/${slug}`;
      const n = bodyWords(guidePage(lang, slug, indexable, false), lang);
      counts.set(key, n);
      if (n >= MIN_WORDS) next.add(key);
    }
    const stable = next.size === indexable.size && [...next].every((k) => indexable.has(k));
    indexable = next;
    if (stable) break;
  }

  return { all, indexable, counts };
}

Delisting a page removes it from other pages' related-links lists. That shortens those pages. Which could, in principle, push a borderline page under the threshold. So the gate iterates to a fixpoint rather than deciding in one pass.

Delisting is not deletion. The page is still written, still reachable, still linked — it just gets noindex,follow and drops out of the sitemap. Related-link rendering degrades the entry to plain text rather than removing it:

return indexable.has(`${lang}/${s}`)
  ? `      <li><a href="${pathFor(lang, s)}">${t}</a></li>`
  : `      <li>${t}</li>`;

And the build reports what it did, so a regression is visible in the log rather than in Search Console eight weeks later:

words: shortest 512, longest 1104
gate: N pages checked, 0 blocked (threshold 450)

Costs and boundaries

The threshold is a proxy, not a measure of quality. Word count does not detect a padded page, a machine-translated page, or a page that says nothing at length. It only catches the specific failure of obviously too short. A page can clear 450 words and still deserve to be blocked; nothing here will tell you.

Rendering during the gate means pages are rendered more than once. runGate renders every page per pass, then the writer renders them again. For a site of this size that is milliseconds and completely irrelevant; if page generation ever became expensive, this loop would need memoizing per indexable set.

The fixpoint has an escape hatch that fails silently. pass < 10. If the set ever oscillates rather than converging — page A's links pushing B over the line while B's absence pushes A under — the loop exits after ten passes with whatever it happens to be holding. Nothing warns you. A console.warn on non-convergence is the obvious missing line.

bodyWords parses HTML with regex. It is correct for this generator because the generator itself emits the <main> tags, so the input is not arbitrary HTML. Point it at hand-authored markup with a nested <main> or a <main> mentioned in a comment and it will misbehave. This is acceptable only because producer and consumer are the same file.

The CJK divisor is a guess. cjk / 1.8 is a heuristic for "characters per equivalent English word." It is not derived from anything. It means the effective threshold is not the same across languages, which is a real inconsistency, just a small one.

Single origin is baked in. ORIGIN is a constant. Preview deployments on a different hostname emit production canonicals — which is arguably the safe failure (previews never compete for indexing) but it does mean the preview's own links are cross-origin absolute in the sitemap.

Zero dependencies is a real constraint, not a badge. No Markdown, no templating language, no image pipeline. Content lives as JavaScript objects and pages are built with template literals. That is genuinely fine at this scale and would be miserable at a hundred times the volume; the honest boundary is "small marketing site with a fixed set of page types."

This generator builds the Kibo site.

AVCaptureMultiCamSession: the real limits, and why the stock Camera app doesn't offer dual recording

TBDS — Sun, 09 Aug 2026 08:52:20 +0000

Modern iPhones can genuinely run the front and back cameras at the same time.
AVCaptureMultiCamSession has been in the SDK since iOS 13 and works on A12 and
later hardware. And yet Apple's own Camera app has no dual-recording mode, no
toggle, no hidden gesture.

That is not a product oversight you can route around. iOS has two separate
concepts: a normal AVCaptureSession runs exactly one camera and can switch
between them, and a multi-camera session runs several inputs concurrently. The
second is a different object with different rules, and almost everything you know
about the first one is wrong for it.

Here is what those rules actually cost, from a shipping dual-camera recorder.

The problem

You want one MP4 containing both cameras, composited live, with a preview that
matches the file. The pieces sound routine: two inputs, two outputs, a compositor,
an AVAssetWriter.

Why the obvious approach fails

`addInput` / `addOutput` simply fail

The convenience API implicitly forms connections for you. On a multi-cam session
that fails outright. You must add inputs and outputs without connections and wire
AVCaptureConnection objects yourself:

private func addVideoBranch(device: AVCaptureDevice,
                            output: AVCaptureVideoDataOutput,
                            mirrored: Bool) throws {
    let input = try AVCaptureDeviceInput(device: device)
    guard session.canAddInput(input) else { throw ConfigError.cannotAdd("\(device.position) input") }
    session.addInputWithNoConnections(input)

    output.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
    output.alwaysDiscardsLateVideoFrames = true
    guard session.canAddOutput(output) else { throw ConfigError.cannotAdd("\(device.position) output") }
    session.addOutputWithNoConnections(output)

    guard let port = input.ports(for: .video,
                                 sourceDeviceType: device.deviceType,
                                 sourceDevicePosition: device.position).first else {
        throw ConfigError.noPort("\(device.position)")
    }

    let conn = AVCaptureConnection(inputPorts: [port], output: output)
    guard session.canAddConnection(conn) else { throw ConfigError.cannotAdd("\(device.position) connection") }
    session.addConnection(conn)

    if conn.isVideoRotationAngleSupported(90) { conn.videoRotationAngle = 90 }
    if mirrored, conn.isVideoMirroringSupported {
        conn.automaticallyAdjustsVideoMirroring = false
        conn.isVideoMirrored = true
    }
}

`sessionPreset` does not work either

A multi-cam session only accepts .inputPriority. Resolution is chosen per device
by picking an activeFormat, only from formats where isMultiCamSupported == true
— and it must be chosen before addInput:

private func configureFormat(for device: AVCaptureDevice, targetWidth: Int32) throws {
    let multiCamFormats = device.formats.filter { $0.isMultiCamSupported }
    guard !multiCamFormats.isEmpty else { throw ConfigError.noFormat("\(device.position)") }

    func supports30fps(_ f: AVCaptureDevice.Format) -> Bool {
        f.videoSupportedFrameRateRanges.contains { $0.minFrameRate <= 30 && $0.maxFrameRate >= 30 }
    }
    func dims(_ f: AVCaptureDevice.Format) -> CMVideoDimensions {
        CMVideoFormatDescriptionGetDimensions(f.formatDescription)
    }

    let chosen = multiCamFormats.first(where: { dims($0).width == targetWidth && supports30fps($0) })
        ?? multiCamFormats.max(by: { a, b in
            Int(dims(a).width) * Int(dims(a).height) < Int(dims(b).width) * Int(dims(b).height)
        })!

    try device.lockForConfiguration()
    defer { device.unlockForConfiguration() }
    device.activeFormat = chosen
    let duration = CMTime(value: 1, timescale: 30)
    device.activeVideoMinFrameDuration = duration
    device.activeVideoMaxFrameDuration = duration
}

Note filter { $0.isMultiCamSupported }. The set of multi-cam formats is a strict
subset of what the device offers when it is the only camera running. This is the
first place the "why not just do what the Camera app does" answer starts to appear:
the Camera app is not choosing from the same menu.

The `hardwareCost` gate — and it cannot be pre-checked

This is the constraint that shapes the whole feature. A multi-cam configuration has
a hardwareCost and a systemPressureCost. If hardwareCost > 1.0 you get a
runtime error and the session dies. And the value only reflects the new
configuration after you commit it. There is no dry run.

So every configuration change has to be: change → commit → read cost → roll back if
over.

session.beginConfiguration()
do {
    try configureFormat(for: back,  targetWidth: r.targetWidth)
    try configureFormat(for: front, targetWidth: r.targetWidth)
} catch {
    session.commitConfiguration()
    errorMessage = L10n.Error.detail(Strings.resolutionFailed, error.localizedDescription)
    await startSession()
    return
}
session.commitConfiguration()

if session.hardwareCost > 1.0 {
    downgraded = true
    applied = previous
    session.beginConfiguration()
    try? configureFormat(for: back,  targetWidth: previous.targetWidth)
    try? configureFormat(for: front, targetWidth: previous.targetWidth)
    session.commitConfiguration()
}

Dual 1080p exceeds the budget easily on many devices. That rollback is not
defensive tidiness — remove it and you ship an app that dies when a user picks a
resolution their phone cannot sustain.

The seamless zoom is not available to you

The stock Camera app's continuous pinch from ultra-wide through wide to telephoto
is built on a virtual capture device that internally owns several physical lenses
and crossfades between them. A multi-cam session connects to specific physical
devices, because that is what the isMultiCamSupported format check operates on. A
virtual device that silently reaches across several lenses is not something that
check can reason about.

The consequence for the UI is that lens choice and zoom become two different
controls, and the lens picker may list fewer lenses than the stock app shows:

/// Running three back lenses at once under MultiCam is impossible and unnecessary —
/// probe only which lenses have a multicam format, then use the chosen one as the
/// single back input.
private func discoverBackLenses() {
    var found: [BackLens] = []
    for lens in BackLens.allCases {
        guard let device = Self.device(for: lens) else { continue }
        guard device.formats.contains(where: { $0.isMultiCamSupported }) else { continue }
        found.append(lens)
    }
    availableBackLenses = found
    if !found.contains(backLens) {
        backLens = found.contains(.wide) ? .wide : (found.first ?? .wide)
    }
}

Front camera zoom, incidentally, is 1.0 and stays 1.0: there is no optical zoom
there, so videoMaxZoomFactor is 1. If the front camera is your main frame,
pinch-to-zoom does nothing, and the honest move is to say so rather than let a
gesture look broken.

What actually works

One clock, one texture

Two camera streams do not arrive in lockstep. The design that avoids an entire
class of bugs is to treat one camera as the clock and hold only the latest frame
from the other, then render one texture that is used for both the preview and
the writer:

func captureOutput(_ output: AVCaptureOutput,
                   didOutput sampleBuffer: CMSampleBuffer,
                   from connection: AVCaptureConnection) {

    if output is AVCaptureAudioDataOutput {
        if recorder.isRecording { recorder.appendAudio(sampleBuffer) }
        return
    }

    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

    guard isBack(output) else {
        latestFront = pixelBuffer   // front camera: keep only the newest frame
        return
    }

    // the back camera is the clock
    guard let front = latestFront else { return }

    let pool: CVPixelBufferPool? = recorder.isRecording ? recorder.pixelBufferPool : nil
    if let composed = compositor.render(back: pixelBuffer, front: front, recordingPool: pool) {
        recorder.appendVideo(composed, pts: CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
    }
}

The composite is rendered directly into a buffer taken from the asset writer's
pixelBufferPool, and that texture is then sampled to the screen. Preview and
recording are structurally incapable of disagreeing. Splitting this into two
independent render paths as an "optimisation" reintroduces the worst bug class in
this domain: looks right on screen, wrong in the file.

All three outputs share one serial data queue and never touch the main actor.

Treat resilience as four named failure paths

Four things end a dual recording, and each one destroys the file if unhandled:

path	consequence if ignored
session interrupted (call, another app takes the camera)	`AVAssetWriter` stuck half-open, corrupt file
`mediaServicesWereReset`	reusing old inputs/outputs silently stops delivering frames — no error, the picture just freezes
thermal state reaches `.critical`	the system kills the session; the whole in-progress take is lost
storage fills	writer fails outright, entire recording lost

The mediaServicesWereReset one is the nastiest because there is no error at all —
you must rebuild the whole session. And all automatic stops must funnel through the
same finalize path the user's own stop button uses. A second teardown path is how
you get one code path that saves the file and another that doesn't.

The thermal and storage policies are pure functions, which is the only way they get
tested:

enum CapturePolicy {
    static let videoBitrate: Int = 10_000_000
    static let minBytesToStart: Int64 = 200 * 1024 * 1024
    static let minBytesToContinue: Int64 = 100 * 1024 * 1024

    static func thermalAction(level: ThermalLevel, isRecording: Bool, isHD1080: Bool) -> ThermalAction {
        switch level {
        case .critical:
            return isRecording ? .stopRecording : .none
        case .serious:
            if !isRecording, isHD1080 { return .dropTo720 }
            return .none
        case .nominal, .fair:
            return .none
        }
    }
}

The asymmetry is deliberate: at serious, drop to 720p only if you have not
started yet — starting a heavy take on an already-hot phone is how you get an
interrupted one — but if you are already recording, warn rather than yank the
resolution mid-file.

Focus coordinates, the silent inversion

focusPointOfInterest is expressed in the unrotated landscape sensor frame. It
does not follow videoRotationAngle or isVideoMirrored. Converting from a
normalised portrait canvas point:

back camera (rotation 90): (u, v) = (y, 1 - x)
front camera (rotation 90 + mirrored): (u, v) = (y, x)

Get it backwards and nothing crashes and nothing logs. Tapping the top-left just
focuses the bottom-right. This is why that conversion is a pure function with
dozens of tests — and those tests were mutation-verified: swapping the front/back
formulas makes them fail immediately.

There is still a hole worth being honest about. The tests lock the contract of the
pure function, and the contract assumes both connections set
videoRotationAngle = 90 with the front additionally mirrored. Change those two
lines in the session and every test stays green while focus is inverted on a real
phone.

Costs and boundaries

Nothing camera-related can be verified in the simulator. AVCaptureMultiCamSession.isMultiCamSupported is permanently false there, so the app correctly shows an unsupported-device screen. Every capture change needs a real A12-or-later device. (Do not hardcode a model list — that system flag is the only authority. Apple documents A12+, which is where "iPhone XS/XR or newer" comes from, but the flag is what you branch on.)
Heat and battery are materially worse than single-camera recording. Two sensors, two ISP paths, a Metal composite per frame and an encoder. Users notice.
Layout crops. Side-by-side portrait means fitting a 720×1280 source into a 540×1920 column — 25% is lost from each side. That is arithmetic, not a bug, but it must be designed around.
Some things get frozen at record time. Muting is implemented by not creating an audio track at start(), which is more robust than writing a silent track, so it cannot be toggled mid-recording. Aspect ratio likewise cannot change during a take without destroying the writer.
Metal has its own traps here. half is a builtin type name and cannot be a variable (the compiler error is wildly misleading); multiple draw calls must use setVertexBytes/setFragmentBytes rather than sharing one MTLBuffer at offset 0; and CVMetalTexture objects must be kept alive until the command buffer completes.
iOS 17+ blocks the easy device screenshot path, so visual verification of a camera app is manual. Plan for it.

This is the capture stack behind DualCam, an iOS app that records both cameras into
a single composited file.

The three hard constraints of an iOS keyboard extension

TBDS — Sun, 09 Aug 2026 08:50:33 +0000

A custom keyboard on iOS looks like a small UIKit app. It is not. It is a
UIInputViewController living inside an extension process with rules that no
normal app has to obey, and the three that actually shape the code are:

a memory ceiling in the tens of megabytes, enforced by silent termination;
a sandbox that is off by default (Allow Full Access), which removes networking, location, the system pasteboard — and, less obviously, a reliably writable App Group container;
a feedback path (sound + haptics) that must be produced without an audio session, inside the same tight budget, on the keystroke path.

Everything below is from a shipping keyboard extension. The numbers are device
measurements, not estimates, and several of them are numbers that proved an
earlier hypothesis wrong.

Constraint 1: the memory ceiling, and why you cannot debug it normally

The ceiling for a keyboard extension is somewhere around 60MB of phys_footprint.
What makes it hard is not the size — it is the failure mode. When you cross it,
jetsam kills the extension: no crash log, no signal, no exception. iOS just
switches the user back to the previously used keyboard. From the user's side, "the
keyboard quit by itself, mid-sentence". From your side, there is nothing to read
afterwards.

Worse, it depends on how much memory the rest of the phone is using, so it is
intermittent and cannot be reproduced on demand.

Why the obvious approach fails

The obvious approach is to guess. You look at the app, decide the biggest thing is
probably the dictionary, shrink it, ship, and ask the person holding the phone.
That loop is a day long and it was wrong repeatedly — first the word index was
blamed, then the pinyin table, while the device sat at 55MB with a peak of 68.

The second obvious approach — write diagnostics to a file in the App Group — fails
for a reason specific to extensions: writes to a shared container from a keyboard
extension are unreliable and fail silently without Full Access. The log would be
missing exactly when it mattered. Unified logging survives the process being
killed, so it is the only thing that reports the last measurement before a kill:

enum MemoryProbe {
    private static let log = Logger(subsystem: "com.douchaojun.kibo", category: "mem")

    /// Resident footprint in megabytes — the same number jetsam measures
    /// (`phys_footprint`: dirty plus compressed pages).
    static func footprintMB() -> Double {
        var info = task_vm_info_data_t()
        var count = mach_msg_type_number_t(MemoryLayout<task_vm_info_data_t>.size
                                           / MemoryLayout<natural_t>.size)
        let result = withUnsafeMutablePointer(to: &info) {
            $0.withMemoryRebound(to: integer_t.self, capacity: Int(count)) {
                task_info(mach_task_self_, task_flavor_t(TASK_VM_INFO), $0, &count)
            }
        }
        guard result == KERN_SUCCESS else { return 0 }
        return Double(info.phys_footprint) / 1024 / 1024
    }
}

That number alone is not enough. A fresh keyboard read 63MB on device while a
cold process read 10MB, memory warnings fired, every cache was dropped, and the
number did not move. Two very different explanations fit that: the process really
holds 63MB of live objects, or it holds a little inside a lot of pages malloc took
from the kernel and never returned. Those have opposite fixes — find what is
retained, versus stop making the large transient allocations that grew the heap. So
live allocations get measured separately, by summing every malloc zone's
size_in_use via malloc_get_all_zones + malloc_zone_statistics, and compared
against the footprint.

What actually works

Two structural findings came out of instrumenting instead of guessing.

Dirty heap is the thing being charged; mapped files are nearly free. The word
list was a Swift string literal parsed into arrays and dictionaries on the first
keystroke — in front of the user, and every byte of it dirty heap. Replacing it
with a memory-mapped binary file, sorted by word so a prefix is a contiguous range,
moved the pages to clean, file-backed memory the kernel can drop and re-read:

init?(url: URL) {
    guard let mapped = try? Data(contentsOf: url, options: [.mappedIfSafe]) else { return nil }
    guard mapped.count >= Self.headerSize else { return nil }
    self.data = mapped
    guard Array(mapped[0..<4]) == Self.magic else { return nil }
    // ... header offsets validated once here, so the per-query path can read
    // without re-checking: a bounds test on the hot path would cost more than
    // the search itself.
}

Note the init?: a truncated or foreign file must degrade to "no suggestions",
never to a crashed keyboard. Baseline went from 52MB to 27MB.

UIInputViewController never releases its view. This is the one that cost
days. An empty subclass with an empty view survives every cycle in a leak harness
while a plain UIViewController survives none. A device probe reported exactly one
live controller, which seemed to exonerate the controller — but both facts are true
at once: iOS builds a new controller for each host app the keyboard appears in,
releases the old one (hence the count of one), and keeps every one of their views.
Checkpoints inside viewDidLoad, which only run for a fresh controller, read
70MB. A brand-new keyboard was being built on top of everything its
predecessors left behind, and no cache purge touches it, because a retained view is
not a cache. Fifty key buttons with a gradient, a glow layer and a rasterised
shadow each is about 3MB — exactly the per-cycle delta the device reported.

The fix is to empty the view in deinit, deferred one main-queue turn rather than
done inside dealloc. Doing the same teardown in viewWillDisappear is wrong: the
same controller is hidden and re-shown many times, so that version pays a rebuild
every time and feeds UIKit's own unpurgeable caches.

Failure modes worth naming

Threads. The probe's mark() appended to a static Swift array from the main thread and from a background composer warm-up. Two threads mutating one array corrupts the buffer and the process dies later, elsewhere, with no exception — occasional, never in the same place. Everything mutable sits behind an NSLock now. A lock costs nothing at a few dozen marks per session.
Threads, again. A warm-up dispatched to DispatchQueue.global() on every language switch: the global queue answers blocked tasks by spawning threads, the tables are static let so all but the first caller block on one initialiser, and half a megabyte of stack each against a ~60MB ceiling adds up fast. One serial queue, each language warmed at most once.
Control groups. A leak test for a keyboard must cycle one controller and include an empty-UIInputViewController baseline. Without the control the numbers mean nothing — an earlier probe set vc.view.frame in every stage including the baseline, which itself loads the view and runs viewDidLoad.

Constraint 2: Full Access is off, and that is the correct default

iOS shows a genuinely frightening warning when you enable Full Access, and most
users decline. Design for declined.

With it off, an extension has no network, no location, and no system pasteboard.
The keyboard's copy/paste buttons therefore cannot silently do nothing:

/// The system pasteboard is only reachable with Full Access. If it's off we
/// say so rather than doing nothing.
private func requireFullAccessForClipboard() -> Bool {
    if hasFullAccess { return true }
    showToast("Turn on Full Access for copy & paste")
    return false
}

The non-obvious cost is storage. An extension is not guaranteed a writable App
Group container without Full Access. This produced a bug that looked nothing like
a permissions problem: switch language, dismiss the keyboard, come back — back to
English. The extension's write went nowhere, silently. The fix is to mirror the
choice into the writer's own defaults, stamp both records, and let the newer one
win, so a change made in the container app still beats a stale mirror in the
extension and vice versa:

static var activeLanguage: KeyboardLanguage {
    get {
        let enabled = languageIDs
        let shared = record(from: defaults)
        let local  = record(from: localDefaults)
        let candidates = (shared.stamp >= local.stamp ? [shared, local] : [local, shared])
        for candidate in candidates where enabled.contains(candidate.id) {
            return KeyboardLanguages.language(id: candidate.id)
        }
        return KeyboardLanguages.language(id: enabled[0])
    }
    set {
        let stamp = Date().timeIntervalSince1970
        for store in [defaults, localDefaults] {
            store.set(newValue.id, forKey: Key.activeLanguageID)
            store.set(stamp, forKey: Key.activeLanguageStamp)
        }
    }
}

A feature that genuinely needs the permission (location, for instance) should
explain itself rather than sit dead. A locked control that hands back nothing is
worse than a control that says why.

Constraint 3: sound and haptics without an audio session

You want a key click. The obvious implementation is AVAudioPlayer, because it has
a volume property. That is the wrong lever, and the bug it produces is invisible
from inside a keyboard: AVAudioPlayer and AudioServicesPlaySystemSound go to
different audio buses. System sounds follow the ringer; AVAudioPlayer follows
media playback. With media volume low — common — every volume below maximum is
inaudible while 100% works. It also needs an audio session, so it is silent again
whenever activation fails, and it keeps decoded buffers alive in a process with a
hard ceiling.

AudioServicesPlaySystemSound needs no audio session and no Full Access, and it
respects the silent switch like Apple's own click. It also has no volume control at
all. So volume is implemented by scaling the sample's PCM once, writing it to a
temp file, and registering that as another system sound — same bus, same ringer
volume, no session, no decoded buffers:

private static func writeScaledCopy(of url: URL, gain: Float, named name: String) -> URL? {
    let destination = FileManager.default.temporaryDirectory
        .appendingPathComponent("kibo-\(name).wav")
    if FileManager.default.fileExists(atPath: destination.path) {
        try? FileManager.default.removeItem(at: destination)
    }
    guard let input = try? AVAudioFile(forReading: url) else { return nil }
    let format = input.processingFormat
    let frames = AVAudioFrameCount(input.length)
    guard frames > 0,
          let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: frames),
          (try? input.read(into: buffer)) != nil,
          let channels = buffer.floatChannelData else { return nil }

    for channel in 0..<Int(format.channelCount) {
        let samples = channels[channel]
        for frame in 0..<Int(buffer.frameLength) { samples[frame] *= gain }
    }

    guard let output = try? AVAudioFile(forWriting: destination,
                                        settings: input.fileFormat.settings) else { return nil }
    guard (try? output.write(from: buffer)) != nil else { return nil }
    return destination
}

Constraint 1 comes straight back here. Volume is rounded into four buckets so a
slider drag cannot generate a hundred temp files, only the style currently in use
is kept registered, and AudioServices sound ids are a system resource that must
be disposed explicitly — dropping the dictionary entry alone leaks them:

if loadedStyle != nil {
    for (_, list) in ids { for id in list { AudioServicesDisposeSystemSoundID(id) } }
    ids = [:]; urls = [:]; purge()
}

Every failure path in that code falls back to the sample at full volume, on
purpose: a click at the wrong volume is a far smaller bug than silence, which is
what the previous implementation degraded to.

Haptics are cheap by comparison — UIImpactFeedbackGenerator needs no permission —
but they belong on the same path as the sound and the visual press effect. When
the suggestion bar didn't call the shared feedback function, chips read as taps that
failed to register, because they're reached by the same finger in the same gesture.

The latency budget

One frame is 16.7ms and the whole suggestion pipeline lives inside it.
Data.copyBytes on a memory-mapped Data is not a pointer read — it goes
through Data's range and bridging machinery, and reading four fields per entry
that way cost more than the binary search over them. One withUnsafeBytes +
loadUnaligned halved the worst case: a long misspelling with no completion, which
runs the full correction scan, went 15.81ms → 11.5ms; a short prefix went 0.31ms →
0.12ms. Assert p95 per keystroke against 16.7ms, not total elapsed over 1000
queries — a total hides where the time went.

Costs and boundaries

You will not get a crash report for the failure that matters most. Budget for building instrumentation you can read off a device, and for the fact that the overlay must be compiled out of release builds — a persisted debug flag will otherwise draw debug text over a reviewer's keyboard.
Nothing here reproduces in the simulator's memory behaviour, and unit tests only help if you write them as budget guards: an index costing more than a set number of megabytes, or five language switches that accumulate, should fail the build.
The Full Access default cascades into storage, not just features. Assume any shared-container write can silently vanish and design a merge.
The audio bus distinction has no compiler help and no runtime error. It is only visible on a device with the media volume turned down.

These constraints and their fixes come from Kibo, a custom iPhone keyboard whose UI
is English-only.

DEV Community: TBDS

Filling a region by writing four bytes: region-id PNGs and an SkSL lookup shader

Why the obvious approaches fail

The representation

The shader

Hit testing, with intent correction

Costs and boundaries

Chalk that looks like chalk: depositing particles instead of stroking paths

Why stroking a path fails

The baker

Deposition happens in the fragment shader

Store points, not pixels

The failure modes — the part worth your time

On-device speech recognition on iOS: the honest boundaries

The problem

Why the obvious approach fails

"Just set requiresOnDeviceRecognition = true and we're private"

"Just hand the whole file to the recognizer"

"The completion handler will tell me when it's done"

What actually works

1. Probe capability, but treat it as a hint

2. Chunk the audio, with overlap

3. Attempt on-device first, fall back explicitly

4. Own your timeout

5. Recording is a separate reliability problem

Costs and boundaries

One function that makes canonical, hreflang and sitemap unable to disagree

Why the usual setup fails

What actually works: one exit point

Head generation is also single-source

The thin-content gate

Costs and boundaries

AVCaptureMultiCamSession: the real limits, and why the stock Camera app doesn't offer dual recording

The problem

Why the obvious approach fails

addInput / addOutput simply fail

sessionPreset does not work either

The hardwareCost gate — and it cannot be pre-checked

The seamless zoom is not available to you

What actually works

One clock, one texture

Treat resilience as four named failure paths

Focus coordinates, the silent inversion

Costs and boundaries

The three hard constraints of an iOS keyboard extension

Constraint 1: the memory ceiling, and why you cannot debug it normally

Why the obvious approach fails

What actually works

Failure modes worth naming

Constraint 2: Full Access is off, and that is the correct default

Constraint 3: sound and haptics without an audio session

The latency budget

Costs and boundaries

"Just set `requiresOnDeviceRecognition = true` and we're private"

`addInput` / `addOutput` simply fail

`sessionPreset` does not work either

The `hardwareCost` gate — and it cannot be pre-checked