DEV Community: prodbymarcu

Your camera's wrong password error can be three defaults deep

prodbymarcu — Fri, 23 Oct 2026 23:11:49 +0000

Two Tapo C210 cameras sit on my home network, static addresses, DNS pointed at my own AdGuard box. On 2026-10-18 the auth flow rejected a password I knew was right, my probes pushed the camera into a lockout, and the error read "Temporary Suspension: try again in ~1800 s". A camera I own, on my own network, telling me to come back in half an hour. The password was right. Three defaults sat between me and that camera, each one failing in a different way, and a fourth layer under them took every service on the box down for 70 minutes the same day.

Layer 1, a toggle that resets on re-bind

Tapo's app hides a per-camera switch called Third-Party Compatibility. Mine was off. With it off, the camera's control API rejects every credential, the correct password included. The setting also resets on a re-bind, so a camera that talked to Home Assistant yesterday can stop talking today after nothing but a re-pair in the vendor app. I flipped it on around 20:00 and both cameras accepted auth on the first try. An auth error never mentions this layer, it just blames the password.

Layer 2, a username the integration guessed for me

Home Assistant's Tapo integration ships cloud setup steps that hardcode the username as "admin". On these cameras the control username is the TP-Link account email, so every "admin" attempt fails no matter what password sits behind it. The winning path was three details: auth with the email, skip_rtsp true, because the RTSP pre-test breaks on the raw @ in the URL, and an empty {} for the cloud password field, skipping it entirely.

The camera's own "camera account" credentials only open RTSP. The control endpoint and ONVIF both reject that account, "Invalid authentication data" from one, "Authority failure" from the other. Every app feature, the pan and tilt, presets, the privacy shutter, the siren, needs the full account password, and the SPAKE2+ handshake behind it computes everything locally without touching a TP-Link server. I typed that password into the integration form myself, my agent never saw it, and I made one attempt, because the camera suspends you again for another ~1800 seconds.

Layer 3, rules that report success and never engage

My AdGuard user_rules section had python-source garbage sitting in it, leftover lines from an earlier quoting crash. AdGuard's behavior here is the nasty part: bad lines make it silently discard the entire section. Every block rule I added reported a clean insert and none of them ever engaged. The fix is unglamorous: 65 valid rules kept, my allowlists untouched, container restarted, then proof. The TP-Link cloud domains the querylog showed, stun.tplinkcloud.com and use1-device-cloudgateway.iot.i.tplinknbu.com among them, now resolve to ::. Five domains, tplinkcloud.com, tplinknbu.com, tplinkra.com, tp-link.com, tapo.com, blocked for the two cameras only, NTP untouched. The cameras went DNS-quiet. The camera's own firmware switch in Home Assistant still reports "on", the update call can't reach anything, the network layer enforces what the UI claims.

What shipped

PTZ proof first: a jog right moved 54% of the frame, a jog back returned it. 45 entity references validated against the live registry, because the earlier warning-sign cards were unvalidated IDs. Four automations: arm on leave, disarm on home, alert while armed with a snapshot push, disarm on HA start when I'm home, and the test push landed on my phone. The first armed recording is a 1.7 MB 30-second clip at 720p, retention 30 days, auto-pruned. Recordings land on the same box, and I mounted them over SMB, browsed them, unmounted, clean. HA went from 2026.6.0 to 2026.10.2 with a backup taken first, every integration survived.

The theme migration cost 4 recovery-mode boots. 2026.10 stopped auto-loading the themes directory, themes must live inline in the config, and YAML comments ate hex colors, empty mode dicts needed explicit braces, and one ghost line survived my regex surgery. Four boots to learn one rule: read the file tail before restarting.

The layer nobody counts

Capacity. The box runs 7 GB total RAM, with ~1.7 GB of that already in swap even at idle. Frigate had 2K recording hot on both cameras, the record ffmpeg stays hot even when disarmed, plus a CPU detector, plus 20 containers. Swap thrash, load average 182, every service dead from 13:10 to 14:20, smart home dark, 70 minutes. docker stop frigate, and the load fell from 182 to 4 in about 4 minutes, HA back. Reconfig: one 720p stream per camera doing both detect and record, detection 5 fps to 3 fps, a 3 GB memory cap on the container. Post-fix the box sits at load 4.4, both cameras capture at 3 fps, detection live, 20/20 automations on.

The decision I keep from it: nothing heavy lands on that box without a free-RAM check first. The clips stay 720p because of the 7 GB constraint, and alerts use 2K snapshots from the camera entities, so the resolution loss lives in recordings I scrub through, never in the push I judge.

The part I keep

The next camera that tells me "wrong password" gets a vendor-app check before the login prompt. So what's the worst default that ate one of your evenings? Mine cost an evening and 70 minutes of a dark house, and the fix was one switch. Drop yours in the comments, I collect these.

A 31M TTS model ran 2.6x faster than the 2.37B incumbent

prodbymarcu — Thu, 22 Oct 2026 22:32:43 +0000

Model size is the least predictive line on a local text-to-speech spec sheet, and the two numbers that decide whether a model belongs on your machine are wall time on the same input and the word error rate of a speech-to-text round trip. I swapped the TTS lane on my Mac mini on those two measurements on 2026-10-20, a 31.2M-parameter model took over from a 2.37B one, the smaller thing ran 2.6x faster on the wall clock, and prose came back at 0.0% WER from Whisper.

What the two lanes are

The old lane was a Kyutai Pocket TTS shim on :8310, torch runtime, ReLU-2.37B weights, 283 MB of cache. The challenger is gradium/phonon-31m-en, 31.2M parameters, 64.5 MB at q8, Rust runtime with no torch anywhere in the loop. I fed both shims the same 129-character line on the same box: the incumbent took 0.96s of wall time to produce 14.08s of audio, and its WAV came back compressed so Python's wave module rejected it with unknown format: 3 before I could count a single frame. Phonon took 0.37s to produce 6.64s of audio with 76% voiced frames, 2.6x faster on the wall clock, roughly 2x tighter output, from weights 76 times smaller.

The swap moved the local lane and left my calls alone. The old shim's only caller was the legacy chained loop, speech-to-text into a model into synthesis, while the live call lane runs a full-duplex cloud voice, so this was never a question of whether the calls got faster. It was a question of what answers the text box.

What the model is

Phonon S 31M is Apache-2.0, the 40M and multilingual builds sit behind CC BY-NC and gating, so the license split matters the moment the audio goes anywhere commercial. The ptts 0.4.0 wheel ships the xn engine and the Mimi decoder as one Python extension, the core is Rust, output is 24 kHz mono 16-bit. Same codec family the Pocket TTS lane was already built on, which is why the size difference lives in the weights and the runtime, the transport is shared.

The numbers on the box

All of it CPU-only on a 16 GB Mac mini at q8: RTF 0.027 to 0.036 steady, 4.4s of audio in 0.16s of wall time, streaming first chunk at 0.20s with 9 chunks covering those 4.4s, fresh-process load 0.17 to 0.19s warm against 4.5s cold, time to first chunk 8 to 23 ms, full CLI call 1.07 to 1.46s. I re-ran the battery today, 2026-10-22, RTF 0.034 and 13 ms to first chunk, same box, nothing tuned between runs.

The check that makes it trustworthy

Synthesized audio is only as good as the transcript that comes back out of it, so every sample went through whisper-large-v3-turbo and got scored. Plain prose: 0.0% WER on Easton, Miles and Preston, verbatim. A line of spoken numbers: 15%, and the misses were normalization, a spoken figure against digits. A line of shell: 50%, launchctl came out launchtel, gui/1000 came out gkg slash 1000.

That last line is the guard. A small model is trustworthy for the domain you tested, and code plus paths are the domain that fails, so the rule shipped with the lane is don't feed it code, or run --lang none when you have to. The WAVs themselves are honest, 68% to 87% voiced frames, max amplitude 14k to 18k, real speech in every one of them.

Where the smaller model loses

What doesn't carry over is cloned voices. The old shim's voices were derived from audio prompts through get_state_for_audio_prompt, Phonon 0.4.0 has no audio-prompt path, add_voice wants .safetensors from Gradium Studio, the HF checkpoints ship no speaker encoder, cloning needs scripts/clone-voice.py against a sample recording. Day-one capability is the 37 English voices in the catalog, 13 female, 12 male, 12 characters, out of 102 total. Speed wins there and parity doesn't come free, it waits on a re-clone from a sample recording.

Three traps ate the week

The streamed WAV from the vendor server claims a data chunk length of 24932 against a file that's larger, strict parsers reject it outright, mp3 or pcm survive, pcm being headerless 16-bit mono at 24 kHz. hermes config set refused the tts.providers.* dot-paths, suggesting providers.phonon.type, and --force wrote them correctly, verified with yaml.safe_load. The pinned streaming provider pointed at an endpoint that had died, curl returned 000 against it, and resolve_streaming_provider never swaps providers to get streaming, so the pin kept hammering the corpse on every call until I removed it, then the dispatcher speaks per-sentence through the sync path.

Before wiring any of that I ran the vendor's stock server, a docker run of ptts-openai-server, and /health returned {"status":"ok"}, /v1/models named gradium/phonon-31m-en, /v1/audio/voices listed the 37 English voices, and a POST to /v1/audio/speech with voice Easton returned 318,764 bytes in 0.67s of wall time at 0.0% WER. That server speaks the exact API the old shim served, so the shim's eventual shape is near zero code.

What shipped

tts.provider: phonon on all three of my configs, the worker a CLI honoring the command-provider contract, text arrives in a temp file, the output extension wins, nonzero exit plus stderr is the failure detail. Default voice Easton, male American, punchy with a crisp attack. Unknown voice returns the clean OpenAI error shape, unknown voice 'bender'; this server has Alfie, ..., so the alias table lives in one place.

Two gates carried the decision, 2.6x on the wall clock and 0.0% WER on prose, and both of them are free to measure before you commit a lane. What's your bar for swapping a local TTS model, and does anyone run the round-trip check before committing?

A 100% progress bar on a 250 GB download proves nothing

prodbymarcu — Thu, 22 Oct 2026 10:38:01 +0000

I pulled unsloth/DeepSeek-V4.1-Flash-1017-GGUF in UD-IQ1_M onto my 3090 box on 2026-10-20, five shards, 250.404 GB, and the tool printed Fetching 5 files: 100%|██████████| 5/5 [58:59<00:00] plus the word Downloaded. That bar is a claim about the transport, it says the bytes the tool asked for came back. It doesn't claim the payload is the payload I wanted, and on a five-shard split the payload is where the interesting failures hide. The check that means something is a diff against the hosting API's file tree, I ran one, five sizes against five sizes, zero mismatches, every shard exact to the byte.

The five sizes, in bytes: 5,780,923, then 49,707,332,128, 49,292,777,280, 40,803,165,856, and 110,595,290,592. Elapsed 58:59, throughput 61 to 68 MB/s from a du delta of 12.7 GB over 180 seconds, my own measurement, the tool's estimate never got used. The downloader had also recorded per-file sha256 in its cache metadata, xet block hashes, sitting there before I did anything. I knew the hashes existed and still ran the diff, because the diff is three lines of Python and the hashes are a format I'd have to trust.

What the shards actually contain

The first shard is 5.78 MB and it has zero tensors in it. I read it with the GGUF reader from the llama.cpp tree and it printed metadata only, the chat template, the tokenizer, keys in a deepseek41.* namespace, draft-model keys. The 110.6 GB at the end of the split is the engram table, IQ4_NL, designed to be mmap'd and paged on demand through --lazy-mode. So the file is 250 GB, and 110.6 GB of it is allowed to live on a disk that pages it. The other 139.8 GB has to be addressable, and that number is the one nobody's progress bar shows.

The gate nobody's bar shows

The vendor publishes a minimum-RAM table for the family with the engram excluded, IQ1_S at 122.4 GB, IQ1_M at 139.8, IQ2_XXS at 165.3, Q2_K_XL at 199.6, and it climbs to 305.6 at Q8_K_XL. My box is 62 GB of RAM plus 24 GB per card on two RTX 3090s, 110 GB addressable. IQ1_M wants 139.8 GB resident, IQ1_S wants 122.4, the box was short by 12 to 30 GB across the two quants. The 125 GB of swap exists, 62.7 GB zram plus disk, and swap-backed expert weights crawl, so the run never got attempted. A 250 GB mmap load on that box would have thrashed the inference lane that was answering on :8080, the one my agents were routing through at that hour.

Placement was a paper decision before the download started. The box has a 2.7 TB spinning HDD mounted cold, ROTA 1, and a spinner makes every mmap page pay a seek, so the engram table belongs on the SSD filesystem, ROTA 0. After the 250 GB landed, that filesystem read 930 GB total with 212 GB free, 78% used. The model fits on the disk and the weights don't fit in the box.

That split is the thesis, disk holds the file, the box cannot hold the weights.

Why the quant took the bigger number

The quant choice came from published agreement, IQ1_S at 82.8% and IQ1_M at 88.2% top-1 on a wiki1100 sample, 5.4 points of agreement on top of 17.4 GB. I took IQ1_M anyway, because the gap between the two quants sits inside the shortfall, both miss the addressable pool, and the bigger one is what you want once the RAM arrives. The family is a 552 B backbone plus 196 B engram parameters at 1M context, arch id 20, and the repo also ships a dspark draft at 7.98 GB in Q8_0 for speculative decoding. That draft waits on the same purchase.

The runner was already there

Twenty minutes of grep decided the build question. llama-server 0.6.0-dev on the box already has the arch, LLM_ARCH_DEEPSEEK4 in src/llama-arch.h, a model struct, and a special-case at llama-model.cpp:387, plus --lazy-mode and --load-mode in the CLI. The other engine on that box is single-arch, its build file lists one arch id, so this model can't run there. No new build was needed. The plan for the next sitting is a llama-swap member pointing at shard1 with lazy mode on, temp 1.0, top-p 1.0, jinja, reasoning effort 75, and it waits on one order, 128 to 192 GB of RAM, because the shortfall lives in the addressable pool.

Rules the night produced

A progress bar reports the transport, a manifest diff reports the payload, and the gap between those two claims is one API call. File size reports where the model lives on disk, the resident table reports where it has to be addressable, and on a split-architecture model those two numbers sit 110.6 GB apart inside the same file. Rotation values belong in the placement decision before the load, ROTA 0 for anything you intend to mmap. And the engine question is a grep before it's a rebuild, an arch id in a header file beats a compile that was never necessary.

The shape of this work as it stands: the download verified, the placement verified, the engine verified, and the run blocked with a number attached, 29.8 GB short at IQ1_M, and a purchase on the way.

Where the doubt belongs

So what would you have to see before you called a 250 GB pull verified? The two places I put doubt are the gap between the bar and the manifest, and the gap between the file and the resident table, both of them questions you can ask today for free. Drop the number your pipeline checks last and I'll tell you what I'd have checked before I believed it.

Three layers said healthy while the engine slept 104 minutes

prodbymarcu — Wed, 21 Oct 2026 10:36:18 +0000

I run two RTX 3090s in the house, one local inference lane answering on :8080 and a second on :8081, with an agent that routes between them so a bad night costs me zero API calls. That setup died on the morning of 2026-10-20 and every status layer above it stayed green. The first signal was a greeting I sent at 07:10, answered by nothing.

systemctl said active. The health endpoint said loaded=true. The router answered with the model identity it was configured to serve. All three statements were about the request that carried them, none was about whether any work happened. A stack that survives on those three checks is a stack that can be asleep and still look alive, and that's why an autonomous rig needs a guard that never asks the thing it's supposed to protect.

What actually happened

The journal had it. The primary lane stopped and got masked at 00:44, the second at 02:23, and a rendering job logged the deliberate GPU reclamation plus the assumption that my agent would quietly fall back to a cloud model. My config had both fallback entries pointed at those same two lanes, so there was no route left to save the request. The greeting at 07:15 landed on a dead chain, the first reply completed at 07:19:47 on the lane I'd just repaired.

A fallback is a pointer to a lane, and the lane is the thing that went down. Fallback config that points at the failure is decoration, it gets exercised by the outage it can't survive.

The guard

The fix that held was one Python file plus a systemd service and a timer, checking the backend health endpoint and the router's model identity every 60 seconds, two failed observations before it starts a repair, a 240 second cooldown so a cold model load can't loop. It restores from saved copies of the production units. No model, no gateway, no cloud call anywhere in the loop, it runs on the box with the electrons that were already there.

I tested it against the exact failure class. I stopped the primary services and replaced the unit with a persistent /dev/null mask, then watched. The watchdog restored the saved unit, started the services, both lanes printed RECOVERED, and a real request aimed at the deliberately dead primary came back through the fallback lane at :8081, which proved the text chain had no cloud route to leak into. The production units finished with NRestarts=0. A repaired lane that's quietly crash-looping wears the same green badge, so I read that counter after every repair.

That test passed, so the guard was worth keeping. It was also incomplete, and the gap cost me the rest of that day.

The failure the guard didn't have

At 11:07 three systemd drop-in files appeared, one per unit, each of them replacing ExecStart with /usr/bin/sleep 86400. A unit running sleep is a unit that starts, holds its state as active forever, and does no work at all. systemctl stayed green, no socket bound, the health endpoint went 404.

The watchdog was right about the endpoint and wrong about what to touch. It restored the masked base unit, and the base unit still had the drop-in sitting on top of it, so every restart brought back the sleep. Same failure, every tick, and the timer kept reporting healthy lanes once the counts reset. From the 11:07 symptom to the repair starting at 12:51 is 104 minutes of a rig I thought was running.

The repair archived all three sleep drop-ins, restored the ExecStart values that run the engine and the router, and widened the watchdog so it archives only the exact known sleep-86400 and Restart=no override contents, which keeps genuine tuning like a RestartSec line intact. I ran that against an isolated test drop-in directory before pointing it at the live units, one override got archived, the tuning survived untouched, both failure counts went back to zero, both lanes printed STRATA_WORKING, and the request I'd sent at 12:46 came back with three completions at 12:52:27, 12:52:49 and 12:53:01 plus the tool calls around them.

The rules that came out of it

A shutdown path needs a gate it reads before it acts. Two separate jobs on that box were reclaiming GPUs on schedules nobody checked against each other, so I put an explicit maintenance marker at the entry of each one, absent marker means return without running a stop command, and I regression-checked the absent-marker branch with a direct call, because a review never runs the branch it reads. The watchdog gets the same marker, a file that says suspend recovery while a maintenance window is open.

Guards get audited for what they can see. Mine could see masks, it couldn't see overrides, and the difference between those two states is one listing of a drop-in directory. Every guard you write needs a second pass over its own blindness, the first pass only proves the guard reacts to the failure you already knew about.

Boundaries stay in the record. A 200 from a health endpoint proves the endpoint answered, it doesn't prove a long inference finishes, and a watchdog can't repair a box that's powered off or a host it isn't running on. Later that evening I moved both lanes to 524288 tokens with yarn rope scaling and restarted only the router that owns the second engine once its slots were idle, both slots reported n_ctx=524288 and a tool-call probe echoed tokens=524288, which verifies the limits loaded and says nothing about a 524K prompt surviving.

What I'd want in my own stack

A health check that asks for a probe with a real answer in it, model identity plus one completion, before a lane counts as up. A recovery loop that reads overrides and drop-ins on top of the base unit. A shutdown that requires an explicit marker. And a log line for each repair with the state that was archived, so the second failure in a day gets found by comparison, and a day full of silent repairs reads back as a list of states I can diff.

So, what does your stack say while it's doing nothing? The question worth answering is which layer in your chain would notice first, and how long you'd wait before a human did. Drop the layer you'd most like to catch your rig asleep and I'll tell you what I'd check before I believed it.

Your smart bulb ate the color and lied about it

prodbymarcu — Tue, 20 Oct 2026 22:15:42 +0000

Two automations in my house went dead on the same night, one on the lamps, one on the TV light trigger, and every layer between me and the fixture answered success. The bulb took a color write, answered 200, then read back [255,163,87], the warm white it had been sitting in for a week. The light trigger never armed because the state I actually arrived from, paused, was never in the list I wrote.

The claim those two failures back reaches past Home Assistant. A status code measures the request you sent, an event filter measures your guess about what the device does, and neither one measures what the device did. Both bugs stayed invisible until I read the state back and dumped the transition trace, and both fixes were one line each.

The write that went nowhere

The two lamps are Leedarson RGBTW bulbs behind a Matter pairing, and on 2026-10-04 around 00:16Z they got re-paired, the old entities died at 00:16:58Z and two new ones were born at 00:15:27Z. Same fixtures, new identity, and the scene file kept pointing at the old ones.

Then every rgb_color write and every hs_color write got eaten by the bulbs in total silence, readback stayed warm white at [255,163,87] after each attempt, while brightness passed through untouched. Only xy_color landed. So the scene entries went from rgb to xy coordinates, left lamp [0.0305, 0.1364], right lamp [0.25, 0.1], and the readback after the change came back [37,157,255] at brightness 128 on the left, [112,0,255] at 128 on the right.

The half-landed write narrows the diagnosis, brightness moved and color didn't, so the integration was fine, the wiring was fine, the bulb understood the request, and one field of that request went out the window. A 200 response cannot separate those cases and nothing else in the chain can either.

The gamut, probed

I didn't pick those coordinates once. [0.33, 0.13] came back too magenta, [0.2, 0.08] came back too blue, and [0.25, 0.1] is the best purple that bulb's gamut can do. Three probes, each one read back before the next, and the color that shipped is a color the device can physically produce.

That's the same rule as the write above, applied one layer down. You can ask a Leedarson for a violet it has no diode for, the ask succeeds, the result is whatever the gamut edge happens to be, and the only way to know which edge you landed on is to look.

The trigger that never armed

The automation fires when the player reaches playing, with for: 5s and a from-list of [off, idle, unavailable, unknown, buffering]. On the broken night the launch path walked off to idle to paused to playing inside about 130 milliseconds, so the transition into playing came from paused, and paused was never in the list. The trigger never armed, the scene never ran, the room stayed warm white.

The night before, 2026-10-04 at 00:11:53Z, it fired, because that session went idle to playing. Same device, same trigger, one path that matched and one that didn't, and the difference between them only shows up in a trace nobody prints by default.

The fix was the word paused, added to the from-list, backed up as automations.yaml.bak-moviemode-20261005-2009. The live test was media_pause then media_play on the player, which moved last_triggered to 2026-10-06T00:13:31.992930+00:00, end to end pass, and I applied the scene manually that night since the TV was already playing by the time I found the cause.

What the plumbing around it reports

The reload calls, POST /api/services/automation/reload and POST /api/services/scene/reload, both returned an empty-body 200, which means the config reloaded and says nothing about whether any lamp changed. Two more traps sit in that same setup and both are cheap to state.

for: 5s needs five seconds of held playing, so a player flapping between idle and paused keeps resetting the timer and nothing arms. And the TV carries two entities, the cast-backed one reports playing, paused and content type, the native-app one only ever reports on and off, so a trigger written against the second entity is dead forever and quiet forever. There's also no movie-mode-off automation in that setup, by original design, the lights keep the movie colors until the next all-lights event.

The guards that make this class of bug visible

After any device write, read the state back and compare it against the requested value, a bulb answering 200 while reporting the previous color is one line of comparison and that line turns a silent no-op into a named failure. Build the from-list from a recorded transition trace, the docs list what a spec allows and the app's real path is what the device sends, those two sets are not the same size. Keep the hold window in mind while you write for:, because a state that crosses in 130 milliseconds can never satisfy a 5 second hold, and a trigger that needs both a fast crossing and a long hold is asking for something the device won't do.

Both fixes cost one line each, and getting there cost reading back every write and dumping one trace. The device layer and the automation layer failed in different ways, the bulb dropped a field and the trigger dropped an event, and the one thing they share is that neither reported a problem.

So for anyone with automations wired to real devices, when did you last read back a color write, and does your trigger list contain every state your player actually emits?

A half-million-token prompt found both needles on my 3090

prodbymarcu — Mon, 19 Oct 2026 22:16:39 +0000

Every stack I've seen sells the context window as a model spec, and every fix for oversize is RAG, and what I measured on my own stack this month says the ceiling you actually feel lives in a config line. Mine was 131072. The model held retrieval exact at 509,084 tokens in one request.

The lie I found first

Pushing the local stack on my 3090 Windows box to a 524,288-token window, the agent kept reporting 131k, and the root cause was my Hermes configs advertising context_length 131072 while the engine underneath ran 262144. Engine, proxy, agent each declare their own number, the smallest one caps the rest, and nobody prints a warning when they disagree, so the gap between 131072 and 262144 is just quietly gone.

That afternoon found the same shape behind the classic "shows 131k" complaint, a stale middle-layer number half the size of the engine under it.

The config that made it real

--max-context 524288
--kv q4_0
--kv-resident 32768
--rope-scaling yarn
--rope-scale 2
--yarn-orig-ctx 262144

llama-swap advertised 524288, the Hermes configs got 524288 too, all three layers finally agreed on one number, restart through nssm brought the engine back in about 40 seconds, and /health printed max_context 524288 before I sent anything.

The run

509,084 prompt tokens in a single request, under the 524,288 cap, and a 1.76M-token test got correctly 400-rejected, so the cap was enforced at the engine. Prefill burned 381 seconds, 1,336 tokens per second. Two facts sat buried in that prompt, one near position 50, one near position 394,540, both retrieved exact, and the needle past 394k sits well past the trained 262k window, so yarn quality held across the scaled range on my model.

The honest cost is the prefill, 381 seconds before the first token, 6.4 minutes. Follow-ups on the same document hit checkpoint reuse and came back near-instant, the checkpoint-hit resend probe measured 0.12s, so you pay the prefill toll once per document and every follow-up rides the checkpoint.

The RAM night

Half a million tokens needed RAM, so I ran a RAM night, four 16GB G.Skill sticks in all four slots, 32GB to 64GB, 63.9 visible. DOCP at 3600 failed POST. Four dual-rank DIMMs under MC load took the box down to 3400, which POSTed and Windows verified every stick at 3400. The trap that cost an hour: BIOS "Auto" after a failed POST silently retrains to 2133, so when speeds look dead, check DOCP is still enabled before chasing anything else.

KV math made VRAM a non-issue at this scale, 262k tokens at q4_0 is 1.69GB, it fits resident and the streaming flags are pointless there.

Calibration I measured

3.11 characters per token on the synthetic prose, 32.2 tokens per ledger line, both measured on the actual document, so multiply the lines in your file and "will this fit" becomes arithmetic before you send.

The probe that ranked it

The TTFT probe I ran before and after, 10k tokens on a fresh prompt with thinking on: 7.62s on the 131k window with 32GB, 5.10s on the 262k window with 64GB. Doubling RAM took 7.62s down to 5.10s, and the bigger window still answered first-token faster than the smaller one did the day before.

What this says about RAG

The claim I'm backing with this: retrieval quality at half a million tokens held on my engine with rope scaling set right, and the failure that bit me was a stale config line half the size of the engine under it. RAG exists because people hit that config ceiling and read it as a model ceiling. The two numbers are cheap to check, /health prints max_context and your agent prints its advertised context_length, and if they disagree the small one wins in silence.

So the question for anyone running a local stack: what does your agent advertise, and what does your engine actually load, because this bug hides in the gap between the two, and mine was 131072 against 262144.

Why does your voice agent go silent mid-call?

prodbymarcu — Mon, 19 Oct 2026 10:04:08 +0000

"I'm on a call, Bender isn't responding. I heard you but you're not responding." That message started a week of digging that ended with six plumbing bugs and one provider stall that looked exactly like them. Not one of them was the model getting dumber. Every one had the same shape: a piece of plumbing gave a confident wrong answer about whether the call was happening, and the code believed it.

If you're building anything that has to know when a human is on the other end, you'll hit these. I hit all six on one Mac.

The watcher died exactly when a call connected

The watcher spawned the voice loop through a hardcoded interpreter path pointing at .venv. The directory had been renamed to venv at some point and the old path was dead, so Popen raised FileNotFoundError inside the watcher, which killed the watcher at the moment a call came in. The call answered. Nothing spawned. No process left running to log why.

The fix is boring on purpose. Resolve the interpreter from a candidate list, probe each one with an import, fall back to sys.executable, and wrap the spawn so a failure gets logged, because the process that watches for failures shouldn't be the one that dies.

Detection was blind, and the watchdog called blindness death

No camera on this Mac, so FaceTime draws the in-call UI somewhere the accessibility tree can't reach: zero AX nodes, and the window itself only says "connect a camera". The helper probe returned idle through a 448 second call. On the camera-less machine both detectors did the same for an entire live call. The watchdog read idle, decided the call was over, and closed a live session about 14 seconds in.

The replacement signal came from the audio side. The loop holds two virtual audio devices during a call, and both read IsRunningSomewhere = True while it's up, False when it isn't. That's the detector now. The watchdog rules changed with it: explicit ended closes immediately, four consecutive bad readings before it hangs up, and an undetermined reading keeps the session alive.

One layer down, the same bug in smaller type. The watcher used to close on a single unknown, and detection blinks to unknown for a poll or two while a history database copy races, so one blink killed calls. Six consecutive unknown readings, about three seconds, is the bar now. Group calls needed their own path too, because they never emit ringing at all, they go straight to connected.

The hangup signal was backwards

CallHistory has no row while a call is up. The row shows up when the call completes, carrying the final duration. The old check read duration == 0 as "still live", which is wrong in both directions, because a failed dial leaves 0.0 sitting there forever.

The signal that survives is a snapshot. Record the newest Z_PK when the session starts, and any newer row means the call ended. That's the only honest hangup source on this machine, since the process holding the audio device can never detect its own call ending.

Audio hangs with the process still alive

sounddevice deadlocked inside OpenAndSetupOneAudioUnit, waiting in mach_msg. sample <pid> showed the exact frame, and a bare InputStream hung the same way. sudo killall coreaudiod unsticks it, because the daemon relaunches on demand and the device open goes back to instant.

It returned a few days later and coreaudiod was innocent. The holder was audiomxd, in R state since boot, 29 days at roughly 94% CPU, with load average between 48 and 88 and zero percent idle. Killing that daemon unstuck audio immediately. The rule now: when an audio open hangs, read process state before you restart the obvious daemon.

The loop kept the failure alive

This one made it look permanent. A stuck loop holds the virtual audio device, so detection reads "connected", so the watcher spawns another loop, and two loops fight over one device. That wedge survives coreaudiod restarts, which is why it read as a hardware problem.

The proof was two runs. With the watcher stopped and stale processes killed, audio opened instantly. With them running, every open hung. The rule that came out: one loop per call, zero loops when idle, enforced from both sides, plus a watchdog that kills duplicates mid-call and stale loops when no call is live.

The call that never rang

Above all of that sat a failure no agent code could touch. A screen sharing session was up, and macOS suppressed the incoming call before it rang. The log said it with no ambiguity: outcome: suppressed; reason: display shared. One notification setting, the one for when you're mirroring or sharing the display, was off. Turning it on gave 48 out of 48 allowed resolutions and zero suppressions over the next five minutes. The agent didn't miss the call, the operating system never let it exist.

The provider added four minutes of dead air

With the plumbing fixed, a live call still went quiet for four minutes. The log had the shape of it: dispatch at 20:19:03, first tool output at 20:23:07, and the tool round itself took 3.1 seconds. The next turn did it again, dispatch at 20:23:18, done at 20:27:31, which is 4 minutes 13 seconds of dead air around 11.6 seconds of work. No client timeout existed anywhere, so a congested provider queue looked exactly like a hang.

The fix is a 35 second socket read timeout on every brain call. Streaming deltas reset it continuously, so it bounds dead air without cutting off a model that's genuinely thinking. Behind it sits a fallback model handed the same conversation, measured at 1.9 seconds through one call path and 1.8 through the other. A separate limit bit me on reconnect: the greeting carried 2,500 characters of call tail, tripped Context append text must not exceed 500 tokens three times, and died silently every time. Capped at 900 characters.

The rules that came out of it

Fail open. An undetermined reading keeps the session alive, and only an explicit ending closes it. A watchdog that treats "I can't tell" as "it's over" will kill live calls on any machine where detection is inherently blind, and that's most machines.

One owner. Exactly one process may hold the audio device per call, and zero when idle. Every duplicate is a future feedback loop.

Process alive is not the same as working. A wedged daemon keeps its PID, its memory, and its state, then stops at the first line of real work. Test the thing itself, not the process table. The health script opens the audio device on every run now, and its silence is what tells me everything is fine.

The self-healer runs every five minutes and has five jobs: kill duplicates during a call, kill a wedged loop so the watcher respawns clean, kill stale loops when no call is live, restart the audio daemon when a device open hangs, and kick the watcher if it's down. It says nothing unless it acts.

What I'd check first

If your agent goes silent, don't start at the model. Start at the last place that reported what was happening, and ask what it prints when it doesn't know.

Where has a watchdog in your stack reported confidence it hadn't earned?

I asked an LLM to spec my dual-GPU power plan, and the second supply could never start

prodbymarcu — Sun, 18 Oct 2026 22:07:25 +0000

Press the button. Nothing happens. No fan spin, no board light. A brand new 700W power supply, wired exactly the way a model told me to wire it, sitting in the case like a brick with a fan screwed to it. That was the end of a two week hunt for a second RTX 3090, and it cost me a weekend.

The hunt

Used 3090s cost money, so I chased one on the marketplaces for a while. Two sellers flaked before I got anything, one promised a video and then disappeared, another sold the PC out from under me the same afternoon I said I'd take it. I passed on a $1,400 firm listing out of stubbornness and regretted it for a week. What landed was a Gigabyte AORUS 3090 picked up in person, cash in hand.

The first card is a 3090 Founders Edition, already in the case. The machine is a 5800X with 64GB of DDR4 in a Lian Li box, eight Corsair fans and a glass side panel, because two graphics cards behind glass are worth looking at. FE up top, AORUS bolted into the lower slot. One card for rendering, one for the models that judge the render, and I wanted them running at once, because a single 3090 doing both is a lesson in patience.

The plan I got handed

There was an 850W unit in the box. Two 3090s at 350W TGP each, plus a 5800X, is already over what I want to run on 850W, so I asked ChatGPT for the right shape and got back a confident, tidy answer. Buy a second power supply, run the Founders Edition off it, PCIe cables only, you don't even need the motherboard connections. Two cards, two supplies, load split. It read like something a person who builds machines would say.

So I bought a 700W unit and wired it exactly like that. The FE pulled its feed from the new supply, the rest of the machine stayed on the old one, I pressed the button, and there was no twitch anywhere on the desk.

What could never start

An ATX power supply does not turn itself on. It waits for PS_ON#, the green wire in the 24-pin connector, to be pulled low by the motherboard before it enables its rails. The board is what says go, and in that plan nothing was connected to the second supply but PCIe cables. No PS_ON link, no shared ground reference back to the first unit, so the rails never came up and the card stayed cold.

Every working dual-supply setup I've read about since uses a jumper on the second unit or an add2PSU style relay board that repeats the first supply's signal, and they tie the grounds together so both units share one reference. None of that was in the answer I got, because the answer came from something that has never had to make a machine boot.

The sizing problem underneath was worse. Ampere cards don't respect their own rating through load transitions. A 3090 sitting around 340W average pulls well above that in millisecond bursts, and two cards spiking at once is what a supply's overcurrent protection exists to catch. That's how renders die mid-run, or how a supply decides it's had enough. One unit with real headroom on a single rail is the sane shape, so the 700W went back to the store and a 1200W went on order.

One connector worth getting right. The Founders Edition uses NVIDIA's 12-pin MicroFit with an adapter to two 8-pin feeds, the 12VHPWR connector everyone argues about belongs to the 40-series cards, and the two get mixed up constantly in threads about power.

The guard that replaced the advice

The model didn't lie to me. It produced a plausible configuration, and I treated plausibility as verification because I wanted the build finished. The failure was mine, and the cost was a 700W paperweight and a weekend of my own time.

What I do now is cheap. Any hardware advice from a model gets converted into mechanism questions before a dollar goes out. What makes this start, what pulls which signal, what happens on the first load spike, what the thing does when one part is missing. If the answer can't survive those questions, the advice is decoration and I don't buy for it. For this build the check was one line. The supply needs PS_ON pulled low and a ground shared with the rest of the machine, and a second supply with neither is a box.

What shipped

The 1200W landed on the evening of 10/17. The box went down around 1:30pm that day for the swap, with the AORUS already seated in the lower slot, and I wouldn't be near the case when it came back, so I armed a watcher on the Mac. It polled ssh every 60 seconds, gave the machine 90 seconds to settle, restored the inference services, waited 45 more, then probed VRAM to find the card without the resident engine and relaunched ComfyUI on that one. A 4 minute health-check on the render server, a smoke render from the Mac, one message with the result. The watcher fired clean at 22:09, which means the machine booted about eight and a half hours after it went down and I never touched it.

Then the payoff I'd been building toward. On one card the render engine and the always-on inference engine were fighting for the same 24GB. A 1920x1088 image-to-video pass OOM'd, that path runs two forward passes per step, and 1664x960 at 50 steps fit only with the card clear, 21.6GB in use at 100% during sampling, at a two to three hour ETA. Worse, the resident engine kept creeping back onto the card mid-day, restarted by something following stale notes, and my first attempt to stop it used pkill -f over ssh, which matched the ssh session's own command line and killed my connection instead of the service. systemctl stop was the fix and that's now the rule on that box.

With two cards that class of problem goes away. The AORUS holds the resident engine at 21.4GB plus a 1.3GB vision sidecar, running 24/7, the other card holds the TTS model at 19.4GB with ComfyUI and the vision critic co-resident beside it, and a full render-and-critique round runs about 40 seconds. The grind that ran on top of it was 46 rounds across 6 briefs against a bar of 9, clearing twice at 8 out of 10 and never once hitting the bar, and the constraint left there is the models' taste.

Two 3090s on one 1200W supply, one switch, one signal, and no relay board to forget. The 700W went back to the store the morning I said it would, and the box has been running dual-card ever since.

My art director agent scored its own render 9/10, then frame stepping called it 4/10

prodbymarcu — Sun, 18 Oct 2026 10:01:38 +0000

I've been building an autonomous art director. Give it a one-line brief, a planner turns that into a shot description, SDXL with Juggernaut XL v9 renders it, a vision model looks at the output and writes fix directives, and the loop re-rolls until the critic clears the bar. Then a video model animates the winning frame.

Hardware is one RTX 3090, 24GB, a 5800X and 64GB of DDR4. That card is the constraint everything else bends around.

The first loop looked perfect

The loop ran end to end at 1:30am on 10/16. Planner, render, critique, verdict, 38 seconds for a full round. The first brief I gave it, a neon rain heist street, cleared at round one with a 9/10. Artifacts on disk: final frame, timeline HTML, run JSON, every round saved.

That should have been suspicious. A critic that hands out 9/10 on the first pass is a rubber stamp with a vision encoder. The loop had a bar of 8 and an or verdict == pass clause in the clear check, so a generous model could wave anything through.

The video model raised the bar by failing

Then I moved to the video half, MiniMax-H3 image-to-video on the same card. First attempts came back as mud. Then the recipe cracked and scores went 2/10, 4/10, 9/10 across three takes. The fix that mattered was CLIPLoader type minimax instead of hidream, plus the official sampler chain: RandomNoise and BasicGuider with no negative prompt and no CFG, KSamplerSelect res_multistep, BasicScheduler simple at 8 steps, the turbo 8-step LoRA through LoraLoaderModelOnly, and the latent pulled from slot 1 of the MiniMaxH3ImageToVideo node. Take 3 held geometry and showed no banding.

So I had a 9/10 clip. I put it in front of the person who has to approve the demo and got "fake as fuck, minecraft rain physics" back. Fair. Heavy explicit rain streaks render as particle sprites, and real night rain reads as mist, beads, drips, puddle rings.

That verdict forced a second look at all of it, and the second look was worse.

Frame stepping is a different measurement

I ran an independent QC pass that ignored the blind score. 13 spread frames per clip plus two 7-frame consecutive runs, stuck onto contact sheets, read frame by frame. Take 6, the max-tier 50-step render, scored 3 to 6 out of 10 in most cells with motion at 4/10. Pseudo-glyph signage morphed between frames. The umbrella figure got re-synthesized every frame like the model was redrawing it from scratch. Ground speckle boiled. Rain read as absent or as sprites. Take 3, my 9/10, scored 4 to 6, with a vendor face hallucinating mid-clip and steam teleporting between frames.

At playback resolution the blind score had been accurate: geometry locked, no banding. It never looked at two consecutive frames and asked whether they belonged to the same world.

Three root causes, in the order they matter. The source still broke the recipe I'd pinned myself, with a human figure and legible signage in frame, and the video model reinvents both every frame. The hi-res pass was soft: lanczos upscale plus img2img at 0.42 denoise left a laplacian of 29.7 against the clip's frame 0 at 30.9, so the model animated mush and preserved its mushiness frame by frame. Generic image-to-video temporal noise did the rest.

The memory wall behind it

Every attempt to fix this by brute force hit the card. 1920x1088 OOM'd because the CFG path needs two forward passes per step, so the hires tier doubles memory before it improves anything. 1664x960 fits at 21.6GB of 24GB against a 1.2GB baseline, and only with the card actually clear. More than once it wasn't: llama-swap and the resident engine crept back onto the GPU mid-render, and a 10:30 cron following stale worklog state is my main suspect.

The nastiest failure was the 124-frame VAE decode at 1664x960 spilling into host RAM. The box went into a swap death spiral where ICMP answered, TCP ports answered, ssh died at banner exchange, and ComfyUI returned an empty HTTP body. A machine that answers a ping and can't say hello has run out of memory, and the network was fine the entire time.

The guards

Four changes landed in the loop, committed as c26c82c.

Bar 8 to 9 with a strict clear: score at least 9 AND a pass verdict. The or verdict == pass bypass is gone.

Chain-aware taste in the planner and the critic. No people or creatures, no legible text or glyphs, ambient motion potential, clean at 1080p. A violation caps the score at 7 no matter how pretty the frame looks.

Fix deltas reach the planner. They were being computed and then discarded, which made the loop decorative. Now the planner receives the problems, the prompt and negative deltas, and the recent prompts, and has to evolve them.

The hires pass became a latent upscale, 1.5x lanczos plus a 30-step KSampler at 0.42 denoise, dpmpp_2m with karras, cfg 5, replacing the soft lanczos plus img2img route.

The gate standard changed too: no clip ships on a single-frame read at playback resolution. 13 spread frames plus consecutive runs, every time.

Audio got its own version of the same treatment. The generated rain audio passed an independent realness score of 0.91, "real or pro-produced", which was encouraging, but the master measured -30.89 LUFS with a -17.33 true peak. Two-pass loudnorm to I=-14, TP=-1.5 landed at -14.7 LUFS, -1.8 dBTP, zero clipped samples. That recipe is a pipeline constant now.

What shipped and what's next

The hardened loop is in. No clip passes the video gate yet. The winner has to come from a still that obeys the chain recipe from the start, rendered by the new loop at bar 9, upscaled properly, then animated at 50 steps with CFG 2.0 and no LoRA, which runs about 65 minutes per clip.

The second 3090 is seated in the lower slot. One card carrying the render and the critic at the same time is why the resident engine kept getting evicted mid-run, and why every hires tier turned into a memory negotiation. Two cards put the critic on its own 24GB and end the competition for VRAM. The machine is down for a power supply swap, and when it comes back the first job is the same brief at the higher bar.

A day and a half of measuring taught me one cheap habit. A model grading its own output at playback resolution will call the frame beautiful. Ask it about frame 47 and frame 48, and it tells you whether the frame holds still.

My self-hosted ad blocker poisoned Apple's push servers and killed iMessage

prodbymarcu — Sat, 17 Oct 2026 22:03:08 +0000

I run AdGuard Home in Docker on a spare Linux box at home. Every device on my network gets it as DNS through DHCP, so the laptops, the phones, the TV, all of them resolve through one container I control. For a few weeks it looked like the cleanest thing in my homelab. 18 filter lists, 85 user rules papering over the false positives, ads gone everywhere including the TV.

Then iMessage stopped sending and receiving on my Mac.

The symptom was confusing. Restart the Mac, messages arrive fine for about 30 minutes, then everything queues and never delivers. The BlueBubbles API returned status 200, message sent, every single time. chat.db told the truth, error 4, is_delivered 0. IDS logs showed NSURLErrorDomain Code=-1001 and response code 2, so Apple's push layer was timing out, nothing in the messaging app itself.

My first instinct was DNS, because it always is, but I checked the obvious thing anyway. nslookup courier.push.apple.com returned 0.0.0.0. The same lookup against 8.8.8.8 returned 17.57.144.152. My own resolver was answering Apple's push domain with a null route.

The rules doing it were explicit, ||mask.icloud.com^ and ||mask-h2.icloud.com^, sitting in a filter list I had turned on for privacy, and those domains are how iCloud Private Relay masks traffic. Blocking them killed Private Relay on the iPhone, which toggled between available and unavailable all day, and the push hostnames got caught by an aggressive list above them.

Why a reboot fixed it for 30 minutes

IDS keeps a DNS cache. At boot it resolves the push hostnames once, gets real IPs while the cache is cold, connects, and messages work. Thirty minutes later the cache expires, IDS re-resolves through the resolver that answers 0.0.0.0, and every connection after that fails. That works-after-reboot, dies-30-minutes-later shape is the fingerprint of DNS poisoning. Nothing in the messaging stack changed between minute 1 and minute 30.

I killed imagent once to force a re-resolve, which was a mistake. It triggered the same XPC sandbox crash I had seen in July, CoreData failed to create NSXPCConnection, contactsd sandbox restriction, xpc_error 159. imagent uses the system resolver on every connection attempt and caches nothing, so killing it bought me nothing and cost me a reboot.

The same failure one layer down

While I was in there I found a second one. AdGuard had two DoH upstreams, Quad9 and Cloudflare, in load_balance mode. Quad9's endpoint intermittently fails on domains with multi-level CNAME chains, Azure Front Door traffic especially. The log line looked like this.

dnsproxy: exchange failed upstream=https://dns10.quad9.net:443/dns-query
  question=";app.evenrealities.com. IN A" duration=3.6s err="unexpected EOF"

An app that worked on cellular and failed on WiFi sent me looking, cellular bypasses my resolver entirely, WiFi goes through it, so the phone was fine until it joined my network. With two upstreams in load_balance, roughly half the queries for an affected domain hit the broken one, which makes the failure look random from the client. Testing one domain and getting an answer proves nothing. It only shows on the CNAME-chain domains, and only on some tries.

The guard

Two fixes, one pattern. First, allowlist rules in user_rules, one per domain, ten for Apple's push and identity stack, eleven more later for X/Twitter when its CDN domains got caught by the same lists.

@@||courier.push.apple.com^$important
@@||init.push.apple.com^$important
@@||mask.icloud.com^$important

The @@ makes it an exception and $important makes it override filter list blocks. Second, I cut the lists themselves from 18 to 1, AdGuard's own DNS filter. The aggressive tiers were catching Apple push, Google login, X, Chegg, and I was writing a rule to unblock each service as it broke, 85 rules to keep 18 lists happy. After the trim, 23.

Then the DoH upstream swap, remove Quad9, add Google DoH next to Cloudflare.

Editing the file without breaking it

One of those edits destroyed the config. I piped a base64 Python script through SSH, something on the way in stripped every control character including the ones YAML needs, and the file came back unreadable. The backup sitting next to it saved me a rebuild. Now the script lands in /tmp as a real file and runs from there, and the edit itself is yaml.safe_load into yaml.dump with sort_keys=False, never sed.

AdGuard reads config at startup, so docker restart on the container, wait about five seconds before testing, the DNS proxy is not ready the instant the process is up. A reload does not re-parse user rules.

Verification is two lookups. Local resolver against a public one, same domain. If local says 0.0.0.0 and 1.1.1.1 says a real address, the block is mine. Then grep the container logs for exchange failures over the last couple of minutes.

What the config looks like today

I re-read the live file this week, one filter list enabled out of the 18 still listed in it, AdGuard's own, 57 user rules up from 23, because Google login, iCloud, gstatic and googletagmanager got allowlisted after they broke the same way. courier.push.apple.com resolves through my resolver again, full CNAME chain, and doubleclick.net still answers 0.0.0.0.

The upstream change is the one that did not stick. Today's config still lists Quad9 and Cloudflare in load_balance, so the intermittent failure is still in there, waiting for a domain with a long CNAME chain and a bad coin flip. The swap is written down and not applied. That is the next five minutes of work on this box.

I almost rebuilt a project for the wrong reason, a 20 minute audit caught it

prodbymarcu — Sat, 17 Oct 2026 15:09:55 +0000

I have an iPhone-driving rig. An ESP32 board plugs into the phone over USB and plays human, screen taps, button presses, the works, so my local agent can actually operate the phone instead of just talking about it. The rig works. It has worked for months. So naturally my first instinct was to rebuild the whole thing from scratch.

The reason I gave myself was security. The codebase came from a mixed-background project, there were Chinese translation files in the repo, a bundled JavaScript front end I had never actually read, and firmware binaries I had flashed without ever opening in a hex editor. In my head the story was already written: old rig is sketchy, new rig is clean, rebuild obviously justified.

Before I burned a month on it I made myself audit the old code first. Four checks, about 20 minutes total. Every single one came back clean, and the audit quietly rewrote my justification for the rebuild.

Check one, scan the repo for CJK text

I grepped the whole tree for CJK ranges across every .py, .js, .ino, .html and .md file. The rule I wanted was blunt: no Chinese anywhere in code I own and run. The scan found exactly two carriers of Chinese text, a UI translation file called i18n.js, which is translation data doing translation work, and a README.zh-CN.md, which is a translated README doing documentation work. Zero Chinese in executable paths. The thing I was going to "fix" by rebuilding did not exist.

Check two, follow every outbound call

I traced every place the old code talks to the network. There is exactly one outbound call in the entire project, one line in llm.py that hits the configured LLM base URL, and that URL is my own box on my own LAN. No telemetry, no analytics, no hardcoded third party endpoints, no phone-home beacon hiding in a helper. A project that talks to exactly one host, and it is mine.

Check three, read the bundled JavaScript like a stranger

The web front end was the part I was most suspicious of, because I had never read it and it ships minified. I ran a proper JavaScript analysis pass over static/, two files, about 23KB, and read what it actually does. Every fetch is same-origin, every request goes to /api/* on the host it came from, nothing touches IPC or any native surface. The bundle is a small local UI talking to its own server. That is the whole story.

Check four, pull the strings out of the firmware

Last one, the ESP32 firmware. I pulled printable strings from the .bin files and read them. What is in there: our own server IP, and Espressif SDK doc URLs from the vendor toolchain. Nothing else. No surprise hosts, no encoded blobs, nothing that looks like it was smuggled in past the source.

What the audit actually changed

Here is the part that matters. The audit did not tell me the old code is good. It told me the old code is clean, and those are different verdicts with different consequences.

The security story was the emotional engine of the rebuild. Clean code killed it, which forced me to write down the real reasons I want a new version, and the honest list looks completely different. I want it native, the old rig is a localhost web app plus a Python service plus a launchd helper, and I want a real Mac app. I want it fast, the old screen pipeline decodes frames in software and I want hardware decode and native rendering. I want auto-calibration, the old rig needs a manual calibration click and it should not. I want ownership, code shaped the way I would shape it, with English comments and a structure I can hold in my head.

None of those reasons say anything is broken. They all say the old rig is working software that I have outgrown. That is a rewrite for architecture, and rewrite-for-architecture makes different decisions than rewrite-for-security. It keeps the parts that already work. The USB HID layer in the old rig is exactly the shape I want, so the new build keeps it instead of redesigning it. The away-mode trick in the new architecture, board joins the phone hotspot and tunnels home over the phone's own network, gets validated before anything builds on it, because there is no security schedule forcing that bet early.

And the old rig stays running untouched while the new one gets built, which I could never justify if the old one were actually unsafe. Clean verdict bought me the luxury of not hurrying.

The rule I am keeping

The rule costs 20 minutes and it is now mandatory on me before any rewrite: audit before you rewrite, and write down what the audit found before you write down why you are rebuilding.

The failure mode it guards against is subtle. Rebuild energy feels productive, and a security justification feels noble, so the combination feels unassailable. You end up a month deep in a new codebase and you never once open the old one in anger, because the whole point was that you did not have to read it. The audit collapses that. If the code is dirty you now know exactly where, and the rebuild has a spec. If the code is clean you lose your noble reason, and good riddance, because a rebuild justified by a feeling is a rebuild that will inherit every problem the old code had plus new ones you add yourself.

Scan the text encodings you do not recognize. Trace the outbound calls and count them. Read the minified bundle like a stranger, not like its author. Pull the strings out of the firmware. Four greps and one string dump, and you will know whether you are fixing a problem or just redecorating.

My old rig was clean. I am rebuilding it anyway, for the real reasons, on the timeline a clean verdict earns. The audit did not stop the rebuild, it made the rebuild honest.

I built a handheld mmWave presence radar, and the first bug was a board that never ran my code

prodbymarcu — Sat, 17 Oct 2026 10:02:32 +0000

One small device noticing one thing. That's the demo genre I keep falling for, and this time it's an M5Stack Cardputer-Adv with an LD2410 mmWave sensor hanging off its EXT header. The firmware turns the sensor's UART stream into a radar scope on the handheld's 240 by 135 screen: a sweep line rotating through the cone the sensor watches, roughly 60 degrees, a contact sitting at a radius equal to the measured distance, a trail fading behind it as the target moves, and a NO PRESENCE state when the room is empty. No app, no phone in between. The device itself knows someone is approaching.

Writing the firmware was the easy part. The hard part was getting a single byte back out of the board.

What the sensor streams

The LD2410 reports presence and distance about every 100 milliseconds at 256000 baud, one target at a time, split into a moving reading and a stationary reading. Frames come wrapped in a fixed envelope: four header bytes up front, F4 F3 F2 F1, four tail bytes at the back, F8 F7 F6 F5, a length and a type in between. The frame I care about carries the basic target state, one byte for the state (nobody, moving, stationary, both), two bytes of moving distance, a moving energy byte, then the same trio again for the stationary side.

Parsing that is a small state machine over the byte stream. Slide until the four header bytes line up, accumulate until the four tail bytes close the frame, decode, repeat. On top of it, a render loop runs at 10 frames a second, the sweep advances 9 degrees per frame, about 40 sweeps a minute, and the contact plots at measured distance against the sensor's 6 meter range, mapped onto the scope radius. Above the scope sits the distance in centimeters, MOVING or STATIC, and two energy bars. And one small flag, ld:OK, that lights when the sensor has sent frames in the last second. That flag ended up being the most important pixel on the screen.

Zero bytes, and where I was looking

First flash, first open of the serial monitor, and the board had zero bytes to say. No garbage bytes, no half frames, nothing.

I reviewed the parser the way you do, baud rate, frame layout, pins. Zero bytes gives you nothing to debug against, though, so I flashed the dumbest possible sketch, a heartbeat that prints one line every two seconds and does nothing else. It streamed zero bytes too. A sketch that can't fail in software, equally silent.

That killed the parser theory. Zero bytes meant the app had never started, and the reasons were mechanical, all of them outside the code.

My build config had ARDUINO_USB_MODE=1, the JTAG option. This card talks serial over native CDC, and the JTAG build left it unresponsive to stdio. One line, ARDUINO_USB_MODE=0, and it's the first line I check in any new ESP32 build config now.

After an upload, esptool and pio leave the board sitting in its bootloader, and the app does not auto-run out of it. Zero bytes is exactly what a board in that state looks like, and the only fix is a press of the device's power button.

Then the one that stung. When I finally pulled the chip info, the board on my bench came back as a plain ESP32-S3 DevKitC, 8MB PSRAM, 16MB of flash. Not a Cardputer. The firmware was written for a handheld with a display, a keyboard, and an M5 library, and the thing on my desk has none of those. I had assumed the one serial device plugged into my Mac was the Cardputer I left there. Wrong board, and no amount of re-reading code was going to put a display on a devkit.

Porting down to what actually exists

So I ported the project to the hardware that was in front of me. devkit_radar/devkit_radar.ino drops the M5 library and the display and keeps the part that matters, the same LD2410 frame parser on GPIO15 and GPIO16, printed straight to serial. A line every two seconds, plus a line any time presence flips or the distance moves more than 5 centimeters, shaped like this:

[TARGET 87cm MOV ld:OK e:0.42]

That line is the piece I'd keep from this stretch, because it makes silence readable. Three states, three different fixes. No bytes at all means the board is parked in the bootloader, press the reset button. Lines arriving with the flag at ld:-- mean the board is running and the sensor is quiet, a wiring problem, and the classic one is swapped TX and RX, which reads as distance stuck at zero. And ld:OK on a TARGET line with a real centimeter value means the chain is alive, and walking toward the sensor should close the number.

The same mirror went into the Cardputer firmware, so both builds talk the same protocol over USB. The radar firmware went onto the board three times, hash verified on every pass, and the devkit flashed through arduino-cli with the USB mode pinned to CDC.

What shipped

The Cardputer side is done in software. Scope, sweep, trails, three color themes on the 0 key, a HUD toggle on the period key, ESC to blank the screen, the serial mirror, the heartbeat. devkit_radar is the headless twin for the bench board. The wiring is written down as a table, EXT pin 13 to the sensor's TX, pin 14 to its RX, 5V and ground, UART straight through since the LD2410 logic runs at 3.3V while the sensor itself takes 5V on VCC.

What's still open is physical, and I won't pretend otherwise. The app needs a power button press after flashing before it runs, the sensor has never been wired, and the walk test hasn't happened yet. The next session is a reset press and four jumper wires, then watch the serial line. If the distance closes, the blip closes on the scope. If it doesn't, the line tells me which link is lying within one boot cycle, and on hardware that counts as a good day.