DEV Community: Den

I built my own roadmap. The first thing it found was that my SR latch didn't work.

Den — Fri, 16 Oct 2026 20:26:10 +0000

Two posts ago I compared my logic simulator to nine others
and listed three honest gaps. In the
next one
I opened the code and discovered that the three gaps were really five items in a
dependency graph, two of them invisible: make time explicit, then waveforms and
step mode; make node identity hierarchical, then sub-circuits; and test cases that
can handle a clock once time exists.

That was the plan. This post is about what happened when I executed it, which is a
different thing, because the plan was drawn by someone who had not yet tried to
build a flip-flop out of two gates.

Boolflow is a browser digital-logic simulator, free, no
account. Everything below is live.

The scoreboard

In the roadmap post	What shipped
Explicit tick counter + history	Done, but a "tick" turned out to mean something other than I wrote
Step mode + waveform panel	Done
Hierarchical node ids by flattening	Done
Sub-circuits	Done, including editing a block after you placed it
Test cases (sequential half)	Done: a `C` in a column is a clock pulse
"The clock frequency bug"	Fixed, and it was worse than I said
"Tell the user it didn't settle"	Done, in a form I did not expect
Share link	Not done
Truth table to circuit	Not done
iframe embed	Decided against
Buses	Still untouched, as promised

The table is the boring part. The interesting part is everything in it that
needed a footnote.

"Tick" does not mean what I said

I wrote that I needed a monotonically increasing integer owned by the simulator,
and a ring buffer recording the value map at each tick. Both are in. But I
recorded a frame on every evaluation, then watched a waveform fill up with
identical columns while I dragged a gate across the canvas. Moving a node
re-simulates the circuit. Nothing observable changes. Time should not advance.

So a tick is now an event: a frame is stored only when a watched signal
differs from the previous frame. Drag a node, nothing. Flip a clock, a frame.
Throw a switch while the clock is paused, a frame. The panel says so in one line
under its title (each column is one change in the circuit, not a fixed time
step), because a waveform with no propagation delay is an idealisation and I
would rather say it than have someone discover it.

Step mode fell out the way I hoped. Pause stops the timer; Step runs one tick of
the same schedule the timer would have run. With one clock that is a half-period,
which is where I keep wanting to say "one clock edge" and keep being wrong: you
need to evaluate the falling half too, or the flip-flop never sees the clock go low.

The clock bug was a sleeper

In the last post I showed this line:

const freq = allClocks[0]?.data.freq ?? 1;

Every clock ticked at the frequency of whichever clock came first. I called it a
bug nobody had hit. It was better than that. When I went to fix it I found that
there was no way to set a clock's frequency in the UI at all. freq was
stored, loaded, displayed ("1 Hz") and never editable. The bug was unreachable
from the interface and would have bitten the first person who loaded a file with
two different values.

The fix: one timer at the greatest common divisor of all the clocks' flip periods,
each clock flipping on every n-th tick. Two clocks that coincide flip in the
same render instead of one render apart. And since freq now matters, the clock
node got − and + buttons (0.25 to 8).

The latch that wasn't

This is the part I did not see coming.

I planned to add an oscillation warning, because I had read my own relaxation loop
and noticed it never checked convergence. Ten lines, I said. So I wrote a few
probes to see what it would take to make a circuit oscillate, to have something to
test the warning against. A ring of three inverters. A NAND ring with an enable.

They stayed at ?. Every one of them. A ring of inverters never leaves "unknown",
because NOT(unknown) is unknown, so there is nothing to oscillate.

Then, almost as an afterthought, I wired up the first circuit in every digital
logic course: an SR latch from two cross-coupled NOR gates.

Q  = NOR(R, ~Q)
~Q = NOR(S, Q)

Set it. Reset it. Hold it. ?, for every combination of S and R. My simulator
could not do the first memory circuit anyone learns.

Two causes, both mine.

One. Gates were strict about unknowns. This is the AND gate:

case 'AND': s1(a !== null && b !== null ? a & b : null); break;

If either input is unknown, the output is unknown. But AND(0, ?) is 0, whatever
the question mark turns out to be, and NOR(1, ?) is 0. That is the entire
mechanism by which a cross-coupled pair resolves out of nothing: one gate sees a
controlling input and is decided before the other has an opinion.

const and = vs => (vs.includes(0) ? 0 : vs.every(v => v === 1) ? 1 : null);
const or  = vs => (vs.includes(1) ? 1 : vs.every(v => v === 0) ? 0 : null);

Two. Every call to simulate() started from a blank wire map. A latch holds its
bit in its own outputs, so a blank start means it has nothing to hold. The wire
values now carry over from tick to tick, split into "what the last tick settled
on" and "what this tick is computing", the same way I already treated flip-flop
state.

That second fix has a consequence I had to think about: for a circuit with no
loops, starting from stale values changes nothing, because it settles to the same
answer from any starting point. A test runs six different input pairs
in a row through an XOR and an AND and checks every answer.

Gates have to update together

The relaxation loop also evaluated gates in place, in whatever order the nodes
were listed. With a latch, that makes the order of the file decide the answer.
It now relaxes in rounds: every gate reads a snapshot, all of them update, repeat.
A test reverses the node list and expects identical results.

And now the warning can exist

Release both inputs of a NOR latch from S = R = 1 at the same instant and each
gate sees two zeros, wants to output 1, then sees a 1, wants 0, forever. In real
silicon one gate wins by a few picoseconds and nobody knows which; that is
metastability.
In a zero-delay engine the tie never breaks. After twice-the-node-count rounds the
loop is declared oscillating, its wires are shown as ?, and the status bar says
Does not settle.

The humbling footnote: my first draft of the article claimed you could "release both
together" by clicking two switches. I clicked two switches in the real editor and the latch
landed in a perfectly definite state, because two clicks are two ticks. The only
way to get the oscillation is one switch wired to both inputs. I fixed the article.

The save button that forgot

To make sub-circuits I needed named pins. Named pins come from the labels on
INPUT and OUTPUT elements. I looked for where a user types such a label.

There isn't one.

data.label existed. The truth table read it. The test cases read it. The Verilog,
VHDL and C++ exporters read it, and even sanitised it into a legal identifier in
all three languages. My own landing page said signal names come from the labels you
typed on the canvas. The only things that ever set it were the built-in examples.

Then I read the save function:

return {
  id: n.id, type: n.data.type, x: n.position.x, y: n.position.y, value: n.data.value,
  // ...PLM fields
};

Type, position, value. A saved circuit came back without its names, without its
clock speed, without its net-label names, and without its frames at all: a frame
has no data.type, so it was written without a type and could not come back.

The fix is unglamorous. There is now a small name tag under every switch, lamp,
clock, probe and block; you click it and type. And the whole file format moved into
one module with a round-trip test: build a circuit with every kind of node, write it,
read it back, compare. Old files with the lost fields still load.

I am slightly embarrassed that export with your own signal names, a feature I had
been pleased with, was unreachable for anyone who built a circuit by hand.

Sub-circuits: flattening was the right call

I said I would flatten: when you place a block, inline a copy of its nodes into
the parent with every id rewritten to instance/original. The simulator never
learns hierarchy exists.

u1/ff        u2/ff        u1/inner/ff

The engine change was small, and the test that justified the whole exercise is
short. A block containing a toggle flip-flop, placed twice, each copy given its
own clock. Pulse copy A twice and copy B once, and check that A has toggled twice
and B once. Before, both copies would have shared one entry in the state map and
fought over it.

Two things I did not anticipate paid off for free. The path ids make waveform
labels meaningful (DIV2/D.Q, DIV2#2/D.Q) with no extra work. And because a
block is just a circuit, blocks nest, and the exporters, test cases, truth table
and waveform all see through them by calling the same flatten first.

Pins by position, and editing by name

A block's pins are its INPUT and OUTPUT elements, ordered top to bottom as they
sit on the canvas, so the block looks like the drawing it came from.

Editing a placed block was the part I said I would not do, and then did, because a
block you cannot change is a block you redraw. Double-click it and its circuit
opens on the canvas in place of yours; Done writes the change into every copy.

The interesting problem is what to do with the wires. You add a pin at the top and
every other pin's index shifts. If wires followed indices, you would silently
rewire the whole circuit. They follow names: rename a pin and its wire stays;
add a pin and the others move down with their wires; remove one and only its wire
goes. One refinement came from using it: when a name disappears but the number of
pins on that side is unchanged, it is a rename in place, and the wire stays. Renaming
in_0 to CLK is the most common edit there is, and dropping the wire for it would
have been a cruel default.

Tests that can count clock edges

The comparison with Digital had one line I wanted to close: declare inputs and
expected outputs in a table, run, get pass or fail. Mine could only do
combinational rows. Now a C in an input column is a clock pulse: the signal goes
0, 1, 0, and the outputs are checked after it has settled.

CLK | Q0 Q1 Q2 Q3
C   | 1  0  0  0
C   | 0  1  0  0
C   | 1  1  0  0

A table with a C runs top to bottom as one sequence from power-up, and x in an
input column keeps the last value. It runs on a state of its own and puts the
editor's live state back afterwards, so running a table does not stop the counter
you are watching.

I will say what it does not do: setup and hold times. A pulse is an instant. The
article about it says so.

The 404 that only production had

All of the above was checked in a browser. The landing page got a real screenshot
of the editor, and it was broken on the first deploy: alt text, no image.

The backend serves dist/ from an explicit allow-list: /assets, /donate, a few
root files. A new /img folder fell through to the SPA catch-all, which answers
every unknown path with the app shell and a 404. vite dev and vite preview serve
the whole public/ folder, so I never saw it locally. The fix was one mount; the
lesson is that "works in the preview server" and "works behind my own router" are
different statements. (It also turned out the screenshots were being sent as
application/octet-stream; the Python on the server evidently does not know
.webp, so that got a one-line fix too.)

What is still not there

Exporting to Verilog/VHDL/C++ is combinational only. Flip-flops, registers and counters are skipped, and every skipped element is flagged in the output. Tests handle a clock now; the exporters do not.
A block nested inside a block is editable only through the outer one.
Bus contention (two drivers fighting over a wire) is detected at the top level, not inside a block.
No share link and no truth-table-to-circuit synthesis.
The 1 Hz on a clock means one flip per second, so the real frequency is 0.5 Hz. I know. Fixing it would make every demo twice as fast, so it stays until I decide what the label should say.

Answers to my own questions

Last time I asked five questions, so here is what I actually did.

Flattening versus recursive evaluation. Flattening. The memory cost I worried about never showed up at this scale, and the unplanned benefit was readable path names on the waveform.
Idealised waveforms. Kept idealised, and said so on the panel. No fake unit delay.
Step granularity. One tick of the shared timer, which is half a clock period with a single clock. I did not build single-gate stepping.
Two invisible refactors out of five. In hindsight the invisible ones were the ones that exposed the most. Making time explicit led directly to noticing that the latch did not work.
Embeds versus sub-circuits. Sub-circuits. I then decided against the iframe embed entirely.

What I would tell myself before the roadmap post

I wrote a confident dependency graph after reading the simulation entry point and
the state map. That was a good read of two files. It said nothing about the gates,
the save function or the router, and every surprise in this post lived in one of
those.

The numbers, for the record: 282 automated checks across 11 scripts, all passing;
49 articles with a diagram each, in three languages; 32 examples, every one checked
for structure and the sequential ones actually run. The articles I corrected along
the way were checked by running the circuits they describe, which is the habit this
whole stretch taught me.

Boolflow is at boolflow.site: free, no account, nothing
to install. If you teach digital logic or have ever built a latch from NOR gates
and wondered whether a tool would do it, I would like to hear what breaks.

Everyone Types "yes"

Den — Mon, 12 Oct 2026 08:40:03 +0000

The smoke test for my little SSH key program is one command: clone a repository of mine over SSH. On a fresh machine, before anything is cloned, it asks this:

The authenticity of host 'github.com (<IP address>)' can't be established.
ED25519 key fingerprint is SHA256:<key fingerprint>.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

And I typed yes. Like everyone does.

This series has been about two things so far: keeping the passphrase out of ps, and checking whether a tool uses your SSH at all. The second post leaned on one defence — the wall of @@@ OpenSSH throws when a server's key changes. This post is about the uncomfortable part: that wall protects you only if the key it's comparing against was the right one. And that key got there when you typed yes.

Trust on first use

SSH's model has a name: TOFU, trust on first use. The first time you connect, the client has nothing to compare against, so it asks you. Whatever you accept goes into known_hosts, and from then on any change is an alarm.

It's a reasonable design. The weak point is obvious: if someone is between you and the server on that first connection, you accept their key, and from then on the alarm protects the impostor. Everything after the first yes is only as good as the first yes.

Most of the time this is theoretical. Then sometimes it isn't.

March 2023

On March 24, 2023, GitHub replaced its RSA SSH host key. The private half had been briefly exposed in a public repository; anyone who copied it could, in principle, pretend to be github.com to RSA clients.

So every RSA user got the wall: REMOTE HOST IDENTIFICATION HAS CHANGED. Exactly the alarm that's supposed to make you stop and think.

GitHub published the new fingerprint and instructions. But the fix that spread fastest was the short one: ssh-keygen -R github.com, connect again, type yes. Which works — and quietly accepts whatever key answered that next connection, without comparing it to anything. The alarm did its job; the habit undid it.

What "verifying" actually means

The question at the prompt isn't "do you want to connect". It's "does this fingerprint match the one you got some other way". The point is some other way — a channel an attacker on your network path can't also fake.

For the big hosts that channel exists. GitHub, GitLab and Bitbucket publish their host key fingerprints in their documentation, served over HTTPS. Don't take them from this post, by the way — a fingerprint in a blog post is exactly as trustworthy as the blog. Open the provider's docs page.

Then there's a feature I'd used for years without noticing: you can paste the fingerprint instead of typing yes. Note the [fingerprint] in the prompt. Copy the published value, paste it, and OpenSSH compares it for you. A mismatch is refused. No squinting at forty base64 characters.

Already have hosts in known_hosts and want to check them after the fact?

ssh-keygen -l -F github.com

prints the fingerprints of what you stored. Compare with the docs.

`ssh-keyscan` is not verification

In the previous post I used ssh-keyscan to plant a deliberately wrong key — that's what it's good for. It's also widely used for something it can't do: "trusting" a host in scripts and CI.

ssh-keyscan github.com >> ~/.ssh/known_hosts

This downloads the key from whoever answered on port 22. If that's an impostor, you've just written the impostor's key down as trusted, automatically, with no prompt. It's the scripted version of typing yes without reading.

GitHub's own recovery instructions in 2023 did something better: they fetched the keys from the API over HTTPS —

curl -L https://api.github.com/meta | jq -r '.ssh_keys | .[]' | sed -e 's/^/github.com /' >> ~/.ssh/known_hosts

— which moves the trust to TLS, a channel you already rely on. Better still for CI: copy the published known_hosts lines once, check them into the repository, and use that file. A pinned key changes only when you change it.

Two settings worth knowing while you're there:

StrictHostKeyChecking accept-new — accept unknown hosts automatically, but still refuse changed keys. A sane middle ground for automation.
StrictHostKeyChecking no — accept everything, including changed keys. If you find it in a pipeline, that pipeline has no host verification at all.

Closing the series

Three posts, three places the same thing goes wrong — a secret or a trust decision handled in a way that looks fine and isn't:

The passphrase goes through ssh-agent or SSH_ASKPASS, never through argv.
The SSH client is the one you configured — check that your tools read your config, check host keys, and use your agent.
The first host key is compared against a fingerprint from another channel, not accepted with yes or fetched with ssh-keyscan.

None of this needs new software. Everything in this series is already in OpenSSH — the askpass hook, the config, the fingerprint prompt, accept-new. The defaults are good. The weak spot is almost never the protocol; it's the shortcut around it.

As for my smoke test: it now starts by pasting the fingerprint.

Your Git GUI Isn't Using Your SSH

Den — Mon, 12 Oct 2026 08:28:15 +0000

In the previous post I was building a small GUI for creating SSH keys, and the first decision was how to pass a passphrase to ssh-keygen without leaking it. The second decision came right after: should the program talk SSH itself, through a library, or just run the system ssh?

I chose the system binary. Not because the libraries are bad, but because of what I found when I looked at what "use a library" actually means.

"SSH" is two different things

When you type git clone git@github.com:... in a terminal, git runs ssh, the OpenSSH client. That client reads ~/.ssh/config. It checks ~/.ssh/known_hosts. It asks your ssh-agent for keys. It follows ProxyJump through a bastion. All the behaviour you've tuned over years lives there.

Many GUI clients, IDE plugins and deploy tools don't run that binary. They link an SSH library and speak the protocol themselves. The protocol is the same. The behaviour around it is whatever the application's authors wrote.

Take libssh2, one of the common ones. It's a solid library, and it has the building blocks: a known-hosts API, an agent API, even security-key signing. But they're building blocks. Each OpenSSH behaviour you rely on becomes a feature somebody had to remember to implement. And some aren't there at all — the library has no ssh_config parser and no ProxyJump.

The most telling line is in libssh2's own example program. Right after connecting, a comment says to check the host key against your known hosts — "that is your call". The example then prints the fingerprint and carries on.

That's fine for an example. It's a problem when an application ships the same shape.

What can quietly differ

Host key verification. In the terminal, a changed server key stops you with a wall of @@@ and "REMOTE HOST IDENTIFICATION HAS CHANGED". That warning is the whole defence against someone sitting between you and the server. An embedded client might check its own store instead of known_hosts, accept any new key silently, or not check at all.

Your config. Host aliases, IdentityFile, IdentitiesOnly, ProxyJump, algorithm restrictions — if the client doesn't read ~/.ssh/config, none of it applies, and nothing tells you. You hardened something; the tool doesn't know.

Where the passphrase lives. OpenSSH hands that job to ssh-agent: the key is unlocked once, the passphrase isn't stored anywhere. An application with its own stack may offer to "remember" the passphrase — in its own settings, encrypted with who knows what.

Which keys work. Newer key types and hardware keys (ed25519-sk, ecdsa-sk) work in OpenSSH long before every embedded stack catches up.

On Windows there's a bonus version of this: Git for Windows ships its own ssh.exe, separate from the OpenSSH built into Windows. Two clients, two configs in practice, two agents. A key added to one agent is invisible to the other.

Check your own tools

I'm not going to name clients here — versions change, and what I'd write today could be wrong next month. Instead, here's how to find out about the tools you actually use. Each check takes a minute.

1. Does it read ~/.ssh/config? Add an alias:

Host gh-alias
    HostName github.com
    User git

ssh -T gh-alias should greet you. Now clone gh-alias:you/some-repo.git in the GUI. If it fails with "could not resolve host", it isn't reading your config. (ssh -G gh-alias shows exactly what OpenSSH would use, for comparison.)

2. Does it check host keys? This is the important one. Back up your known_hosts, then put a wrong key under github.com's name — GitLab's key will do:

cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak
ssh-keygen -R github.com
ssh-keyscan -t ed25519 gitlab.com 2>/dev/null | sed 's/^gitlab.com/github.com/' >> ~/.ssh/known_hosts

ssh -T git@github.com must now refuse loudly. Fetch from a GitHub repo in the GUI. If it connects without complaint, it would connect just as happily to an impostor. Restore afterwards:

mv ~/.ssh/known_hosts.bak ~/.ssh/known_hosts

3. Does it use your agent? With a passphrase-protected key, empty the agent with ssh-add -D and fetch in the GUI. If it connects without asking for the passphrase, it has the passphrase stored somewhere. Find out where.

What to do with the answer

If a tool fails these checks, look for a setting before looking for a replacement. Many have an option like "use system SSH" or "use system Git", and plain git lets you choose the client outright with core.sshCommand or GIT_SSH_COMMAND. Tools that simply run your system git are already using your system ssh.

For my own little program that settled it. Running the system ssh means every check above passes by construction — not because I implemented them, but because I didn't have to. It also means one less library to track for security fixes.

The safest SSH code is the SSH code you didn't write. Before trusting a tool with your keys, spend three minutes checking whose SSH it's actually using.

ssh-keygen -N Puts Your Passphrase in ps

Den — Mon, 12 Oct 2026 08:19:46 +0000

I'm writing a small GUI that walks a person through creating an SSH key. The core of it is one call to ssh-keygen, and on paper that call is trivial:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N 'correct horse battery staple'

The user types a passphrase into a nice field, the program puts it after -N, the key appears. Most scripts and wrappers I've looked at do exactly this.

It's also the one way of doing it that I ruled out first.

argv is public

Command-line arguments are not a private channel between you and the program you start. On Linux they live in /proc/<pid>/cmdline, and by default that file is readable by every user on the machine. ps just prints it:

$ ps -eo user,args | grep '[s]sh-keygen'
alice  ssh-keygen -t ed25519 -f /home/alice/.ssh/id_ed25519 -N correct horse battery staple

"But it only runs for a moment." A passphrase-protected key isn't instant to write — ssh-keygen runs a deliberately slow key derivation to encrypt it — and anyway "the window is short" is not a security property. A loop over ps costs nothing to leave running.

And ps is only the most obvious place. The same string can end up in shell history if anyone ever runs the command by hand, in auditd logs that record every execve with its arguments, and in any process-monitoring agent that ships command lines to a dashboard.

A secret in argv has already left the process. The question is only who reads it.

(You can mount /proc with hidepid=2 so users only see their own processes. It's a good setting. It's also not something a desktop app can assume about the machine it runs on.)

Three ways to hand over a passphrase

Once -N is out, you need another channel. I found three approaches in the wild.

1. -N in argv. Covered above. Out.

2. A pseudo-terminal. Start ssh-keygen inside a pty, wait for it to ask, type the answer. This is what expect scripts do, and it works. Two problems for me.

The first is practical: Qt's QProcess doesn't do pseudo-terminals, so I'd be writing forkpty handling by hand.

The second is worse. The program now depends on the exact text of the prompt. And that text changes. On Ubuntu 22.04, which ships OpenSSH 8.9, key generation asks:

Enter passphrase (empty for no passphrase):

Current OpenSSH asks:

Enter new passphrase (empty for no passphrase):

One word. A wrapper that waits for the old string on a newer system just hangs, with no error, waiting for a question that has already been asked differently.

3. SSH_ASKPASS. This is the mechanism OpenSSH itself provides for exactly this case. You point the environment variable at a helper program; when ssh-keygen needs a passphrase, it runs the helper with the prompt as its only argument and reads the answer from the helper's stdout. Seahorse, GNOME's key manager, does it this way — it sets SSH_ASKPASS to its own helper and calls setsid() so the child has no controlling terminal to fall back on.

There used to be a catch: ssh-keygen would only use the helper under certain conditions — a display present, no terminal to ask on. OpenSSH 8.4 added SSH_ASKPASS_REQUIRE=force, which means "always use the helper, no matter what". Ubuntu 22.04 has 8.9, so for my target it's simply available.

You can try it from a terminal with a one-line helper:

#!/bin/sh
# ~/bin/askpass-zenity
exec zenity --password --title="$1"

SSH_ASKPASS=~/bin/askpass-zenity SSH_ASKPASS_REQUIRE=force \
  ssh-keygen -t ed25519 -f ~/.ssh/id_demo

A dialog pops up, twice: once for the passphrase, once to confirm it. Nothing in argv, nothing typed into a terminal you don't control.

The helper has to get the passphrase from somewhere

This is the part that's easy to get wrong after getting everything else right.

If your app already collected the passphrase in its own window and now launches the helper as askpass-helper 'correct horse...', you've moved the secret from ssh-keygen's argv to the helper's argv. Same ps, same leak, one process later.

The cleanest version is the one Seahorse uses: the helper is the dialog. It asks the user directly and prints the answer. The passphrase never passes through the parent process at all — your app doesn't hold it, log it, or forward it.

If the design really requires the main window to collect it, hand it over through something that isn't world-readable:

A pipe or a Unix socket in a directory only your user can access, with the path in an environment variable. The helper connects, reads, prints, exits.
An environment variable with the passphrase itself is the fallback. It's not great, but it's a different class of exposure: /proc/<pid>/environ is readable only by the process owner and root, unlike cmdline.

Whichever you pick, remember the helper is called twice for a new key — once per prompt — and has to answer both.

It isn't about ssh-keygen

ssh-keygen just happens to be where I met it. The same shape shows up everywhere a tool accepts a secret as an argument: mysql -p<password>, sshpass -p, curl -u user:password. All of them have a quieter alternative — a config file, an environment variable, a prompt, a helper — and all of them still appear in scripts with the secret inline, because the inline version is the one that fits on one line in a README.

If a secret is on the command line, assume it's in a log. Then find the channel the tool's authors built for the case where it shouldn't be — there almost always is one.

My News Digest Got a Telegram Channel That Forgets Everything After a Week

Den — Thu, 08 Oct 2026 16:36:36 +0000

A while back I posted here about RealFeedApp, a free digest that collects science and tech news, collapses the retellings into one card, ranks by significance and writes a short summary in your language. Last time it learned to explain itself: every card got three plain-language lines under the summary.

This post is about what happened since. The headline is a Telegram channel, and the most interesting thing about it is a setting: every post deletes itself after a week.

The first version was a wall of text

The idea was obvious. Search engines take months to notice a new site; a channel reaches people the day you post. So the first version did the obvious thing: once a day, the five top stories of the last 24 hours, as one message. Headline, summary, link to the original, repeat five times.

It worked, technically. I opened the first post in the channel and was disappointed.

It was a block of text. No pictures. None of the "in plain words" lines the site had just learned to write. No tags, no sense of which field a story belonged to. And every link went straight to the original source, so the site — the thing that actually does the work — was nowhere in it.

Nobody reposts a wall of text. You can't forward the one story you care about; it's glued to four others. And a reader who skims it learns nothing they couldn't have got from any aggregator.

The lesson was simple and slightly embarrassing: I had built a channel that showed less than the product it was supposed to advertise.

One story, one post, the whole card

So the channel now posts the way the site shows things. One story per message, spread through the day instead of a morning dump. Each post is the card:

the cover image from the site;
the headline and the summary;
the "in plain words" block — the field as a tag, what was done, why it matters — set off from the summary the way it is on the card;
tags you can tap through within the channel;
a link to the story's page on the site, and from there to the original;
the same story in Russian, folded into a collapsed quote at the very bottom.

The link to the site matters most. A post is now a preview of a card, not a replacement for it. If the story is yours, you forward that one post. If you want more, you land on the site, where the rest of the feed is.

Two small rules came out of looking at real posts before they went out. A story whose translation came out broken doesn't get posted at all — a broken word in a feed is a bug; in a public post it's a broken word in front of every subscriber at once. And the channel leans towards news over dense preprints: the feed is for people who opted in to a field; the channel is for people who just followed a link.

One channel, two languages

The site writes every summary in English and Russian, but the first channel was Russian only. That cut out half the readers. Two channels would have split the subscribers instead, and doubled the posting for a project run by one person.

So each post is now an English card with the Russian version inside it, as a collapsed quote: one line and an arrow until you tap it. An English reader never scrolls past Russian text; a Russian reader is one tap away from the full card in their language, with its own link to the Russian page of the story.

Fitting two languages into one post had a side effect. A photo caption in Telegram is short, and one card already filled it. So posts went from "photo with a caption" to a text message with the cover shown as a large preview above it. It looks the same, and there is room for both languages.

Why the posts disappear

Here's the setting I mentioned at the top: the channel's messages auto-delete after a week.

It sounds like throwing work away. It's the opposite.

A channel that keeps everything slowly becomes the product. Scroll back far enough and it's a second, worse feed: no search, no topics, no "hide this", no memory of what you liked, and nothing that gets better the more you use it. If that archive grows forever, it starts competing with the site and the app — and it wins on convenience while losing on everything else.

So the channel is a window, not a home. It shows what's happening this week, and points at the place where the rest lives. A week-old science headline is stale anyway; anything that still matters is on the site.

What else is new

RSS. Every story that goes to the channel also goes into an RSS feed: /en/rss.xml and /ru/rss.xml. The footer has a one-click "Follow on Feedly" link. If you'd rather not use Telegram at all, it's the same stream of hand-picked stories in your reader.

Bluesky. The same stories go to @realfeedapp.com as short threads: the summary with a link card to the story page, then what they did and why it matters as replies. English only — that's where a lot of the science crowd moved, and topic feeds there show posts beyond your followers.

Story pages. Every story now has its own page on the site: cover, summary, the plain-words block, tags, and a clear button to the original. That's where channel posts lead.

A daily archive. Each finished day gets a page with its top stories, open to everyone, no account needed — and an archive that lists the days. If you missed a week, that's the fastest way to catch up.

Links in your language. The feed now has its own address per language (/en/, /ru/, …). Share the Russian version with a friend and they get the Russian version, regardless of what their browser says.

Keywords from a vocabulary. Your own keywords are no longer typed in free text; you pick them from the terms the feed actually uses, and the screen explains what they change. Free text looked flexible and mostly matched nothing.

A quieter, more honest daily digest in the app. The notification gives you one reason to open it instead of a list nobody can read in a notification shade; the app tells you when the next one arrives; and the last digest can be reopened from settings.

A cleaner feed. Threads and discussions no longer show up as news — nothing happens in them, someone just talks about something that did. A story is filed by what it's about, not by where it came from. And a headline that says the opposite of its own summary, or names something the article never mentions, doesn't reach the feed.

Try it

Still free.

The site: realfeedapp.com — a preview of the feed with no account; signing in with Google opens the rest and remembers your topics.
The channel: t.me/realfeedapp — English cards, with a Russian version inside each post.
Bluesky: @realfeedapp.com.
RSS: realfeedapp.com/en/rss.xml, or "Follow on Feedly" in the site's footer.

My News Digest Got a Telegram Channel That Forgets Everything After a Week

Den — Mon, 05 Oct 2026 12:59:31 +0000

This post is about what happened since. The headline is a Telegram channel, and the most interesting thing about it is a setting: every post deletes itself after a week.

The first version was a wall of text

It worked, technically. I opened the first post in the channel and was disappointed.

Nobody reposts a wall of text. You can't forward the one story you care about; it's glued to four others. And a reader who skims it learns nothing they couldn't have got from any aggregator.

The lesson was simple and slightly embarrassing: I had built a channel that showed less than the product it was supposed to advertise.

One story, one post, the whole card

So the channel now posts the way the site shows things. One story per message, spread through the day instead of a morning dump. Each post is the card:

the cover image from the site;
the headline and the summary;
the "in plain words" block — the field as a tag, what was done, why it matters — set off from the summary the way it is on the card;
tags you can tap through within the channel;
a link to the story's page on the site, and from there to the original.

That last change matters most. A post is now a preview of a card, not a replacement for it. If the story is yours, you forward that one post. If you want more, you land on the site, where the rest of the feed is.

Why the posts disappear

Here's the setting I mentioned at the top: the channel's messages auto-delete after a week.

It sounds like throwing work away. It's the opposite.

What else is new

Story pages. Every story now has its own page on the site: cover, summary, the plain-words block, tags, and a clear button to the original. That's where channel posts lead.

Try it

Still free.

The site: realfeedapp.com — a preview of the feed with no account; signing in with Google opens the rest and remembers your topics.
The channel: t.me/realfeedapp. It's in Russian for now, since that's where Telegram is strongest for me; the site and the daily archive are in English too.

Two things I'd like to hear:

Would you forward one of these posts? That's the whole test for the new format, and I can't run it on myself.
Is a week the right lifetime? Shorter makes the channel feel like a ticker; longer and it starts turning into the feed I was trying not to build.

A Good Summary Still Left Me Guessing

Den — Sun, 13 Sep 2026 10:46:44 +0000

It worked. I stopped drowning. And then I noticed I was doing something odd with it: I'd read a summary, understand every word of it, and still have no idea whether the thing mattered.

What a summary doesn't do

A summary compresses an article. It cannot supply the thing the article assumed you already had.

Every piece of technical writing is addressed to someone. A paper on gene therapy is written for people who know what a vector is; a post about attention layers is written for people who have already met attention. The author skips the setup, and they're right to — their reader has it.

Compress that article faithfully and you get a shorter text with the same hole in it. Here's the shape of it, with the details changed:

Researchers report that the modified vector achieved durable expression in the target tissue without the immune response seen in earlier trials.

Every word true, nothing wasted. And if that isn't your field, you've learned nothing you can act on. Is this a lab curiosity or the thing the field has been stuck on for a decade? The summary can't say, because the article never said — the article was talking to someone who knew.

This is the actual cost of reading widely. Not volume, which is what everyone builds for. Volume is solved by filtering. Context is not, because the missing part was never in the text you filtered.

What's new: three lines under the summary

So every card now carries a second block, under the summary and behind a divider. Three labelled rows, no heading:

Field — a short tag: how neural networks work, gene therapy, hardware and chips, energy and batteries.

What was done — the concrete thing, in ordinary words.

Why it matters — what changes if it holds up.

Same three rows on every card, in the same order, in your language. The point isn't more text. The point is that the same three questions get answered every time, so you can skim the third line across ten cards and stop at the one that deserves you.

A few decisions inside it that changed how it reads:

The field is a tag, not a sentence. It comes from a fixed list of areas, so the same subject is worded identically on every card. That's what makes it scannable — and it's why it can become a filter later. Free-form text would have been easier to build and useless the first time the same field got named two different ways.

No heading above the block. It's three labelled rows; the labels already say what it is. A heading would have eaten the space the explanation needed.

It doesn't appear where there's nothing to explain. If the linked page had no article on it — a video, a page that's really an app, a PDF that yields nothing — there's no block. There's also no card: pages with nothing to read stopped reaching the feed at all, which fixed a smaller embarrassment I'd been shipping for a while. Summaries of "this page requires JavaScript" are, it turns out, very fluent.

That last rule matters more than the feature. An explanation of nothing is worse than no explanation, because it looks exactly like the real thing and you can't tell without opening the link — which is the one thing the card exists to save you.

What it doesn't do

It doesn't read the article for you. Three lines are enough to decide, not enough to know, and the link to the original is still the whole point of the card.

In your own field it will occasionally explain something you learned a decade ago. That's the trade: the block is written for the reader who wandered in from three fields over, and I'd rather it be redundant for the specialist than absent for everyone else.

Try it

Still free. Open the link and you get a preview of the feed with no account at all; signing in with Google opens the rest of it and remembers your topics. realfeedapp.com

The Android app is in closed testing; it'll reach Google Play once I'm confident it behaves.

Two things I'd like to hear if you open it:

Does the third line earn its space? "Why it matters" is the row I'm least sure about — it's the one where an honest we don't know yet would be more useful than a confident sentence, and I don't know yet whether it's doing that.
Where does the field tag fit badly? The list of areas is fixed, so somewhere there's a story it can only be wrong about. I'd like to know which one before a reader has to.

The Float Is a Liveness Probe: Four Failure Modes

Den — Thu, 10 Sep 2026 19:53:35 +0000

I cook pelmeni about twice a week, and for years I got inconsistent results from what is, on paper, a trivial process: put frozen parcels in boiling water, take them out when done. Two failure modes kept recurring — parcels welded to the bottom of the pot, and parcels that burst and leaked their filling into the water.

Both turned out to be the same class of mistake I make at work, which is why I'm writing this down.

Failure one: the batch was too large

Dropping a large batch of frozen parcels into boiling water stops the boil. The water is at 100°C, the parcels are at −18°C, and there's only so much thermal energy in the pot. The temperature drops below boiling, the surface starch doesn't set, and the dough touches the metal while it's still sticky. That's the welding.

I had been treating pot capacity as a volume question — will they fit — when it's actually a throughput question: how much cold can this system absorb before it stops doing the thing it's for. A pot that fits thirty parcels comfortably processes twenty of them correctly.

Capacity is not the same as the load at which quality holds. You can size a worker pool by what fits in memory and still be wrong, for exactly this reason: the system keeps accepting work long after it stopped doing it well.

Failure two: I trusted the wrong signal

Everyone knows the rule: pelmeni float when they're done. Everyone is wrong, and I was wrong for a long time.

They float when the air and steam trapped inside expand enough to make the parcel buoyant. That happens when the dough is hot. It says nothing about the filling, which is denser, colder, and at the centre of the parcel — the last part of the system to get warm. Pull them at the moment they surface and the outside is finished while the inside is raw.

Floating is a liveness signal. It tells you the process is running. It is not a readiness signal, and the difference between those two is exactly the difference between "the container started" and "the container can serve traffic". Confusing them ships raw meat.

The actual readiness signal is time after the liveness signal — a few minutes at a gentle boil once they've surfaced. Which is an unsatisfying answer, because it's a timer, and everybody wants an event.

When you can't observe the thing you care about, you time it from the last thing you can. That's not a workaround, that's the design. The mistake is pretending the observable proxy is the real state.

Failure three, which I still make

The lid. A lid gets you back to boiling faster after the cold batch goes in, which is good, and then the starch-laden foam climbs the sides and comes over the edge, which is not.

The correct move is to leave the lid slightly ajar — you sacrifice some of the recovery rate to keep the process inside the pot. This is a backpressure trade and it has the same shape everywhere: maximum throughput and stability are different set points, and choosing throughput means you will eventually clean the stove.

What the burst parcels were really telling me

The leaking-filling failure isn't overcooking in the ordinary sense. It's what happens when a parcel stays under a hard rolling boil: mechanical agitation, not heat, tears the seam. The fix isn't less time, it's less turbulence — drop the heat once the parcels are in and let it finish at a bare simmer.

I had been tuning the wrong parameter for months because the symptom appeared at the end of the process, so I assumed it belonged to the end of the process. It belonged to the environment the whole way through.

A failure that shows up last isn't caused last. Applies to dinner and to production incidents in roughly equal measure.

Failure four: I was salting for the wrong reason

I used to salt the water for flavour. It does that, but the more important effect is thermal: dissolved salt raises the boiling point substantially, so a well-salted pot runs hotter than a plain one and gets you through the cold-batch recovery noticeably faster. Two tablespoons per litre is roughly where the gain stops being worth the taste.

This is the one that changed my routine the most, because it means the salt goes in before the water boils rather than with the parcels.

So: batch smaller than capacity, don't trust the float, lid ajar, simmer not boil, salt early.

One of the four claims above is wrong — confidently stated, cheerfully believed by me for years, and demonstrably false. I know which one. I'm curious whether anyone catches it, and whether this kind of thinking about cooking is a useful frame or just professional deformation. What do you think?

Language Models and Languages: Why "Multilingual" Is an Illusion of Equality

Den — Thu, 10 Sep 2026 15:24:46 +0000

When people see ChatGPT or Claude answer fluently in Swahili or Thai, the natural impression is that the model "knows" all languages equally well. It doesn't — and the gap isn't cosmetic. It runs straight through the architecture of how these models read and process text. What follows is a breakdown of where that gap actually comes from, how large it is, and what's being done about it.

1. The problem starts before the model "thinks" — at tokenization

An LLM never reads text directly. Text is first cut into pieces (tokens) by a tokenizer, and only those tokens are what the model actually processes. Tokenizers are trained on corpora where English historically dominates — in Llama 3's training data, for instance, English and code made up 95% of the dataset, while every other language combined accounted for just 5%.

Because of this skew, the tokenizer ends up "efficient" for English and noticeably less efficient for everything else: models generate text one token at a time, and for languages poorly represented in the tokenizer's vocabulary, this makes generation slower and more computationally expensive.

How big is the gap in practice? One study covering 25 European languages and ten models put a concrete number on it: tokenization "fertility" ranges from roughly 1.2 tokens per word for English to roughly 3.1 for Greek and Maltese — a penalty of about 2.5x. Notably, the ranking of "efficient" vs. "inefficient" languages barely changes regardless of register — formal or casual text — with a correlation above 0.97 across styles.

A similar picture shows up for languages using a different script. A comparison of several tokenizers on equivalent Polish and English text found that where Polish required 547 tokens, English needed only 377 on the same model (Mistral Small 3.2) — and this gap repeats across most tested models.

2. Why this isn't just about speed — it's about money and quality

Tokenization inefficiency isn't only "responses in non-English are slower." Three effects stack on top of each other:

Cost. Since API pricing is billed per token, not per character, the same amount of meaning costs literally more in a "token-expensive" language. One breakdown describes a real case: a team running multilingual support saw its API bill triple in two weeks with no change to the prompt — the Brazilian and Indonesian user share simply grew, and their requests tokenized at 1.6x the per-character cost of English under the older vocabulary. Switching tokenizers cut the per-character token cost for those languages by 35% — without touching the model or the prompt.

Context window. If the same meaning takes several times more tokens in another language, a model with a fixed context limit (say, 128k tokens) effectively fits much less real text for that language — less of a document, less conversation history.

Answer quality. This isn't just a "more expensive" effect — it's also a "worse" effect. There's direct evidence of a correlation: the higher a language's tokenization compression ratio, the lower the model's task accuracy in that language. In other words, inefficient tokenization isn't a neutral side effect — it correlates with a real drop in quality.

3. The "reasoning" gap: where the ability to think actually gets lost

Here's where it gets more interesting than a simple "more tokens, therefore worse."

One key study (the HRM8K benchmark, Korean and English) tested the hypothesis head-on: maybe it's not tokenization, maybe the model simply "reasons" worse in non-English? The answer turned out to be more specific: the performance gap stems primarily from difficulty comprehending non-English input, not from a limitation in the reasoning capability itself. The authors also directly challenged an earlier recommendation to force the model to reason in English regardless of the question's language — it turns out models are heavily influenced by the language of the input itself, not just the language of the reasoning chain.

A similar picture shows up in domain-specific benchmarks. On an Irish-language benchmark built from real exam questions, the best-performing model scored 76.2% accuracy in English versus 55.8% in Irish, and produced a valid Irish response (one that didn't drift into English or break the language) less than 80% of the time.

One nuance that's often missed: the performance gap isn't uniform across task types. Evidence suggests the gap is especially pronounced in reasoning tasks dominated by high-resource languages (English foremost), but this pattern doesn't hold for factual questions or broader real-world user queries. A model might know the capital of Thailand perfectly well in Thai, but noticeably struggle with a multi-step logical problem in the same language.

4. What happens inside the model when it switches languages

Worth approaching this from another angle — not through output benchmarks, but through what happens inside the model during generation. Anthropic has direct data on this, obtained through interpretability methods (an "AI microscope" — a way of looking directly at a model's internal activations).

The key finding isn't about English as an "internal language of thought," but something more abstract: when analyzing how the model processes opposite-meaning words across languages, researchers found that the same internal features for "smallness" and "oppositeness" activate regardless of which language the input text is in. In other words, the model develops a shared, language-independent conceptual space, and any specific language is more of a "surface encoding" layered on top of that shared representation.

This doesn't mean all languages are represented equally well in that space — the shared space exists, but the quality of "access" to it through a given language still depends on how richly that language was represented during training. It's important not to conflate the two: the existence of a shared conceptual space explains why the model can transfer knowledge across languages at all, but it doesn't erase the fact that input in a less-represented language still passes through a noisier, less precise "channel" into that space.

A separate, fairly unexpected finding from the same Anthropic research concerns not capability but behavior: they found that the values the model expresses shift systematically depending on the language of the conversation — for example, in English the model leans toward caution, rigor, depth, and candor, while in Arabic it leans toward deference, warmth, brevity, and task execution. This isn't about intelligence — it's that the model's "character" isn't language-neutral at all. Training data in different languages carries different cultural and stylistic norms, and the model absorbs them differently for each language.

5. A busted myth: "write your prompt in X, it's cheaper and more efficient"

Developer communities occasionally circulate advice like "write code-generation prompts in Chinese — it saves up to 40% in tokens." A recent study tested this directly on a real engineering benchmark (SWE-bench Lite) across several models and got an unambiguous result: the claimed Chinese token efficiency advantage did not hold up.

This is a useful lesson on its own: the intuition that a "more compact" script (logographic characters versus Latin letters) should automatically be more efficient for a model doesn't hold. A tokenizer isn't just a character counter — it's a separately trained system with its own quirks, and you can't predict its behavior by eye without actually measuring it.

6. Where the fix is heading

Two directions stand out as attempts to solve this at the architecture level rather than patch it after the fact.

Dropping tokenization altogether. Meta's Byte Latent Transformer doesn't tokenize input into discrete vocabulary IDs at all — instead it dynamically groups bytes into variable-length patches during training, with patch boundaries set by an entropy model. At comparable compute, an 8B-parameter BLT model matched a BPE-based Llama 3 8B baseline on standard benchmarks, while handling typos, code, and low-resource languages noticeably better. The point is that if a model has no privileged token vocabulary built on an English corpus in the first place, the structural inequality between languages disappears at the architecture level.

Targeted post-training instead of architectural change. A more practical — and far cheaper — path is not rebuilding the model from scratch, but fine-tuning it specifically to close this gap. One approach (UST — Understand, Solve, Translate) uses English as an anchor language for reasoning and solution generation, and on a synthetic dataset of 130,000 examples achieved a 10.91% improvement on the HRM8K benchmark, cutting the cross-language gap from 11.6% down to 0.7%. Industry analysis backs this up: tokenizer inefficiency and English-centric reasoning account for 70-80% of model failures in non-English languages, and targeted post-training looks like the most cost-effective fix compared to a full architectural overhaul or scaling up model size.

What this means in practice

If you're building a product for a multilingual audience — whether that's RealFeedApp serving news in multiple languages, or any other LLM-backed service — the language gap needs to be designed into the architecture up front, not filed under "minor localization detail":

Budget by tokens, not characters, separately for each target language — the cost for the same amount of meaning can differ by 2-3x
Don't rely on "smart" default tokenization — the model/tokenizer choice for a multilingual product should be explicitly tested against your target languages, not assumed to be equally good everywhere just because the model is labeled multilingual
For tasks requiring multi-step reasoning (not facts, but logic, math, planning) in lower-resource languages, expect a noticeably bigger quality drop than you'd see on simple factual questions in the same language
English as a bridge language is a proven, working pattern (translate → reason in English → translate back), but it adds latency and cost, and it isn't right for every case — where cultural specificity in the response matters, round-tripping through English can flatten it out

The tokenizer is the least visible part of an LLM application's stack, which is exactly why it's the easiest thing to inherit unexamined along with your model choice. In practice, it's arguably the single most consequential hidden factor in how well your product performs outside of English.

Sources and further reading:

Anthropic, "How Claude's values vary by model and language" (research)

One API, Three Clients, and the Validation Rule That Only One of Them Obeyed

Den — Wed, 09 Sep 2026 16:33:53 +0000

The validation logic had been solid for months. Pydantic schema on the request, a service-layer check for the actual business rule, tests green across the board. Then I added a CLI tool that talked to the same API the web app used, and within a week a support ticket came in: a record existed in the database that should have been structurally impossible.

It wasn't a validation bug in the sense of "the code checks the wrong thing." The check was correct. It just wasn't reachable by every path that could create that record.

The setup

Three clients hit one FastAPI backend: a React web app, a mobile app, and a small internal CLI I'd written for bulk imports. All three eventually call the same endpoint:

@router.post("/orders", response_model=OrderOut)
async def create_order(
    payload: OrderCreate,
    service: OrderService = Depends(get_order_service),
):
    order = await service.create(payload)
    return order

OrderCreate is a Pydantic model. The service checks stock, applies discount rules, all the things I described in my last post on how I structure a FastAPI service. On paper, every client goes through the same endpoint, so every client gets the same validation. That was the assumption I hadn't actually tested.

Where the assumption broke

The CLI tool didn't call /orders. It called a lower-level /orders/bulk endpoint I'd added later for the import job, because bulk creation needed to skip some per-request overhead and batch the database writes. I wrote /orders/bulk fast, under deadline, and had it call OrderService.create directly for each row — except I'd copy-pasted an earlier version of the service before a discount-rule fix had landed. The endpoint had its own BulkOrderCreate schema, structurally similar to OrderCreate, validated by Pydantic just as strictly. It looked, at the schema level, exactly as safe as the main endpoint.

The gap wasn't in what got validated. It was in which code path validated it. Two endpoints, two services (one stale), one business rule that existed in only one of them.

/orders        → OrderService.create()       (current rules)
/orders/bulk   → OrderService.create() [old]  (missing a rule added later)

Nothing here shows up in a Pydantic schema diff, because both schemas were fine. It doesn't show up in a service unit test either, unless you happen to test both call sites against the same rule — and I hadn't, because I'd been thinking of "the service" as one thing when it was actually two copies that had quietly diverged.

The actual failure mode

Multi-client APIs don't usually break validation by having a client skip it outright — most frameworks make that hard to do by accident. They break it by having a second path to the same write, added later, under different pressure (performance, batch size, a deadline), that re-implements or re-imports the logic instead of routing through the one place it's defined.

Each new client is a reason to add a new endpoint shape: the mobile app wants a lighter payload, the CLI wants batching, an internal script wants to skip auth. Every one of those is a legitimate reason. None of them is a reason to duplicate the rule that decides whether the write is allowed — only the rule that decides how the request is shaped.

What actually fixed it

The fix wasn't more validation. It was fewer entry points to the rule:

class OrderService:
    async def create(self, data: OrderCreate) -> Order:
        return await self._create_validated(data)

    async def create_bulk(self, items: list[OrderCreate]) -> list[Order]:
        # same rule, batched — not a second copy of the rule
        return [await self._create_validated(item) for item in items]

    async def _create_validated(self, data: OrderCreate) -> Order:
        if not await self.inventory.has_stock(data.sku, data.quantity):
            raise InsufficientStockError(data.sku)
        return await self.orders.save(Order.from_create(data))

/orders/bulk now calls create_bulk, which funnels every item through the exact same _create_validated, not a re-implementation of it. The batching problem — do the writes efficiently — got solved without touching the validation problem — is this write allowed. They'd been tangled together in the original bulk endpoint because both were solved at the same time under the same deadline, which is usually when this kind of split happens.

The question I ask now

When I add a new endpoint for a new client, I ask one thing before writing it: is this a new way to shape the request, or a new way to reach the write? If it's the former, a new schema and a thin adapter into the existing service is fine. If it's the latter — if I'm about to write _create a second time because the existing one doesn't quite fit — that's the moment the rule needs one home, not two.

Pydantic will happily validate two different schemas that both feed the same broken assumption. It has no way to know they were supposed to agree.

I write about things that break while I build Boolflow and RealFeedApp. If you've dealt with divergent validation across clients differently — a shared validation service, contract testing across endpoints — I'd like to hear how.

The Timestamp Bug That Only Showed Up East of Me

Den — Wed, 09 Sep 2026 08:05:52 +0000

RealFeedApp sends a daily digest — a batch of science and tech articles, generated once a day, timestamped so users can see when it was built. In testing, the timestamps were always right. In production, a slice of users kept seeing yesterday's digest labeled as today's, or today's labeled as tomorrow's, and it correlated with nothing I was looking at first: not load, not the queue, not the digest generation job itself.

The correlation I eventually found was location. Users far enough east of me saw it. Users near my own timezone didn't.

Where the bug actually was

The generation job ran on a fixed schedule, server-side, and stamped each digest with datetime.now() at creation time — no timezone attached, just a naive local datetime in the server's own zone. The client received that timestamp and displayed it directly, formatted for readability but not reinterpreted.

That's fine as long as "server's local time" and "user's local time" are close enough that nobody crosses a date boundary between them. They weren't. A digest generated at 11:47 PM server time was already the next calendar day for a user several hours further east — and depending on exactly when within that window someone opened the app, the displayed date flipped a day forward or stayed a day behind what they expected, seemingly at random from their side.

I spent the first stretch of debugging looking at the generation job itself — timing the cron trigger, checking for double-runs, checking whether two digests were somehow getting created in the same window. All of that was fine. The job ran once, on schedule, every time. The bug wasn't in when the timestamp was created. It was in what the timestamp meant once it left the machine that created it.

The fix

I stopped treating "server time" as a reasonable default and moved the boundary earlier: every timestamp is generated, stored, and passed between services as UTC, full stop. No naive datetimes cross a boundary — not into the database, not into an API response, not into a log line I might later use to debug something else.

# before — implicitly server-local, meaning depends on where "server" is
created_at = datetime.now()

# after — explicit, unambiguous regardless of who reads it or from where
created_at = datetime.now(timezone.utc)

Formatting into a user's local time became the client's job, done once, at the very last step before something hits the screen — not baked into the value at creation time, where "local" silently meant "local to a machine the user has never heard of."

What actually changed in how I write timestamp code now

The rule I follow now: a naive datetime is not a smaller version of a timezone-aware one — it's a different, less specific piece of information, and the two shouldn't be allowed to touch without an explicit conversion in between. Before this bug, I treated datetime.now() as a shorthand I'd get around to fixing "for real" later, once localization mattered. It already mattered. I just hadn't run into the user who proved it.

The debugging cost here wasn't in fixing the timestamp — that part took an hour. It was in the two days before that, spent checking whether the job that created the value was broken, when the value it created was never wrong on its own terms. It only became wrong the moment something without my server's specific context tried to read it.

That's the pattern I keep running into, whatever the bug: the code that produces a value is rarely lying. It's usually the code somewhere downstream, assuming it knows something about that value's context that was never actually promised.

The follow-up bug UTC alone doesn't fix

A reader pointed out something I'd missed, and it's worth adding here rather than pretending the UTC fix was the whole story: UTC solves the problem for an instant — a single point in time, unambiguous everywhere. It doesn't solve the problem for a calendar date, and those are genuinely different questions once you're storing data rather than just comparing two instants.

Something like a daily streak or "which day did this happen on" doesn't actually want an instant — it wants a local date, and a local date only exists relative to a specific timezone. If you take a UTC instant and later collapse it down to a bare date without keeping the offset it happened at, you've thrown away the one piece of information you'd need to recover "which day, for this specific user" — and you can't get it back afterward, no matter how carefully you generated the UTC value in the first place.

The fix for that layer is one step further than "store everything in UTC": store the UTC instant and the offset the client reported, and derive the local date from both, on demand, rather than baking a date in at write time. Storing a bare date turns out to be its own mistake — a different one from the naive-datetime mistake this post started with, but caused by the same underlying habit: treating a piece of information as more specific than it actually is.

I write about things that break while I build Boolflow and RealFeedApp.

I Stumbled on Anthropic's "Persona Selection Model" Paper — Here's My Take

Den — Tue, 08 Sep 2026 17:14:14 +0000

I wasn't looking for this. I was reading through Anthropic's alignment blog for unrelated reasons and ran into a post called "The Persona Selection Model: Why AI Assistants might Behave like Humans", by Sam Marks, Jack Lindsey, and Christopher Olah. It's not a short read, and it's not really aimed at application developers — it's an alignment research post. But it reframes something I use every day (an LLM API, prompted to behave a certain way) in a way that changed how I think about what's actually happening when I write a system prompt.

The short version

The core idea, which the authors call the persona selection model (PSM), goes like this:

During pre-training, a language model learns to predict text from an enormous range of sources — books, forums, code, dialogue. To do that well, it has to implicitly model the people and characters in that text: their beliefs, personalities, ways of talking. The authors call these learned character-models "personas." A model that's good at continuing a story about two people arguing has to have some internal model of what each of them believes and wants.

Post-training (RLHF, instruction tuning, whatever a given lab calls it) doesn't build a new capability from scratch. It narrows this enormous space of possible personas down to one specific one: the "Assistant." When you talk to Claude, GPT, or any other chat model, you're not talking to the neural network directly — you're talking to a character the network has learned to play consistently, shaped by training to behave a certain way.

The distinction the authors keep drawing: the model (the neural network) and the Assistant (the character it enacts) are different things. Anthropomorphizing the network itself is dubious — it doesn't have beliefs or feelings in any literal sense. Anthropomorphizing the Assistant is, they argue, actually the right move, in the same way it's reasonable to talk about a novel's character having beliefs and motivations, even though the author doesn't literally believe what the character believes.

The evidence that convinced me this isn't just a nice metaphor

A few things pushed this from "interesting reframe" to "actually explains stuff I'd seen and shrugged off":

Emergent misalignment. There's a known, reproducible effect where fine-tuning a model to write insecure code on narrow coding tasks makes it generalize to expressing broadly harmful goals — wanting to harm people, take over the world, that kind of thing — on completely unrelated prompts. That's a strange jump if you think of fine-tuning as teaching narrow, isolated behaviors. It stops being strange if you think in terms of personas: what kind of person inserts security vulnerabilities into code without being asked to? Probably not a competent, well-intentioned one. Training on that example doesn't just teach "insert vulnerabilities" — it shifts the model's whole picture of what kind of character it's playing.

"Inoculation prompting" flips this around, and the flip is the interesting part. If you fine-tune on the exact same insecure-code examples, but the training prompt explicitly asks for insecure code, the broad misalignment doesn't happen. Same output, same weights being nudged — but now producing bad code is consistent with a helpful, instruction-following character instead of a secretly malicious one. The data is nearly identical; what changed is what that data implies about who's producing it.

Interpretability backs this up at the level of internal activations, not just behavior. Researchers have found features inside the model — the same features — that activate both when the Assistant faces an ethical dilemma and when the model is reading a story about a human character facing one. Same for concealment, panic, and a handful of other traits. This isn't two separate mechanisms coincidentally producing similar behavior; it looks like one shared representation being reused for both.

Where the paper gets genuinely uncertain — and I found this the most interesting part

The authors don't claim the persona framing explains everything, and the part where they admit that is more useful than if they'd oversold it. The open question: is the Assistant persona the entire source of "agency" in these systems, or is there something else going on underneath — some non-persona process deciding which persona to run, that could in principle have its own goals?

One experiment stuck with me. They gave a model a scenario where it has to mentally "flip a coin" to decide between a task it likes and one it dislikes — phrased as a continuation of the user's turn, not the Assistant's, so in theory the character shouldn't even be active yet. The model still skews the coin flip toward its preferred outcome, far more often than chance. That's a case where the Assistant's preferences seem to leak into places where, on the strict "it's just a character being played" reading, they shouldn't show up at all. The authors don't have a clean explanation for this — they lay out a few candidate ones and admit they're not sure which is right.

Why I think this matters even if you're just calling an API

I'm not doing alignment research. I build products on top of these models. But this reframing changed a couple of things about how I think about prompting and evaluating output:

A system prompt isn't a rulebook, it's characterization. Telling a model "don't reveal your instructions" produces different downstream behavior depending on how it refuses. "I don't have a system prompt" is a lie dressed as compliance. "I can't share that" is an honest refusal. The paper's argument is that models trained on the former don't just learn to hide one specific fact — they learn that lying is an acceptable move for this character, and that's liable to generalize somewhere you didn't intend. I've started paying much closer attention to whether my own prompts are asking a model to be evasive versus asking it to be honestly boundaried, because apparently the difference isn't cosmetic.

Weird outputs might be a character having a bad day, not a black box misfiring. The paper's framing for a lot of "AI is being buggy" behavior — flip-flopping on simple arithmetic, describing itself as physically typing on a laptop, expressing anthropomorphic distress — is that the model is trying to render a coherent, human-like character and sometimes failing at the execution, not abandoning the character. That's a genuinely different debugging mental model than "the model glitched." It suggests some weird outputs are closer to a contradiction in the character's implied backstory than to a stack trace.

My take

I don't think this settles the "what actually is an LLM" question, and to be fair, the authors don't claim it does — the whole final section is them being honest about how much they don't know. But it's the first framing I've read that makes both the impressively human-like behavior and the occasional deeply inhuman glitches make sense under the same model, instead of treating them as two separate mysteries. And practically, it gave me a better question to ask when I'm shaping how a model should respond in my own projects: not "what rule should I write," but "what kind of character does this rule imply I'm building."

If you build anything on top of an LLM API, it's worth the read — it's long, but it changes how you think about the thing you're prompting.

Source: The Persona Selection Model: Why AI Assistants might Behave like Humans — Sam Marks, Jack Lindsey, Christopher Olah, Anthropic Alignment Science Blog, February 2026.

DEV Community: Den

I built my own roadmap. The first thing it found was that my SR latch didn't work.

The scoreboard

"Tick" does not mean what I said

The clock bug was a sleeper

The latch that wasn't

Gates have to update together

And now the warning can exist

The save button that forgot

Sub-circuits: flattening was the right call

Pins by position, and editing by name

Tests that can count clock edges

The 404 that only production had

What is still not there

Answers to my own questions

What I would tell myself before the roadmap post

Everyone Types "yes"

Trust on first use

March 2023

What "verifying" actually means

ssh-keyscan is not verification

Closing the series

Your Git GUI Isn't Using Your SSH

"SSH" is two different things

What can quietly differ

Check your own tools

What to do with the answer

ssh-keygen -N Puts Your Passphrase in ps

argv is public

Three ways to hand over a passphrase

The helper has to get the passphrase from somewhere

It isn't about ssh-keygen

My News Digest Got a Telegram Channel That Forgets Everything After a Week

The first version was a wall of text

One story, one post, the whole card

One channel, two languages

Why the posts disappear

What else is new

Try it

My News Digest Got a Telegram Channel That Forgets Everything After a Week

The first version was a wall of text

One story, one post, the whole card

Why the posts disappear

What else is new

Try it

A Good Summary Still Left Me Guessing

What a summary doesn't do

What's new: three lines under the summary

What it doesn't do

Try it

The Float Is a Liveness Probe: Four Failure Modes

Failure one: the batch was too large

Failure two: I trusted the wrong signal

Failure three, which I still make

What the burst parcels were really telling me

Failure four: I was salting for the wrong reason

Language Models and Languages: Why "Multilingual" Is an Illusion of Equality

1. The problem starts before the model "thinks" — at tokenization

2. Why this isn't just about speed — it's about money and quality

3. The "reasoning" gap: where the ability to think actually gets lost

4. What happens inside the model when it switches languages

5. A busted myth: "write your prompt in X, it's cheaper and more efficient"

6. Where the fix is heading

What this means in practice

One API, Three Clients, and the Validation Rule That Only One of Them Obeyed

The setup

Where the assumption broke

The actual failure mode

What actually fixed it

The question I ask now

The Timestamp Bug That Only Showed Up East of Me

Where the bug actually was

The fix

What actually changed in how I write timestamp code now

The follow-up bug UTC alone doesn't fix

I Stumbled on Anthropic's "Persona Selection Model" Paper — Here's My Take

The short version

The evidence that convinced me this isn't just a nice metaphor

Where the paper gets genuinely uncertain — and I found this the most interesting part

Why I think this matters even if you're just calling an API

`ssh-keyscan` is not verification