Writing for four-year-olds, in Polish
The pipeline writes narration for a preschool channel. A reviewer watched the first episodes and gave one note about the language: the narrator was using strange words, words a small child would not know. Fair, specific, and worth enforcing mechanically so it does not come back.
I reached for the obvious proxy. Long words are hard words, so cap word length. Twelve characters felt generous. It shipped, it fired on a few drafts, the drafts got repaired, and it looked like a working gate.
Polish inflects, and the rule did not know
Polish conjugates and declines. A perfectly ordinary second-person plural past tense verb runs to thirteen or fourteen characters, and preschoolers hear those forms constantly. The gate started rejecting them:
zobaczyliście 13 "you (all) saw" rejected
policzyliście 13 "you (all) counted" rejected
niespodzianką 13 "surprise" (instr.) rejected
pomarańczowego 14 "orange" (gen.) would be rejected
That last one is the line where the rule stopped being merely annoying. Orange is one of the six colours the channel teaches. The declined form is fourteen characters, which means the colour lesson was forbidden from declining the colour it was teaching. A gate that blocks the lesson is not a quality gate.
It also cost money in a way that is easy to miss. Every rejection triggers a repair call to the model. The model dutifully swapped a normal word for a rarer one, which sometimes tripped a different rule, and the episode came back a third time. I was paying for a language model to make text worse, at the instruction of a rule I wrote.
Length was never the property
The reviewer had not said "the words are long". He said the words were strange. What he was hearing was register: abstract nouns in a lesson about a snail. "Safety" is eight letters and abstract. "You saw" is thirteen and concrete. Length and difficulty correlate weakly in English and barely at all in an inflected language, and I had imported an English-shaped intuition without noticing it was one.
Worse, the proxy had looked correct precisely because my data was uniform when I wrote it. Every episode at that point came from a single structural template with a narrow vocabulary. The rule fitted that sample. It broke the moment the content diversified, which is the general shape of a bad proxy: it works on the data that inspired it and fails on the data you built it for.
I had the same failure twice in one day. A second gate required the numerals in a counting episode to strictly ascend, which rejects 1, 1, 2, 1, 2, 3. That sequence is not a bug. It is how a four-year-old is taught to count: one, then one-two, then one-two-three. Same root cause, a rule written against one skeleton and mistaken for a rule about the subject.
Prevention where judgement lives, detection where it does not
I could not write a reliable detector for "too abstract for a preschooler". Nothing in a regex knows that. So I stopped pretending, and split the rule in two.
Prevention went into the brief, where a language model can actually exercise judgement, phrased as a test rather than a threshold:
Detection became an explicit list of words actually caught in review, with a comment saying so and inviting it to grow. Fifteen entries, not a formula. The only length check left is a typo guard at eighteen characters, framed as "this does not look like a word" rather than as a vocabulary rule.
One caution from doing this: I immediately over-added. I put an ordinary adjective on the banned list next to its abstract noun sibling, and it rejected a correct episode about rabbits within the hour. An explicit list is only honest if each entry earns its place.
Questions to ask before adding a threshold
- Name the property, then ask whether the number measures it or merely correlates with it. If it correlates, look for a way to observe the thing itself. Sometimes there is one, and it is not much harder.
- If the property needs judgement, do not fake a formula. Prevention belongs in the instructions; detection belongs in a list of real cases. Half a rule honestly labelled beats a whole rule that lies.
- A gate that passes on old data and fails on new data is suspect. Check whether it encoded the shape of the old data before blaming the generator.
- Count the cost of a false rejection. With a human it is friction. With a model in a repair loop it is money, latency, and output that gets worse each pass.
- If a repair fails twice on the same class of problem, the rule is the suspect, not the model. A third call will not discover that.
The sentence I keep coming back to: a measure that fails correct work is worse than no measure. No measure leaves you looking. A wrong one convinces you that you already looked.
Originally published at neuragrowth.co. I run a one-person digital-products studio and write up what breaks in production.
If you write CLAUDE.md files, I keep a set of working templates here: neuragrowth.co/free/claude-md-templates.
Top comments (0)