DEV Community: robzepdev

Structured output guarantees the shape, not the lengths

robzepdev — Wed, 07 Oct 2026 07:29:50 +0000

I run the first real batch of my release tracker on 2026-09-28: 126 releases, one Message Batches
call, done in 17 minutes. The API reported 126 successes out of 126. My pipeline stored 122.

Four summaries were thrown away by my own validation. Not because the model hallucinated, not
because the JSON was malformed, not because a field was missing. Because the title was six
characters too long.

IMAGE 1: the Axios 1.19.0 page in production, showing "error: summary unavailable"

The four

Here they are, with their length against a limit of 120:

Release	Title	Characters
Axios 1.19.0	Axios 1.19.0 raises the `form-data` floor to fix a CRLF injection advisory and fixes NO_PROXY, interceptor and progress bugs	124
Node.js 26.8.0	Node.js 26.8.0 adds ZIP APIs in `zlib`, AES-SIV cipher modes, REPL syntax highlighting and new `sqlite` statement methods	121
Turborepo	Workspace task orchestration replaces topological chunking, plus remote shared build artifacts and `sideEffectsCache` rework.	125
pnpm 11.27.0	pnpm 11.27.0 isolates registry metadata caches per URL path, adds global `nodeDownloadMirrors`, and fixes a store symlink flaw	126

Read them. They are good titles. Every other field was correct: the key points were within the
allowed count, every citation resolved to a real line of the release notes, the link between the
breaking changes and the "action required" verdict held, and stop_reason was end_turn on all
four. These were paid, correct summaries, discarded over punctuation-sized overshoot.

Why structured output did not stop it

My request carries the Zod schema as output_config.format. Constrained decoding is doing real work
there: I have never once received malformed JSON, a missing required field, or a string where an
array belongs. That part is a guarantee, and it is worth having.

But the guarantee is about the shape of the JSON, not about the values inside it. A
maxLength on a string is not a structural constraint that a decoder can enforce token by token,
the way it can enforce that an object closes or that a key is one of five allowed names. By the time
the model is 118 characters into a title, nothing in the machinery is counting, and nothing is going
to stop it at 120.

So counting characters stays something the model does by eye. And by eye, it misses by a little.

Not for lack of asking. My prompt is explicit:

Hard limit: 120 characters, spaces included. Count them before answering.

It aimed well: 122 of 126 landed under the limit, and the four that missed were 1 to 6 characters
over. That is not a model that ignored the instruction. That is a model doing arithmetic on a string
it is in the middle of writing.

The part that cannot be expressed at all

Worse than a maxLength that is ignored is a rule that never reaches the schema in the first place.

My summary schema has a .refine() on it: a summary that lists breaking changes cannot also report
"safe to upgrade". That is the whole product in one line, and it is the one rule I would least like
to be decorative.

It does not survive the trip to JSON Schema. A cross-field invariant is not expressible there at
all, so it exists only in my own parsing step, after the response comes back. On this batch it held
on all 126, and I believe it mostly holds because the prompt makes the relationship obvious. But
"it held 126 times" is an observation, not a guarantee, and I should not confuse the two. The
.refine() stays, and it stays as the last word: anything that fails it is not shown.

The fix is a gap, not a number

The obvious move is to raise the limit. I did, but the interesting part is what I did not raise.

The schema maximum went from 120 to 140. The prompt still asks for 120.

That distance is the entire fix. The model keeps aiming at 120, which is the length I actually want,
and the few characters it overshoots land in a margin instead of destroying a paid summary. If I had
raised the prompt to 140 as well, I would have moved the same problem to 141 and learned nothing.

The prompt text is untouched, so PROMPT_VERSION stays at v4 and my evaluation set stays
comparable across the change. A schema widening is not a prompt change, and it was useful that the
versioning made me say which one this was.

I also considered truncating at the limit. I decided against it: a title cut mid-word is worse than
a title that runs slightly long, and it would hide the problem instead of measuring it. I checked
the layout with a real 140-character title instead: two lines at 1280px, four lines at 390px, no
overflow. The title lives in exactly one place, the <h2> of the release page, which made that
cheap to verify.

IMAGE 2: React 19.3.0 in production, a title that fit, with its five key points and line citations

Do not pay twice for the same answer

The four summaries existed. They were generated, they were billed, and they were correct. Resending
them would have cost about 12 cents, which is nothing, and would also have been the wrong habit.

So the repair reads the saved batch results from disk, revalidates them through exactly the same
parser the pages use, and writes UPDATE statements to a .sql file. All four pass at 140. The
script touches no database: applying the file stays a deliberate gesture in the SQL editor of the
environment I mean.

There is a trap hiding in that sentence, and it nearly cost me the option. The expires_at of a
batch is 24 hours from creation, not the result retention period of 29 days that the
documentation makes easy to assume. I had saved the 126 results to a local file the same day, out of
habit rather than foresight, and that habit is the only reason this repair was free. Save the
results file. It costs nothing and it buys you every repair you have not thought of yet.

What it cost to learn

The batch billed $4.16: a mean of $0.033 per summary, a median of $0.0245. I had estimated $2.94,
measured on 14 releases I had picked by hand. The median was nearly right. The tail was not: real
release notes are longer than the ones I chose to test with, and the ten most expensive summaries
were 21% of the spend. That is a separate lesson and it deserves its own post.

The 3% discard is what I actually came away with. It is not a bug I fixed, it is a number I did not
have before: the measured distance between what structured output promises and what I had assumed it
promised.

Honest status at the time of writing: that SQL has not been applied yet, so those four releases
still read "summary unavailable" in production. The screenshot above is the live page, not a staged
one. The repair is written and verified; running it is a manual step still sitting in my queue.

The tracker is a side project: it watches releases of languages, frameworks and libraries and turns
their release notes into summaries that say whether you need to act before upgrading. Everything
above is from the real run, not a reconstruction.

Enforce your free-plan limit in the database, not in the UI

robzepdev — Mon, 28 Sep 2026 10:29:19 +0000

Your free plan says "up to 10". Where does that 10 actually live?

In most apps I have read, it lives in the interface: a button that disables itself at ten, a count in a store, maybe a check in the route handler that writes the row. That is a suggestion, not a limit. A second tab, a page left open since yesterday, a direct call to the API with a token from devtools, and the eleventh row is in the table.

I put mine in Postgres. This is what that took, and the three things I got wrong on the way.

The setup

The product is a release tracker: you follow the technologies in your stack and it tells you what shipped and whether upgrading needs work. The Free plan follows up to 10.

Follows are the one thing a signed in developer writes, and row level security already limits every row to their own user id, so the browser talks to the database directly. There is no route of mine in the middle:

const { error } = follow
  ? await follows.insert({ user_id: developerId, technology_id: technologyId })
  : await follows.delete().eq("technology_id", technologyId);

That is a deliberate choice, and it is what makes the rest of this possible: if the database refuses, the refusal arrives in the mutation that asked, not in a log I will read next week.

The rule, in the database

create function public.enforce_follow_rules()
returns trigger
language plpgsql
security definer
set search_path = ''
as $$
declare
  followed integer;
begin
  if (select retired from public.technologies where id = new.technology_id) then
    raise exception 'A retired Technology cannot be followed'
      using errcode = 'FRTRD';
  end if;

  perform pg_advisory_xact_lock(hashtextextended(new.user_id::text, 0));

  if coalesce((select plan from public.plans where user_id = new.user_id), 'free') = 'free' then
    select count(*) into followed from public.follows where user_id = new.user_id;
    if followed >= 10 then
      raise exception 'The Free Plan follows at most 10 Technologies'
        using errcode = 'FLIMT';
    end if;
  end if;

  return new;
end;
$$;

create trigger enforce_follow_rules
  before insert on public.follows
  for each row execute function public.enforce_follow_rules();

Three details in there are worth more than the rest.

The advisory lock. Without it the whole thing is theatre. Two tabs inserting at the same moment both run the count, both read nine, both pass. pg_advisory_xact_lock on a hash of the user id serialises the decision per developer and releases at the end of the transaction. It is exactly the "added from another session" case the feature is supposed to survive, and it is the one I would have shipped broken if I had not written a test that inserts a follow from outside the browser first.

security definer. The trigger applies to the service role too. That is what I want, and it also means my own test fixtures cannot set up a followed retired technology directly: they have to follow first and retire after, which is the only way the case appears in real life anyway.

The plan check. The limit is for free only. What a paid plan allows is not decided yet, and the trigger is the place where that decision will land.

Refusals should be codes, not sentences

The interesting part is not raising the error, it is getting it to the row that has to explain itself.

I gave each rule an SQLSTATE of my own: FLIMT for the limit, FRTRD for a retired technology. PostgREST carries the code to the browser in the error object, so the page never parses a message:

const BY_CODE: Record<string, FollowRejection> = { FLIMT: "limit", FRTRD: "retired" };

export function followRejectionFor(code: string | undefined): FollowRejection {
  return (code && BY_CODE[code]) || "unknown";
}

Anything I did not plan for, a dropped connection or an expired session, becomes unknown, which still has a remedy to offer ("try again") instead of failing silently. Two rules, three outcomes, and the wording of the database error stays where it belongs: in the console.

Pick codes in the XX000 custom space and write them down somewhere the next person will look. Mine are in the migration, in a comment right above the function.

The bug the first real refusal found

The UI follows optimistically: the row flips, the count moves, and if the database says no the previous state goes back. I had built that with TanStack Query, with one useMutation inside the follow toggle component.

It worked perfectly until the first refusal arrived, and then the refusal was invisible.

One mutation per row means the error lives in the component that was pressed. The row that has to show "this technology is retired" is a sibling; it never sees it. Worse, the count in the top bar is computed somewhere else again, so three parts of the same page can each hold a different idea of what just happened.

The fix is dull and it is the whole lesson: one mutation for the page, in the provider, exposed through context.

const value: Follows = {
  count: following.size,
  atLimit: following.size >= FREE_FOLLOW_LIMIT,
  isFollowing: (technologyId) => following.has(technologyId),
  toggle: (technologyId) => mutation.mutate({ technologyId, follow: !following.has(technologyId) }),
  isPending: (technologyId) => mutation.isPending && mutation.variables?.technologyId === technologyId,
  rejectionOn: (technologyId) => (technologyId === rejected ? rejection : null),
};

The count, the block at ten and the refusal now come from one place, so they cannot drift apart.

What the interface still owes you

Enforcing the rule in the database does not excuse the UI from anticipating it. At ten of ten the unpressed buttons get aria-disabled="true" and point at a note that says what to do, and they stay in the tab order. A real disabled would take them out of it, and a keyboard user would meet a control that has silently vanished and no explanation of why.

There is a nice side effect. Playwright refuses to click a control marked aria-disabled, so the test has to say click({ force: true }). That line is the proof that the block is there: if the attribute went missing, the forced click would go through and the assertion after it would fail.

The number 10 is now written in two places, FREE_FOLLOW_LIMIT in the code and the trigger in the migration. That duplication bothers me a little, and it is the correct trade: the one in the migration is the one that decides, the one in the code is how the page explains itself before asking.

Was it worth it

The trigger is 25 lines. It ships with the schema, it applies to every client that will ever exist, including the ones I have not written yet, and it turned a plan limit from a claim into a fact.

What it does not do is tell your users anything. That is still the interface's job, and the trigger only makes it easier, because now the page reacts to a rule instead of pretending to be one.

How I keep an LLM from inventing breaking changes

robzepdev — Thu, 24 Sep 2026 14:13:05 +0000

I'm building a tool that reads every new release of React, TypeScript, Vite and friends, and tells you two things: what changed, and whether you need to touch your code before upgrading.

The summarising part is easy. Any decent model will write you a tidy paragraph about a release.

The part that decides whether the tool is worth anything is different: when it says "action required", is that true? A tracker that invents a breaking change is worse than no tracker, because you'll go looking for a migration that doesn't exist. And a tracker that misses one is worse still.

Here is everything I do about that, in the order I had to learn it.

1. The model only sees the release notes

The system prompt says it in the second line:

Use only the release notes in the user message. Do not use what you know about the project from elsewhere.

This matters more than it sounds. A model asked about "React 19.2.0" has opinions about React 19 from training. Those opinions are often right, which is exactly what makes them dangerous: they're right until the release where they aren't, and you have no way to tell which one you're looking at.

So the notes go into the user message, wrapped in a tag, and nothing else about the project does.

Two consequences follow immediately:

No notes, no summary. If the release has no notes, the request is never built. The release stays in the catalogue with no summary rather than getting a plausible one.
Notes are never truncated. There's a hard ceiling of 80,000 characters. Over it, the summary fails with a reason instead of being generated from the first half. A truncated summary is a confidently wrong one: the breaking change was probably in the part you cut.

if (notes.trim() === "") return { ok: false, reason: "release notes are empty" };
if (notes.length > MAX_NOTES_LENGTH) {
  return { ok: false, reason: `release notes are ${notes.length} characters, over the ${MAX_NOTES_LENGTH} limit` };
}

Both are failures I'd rather see than paper over.

2. "Action required" is a definition, not a judgement

The prompt doesn't ask the model whether you should act. It asks for a list:

"breakingChanges": every change the notes state or clearly describe as breaking, removed or incompatible, so that users must change code, configuration or runtime to upgrade.

"actionRequired": true exactly when "breakingChanges" is not empty.

And then, flatly:

Never infer a breaking change the notes do not mention, even if the version number suggests one.

That last line earns its place. Without it, a x.0.0 invites the model to reason "this is a major, majors break things" and produce something. Version numbers are a classification input for me, never evidence about content.

The output shape is constrained with a schema, so a reply that isn't that shape never reaches the database. But keep reading, schemas do less than you think.

3. I never ask the model for line numbers

Each key point and each breaking change links back into the official notes, to the line it came from. On the page it looks like release-notes:20, and clicking it opens the notes, scrolls to line 20 and puts a caret at the right column.

The obvious implementation is to ask the model for the line number. Don't. Models count badly, and, worse, they count badly invisibly: you get 42, it looks like a line number, and it's off by nine.

So the model returns a verbatim quote instead:

A "quote" is the stretch of the release notes the item comes from, copied character for character [...] Never adjust, shorten with an ellipsis or rewrite a quote to fit: a quote that does not appear verbatim in the notes is dropped.

And I compute the position myself, by searching my stored copy of the notes for that string:

locateCitation(notes, quote) // → { fromLine, toLine, column, length } | null

null means the quote isn't in the notes. Which means the model made it up. Which means no pointer is rendered.

This is the part I'm most pleased with, because of what it collapses. "The model must not hallucinate citations" and "references must stay inside the document" sound like two separate things to test. They're the same thing, and it's a string search. The column offset for the caret falls out for free.

4. Then I stopped guessing and measured

Everything above is reasoning about a prompt. Reasoning about a prompt is how you end up with a prompt that feels great and is wrong.

So: an evaluation set. 14 real releases, from 8 projects, 5 of them not even in my catalogue, so I couldn't accidentally tune the product to the test. The notes are copied into the repo, not read from the database. The catalogue changes; the set must not, or two reports a month apart mean nothing.

The human judgement lives in a set.json next to them, and it's more nuanced than a pass/fail:

required breaking changes: miss one and the case fails;
tolerated breaking changes: legitimately arguable, listing one isn't an error;
actionRequired, which can be true, false, or "either" for releases whose notes describe a behaviour change without ever calling it breaking. Some releases genuinely are a judgement call, and pretending otherwise just bakes my mood on the day into the test.

The comparison itself is a pure function with its own unit tests. The script around it only does I/O. Two prompt versions, one batch each:

	v3	v4
cases evaluated	13/14	14/14
"action required" correct	13/13	14/14
missing breaking changes	0	0
invented citations	1	0
verbatim quotes	66 of 72	77 of 81

Six of the fourteen are deliberate duds: patch releases whose notes say nothing but "see the changelog". All six came back "no action needed". That was the point of including them: I wanted evidence that the model shuts up when there's nothing to say, not just that it's eloquent when there is.

Two full runs cost about 50 cents through the Batch API. Roughly $0.02 a summary.

5. Two bugs that only a real call could find

Both of these survived a fake adapter, unit tests and integration tests. Both died within minutes of the first real batch.

Line endings. Release notes fetched from GitHub contain CRLF. The model quotes them back with LF. The page searches the normalised text. So any quote spanning more than one line silently failed to locate. No crash, no log, just a pointer that quietly wasn't there. In production. locateCitation now normalises both sides.

A sixth key point. The prompt asks for 3 to 5 key points, and the schema says so too. A reply came back with six, failed validation, and threw away a summary I had already paid for.

The lesson is worth stating plainly: structured output constrains the shape of a reply, never its counts. You can guarantee "an array of objects with these fields". You cannot guarantee "at most five". So now the parser keeps the first five — they arrive in order of importance — and the prompt states the limit as a hard rule with the consequence spelled out. That change is what took the prompt to v4, and it's why the "cases evaluated" row went from 13 to 14.

6. Running out of credit is not a failure

A smaller thing, but it's the same instinct. When the Anthropic account runs out of credit, the affected releases go back to pending, not to failed. failed is final — those summaries would never have regenerated, even after topping up.

And the case is typed, not sniffed out of an error message: submit throws a CreditExhaustedError, collect returns credit_exhausted. "Other errors must not be mistaken for exhausted credit" is then something the compiler enforces, rather than something I promise.

What I'd keep

If I did this again from scratch, three things carry over:

Ground the model in a document you also store. Not because the model is untrustworthy, but because it gives you something to check the answer against, mechanically.
Ask for quotes, derive everything else. Anything positional the model computes is a silent error waiting to happen.
Spend the fifty cents. A fake adapter tells you your code works. A real call tells you your assumptions do. I had two bugs in production-bound code that no amount of local testing was ever going to surface.

The tracker is live and collecting releases while I build the rest of it. The summaries aren't switched on in production yet — that's the next thing, and it's the first part of this project that costs money to leave running.

https://release-tracker-drab.vercel.app/technologies/react

TypeScript never ships x.y.0: classifying releases the way projects actually version

robzepdev — Tue, 22 Sep 2026 14:13:13 +0000

I'm building a tracker that tells developers which releases of their stack actually matter.
The first problem wasn't AI or scraping. It was a deceptively simple question:

Given a version string, how important is this release?

Why "just use SemVer" doesn't work

The naive rule: x.0.0 is a major, x.y.0 is a minor, everything else is a patch.
It holds for many libraries, and breaks for some of the most popular ones.

I ended up with one classification strategy per project, mapping a version
to one of four buckets: Spotlight (you should look), Latest (new stuff),
Digest (patches, batched weekly) and Preview (beta and rc).

1. `semver`: the default

x.0.0 → Spotlight, x.y.0 → Latest, patches → Digest. Vite fits perfectly:
after 8.0.0 came 16 patches, 8.0.1 through 8.0.16, in under three months.
A good tracker shows you 8.0.0 and quietly folds the rest into a digest.

2. `node`: even majors become LTS

Node's even majors (20, 22, 24) become Long Term Support; odd ones don't.
So an even x.0.0 goes to Spotlight, while odd majors and x.y.0 go to Latest.
Same version shape, different meaning.

3. `typescript`: there is no stable x.y.0

TypeScript doesn't follow SemVer: every x.y can contain breaking changes.
And the numbering is unusual:

5.9.0-beta → 5.9.1-rc → 5.9.2   ← first stable 5.9

A rule that looks for x.y.0 would never flag TypeScript 5.9 at all.
So the strategy is: the first stable version of each x.y that the sources observed is Spotlight;
later ones are Digest. The strategy has to look at history, not just the string.

4. `zerover`: in 0.x the minor breaks

For libraries still in 0.x, 0.y.0 is effectively a major.
So 0.y.0 → Spotlight, other 0.y.z → Digest, and from 1.0.0 on it falls back to semver.

Previews and noise

Every strategy treats pre-releases the same way. Only suffixes that are exactly
beta or rc (optionally numbered, like -rc.1) count as previews.
Canary, nightly, experimental, alpha and hash- or date-stamped builds
(-rc.0-next-…) aren't tracked at all. They're the bulk of the noise.

Keep it pure, test it on real versions

Each strategy is a pure TypeScript function, with no I/O,
tested against real version histories from npm and GitHub.
Real data catches what toy examples don't: TypeScript's .2, React's
-rc.0-next-<hash> tags, v-prefixed tags.

And when a rule is still wrong for an edge case, a manual override lives
in the database and is never overwritten by the next sync.

What's next

Classification decides what deserves your attention. The next step decides
what to do about it: an AI summary of each important release, with breaking changes,
migration steps and an explicit Action required / Safe to upgrade verdict.
It will be grounded only in the official release notes. If they don't mention
a breaking change, neither will the summary.

You can see the classification at work on an early preview:
Vite's release page: 8.0.0 up front,
the 16 patches folded into a digest. Name and domain are still provisional.

I'm building this in public. If your stack has a versioning scheme that would
break these rules, I'd love to hear about it in the comments.