I'm building a tool that reads every new release of React, TypeScript, Vite and friends, and tells you two things: what changed, and whether you need to touch your code before upgrading.
The summarising part is easy. Any decent model will write you a tidy paragraph about a release.
The part that decides whether the tool is worth anything is different: when it says "action required", is that true? A tracker that invents a breaking change is worse than no tracker, because you'll go looking for a migration that doesn't exist. And a tracker that misses one is worse still.
Here is everything I do about that, in the order I had to learn it.
1. The model only sees the release notes
The system prompt says it in the second line:
Use only the release notes in the user message. Do not use what you know about the project from elsewhere.
This matters more than it sounds. A model asked about "React 19.2.0" has opinions about React 19 from training. Those opinions are often right, which is exactly what makes them dangerous — they're right until the release where they aren't, and you have no way to tell which one you're looking at.
So the notes go into the user message, wrapped in a tag, and nothing else about the project does.
Two consequences follow immediately:
- No notes, no summary. If the release has no notes, the request is never built. The release stays in the catalogue with no summary rather than getting a plausible one.
- Notes are never truncated. There's a hard ceiling of 80,000 characters. Over it, the summary fails with a reason instead of being generated from the first half. A truncated summary is a confidently wrong one: the breaking change was probably in the part you cut.
if (notes.trim() === "") return { ok: false, reason: "release notes are empty" };
if (notes.length > MAX_NOTES_LENGTH) {
return { ok: false, reason: `release notes are ${notes.length} characters, over the ${MAX_NOTES_LENGTH} limit` };
}
Both are failures I'd rather see than paper over.
2. "Action required" is a definition, not a judgement
The prompt doesn't ask the model whether you should act. It asks for a list:
"breakingChanges": every change the notes state or clearly describe as breaking, removed or incompatible, so that users must change code, configuration or runtime to upgrade.
"actionRequired": true exactly when "breakingChanges" is not empty.
And then, flatly:
Never infer a breaking change the notes do not mention, even if the version number suggests one.
That last line earns its place. Without it, a x.0.0 invites the model to reason "this is a major, majors break things" and produce something. Version numbers are a classification input for me, never evidence about content.
The output shape is constrained with a schema, so a reply that isn't that shape never reaches the database. But keep reading — schemas do less than you think.
3. I never ask the model for line numbers
Each key point and each breaking change links back into the official notes, to the line it came from. On the page it looks like release-notes:20, and clicking it opens the notes, scrolls to line 20 and puts a caret at the right column.
The obvious implementation is to ask the model for the line number. Don't. Models count badly, and — worse — they count badly invisibly: you get 42, it looks like a line number, and it's off by nine.
So the model returns a verbatim quote instead:
A "quote" is the stretch of the release notes the item comes from, copied character for character [...] Never adjust, shorten with an ellipsis or rewrite a quote to fit: a quote that does not appear verbatim in the notes is dropped.
And I compute the position myself, by searching my stored copy of the notes for that string:
locateCitation(notes, quote) // → { fromLine, toLine, column, length } | null
null means the quote isn't in the notes. Which means the model made it up. Which means no pointer is rendered.
This is the part I'm most pleased with, because of what it collapses. "The model must not hallucinate citations" and "references must stay inside the document" sound like two separate things to test. They're the same thing, and it's a string search. The column offset for the caret falls out for free.
4. Then I stopped guessing and measured
Everything above is reasoning about a prompt. Reasoning about a prompt is how you end up with a prompt that feels great and is wrong.
So: an evaluation set. 14 real releases, from 8 projects — 5 of them not even in my catalogue, so I couldn't accidentally tune the product to the test. The notes are copied into the repo, not read from the database. The catalogue changes; the set must not, or two reports a month apart mean nothing.
The human judgement lives in a set.json next to them, and it's more nuanced than a pass/fail:
- required breaking changes — miss one and the case fails;
- tolerated breaking changes — legitimately arguable, listing one isn't an error;
-
actionRequired, which can betrue,false, or"either"for releases whose notes describe a behaviour change without ever calling it breaking. Some releases genuinely are a judgement call, and pretending otherwise just bakes my mood on the day into the test.
The comparison itself is a pure function with its own unit tests. The script around it only does I/O. Two prompt versions, one batch each:
| v3 | v4 | |
|---|---|---|
| cases evaluated | 13/14 | 14/14 |
| "action required" correct | 13/13 | 14/14 |
| missing breaking changes | 0 | 0 |
| invented citations | 1 | 0 |
| verbatim quotes | 66 of 72 | 77 of 81 |
Six of the fourteen are deliberate duds: patch releases whose notes say nothing but "see the changelog". All six came back "no action needed". That was the point of including them — I wanted evidence that the model shuts up when there's nothing to say, not just that it's eloquent when there is.
Two full runs cost about 50 cents through the Batch API. Roughly $0.02 a summary.
5. Two bugs that only a real call could find
Both of these survived a fake adapter, unit tests and integration tests. Both died within minutes of the first real batch.
Line endings. Release notes fetched from GitHub contain CRLF. The model quotes them back with LF. The page searches the normalised text. So any quote spanning more than one line silently failed to locate — no crash, no log, just a pointer that quietly wasn't there. In production. locateCitation now normalises both sides.
A sixth key point. The prompt asks for 3 to 5 key points, and the schema says so too. A reply came back with six, failed validation, and threw away a summary I had already paid for.
The lesson is worth stating plainly: structured output constrains the shape of a reply, never its counts. You can guarantee "an array of objects with these fields". You cannot guarantee "at most five". So now the parser keeps the first five — they arrive in order of importance — and the prompt states the limit as a hard rule with the consequence spelled out. That change is what took the prompt to v4, and it's why the "cases evaluated" row went from 13 to 14.
6. Running out of credit is not a failure
A smaller thing, but it's the same instinct. When the Anthropic account runs out of credit, the affected releases go back to pending, not to failed. failed is final — those summaries would never have regenerated, even after topping up.
And the case is typed, not sniffed out of an error message: submit throws a CreditExhaustedError, collect returns credit_exhausted. "Other errors must not be mistaken for exhausted credit" is then something the compiler enforces, rather than something I promise.
What I'd keep
If I did this again from scratch, three things carry over:
- Ground the model in a document you also store. Not because the model is untrustworthy, but because it gives you something to check the answer against, mechanically.
- Ask for quotes, derive everything else. Anything positional the model computes is a silent error waiting to happen.
- Spend the fifty cents. A fake adapter tells you your code works. A real call tells you your assumptions do. I had two bugs in production-bound code that no amount of local testing was ever going to surface.
The tracker is live and collecting releases while I build the rest of it. The summaries aren't switched on in production yet — that's the next thing, and it's the first part of this project that costs money to leave running.
Top comments (0)