I turned git pushes into changelog drafts with an LLM. The prompt was the easy part.

If you are adding an LLM feature to an existing product, you will spend a week on the prompt and then discover the prompt was never the risky part. The risky parts are what you feed it, how often you call it, what you do with a bad response, and what you keep afterwards.

I just shipped a feature that turns pushes to a Git repo into draft changelog entries. Here are the five decisions that actually determined whether it worked, none of which are prompt engineering.

1. Do not generate on every event

The obvious design is: push arrives, generate an entry, done. It is wrong for two reasons.

The first is volume. A normal working day produces pushes like "fix typo", "wip", "actually fix it". One entry per push gives you twenty drafts a day and a review queue nobody opens.

The second is worse. A single feature usually lands across several pushes. Generating per push means the model never sees the whole change, so it writes three half descriptions of one thing instead of one description of the thing.

So pushes go into a buffer, and generation runs on a cadence. Each run gets a window of work and can see the shape of it. The cadence is the user's choice (hourly for a repo that ships all day, weekly for a side project), and the scheduler ticks more often than the shortest cadence so that "hourly" is not secretly "up to an hour late".

Buffering also splits the two halves at exactly the right seam. Ingestion has to be fast, because a CI step is waiting on the HTTP response. Generation takes minutes. Put them in the same request and you have a CI job that hangs on a language model.

2. Decide the output granularity before you write the prompt

"Summarize this week's work" gives you one blob of text. That reads like a status report, and status reports are not what anyone subscribes to.

The right unit was already sitting in the database schema. Each changelog entry has one type (feature, fix, improvement, security) and renders as one item with one badge. So the contract is N entries, one per user-facing change, and the model is told to cluster related commits into a single entry.

The general lesson: your existing schema usually already encodes the granularity your users think in. Match it. If the model's output shape and your storage shape disagree, you will either mangle the output on the way in or build a second review screen you did not need.

3. An empty response is a success

This one is easy to get wrong and expensive to leave wrong.

A week of dependency bumps, refactors, CI fixes and test changes contains nothing a user can see. The correct output for that week is nothing at all.

If you do not say that explicitly, the model will invent something. It has been handed a pile of work and asked for changelog entries, and returning nothing feels like failure, so you get "Improved performance and stability" on a week where you renamed a variable. That is how changelogs become noise, and it is the model doing exactly what it was asked.

So the contract has an explicit success case for nothing:

{"entries": []}

and the prompt says, in as many words, that this is a correct and expected answer, not a failure, and that inventing an entry to avoid an empty result is worse than the empty result.

Worth generalizing: for any extraction or summarization feature, define the empty answer as a first class output and say so in the prompt. Otherwise "no signal here" becomes "here is some noise".

4. House rules belong in a validator, not only in the prompt

I have a hard rule that nothing I publish contains an em dash or an en dash. It is the single most recognizable tell of generated text, and these drafts become public changelog copy on somebody else's website.

Putting that rule in the prompt is necessary and not sufficient. Prompts are a request. Models comply most of the time, which is precisely the failure mode you cannot ship on, because "most of the time" means it appears in production copy on a day you are not looking.

So the rule is also a validator. Every title and body is checked, and a response containing a banned character is rejected in full, then retried once with the rejection reason fed back so the retry knows what to fix. If it fails again, the job fails and creates nothing.

Three properties of that design are worth stealing:

Reject the whole response, not the bad entry. Persisting three of four drafts leaves a human to work out which one is missing and why. Rejecting everything costs one retry and keeps the state simple.

Feed the reason back. "Your previous reply was rejected: a title contained an em dash. Fix that and reply again." is a much better retry than the same prompt sent twice.

Losing a run beats shipping a violation. Whatever your equivalent rule is (no pricing claims, no customer names, no promises about unreleased features), decide up front which side you fail on. Write the rule as code, because a rule that lives only in the prompt is a preference.

And do not fix it with find and replace. Swapping an em dash for a comma leaves the same rhythm behind, which is the actual tell. Ask for a rewrite or drop the response.

5. Delete the input as soon as you are done with it

To write a decent entry from commit messages like "fix stuff", the model needs the diff. Which means the feature asks people to send their source code to a third party, and how that is handled is a real question, not a paragraph in a privacy policy.

The design I settled on:

• Exclusions are applied in CI, before anything leaves the repo, so excluded paths are never transmitted rather than transmitted and then dropped.
• The diff is stored only until a run consumes it, then nulled in the same transaction that writes the drafts.
• A sweep nulls diffs on anything a run never reached, so source does not linger because generation kept failing.
• Commit messages and stats stay, because they are the audit trail for where a draft came from and they carry no source.

There is a real cost to this, and it is worth naming rather than hiding: once the diffs are gone, a draft cannot be regenerated. You get the drafts from that run or you write it yourself. I took that trade because "we delete your code" is only worth saying if it is literally true.

The pattern underneath

Every one of these is the same shape. The model is a component with a contract: defined input, defined output, defined failure. The prompt describes the contract, the code enforces it, and the retention policy decides what the component is allowed to remember.

Teams that treat the prompt as the feature end up with a system whose behaviour is a matter of opinion. Teams that treat the prompt as one part of a contract can actually ship the thing.

I build Patchlog, a changelog widget for small SaaS, which is where this feature lives (it is on the Pro plan, and it is GitHub only for now, one repo per project, push triggers only). Everything above is how it is built, so you can take the design without taking the product.