Nobody writes the post-mortem, so my bug fixes write their own

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

In March 2014, requests fixed a leak. When a redirect sent your request somewhere else, the library kept replaying your proxy password. The fix added a function, rebuild_proxies(), that recomputes the proxy credentials for each new destination.

In May 2023, requests fixed another leak. On an HTTPS redirect, your proxy password was reaching the server at the end of the chain. The culprit was rebuild_proxies().

Same function, nine years apart. The lesson was never lost: it sat in the git history the whole time, between thousands of commits nobody rereads. And it isn't just requests. Keeping credentials when a redirect changes host, scheme or port has been fixed at least 47 times in 22 projects, from curl and Go to undici and urllib3, between 2014 and this year.

I'm not better than any of them. After a fix, I write a commit message, a test if I'm in a good mood, and I move on. Writing down what the bug taught me, and reading it again before the next change, is exactly the kind of work that never gets done. Nothing breaks when you skip it. Not that day.

So I gave that job to a sloth.

Bug Diary keeps a diary of the bugs you fixed, and reads it back to you before you repeat one:

• After a fix, it reads the diff, the new test, the commit message, and the issue if there is one, and Gemma 4 writes the post-mortem nobody writes: symptom, root cause, pattern, the guard the fix added, and one lesson for next time.
• Before a commit, it describes what your change does, finds the closest lessons in the diary, and checks whether your change repeats one of them. If it does, it shows you the lines, the old bug, and a test to write.
• One diary for all your repositories. A lesson learned in one project is checked in all the others.
• You don't run anything. A git hook learns from every commit whose message starts with fix, in the background, and checks every commit before it lands.

Everything runs on my laptop, slowly and steadily: Gemma 4 and EmbeddingGemma in Ollama, a SQLite file in my home directory.

Who gets to be lazier? Anyone who fixes bugs and never writes the post-mortem, and anyone who reviews code without remembering an incident from three years ago in another repository.

Demo

python demo/toy.py builds two small repositories. The first one, api, fixes a race: two requests refresh an expired token at the same time, and the second refresh uses a token the first one already rotated. Bug Diary learns the fix. Then a teammate writes a billing client in another repository, with its own token refresh:

billing-worker $ bugdiary check
⚠ Bug pattern already met in another project: race condition in asynchronous resource rotation
  Precedent   BUG-0001 · api · 72ac6e21 · fixed 2026-10-21 · production
  Past bug    users logged out when two requests refresh an expired token at once
  Cause       The second refresh sent a token the first one had already rotated, so the server
              revoked the whole session.
  Lesson      Use a shared promise to deduplicate concurrent asynchronous operations that mutate
              shared state.
  In this change
    src/client.ts:8  if (this.token.expired) {
    src/client.ts:9  this.token = await this.auth.refresh(this.token.refreshToken);
  Why         The bearer method performs an asynchronous token refresh without any mechanism to
              deduplicate concurrent calls, meaning multiple simultaneous requests will trigger
              multiple refresh calls.
  Check       Are you using a shared promise or a locking mechanism to ensure that multiple
              concurrent calls to bearer() do not trigger multiple calls to auth.refresh()?
  Test        Call billingClient.invoices() and billingClient.payments() simultaneously using
              Promise.all and verify that auth.refresh is called exactly once.

When the client shares the refresh, it stays quiet:

billing-worker $ bugdiary check
No known bug pattern in this change.
✓ BUG-0001 (race condition in asynchronous resource rotation) is already guarded at src/client.ts:11
Compared the change with 1 of 1 lessons in 13 s.

The same check on real history. The diary below only knows fixes published before 2021, and it reads node-fetch's redirect code as it was just before its January 2022 fix:

And the same code with the fix:

Code

sintineddi / bug-diary

Keeps a diary of the bugs you fixed, and reads it back before you repeat one. Runs locally with Gemma 4.

How I Built It

Who does what

Piece	Job
Gemma 4 26b (Ollama)	Reads a fix and writes the card; describes a new change; compares it with a past bug
EmbeddingGemma (Ollama)	Finds the lessons closest to what a change does
My code	Checks every card and every alert against the sources, stores the diary
SQLite	One diary file for all repositories
Git hooks	Learn after a fix commit, check before every commit

The model writes, the code checks

A post-mortem written by a model has an obvious failure: it sounds right. Give it a one-line diff that adds a cache timeout, and it will happily tell you that customers saw stale prices for hours. Nothing in the diff says so.

So every card says where each claim comes from, and my code checks it:

• A symptom or a cause marked stated must be found, nearly word for word, in the commit message, the issue, the changelog or the new test. The card keeps the real excerpt, not the model's version of it. If the quote isn't there, the claim becomes inferred from the code, or unknown.
• Function names the model cites must exist in the fix.
• A test found in the fix is recorded as found, not as run.

The same rule holds when checking a new change. An alert must quote lines that are really in the change; a verdict of already guarded must quote the guard. Otherwise it is dropped. Similar words are not enough: the comparison has to point at code.

Could the diary have warned them?

To test this on real history, I needed bugs that verifiably came back, with dates. Security advisories are the best-documented recurring bugs in open source: each one has an ID, a date and a fix commit. So I collected 153 of them, all checked against their fix commits, in four families:

Family	Instances	Repositories
Credentials kept when a redirect changes host, scheme or port	47	22
Archive extraction writes outside its folder (zip slip, tar slip)	48	22
Prototype pollution through __proto__ or constructor	42	24
HTTP request smuggling (lenient Content-Length / Transfer-Encoding)	16	6

Then I froze time. The diary only learned the fixes published before January 1, 2021: 60 of them, plus 6 ordinary bug fixes from the same repositories as noise. For each of the 92 later bugs (the 93rd couldn't be tested), Bug Diary checked two versions of the same code: the vulnerable one, as if someone were writing it today, and the fixed one, which should pass. It also checked 21 ordinary commits from the same repositories. No fix made after 2020 entered the diary.

What a warning is worth

A tool that warns about everything is a tool you stop reading. So the question isn't only "did it see the bug?", but "did it warn on the vulnerable code and stay quiet on the fixed one?" For each of the 92 later bugs, I compared the two answers.

	Warns on the vulnerable version only	Warns on both	Warns on neither	Warns on the fixed version only	Ordinary commits with a warning
Gemma 4 alone, asked to review the change	1	89	1	1	18 of 21
Bug Diary	24	9	54	5	1 of 21

Without the diary, the same model finds a problem in almost everything: both versions of 89 bugs out of 92, and 18 of the 21 ordinary commits. Even when it points at the lines the real fix changed, it does so on the fixed version of 62 bugs and on the vulnerable version of only 52. The new guard is the most complicated code in the change, so it gets the suspicion.

With the diary, Gemma has something to compare with. For 90 of the 92 later bugs, the closest lessons included at least one of the same family. And it can stay quiet. Bug Diary warned on the vulnerable version of 33 bugs and on the fixed version of 14: for 24 bugs it warned on the vulnerable version only, and for 5 the other way round. Of the 21 ordinary commits, it warned on one.

It misses a lot: 54 bugs got no warning at all. But when it speaks, it usually speaks about the right code, and it mostly speaks about the vulnerable one.

The lessons travel. Of the 33 warnings, 30 relied only on lessons from other repositories, and 14 only on lessons learned from code in another language. Until January 2022, node-fetch's redirect code did this:

// HTTP-redirect fetch step 6 (counter increment)
// Create a new Request object.
const requestOptions = {
    headers: new Headers(request.headers),
    follow: request.follow,
    counter: request.counter + 1,

Every header of the original request, Authorization and Cookie included, was copied into the request sent to the new host. With a diary frozen a year earlier, Bug Diary flagged line 178 with three lessons: two from requests (Python, 2014) and one from GitHub's http-client (TypeScript, 2020). On the fixed version, it quoted the new guard and stayed quiet. In the benchmark record of that check, it still left the question a good reviewer would ask: could attacker-example.com pass for a subdomain of example.com?

Half a guard looks like a guard

The bugs that come back are rarely new. They come back because the last fix was incomplete. merge-deep blocked __proto__ in 2018; in 2021 the same merge went through constructor.prototype. minimist blocked __proto__ in 2020; in 2022, constructor got through. follow-redirects removed the Authorization header when a redirect changed host, then had to add the cookies, then the change from HTTPS to HTTP, then Proxy-Authorization.

That is exactly where a diary should help, and it is how Bug Diary fails. Of the 59 vulnerable versions it didn't warn about, it called 55 already guarded. The vulnerable code had a guard, and the model trusted it. For 23 of them, the repository had fixed a bug of the same family before 2021, so the code usually carried the guard of that earlier fix. The lesson says block __proto__. The model sees the block and answers already guarded.

So I tried showing the model the whole family: the lesson, plus the other fixes of the same kind of bug in the diary, and one sentence: a guard can be incomplete. It warned on the vulnerable version of 53 bugs instead of 33. It also started doubting every guard, and warned on both versions of 40 bugs. One sentence turned a reviewer that trusted too much into one that trusts nothing.

Without that sentence, with the family shown only as context, the model went the other way and became more careful: it warned on 21 vulnerable versions and 6 fixed ones. That is now the --by-kind option. It makes one model call per kind of bug instead of one per lesson, so a check takes 22 seconds instead of 38, for fewer warnings in both directions. More context didn't make the model better at judging a guard. It moved where the model draws the line. By default, Bug Diary compares each lesson alone, which gave the best balance.

How much can a model learn from a fix alone?

From the diff alone, Gemma left the symptom unknown for 37 of the 66 fixes and inferred it from the code for the other 29. For five of those, it quoted the diff itself, and the check refused the quotes. With the commit message, the pull request and the advisory, 62 of the 66 symptoms were quoted from a source, and the code checked every quote.

The test and the commit message changed less than I expected. With the new test, Gemma inferred a symptom for 57 fixes instead of 29: a test describes the behaviour that broke, so the model can work it out. With the commit message as well, it could quote a symptom for only 9: commit messages say what was changed, rarely what went wrong. In the full run, 56 of the 62 quoted symptoms came from the advisory or the pull request, 5 from the commit message, and 1 from a test. A diary can only know what went wrong if someone wrote it down, and it is rarely in the commit.

Does it matter for catching the next bug? Less than I thought for finding it, more for judging it. A diary learned from the diffs alone still found the right family for 89 of the 92 later bugs, and warned on vulnerable code just as often. But it told the two versions apart less well: it warned on both versions of 17 bugs. With the tests, that fell to 7, and the full context gave the most clean calls, 24 bugs warned on the vulnerable version only, against 19 from the diffs alone.

Why Does Open Innovation Matter?

A security fix is a secret until it ships. Sixteen of the fixes in my benchmark are literally called Merge pull request from GHSA-… or Merge commit from fork: they were written in a private fork and merged shortly before the advisory was published. Until then, the patch is the most sensitive file in the project. Nobody should paste it into someone else's API to get a post-mortem. A model on my laptop can read it the day it is written.

It runs on every commit, for free. Learning the 66 fixes of the benchmark took 18.5 minutes of laptop time. A check takes 38 seconds at the median in the benchmark, and nearly one in four takes more than a minute. Billed per token, I would think twice before running it on every commit of every repository. Here the only bill is electricity.

I can measure it, and measure it again next year. The weights don't change under me. Gemma 4 is Apache 2.0, EmbeddingGemma comes with Gemma's terms, and --model takes any other model in Ollama, so the benchmark can be rerun with whatever comes next.

The diary is a file. ~/.bugdiary/diary.db is SQLite. It belongs to whoever fixed the bugs, not to a vendor.

The Lazy Payoff

Before publishing, I had Bug Diary's own code reviewed. The review found six real bugs: the --strict hook that never blocked anything, a large change reported clean although part of it was never read, a case-insensitive comparison of code, and three more. So I installed the hooks on Bug Diary itself and committed each fix with a message starting with fix.

Each fix became a card in the background, in about 30 seconds of laptop time, and each later commit was checked against the cards before it, in 25 to 78 seconds. Two more fixes followed: two processes could upgrade the same diary at once, and the one below. The diary of Bug Diary now holds eight lessons. I wrote zero lines of post-mortem. Honestly, that isn't time saved: I never wrote post-mortems before either. It is eight post-mortems that exist now and didn't before, for about four minutes of laptop time I didn't watch.

The best moment was a mistake. Pasting the commit messages, I pasted the second one twice, and the next three commits each got the message of the fix before. The diary believed the messages. The case-sensitivity fix became a card about large changes, with a fix it had partly invented from the surrounding code. My quote check was satisfied: the words were really in the message. They were just about another change.

So Bug Diary got one more fix. Before writing a card, Gemma now says whether the message describes the diff. If it says no, the card is written again without the message, and says so. On my nine commits, it caught two of the three pasted messages, and none of the six good ones. On the 66 real fixes of the benchmark, it flagged two messages: a merge commit with no description at all, and the fix for CVE-2019-11249, a directory traversal in kubectl cp, whose title is Refactors to kubectl CP command.

Now it at least checks that the message and the diff talk about the same change.

Then it paid me back. On one of the last commits before publishing, the hook printed this before the commit went through:

bug-diary $ git commit -m "Challenge is coming"
⚠ Possible return of a known bug: trusting incorrect commit messages as symptoms
  Precedent   BUG-0008 · bug-diary · 8fb97e53 · fixed 2026-10-21
  Lesson      Verify that the commit message accurately describes the actual code changes before
              using it to extract symptoms.
  In this change
    bugdiary/learn.py:199  if "message" in texts and raw.get("message_matches") in ("no", "part…
  Why         The change modifies the logic in learn.py to no longer treat 'partly' as a reason to
              remove the message and re-query, which is exactly the behavior the bug fix was
              intended to prevent.
  Check       Does the new logic in learn.py still ensure that a 'partly' match doesn't lead to
              using an untrustworthy commit message?
  Test        Given a commit message that matches the code changes only 'partly'; when the learn
              function processes the fix; then the message should be removed from texts and the LLM
              re-queried, just as it was before the bug was fixed.

2 other lessons do not apply or stay unclear (--all shows them).
Compared the change with 3 of 8 lessons in 156 s.

It was right. I had narrowed that rule on purpose: on the 66 real fixes, counting partly as a mismatch threw away eight messages that were incomplete or vague, but true. Bug Diary couldn't know that, and it didn't block anything. It made sure I was undoing part of a fix knowingly, with the old bug, the line and a test in front of me.

My Agent Session

I built Bug Diary with Claude Code, from my design note. Claude Code wrote the code, the tests and the benchmark harness, and ran the benchmark on my laptop.

Prize Categories

Best Use of Gemma: Gemma 4 writes and compares the cards, EmbeddingGemma finds them again.