I couldn't keep track of project knowledge, so I automated it

#development #documentation #automation #showdev

When working on one or two long-term projects, you can usually remember most details about how things work. But there's a breaking point where the codebase gets too large and details start slipping through the cracks.

This gets way worse when you're juggling multiple projects at once.

As a developer working across several projects, I learned the hard way how painful it is to not have written documentation about what each project does and how it works. I became the go-to person to ask about how a feature worked and whether it was implemented, when docs should have been the place to go.

My failed attempt at documentation

My attempts at documenting every change fell short. I couldn't actually keep up with updating the docs for every single PR. The workload I was dealing with meant that spending extra time on documentation felt like a strain one more thing to remember before closing a ticket.

And all it took for the docs to become outdated was forgetting to update them a few times.

The idea

Every change I made already had documentation in some form:

Pull requests had descriptions of what changed
Tickets explained what needed to be changed and why
Comments in PRs captured context and decisions

All that knowledge existed. It just wasn't in the actual documentation.

So I built a simple automated flow:

Changes made + Current docs = Updated docs

How it works

I created Taktibt to handle this automatically. The flow is straightforward:

You configure which repos contain your tickets/PRs (the knowledge source)
You configure which repo contains your documentation (the knowledge destination)
When you complete a ticket or merge a PR, the system reads both the change and your existing docs
It generates a documentation update PR for you to review

The key was making it smart enough to:

Skip content that's already documented (avoid duplication)
Place updates in the correct sections (understand structure)
Maintain current state, not a changelog (keep it readable)
Follow project-specific conventions (via custom instructions)

What I learned building this

My first attempt at this was back when GPT-3.5-turbo was the latest accessible model.

The results were horrible. I couldn't get it to properly output a diff of changes, mainly because it couldn't follow instructions that well, but also because my first approach relied on line numbers, and LLMs really struggle with numbering lines.

Then I switched to an exact content replacement strategy using basic delimiters to determine what content should be removed, replaced, or added. It was better, but the model still struggled too much.

Fast-forward to early 2025, and we have much more capable models. Today the app uses Claude Haiku 4.5, which is both cost-effective and capable enough to handle the complexity.

As simple as the idea was, dealing with context limits, LLM costs, the right automation flow that can recover from failures, and getting the LLM to do exactly (or close to exactly) what I expected, were challenges I hadn't anticipated.

Prompting your docs

A late addition that turned out to be really useful: I embedded all the documentation content and each ticket/PR (the changes), then hooked that up so you can search from directly within GitHub comments.

Just drop !ttt:search {query} in any issue or PR comment, and get an AI-powered answer that references your actual docs and change history. Saves a ton of context switching.