This is a follow-up to How I Designed Invariants for AI-Human Collaboration. If you haven't read that, the short version: I built three rules into a collaborative AI system that made it trustworthy. One of those rules was a boolean flag called isManuallyEdited. This post is about what happened next.
How isManuallyEdited Trains My Local AI Through Dialogue
It started as a simple boolean. A single field in a database. A marker that said: "A human touched this."
That flag is isManuallyEdited.
And it turned out to be the beginning of a feedback loop I hadn't fully planned.
The problem: an AI that didn't learn from its mistakes
When I first built my local AI system, I ran into a familiar frustration.
The AI would generate responses. I would correct them. The AI would make the same mistakes again.
There was no memory. No learning. No connection between the correction and the next generation.
The machine wasn't dumb because it lacked data. It was dumb because it lacked a mechanism to receive feedback.
The solution: treating isManuallyEdited as training signal
In the original system, isManuallyEdited protected human corrections from being overwritten. The flag flipped to true, and the AI couldn't touch that row again.
That was Invariant 2 from the last post: the override-is-sacred rule.
But I realised the flag was also sitting on a goldmine of signal I wasn't using.
Every isManuallyEdited = true record is a pair: what the AI produced, and what a human decided it should actually say. That's a labeled training example. And I had hundreds of them.
So I added one more step to the pipeline: store the edit, and queue it for retraining.
class AIGeneratedContent:
content: str
isManuallyEdited: bool = False
originalContent: str
editedContent: Optional[str] = None
editHistory: List[Edit] = []
def edit(self, new_content: str) -> None:
self.isManuallyEdited = True
self.editedContent = new_content
self.editHistory.append(Edit(
before=self.content,
after=new_content,
timestamp=datetime.now()
))
self.queue_for_retraining()
The change was small. The effect was not.
The loop
AI generates output
↓
Human corrects it
↓
isManuallyEdited = True
↓
Edit stored as labeled pair (original → corrected)
↓
Model fine-tuned on accumulated edits
↓
AI generates better output
The human speaks through corrections. The model listens through retraining. Each edit becomes data. The loop runs continuously instead of in scheduled monthly batches.
What the loop produced
Over the months since I added the retraining pipeline, I tracked every edit.
| Metric | Before | After |
|---|---|---|
| Outputs requiring manual correction | ~40% | ~12% |
| Accuracy in my specific domain | 62% | 89% |
| Retraining schedule | Manual, monthly | Automated, continuous |
The model didn't just improve on average — it improved on the specific mistakes that actually mattered to my workflow. Because those were the mistakes I was correcting.
Generic fine-tuning on public datasets could not have done that. Only my own corrections, on my own data, in my own domain, could.
How to implement this in your own system
If you already have an isManuallyEdited flag (or similar provenance tracking), you're most of the way there.
Step 1: Flag every manual intervention
ALTER TABLE ai_outputs ADD COLUMN is_manually_edited BOOLEAN DEFAULT FALSE;
ALTER TABLE ai_outputs ADD COLUMN original_text TEXT;
ALTER TABLE ai_outputs ADD COLUMN edited_text TEXT;
Step 2: Store the full edit history
class EditHistory:
edits: List[Dict] = []
def record_edit(self, before: str, after: str, user_id: str):
self.edits.append({
"before": before,
"after": after,
"user": user_id,
"timestamp": datetime.utcnow()
})
Step 3: Build a retraining pipeline
def retrain_on_edits():
edits = get_all_manually_edited_outputs()
training_data = [
{
"input": edit.original_text,
"expected_output": edit.edited_text
}
for edit in edits
]
model.fine_tune(training_data)
Step 4: Measure the sharpening
Track:
- Edit rate over time (should decrease)
- User correction effort (should decrease)
- Accuracy benchmarks against your domain (should increase)
The metrics tell you whether the loop is working. If edit rate stops falling, something in the pipeline is broken — or the model has hit the ceiling of what your current corrections can teach it.
What this is not
I want to be honest about scope, the same way I was in the last post.
This is not RLHF. I'm not training a reward model or doing reinforcement learning. I'm doing supervised fine-tuning on correction pairs. It's simpler, more brittle, and appropriate for my scale.
This does not replace prompt engineering. Some mistakes are better fixed by improving the prompt than by retraining. I do both. When I see a pattern in the corrections, I often fix the prompt first and retrain second.
This works at small scale. Hundreds of labeled corrections, a local model, a single domain. At production scale with millions of users, you'd need more infrastructure and more careful data curation. But the principle holds.
The AI is not conscious of the feedback. It doesn't "know" it's being corrected. The learning happens through gradient updates, not understanding. I mention this because it's easy to slip into anthropomorphizing the loop.
The test of time
The isManuallyEdited flag was first written on April 27, 2026.
It still hasn't changed.
What changed is what I do with it. The flag was always there. The retraining pipeline was the thing I added later, once I realized I was sitting on training data I wasn't using.
If you already track manual edits in your system, you may be in the same position. The signal is there. The question is whether you're listening to it.
Kiprono Ngetich builds AI-assisted collaboration tools. He thinks too much about data provenance and feedback loops.
Top comments (0)