The Pain: The landing trio (onboarding checklist, checkpoint gates, rollback) lets a new hire take over your system in a week. Then you find a harsher truth: incidents still come, and they love hitting the exact same spot. My real record: Aug 3, three finished articles never entered the publish queue, patched that night; Aug 5, the same gap again - the English versions were translated but publishing and registration were skipped, patched again; Aug 6, the third time - all five Series D articles were live, and the queue still did not list them. The same "finished article not registered" pit, stepped into three times in five days.
What You'll Learn:
- Why "knows how to fix" is not "has immunity" - fixing solves this one, immunity solves the next one
- The 4-step organizational immunity loop: incident → log → fix → feed rules back
- How each step maps to a real artifact: error-ledger entries, re-runnable verification, rules sunk into skills/scripts/gates
- How to verify immunity is actually working - two signals you can check with one command
Opening: the same pit, we stepped into it three times
In the previous article - From Documents to Mechanisms: Onboarding Checklist, Checkpoint Gates, and Rollback - the trio got a new hire productive in one week. Then you discover a harsher fact: incidents still come, and they love hitting the exact same spot.
My real record: on Aug 3, during multi-platform publishing, we found that articles 17/19/20 had never entered the publish queue even though they were finished - patched that night. On Aug 5, the same gap again: the English versions were translated, but publishing and registration were not done - patched again. On Aug 6, the third time: all five Series D articles were already live, and the queue still had no trace of them. The same "finished article not registered" pit, five days, three times.
The team is not lazy; the system has no immunity. One person stepping into a pit is a lesson; an organization stepping into the same pit repeatedly is amnesia. Today's cure is a four-step loop: incident → log → fix → feed rules back. This loop is not a concept; it is a component running in my own system right now, and every command can be reproduced on your machine.
Why "knows how to fix" is not "has immunity"
Most teams' "review" looks like this: incident happens → the person in charge fixes it overnight → a message goes into the group chat: "fixed, be careful next time" → three months later the same pit explodes again. Why? Because fixing solves "this time"; immunity solves "next time".
| Gap | What it looks like | Consequence |
|---|---|---|
| No logging | Fixed and done, no record | The same troubleshooting starts from zero again |
| No feedback | The record lies in a document | The rule never reaches the execution layer |
| No interception | The rule depends on people remembering it | Newcomers do not know the old pits |
The essential difference: fixing puts the system "back to normal"; immunity makes the system "unlikely to break in the future". To achieve the latter, every incident must become an increment of the rules - this is what Loop Engineering looks like at the organizational level.
Step 1: Log the incident - turn "pits you stepped into" into "organizational assets"
Problem: incident memory lives in individual heads. The person involved remembers it clearly, but the organization has no memory. The person leaves, the lesson leaves with them.
Solution: create an error ledger and log in four parts. First, create the error knowledge base file /root/shared/error-ledger.md; second, every incident gets one entry with a fixed four-part structure: symptom → root cause → fix → status. Our knowledge base has accumulated more than 30 real incidents. Here is a real entry (the 2026-08-01 Hashnode duplicate-publish incident):
## 2026-08-01 · Hashnode duplicate publish incident
**Symptom:** A1/A2/A3/A4/A5/A8 each appeared twice on Hashnode
**Root cause:** the "Draft not found" error actually meant the publish had
succeeded; the retry logic caused duplicates
**Fix:** dedupe before publishing - skip when the title already exists;
when "Draft not found", first check whether the article is published
**Status:** dedupe logic added to multi_publish.py
The four-part structure is not a format; it forces you to think clearly: symptom states "what you saw", root cause digs to the mechanism layer, fix goes down to command level, and status states clearly whether the loop is closed. A pit that is not closed is written in the open, not hidden in your head.
✅ Verification: whenever you hit a new error, the first step is not to fix it - it is to open the error ledger and search history. Run this command:
grep -n "duplicate\|429\|Broken pipe" /root/shared/error-ledger.md
10 seconds to a historical answer, no need to troubleshoot from zero.
🩸 Pitfall: on Aug 2, the daily report trend date showed question marks; after troubleshooting we found the collected_at field was missing - but this pit had never been logged, so the lesson was wasted. The rule now: log today's incident today, do not wait for the review meeting.
▸ Cognitive shift: logging is essentially turning "pits you stepped into" from individual memory into organizational assets - assets appreciate, memories depreciate.
Step 2: Verify the fix - "fixed" is not the end, gate passed is
Problem: announcing "fixed" right after fixing is the fastest path to recurrence. Whether a fix actually worked cannot rely on feeling; it relies on evidence.
Solution: every fix must come with verification, and the verification must be re-runnable. When we handled the "gate_en_articles.py Chinese detection false positive" incident, the fix was changing the detection logic (character count → line count), and the verification was running three regression cases (compliant article passes / Chinese residue blocked / passes after the fix):
# After the fix, run the verification; 4/4 PASS is the real closure
python3 /root/hermes-harness/scripts/validate_article.py check article-25.md
Real output (I ran it on my machine):
check: article-25.md | PASS
body: 3666 chars
quality score: 100/100 (16/16 checks passed)
ALL CHECKS PASSED - article meets the publish standard
🩸 Pitfall: on Aug 5, fixing the Dev.to tag error (the tag contained a hyphen ab-testing, and the platform only allows alphanumeric characters - fixed to abtesting). Right after fixing, we re-ran the publish and confirmed the 422 was gone, and only then changed the status in the error ledger to "fixed". Verification is not optional; it is the switch that closes the loop.
▸ Cognitive shift: verification is essentially turning "I fixed it" from a self-report into a set of re-runnable evidence - evidence speaks, self-reports lie.
Step 3: Feed the rules back - stop the next incident before it happens
Problem: logged, fixed - and the rule still lies in the document. The next incident happens anyway. The key step of an immune system is to sink rules from the "knowledge layer" down to the "execution layer" - into skills, into scripts, into gates.
Our real feedback actions, three paths:
1. Sink into the skill: on Jul 31, the WeChat push returned 501 (consecutive SQL keywords in the body triggered security risk control); the fix was to express such patterns as pseudocode/field descriptions, fed back into the content-repurpose skill, so future articles avoid it automatically.
2. Sink into the script: on Aug 1, the Dev.to 429 rate limit (about 5 posts per 30-second window); the fix was publish interval >= 30 seconds and <= 3 posts per batch, fed back into multi_publish.py with an automatic sleep; the same day's Hashnode duplicate-publish incident got its dedupe logic written directly into the publish script.
3. Sink into the config: on Aug 1, we found the legacy publish CLI was broken (only accepts JSON, depends on a local service, cover URL upload returns 40113); the fix was switching the publish path; on Aug 8, the old command and plaintext credentials were removed from the cron task together, fed back into the task config, never misused again.
# Real effect after feedback: the publish script auto-dedupes and auto-rate-limits
# same title already exists -> skip; between publishes -> auto sleep 30s
grep -n "dedupe\|sleep\|30" /root/tools/baoyu-post-to-wechat/scripts/wechat-api.ts | head -5
🩸 Pitfall: feedback is the easiest step to skip. In the Aug 3 queue gap, the fix was "re-publish", but the feedback action (registering the queue right after writing) was not done at the time - so Aug 5 and Aug 6 it happened again, back to back. A fix without feedback is burying the pit again.
▸ Cognitive shift: feedback is essentially turning "be careful next time" from a verbal reminder into a line of checking code - once the rule enters the execution layer, it does not depend on anyone remembering it.
Step 4: Verify immunity - the same pit does not recur, and if it recurs you can look it up in seconds
Problem: after the feedback, how do you know immunity actually works?
Solution: two signals. First, the same error no longer recurs the same way; second, even if it recurs, you can find the historical answer within 10 seconds.
Our cron task crashed three days in a row (Aug 4/7/8, dying at startup). The first diagnosis was "one-night infrastructure failure"; after the Aug 7 recurrence it was corrected to "recurrent occasional failure"; after the third time on Aug 8 it was upgraded to "persistent occasional failure" and physical governance began (removing the broken command, updating the task config). This diagnosis-correction chain itself is the immune system at work - every recurrence makes the understanding more accurate.
Another real case: after the queue gap recurred three times (Aug 3/5/6), we hardened "register the queue immediately when writing is done" into the process, and that class of gap has not appeared since.
✅ Verification: one command tells you whether immunity is working:
grep -c "fixed" /root/shared/error-ledger.md # fixed entries keep growing
grep -n "queue gap" /root/shared/error-ledger.md # when did it last recur
▸ Cognitive shift: verifying immunity is essentially turning "we will not do it again" from an optimistic expectation into checkable history - the thickness of the error ledger is the scale of organizational immunity.
Before vs After: the same incident, two endings
| Stage | Before: individual debugging | After: organizational immunity |
|---|---|---|
| Incident record | Fixed and forgotten, "fixed" in the group chat | Logged in four parts the same day, everyone can look it up |
| Troubleshooting cost | The same error investigated from scratch | grep gives the historical answer in 10 seconds |
| Where the rule lives | In the lead's head | In skills / scripts / gates |
| Recurrence rate | The same class of pit stepped into repeatedly | Rules intercept, the same pit does not recur |
| Newcomer onboarding | Step into every pit from scratch | The error ledger is a pit-avoidance map |
Why an immune system beats a perfect system
A lot of teams chase "never making mistakes". Wrong direction. Chasing never making mistakes only makes mistakes get hidden; chasing immunity makes mistakes become nutrition.
The immune system has three properties, and they map exactly to the three things we built: memory - the error ledger makes every incident remembered; learning - rule feedback makes every fix change system behavior; detection - gates and verification make recurrence discoverable. Remove any one of the three and it is not immunity, it is just "fixing a bit faster".
Go one layer deeper: the immune system solves the "survival problem" of organizational knowledge. When individuals debug, knowledge lives in human heads - the person leaves, the knowledge leaves. When organizations immunize, knowledge lives in mechanisms - as long as the mechanism is there, the knowledge is there. The third real question of digital transformation is not "make the system smarter"; it is letting the organization evolve from "knows how to fix" to "does not break" - growing a memory every time it makes a mistake. This is the complete shape of Loop Engineering at the organizational level.
Summary: 3 things you can start today
- Create an error ledger: four parts (symptom/root cause/fix/status), log today's incident today
- Every fix must come with verification: after fixing, run a re-runnable verification command; only pass means change the status
- Force the feedback: after every fix, sink the rule into a skill, a script, or a gate - pick one of the three
Do these three and your organization starts to have immunity. This is the third foundation stone of digital transformation - from "knows how to fix" to "does not break", from "individuals remember" to "mechanisms remember".
Next article: LLM-as-Judge: How to Calibrate Your Judge
After rules are fed back into the system, a new question appears: if the judge deciding "right or wrong" is itself an AI, who verifies that the judge is not blind? Next we enter the Agent evaluation and observability series, and talk about calibrating the AI judge - the last link of organizational immunity: even "judgment" itself must be examined.
🧭 Cognitive Index
🏷️ Entities: error-ledger · multi_publish.py · content-repurpose · four-part logging
💼 Value: organizational immunity · incident closure · knowledge consolidation
🧠 Cognition: from "knows how to fix" to "does not break" - the thickness of the error ledger is the scale of immunity
About the author: Wu Ji (无记) — AI / Agent / digital transformation practitioner. Only writes things that actually ran end to end, no concepts for their own sake. Follow along and it just works.
Top comments (0)