A story about knowledge extraction, Kafka consumer rebalance, and what happens when a company discovers their AI Skill knows the past — but not the present.
Based on a submission from a community member. If you have a similar story or something you need to get off your chest — reach out. The next one could be yours.
Has your company ever extracted your experience into a system — then decided the system was good enough without you?
Have you watched an AI Skill handle 312 scenarios correctly, and wondered whether you'd be around when number 313 finally showed up?
This is that story.
Act 1 · Knowledge Extraction
The conference room had no windows. Three months.
Every Monday through Thursday I sat in that plastic chair — a voice recorder, a laptop, a mug on the table. Across from me sat an engineer named Caleb. His job was to ask me questions.
"Why PostgreSQL over MongoDB?"
"Why is the retry interval 450 milliseconds?"
"How did you calculate that alert threshold?"
I answered. He wrote it down. The red light on the recorder stayed on.
They called it a "Knowledge Transfer Initiative." The CTO's all-hands email was polished: We're preserving decades of institutional knowledge for the next generation of engineers.
In plain English: your experience is too expensive. We're packaging it into a Skill.
I didn't take it seriously at first. Every company has knowledge management. Then month three rolled around. Caleb stopped asking questions. He started validating.
He pulled up an incident I'd debugged the year before — a production outage. He asked me to reproduce the diagnosis. Then he let the system do it.
The system said: Message broker latency spike detected. Retry logic at 450ms interval will amplify under queue backpressure. Recommend adaptive backoff.
I read it three times. It was right. It was my chain of reasoning — but it said it faster than I could.
The CTO stood next to me, smiling. I've seen that smile too many times in Silicon Valley. It's not happiness. It's sign-off.
Act 2 · "96.8% Accuracy"
The moment I knew something was wrong was the day the validation report came out.
A full row of people sat across the table — CTO, HRBP, VP of Engineering, and a woman I didn't recognize with a consulting firm logo on her blazer.
The projector displayed one number: Knowledge Retention Rate: 96.8%
Caleb delivered his findings. "After three rounds of validation, the AI Skill achieved 96.8% diagnostic accuracy across 312 historical failure scenarios. The remaining 3.2% deviation is suboptimal recommendations due to insufficient context — correctable with guidance."
Applause. The CEO turned to me and said something I'll never forget.
"Mark, you created your own replacement."
Everyone laughed. I laughed too. What else was there to do?
After the meeting I sat in the parking lot for a long time. My coffee had gone cold. Outside the window, that unchanging California blue — the same one it's always been. My phone buzzed. A text from my wife: What time are you coming home?
I typed: "Soon." Then I deleted it.
I didn't know how to tell her I'd spent three months turning myself into a manual. And once you write the manual, you don't need the original.
Act 3 · The Severance
The Monday after, the CTO closed the door for our one-on-one.
"Mark, I won't sugarcoat it. Your position's been eliminated as part of the restructuring."
He slid an envelope across the table.
N+3. Three months of severance. The standard Silicon Valley we're not firing you, we're helping you transition treatment.
I signed without stopping. Not because I didn't care — because I'd already played out how this ended before I walked in.
My last day was a Friday. It took me under forty minutes to clear my desk. One backpack. One monitor stand. Three pen caps — I still don't know how three pen caps ended up in my drawer.
I sat in the car and searched three things on my phone:
- LLC registration (California, single-member, $70, 15 minutes)
- Professional liability insurance (industry standard for consultants, ~$1,200/year)
- Rate benchmarking for senior infrastructure consultants (median: $215/hour)
That night I registered a company. Mark Johnson Consulting LLC. No office, no employees, no VC pitch deck. One clause, written clear: project-based only. No AI in the delivery chain.
Act 4 · AI Skill Goes Live
The first full quarter was a showpiece.
The AI Skill absorbed 70% of tier-2 operations tickets. New engineers went from six months to three weeks to reach productivity. The CEO used a slick slide at the all-hands:
Before: 12 years of Mark's experience locked in his head.
After: 12 years of Mark's experience available as a prompt.
Nobody mentioned Mark himself. They didn't need to. He was already packaged.
My wife noticed before I did. The weekend after I left, she looked at me across the kitchen and said something I wasn't expecting.
"You seem different."
"Different how?"
"When you sit at your computer and don't talk — that tension's gone. You used to be waiting for things to break. Now you're waiting for clients."
She was right. I was waiting.
I saw on LinkedIn they'd hired a new Platform Lead. First thing he did was rebuild the monitoring stack. The Skill? Nobody re-ran validation after they migrated to Kafka. Why would they? The person who built it wasn't there anymore.
Act 5 · Crash
It happened at 3:47 AM on a Wednesday.
The distributed tracing system lit up. P99 latency on the core payment chain went from 80 milliseconds to 12 seconds. Three minutes in, the first transaction timed out and rolled back. At ten minutes, the payment queue started backing up. At twenty-three minutes, the rollbacks triggered a cascade — two downstream services began refusing requests.
The on-call engineer pulled up the AI Skill. It scanned the logs, identified the pattern, and returned a diagnosis.
Detected message broker latency. Applying known mitigation: activate retry queue with 450ms backoff.
That diagnosis had been correct 312 times out of 312 historical scenarios.
This was number 313.
The 450ms retry window was a compatibility shim I'd written five years earlier for RabbitMQ. The number wasn't random — I'd spent two weeks load-testing on a RabbitMQ cluster to find the exact gap that cleared its Erlang VM GC cycle.
But that was RabbitMQ. Retired for three years. They were running Kafka now.
Kafka's consumer groups use a poll-based protocol — every consumer has to keep polling the coordinator at a configured interval, or the coordinator marks it dead and triggers a rebalance. My retry worker was synchronous — process one message, pull the next. At 450ms a round, a queue of a few dozen messages stretched the poll window by multiple factors.
# What the Skill did (the 450ms retry window)
def handle_queue():
while queue_not_empty():
msg = fetch_one_message() # one at a time
retry_with_backoff(450ms) # ⏱ 450ms per message
process(msg)
poll_consumer() # "still alive?" → into debt
# What happened with 60 queued messages
poll_timeout = 5000 # 5s before coordinator marks dead
time_per_round = 60 * 450 # 60 messages × 450ms = 27,000ms
# 27,000ms > 5,000ms → poll timeout missed → rebalance triggered
Once the rebalance kicked in, the whole group started oscillating. New consumers claimed partitions. Old messages got redelivered. Latency climbed again. A sticky problem turned into a snowball.
The AI executed a strategy that was 100% correct on the old architecture. On the current one, it was pouring gasoline on the fire.
The AI didn't fail because it was wrong. It failed because it was right about yesterday — and yesterday wasn't running anymore.
Act 6 · The Call
My phone rang at 4:12 AM.
A name I hadn't called in months — my former CTO.
His voice was calm. Too calm. When an engineer calls at 4 AM and sounds that steady, something's on fire.
"Mark. I need to ask you something."
"Ask."
He laid out the incident in three sentences. I thought he'd ask about rollback strategy. Damage containment. He didn't.
"You left a note in the RabbitMQ migration docs. At the time, nobody understood it. We thought it was a stale config comment. I think I understand it now."
I leaned back against the headboard. The only light in the room was the phone screen. My wife stirred beside me. Didn't wake up.
The note said: 450ms matches RabbitMQ GC window. Do not reuse outside this context.
Silence on the other end. About four seconds.
"I know," he said. "I'm out of time, Mark. Come back."
"I'm not asking if you'll come back." His voice shifted. "I'm asking you to. I'll pause everything for the rest of the month. You call the shots. Name your price."
Act 7 · The Price
"Five times my old salary."
I heard his breath catch.
"… Deal. But I need you on-site for two weeks. Contract. Bring your own equipment. No AI touches your delivery chain."
"Done. Send the contract to my lawyer."
I didn't have a lawyer. What I meant was: this runs at my pace.
After I hung up, I scrolled to a name I hadn't called in nearly a year — Mike, college roommate, law school grad. Sent one message.
Got a contract. Mind glancing at it?
He didn't reply. At 4 AM, who's awake except people in IT?
I waited. Then I typed five more words. I sent them to him, but they were really for me.
This time I set the price.
You've been through this before? Your knowledge extracted, packaged into a Skill, turned into the reason you weren't needed anymore. And when the Skill hit something it didn't understand, they came back. Is there a better answer than "no"?
The Skill knew every past scenario. It just couldn't see that the infrastructure had changed.
The company saved six figures on headcount. They just forgot validation only works when the person who wrote it is still there to re-run it.
When the system went down — who did they call?
Follow for more stories about AI extraction, implicit knowledge, and what happens when companies discover their Skill only remembers the past.
I couldn't teach the AI Skill to understand context. But the stories stay open. buy me a coffee ☕
Top comments (4)
.... and i should read this before read the new one! thats awesome!
"It was right about yesterday — and yesterday wasn't running anymore."
That single sentence captures the most dangerous blind spot in AI knowledge extraction.
The Skill didn't fail because it was poorly built. It failed because the world moved and nobody re‑validated the assumptions. 312 correct answers created a false sense of permanence. Number 313 exposed the truth: the Skill was frozen in time. Mark wasn't.
What strikes me most is the validation gap. Companies love the 96.8% number. They put it on slides, in board decks. But validation is a snapshot, not a contract. Once the person who holds the context leaves, who re‑runs the tests? Who knows which edge cases were borderline? Who remembers that 450ms was RabbitMQ‑specific, not a universal constant?
The CTO's call at 4 AM says everything. They didn't need the Skill. They needed Mark. But Mark's price was 5x. That's the "context tax" — what you pay when you realize that documentation and AI can't replicate years of lived experience.
I've been building SHALA (a supportive agent for Human) and working on LLM security audits. This story reinforces something I keep telling teams: AI can compress experience, but it cannot own the responsibility for when that compression fails. Someone still has to hold the bag. And that someone should be paid accordingly.
Thanks for sharing this (and to the original storyteller). It's a necessary counterweight to the "just extract and replace" narrative.
Cheers,
Jack
DEV.to/ggle.in
Jack, thanks for reading — and for picking out that sentence. It was the one I kept coming back to while writing.
"It was right about yesterday — and yesterday wasn't running anymore."
You're spot on about validation being a snapshot, not a contract. What I think makes it even more dangerous is the gap between snapshots. Companies love to freeze-frame at 96.8%, frame it, put it on slides. But nobody accounts for what drifts between screenshot A and screenshot B. Configs change. Dependencies roll. Data distributions shift. None of that shows up on a dashboard, but it's exactly what kills you at 4 AM.
The CTO calling at 4 AM wasn't paying for Mark's knowledge. He was paying for someone willing to be the person who picks up at 4 AM. You can compress knowledge into a Skill. You can't compress "I'll take the call" into one. That's the real context tax — and there's no automation that avoids it.
SHALA and LLM security audits sound like exactly the kind of work the industry needs more of. Would love to hear how it goes. The line between "compression works" and "compression failed" — I think that's the question none of us have a good answer for yet, and we're all figuring it out in real time.
Cheers,
Xu
Karma has a ticket price, and it's 5x your old salary. Hope you made them prepay.
Nah, I invoiced them net-30. Still waiting. 😅