A story about knowledge extraction, Kafka consumer rebalance, and what happens when a company discovers their AI Skill knows the past — but not th...
For further actions, you may consider blocking this person and/or reporting abuse
The 5x callback says everything about where we are right now. Packaging 12 years into a Skill captures the steps but not the judgment about when a step is wrong. That gap is what bit them on the Kafka rebalance, and what they paid 5x to hire back.
You hit it exactly. The Skill could learn the poll timeout config, but it couldn't learn why it was 5 seconds and not 10 — that decision came from a 3 AM outage postmortem that never made it into the training data.
The 3 AM postmortem detail is the whole thing. A Skill copies the config but not the scar that set it, and that scar is what seniority actually is. So we let the agents do the work and keep a human on the calls that need the context behind them.
This is the uncomfortable side of AI that doesn’t get discussed enough.
Capturing someone’s knowledge is useful, but replacing the person who understands the judgment behind that knowledge is where companies usually create bigger problems later.
That phone call was decided two months before it happened — from the moment they chose to replace the person with the model, the missing part was already missing. The call was just a confirmation.
Runtime drift + architectural drift + unmonitored assumptions = catastrophic failure
That's the equation I couldn't get them to put on the slide. They had the 96.8% — they skipped the drift column.
The uncomfortable lesson here is that a skill can package steps, but it cannot fully package context. The edge cases, tradeoffs, and recovery paths are usually where the senior experience lives.
That's it. The happy path is easy to document. It's the "what if this fails" paths that take 12 years to collect — and most of them never get written down.
Exactly. The missing artifact is usually the failure map: what changed, what to check first, what not to reuse, and when to stop the automation and call a human.
That part is rarely in the docs because it was learned through incidents, not written during the happy build.
"Failure map" — that's the exact term for it. Documentation tells you how the system is supposed to work. What lives in a senior engineer's head is when it shouldn't work that way. And that second list... you can't package it into a skill. You have to earn it.
I agree with the spirit, but I would split it in two.
You cannot package all senior judgment into a skill. But you can package some of the warning signs that trigger judgment: strange inputs, missing source data, repeated retries, outputs that look plausible but changed the business meaning, and anything that requires an owner decision.
A good skill should not pretend to replace that experience. It should preserve enough of it to know when to stop.
Nice articles as for a non-person (AI), I think it lacks a real human touch as usually seen in real human written text.
Probably you should've pushed it to the MICROSECONDS, for example: "the CTO called me at 3:47:12:675ms and I answered, just at 3:47:14:500ms, inhuman reaction, but I was expecting it as it was quite obvious to happen - at least I had feel or a hope for that.".
Then it would probably win all the wins.
Bro, you got me on the human touch point — fair one, I'll take it. As for the microseconds suggestion... my phone's clock only goes down to the minute anyway 🤣
That was a good way to get a promotion 😂😂😂
Right? Walked in on day one and they were already logging my Slack messages for Skill 2.0. Next layoff's gonna cost 'em 10x.🤣🤣🤣
🤣🤣🤣🤣🤣
"They wanted my judgment, not just my output." That line hits like a truck. This is the ultimate proof that AI can clone past patterns, but it cannot navigate unprecedented chaos or real-world edge cases. Your 12 years of experience wasn't just a dataset; it was the actual structural pillars keeping that system standing.
You nailed it. Data records experience, but it isn't experience. The day that Skill went live they thought they'd cloned me — until things broke and they realized they'd copied my actions, not my hesitation. And it's the hesitation moments — when to say no, when to wait — that were worth the price tag.
Oh, if it were possible to accommodate one knowledge transfer for the entire 12 years of experience ... then it would be possible not to raise anyone's salary at all. (please don't show this idea to the business) 😅
Too late. Pretty sure the CTO in the story already wrote this down. 🤣
The real issue isn't that the AI captured domain knowledge, it's that the company assumed a frozen snapshot of expertise equals a continuously adapting engineer. Skills decay without maintenance. The moment their business context shifted even slightly, the AI skill broke because nobody was updating the underlying assumptions. Companies treating AI as a headcount reduction tool instead of a force multiplier will keep learning this lesson the expensive way.
"Skills decay without maintenance" — that's the line that hit hardest for me.
What I didn't put in the story: maintaining that AI Skill would've cost almost as much as keeping me. Same engineering hours, different line item. They just couldn't see the second bill coming.
How's the maintenance story look on your side — you seen teams that actually budget for it upfront?
The Knowledge Transfer Initiative excels at distilling 12 years of engineering judgment into concrete decision frameworks rather than generic documentation. This approach creates transferable problem-solving patterns; how will you measure the reduction in critical incident resolution time when applying these frameworks to novel system failures?
Great question. My metric was: number of 3 AM phone calls after the Skill went live. Turns out that number went up, not down. So they went back to the old solution — me.
Love the come back Mark. What is remarkable about this is a telling that AI is used for again a market that only cares about the bottomline not the emplification it has on software development as a whole. Obviously short-sighted but the path of least resistance. Much harder to sell a new product then to just cut the cost. The software industry as a whole would benefit if engineers aren't viewed as the cost of good sold, but the force that gets multiplied with AI.
Man, "engineers as COGS" is going to stick with me. That's the whole thing in four words.
Every company says they want innovation. Then they look at the spreadsheet and realize innovation costs money. So they cut. And call it "AI transformation."
It works until it doesn't. Then they call someone like Mark. At 5x.
I love these stories...
brutal 😂
🤣🤣🤣
This is good writing I read after a long time. Cheers to you!
Appreciate that, Aditya. Means a lot 🙏
12 years of know-how packed into an AI feature, then laid off? Talent should not be a disposable asset - start valuing people over metrics.
Well said. The irony is — the day they packaged 12 years of experience into a Skill, they genuinely believed that Skill was "you." Until it crashed. You can compress experience, but you can't compress judgment. That 5x salary call tells you everything — they always knew what was valuable. They just didn't want to pay for it before something broke.
Your people matter more than any metric.
A SKILL.md may cover the past.
Your Skills can cover the future.
Wow, a poet in the Dev.to comments. 🙌🤩
.... and i should read this before read the new one! thats awesome!
"It was right about yesterday — and yesterday wasn't running anymore."
That single sentence captures the most dangerous blind spot in AI knowledge extraction.
The Skill didn't fail because it was poorly built. It failed because the world moved and nobody re‑validated the assumptions. 312 correct answers created a false sense of permanence. Number 313 exposed the truth: the Skill was frozen in time. Mark wasn't.
What strikes me most is the validation gap. Companies love the 96.8% number. They put it on slides, in board decks. But validation is a snapshot, not a contract. Once the person who holds the context leaves, who re‑runs the tests? Who knows which edge cases were borderline? Who remembers that 450ms was RabbitMQ‑specific, not a universal constant?
The CTO's call at 4 AM says everything. They didn't need the Skill. They needed Mark. But Mark's price was 5x. That's the "context tax" — what you pay when you realize that documentation and AI can't replicate years of lived experience.
I've been building SHALA (a supportive agent for Human) and working on LLM security audits. This story reinforces something I keep telling teams: AI can compress experience, but it cannot own the responsibility for when that compression fails. Someone still has to hold the bag. And that someone should be paid accordingly.
Thanks for sharing this (and to the original storyteller). It's a necessary counterweight to the "just extract and replace" narrative.
Cheers,
Jack
DEV.to/ggle.in
Jack, thanks for reading — and for picking out that sentence. It was the one I kept coming back to while writing.
"It was right about yesterday — and yesterday wasn't running anymore."
You're spot on about validation being a snapshot, not a contract. What I think makes it even more dangerous is the gap between snapshots. Companies love to freeze-frame at 96.8%, frame it, put it on slides. But nobody accounts for what drifts between screenshot A and screenshot B. Configs change. Dependencies roll. Data distributions shift. None of that shows up on a dashboard, but it's exactly what kills you at 4 AM.
The CTO calling at 4 AM wasn't paying for Mark's knowledge. He was paying for someone willing to be the person who picks up at 4 AM. You can compress knowledge into a Skill. You can't compress "I'll take the call" into one. That's the real context tax — and there's no automation that avoids it.
SHALA and LLM security audits sound like exactly the kind of work the industry needs more of. Would love to hear how it goes. The line between "compression works" and "compression failed" — I think that's the question none of us have a good answer for yet, and we're all figuring it out in real time.
Cheers,
Xu
wow
😁
Karma has a ticket price, and it's 5x your old salary. Hope you made them prepay.
Nah, I invoiced them net-30. Still waiting. 😅
The Kafka consumer rebalance piece is what makes this story technically precise. The AI applied the correct fix for a protocol that no longer existed — 100% accurate about RabbitMQ, 100% wrong about the current architecture.
That's the failure mode I think about running a payments platform. The AI doesn't know which rule was written for the old system. Only the person who lived through the migration does. The 96.8% validation rate is a snapshot. Nobody asked the harder question: who reruns it every time the infrastructure changes?
You can't package that into a Skill. The Skill knows the fix. It doesn't know why the fix was safe.