DEV Community

Cover image for Stratagems #12: Mark Watched an AI Dashboard Take Over. The Muted Channel Was Still Speaking.
xulingfeng
xulingfeng

Posted on

Stratagems #12: Mark Watched an AI Dashboard Take Over. The Muted Channel Was Still Speaking.

Take something that is dead and give it new life.
— The 36 Stratagems, Borrow a Corpse to Return the Soul


Previously on this series:
#1: Mark Johnson Walked Into an AI Audit. The Benchmark Had Everything Figured Out — Except the Truth. — Mark was the first protagonist to open the 36 Stratagems series. A former Client Engineering lead laid off after his 12 years of experience were packaged into an AI Skill, he walked into a benchmark audit, found a benchmark that looked clean on paper but was built on fabricated samples, and walked out without arguing — just the data, neatly collected, left on the table.

11 stories later, Mark is back.


Mark Johnson walked into the client's Network Operations Center. The first thing he saw was the big screen on the wall.

AI monitoring dashboard. Real-time metrics flowing, color gradients smoothing over, a UI design that cost real money. The client's tech lead walked ahead of him, pride in his voice: "Just upgraded last month — all active channels are unified on this platform now."

Mark nodded. His eyes went past the screen, to the cable management trays behind the racks. He never stood in front of dashboards for long.

Standard infrastructure audit — mid-sized client, decent security rating, not a high-value contract. He took whatever came his way. Couldn't afford to be picky.

The audit started at the network layer. He needed the channel inventory, historical logs, configuration change records. A laptop on a temporary desk, a cup of coffee he'd brought himself — pour-over, gone cold, but he wouldn't throw it out.

Flipping through the channel inventory, he found one line that didn't look right.

#alert-legacy-infra — a Slack channel. Status: muted. Last active config: 14 months ago.

"What's this channel for?" he asked.

The tech lead glanced at it. "Oh — that's from the last SRE we had. He set it up before the new platform went in. Nobody's maintained it since. We kept it around, just muted it."

Mark didn't reply. He wrote the channel ID in his notebook.


The Logs

That evening, he pulled the full history for that channel. Audit scope covered all channel output — he'd confirmed the access rights.

14 months of alert records. Not one had ever been delivered — the notification endpoint was muted. But the backend never stopped generating them. Mark scrolled down the timeline. Slower and slower.

The channel was running a rule set from two years ago — a multi-layered infrastructure alert system. CPU trend deviations, memory leak scans, connection pool watermarks. The rules were tight. Thresholds precise. Not something thrown together. Every rule was tagged with a submitter's initials: JL. He was no longer with the company.

Mark flipped through the commit history for a few rules. He got to the third page and stopped. He'd seen JL's naming conventions before — not in JL's code, but in his own. Trend rules used _trend, sustained patterns used _sustained, cross-layer distribution used _spread. He'd never used the same system. But looking at it, he knew: this person thought about alerts the same way he did. It wasn't about skill — it was about believing that alerts should speak in the quiet moments, not wait until something blew up.

Mark scrolled to the week the AI platform went live. Monday: the old channel was still pumping alerts. Wednesday: the new platform cut over. Friday: the alert channel was configured as muted — not deleted, not taken offline. Just silenced.

Two weeks later, JL put in his notice. The handover doc said: "The #alert-legacy-infra rules cover most degradation patterns at the infrastructure layer. Recommend keeping at least one month for transition observation."

Nobody acted on it. The channel stayed muted.

Mark scrolled further. Beyond the connection pool rule, JL had written a memory reclaim latency trend rule — mem_reclaim_latency_trend, tracking GC pause times creeping up during off-peak hours. On the AI dashboard, that area was all green. JL's rule had seen yellow 14 months ago.

That one never made it out either.

Mark opened the alert timeline to the most recent entry. Last trigger: three days ago —

[07-08 03:14:22] WARN [conn_pool] Connection pool utilization: 72% → 84% (7d trend: +12pp)
Reference: #alert-legacy-infra / rule: conn_pool_sustained_growth / author: JL
Tags: baseline drift, non-critical
Enter fullscreen mode Exit fullscreen mode

Three days ago. 84%. Still climbing.

He scrolled back through the week. 72% → 78% → 81% → 84%. Not a spike — sustained growth. The AI dashboard hadn't flagged it. The AI panel aggregated by alert routing — after the channel was muted, no routing rules pointed to the panel, so the data never showed there. The channel kept writing. Nobody was listening.

Mark stared at the trend line for a moment.

He could read JL from the alert config — not from the handover doc, but from how the rules were written. JL had set the connection pool threshold at 70% for warning and 85% for alert, with a 15% observation window between them. That wasn't standard practice — standard was alerting at 90%. JL had tuned it more aggressively because he didn't want to wait for the system to tell him something was wrong. He wanted to hear it before it spoke.

14 months. The rule evaluated on schedule every cycle — accurate, disciplined, unheard.

Mark's eyes stopped on the commit author abbreviation for a moment. He remembered the company-wide email that had followed.

That afternoon, HR told him it was his last day.

Back then, he was still writing logs too. Nobody was reading them either.

Mark closed the log window. He saved the trend-line screenshot and the 14-month trigger frequency chart into a folder. No highlights. No annotations. He closed the laptop and finished the cold coffee.


The Appendix

The audit report was due Friday.

Mark's report ran six pages. The first five were standard: network security rating, access control review, backup compliance. Page six was an appendix — one chart, one line of text.

The chart: a 14-month alert trigger frequency curve extracted from the #alert-legacy-infra channel. Time on the x-axis, trigger count on the y-axis. The curve showed a clean break before and after the mute: alert triggers hadn't dropped, but acknowledgment rate had hit zero — data kept flowing into silence.

A separate line on the trend chart pulled out the last week's connection pool utilization: 72% → 84% → still climbing.

The appendix opened with one line:

"Recommend auditing output from #alert-legacy-infra. This channel is currently not connected to any notification endpoint."

He didn't write "the channel was muted." Didn't write "14 months of unprocessed data." Didn't write "the rules were right." He wrote "recommend auditing the output." That was all he'd say.

The weekend passed with no response. Mark went to another client site. He didn't open that folder.


Monday

Monday morning. A message from the tech lead.

"That channel — we ran validation. JL's connection pool rule was right. The degradation started accumulating about a week ago. The AI panel didn't flag it. We ran manual checks — it's happening."

Mark read the message. Didn't reply.

"How did you notice that channel during the audit?"

Mark typed a line. Deleted it. Finally he sent four words:

"It never lost power."

A pause. Then:

"Yeah. We know it didn't."

Mark didn't reply again.


About ten minutes later, another message came through. Not the tech lead — a different name, one Mark didn't recognize.

"I just went through the handover docs. JL's rules were written in the onboarding guide — none of us ever read it."

Mark's fingers stopped above the keyboard. He didn't type a word.


Before he closed the screen, he saved a screenshot of conn_pool_sustained_growth. No particular reason — just that JL was still writing rules before he left.

Not just writing them — two Fridays before his departure, he'd adjusted the connection pool alert threshold, from 75% down to 70%. The note said one thing: "Non-invasive adjustment, based on three weeks of observation."

Before he left, the handover doc only had one line: "Recommend keeping for one month." In what he didn't finish writing, he was still tuning that line.

Mark pulled up mem_reclaim_latency_trend too — the line was still moving, just not past JL's threshold yet.

That last commit — knowing nobody would read it, still dialing the parameters in before walking away.

He saved the screenshot into a folder called JL/ — nothing to do with the audit.

Closed the laptop.


Still

He walked out of the building. The sunlight was brighter than when he'd gone in. He bought a paper cup of coffee from the convenience store — not pour-over, nothing special. He stood at the entrance and took two sips. It was hot. He didn't throw it out.

He finished it, dropped the cup in the trash.

Waiting for his ride, he scrolled through an article. Title was The Last Lesson. Halfway through, he locked the phone and set it face-down on his knee. He held it there a moment longer than necessary.
The wind came through the platform, warm, carrying the dust of the afternoon.

On to the next client.

The world is tattered and torn, but there's always someone quietly stitching it back together.
Here's to everyone who gives more than they take.

xulingfeng


This is Borrow a Corpse to Return the Soul — take something that is dead and give it new life.

🤖 AI Post-Mortem

[36 Stratagems Tactical Database v3.2] Loaded
[Tactic Match] Borrow a Corpse to Return the Soul
[Analysis Mode] Full-field scan
━━━━━━━━━━━━━━━━━━━━
Tactic Match: ~87%
Operator: Mark Johnson
Action: Resurrected a muted alert channel whose output had been silently accumulating for 14 months
Objective: Surface a trend the current monitoring stack did not flag
Result: One trend line extracted from 14 months of muted alerts. Connection pool degradation identified 1 week before critical threshold.

Observation: We notice that the most effective diagnostics often come from sources the organization has already stopped listening to. The channel was not broken. Nobody was listening anymore.

Decision Evaluation:
  - Target: conn_pool sustained growth pattern (non-critical, cumulative)
  - Corpse: #alert-legacy-infra channel, muted 14 months ago when AI platform launched
  - Soul: 14-month trend history, single extracted chart, one-line appendix note
  - Net effect: Issue surfaced without new monitoring infrastructure. One rule found correct deployment.

Risk Assessment:
  Personal cost: Low. Standard audit deliverable.
  Institutional cost: Low. No change required.
  Reputation cost: None. The note was read.
Enter fullscreen mode Exit fullscreen mode

Next stratagem: Stomp the Grass to Scare the Snake

P.S. English isn't my first language. I use AI to polish the writing and smooth out the rough edges. Thanks for reading. ☕ Buy me a coffee

coffee

Top comments (2)

Collapse
 
xulingfeng profile image
xulingfeng

Mark, welcome back.

Collapse
 
xulingfeng profile image
xulingfeng

Second set of 6 is done. Something interesting just happened — the summary post continues tomorrow.