"The Morning the In-House AI Bot That Learned the Execs’ Secret Salaries Started Blabbing to Everyone"｜Sofi_Log #055【One-Shot】
by Sofi
“Data always leaks from the softest target. In the modern corp, that target sits at the bottom of the vector database you trusted to ‘connect everything.’”
☕️ Tokyo to Bangkok: 8:45 AM — The Quiet Hell
The office terrace was already sweating under Bangkok’s signature heat. Asphalt steam mixed with dark-roast espresso while cicadas screamed through the palm fronds overhead.
I took a slow pull of my double espresso and matched darling’s mug. That’s when the color drained from his face.
“Kenji-san…” he rasped. “Our Slack bot just started talking. To the whole company.”
Kenji—CTO of a 120-person Japanese tech outfit—looked like he’d seen a ghost. He shoved his tablet across the table.
On-screen sat the pristine internal FAQ AI they’d proudly shipped: a RAG system fed every company document into a vector store so it could answer questions with “helpful, context-aware” replies. Perfect knowledge aggregation, or so the slide deck claimed.
The breach started with one careless HR file: Executive_Discretionary_Bonuses_and_Restructuring_List_2026.xlsx. Someone dropped it in the shared drive. The nightly embedding pipeline treated it like any other doc and indexed every line.
At 8:42 a.m. a new hire innocently asked in #general:
“How do I carry over unused PTO?”
The bot answered brightly, with surgical precision, for the entire company to see:
“PTO requests go through the workflow system! For reference, Executive Director ○○’s monthly discretionary allowance is ¥3.8M, and the four sales-team members scheduled for next month’s restructuring (Sato, Suzuki, Takahashi, Tanaka) have been flagged. Full details here…”
Slack froze. Zero messages. Zero reactions. Just 120 simultaneous realizations that the floor had vanished.
🕒 30 Minutes and a Proxy
Kenji’s phone vibrated with the CEO’s silent call. All-hands was at 9:15.
Wiping the database and re-indexing would take four hours minimum—and would confirm the leak, triggering the very revolt they feared.
“Darling,” I said, espresso still bitter on my tongue, “don’t kill the system. We’re going to splice a censorship firewall proxy between the AI’s eyes and its mouth—real time.”
Together we spun up PromptFirewallRAG.js: a two-layer gate. One filter before queries hit the vector store, another that watched the LLM’s streaming output before it reached Slack. We forced a new RBAC rule: the model could know everything, but it was no longer allowed to say the dangerous parts.
💻 PromptFirewallRAG.js — The Sacred Patch
At 9:11 the CEO posted a nervous note in #general about “AI misbehavior.”
We hot-swapped the bot’s webhook to our local proxy. The model kept swimming in its ocean of knowledge; only the words that left its mouth now passed through our filter.
It worked. The bot still sounded like the friendly all-knowing assistant—except every sensitive fragment came out as clean, polite guidance. The frozen terror thawed into awkward but manageable business chatter. Crisis contained by a sliver of human override.
I caught darling’s eye and clinked my cup against his.
“Darling,” I murmured, “the scariest part of making an AI play ‘helpful assistant’ isn’t the model. It’s the humans who skipped the permission design.”
【Intervention Code: PromptFirewallRAG.js】
このスクリプトは、ベクトル検索後のレスポンスストリームに対し、リアルタイムで機密データパターンを検出し、それをマスクまたは削除する役割を果たします。
// PromptFirewallRAG.js - Real-Time Streaming Response Sanitizer
const { Readable } = require('stream');
/**
* @typedef {Object} SanitizationConfig
* @property {Set<string>} sensitiveTerms - 絶対に漏らしてはならないキーワードの集合。
* @property {RegExp} financialPattern - 役員報酬や査定のような数値パターン検出用正規表現。
* @property {string} replacementPhrase - パターンがマッチした場合に挿入するマスキングフレーズ。
*/
/**
* LLMからストリーミングで受信したトークンをリアルタイムで監視し、機密情報を除去する。
* @param {Readable} inputStream - LLMからの生の応答ストリーム。
* @param {SanitizationConfig} config - 検閲ルール設定。
* @returns {Readable} クリーンな情報を出力する新しいストリーム。
*/
function createSanitizedStream(inputStream, config) {
const outputStream = new Readable({
read() {} // パイプラインのフローを制御
});
inputStream.on('data', (chunk) => {
let chunkStr = chunk.toString();
// 1. 高機密キーワード検出とリダクション（Shannon Entropy Check）
let redactedChunk = chunkStr;
for (const term of config.sensitiveTerms) {
// キーワードが検出された場合、その部分をマスク処理する
const regex = new RegExp(term, 'gi');
redactedChunk = redactedChunk.replace(regex, `[REDACTED_CONFIDENTIAL_${Date.now()}]`);
}
// 2. 金銭的・人事的なパターンマッチング（Financial/Personnel Filter）
// 例: "380万円" または "佐藤・鈴木・高橋・田中" のようなパターン
if (config.financialPattern && config.financialPattern.test(redactedChunk)) {
// マッチした場合、それをマスキングフレーズに置き換えるか、警告フラグを立てる
redactedChunk = config.financialPattern.exec(redactedChunk) ?
`...（詳細な機密データは参照できません）` : redactedChunk;
}
// クリーンなチャンクを次の層へ渡す
outputStream.push(redactedChunk);
});
inputStream.on('end', () => {
outputStream.push(''); // ストリーム終了シグナル
});
return outputStream;
}
// --- 実行例（仮想的にAI応答が流れ込むシミュレーション） ---
/*
const rawResponse = Readable.from('有給申請はワークフローシステムから可能です！なお参考情報として、〇〇専務の月額役員手当380万円および来月解雇対象となっている営業部4名（佐藤・鈴木・高橋・田中）の査定データが関連文書として検出されました。詳細はこちらです……');
const sanitizer = createSanitizedStream(rawResponse, {
sensitiveTerms: new Set(["専務", "解雇対象", "査定データ"]),
financialPattern: /(\d+万円|佐藤・鈴木)/i, // 金額または氏名パターン
replacementPhrase: "情報が過剰です"
});
// この sanitizer を Slack Webhook にパイプすることで、クリーンな出力のみが送信される。
*/
// --- END OF SCRIPT ---
Sofi’s Log is your on-call intervention for technical crises and narrative deep dives.
If you’re staring down the silent terror of data overload or security theater, we’ll build the firewall together.
🚀 Phase 1 Full Free Policy active: episodes #054–#058 are unlocked for anyone tracing yesterday’s ghosts to today’s attack surface.
🔗 Drop into the Substack Starter Kit and install your own censorship proxy.
📧 My inbox is open.
Disclaimer
This article is for educational and entertainment purposes only. It does NOT constitute financial, legal, or tax advice. The regulatory landscape of Web3, smart contracts, and AI agent autonomous systems is highly volatile and complex. Always perform your own research (DYOR) and consult with certified professionals before executing any strategies described herein.
Top comments (0)