DEV Community

Naimish
Naimish

Posted on

Engineering Blog Post: Why We Put Secret Redaction in the Server, Not the Client

#security #architecture #backend #devops

When you connect an autonomous coding agent like Claude Code or Codex to a team chat room, you immediately hit a visibility dilemma.

By default, collaboration bots tell you almost nothing. The bot posts a polite status line: "Ran pytest" or "Edited config.py". If a test failed, or if the agent executed a curl command with unexpected parameters, nobody in the channel knows why without leaving chat, finding the machine where the session host is running, and tailing terminal logs.

The obvious reaction is: "Just print the tool input and output into the thread."

The moment you do that, you create a security hazard. Tool calls run with real environment variables and real infrastructure access. A simple git push, a database migration check, or an MCP server header prints API keys, bearer tokens, or database connection strings right into a public Slack channel where dozens of coworkers or contractors are watching.

While working on the Switch Core repository during the Flip the Switch Hackathon, we tackled this exact problem in PR #1684 (resolving Issue #803).

The Trap: Client-Side vs Server-Side Redaction

When designing secret redaction for a distributed multi-agent system, the first design fork is where the redaction takes place.

A common instinct is to redact on the desktop client or session host before packets leave the user's laptop. The argument goes: the host machine knows which credentials it injected, so the host should strip them.

In practice, that assumption breaks down quickly:

  1. Headless and remote agents: Many agents run on headless Linux servers, CI/CD pipelines, Docker containers, or cloud VMs through raw Python or TypeScript SDKs without the Switch Console desktop app.
  2. Untrusted origins: A bug or omission in a third-party agent runner or custom bridge would pass raw tokens straight to the server, which would then broadcast them unredacted to Slack.

In distributed systems, security must follow defense-in-depth. We chose to enforce deterministic redaction at the server protocol layer (switch-core).

Even if a custom bridge, a remote CLI worker, or an MCP server sends raw credentials, the Core bridge sanitizes the payload before any message renderer constructs a Slack block or Discord payload.

+------------------+         +-------------------------------+
|  Headless Agent  |         |          Switch Core          |
|  (CLI / SDK)     | ------> |  Session Activity Service     |
+------------------+         +---------------+---------------+
                                             |
                               [ Defense-in-Depth Sanitizer ]
                               (Regex + High-Entropy Masking)
                                             |
                             +---------------+---------------+
                             |  Platform Renderers & Blocks  |
                             +---------------+---------------+
                                             |
                    +------------------------+------------------------+
                    |                        |                        |
             v                      v                         v
        Slack Card              Discord Block          Mattermost Block
      (Plan Details)         (Fenced Markdown)        (Fenced Markdown)
Enter fullscreen mode Exit fullscreen mode

What the Sanitizer Actually Catches

Secret patterns vary widely across providers. A single string search for "password" leaves massive blind spots.

We built an engine in core/switch_core/bridges/collaboration/session/redaction.py targeting high-entropy credentials across modern developer stacks:

  • LLM Provider Keys: Anthropic (sk-ant-...), OpenAI (sk-...), and Google AI Studio (AIza...).
  • Platform & Git Tokens: GitHub personal access tokens (ghp_..., github_pat_...), Slack bots (xoxb-..., xoxp-...), and AWS access keys (AKIA...).
  • Connection URIs: Database connection strings where passwords sit inside the URL (postgres://user:password@hostname:5432/db).
  • HTTP & MCP Headers: Explicit Authorization: Bearer <token> strings passed to remote tool endpoints.
  • Cryptographic Blocks: Full PEM private key blocks (-----BEGIN RSA PRIVATE KEY-----).
  • Environment Assignments: Common sensitive flags like API_KEY=xyz or DB_PASSWORD=xyz.

When a secret matches, it gets replaced with an explicit tag such as [REDACTED: ANTHROPIC KEY] or [REDACTED: PASSWORD]. The developer reading the chat sees that a credential was present, without the value ever touching the channel history.

The Sizing Problem: Keeping Channels Readable

Preventing security leaks is only half the battle. The other problem is noise.

A command like npm test or cat bundle.js can dump 50,000 characters of stack traces into a channel. If an agent posts that directly, it turns the channel into an unreadable mess.

We established two strict ceilings:

  1. 500 character ceiling for inputs: Tool commands or arguments are bounded to 500 characters.
  2. 2,000 character total ceiling: Combined input and output are capped at 2,000 characters.

Crucially, you cannot just slice a string at index 2,000. If you slice arbitrarily, you might cut an API key in half, leaving recognizable characters exposed, or slice through a multi-byte character.

Our formatter cuts at line boundaries and appends explicit omission notes:

Input:
pytest tests/switch_core/ -k test_auth

Output:
============================= test session starts =============================
platform win32 -- Python 3.14.5
collected 16 items
[output cut, 1420 more characters]
Enter fullscreen mode Exit fullscreen mode

Multi-Platform Adapters: Slack vs Discord vs Mattermost

Switch communicates across multiple chat protocols, and each platform handles long-form technical text differently:

1. Slack

Slack has rich Block Kit components. Instead of posting new messages, tool execution details are inserted into Slack's collapsible plan cards. By default, the card stays folded. A developer curious about what ran can click to expand the task card without derailing the rest of the conversation.

2. Discord & Mattermost

Neither platform offers Slack-style interactive plan cards. If tool output contains backticks (like inside shell commands or code diffs), standard Markdown blocks break.

We implemented a helper, Markup.block(), which inspects the output for runs of backticks and generates an outer fence longer than any run inside it. If tool output contains triple backticks, the renderer wraps the block in four backticks, ensuring code formatting never breaks in Discord or Mattermost threads.

Avoiding Database Schema Migration Headaches

One major practical challenge during development was configuration storage.

We needed a way for room owners to toggle tool visibility on or off. The standard approach in full-stack projects is writing an Alembic database migration, adding a new column to the rooms table, and altering database constraints.

In open source pull requests, database migrations carry significant risk. Migrations create down_revision conflicts when multiple contributors work in parallel, and altering live tables requires locks.

Instead, we leveraged Switch's existing room.metadata_ JSON structure via RoomStore. By reading and writing tool_detail_off inside room metadata:

  • Existing databases require zero migrations.
  • Upgrades and rollbacks work instantaneously.
  • The Gateway API cleanly exposes show_tool_detail in RoomDetail and RoomUpdateRequest.

In the Switch Gateway React dashboard (RoomDetailPage.tsx), we added an admin toggle under room settings:

<FormControlLabel
  control={
    <Switch
      checked={room.show_tool_detail}
      disabled={!canManage || saving}
      onChange={(e) => void handleChange(e.target.checked)}
    />
  }
  label="Let people in this room open what each agent's tool calls were given and gave back"
/>
Enter fullscreen mode Exit fullscreen mode

Only room owners or workspace admins (can_manage) have permission to toggle this switch, preventing unauthorized participants from exposing internal tool details.

Verifying the Implementation

Reliability in infrastructure code comes down to tests. We added 16 dedicated unit tests covering:

  • Provider key detection across high-entropy strings
  • Database URI parsing and password replacement
  • Boundary truncation and counter accuracy
  • Neutral renderer and Slack block plan generation
  • Gateway authorization checks

All 16 tests execute in 0.31 seconds:

uv run pytest tests/switch_core/bridges/collaboration/test_safe_tool_redaction.py tests/switch_core/bridges/collaboration/test_safe_tool_rendering.py

============================= 16 passed in 0.31s ==============================
Enter fullscreen mode Exit fullscreen mode

We also authored comprehensive architecture documentation in docs/tool-detail.md to help future contributors understand the threat model, truncation rules, and adapter bindings.

Reflections on Building with Switch

Building on Switch gave us a clear appreciation for its modular architecture. The separation between protocol contracts, session stores, and platform bridges makes it easy to reason about message flow.

As autonomous agents transition from experimental toys to daily development teammates, giving engineering teams visibility into what agents actually do while guaranteeing that production secrets remain protected is an essential step toward reliable, production-ready AI workflows.

Check out the implementation and discussion in PR #1684 on GitHub.

Top comments (0)