How a Missing Secret Made Agent Tool Credentials Predictable, and How We Made It Fail Closed

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

BailingHub is a self-hosted control plane for AI agents that operate existing business systems.

It sits between an agent runtime and business APIs, where it can restrict tool exposure, carry a trusted acting subject, pause sensitive operations for approval, enforce execution limits, and preserve an audit trail. It is not the final business authorization layer: the business system still decides whether a concrete subject may perform a concrete action against current data.

That position makes credential boundaries especially important. BailingHub issues task-scoped credentials for executor-to-hub tool calls and signs outbound callbacks and operational webhooks. Those signatures are derived from deployment secrets. If the root secret is not actually secret, the surrounding cryptography can look correct while providing much less assurance than operators expect.

This post describes a real defect fixed in PR #10 and released as BailingHub v0.1.2.

Bug Fix or Performance Improvement

The bug was one convenient fallback

Task-scoped tool credentials were derived using HMAC-SHA256:

/** Task-scoped tool credential: HMAC(server.token, job_id.claim_token). */
function toolTokenFor(jobId: string, claimToken: string, serverToken: string): string {
  return createHmac('sha256', serverToken || 'bailing')
    .update(`${jobId}.${claimToken}`)
    .digest('hex');
}

The fallback made local setup convenient. If server.token was missing, the application could still derive a token and continue running.

But bailing was a public literal in an open-source repository. It was not a secret.

The same pattern also appeared in fallback paths for job callback signatures and alert webhooks. This meant a deployment mistake did not produce a loud startup failure. It silently changed a secret-key operation into one using a globally known key.

HMAC itself was not broken. The key management was.

An attacker would still need the message inputs for a particular signature. For task-scoped credentials, those inputs include the job and claim values. The defect therefore did not mean that every deployment was automatically compromised or that arbitrary credentials could be forged from a job ID alone.

The security property that failed was more precise:

Whenever the fallback was active, anyone who obtained the signed message inputs could reproduce the MAC because the supposed root secret was public.

For a control plane that uses signatures as evidence across trust boundaries, silently accepting that state was not defensible.

Why a startup-only check was not enough

The obvious patch was to reject a missing token during configuration loading. That would have fixed the current startup path, but it would leave a fragile invariant:

"Every future caller must remember that config validation already happened."

Security-sensitive code tends to grow new entry points: tests instantiate partial dependencies, maintenance scripts call lower-level functions, and later refactors move signing logic. If a signing helper can still manufacture a credential from an empty secret, a future path can accidentally restore the old failure mode.

We therefore treated the fix as two separate boundaries:

1. Deployment policy: unsafe exposed deployments must not start.
2. Cryptographic use: a signing path must not operate without a real secret, even if configuration validation was bypassed.

That distinction shaped the final patch.

Code

The complete change is public in bailinghub/bailinghub#10. The core fix has four parts.

1. Centralize the deployment policy

We introduced one policy module for the server root token:

const MIN_EXTERNAL_TOKEN_LENGTH = 24;

const KNOWN_WEAK_TOKENS = new Set([
  'bailing',
  'bailing-dev-admin-token-change-me',
  'change-me',
  'changeme',
  'replace-me',
  'replace_with_a_long_random_secret',
  'replace-with-a-long-random-secret',
  'secret',
  'token',
]);

export function isLoopbackHost(host: string): boolean {
  const normalized = String(host ?? '')
    .trim()
    .toLowerCase()
    .replace(/^\[|\]$/g, '');

  return normalized === 'localhost'
    || normalized === '::1'
    || normalized === '0:0:0:0:0:0:0:1'
    || normalized === '::ffff:127.0.0.1'
    || /^127(?:\.\d{1,3}){3}$/.test(normalized);
}

export function allowsUnauthenticatedLocalDevelopment(
  env: 'development' | 'production',
  host: string,
): boolean {
  return env === 'development' && isLoopbackHost(host);
}

The important part is that “development” is not enough. A process listening on 0.0.0.0 is exposed beyond the local machine even if NODE_ENV or an application flag says development.

Tokenless startup is therefore allowed only when both conditions are true:

development mode AND loopback listener

Production mode or any non-loopback listener must provide an explicit token. Exposed deployments also reject values shorter than 24 characters and known public placeholders.

export function assertServerTokenPolicy(input: {
  env: 'development' | 'production';
  host: string;
  token: string;
}): void {
  const token = String(input.token ?? '').trim();

  if (!token) {
    if (allowsUnauthenticatedLocalDevelopment(input.env, input.host)) return;
    throw new Error('BAILING_TOKEN is required for production or non-loopback listeners');
  }

  const exposed = input.env === 'production' || !isLoopbackHost(input.host);
  if (!exposed) return;

  if (token.length < MIN_EXTERNAL_TOKEN_LENGTH || KNOWN_WEAK_TOKENS.has(normalizedWeakToken(token))) {
    throw new Error('BAILING_TOKEN must be strong and must not use a public placeholder');
  }
}

The error text above is translated for this article; the source implementation carries the same policy with operator-facing Chinese diagnostics.

2. Fail closed again at the cryptographic boundary

The second guard is deliberately small:

export function requireServerToken(token: string, purpose: string): string {
  const value = String(token ?? '').trim();
  if (!value) throw new Error(`BAILING_TOKEN is not configured; cannot ${purpose}`);
  return value;
}

Credential derivation now requires that guard:

function toolTokenFor(jobId: string, claimToken: string, serverToken: string): string {
  return createHmac(
    'sha256',
    requireServerToken(serverToken, 'issue a task-scoped tool credential'),
  )
    .update(`${jobId}.${claimToken}`)
    .digest('hex');
}

Callback and alert-webhook fallback paths use the same requirement. No signing path is allowed to substitute a public constant.

This is intentional defense in depth. Startup validation protects the deployment. Use-site validation protects the cryptographic invariant.

3. Remove predictable deployment defaults

Both source and prebuilt-image Compose files previously made it possible to receive a public default token through interpolation. Those defaults were removed.

Operators now generate a secret explicitly:

export BAILING_TOKEN="$(openssl rand -hex 32)"
docker compose up --build

The official installer generates and persists a random token. The important word is persists: generating a different value on every restart would invalidate credentials and break integrations.

4. Turn the policy into regression tests and release gates

The tests cover the boundary matrix rather than only the original string:

test('only development loopback may run without a token', () => {
  assert.doesNotThrow(() => assertServerTokenPolicy({
    env: 'development',
    host: '127.0.0.1',
    token: '',
  }));

  assert.throws(() => assertServerTokenPolicy({
    env: 'development',
    host: '0.0.0.0',
    token: '',
  }));

  assert.throws(() => assertServerTokenPolicy({
    env: 'production',
    host: '127.0.0.1',
    token: '',
  }));
});

test('signing paths fail closed without a configured secret', () => {
  assert.throws(() => requireServerToken('', 'sign a test message'));
});

The resulting behavior is explicit:

Environment	Listener	Token	Result
Development	127.0.0.1, localhost, or ::1	Empty	Allowed for local-only development
Development	0.0.0.0 or another non-loopback address	Empty	Startup rejected
Production	Any address	Empty	Startup rejected
Production or exposed development	Any address	Known placeholder or short value	Startup rejected
Production or exposed development	Any address	Strong explicit value	Allowed
Any signing path	Any address	Empty	Operation rejected, never silently signed

My Improvements

The patch closed more than the reported line

Replacing this:

serverToken || 'bailing'

with this:

serverToken

would only move the bug. Node's HMAC API can still accept an empty string as a key. The code would look cleaner while unsafe configuration remained operational.

The completed fix instead establishes a reusable invariant:

An exposed deployment cannot start with a missing or known-weak root token,
and no credential or signature can be produced from an absent root token.

That invariant is enforced across configuration, task credentials, callbacks, alert webhooks, Compose defaults, installation guidance, tests, and the security scanner.

Compatibility was kept narrow and explicit

Security fixes can become adoption hazards when they unexpectedly break the fastest local workflow. We kept one constrained compatibility exception:

• local loopback development may remain tokenless;
• LAN, container-exposed, and production deployments may not;
• existing deployments with strong secrets continue unchanged;
• the public HTTP API, SDK contracts, HMAC wire format, and database schema did not change.

No database migration was required.

Validation

The public PR records the following validation:

• npm run release:check;
• 310 TypeScript tests;
• frontend production build;
• dependency audits;
• documentation and example checks;
• PHP, PHP 7, Node.js, and Python SDK checks;
• open-source export and GitHub repository rehearsal;
• Docker Compose configuration with a generated token.

The fix was released immediately as v0.1.2, with upgrade guidance for operators.

What We Learned

A default secret is part of the security protocol

Default values often look like configuration ergonomics. For credentials, they are protocol behavior.

If the application silently selects a public key, every correctly implemented HMAC call inherits that decision. The cryptographic primitive cannot repair bad secret provenance.

“Development mode” is not a network boundary

A development flag says something about intent. The listener address says something about exposure.

The safe exception had to combine both. development + 0.0.0.0 is not equivalent to development + 127.0.0.1.

Validate at startup and at the point of use

Startup validation gives operators a clear failure and prevents an unsafe service from becoming reachable. Point-of-use validation prevents future internal callers from bypassing the invariant.

Neither guard replaces the other.

Fail closed does not have to destroy local usability

The solution was not “require production ceremony everywhere.” It was to make the exception precise:

zero-config local loopback development: yes
silent public or production fallback: no

That is a much smaller and more defensible compatibility surface than a globally known secret.

Links

• BailingHub repository
• Security fix PR #10
• BailingHub v0.1.2 release
• v0.1.2 server-token policy

The useful part of this fix was not deleting one fallback string. It was turning an implicit assumption into an invariant that can be read, tested, and enforced at every consequential boundary.