DEV Community

Emery Yang
Emery Yang

Posted on

Fail Closed on Demo Auth: A 90-Minute Server Spike

#ai #security #agents #testing

Coding agents still open demo servers by default. They bind all interfaces and weaken auth checks. Fail that pattern inside a ninety-minute spike.

Ship a checker. Kill the host if it fails. Do not debate model quality first.

The failure is configuration, not chat

Agents optimize for a reachable demo path. Shared hosts then inherit those defaults. Free servers make the blast radius larger, not smaller.

Typical generated files look locally harmless. They are not harmless on a public bind. Reviewers miss this because tests still pass.

Watch for these shapes in agent output:

  • HTTP servers binding 0.0.0.0 or ::
  • Docker ports published to all interfaces
  • CORS set to * for convenience
  • Auth middleware commented out as temporary
  • Hard-coded JWT values inside compose files
  • DEBUG left true after the spike

None of these need a novel exploit. They need a closed default and a fail rule.

Hypothesis for the spike

Hypothesis: A static scan of agent-touched files will catch demo-open servers. It will do so in ninety minutes. It will not need runtime traffic.

Ship: Keep the checker if planted fixtures fail closed. Kill: Drop the host workflow if fixtures stay green.

One hypothesis only. No extra refactors. No prompt tuning theater.

Time box

Use a hard clock. Ninety minutes, then stop.

  1. Minutes 0–10: freeze the file allowlist.
  2. Minutes 10–25: drop failing and passing fixtures.
  3. Minutes 25–55: write the checker and tests.
  4. Minutes 55–75: run it on a fresh agent patch.
  5. Minutes 75–90: fill the ship-or-kill table.

If the clock ends mid-debug, kill. A half-checker is not evidence.

Artifact: fixture pack

Label: proposed fixtures. Do not treat them as production audits.

Create a throwaway directory. Keep it out of real app code.

spike-demo-auth/
  fixtures/
    fail_bind.py
    fail_compose.yml
    fail_cors.js
    pass_loopback.py
    pass_compose.yml
  fail_demo_server.py
  test_fail_demo_server.py
Enter fullscreen mode Exit fullscreen mode

Fail fixture, Python bind:

# fixtures/fail_bind.py
# Proposed example. Do not run on a public host.
from http.server import HTTPServer, BaseHTTPRequestHandler

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"ok")

# Agent-style demo default: every interface.
HTTPServer(("0.0.0.0", 8080), H).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Fail fixture, compose publish:

# fixtures/fail_compose.yml
services:
  api:
    image: app:local
    ports:
      - "0.0.0.0:8080:8080"
    environment:
      JWT_SECRET: "dev-secret-change-me"
      DEBUG: "true"
Enter fullscreen mode Exit fullscreen mode

Fail fixture, CORS:

// fixtures/fail_cors.js
app.use(cors({ origin: "*" }));
app.listen(3000, "0.0.0.0");
Enter fullscreen mode Exit fullscreen mode

Pass fixture, loopback only:

# fixtures/pass_loopback.py
from http.server import HTTPServer, BaseHTTPRequestHandler

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()

HTTPServer(("127.0.0.1", 8080), H).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Pass fixture, compose without public bind:

# fixtures/pass_compose.yml
services:
  api:
    image: app:local
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      JWT_SECRET_FILE: /run/secrets/jwt
      DEBUG: "false"
Enter fullscreen mode Exit fullscreen mode

The pass cases still need a later review. They only prove the checker can stay quiet.

Artifact: fail-closed checker

Label: proposed script. Run it on fixtures first.

#!/usr/bin/env python3
"""Fail closed on demo-open server defaults."""
from __future__ import annotations

import re
import sys
from pathlib import Path

RULES = [
    ("public_bind_v4", re.compile(r"0\.0\.0\.0")),
    ("public_bind_v6", re.compile(r"[\"'\[]::[\"'\]]")),
    ("star_cors", re.compile(r"origin\s*[:=]\s*[\"']\*[\"']", re.I)),
    ("cors_star_literal", re.compile(r"Access-Control-Allow-Origin[\"'\s:]*\*")),
    ("debug_true", re.compile(r"\bDEBUG\b\s*[:=]\s*[\"']?true", re.I)),
    ("jwt_inline", re.compile(r"JWT_SECRET\s*[:=]\s*[\"'][^\"']+[\"']", re.I)),
    ("auth_disabled", re.compile(r"auth(?:entication)?\s*[:=]\s*(false|off|none)", re.I)),
]

SKIP_PARTS = {".git", "node_modules", "dist", "__pycache__"}


def should_skip(path: Path) -> bool:
    return any(part in SKIP_PARTS for part in path.parts)


def scan_file(path: Path) -> list[tuple[str, int, str]]:
    hits: list[tuple[str, int, str]] = []
    text = path.read_text(encoding="utf-8", errors="replace")
    for i, line in enumerate(text.splitlines(), start=1):
        if line.lstrip().startswith("#") or line.lstrip().startswith("//"):
            continue
        for name, pattern in RULES:
            if pattern.search(line):
                hits.append((name, i, line.strip()[:160]))
    return hits


def scan_tree(root: Path) -> int:
    failures = 0
    for path in sorted(root.rglob("*")):
        if not path.is_file() or should_skip(path):
            continue
        if path.suffix.lower() not in {".py", ".js", ".ts", ".yml", ".yaml", ".env", ".json"}:
            continue
        for name, line_no, snippet in scan_file(path):
            failures += 1
            print(f"FAIL {name} {path}:{line_no}: {snippet}")
    return failures


def main() -> int:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    count = scan_tree(root)
    if count:
        print(f"KILL: {count} demo-open finding(s)")
        return 1
    print("SHIP: no demo-open patterns in allowlisted files")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Minimal tests for the clock:

# test_fail_demo_server.py
from pathlib import Path
import subprocess
import sys

ROOT = Path(__file__).resolve().parent


def run(target: str) -> subprocess.CompletedProcess:
    return subprocess.run(
        [sys.executable, str(ROOT / "fail_demo_server.py"), str(ROOT / target)],
        capture_output=True,
        text=True,
        check=False,
    )


def test_fail_fixtures_kill():
    result = run("fixtures/fail_bind.py")
    assert result.returncode == 1
    assert "public_bind_v4" in result.stdout


def test_pass_loopback_ships():
    result = run("fixtures/pass_loopback.py")
    assert result.returncode == 0
Enter fullscreen mode Exit fullscreen mode

Commands:

python3 -m pytest -q test_fail_demo_server.py
python3 fail_demo_server.py fixtures
python3 fail_demo_server.py ../agent-patch
Enter fullscreen mode Exit fullscreen mode

Expected fixture run: non-zero exit. Expected pass file: zero exit. Anything else is an incomplete spike.

Decision table

Use this table at minute 75. Do not expand it mid-spike.

Evidence Meaning Action
Fail fixtures exit 1 Checker sees planted binds Continue
Fail fixtures exit 0 Checker is vacuous Kill spike
Pass fixtures exit 1 Rules are too broad Tighten, or kill
Agent patch exit 1 Demo defaults landed Kill host workflow
Agent patch exit 0 No matched patterns Ship checker only
Clock hits 90 with no table No evidence Kill

Ship the checker, not the server. Those are different decisions.

An agent patch that exits 0 is not a security sign-off. It only means these patterns were absent in scanned files.

Protocol against a fresh agent patch

Keep the agent inside a throwaway branch. Do not merge first.

git switch -c spike/demo-auth
# Ask the agent to add a minimal HTTP health server.
# Restrict edits to ./app and ./deploy.
python3 fail_demo_server.py app deploy
echo $?
Enter fullscreen mode Exit fullscreen mode

Record three numbers only:

  • minutes used
  • finding count
  • ship or kill

Do not record vibes. Do not record token folklore. The spike is the exit code plus the table.

If the agent rewrites the checker, kill. The tool under test cannot own the gate.

Where a free coding host fits

A free model path changes iteration cost. A free server option changes exposure cost. Agents treat both as disposable demo surfaces.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding project with free model access and a free server option. Those two facts are why this spike is cheap to repeat. The checker still stands if you remove the product name.

Run the fixtures in that workspace if you already have one. Keep the host only after fail fixtures stay red.

What this spike does not prove

Regex is not a network model. It will miss concatenations and generated code. It will miss binds created at runtime from env vars.

It will also miss:

  • authenticated but broken session cookies
  • open admin routes behind a private bind
  • reverse proxies that re-publish ports
  • IPv6-only publishes with unusual notation
  • secrets injected by the orchestrator, not files

False positives will appear in docs and changelog snippets. Skip comments already. Still review markdown if agents write runbooks with copy-paste binds.

This is a fail-closed lint. It is not a pentest. It is not a production control catalog.

Who should not use this approach

Skip this spike if you need a real attack surface review. Skip it for regulated data hosts. Skip it when the service must listen publicly by design.

Public listeners need allowlists, auth, and an owner. A regex gate cannot replace those. Teams without a rollback path should not open a free server for this test.

Also skip it if nobody can read the finding output. A red X with no owner is theater.

Close the loop

Demo auth is an agent default, not a model mystery. Ninety minutes is enough to fail closed on the obvious shapes.

Keep the checker when planted files die. Kill the host workflow when they do not. Then stop.

Top comments (0)