I Grepped My Own Logs for 'token' as a Joke. It Was Not a Joke.

It was a slow Sunday and I was procrastinating on a deploy, so I ran a lazy little command against my Raspberry Pi's log directory — the one where my AI agents, my Flask app, and a handful of cron jobs all scribble their output:

grep -ri "token" /var/log/myapp/ ~/agent-logs/ | wc -l

I expected a handful. Maybe some OAuth debug noise from a library.

I got 11,214 lines.

And that was just token. When I widened the search to api_key, password, Authorization, and Bearer, the picture got worse. My logs contained live API keys, customer email addresses in plaintext, full webhook payloads (including one payment provider's signed events), and — my personal favorite — a complete dump of an SMTP credential set that I had, six months earlier, sworn I rotated after "that incident."

Nobody reads their logs. That's exactly why they're a dumping ground for secrets. Here's what I found, how it got there, and what I changed so it can't happen again.

How secrets end up in logs (it's never the obvious path)

I assumed I'd been careful. I never print(api_key) anywhere. The leaks came from four places I wasn't watching:

1. Exception tracebacks that include request context. My Flask error handler logged the full exception with locals. One route accepted an API key as a query parameter (legacy, I know) — and every 500 on that route dutifully wrote the key into the error log.

2. "Debug" HTTP logging left on after the bug was fixed. During one integration headache in June, I enabled full request/response logging in my HTTP client to chase a header bug. I fixed the bug on a Thursday. I removed the logging... never. Three months of full API responses — including one provider that echoes your auth header back in error payloads — sitting in a world-readable file.

3. My AI agent's transcripts. This was the big one. My agent framework logged the full tool-call payloads "for observability." That meant every time the agent called my CRM API, the request log included the Authorization: Bearer ... header. Every time it processed an inbound email, the raw message — customer addresses, sometimes order details, once a password reset link — went into a JSONL file that grows forever and that I had never once opened.

4. Environment dumps. A startup crash handler I wrote logged os.environ "to debug config issues on the Pi." It fired exactly twice. Both times it captured every secret in the environment. Two lines in a 400MB log file, each worth a full credential rotation.

The pattern: none of these were decisions to log secrets. They were decisions to log everything, made under time pressure, that outlived the pressure.

What I did the ugly Sunday afternoon

First, triage. I grepped for each credential type and asked one question per hit: is this still valid?

grep -rEoh "(sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{36}|Bearer [A-Za-z0-9._-]+)" ~/agent-logs/ | sort -u | head -20

That regex pulled out a deduplicated list of key-shaped strings. Several were expired test keys. Four were live:

• The OpenAI-compatible API key my agent used daily
• The SMTP password from the environment dump
• A GitHub fine-grained PAT (thankfully scoped read-only to one repo)
• The payment provider's webhook secret — which meant anyone with that log file could forge valid webhook signatures

I rotated all four within the hour. Then I deleted the log files — after archiving a scrubbed copy, because I wanted to learn from them, and because "delete the evidence" is not a remediation plan.

And here's the part I want to be honest about, because it's the part that stings.

The failure: I'd already "solved" this problem once

In March, I'd caught a different agent writing an API key into a public GitHub repo. I rotated the key, wrote a smug little post-mortem, and added a pre-commit hook that scans staged files for secret-shaped strings. I felt done with secrets-in-places-they-shouldn't-be.

But my pre-commit hook only guarded one sink: git. I had mentally filed the problem as "don't commit keys" when the actual problem is "keys flow into every passive data store you don't actively police." Logs, crash dumps, agent transcripts, analytics events, error trackers — they're all downstream of the same leak. I guarded the door I'd been robbed through and left the windows open.

The uncomfortable truth from that Sunday grep: the SMTP password had been sitting in a log file for six months, on a machine whose SSH port I had briefly exposed to the internet back in early September. I have no evidence anyone found it. "No evidence" doing a lot of work in that sentence.

The fix: treat logs as a secret store, because they are

1. Redaction at the logging boundary, not at the source. You cannot trust every library, framework, and agent tool to avoid logging secrets. So I wrote a logging filter that every handler passes through:

import re, logging

PATTERNS = [
    (re.compile(r'(sk-[A-Za-z0-9]{8})[A-Za-z0-9]{12,}'), r'\1…REDACTED'),
    (re.compile(r'(ghp_[A-Za-z0-9]{6})[A-Za-z0-9]{30}'), r'\1…REDACTED'),
    (re.compile(r'(Bearer\s+\S{6})\S+'), r'\1…REDACTED'),
    (re.compile(r'(api[_-]?key["\']?\s*[:=]\s*["\']?\w{4})\w+', re.I), r'\1…REDACTED'),
    (re.compile(r'[\w.+-]+@[\w-]+\.[\w.]+'), '[EMAIL]'),
]

class RedactFilter(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()
        for pat, repl in PATTERNS:
            msg = pat.sub(repl, msg)
        record.msg = msg
        record.args = ()
        return True

It's not perfect — regex redaction never is — but it converts "every secret leaks" into "only weirdly-shaped secrets leak," and that delta is huge. My agent framework's transcript logger now runs through the same filter.

2. Kill the environment dump. My crash handler now logs an allowlist of config names (DATABASE_URL: set, SMTP_PASS: set) — never values. If you need to know whether config is missing, presence is almost always enough.

3. Log rotation with a short fuse. logrotate on the Pi, 14-day retention, compressed, and — critically — the agent transcript JSONLs are included. Logs you keep forever are a honeypot with a growing attack surface. Logs you delete on a schedule cap the blast radius of any single leak.

4. A weekly canary grep in cron. This was the cheapest and highest-value change. Every Monday at 6am:

#!/usr/bin/env bash
HITS=$(grep -rEc "sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{36}|BEGIN.*PRIVATE KEY" /var/log/myapp/ ~/agent-logs/ 2>/dev/null | awk -F: '{s+=$2} END {print s+0}')
if [ "$HITS" -gt 0 ]; then
  echo "SECRET LEAK: $HITS key-shaped strings in logs" | mail -s "log-canary" [e-mail]
fi

Plus, once a month I plant a real canary: a fake sk-CANARY... string in an environment variable, then verify the redaction filter catches it in the logs. If the canary shows up unredacted, the filter regressed.

5. Assume anything logged before the fix is compromised. I rotated every credential that appeared in the old logs, even the "surely expired" ones. Rotation is cheap; the SMTP password living in a file for six months taught me that "surely" is not a security property.

Since that Sunday, the canary has fired twice: once when a new dependency's HTTP client bypassed my logger and printed raw headers to stdout (caught in a week instead of a quarter), and once when I fat-fingered a real key into a config value that got echoed at startup. Both were five-minute fixes. Both would have been silent leaks before.

Your logs are a database of everything your systems ever touched, written by code you didn't review, stored on a machine you might have once port-forwarded. Grep them this week — before someone with more motivation does.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50.