Every distributed job system eventually hits the same wall.
Your executor needs credentials to do its job: a GitHub token, a database password, an API key. So you inject them as env vars and move on. That works until it doesn't. A compromised executor, a log line with a secret in it, a misconfigured container and the credential is out.
The real question is:
How do you give an executor the ability to use a secret without ever sending it the secret?
The core idea
Instead of sending credentials, the coordinator sends a signed payload with credential references: pointer strings that look like this:
"github_token": "cref://tenant_abc/github_token"
That string is useless on its own. To get the real value, the executor has to call a credential store and prove:
- It has a one-time request token proving this job was legitimately issued
- Its own registered client identity matches
- It is within the correct tenant scope
The credential store validates all three, marks the resolution as used, and returns a short-lived secret valid only for the expected duration of the job.
The executor uses it. It expires. Even if intercepted mid-flight, it is dead.
6 independent security layers
What makes this robust is not one clever trick. It is six independent layers stacked on top of each other:
- HMAC signature - payload is signed with a per-executor shared secret. Can't forge without the key.
- RS256 JWT (one-time) - issued per job, 5 minute TTL. Executors can verify but cannot create tokens.
- Nonce burn - nonce stored in Redis, burned on first use. Replay impossible even with a valid JWT.
- Credential store auth chain - refs only resolvable with valid token + matching client ID + matching tenant.
- Mutual TLS (mTLS) - both sides present CA-signed certs. The endpoint URL alone is useless without one.
- Network isolation - egress firewall on executor. Compromised executor cannot exfiltrate to arbitrary endpoints.
Bypassing one layer does not help with the others. An attacker who steals the request token still hits the nonce burn. A valid nonce without the HMAC gets nowhere.
7 attack scenarios, all blocked
The full post walks through each one: direct calls with no token, replay attacks, stolen credential references, stolen request tokens, expired token replays, misdirected payloads, and compromised executor exfiltration. Each one hits a different layer and stops cold.
The weak point (and it is not what you think)
The shared secret is a long-lived symmetric key sitting in an env var and it is the operational weak link, not the pattern itself. The post covers the full lifecycle: registration, runtime signing, rotation with a 5-minute grace TTL for in-flight jobs, and exactly where the raw secret lives (and where it must never appear).
Read the full deep-dive
This is a teaser. The full post has the complete request payload structure, the 8-step validation chain, step-by-step credential resolution flow, all 7 attack scenarios with exact rejection points, gotchas (canonical JSON, Redis as a dependency, clock sync), and full code snippets.
Read the full post on anonymouscoderdev.dev: https://anonymouscoderdev.dev/the-zero-trust-credential-pattern-nobody-talks-about/
No paywall. No sign-up. MIT forever.
Building in the dark. Knowledge is free.
Top comments (0)