Team-Level LLM API Key Management: The Guide I Wish Someone Gave Me

#llm #security #devops #api

When your app was a side project, one .env file with an OpenAI key was fine. The moment a second engineer joins, that setup turns into a small disaster: keys leak into git, a junior's experiment blows the budget, and no one knows who spent what.

Here's the system we ended up with after burning two budgets and rotating three keys.

1. Never share one key across the team

The instinct is to put the production key in 1Password and call it a day. The problem:

You can't revoke one person's access without killing everyone's
You can't attribute spend
When that key leaks (and it will), you rotate it for the whole team

Give each developer their own key. Scope it down.

2. Separate environments from users

Two axes to think about:

Axis	Values
Environment	dev / staging / prod
Owner	each engineer + CI bots

That's a matrix of keys, not one key. A CI key should never have production write access. An intern's dev key should never work in staging.

3. Budget per key, not per account

OpenAI's account-level limit is too coarse — one runaway loop burns everyone's quota. You want per-key spend caps:

# Pseudocode: what the gateway should enforce
per_key_budget = {
  "dev-llong":      {"monthly": 20,  "models": ["gpt-4o-mini"]},
  "prod-service-a": {"monthly": 500, "models": ["gpt-4o","claude-3-5-sonnet"]},
  "ci-bot":         {"monthly": 5,   "models": ["gpt-4o-mini"]},
}

4. Rotate, but make it cheap

Key rotation should be a 5-minute operation, not a sprint. If every rotation means updating 30 service configs, people stop doing it. A gateway layer with one ANTHROPIC_API_KEY-equivalent that routes to upstream providers lets you rotate the upstream key without touching the app.

5. Audit log everything

You want to know, after an incident:

Which key made the call
Which model and how many tokens
From which service

Without this, "who burned $400 last night?" becomes a he-said-she-said.

6. What we actually use

We ran through Envoy + custom Lua scripts, then a vendor-specific dashboard, before landing on a gateway that does per-key budgets, per-model routing, and a unified audit log out of the box. Agent Token is what we use today — one OpenAI-compatible key, per-key spend caps, team usage dashboard across GPT / Claude / Gemini.

The pattern matters more than the tool, though: one key per person per environment, hard budgets, and an audit log you'd actually check.

How does your team manage LLM keys? Genuinely curious what works for bigger orgs.

DEV Community