DEV Community

Cover image for Observability without prompt storage: securing LLM API keys in production
OROO Kids
OROO Kids

Posted on Originally published at orookids.blogspot.com

Observability without prompt storage: securing LLM API keys in production

When you ship an LLM feature, the default instinct is to log full prompts and completions. It makes debugging easy, but it also builds a store of user content you now have to secure and expire. For most cost and reliability questions you don't need it.

1. Log metadata, not content

Request ID, model, input/output tokens, status and latency answer "why did spend jump?" without retaining user text:

{ "request_id": "req_123", "key": "prod-web", "model": "oroo-chatgpt", "input_tokens": 412, "output_tokens": 188, "status": 200, "ms": 940 }
Enter fullscreen mode Exit fullscreen mode

(Example log shape only; values are illustrative.)

2. One key per app and environment, each with limits

Separate keys make usage attributable and shrink the blast radius. Set allowed models, an output-token cap and requests-per-minute per key so a runaway loop fails fast.

3. Treat key secrets like passwords

Server-side only, never in a client bundle. Show the secret once, store only a hash, rotate on a schedule.

4. IP allowlists for production keys

A leaked key that only works from your own servers is far less useful to an attacker.

5. A one-minute freeze procedure

Document how to freeze, rotate or revoke a key and make sure more than one person can do it.

6. Idempotent retries

Send a new Idempotency-Key for every new request and reuse it only when retrying that request, so a network retry doesn't trigger a second billable model call.

Common mistakes to avoid

  • Shipping a key inside a mobile app "temporarily"
  • Sharing one key between staging and production
  • Retry loops with no ceiling and no idempotency
  • Keeping raw prompts in logs forever because nobody set a retention period
  • Only one person knowing how to revoke a key

Each of these is cheap to fix before launch and expensive to fix after an incident.

Where a gateway helps

I work on OROO Blue, a secure AI gateway for Indian teams that bakes these patterns in: one API and one OROO key for OpenAI, Google and Anthropic models; per-key usage and activity; secrets shown once and stored as a hash; optional IP allowlists; freeze/rotate/revoke; idempotent replays that aren't charged again; and OROO does not store prompts or completions. Billing is a prepaid INR wallet.

The patterns above apply whatever stack you use.

Full write-up: https://orookids.blogspot.com/2026/11/usage-without-secrets-6-privacy-habits.html
OROO Blue: https://orooblue.com

Top comments (0)