When you ship an LLM feature, the default instinct is to log full prompts and completions. It makes debugging easy, but it also builds a store of user content you now have to secure and expire. For most cost and reliability questions you don't need it.
1. Log metadata, not content
Request ID, model, input/output tokens, status and latency answer "why did spend jump?" without retaining user text:
{ "request_id": "req_123", "key": "prod-web", "model": "oroo-chatgpt", "input_tokens": 412, "output_tokens": 188, "status": 200, "ms": 940 }
(Example log shape only; values are illustrative.)
2. One key per app and environment, each with limits
Separate keys make usage attributable and shrink the blast radius. Set allowed models, an output-token cap and requests-per-minute per key so a runaway loop fails fast.
3. Treat key secrets like passwords
Server-side only, never in a client bundle. Show the secret once, store only a hash, rotate on a schedule.
4. IP allowlists for production keys
A leaked key that only works from your own servers is far less useful to an attacker.
5. A one-minute freeze procedure
Document how to freeze, rotate or revoke a key and make sure more than one person can do it.
6. Idempotent retries
Send a new Idempotency-Key for every new request and reuse it only when retrying that request, so a network retry doesn't trigger a second billable model call.
Common mistakes to avoid
- Shipping a key inside a mobile app "temporarily"
- Sharing one key between staging and production
- Retry loops with no ceiling and no idempotency
- Keeping raw prompts in logs forever because nobody set a retention period
- Only one person knowing how to revoke a key
Each of these is cheap to fix before launch and expensive to fix after an incident.
Where a gateway helps
I work on OROO Blue, a secure AI gateway for Indian teams that bakes these patterns in: one API and one OROO key for OpenAI, Google and Anthropic models; per-key usage and activity; secrets shown once and stored as a hash; optional IP allowlists; freeze/rotate/revoke; idempotent replays that aren't charged again; and OROO does not store prompts or completions. Billing is a prepaid INR wallet.
The patterns above apply whatever stack you use.
Full write-up: https://orookids.blogspot.com/2026/11/usage-without-secrets-6-privacy-habits.html
OROO Blue: https://orooblue.com
Top comments (0)