DEV Community

Cover image for Your GitHub Actions Logs Are Leaking LLM Keys and Your SIEM Isn't Catching It
Teycir Ben Soltane
Teycir Ben Soltane

Posted on

Your GitHub Actions Logs Are Leaking LLM Keys and Your SIEM Isn't Catching It

#devsecops #osint #api #llm

You've locked down your AWS credentials. You've got secret scanning on your repos. You rotate your database passwords.

But LLM API keys? Those are sitting in plaintext in your pipeline — and nobody's rotating them.

The problem nobody's talking about yet

LLM API keys exploded in the last two years. Every team has them now: OpenAI for the chatbot, Anthropic for the internal tool, Groq because someone read a benchmark. They get pasted into CI/CD workflows, hardcoded into Dockerfiles, committed in .env.example with real values, echoed in build logs.

The usual secrets scanning tools weren't built for them. GitLeaks and TruffleHog have patterns for AWS and Stripe, but coverage for sk-ant-api03-... or gsk_... is inconsistent. And unlike a database password, a leaked LLM key doesn't crash your app — it just silently drains your quota and potentially exposes your prompts.

What I keep finding in pipeline audits

During a recent audit of a client's GitHub Actions setup, I found three LLM API keys across two workflows:

  • One OpenAI key echoed in a debug step from 8 months ago
  • One Anthropic key hardcoded in a Docker build arg
  • One Groq key in a .env.staging file committed "temporarily"

All three were still live.

The hard part wasn't finding them — it was quickly assessing blast radius before writing the report. Which models do these keys unlock? Are they on a paid plan? What rate limits are attached? Writing provider-specific curl scripts for each one wastes time you don't have during an engagement.

Triaging fast with CheckAPIs

I've been using CheckAPIs for this step. Paste the keys, get back:

  • ✅ Valid / invalid status
  • 🤖 Full model list the key unlocks
  • ⏱ Response latency
  • 🏢 Organization/account info where available
  • 🚦 Rate limit headers

Supports 12+ providers: OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, HuggingFace, Replicate, Together AI, Perplexity, Azure, AWS Bedrock.

The important part for client work: everything runs client-side. The validation calls go directly from your browser to the provider's API — no proxy, no logging, no third party ever sees the key.

# Or use the API for automation
curl -X POST https://checkapis.pages.dev/api/check \
  -H "Content-Type: application/json" \
  -d '{"keys": ["sk-proj-...", "sk-ant-api03-..."]}'
Enter fullscreen mode Exit fullscreen mode

What to actually fix

Finding the key is step one. Here's the remediation checklist I hand off:

Immediate

  • Rotate every exposed key, assume it's compromised
  • Audit provider dashboards for unexpected usage spikes
  • Check if keys have org-level or project-level scope and tighten it

Pipeline hardening

  • Move all LLM keys to your secrets manager (Vault, AWS Secrets Manager, GitHub Secrets)
  • Never echo secrets in build steps — add set +x before any step that uses them
  • Add provider-specific patterns to your GitLeaks/TruffleHog config

Detection

  • Alert on LLM API spend anomalies — most providers have webhook or email alerts for quota thresholds
  • Add LLM key patterns to your SIEM if you're ingesting repo events

The bigger picture

LLM keys are credentials. They have blast radius: financial (quota drain), data (prompt/completion logs on the provider side), and reputational (your key used for abuse). Treat them exactly like you'd treat an AWS access key.

The tooling hasn't caught up yet — which means right now, in most orgs, they're the path of least resistance.

CheckAPIs is open source — github.com/Teycir/CheckAPI

Top comments (0)