DEV Community

Cover image for The $100K Mistake: How One Exposed API Key Cost a Startup Everything
M.Azeem
M.Azeem

Posted on

The $100K Mistake: How One Exposed API Key Cost a Startup Everything

#github #security #ai #learning

A Developer's Guide to Preventing Secret Leaks in Your Code

It was 3 AM when Sarah's phone started buzzing non-stop.

Slack notifications. Email alerts. Her phone's emergency bypass alarm screaming. She grabbed her phone with trembling hands, already knowing something was catastrophically wrong.

Her AWS account was maxed out. Someone had found her OpenAI API key in a public GitHub commit from 6 months ago—a commit she'd made during a late-night coding session and completely forgotten about. The key was buried in a config file, pushed alongside a "quick fix" at 2:47 AM on a Tuesday.

What started as a small side project just cost her startup $97,000 in unauthorized API usage.

The bots had found her key within 4 hours of the commit. They'd been quietly draining her accounts for months.

You're More Vulnerable Than You Think

Sarah's story isn't unique. It's not even rare.

Here's the uncomfortable truth:

  • Over 6 million secrets are exposed on GitHub every year (GitGuardian 2023 Report)
  • The average time to discover an exposed secret: 20 days
  • Average cost of a data breach: $4.45 million (IBM 2023)
  • 72% of developers have accidentally committed secrets at least once
  • Automated bots scan GitHub every 30 seconds looking for fresh API keys

And here's the kicker: It's not happening to careless developers. It's happening to senior engineers at Fortune 500 companies.

Real Examples (Anonymized)

Case 1: The Senior Engineer
A principal engineer at a major tech company pushed a .env.backup file to a public repo during a refactoring. The file contained production database credentials. Within 12 minutes, attackers had accessed their customer database. Cost: $2.3M in breach notifications and legal fees.

Case 2: The Open Source Contributor
An open-source maintainer accidentally included their Stripe API key in an example config. Before they noticed (3 days later), $14,000 in fraudulent charges had been processed.

Case 3: The Startup CTO
A CTO made a private repo public to showcase their work for investors. They forgot about the AWS credentials in a 2-year-old commit. Their entire infrastructure was cryptomined within 6 hours. Cost: $43,000 + downtime.

The Anatomy of a Leak: How Smart Developers Make These Mistakes

Let's be honest: you know better. You've read the articles. You've sat through the security trainings. You've probably even judged other developers for making these mistakes.

So why does it keep happening?

1. The Late-Night Commit

It's 11:47 PM. You've been debugging for 4 hours. You finally got it working. Your brain is fried. You type:

git add .
git commit -m "fix: finally working"
git push
Enter fullscreen mode Exit fullscreen mode

You just committed your .env.local file. Your .gitignore had .env but not .env.local.

2. The "Quick Test" That Became Permanent 

// TODO: Move to env variables
const OPENAI_API_KEY = "sk-proj-abc123...";
Enter fullscreen mode Exit fullscreen mode

That TODO has been there for 8 months. You've committed it 47 times across different branches.

3. The Copy-Paste Cascade

You copy a working config from Slack to test something quickly:

# docker-compose.yml
environment:
  - DATABASE_URL=postgresql://admin:MySecretPass123@prod-db.company.com/users
  - API_KEY=sk-proj-real-key-here
Enter fullscreen mode Exit fullscreen mode

"I'll change this before committing," you think. But GitHub Copilot auto-commits, or you're using a GUI that stages everything by default.

4. The Git History Time Bomb

You realize your mistake and remove the secret:

git rm .env
git commit -m "remove env file"
git push
Enter fullscreen mode Exit fullscreen mode

Problem: The secret is still in your Git history. Forever. Anyone can access it with:

git log --all --full-history --source -- .env
Enter fullscreen mode Exit fullscreen mode

5. The "Private Repo" Illusion

Your repo is private today. But:

  • You might make it public for your portfolio
  • You might fork it to a public personal account
  • A collaborator might fork it
  • Your company might open-source it
  • GitHub might have a security breach

Private ≠ Secure

The Bot Army That Never Sleeps

While you sleep, an army of automated bots scans GitHub 24/7. They're not sophisticated AI—they're simple pattern-matching scripts that search for:

sk-proj-[a-zA-Z0-9]{20,}          # OpenAI keys
ghp_[a-zA-Z0-9]{36}                # GitHub tokens  
AKIA[0-9A-Z]{16}                   # AWS access keys
-----BEGIN PRIVATE KEY-----        # SSH/SSL keys
mongodb://.*:.*@                   # Database URLs
Enter fullscreen mode Exit fullscreen mode

The moment you push a commit with a pattern match:

  1. Seconds 0-30: Bot discovers your key
  2. Seconds 30-60: Key is validated (tested against the API)
  3. Minutes 1-5: If valid, key is sold on dark web marketplaces ($5-$500 depending on type)
  4. Minutes 5-∞: Attackers use your key until you notice

You're in a race against machines that never blink, never sleep, and never get tired.

Prevention Strategies: A Multi-Layered Approach

Here's what doesn't work:

❌ "I'll be more careful" - You will forget

❌ "I'll manually check each commit" - You'll miss things

❌ "We'll do quarterly security audits" - Too late

Here's what does work:

✅ Automated scanning before every push

✅ Real-time monitoring of your repositories

✅ Proper secret management tools

✅ Team education and culture change

✅ Multiple layers of defense

Layer 1: Prevention at the Source

Use Environment Variables Properly

# .env (NEVER commit this)
OPENAI_API_KEY=sk-proj-your-key
DATABASE_URL=postgresql://...

# .env.example (Safe to commit)
OPENAI_API_KEY=your-openai-key-here
DATABASE_URL=postgresql://user:password@host/db
Enter fullscreen mode Exit fullscreen mode

Update Your .gitignore

# Environment files
.env
.env.local
.env.*.local
.env.development
.env.production
.env.test

# Config files that might contain secrets
config/secrets.yml
config/database.yml
config/credentials.yml

# IDE-specific files
.vscode/settings.json
.idea/workspace.xml

# OS files
.DS_Store
Thumbs.db
Enter fullscreen mode Exit fullscreen mode

Use Secret Management Services

Instead of environment variables, use proper secret management:

  • AWS Secrets Manager - For AWS infrastructure
  • HashiCorp Vault - For multi-cloud environments
  • Azure Key Vault - For Azure deployments
  • Google Cloud Secret Manager - For GCP
  • Doppler - Developer-friendly secret management
  • 1Password - For team secret sharing 
// Instead of this:
const apiKey = process.env.OPENAI_API_KEY;

// Do this:
const secretsManager = new AWS.SecretsManager();
const apiKey = await secretsManager.getSecretValue({ 
  SecretId: 'production/openai/api-key' 
}).promise();
Enter fullscreen mode Exit fullscreen mode

Layer 2: Pre-Commit Detection

Use Pre-Commit Hooks

Pre-commit hooks run before your code is committed, catching secrets before they enter Git history.

# Install pre-commit framework
pip install pre-commit

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']
Enter fullscreen mode Exit fullscreen mode

Popular Pre-Commit Tools:

  1. detect-secrets (Yelp)

    • Prevents secrets from entering codebase
    • Establishes baseline of known secrets
    • pip install detect-secrets

  2. git-secrets (AWS Labs)

    • Scans commits for AWS credentials
    • Can be integrated globally
    • brew install git-secrets

  3. gitleaks

    • Fast, configurable secret scanner
    • Works offline
    • brew install gitleaks

  4. TruffleHog

    • High entropy string detection
    • Scans entire Git history
    • pip install truffleHog

Layer 3: Repository Scanning

Enable GitHub's Native Secret Scanning

GitHub provides free secret scanning for public repositories and GitHub Advanced Security for private repos.

How to enable:

  1. Go to repository Settings → Security → Code security and analysis
  2. Enable "Secret scanning"
  3. Enable "Push protection" (blocks pushes with secrets)

What GitHub detects:

  • API keys from 100+ services
  • Private keys
  • Database connection strings
  • Cloud provider credentials

Third-Party Scanning Tools:

  1. GitGuardian (Free tier available)

  2. TruffleHog (Open source)

    • Scans Git repositories for secrets
    • High entropy detection
    • Can scan S3 buckets, filesystems
    • docker run trufflesecurity/trufflehog github --org=yourorg

  3. Gitleaks (Open source)

    • Fast and configurable
    • CI/CD integration
    • Custom rule support
    • gitleaks detect --source . --verbose

  4. Custom Scripts

    • Build your own scanner for organization-specific patterns
    • Integrate with your existing tools
    • Example: Simple Python scanner using regex patterns

For example, I built a lightweight Python scanner that searches GitHub using their API for common secret patterns. It's useful for ad-hoc scans and can be customized for your specific needs. There are many similar tools available - the key is to use something rather than nothing.

Layer 4: CI/CD Integration

Automate Scans in Your Pipeline

Add secret scanning to your CI/CD pipeline:

# GitHub Actions example
name: Security Scan
on: [push, pull_request]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0  # Full history for scanning

      - name: Run Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Run TruffleHog
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: main
Enter fullscreen mode Exit fullscreen mode

GitLab CI Example:

secret_detection:
  image: python:3.9
  script:
    - pip install detect-secrets
    - detect-secrets scan --all-files --force-use-all-plugins
  only:
    - merge_requests
    - main
Enter fullscreen mode Exit fullscreen mode

Layer 5: Continuous Monitoring

Set Up Automated Scanning

# Cron job for nightly scans (Linux/Mac)
# crontab -e
0 2 * * * cd /path/to/repos && gitleaks detect --source . --report-path daily-scan.json

# Send alerts to Slack
0 2 * * * /path/to/scan-and-alert.sh
Enter fullscreen mode Exit fullscreen mode

scan-and-alert.sh example:

#!/bin/bash
RESULTS=$(gitleaks detect --source /path/to/repo --no-git)

if [ $? -ne 0 ]; then
  curl -X POST -H 'Content-type: application/json' \
    --data "{\"text\":\"⚠️ Secrets detected in repository! Check the logs.\"}" \
    $SLACK_WEBHOOK_URL
fi
Enter fullscreen mode Exit fullscreen mode

The Developer's Pre-Push Security Checklist

Print this out and put it next to your monitor:

Before Every Push:

# 1. Review what you're committing
git diff --staged

# 2. Check for common secret patterns
git diff --staged | grep -E "(api[_-]?key|secret|password|token|private.*key)"

# 3. Verify .gitignore is working
git status --ignored

# 4. Run your pre-commit hooks manually (if not automatic)
pre-commit run --all-files

# 5. Review commit history
git log --oneline -5
Enter fullscreen mode Exit fullscreen mode

Weekly Security Habits:

☐ Run a full repository scan with gitleaks or similar

☐ Review access logs for your API keys

☐ Check for unusual billing activity

☐ Rotate any keys that seem suspicious

☐ Update team on any security incidents

Monthly Team Practices:

☐ Security training session (30 minutes)

☐ Review organization-wide scan results

☐ Update .gitignore templates

☐ Audit team member access

☐ Review secret management procedures

What To Do If You've Exposed a Secret

Don't panic. Act fast.

Immediate Response (Within 5 Minutes)

1. Revoke the exposed secret immediately

Don't wait to investigate. Revoke first, ask questions later.

  • AWS: IAM Console → Users → Security Credentials → Delete Access Key
  • GitHub: Settings → Developer settings → Personal access tokens → Delete
  • OpenAI: Platform → API Keys → Revoke key
  • Stripe: Developers → API keys → Roll key
  • Google Cloud: IAM & Admin → Service Accounts → Delete key

2. Generate a new secret

  • Use a password manager or key generation tool
  • Don't reuse any part of the old secret
  • Document the rotation in your incident log

3. Update all services

  • Production environments
  • Staging/dev environments
  • Team members' local configurations
  • CI/CD pipelines
  • Documentation (if key was used in examples)

Short-Term Response (Within 1 Hour)

4. Remove the secret from Git history

⚠️ Warning: This rewrites history. Coordinate with your team first.

# Option 1: BFG Repo-Cleaner (fastest, recommended)
# Download from: https://rtyley.github.io/bfg-repo-cleaner/

# Create a file with the secret to remove
echo "sk-proj-abc123..." > secrets.txt

# Run BFG
bfg --replace-text secrets.txt your-repo.git

cd your-repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force

# Option 2: git filter-branch (slower, more control)
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch .env" \
  --prune-empty --tag-name-filter cat -- --all

# Option 3: git filter-repo (modern replacement for filter-branch)
pip install git-filter-repo
git filter-repo --invert-paths --path .env
Enter fullscreen mode Exit fullscreen mode

5. Check for unauthorized usage

Look for evidence of compromise:

  • AWS CloudTrail: Check for unusual API calls
  • API dashboards: Look for spike in usage
  • Billing alerts: Unexpected charges
  • Database logs: Unauthorized access attempts
  • Application logs: Unusual patterns 
# AWS CLI example: Check recent API calls
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=AKIA... \
  --max-items 100 \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S)
Enter fullscreen mode Exit fullscreen mode

6. Document the incident

Create an incident report:

  • What was exposed (type of secret, service)
  • When it was committed (timestamp, commit hash)
  • How long it was public (approximate exposure window)
  • What actions were taken (timeline)
  • Evidence of unauthorized access (if any)
  • Lessons learned

Long-Term Response (Within 1 Week)

7. Implement preventive measures

Don't let it happen again:

  • Set up pre-commit hooks (detect-secrets, gitleaks)
  • Enable GitHub secret scanning
  • Configure push protection
  • Add the tool/service to your CI/CD pipeline
  • Set up billing alerts

8. Review your security practices

  • Audit all repositories for similar issues
  • Update .gitignore templates organization-wide
  • Implement proper secret management (Vault, AWS Secrets Manager)
  • Train team on security best practices
  • Create runbooks for common scenarios

9. Consider legal/compliance requirements

Depending on the severity:

  • Notify affected users (if data was accessed)
  • File security incident reports (if required)
  • Contact legal/compliance teams
  • Update security documentation
  • Review insurance coverage

Building a Security-First Culture

Technology alone won't solve this problem. You need systems that make the right thing the easy thing.

1. Make Secrets Management Painless

Bad approach (but easy):

// Hardcoded - developers will do this if it's easier
const apiKey = "sk-proj-abc123";
Enter fullscreen mode Exit fullscreen mode

Better approach (still manual):

// Environment variables
const apiKey = process.env.OPENAI_API_KEY;
Enter fullscreen mode Exit fullscreen mode

Best approach (automated + audited):

// Secret management service
const apiKey = await vault.getSecret('openai-api-key');
Enter fullscreen mode Exit fullscreen mode

Make it easy:

# Create a CLI tool for your team
secrets get openai-api-key

# Or a simple wrapper
source <(secrets env production)
Enter fullscreen mode Exit fullscreen mode

2. Automate Everything

If it requires a human to remember, it will eventually fail.

Automation checklist:

  • ✅ Pre-commit hooks that auto-scan
  • ✅ CI/CD that blocks merges with secrets
  • ✅ Scheduled scans of all repositories
  • ✅ Automatic alerts to Slack/email
  • ✅ Automated secret rotation (where possible)
  • ✅ Onboarding automation (new devs get hooks configured)

3. Blameless Post-Mortems

When someone exposes a secret (and they will), don't blame the person. Fix the system.

Bad response: "You should have been more careful!"

Good response: "What can we automate to prevent this?"

Post-mortem template:

## Incident: Exposed API Key - 2024-02-14

**What happened:** OpenAI API key committed to public repo
**Impact:** $3,200 in unauthorized usage
**Root cause:** .env.local not in .gitignore
**Detection:** Billing alert triggered

**Timeline:**
- 14:23: Secret committed
- 14:45: Bots discovered and began using
- 09:15 (next day): Billing alert fired
- 09:22: Investigation began
- 09:30: Key revoked

**What went well:**
- Billing alerts caught it within 24h
- Team responded quickly
- Clear incident response process

**What went wrong:**
- No pre-commit hooks installed
- .gitignore was incomplete
- No developer training on secret management

**Action items:**
- [ ] Add .env.local to organization .gitignore template
- [ ] Install pre-commit hooks on all developer machines
- [ ] Add gitleaks to CI/CD pipeline
- [ ] Schedule security training session
- [ ] Document secret management procedures
Enter fullscreen mode Exit fullscreen mode

4. Make Security Visible

Create a dashboard showing:

  • ✅ Last scan date for each repository
  • ✅ Number of secrets found/resolved
  • ✅ Mean Time To Remediation (MTTR)
  • ✅ Team compliance rate
  • ✅ Days since last incident

Gamify it:
"Our team has gone 47 days without a secret exposure! 🎉"

5. Education Over Punishment

Monthly "Security Fridays":

  • 30-minute team session
  • Real-world case studies
  • Hands-on practice with security tools
  • Guest speakers from security team
  • Share wins and near-misses

Onboarding checklist for new developers:

## Security Onboarding

- [ ] Review secret management policy
- [ ] Install pre-commit hooks
- [ ] Set up 1Password/secret manager
- [ ] Enable 2FA on all accounts
- [ ] Complete security training module
- [ ] Shadow an incident response drill
Enter fullscreen mode Exit fullscreen mode

The Economics of Prevention

Let's do the math:

Cost of Prevention:

Tools (per year):

  • GitHub Advanced Security: $0 (public repos) or $49/user/month
  • GitGuardian: Free tier available, $18/dev/month for teams
  • Pre-commit hooks: Free (open source)
  • Gitleaks/TruffleHog: Free (open source)
  • Secret management (Doppler/Vault): $0-12/user/month

Time investment:

  • Initial setup: 2-4 hours
  • Training: 2 hours/year per developer
  • Ongoing maintenance: 30 minutes/week
  • Pre-commit hook slowdown: 2-5 seconds per commit

Total annual cost for a 10-person team: ~$5,000 - $10,000

Cost of Recovery (Average Incident):

Direct costs:

  • Emergency response: 8 hours × $150/hr = $1,200
  • Unauthorized API usage: $5,000 - $100,000+
  • Legal fees: $25,000+
  • Compliance fines: $50,000 - $500,000
  • Customer notifications: $100,000+

Indirect costs:

  • Customer churn
  • Brand damage
  • Lost productivity
  • Insurance premium increases
  • Recruiting challenges

Total incident cost: $180,000 - $1,000,000+

ROI of prevention: 18:1 to 200:1

You literally cannot afford NOT to invest in prevention.

Tools Comparison Matrix

Here's a quick reference for choosing the right tools for your needs:

Tool Type Cost Best For Integration
GitHub Secret Scanning SaaS Free/Paid GitHub users Native
GitGuardian SaaS Free/Paid Teams, enterprise CI/CD, IDE
Gitleaks CLI Free Fast scanning, CI/CD GitHub Actions
TruffleHog CLI Free Deep history scans Docker, CI/CD
detect-secrets CLI Free Pre-commit hooks Git hooks
git-secrets CLI Free AWS-focused teams Git hooks
Custom scanners Script Free/DIY Specific patterns Flexible

Recommendation: Use a layered approach:

  1. Pre-commit hooks (detect-secrets or git-secrets)
  2. CI/CD scanning (gitleaks or TruffleHog)
  3. Continuous monitoring (GitGuardian or GitHub Advanced Security)
  4. Periodic deep scans (custom scripts or TruffleHog)

Getting Started Today (10-Minute Setup)

You don't need to implement everything at once. Start small:

Quick Win #1: Enable GitHub Secret Scanning (2 minutes) 

1. Go to your repository
2. Settings → Security → Code security and analysis
3. Enable "Secret scanning" and "Push protection"
4. Done!
Enter fullscreen mode Exit fullscreen mode

Quick Win #2: Add Pre-Commit Hook (5 minutes) 

# Install detect-secrets
pip install detect-secrets

# Initialize in your repo
cd your-repo
detect-secrets scan > .secrets.baseline

# Create pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
detect-secrets scan --baseline .secrets.baseline
if [ $? -ne 0 ]; then
    echo "❌ Secrets detected! Commit blocked."
    exit 1
fi
EOF

chmod +x .git/hooks/pre-commit
Enter fullscreen mode Exit fullscreen mode

Quick Win #3: Update .gitignore (3 minutes) 

# Add these to your .gitignore
cat >> .gitignore << 'EOF'

# Environment files
.env
.env.*
!.env.example

# Secret files
secrets/
*.key
*.pem
credentials.json
EOF
Enter fullscreen mode Exit fullscreen mode

Next Steps:

  1. This week: Install pre-commit hooks on all developer machines
  2. This month: Add scanning to CI/CD pipeline
  3. This quarter: Implement proper secret management (Vault/AWS Secrets Manager)
  4. Ongoing: Monthly security reviews and training

Common Questions

Q: What if I find secrets in old commits from years ago?

A: Assume they're compromised and rotate them immediately. Then decide if you need to rewrite Git history (breaking change) or just document the rotation.

Q: Are private repositories safe?

A: Safer, but not safe. Always treat secrets as if they could become public tomorrow.

Q: What about secrets in dependencies?

A: Use tools like npm audit, pip-audit, or bundler-audit to check for compromised packages. Consider using Dependabot or Renovate for automated updates.

Q: How do I handle secrets in local development?

A: Use .env files (gitignored) or a secret manager. Never hardcode, even locally - it's too easy to forget and commit.

Q: Should I rotate all keys after setting up scanning?

A: At minimum, audit them. If you've never scanned before, assume some were exposed and rotate high-value keys (payment, production databases).

Q: What about secrets in screenshots/documentation?

A: Always redact secrets in screenshots. Use fake/example keys in documentation. Tools like redacted can help automate this.

The Bottom Line

The next Sarah could be you, your colleague, or someone on your team.

But you have a choice. You can take 10 minutes today to:

  • Enable GitHub secret scanning
  • Install a pre-commit hook
  • Update your .gitignore
  • Scan your existing repositories

Or you can wait until 3 AM when your phone starts buzzing.

Prevention is always cheaper than recovery.

Resources & Further Reading

Tools Mentioned:

Secret Management:

Learning Resources:

Take Action Now

Don't wait for an incident. Start securing your code today:

  1. ⚡ Scan your repos - Run gitleaks or TruffleHog on your repositories
  2. 🔒 Enable protection - Turn on GitHub secret scanning and push protection
  3. 🪝 Install hooks - Set up pre-commit hooks with detect-secrets
  4. 📚 Educate your team - Share this guide with your colleagues
  5. 🔄 Rotate suspicious keys - If you find anything, rotate immediately

Help Spread Awareness

If this guide helped you:

  • 📢 Share it with your team
  • 🐦 Post on Twitter/LinkedIn
  • 💬 Discuss in your community
  • ✍️ Write about your own experiences

Together, we can make the developer ecosystem more secure.

Have you ever accidentally exposed a secret? You're not alone. Share your story (anonymously) in the comments. Let's learn from each other's mistakes and build better systems together.

About This Guide

This guide is meant to educate developers about the risks of exposed secrets and provide practical, actionable steps to prevent them. Security is everyone's responsibility, and with the right tools and practices, we can all write safer code.

Contributing: Found an error or have a suggestion? This guide is open to community improvements.

Last updated: February 2026

Top comments (0)