DEV Community: Dwayne McDaniel

Stop Leaking API Keys: The Backend for Frontend (BFF) Pattern Explained

Dwayne McDaniel — Thu, 13 Aug 2026 12:54:39 +0000

👉 TL;DR: Frontend applications (SPAs, mobile apps, desktop clients) cannot securely store secrets: any embedded API key is extractable by users and attackers.
The Backend for Frontend (BFF) pattern solves this by placing a server-side layer between your frontend and third-party APIs. The BFF holds the secrets; the frontend never sees them.
For production deployments, use a secrets manager (AWS Secrets Manager, HashiCorp Vault) rather than environment variables to enable rotation and auditing.
A BFF adds infrastructure complexity, but for any API key with financial or administrative implications, the tradeoff is worth it.

Frontends are notoriously leaky environments. Cybernews found in 2022 that 56% of Android apps on the Google Play Store contained hardcoded secrets extractable through basic automation. A similar study in 2025 concluded that iOS apps are not better, with over 815,000 secrets harvested from 156,000+ apps (71% leaking at least one credential).

These studies plainly expose the widespread issue of hard-coding secrets in production-deployed frontend code. This article aims to warn developers about this risk and present a simple, reusable pattern for safeguarding their applications: the Backend for Frontend (BFF) pattern.

Before we start, let's be clear on the crucial point: Whether you are building a React Single Page Application (SPA), a mobile app, or a desktop client, if the code runs on the user's device, the user (and potential attackers) can always inspect it. The solution isn't to try and hide the keys better; it's to move them somewhere safe.

"Public Clients" vs. "Confidential Clients"

In OAuth terminology, there are two types of clients, with completely different security models:

Confidential Clients: Applications running on a secure server (e.g., a Node.js backend, Python API) that can securely store secrets (like a CLIENT_SECRET) because end-users don't have access to the server's file system or memory.
Public Clients: Applications running in an environment you don't control (e.g., browsers, mobile devices). No matter how you obfuscate your code or use .env files during the build process, the final artifact (JS bundle, APK) is distributed to the user. This inherent vulnerability is why hardcoded secrets remain a staple of the OWASP Mobile Top 10.

A common misconception is that using environment variables in a frontend framework (like REACT_APP_API_KEY) secures the key. In reality, during the build process, these variables are embedded directly into the JavaScript strings. Anyone can grep your bundled code or inspect the Network tab in their browser to see the API key being sent in headers.

This is why you should never embed secrets in a frontend application, but instead rely on a lightweight backend service to handle authorized requests on behalf of the frontend.

The Backend for Frontend (BFF) Pattern

The Backend for Frontend (BFF) pattern (originally popularized by SoundCloud) involves creating a dedicated backend layer specifically for your frontend application. While it solves many problems (like data aggregation), its security benefits are arguably its strongest asset. In a BFF architecture, your frontend never communicates directly with the sensitive third-party API (e.g., Stripe, OpenAI, Contentful). Instead, it talks to your BFF, and your BFF talks to the service.

Here is an example flow to process a payment:

The frontend sends a request to the BFF (e.g., POST /api/process-payment). No API keys are needed here, just the user's session cookie.
The BFF validates the user's session. It then retrieves the necessary secrets (e.g., STRIPE_SECRET_KEY) from its own secure server-side environment variables or a secrets manager.
The BFF attaches the secret key and forwards the request to the external service.
The service (Stripe) responds to the BFF, which can then filter or format the data before sending it back to the frontend.

How do you deploy it?

There is no single "correct" way to deploy a BFF, but there are two common patterns depending on your team's size and complexity.

1. "Integrated" BFF

For modern web development using frameworks like Next.js, Nuxt, Remix, or SvelteKit, the BFF is often built directly into the frontend project structure. The code lives in the same repository, and uses the same deployment workflow as the frontend code.

With React Server Components (RSC), the server-side logic never even gets sent to the browser bundle. The UI components run in the user's browser (Public Client), while API routes, Server Actions, and Server Components run on a secure Node.js/Edge runtime (Confidential Client). This is your typical Vercel, Netlify, or similar platform deployment.

2. "Standalone" BFF

For mobile apps or complex enterprise systems, a separate backend service is often used. The BFF would live in a separate repository, deployed as a Docker container or Serverless function.

The big advantage is a strict separation of concerns: the BFF can be maintained by a different team, which is convenient for microservices architectures, or when the frontend is a pure SPA (e.g., plain Vite + React) without server capabilities.

This is also the mandatory pattern for mobile apps, since native apps don't have server-side rendering. There are also some specific points to keep in mind for mobile development:

Session management: Cookies work on mobile but require explicit handling (e.g., CookieJar in OkHttp for Android, HTTPCookieStorage on iOS). The best way is to use short-lived tokens stored in secure platform storage (Android Keystore, iOS Keychain).
Certificate pinning: Consider pinning your BFF's TLS certificate to prevent man-in-the-middle attacks on the app-to-BFF connection.
For apps already in the Firebase ecosystem, Firebase Cloud Functions can serve as a lightweight BFF without managing additional infrastructure.

3. Managed Solutions: Serverless BFF

You don't need to manage servers to run a BFF. Cloud providers offer fully managed compute that fits the pattern well.

AWS API Gateway + Lambda is the most common serverless BFF stack:

API Gateway acts as the entry point, handling HTTPS termination, request routing, and optional request validation
Lambda functions execute your BFF logic: validating sessions, fetching secrets, and calling third-party APIs
Secrets Manager stores API keys that Lambda retrieves at runtime
CloudWatch provides logging and monitoring out of the box

This architecture scales automatically and costs effectively nothing when idle (you pay per request, not per hour). Google Cloud offers an equivalent pattern with Cloud Functions or Cloud Run fronted by API Gateway or Cloud Endpoints, pulling secrets from Secret Manager.

When to use serverless vs. containers: Serverless BFFs work well when your API proxying is straightforward and request volumes are moderate or spiky. If you need persistent connections (WebSockets), have consistent high traffic, or require complex request processing, a containerized BFF on ECS, Cloud Run, or Kubernetes may be more cost-effective and flexible.

BFF vs. API Gateway

It's easy to confuse a BFF with an API gateway, as both sit between the client and backend services. However, they serve different purposes:

An API gateway is a global entry point for all clients (web, mobile, partners) which handles cross-cutting concerns like authentication, rate limiting, and SSL termination. Typically owned by the Platform/DevOps team.
A BFF is usually specific to one frontend (e.g., the mobile app BFF, the web dashboard BFF). It focuses on data formatting, aggregation, and UI-specific logic. Typically owned by the Frontend team.

In mature architectures, the BFF often sits behind the API gateway: the gateway handles infrastructure concerns while the BFF optimizes the interaction for its specific client.

Implementation: Securing the BFF

Simply adding a Node.js middleware doesn't automatically solve everything. You must ensure the channel between the Frontend and the BFF is also secure.

1. Cookie-Based Sessions over Tokens

Instead of sending a JSON Web Token (JWT) to the frontend to be stored in localStorage (where it is vulnerable to XSS attacks), the BFF should handle authentication and issue an HttpOnly, Secure, SameSite cookie.

HttpOnly: JavaScript cannot read the cookie, preventing XSS theft.
Secure: Cookie is only sent over HTTPS.
SameSite: Mitigates CSRF attacks. Use SameSite=Strict for sensitive actions, or SameSite=Lax if you need the cookie sent on top-level navigations.

2. Proxying API Requests

The BFF acts as a proxy between your frontend and external services. In Next.js, a Route Handler can forward requests while injecting credentials the client never sees:

// app/api/external-service/route.ts
import { NextRequest, NextResponse } from 'next/server';
export async function POST(request: NextRequest) {
  const body = await request.json();
  // Forward the request to the external API with the secret key
  const response = await fetch('https://api.external-service.com/endpoint', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${process.env.SERVICE_API_KEY}`, // Secret stays on server
    },
    body: JSON.stringify(body),
  });
  const data = await response.json();
  return NextResponse.json(data);
}

3. Secrets Management: Beyond Environment Variables

Using process.env.API_KEY on your BFF is a starting point, but production deployments need a proper secrets manager. The reason is that sensitive API keys eventually need rotation.

Side note: not all API keys are sensitive. If unsure, ask yourself: does this key cost money, access user data, or grant write permissions?

This becomes critical during incident response. If a secret detection tool like GitGuardian alerts you to a leaked credential, you need to rotate the key immediately, ideally within minutes, not hours. A secrets manager makes this possible: update the secret in one place, and all consuming services pick up the new value without code changes or redeployments.

In addition, a dedicated secrets manager provides:

Centralization: Update a secret once, and all services consuming it pick up the change
Scheduled rotations
Audit trails (who accessed which secrets and when)
Access control: Grant the BFF permission to specific secrets without exposing your entire credential store

4. Injecting Secrets During CI/CD

The BFF deployment pipeline must get secrets into the runtime environment without storing them in code or build artifacts:

For serverless: reference the secrets manager directly in your function code via IAM role permissions (no environment variables needed in deployment config).
For containers (ECS, Kubernetes, Cloud Run): Inject secrets at runtime, not build time. The container image itself should contain no credentials!
For integrated BFFs (Vercel, Netlify): Set secrets through the platform's dashboard or CLI. Never commit them to your repository, even in a .env.production file!

Hardening the BFF Layer

Moving secrets to the BFF eliminates frontend exposure, but the BFF itself becomes a security-critical component. A few considerations:

Rate limiting: Implement request throttling at the API gateway level or within your BFF code.
Input validation: Validate and sanitize all inputs before forwarding requests. Don't blindly proxy user-supplied parameters to external APIs.
Least privilege: Grant the BFF access only to the secrets it needs. Use separate secrets for different services rather than a single credential store with everything.
Logging and monitoring: Log API usage patterns (without logging the secrets themselves) to detect anomalies.
Session validation: Every request to the BFF should validate the user's session before proxying to external services.

Conclusion

Frontend applications are public clients by design. Any secret embedded in browser JavaScript, a mobile app bundle, or a desktop executable is a secret shared with every user and every attacker willing to spend five minutes with developer tools.

The BFF pattern addresses this by placing a server-controlled layer between your untrusted client and the services that require authentication. Combined with a secrets manager for rotation and proper CI/CD practices for deployment, this architecture keeps credentials where they belong: on infrastructure you control.

Is it more complexity than shipping a React app with an API key in the bundle? Yes. But that complexity buys you secrets that rotate, access that audits, and credentials that don't appear in security researcher blog posts about the next batch of leaked API keys.

What AI Agents Can Teach Us About NHI Governance

Dwayne McDaniel — Tue, 11 Aug 2026 19:54:50 +0000

Artificial intelligence (AI) is a broad field with many practical applications. Over the past few years, we have seen explosive growth in generative AI, driven by systems like ChatGPT, Copilot, and other interactive tools that help developers write code and users create content. More recently, we have also seen the rise of "Agentic AI," in which orchestrators coordinate actions across one or more AI agents to perform tasks on behalf of a user.

While that can sound futuristic, even here in 2026, the reality is a little simpler.

AI systems, no matter how they are deployed, are just processes running on machines. They may live on a laptop, in a container, inside a virtual machine, or deep in a cloud environment. Fundamentally, they are software executing instructions, albeit probabilistic ones rather than hard-coded deterministic programs. And like every other subsystem we have ever built, they need a way to communicate safely.

This is where our real problem begins.

As we rush to adopt agentic AI, we are repeating a familiar mistake. We are focusing on capability and speed while leaving non-human identity (NHI) security and governance as an afterthought, by connecting AI tools to sensitive systems (repos, cloud, ticketing, secrets) without consistently applying least privilege. That gap has existed for years with CI systems, background jobs, service accounts, and automation.

Agentic AI is not inventing the gap, but it is quickly widening it.

Trust Decides Everything

Security in any system, whether it is a chatbot, a CI worker, or a long-running daemon, eventually boils down to trust.

Who is making the request? What are they allowed to do? And under what specific conditions?

Modern architectures focus on "zero trust," but many of these systems still rely on a single fragile factor for access: long-lived secrets, often taking the form of API keys or never-expiring certificates. The risk really comes from keeping static tokens in environment variables, dotfiles, build logs, or shared vault paths accessible to many processes, including AI tools. When those access keys leak, anyone who finds or steals them can grab everything the secret allows access to, until someone notices.

What zero trust actually requires is separation of concerns.

Authentication should prove that an entity is who or what it claims to be. Authorization should define exactly what that entity is allowed to do. Without that separation, we end up granting broad, standing permissions that are difficult to track, difficult to revoke, and extremely attractive to attackers.

This is the heart of non-human identity (NHI) governance. NHIs are broadly defined as anything that is not a human but still authenticates and connects to other running systems. This includes bots, scripts, workloads, service accounts, and CI jobs. And now, that list includes agents and agentic systems.

The control plane is the same. Identity, ownership, lifecycle, and permissions. Agentic AI, fundamentally, does not alter this reality. Under the hood, these systems are all advanced math, running on a chip somewhere.

What feels different is how we interact with them. We give them names like Claude and Cursor. We assign them personalities. We treat them like coworkers rather than workloads.

That anthropomorphization subtly shifts how we, as humans, think about trust and responsibility.

Ironically, that shift can be helpful.

Agents: NHIs in CI Pipelines, Terminals, and Browsers

Once you start treating an agent like an actor, not too dissimilar from a human, it becomes easier to apply time-tested identity patterns. OAuth, a delegated authorization standard, and OpenID Connect, the identity layer on top of OAuth, exist because we learned, over decades, that standing privilege does not securely scale. Short-lived, verifiable credentials usually beat permanent keys. Scoped permissions beat "just give it access" every time, from a safety perspective.

This "agents as if they were human actors" analogy does break down in some key places, though. For example, since agents have no fingerprints or ability to stop and pull out their phone, traditional multi-factor authentication cannot be bolted onto a background process. But the core idea holds: every entity accessing a system should be provable, attributable, and constrained.

That becomes obvious when you follow agents into the environments where they actually operate.

Where CI Meets The Robot

Continuous integration pipelines are no longer just build scripts. It is the complex dance where code becomes reality, pulling in dependencies, relying on task runners, and hopefully passing thorough testing and review processes.

If an agent can read a failing build, modify code, open a pull request, and iterate until green, that is a productivity win. It also means the agent needs access to repositories, build logs, artifacts, and sometimes deployment pathways. CI already has a history of being over-permissioned because teams value uptime and speed over locked-down security postures.

Agents tend to inherit that bias. "Give it what it needs" becomes "give it everything so it does not block the sprint."

Agents In The Command Line

In terminals, the risk is less glamorous and more common. Terminals are full of implicit trust, which looks like environment variables, config files, copied tokens, and debugging output. These habits all made sense when only a human was driving or could access the guarded developer's machine.

Agents in a terminal context can act quickly, which is the point. They can also surface secrets quickly, copy them into logs, paste them into tickets, or echo them into places you did not intend, as we saw in repeated attacks across 2025, such as Nx's S1ngularity and Shai Hulud. It's enough to have secrets stored in places that are routinely copied, logged, or shared during debugging. Agentic tools can increase the volume and speed of this copying, which raises the likelihood of accidental exposure unless guardrails, including redaction, scanning, and review, are in place.

You do not need sci-fi for things to go wrong. You just need carelessness with credentials and a system that rewards moving fast.

On Your Behalf Across The Internet

In browsers, the stakes are higher because browser agents often operate within authenticated sessions. This is the sharp edge of "acting on your behalf." If an agent can click around internal tools, approve actions, download data, or change configuration through an admin console, the question is which identity is it using? What permissions does it have? Can you reconstruct what happened later and tune for those edge cases?

This is why framing agents as NHIs is so useful. It avoids the fantasy. It forces the boring questions:

Who owns the agent, and who gave it those permissions?
What is its lifecycle?
What is it allowed to do, precisely?
How do we observe and audit its actions?

If you cannot answer those, your organization is not "doing agentic AI." It is running ungoverned automation at scale. And that is dangerous.

Governance that scales starts with inventory and alignment

Treating NHIs, including AI agents, with rigor sounds obvious until you try to do it. The hard part is not the principle. It is the grind.

The work starts with understanding what you already have. Inventory is unavoidable.

There is no shortcut around accounting for existing credentials, services, agents, and automations. This traditionally has been slow, difficult, and often uncomfortable work, especially in legacy environments. But it is also where progress begins, because you cannot reduce risk you cannot see.

Inventory also forces a second reality into the open: accountability for permissions.

Who Owns That Agent?

If an agent can access a system, that access was provisioned by someone: either explicitly, via tokens, OAuth grants, service accounts, or implicitly, by running inside an already-authenticated session. The governance challenge is ensuring every access path has a named owner, least-privilege scope, auditability, and a rotation/revocation process.

While it would be convenient to have a person to blame, like the developer who set everything up and then left, the reality of ownership and accountability is more nuanced and requires a shift in how we think about responsibility models.

Every team needs to be accountable for rotation, revocation, and incident response. This is the difference between governance and hope. When ownership is unclear, offboarding fails, cleanup stalls, and leaks become prolonged incidents.

There is also the organizational challenge and reality that, to get NHI governance under control, it will require creative collaboration across traditionally siloed parts of the business. No single team can own identity and access management at scale across all humans, workloads, agents, and whatever evolves next.

Who Accounts For That Agent's Access?

IAM, DevSecOps, security, platform, and development teams all touch this problem from different angles. Success depends on alignment around shared north stars, not isolated tooling decisions made under delivery pressure.

For years, non-human identity evolved bottom-up. Developers solved immediate problems with whatever mechanism worked. That flexibility helped systems scale, but it also created fragmentation, duplication, and governance gaps. As agentic AI becomes operationally significant and strategically imperative, the pendulum is swinging back.

Compliance, auditability, and cost control are pushing organizations toward more standardized approaches.

The good news is that the ecosystem is catching up. Workload identity, short-lived credentials, and policy-driven access are no longer niche ideas. They are becoming default building blocks.

Tools that help organizations discover and govern secrets and NHIs, including GitGuardian's NHI Security and Governance platform, are moving from "nice-to-have" to foundational. Not because they are trendy, but because you cannot govern what you cannot find, and you cannot respond to leaks without context.

GitGuardian's NHI Governance Inventory View

What We Should Learn From Agentic AI About NHIs

Agentic AI is a forcing function. It is showing us, in bright light, the identity failures we have tolerated for years. What's new is how quickly a mis-scoped token, inherited session, or leaky log can be exercised.

Any entity that can act on your behalf must be treated with the same seriousness as human access, regardless of whether it runs for a millisecond or maintains a long-lived connection.

Transformation will not happen overnight. Inventory, cleanup, and migration take time. They have to scale across teams and technologies. That is normal. Plan for it.

Alignment matters more than tools. Without shared strategy and governance, agentic AI will simply accelerate the same failures we have been living with, including long-lived keys, unclear ownership, broad standing permissions, and poor offboarding. Those, in turn, lead to more breaches and incidents.

All non-human identities face the same fundamental problems. They all need verifiable identity, least-privilege access, and continuous oversight. Agentic AI is not a special case. It is a stress test.

The teams that succeed will be the ones that treat it that way, early, and build the governance that makes speed survivable.

HMAC Secrets Explained: Authentication You Can Actually Implement

Dwayne McDaniel — Fri, 07 Aug 2026 15:14:31 +0000

HMAC (Hash-based Message Authentication Code) secrets are the industry standard for webhook signatures, internal API authentication, and session tokens. They provide a fast, simple way to verify that a message hasn't been altered and came from a trusted source.

While services like Stripe, GitHub, and Slack make HMAC easy to consume, implementing it securely requires attention to detail. This guide covers how HMAC works, how to implement it correctly, and how to avoid common security pitfalls like timing attacks and hardcoded secrets.

What Is an HMAC Secret?

An HMAC secret is a shared cryptographic key used to generate and verify message authentication codes. These act as a digital signature that proves a message hasn't been modified and originated from a trusted source.

Unlike public-key cryptography (which uses two different keys), HMAC uses a single symmetric key known only to the sender and receiver. This secret is a high-entropy value (usually a 256-bit random string) that requires special caution. Combined with the message (the raw data being authenticated, e.g., a webhook payload or API body), it gets hashed through a cryptographic algorithm like SHA-256 to produce the signature.

Common Use Cases:

Webhook Verification: Verifying that events from Stripe, GitHub, or Slack are legitimate.
API Authentication: Securing internal service-to-service communication.
Session Tokens: Signing cookies or JWTs (HS256) to prevent client-side tampering.

How HMAC Works

Here is an important fact: HMAC is more than just hash(key + message). That simple approach is vulnerable to length extension attacks. Instead, HMAC (defined in RFC 2104) uses a two-pass "hash-of-hashes" construction:

Inner Hash: The key is XORed with a constant (ipad) and hashed with the message.
Outer Hash: The key is XORed with a different constant (opad) and hashed with the result of the inner hash.

This structure ensures cryptographic strength even if the underlying hash function has minor weaknesses. As a developer, you don't need to implement this: standard libraries handle it. Your focus should be on key management and secure verification (more on that later).

Implementing HMAC Authentication

Generating Signatures

Always use your language's standard crypto library. Never roll your own crypto.

Python:

import hmac
import hashlib
def generate_signature(secret: str, payload: str) -> str:
    return hmac.new(
        secret.encode('utf-8'),
        payload.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()

Node.js:

const crypto = require('crypto');
function generateSignature(secret, payload) {
  return crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
}

Verifying Signatures (Crucial!)

Verification is where most vulnerabilities occur. You must use constant-time comparison to prevent timing attacks: Standard string comparisons (==) stop as soon as they find a mismatch. An attacker can measure how long the server takes to respond and guess the signature one byte at a time.

To prevent these attacks, it is crucial to use comparison functions that always take the same amount of time, regardless of where the mismatch occurs.

Python (Flask Example):

import hmac
import hashlib
from flask import Flask, request, abort
app = Flask(__name__)
# In production, load this from a secure secrets manager
WEBHOOK_SECRET = "your-secure-random-hex-string"
@app.route('/webhook', methods=['POST'])
def handle_webhook():
    signature = request.headers.get('X-Signature')
    # Verify the RAW body bytes, not parsed JSON
    payload = request.get_data()

    expected = hmac.new(
        WEBHOOK_SECRET.encode(), 
        payload, 
        hashlib.sha256
    ).hexdigest()

    # SECURE: Constant-time comparison
    if not hmac.compare_digest(signature, expected):
        abort(403)

    return "Verified", 200

Node.js (Express Example):

const express = require('express');
const crypto = require('crypto');
const app = express();
// Middleware to save raw body for verification
app.use(express.json({
  verify: (req, res, buf) => { req.rawBody = buf; }
}));
app.post('/webhook', (req, res) => {
  const signature = req.headers['x-signature'];
  const expected = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET)
    .update(req.rawBody)
    .digest('hex');
  // SECURE: Constant-time comparison
  const sigBuffer = Buffer.from(signature, 'hex');
  const expBuffer = Buffer.from(expected, 'hex');

  if (sigBuffer.length !== expBuffer.length || 
      !crypto.timingSafeEqual(sigBuffer, expBuffer)) {
    return res.status(403).send('Invalid signature');
  }
  res.send('Verified');
});

Security Best Practices

1. Key Management & Storage

Your HMAC security is only as good as your secret key.

Generate Strong Keys: Use a cryptographically secure random number generator. Keys should be at least 32 bytes (256 bits).
- openssl rand -hex 32
Never Commit Secrets: Hardcoding secrets in source code is the #1 cause of data breaches. Use environment variables or dedicated secrets managers (AWS Secrets Manager, HashiCorp Vault).
- Read more on The Extent of Hardcoded Secrets.
Scan for Leaks: Use tools like GitGuardian to automatically detect if an HMAC secret is accidentally committed to your repository.

2. Prevent Replay Attacks

A valid signature is valid forever unless you add a timestamp. An attacker could capture a legitimate request and "replay" it later.

Include a Timestamp: Send a timestamp header (e.g., X-Timestamp).
Verify Age: In your verification logic, reject requests older than 5 minutes.
Sign the Timestamp: Include the timestamp in the HMAC signature payload so it can't be modified.

Note: Timestamps reduce the replay window but don't eliminate it entirely. For high-security scenarios (financial transactions, sensitive mutations), consider nonce-based replay prevention where the server issues a one-time token that's included in the signature and invalidated after use.

3. Prevent Destination Replay (Context Binding)

For internal APIs, signing only the body isn't enough. An attacker could intercept a valid request to /api/user/preferences and replay it to /api/admin/create if the bodies are compatible.

Bind Context: Sign a "canonical string" that includes the host, HTTP method, and path, not just the body.
- Example: HMAC(secret, host + "POST" + "/api/v1/resource" + timestamp + body)

4. Choose the Right Algorithm

Recommended: HMAC-SHA256. It's widely supported, fast, and secure.
Acceptable: HMAC-SHA1 is theoretically still secure for HMAC constructions (unlike direct hashing), but it's best to avoid it to prevent compliance flags.
Avoid: HMAC-MD5.

5. API Key Security

If you are using HMAC for API authentication, treat the shared secrets with the same care as API keys.

Rotate: Change secrets periodically. Use a dual-phase approach (supporting both old and new keys simultaneously) to rotate without downtime.
Scope: If using multiple HMAC keys, bind each to specific clients or API scopes to limit blast radius if one is compromised.

Learn more about API Key Security Best Practices.

HMAC vs. JWT vs. OAuth

Developers often ask which approach fits their use case.

In reality these aren't strictly comparable: HMAC is a cryptographic primitive, JWT is a token format, and OAuth is an authorization framework.

The table below might be useful for you depending on your practical needs:

Feature	HMAC	JWT (HS256)	OAuth 2.0
Key Type	Symmetric (Shared Secret)	Symmetric or Asymmetric	Asymmetric (Tokens)
Complexity	Low	Medium	High
Best For	Webhooks, Internal APIs	Stateless Sessions	User Login, 3rd Party Access
Non-repudiation	No	No (HS256) / Yes (RS256)	Yes

Use HMAC when you control both ends of the connection or have a direct 1:1 relationship (like a webhook provider). Use OAuth for user-facing applications and JWTs for distributed session management.

Troubleshooting Checklist

If your signatures aren't matching:

Check Encoding: Are you hashing the hex string or the raw bytes of the key? (Usually raw bytes).
Raw Body: Are you hashing the exact raw body received? Even a single missing space or newline will break the signature. Do not re-serialize parsed JSON.
Algorithm: Confirm both sides are using the same hash function (e.g., SHA-256).
Headers: Ensure you are extracting the signature from the correct header key (case-sensitivity matters).

Conclusion

HMAC provides a robust, efficient way to authenticate messages between trusted parties. By generating strong keys, using constant-time verification, and protecting your secrets from being hardcoded, you can secure your webhooks and APIs against tampering and impersonation.

Ready to secure your secrets? Start by scanning your repositories for hardcoded keys with GitGuardian to ensure your HMAC secrets remain private.

Frequently Asked Questions (FAQ)

How do I find my HMAC secret key?

If you are verifying webhooks from a service like Stripe or GitHub, the HMAC secret is generated by the provider. You can typically find it in their developer dashboard under "Webhooks" or "API Security." If you are implementing your own HMAC system, you must generate a cryptographically strong random key yourself using a secure random number generator (e.g., openssl rand -hex 32).

What is HMAC (for dummies)?

Think of an HMAC as a digital wax seal on an envelope.

The message is the letter inside.
The secret key is the unique signet ring used to press the wax.
The signature is the resulting wax seal.

Only someone with the exact same signet ring (the secret key) can create a matching seal. If the letter is opened or changed, the seal breaks, and the receiver knows it was tampered with.

Is HMAC-SHA1 still secure?

Yes, HMAC-SHA1 is generally considered secure for message authentication, unlike SHA-1 for digital signatures or file hashing. The "double-hashing" structure of HMAC protects it from the collision attacks that broke SHA-1. However, for all new applications, industry best practices recommend using HMAC-SHA256 to ensure long-term security and compliance.

Boards Focus On Risk, Resilience, and Operational Realities: Where NHI Governance Fits In

Dwayne McDaniel — Thu, 06 Aug 2026 16:04:51 +0000

Boards Focus On Risk, Resilience, and Operational Realities: Where NHI Governance Fits In

Learn how GitGuardian helps boards and CISOs align on cyber risk, operational resilience, and the rising impact of unmanaged workload identities at scale.

By Dwayne McDaniel • 22 Jan 2026 • 9 min read

Boards of Directors (BoDs) do three things exceptionally well when cyber is framed correctly. They set risk appetite, they allocate capital, and they demand evidence that the business can withstand disruption without losing momentum.

Boards do not want a deep dive on token formats, vault policies, or why one Kubernetes access pattern is better than another. They care about outcomes that map to enterprise value, including material exposure, downtime, and regulatory risk. They care about customer impact, and they are deeply concerned about whether the company can keep operating under stress.

Many cyber conversations fail at the very first step. Security leaders too often walk into the boardroom with a pile of findings and expect trust. The board hears complexity, uncertainty, and loudest of all, costs.

Most BoDs want a small, stable set of business indicators that show risk is going down and resilience is going up over time. If leaders adopt that lens, the scope naturally broadens beyond "cybersecurity." It becomes operational resilience, and it includes efficiency.

Why Cyber Keeps Becoming A Board Topic

A board generally spends the most time addressing cyber when they have to, not by choice. If there is a material incident, cyber temporarily becomes the board's number one issue because disclosure, customer impact, and financial exposure collapse into one event.

For example, U.S. public companies are required by the SEC's cybersecurity disclosure rules to disclose material cybersecurity incidents under Item 1.05 of Form 8-K, generally within four business days of determining materiality. They must also describe governance and oversight, including the board's role, in annual disclosures.

Oversight expectations are also rising. Deloitte's Audit Committee Practices reporting shows that cyber sits squarely in audit committee priorities, with 50% of respondents identifying cybersecurity as the number one area of focus for their audit committee over the next 12 months. That same report found that 62% said audit committees have primary oversight of cybersecurity risk.

Balancing Risk With Keeping Up With Tech

Boards are also pushing innovation. BDO's 2025 Board Survey summary highlights that many directors see emerging technology as both an opportunity and a governance burden, with a meaningful share saying tech advancements will require significant board attention. The same report also notes that 63% of directors plan to increase strategic investment in cybersecurity in the year ahead.

Boards are balancing two pressures that often collide. They need to move faster on technology, while at the same time reducing exposure resulting from any change in technology. The only sustainable way to do that is to treat cyber as operational resilience rather than a separate technical function.

Operational Resilience Is The Bridge Between Board Priorities And Security Reality

Operational resilience is the ability to keep delivering strategy through disruption. That includes preventing incidents, but it also includes reducing fragility, shrinking blast radius, and maintaining delivery speed when conditions are imperfect.

This framing aligns with what boards already recognize:

The organization can accept some risk, but only if you have controls, monitoring, and response capacity that keep losses within acceptable limits.
The organization can invest in transformation, but only if transformation does not turn into operational chaos and cause spikes in costs.
The organization can innovate, but only if the basics are disciplined enough to survive the consequences of change.

But this goal of investing wisely and safely in tech does not match the planning for many organizations. According to PwC's Global Digital Trust Insights, only 24% of organizations report spending significantly more on proactive measures than reactive measures, while 67% report spending is roughly even across both categories. PwC explicitly frames proactive investment as the healthier posture, and warns that reactive costs are often underestimated because they are dispersed across the business.

Boards should understand this intuitively. They already know it is cheaper to maintain a factory than to rebuild it after a disaster. The same logic should apply to identity, access, and the infrastructure that keeps digital operations running. The disconnect is partly a result of how fast change is happening with regard to AI and non-human identity governance needs.

The Identity Layer Is Now The Operations Layer, Especially For Non-Human Identities

When boards talk about identity, they often default to humans and initiatives around MFA adoption, onboarding, privileged access reviews, and insider risk. Those are important, but they are no longer the whole identity story.

Modern businesses run on systems made up of non-human identities. Service accounts, API keys, CI tokens, OAuth apps, workload identities, and agent credentials now power all the integrations, automation, cloud workloads, and data pipelines that keep our customers using our digital products. As the number of machines and workloads needing access grows, the governance surface area continually expands.

CyberArk has reported that machine identities outnumber human identities by more than 80-to-1. Other estimates across the industry think we have crossed the 100-to-1 threshold as automation accelerates.

In most organizations, the majority of secrets exist because, traditionally, that was how legacy systems were safely connected to other systems. We needed an access mechanism for these identities, so we again reached for passwords, in the form of API keys and tokens, to get the job done. But those long-lived access keys, most of which grant more permissions than strictly necessary, have a tendency to sprawl by the millions.

In other words, "secrets sprawl," the leaking of credentials into plaintext across systems, is really a symptom of non-human identity sprawl.

The conversation needs to change from "how do we stop developers from making mistakes" to "how do we govern machine access at enterprise scale without slowing the business down." Helping the board have this exact conversation is the difference between fighting for security budgets and finding paths forward for real organizational change.

Making The Business Case With GitGuardian's Insights

GitGuardian's "Hidden Cost of Secrets Sprawl" report is useful precisely because it quantifies what boards tend to suspect: credential chaos is a productivity tax.

This report makes it clear that manual secrets management costs organizations $172,000+ annually per 10 developers. The math is grounded in three hours per week per developer, and a fully loaded cost model that puts senior developer time at around $120 per hour.

The more important point for a board is that the cost shows up in several predictable places across the lifecycle of delivery and response.

Costs show up in engineering throughput. When developers spend hours requesting, finding, rotating, or debugging credentials, they are not building features. GitGuardian describes how this friction compounds as organizations grow, turning a small tax into a competitive disadvantage.

Costs show up in security and operations bandwidth. Alert fatigue and manual investigation pull teams away from strategic work, and estimates show that automation can recover at least 1.2 FTE worth of capacity.

Costs show up in onboarding and time to productivity. Access and credential setup can stretch onboarding timelines, keeping new hires from becoming productive while they wait for access and learn informal processes.

Costs show up during incidents. When a key leaks, teams often lose time simply understanding what the key can touch, where it is used, and what needs to be rotated. GitGuardian includes customer examples that emphasize how long this mapping can take in real environments.

Costs show up in audit readiness. Audit prep is frequently a scavenger hunt for evidence of control and ownership, made worse when credentials and machine access are spread across teams and tools. This is an avoidable operational burden, not just a compliance annoyance.

The "hidden cost" is operational inefficiency that increases risk by consuming the exact bandwidth you need to improve resilience.

The Board Question To Anchor The Conversation

If you want to boil it down to one board-level question that forces clarity without dragging the room into technical weeds, it is this:

How are we governing non-human identities and their access, and what is our confidence in the inventory?

That single question ties to everything boards already care about:

Risk appetite - unknown access creates unknowable exposure.
Continuity - fragile access breaks operations during change.
Accountability - "no owner" means "no control."
Cost - manual access work is a measurable operational drag.
Crisis Response - containment speed depends on visibility and ownership.

The job of a security leader is not to get them to ask this specific question; it is to answer it proactively before they know to ask.

Framing security, DevOps, and IAM work as a unified front that can speed innovation while limiting risks sets you up for wider success than focusing on a single tool choice or team-siloed initiative ever could.

The Challenge Of Up-Front Investment To Move Away From Long-Lived Secrets

Boards are often comfortable with funding end states. But engineering teams live in transitions. Bridging that gap, without bogging down in the weeds, is the real challenge.

Moving from long-lived secrets to identity-based authentication for NHIs is a real modernization effort. It can require refactoring authentication patterns, adjusting CI pipelines, replacing brittle integrations, and introducing new controls like workload identity or signed assertions. It also requires building governance muscles that many organizations never had to build when credentials were informal.

That is why so many of these types of programs stall. The product always has to ship. Reliability work competes with feature work, and security initiatives compete with operational debt.

This is where board oversight becomes valuable. The board can do what engineering teams often cannot do alone, like protecting time for foundational work, insisting on measurable progress, and preventing risk from silently compounding while the company scales.

Prepping For Operational Realities

In practical terms, the board should expect a phased approach to stabilize what exists by reducing unmanaged long-lived credentials and improving discovery, ownership, and rotation discipline.

Part of the data BoDs should demand is exactly which systems are mission-critical, meaning they would cause loss if they were affected by an incident or outage. Shifting workloads and integrations toward short-lived, identity-based access takes time, so priority should be given to "critical" infrastructure and systems first.

Boards must expect the orgs' executives and leaders to institutionalize governance across NHIs. The goal is to ensure machine access does not drift back into chaos as teams change and new systems arrive. This is a goal most organizations are just beginning to grapple with, and where partnering with the right technologies can make a significant difference.

The board does not need to choose individual protocols or tools. BoDs need to fund the journey and demand evidence that the journey is reducing both risk and operational drag.

Move Toward Full NHI Governance With GitGuardian

GitGuardian can help you move towards true NHI Governance because our platform starts where the pain is most visible and measurable, then expands into where the long-term control must exist.

When most people think of GitGuardian, they immediately think of the State of Secrets Sprawl, our annual report on finding millions of publicly leaked credentials. That is indeed where we started our journey as an organization, focused on secrets. Along the way, we realized that secrets don't exist in a vacuum, and what we have really been tracking all along is access mechanisms for identities, in particular, non-human identities. This shift might seem subtle at first, but the sea change is evident in the platform's recent release notes, where we talk about expanding NHI Governance with integrations that discover and enumerate NHIs tied to platforms like Airbyte, Anthropic, N8n, OpenAI, CyberArk Secrets Manager Self Hosted, and Slack. The emphasis is on identity context, permissions, accessed resources, and an identity-first inventory view, not on new types of detectors.

And right before we ended 2025, GitGuardian expanded NHI Governance coverage into additional critical platforms, including Datadog, Snowflake, Okta, and Auth0. We are now helping teams close blind spots and enabling unified identity risk assessment.

GitGuardian's Analytics views give you the right information in real time

This aligns well with BoDs' mindsets. It begins with exposure and operational friction that is already costing money, then builds toward a durable identity governance layer that scales with the business.

What Boards Should Demand, And What Management Should Deliver

Boards do not need to become security architects. They need to govern the enterprise in the reality in which it operates. Today, that reality means identity is the control plane for resilience, and non-human identities are the fastest-growing part of it.

Non-human identities are now a core part of that resilience equation because they represent scaled access to systems, data, and automation. If they are unmanaged, your exposure is unknowable, and your operations are fragile. If they are well-governed, you reduce the blast radius and reclaim operational capacity at the same time. This is why GitGuardian matters when communicating risks and your strategy to your board.

Partnering with GitGuardian can help you report periodically on NHI governance with trend lines, not just point-in-time snapshots. This includes progress on your modernization path away from long-lived credentials. Our platform will help you treat incident readiness as a control, including containment speed and the ability to rotate or revoke access without downtime.

We would be happy to set up a demo and help you align your next board of directors conversations with your operational realities and needs.

Responding to Exposed Secrets - An SRE's Incident Response Playbook

Dwayne McDaniel — Tue, 04 Aug 2026 15:10:20 +0000

Responding to Exposed Secrets - An SRE's Incident Response Playbook

Today, let's take a closer look at incident response playbooks: how to build one, tailor it for secret leaks, take actions, and learn from incidents.

By Tiexin Guo • 27 Jan 2026 • 10 min read

Imagine this: It's a regular Thursday morning when disaster strikes. The on-call engineer receives a flood of alerts -- unusually high API failures of an internal service. Reverting to a previous version changes nothing. Now, colleagues are panicking in crisis mode. Just when things couldn't get worse, it does: another deployment triggered by a code merge takes everything down. The entire platform is now completely offline.

Above is a typical "unmanaged" incident, where a series of blunders and a lack of coordination lead to a meltdown. Humans tend to make mistakes, especially under pressure; that's why we want to use standardized, structured processes to manage incidents efficiently, minimizing disruptions and restoring operations quickly. And that is precisely what an incident response playbook is for.

1. Preparation

Although you might think so, an incident response playbook doesn't really start with "if alert X happens, do Y".

Before handling incidents, there are other important things to figure out:

Goals and objectives: What is the objective of the playbook? What is the scope? What types of incidents does it try to cover?
Roles and teams: What are the roles involved, and what are their responsibilities? Do you need an incident commander or an ops lead? Which team/team members are required? Do they have the necessary skills? If not, are there trainings? Are they available? Is assembling a dedicated incident response team needed?
Communication, documentation, and coordination: What channels to use for internal and external communication? How to notify stakeholders? How to transfer command clearly, especially across time zones? What format/template to use for reports, updates, and logs? How to live-update the state of an incident (e.g., a shared Google Doc) so that everyone can work synchronously? What info needs to be documented during an incident?

The above isn't an exhaustive list. There is other stuff to think about beforehand, like how to categorize incidents, how incident levels are defined, whether existing company/team policies and requirements need to be integrated with, etc.

Even if the whole document is finished, it doesn't mean the end of the preparation - remember to test the playbook! Train related team members on the playbook, do simulations to test its effectiveness, and adjust procedures based on feedback.

One can never be too prepared!

2. Secret Leak Incident

As briefly mentioned, there are different types of incidents, and one incident response playbook couldn't possibly cover all. It's a good practice to tailor our playbooks to different types of incidents.

Here, I want to single out one specific type of incident - secret leaks, because it is different in many ways.

2.1 Alerting and Detection

From the alerting and detection standpoint, it's more challenging to spot secret leaks.

The "service/server/network down" type of incidents typically show up clearly, and immediately, on our dashboard because we get alerts on unreachable resources, high latencies, and elevated error rates. However, traditional metrics and monitoring systems are less effective in detecting secret leaks: CPU usage, latency, and error rates might only increase slightly, and it isn't always immediately obvious since it takes time for malicious actors to find it out and exploit.

We can create specific metrics and alerts for secret leaks. For example:

API usage: Is there unexpected traffic like sudden, unexplained increases? Are there requests from unfamiliar IP addresses or locations? Does error rate increase, but not to 100% (since a surge in errors like 401 Unauthorized/403 Forbidden might indicate that someone is trying to use the leaked secret improperly or is attempting to brute-force access, instead of a service going down completely)? Is there a high volume of requests from a single IP?
Cloud monitoring: Is there unusual resource consumption (a leaked key could be used to spin up cloud resources)? Are there unauthorized IAM activities? Infrastructure as Code for IAM could be our friend to track changes to IAM roles and permissions. Is there network traffic to unfamiliar destinations or unusual data transfer patterns?
Database Monitoring: Are there unusual queries, such as attempts to access sensitive data or perform unauthorized modifications? Is there a high number of failed login attempts for database accounts, which could indicate that someone is trying to gain access using compromised credentials?

Besides monitoring, we can aggregate logs from all systems, so that machine-learning-based (or simply rule-based) anomaly detection can be used to detect patterns that deviate from the norm.

Last but not least, perform regular scans on code repos, logs, and configuration files for exposed secrets, and they can be integrated into our CI/CD pipelines to prevent secrets from being committed.

2.2 Impact, Scope, and Investigation

Typical "server down" types of incidents have a more localized and immediate impact, and while the results can be severe, the scope is usually contained. And, the investigation process focuses on identifying the root cause, such as:

analyzing logs to identify errors;
examining performance metrics to identify bottlenecks or resource constraints;
reviewing recent configuration changes that may have caused the issue;
using standard troubleshooting techniques to pinpoint the problem and find a solution.

On the other hand, the impact of secret leaks can be long-lasting, since a compromised secret can grant unauthorized access to sensitive data and services, and the scope can extend beyond the immediate system where the leak occurred, potentially affecting multiple apps and services, or even an entire environment. So, the investigation requires a more comprehensive approach, and understanding the blast radius is crucial for secret leaks. When working on a secret leak incident, the first thing to do usually isn't to rotate the leaked secret, but to identify the scope:

Identify the leaked secret: How to determine what systems and data are accessible with the exposed secret?
Determine the scope of the compromise: How critical is it? For example, can the secret be used to access personally identifiable information (PII) or financial data? Is the affected environment non-prod or production?
What are the tools and techniques to identify the scope? Are there tools and scripts to parse access logs? Is network traffic analysis possible?
Revoke and rotate: How to invalidate the leaked secret and rotate all related secrets while maintaining reliability?
Collaboration: Is collaboration with another team, like a dedicated security team or even a legal team, necessary to assess the potential impact?

How to Become Great at API Key Rotation: Best Practices and Tips

2.3 Prevention

To prevent "server down" type of incidents, traditional methods are monitoring, redundancy, capacity planning, autoscaling, and change management.

For secret leaks, however, prevention requires a multi-layered approach:

Secret management: Use a dedicated secret management system (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) to store and manage secrets securely.
Least Privilege Principle: Grant only the necessary permissions to access secrets.
Code scanning: Integrate secret scanning into the CI/CD pipeline to prevent secrets from being committed to code repositories.
Continuous learning: Train developers and operations staff on secure coding practices and the importance of secret management.

Secrets Management Simplified with Multi-Vault Integrations

3. Real-Time Incident Response: The SRE Action Plan

After detection and analysis, it's time to contain, eradicate, and recover. This part of the playbook needs to be as specific as possible with easy-to-follow instructions to avoid any ambiguity and human errors. Detailed response steps should be outlined, and they differ for different types of incidents.

First, we want to have detailed steps for isolating affected systems to prevent further access. Prioritize efforts based on the importance of the affected systems and the potential impact, focus on high-value assets, sensitive data, and publicly accessible systems. If possible, isolate affected systems from the network to prevent further traffic. This can be achieved through firewall rules or network segmentation. If the exposed secret is associated with a user account, temporarily disable the account to prevent further use. If the exposed secret is an API key or access token, revoke it, but as said in earlier chapters, figure out the blast radius should precede the operation. Cloud provider firewalls and IAM systems can be used to help revoke access.

Then we would like to revoke the compromised secret and generate a new one. Pinpoint where the compromised secret is stored (e.g., environment variables, secrets management system); delete the secret from the secrets management system, invalidate the API key, or change the password; create a new, strong secret using a cryptographically secure random number generator; store the new secret securely; finally, update the config. Here, we want to use automation to speed up incident response time and reduce human error. Use tools like Cyberark Conjur, HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault to securely store and manage secrets, tools like Ansible to automate the process of updating configuration files, and tools like continuous deployment/GitOps for secret injection to update environment variables.

Before declaring the result and notifying impacted teams and stakeholders, make sure to test that the new secret is working correctly and that the old secret is no longer valid.

It's worth mentioning that rotating secrets in a production environment can be challenging, since it can potentially disrupt services and cause downtime. We can take advantage of different deployment strategies to minimize downtime:

Blue/Green Deployments: Deploy changes to a separate environment before switching over. This allows us to test the changes without impacting production.
Feature Flags: Enable or disable features without deploying new code. This allows us to control the rollout of changes and quickly revert.
Canary Releases: Roll out changes to a small subset of users before a full deployment. This allows us to identify any issues before they affect a large number of users.

It's recommended to monitor key metrics during and after the rotation, which will help identify any potential issues that may arise during the process. It's also necessary to have a rollback plan, which should include steps to revert to the previous state.

4. Post-Incident Analysis and Proactive Measures

This section focuses on learning from incidents, implementing proactive measures to prevent future incidents, and making sure the playbook is up-to-date.

4.1 Post-Incident Analysis: Learning from the Leak

A thorough post-incident review is critical for understanding what happened, why it happened, and how to prevent similar incidents from happening in the future. Remember, the review isn't about assigning blame, but identifying weaknesses in the system.

Before the review, gather information, identify all factors, and determine the root cause. For example, it could be:

lack of security awareness among developers
misconfigured systems or apps
insecure secret management practices
vulnerabilities in third-party libs or components
access controls

A comprehensive record of the incident should be created so that next time, when a similar incident happens, we have something to refer to. Key elements to include: timeline, impact assessment, actions taken, root cause analysis, and lessons learned.

4.2 Proactive Measures: Preventing Future Leaks

To implement measures that prevent similar incidents from occurring in the future, many measures can be taken. Although there are incident-specific measures for each incident, there are a few generic action items, such as:

Shift-left: Integrate security practices earlier in the development lifecycle automatically to identify vulnerabilities before they even reach production. This reduces the cost and effort of fixing issues, improves the overall security, and fosters a culture of security awareness among developers.
Prevent secrets from being committed: Make use of tools like pre-commit hooks, .gitignore files, environment variables (12-factor app) to avoid hard-coded secrets, and use static and dynamic analysis tools for secret scanning.
Security awareness training and knowledge sharing: Educate developers on secure coding practices and security best practices.
Use secret managers.

How To Use ggshield To Avoid Hardcoded Secrets

4.3 Playbook Review, Update, and Distribution

To ensure the playbook remains up-to-date, effective, and accessible, regular review, update, and test are required.

Continuously monitor the threat landscape for new types of threats and vulnerabilities, conduct periodic reviews to ensure it remains relevant, integrate feedback from actual incidents to improve the playbook's effectiveness, and update the playbook to reflect changes in the tech stack and infrastructure.

Also, playbooks need to be accessible to all relevant team members. Make sure they are version-controlled to track the changes to the playbook. Store them in a centralized repository that is easily accessible, shareable, and searchable. Also, provide training on the playbook to ensure that personnel understand its contents and how to use it.

5. Summary: Securing the Secrets, Ensuring Reliability

In this post, we've walked through the essential components of an SRE incident response playbook tailored for the threat of exposed secrets. From detailed preparation and proactive detection, to rapid response and continuous learning, a well-defined playbook is our best friend defending against potential breaches.

A quick recap of the key steps:

Preparation: Define goals, roles, communication channels, and documentation standards before an incident occurs.
Detection: Implement specific metrics and alerts to identify potential secret leaks, including API usage anomalies, cloud monitoring, and log analysis. Don't forget secret scanning!
Impact assessment: Prioritize identifying the scope of the leak before taking action. Determine what systems and data are at risk.
Containment, eradication, and recovery: Isolate affected systems, revoke compromised secrets, generate new ones, and update configurations. Leveraging automation and modern deployment strategies.
Post-mortem: Conduct a thorough review to understand the root cause and identify areas for improvement.
Proactive measures: Implement preventative measures such as secret management systems, the principle of least privilege, code scanning, and security awareness training.
Continuous improvement: Regularly review, update, and test the playbook to ensure it remains effective and relevant.

An incident response playbook is not a "static" document. It's a living guide that evolves with our infrastructure, security landscape, and lessons learned from incidents. Effective communication, clear roles, and a culture of continuous improvement are key to effective incident management.

As SREs, we are the guardians of both service reliability, and security. By implementing these practices, we can minimize the impact of exposed secrets, maintain the integrity of our systems, and ensure the trust of our users.

Now, it's your turn. Take these insights and implement them within your own organizations. Develop, test, and improve your incident response playbooks. Your efforts today will pay dividends in the long run!

Your Secrets Need a VDP, Not Just a Bug Bounty

Dwayne McDaniel — Mon, 03 Aug 2026 18:57:04 +0000

Your Secrets Need a VDP, Not Just a Bug Bounty

Bug bounty programs are valuable -- until they replace disclosure policies. Learn how unreasonable PoC demands or scope exclusions create security blind spots when it comes to leaked secrets.

By Gaetan Ferry • 6 Feb 2026 • 8 min read

In recent years, more and more companies have launched bug bounty programs as proof of their commitment to security and as a way to implement continuous monitoring of their corporate attack surface. Those programs sometimes offer generous payouts to vulnerability reporters, and often partner with dedicated platforms that offer various services such as:

Payment and billing management
Triaging as a Service
Investigation assistance

Platforms rely on a "hacker community", a group of people who hack on the programs to discover vulnerabilities and earn bounty money. Most of those "hackers" are self-employed in a way that allows them to comply with local applicable tax laws.

Bug bounty programs are a great way to have a corporate perimeter or set of applications audited by a large set of people, nearly continuously. They can be a great addition to a company's security policy. In fact, GitGuardian has been running a bug bounty program for multiple years, as a complement to our periodic audits and overall security strategy.

Bug Bounty Done Wrong

The problem with bug bounty programs starts when they try to substitute for a proper Vulnerability Disclosure Policy. When they do, they no longer improve your security posture; they undermine it.

Bug bounties, by design, are selective. From a "hacker" perspective, they come with limited scopes, opaque triage processes, gatekeeping platforms, or even eligibility requirements. As a result, valid, good-faith vulnerability reports can get ignored, rejected, or buried -- not because they lack accuracy or merit, but because they fall outside of the boundaries of the programs' terms or the opaque decision of a third-party triager. Payout levels also undermine this testing model, turning continuous monitoring into a blind spot shaped by market incentives; why search for or report vulnerabilities when they pay little or nothing?

Using a bug bounty platform as the only possible communication channel for vulnerability disclosure creates unnecessary friction:

Mandatory registration forces researchers to trade their privacy for participation.
Non-disclosure clauses can silence conversation about systemic risks, and more generally hinder information sharing.
Platform gatekeeping can discourage reporters.
Worse: out-of-scope dismissals allow serious vulnerability reports to be voided, and never reported to security teams

These blind spots don't make an organization more secure. They make it easier to overestimate the security posture, thinking that fewer reports mean fewer problems.

A good Vulnerability Disclosure Policy (VDP) should promote openness. It should be a clear and accessible way for anyone -- a professional researcher, a student, or a concerned user -- to report a security issue safely, privately, and without process complexity. A good disclosure policy should enable communication rather than controlling it.

One particular issue that highlights how bug bounty can fail as a disclosure channel lies in how they handle secret leak reports.

GitGuardian's experience

One of the core foundations of GitGuardian is the detection and remediation of secrets leaked in public spaces. Over the course of the past year, while working on improving our understanding of the secret sprawl issue, we performed responsible disclosures to hundreds of companies.

GitGuardian's cybersecurity research team is not a bug bounty crew. We do not seek any reward for reporting incidents. For this reason, we usually attempt to contact affected companies directly, preferably via email, and sometimes through online forms dedicated to security incident reporting. We only fallback to the bug bounty program channel as a last resort, or when directly prompted to do so.

While working with platforms, we experienced a variety of situations and answers that illustrate how bug bounty can fail as a disclosure channel.

400 PoC or GTFO

As a result of a large-scale research project, we recently reported leaked private keys related to valid X.509 certificates. The risk of such incidents can generally be considered high, as a leaked key can be used to set up Man-In-The-Middle attacks against the company's public assets. Some of our reports had to go through bug bounty platforms, which already create friction. As much as we can automate the sending of hundreds of e-mails, filling bug bounty reports at scale is challenging.

In all our reports, the triagers asked for a proof of concept exploitation.

HackerOne response asking for a proof of concept after private key leak

BugCrowd response asking for a proof of concept after valid credential leak

First, proving a credential's impact has a clear ethical boundary: demonstrate the potential for harm without causing actual harm. This means verifying credentials are valid, confirming what resources they access, and documenting their privilege level, but without reading production data, modifying systems, or performing harmful actions. This is not always possible, depending on the credential type. In the case of leaked X.509 certificate private keys, creating such a proof-of-concept would have required decrypting real traffic or impersonating production services -- crossing from validation into active attack -- which could have severe legal consequences.

Then, the main question is: what happens after the report gets closed as informative? There is a chance that no action will be taken. In some cases, the issue might never pass the triaging filter and reach the corporate security team.

In our case, most reports were actually closed as informative, and none of the related certificates were revoked. Worst of all, some GitHub repositories containing leaked private keys have never been deleted. We later contacted the related certificates' issuer authorities to have the keys black listed and certificates revoked.

403 Private Program

Bug bounty programs can either be public or private. Public programs can be viewed, accessed, and interacted with by anyone. On the other hand, private programs are invite-only, so only selected members of the platform's community can report vulnerabilities.

In that case, obviously, the program can not be considered a proper disclosure channel. However, there is a reporting flow that overlooks this issue:

You discover a vulnerability and attempt to report it through standard channels (security@, contact forms).
You receive a response: 'Please submit via our Bug Bounty Program.'
You navigate to the platform, only to find it's private and invitation-required.
Without an invitation, you hit a dead end with no alternative channel.

Our team has faced this situation once, making the reporting process painful and highlighting how companies often lack awareness about vulnerability disclosure practices.

Similarly, a documented program can have expired or been decommissioned. In this case, the communication channel is effectively nonexistent. This was the case when we reported a leaked API key to xAI in 2025.

404 Secret Not Found In Scope

The scope of a program includes the list of assets that are authorized to be worked on. It also includes the list of vulnerabilities that are accepted in reports. The purpose of this restriction is to limit the number of low-quality reports or reports for vulnerabilities that are widely recognized as lacking real-world impact.

However, if the scope of a program is too restricted, valid and severe issues might get discarded by the triaging team without further notice. In that spirit, we faced bug bounty programs that explicitly marked leaked credentials as out of scope. The platforms sometimes even encourage their customers to ban secrets. The reason behind this is tied to the origin of the credentials, as we discussed with a platform representative:

We strongly advise our clients to exclude leaked secrets from their bug bounty program scope. The reality is that compromised credentials frequently originate from illicit sources. There's a thriving underground market for stolen credentials, and by offering bounties for leaked secrets, we risk inadvertently incentivizing and legitimizing a secondary marketplace for compromised authentication data.

While this concern is understandable, excluding leaked secrets creates a dangerous blind spot. Valid credentials represent immediate security risks: unauthorized access, data breaches, or compromised systems.

The solution isn't to ban secret reports -- it's to require source transparency. Researchers should disclose where the credentials were found. This approach enables security teams to investigate the leak's origin and take appropriate remediation action, while distinguishing legitimate research from illicit activity.

500 Triager error

Triager gatekeeping can also be an issue in case of a misunderstanding about a security issue. While misunderstandings can occur with corporate security teams, triagers can close the communication channel when they deem an issue uninteresting. While it is often possible to ask to reopen closed reports or ask for mediation, this can prevent legitimate reports from reaching the corporate teams and create unnecessary friction.

In the above case, the triager closed the issue while the affected credentials were still valid. Such behaviors create frustration, discourage reporters and, again, prevent secrets from being reported.

302 Redirect To Bug Bounty

Even when a direct communication channel with corporate security teams exists, it happens that those teams redirect mailed reports to a bug bounty platform. The rationale is understandable: centralizing all vulnerability reports in one place simplifies triaging and tracking.

Doing so not only slows the remediation process down, but also creates a dangerous bottleneck as the submission will likely have to comply with the bug bounty rules and scope definition, with the same pitfall as issues directly reported on platforms.

It also goes against the potential privacy requirements of the reporter, who would have to create an account on the bug bounty platform and sometimes even fill out tax regulation documents.

We received such a response when we contacted xAI for a leaked token last year. In that case, the corporate team also fixed the issue in the background, even before we could submit it to their program, demonstrating a clear lack of transparency.

Dear Gaëtan,
Thank you for your email.
For us to analyze and also for you to receive proper credit, if applicable, would you please submit this to xAI's Bug Bounty Program on HackerOne?
https://hackerone.com/x?type=team
Thanks!
xAI Team

Vulnerability Disclosure Policy done right

Writing a clear Vulnerability Disclosure Policy that provides an open and transparent communication channel is of prime importance to ensure your company receives vulnerability reports properly. As we explained above, such a policy should promote openness, transparency, and reporters' safety. Privacy is also a core concept of any proper VDP and should be emphasized, as is explained in documents from the US Cybersecurity & Infrastructure Security Agency:

How should my agency treat vulnerability reports from anonymous sources?
These reports should be treated the same as all other reports: like a gift. Knowing the source of a report can be a real benefit because it allows for rapport to develop. However, if the person who submits a report isn't known, the claim should simply be evaluated on its merits -- like every other report.

When bug bounty platforms are your only security communication channel, such privacy can not be appropriately granted to vulnerability reporters.

In fact, CISA published a complete template for Vulnerability Disclosure Policy that emphasizes those openness and transparency concepts. The document is meant to be a regulatory requirement for government agencies, but it can be used as a basis to write the VDP of any company.

At GitGuardian, we are not against bug bounty programs, as we think they can be a great addition to a company's security policy. However, it is of prime importance to understand the limits and blind spots created by those platforms.

Most importantly, bug bounty programs must complement -- not replace -- a public Vulnerability Disclosure Policy. Private, invitation-only programs create insurmountable barriers for new researchers and should never be the sole disclosure channel. Companies should maintain accessible public VDPs alongside any BBP, with clear escalation paths that allow critical reports to bypass platform restrictions when necessary. Direct reports to security@ should remain direct -- triaged by internal teams who understand the full context of their infrastructure, not filtered through external platform scopes that may dismiss legitimate threats on technicalities.

Especially, if you manage a bug bounty program, make sure to include leaked credentials in its scope. Credentials-based attacks have become the number one cyber threat in the modern world, so those incidents should not be disregarded. Asking and verifying the source of the leaks will allow better investigation of the leak issue while reducing the risk of buying stolen credentials from the black market.

To conclude, whatever communication channel you choose for your vulnerability reports, make sure to promote it and make it as visible as possible, for example, with an RFC 9116 security.txt file. There is nothing worse than a communication channel no one knows about.

The Streak Continues: Four More Supply Chain Attacks Hit npm and PyPI

Dwayne McDaniel — Fri, 31 Jul 2026 13:01:06 +0000

The Streak Continues: Four More Supply Chain Attacks Hit npm and PyPI

Between early June and July 14, four more supply chain attacks hit npm and PyPI: a Shai-Hulud worm variant, typosquatted payment SDKs, a stolen publishing token, and a hijacked CI pipeline. Different entry points, one target: the credentials in developer environments and build pipelines.

By Cybersecurity research team • 22 Jul 2026 • 5 min read

Between early June and July 14, 2026, four more supply chain attacks hit npm and PyPI.

The Miasma worm we flagged in our last post did not stop at Red Hat's packages. It kept spreading across npm through June 5, reaching the Vapi server SDK and a string of smaller packages before researchers finished counting. Miasma was not the only worm working in that window. In early June, JFrog uncovered IronWorm, a Rust-built infostealer planted in 36 npm packages that hid behind an eBPF kernel rootkit and spread by using stolen npm credentials to publish trojanized versions of its victims' packages. That campaign was stopped before it reached widely used packages. Four more attacks followed. A PyPI sibling, two typosquat and credential-theft campaigns, and a CI token theft that backdoored packages with millions of weekly downloads. Different entry points, one objective: land where the credentials live and leave with them.

Attack 1: Hades Brings the Shai-Hulud Worm to PyPI

Shortly after Miasma spread across npm, a matching variant surfaced on PyPI carrying the string "Hades - The End for the Damned." Socket identified it as the PyPI branch of Miasma: the same credential harvesting, the same self-spreading logic, and the same Shai-Hulud habit of exfiltrating stolen data by publishing it to fresh GitHub repositories.

The delivery mechanism was new. Instead of an install hook, Hades shipped a *-setup.pth file that runs automatically at Python startup, fetches the Bun JavaScript runtime, and executes the payload. An initial wave hit roughly 19 packages. A second wave on June 8 pushed the count to at least 29, targeting bioinformatics, graph machine learning, and Model Context Protocol (MCP) libraries. Socket also caught the worm mutating mid-campaign, splitting the loader and payload across sys.path to slip past scanners. Counting both ecosystems, SecurityWeek put the Miasma and Hades total north of 100 packages and 471 malicious artifacts.

Attack 2: Fake Payment SDKs Harvest CI Secrets on npm and PyPI

On July 7, Socket's scanners flagged a coordinated cluster of roughly 17 malicious packages across npm and PyPI, all dressed up as SDKs for well-known payment providers including PaySafe, Skrill, and Neteller. Names like paysafe-checkout, paysafe-node, and neteller were built to be grabbed by a developer moving fast.

The facade was clever. Call payments.create and the package returns a success response immediately, so nothing looks broken, while a delayed background routine sweeps environment variables matching KEY, SECRET, TOKEN, PASS, AUTH, or API and ships them to a C2 host behind an ngrok tunnel. The filter explicitly targets AWS_SECRET_ACCESS_KEY, GITHUB_TOKEN, and NPM_TOKEN, the exact secrets sitting in a CI runner's environment. The malware even bailed out early when it detected sandbox hostnames or fewer than two CPU cores, and it ran the same playbook in both JavaScript and Python. Detection came fast, within about six minutes of publication, but the design assumed it would.

Attack 3: A Stolen npm Token Poisons Jscrambler

On July 11, an attacker used stolen publishing credentials to push five malicious versions of jscrambler, plus poisoned releases of its webpack, gulp, grunt, and metro plugins, to npm. Per the Socket and StepSecurity analyses, each version carried a native binary compiled for Linux, Windows, and macOS.

The first versions ran the binary from a preinstall hook. Then the attacker changed tactics: later versions dropped the install hook and instead executed on import or when the CLI ran, which meant npm install --ignore-scripts no longer offered any protection. The payload went hunting well beyond the usual cloud keys, reaching for cryptocurrency wallets and, notably, the credential stores of AI coding assistants like Claude Desktop, Cursor, and Windsurf. JFrog identified the implant as an evolved variant of IronWorm, the Rust infostealer from early June, rebuilt to cover macOS and Windows and to automate its own npm propagation. Jscrambler deprecated the bad versions and shipped a clean 8.22.0 the same day; the official advisory counted 1,479 downloads before removal.

Attack 4: AsyncAPI's Own CI Pipeline Turned Against It

On July 14, an attacker opened 37 pull requests against the AsyncAPI generator repository. Nearly all were noise, proposing a fake charity donation page. Buried in the traffic, one PR exploited a pull_request_target workflow that checked out and ran untrusted pull request code, a "pwn request," to steal the highly privileged asyncapi-bot personal access token.

That workflow was a known risk. A contributor had flagged it 58 days earlier and proposed a fix that was still sitting unmerged when the attacker struck. With the stolen token, the attacker pushed malicious commits to two AsyncAPI repositories under a placeholder git identity and let each repository's own release workflow do the publishing through npm's OIDC trusted publisher integration. Four packages went out across five versions: @asyncapi/generator, @asyncapi/generator-helpers, @asyncapi/generator-components, and @asyncapi/specs, with combined weekly downloads above 2.25 million. The malicious versions were live for roughly four hours, between 07:10 and 11:18 UTC, before they were pulled from the registry, and lock files generated in that window can still resolve to the poisoned releases. The payload fires on require() rather than install, pulls a multi-stage remote access trojan from IPFS, persists through a systemd service, and holds open command-and-control channels over HTTP, Nostr, an Ethereum smart contract, and a peer-to-peer mesh. It targets browser passwords, SSH keys, npm and GitHub tokens, cloud credentials, and crypto wallets, and it can fetch Gitleaks and HackBrowserData to help with collection. One caveat from Aikido's analysis: the automated harvesting routines are broken and exit before gathering anything. The shell access is not, so an operator can still take everything by hand.

The payload also ties this attack back to where this post began: it carries Miasma branding. SafeDep reads it as either a private build by the same operators or a copycat that picked up the name after the worm's source code leaked on GitHub.

The most concerning piece of this for security teams, and something Chainguard called out directly, was that those backdoored packages carried valid Sigstore and SLSA provenance. The attacker did not forge anything. They compromised the CI identity, so the build system signed the malware honestly. Provenance proved where the package came from. It said nothing about whether the source was trustworthy when it was built.

The Common Thread

A worm hiding in a Python startup file, a typosquatted payment SDK, a stolen publishing token, a hijacked CI identity. Four different entry points, all aimed at the same prize: the credentials sitting in developer environments and build pipelines.

The platforms are beginning to close some of the paths attackers have relied on. On July 8, npm v12 disabled install scripts by default, reducing what GitHub called the ecosystem's largest code-execution surface. But the Jscrambler and AsyncAPI attacks show how quickly that protection can be routed around. Their payloads executed when the package was imported or required, not when it was installed.

The same problem exists in CI. The pull_request_target weakness behind the AsyncAPI compromise belongs to the same class of workflow flaw exploited in the March 2025 tj-actions/changed-files incident. GitHub has since introduced safer defaults in actions/checkout v7, but those protections only matter once teams adopt them. AsyncAPI's vulnerable workflow remained in place for 58 days after a contributor proposed a fix.

Even provenance could not distinguish a legitimate release from a malicious one. Because the attacker controlled the CI identity, the compromised pipeline produced valid Sigstore and SLSA attestations for the backdoored packages. The signatures accurately proved where the packages came from. They could not prove that the code entering the build was safe.

That is the limit of ecosystem guardrails. They can close known execution paths, harden workflows, and verify how an artifact was produced. They cannot tell a compromised team which credentials the malicious code reached, whether those credentials are still valid, or what access they provide.

By the time a supply chain attack is discovered, those are the questions that determine the blast radius. Answering them quickly requires visibility that exists before the incident: an inventory of secrets across repositories, CI configuration, environment variables, developer machines, and the identities and systems behind them.

Aligning NHI Governance With Financial Services Regulatory Expectations

Dwayne McDaniel — Thu, 30 Jul 2026 13:26:21 +0000

Aligning NHI Governance With Financial Services Regulatory Expectations

Explore how NHI governance, secrets management, and risk framing support regulatory compliance, audit assurance, and operational resilience in the financial industry

By Dwayne McDaniel • 9 Feb 2026 • 8 min read

Senior security leaders in banking and financial services operate in a continuous translation role. Security organizations generate high volumes of alerts, findings, and technical metrics. Boards of directors, audit committees, and supervisory authorities evaluate performance through a different lens: enterprise risk, regulatory exposure, and operational resilience. Alignment depends on whether security activity can be translated into those terms in a credible and repeatable way.

The board's role is to define risk appetite, allocate capital, and ensure the institution can continue operating through disruption. Security teams must demonstrate measurable influence over loss exposure, supervisory confidence, and service continuity when thinking about selecting or implementing a specific technology or program. Controls, architectures, and tooling are inputs. Reduced likelihood of material loss and improved resilience are the outcomes that security leadership should strive to achieve.

In regulated financial institutions, compliance frameworks often serve as the initial proxy for risk management. They provide defensibility and a shared vocabulary in environments where the consequences are material. We can call this governance, and it shows that risk is intentionally managed, not just that security work is being performed.

Governance maturity develops when compliance evidence is consistently connected to changes in impact. That connection increasingly runs through identity governance and secrets security.

What Audit and Control Failures Have Cost Financial Institutions

Let's take a look at the realities that keep enterprise leaders up at night. The financial consequences of regulatory enforcement clearly show what is at stake:

OCC (2020) -- Capital One Bank An $80 million civil money penalty was imposed for information security deficiencies and noncompliance with "Interagency Guidelines" following a major unauthorized access incident.
FCA (2018) -- Tesco Personal Finance A £16.4 million fine after a cyber attack enabled unauthorized transactions. The FCA cited deficiencies that exposed customers to avoidable harm and highlighted weaknesses in access controls and monitoring.
SEC (2022) -- Morgan Stanley Smith Barney A $35 million penalty related to failures to safeguard customer personal information, with regulators pointing to deficiencies in protective controls and oversight mechanisms.
Poland DPA via EDPB (2025) -- mBank An administrative fine of €928,498.06 related to GDPR Article 34 violations.

Across jurisdictions, the pattern is consistent. Regulatory cost increases when unauthorized access occurs, when access privileges are weakly governed, or when institutions cannot demonstrate that controls operate effectively and consistently over time. The absence of evidence often matters as much as the presence of technical safeguards.

Why Risk Framing Determines Security Credibility

Vulnerabilities, misconfigurations, and exposed credentials are operational observations. Security professionals need to articulate how those observations translate into potential losses, how likely those losses are to occur, and what factors influence their magnitude.

Boards, regulators, and auditors all focus on loss outcomes. Their concerns center on customer harm, financial impact, service disruption, regulatory notification thresholds, and reputational damage. Security updates gain relevance when they clearly show how changes in tools and programs drive those risks down.

Risk models, such as Open FAIR, provide a shared structure for this conversation by defining risk as "the probable frequency and probable magnitude of future loss." Frequency is influenced by threat activity and the strength of resistance measures you have implemented. Loss magnitude is shaped by reachability, privilege concentration, and the speed and effectiveness of containment.

Within financial institutions, identity failures frequently act as the enabling mechanism for loss. This is magnified when you consider the scale of non-human identities (NHIs) outnumber humans: at least 80:1. Exposed credentials and poorly governed non-human identities provide direct access paths into systems that process payments, store customer data, or support trading, clearing, and settlement activity.

Auditors assess whether controls operate consistently and whether their operation plausibly reduces loss frequency or magnitude. Evidence accumulated over time forms the basis for that assessment, not isolated point-in-time attestations. And that reassurance, that over time risk has and will go down, is exactly what the board wants from security.

How Financial Regulation Reinforces Identity and Secrets Governance

Regulations, especially for the financial services industry, emphasize outcomes rather than prescribing specific technologies. Across jurisdictions, supervisory expectations repeatedly focus on understanding, controlling, monitoring, and recovering access pathways.

In the United States, Gramm-Leach-Bliley Act (GLBA) and the Federal Reserve's Interagency Guidelines Establishing Information Security Standards tie program adequacy to safeguards that protect customer information from unauthorized access.

SEC Regulation S-P: Privacy of Consumer Financial Information and Safeguarding Customer Information formalizes expectations for preventing and responding to unauthorized access within SEC-regulated entities.

Some states have their own regulations, such as New York State Department of Financial Services 23 NYCRR Part 500, which establishes explicit requirements around access privileges, authentication, governance, and ongoing monitoring.

Response speed is a measurable regulatory variable. The OCC, FDIC, and other bodies have a 36-hour notification rule. Credential validity duration is part of the determination of whether some thresholds are crossed and whether incidents escalate into reportable events.

Operational resilience frameworks expand the scope further. European regulations, like DORA and GDPR, emphasize tested controls and third-party access governance. UK operational resilience guidance from the FCA explicitly links identity failures to the disruption of important business services. Comparable expectations globally appear in Singapore's MAS TRM, Australia's APRA CPS 234, Canada's OSFI Guideline B-13, and the global industry's SWIFT Customer Security Programme.

Across regions and countries, access governance and identity control effectiveness remain in scope even when terminology varies. The question remains consistent: can the institution demonstrate effective control over who or what can access critical systems, and how quickly that access can be withdrawn when conditions change?

The Structural Growth of Non-Human Identities in Banking

The emergence of non-human identities, and the scale and rate at which we are deploying them, represents a major shift in access governance across financial institutions. Designed for machine-to-machine interaction, these identities enable applications, services, automation, and APIs to access systems without human involvement. They underpin payment processing, reconciliation, trading operations, cloud infrastructure, and third-party integrations that require reliable, automated access at scale.

These identities authenticate using secrets such as API keys, tokens, certificates, and service account credentials. As automation expands, so does the number of secrets required to support it. This growth is often decentralized and poorly documented, leading to secrets sprawl across code, pipelines, configurations, and cloud services.

Each secret represents an access path. When exposed, attackers can gain machine-level access without compromising a human account, often bypassing controls designed for interactive users. NHIs lack direct human oversight, so misuse is much harder to detect. In other words, "Attackers are not breaking in, they're logging in."

As the number of NHIs increases with every new deployment, mismanagement becomes more likely. Secrets are frequently long-lived, over-permissioned, or reused. Some are orphaned, with unclear ownership and no clear path to revocation or remediation. Boards recognize this as a governance tradeoff between efficiency and risk. Auditors assess whether institutions can inventory access mechanisms, explain the scope of any access, and revoke access quickly when needed. This requires proofs throughout each NHIs lifecycle. Gaps in visibility and control increasingly surface as audit findings.

Connecting NHI Governance to Enterprise Risk and Resilience

Effective alignment depends on presenting secrets and non-human identity risks in a structure that aligns with regulatory expectations. Again, we can look to guidance from Open FAIR. Four dimensions consistently resonate with boards, auditors, and supervisors.

Exposure channels - Secrets appear across repositories, configuration files, CI/CD pipelines, collaboration platforms, ticketing systems, documents, containers, and cloud services. This distribution broadens the attack surface and complicates assurance.

Time - Risk increases with the duration a credential remains usable after exposure. Time-to-revoke becomes a practical proxy for incident response effectiveness and regulatory exposure.

Identity sprawl - Non-human identities expand faster than human identities, receive less routine review, and frequently hold elevated permissions. Governance frameworks increasingly associate this sprawl with operational resilience and third-party risk.

Containment - Loss magnitude depends on how quickly access can be revoked and how accurately the impact scope can be determined. Delays increase financial, regulatory, and reputational consequences.

Mapped to a risk model, exposure and sprawl influence resistance strength, time influences loss event frequency, and containment influences loss magnitude. Boards can evaluate these relationships. Auditors can test the associated controls.

Translating Security Activity Into Risk Reduction

Security activity supports governance and proves risk management when it directly addresses loss drivers rather than operational artifacts.

Eliminating exposed credentials reduces the probability that threat activity escalates into loss.

Shorter credential lifetimes reduce misuse frequency.

Constrained permissions limit potential impact.

Faster detection and revocation influence both frequency and magnitude.

When remediation is expressed in these terms, it demonstrates proactive risk management rather than reactionary operational cleanup. Reduced high-risk exposure, improved revocation speed, and defined accountability for non-human identities provide evidence that auditors can validate.

The history of regulatory enforcement, particularly in the financial industry, shows that supervisors examine these same outcomes when assessing control effectiveness, regardless of the underlying tooling.

Aligning GitGuardian Signals With Board and Audit Expectations

From an auditor or regulator's perspective, security programs are evaluated by their ability to produce credible, repeatable evidence of risk management. The goal is to assess whether access risk is actively governed, whether controls operate consistently, and whether trends demonstrate sustained improvement. GitGuardian supports this alignment by translating secrets exposure and NHI risk into signals that map directly to governance expectations.

At the foundation is broad coverage across exposure channels. GitGuardian continuously monitors for leaked secrets in source code repositories, both publicly on GitHub and private repos on any major provider. The platform also connects with CI/CD pipelines, collaboration platforms, ticketing systems, and other systems. This visibility enables institutions to demonstrate where authentication artifacts are monitored and where residual gaps remain, a prerequisite for audit assurance. This problem of secret sprawl is, unfortunately, only getting worse.

GitGuardian NHI Governance platform adds contextual enrichment by associating secrets with repositories, services, environments, and identity types. This context clarifies reachability and privilege concentration, allowing governance stakeholders to understand not just that a secret exists, but what systems it enables and why it matters from a risk perspective.

GitGuardian NHI Governance Identities view

GitGuardian's Risk Score classification capability, driven by machine learning, helps teams quickly distinguish between low-risk artifacts and credentials that represent active access paths. This supports prioritization based on potential loss impact rather than alert volume.

GitGuardian Incident Risk Score Example

Risk-based prioritization and remediation signals further align security activity with governance outcomes. GitGuardian analytics provide metrics such as "Median time to remediate," which are measurable indicators of response effectiveness, particularly when notification thresholds depend on response speed. Workflow integration and ownership assignment reinforce accountability and support audit traceability.

GitGuardian Analytics

Over time, trend reporting converts operational activity into governance artifacts. Boards see directional risk reduction, auditors observe consistent control operation, and supervisors gain confidence that access risk related to non-human identities and secrets sprawl is actively managed.

In this way, GitGuardian functions as both a detection platform and a governance enabler, producing evidence that aligns security operations with enterprise risk oversight.

Communicating Security Risk With Boards and Auditors

Board reporting focuses on movement in risk drivers. Expecting exposure monitoring to expand and credential validity windows shrink. High-risk identities should get more controls implemented around them. And the evidence always needs to be available for review.

Audits emphasize structure and consistency. They need evidence that controls are in place and operate as designed. Trends from audits demonstrate improvement over time.

Financial services enforcement history demonstrates the cost of access control failures. Identity and secrets governance represent a domain where loss mechanisms are well understood and increasingly scrutinized.

When security signals align with loss frequency and magnitude, security leadership supports decision-making, audit outcomes, and regulatory confidence. At that point, security operations function as enterprise risk management.

We would love to help you get started with aligning your NHI governance with your compliance goals.

Blast Radius: What a Leaked Secret Breaks

Dwayne McDaniel — Wed, 29 Jul 2026 12:30:43 +0000

Why identity-local signals and topology signals are two layers of the same blast radius

The credential with the widest blast radius sometimes has no secret to flag. See how GitGuardian and Anyshift rank risk by what actually breaks.

By Louis Fradin • 23 Jul 2026 • 7 min read

👉 TL;DR: Identity-local signals show whether a credential or machine identity is risky. Topology signals show what breaks if that identity is abused. GitGuardian identifies and ranks exposed credentials and risky machine identities; Anyshift's graph adds downstream context by showing which services depend on the resources those identities reach. Together, they help teams prioritize by both credential severity and operational blast radius.

A leaked credential is also a topology problem

A leaked credential creates risk beyond the identity itself. Its real impact depends on the services and resources connected to what that credential can access.

Identity-local signals answer the first question: how risky is this credential or machine identity on its own? Is it plaintext? Guessable? Stale? Overprivileged? Production-exposed? Tied to an admin identity? Those signals matter because they identify the secrets and machine identities most likely to be abused.

But they do not answer the next question: what breaks if that credential is used?

That answer lives in the topology around the credential. A database credential may sit on one pod and unlock one datastore, but the operational blast radius extends to every service that depends on that datastore. Some of those services never hold the credential at all. Some may not even have a secret signal to score.

Want to run the same analysis on your own stack? Explore the Anyshift Graph API to query dependencies, blast radius, and production impact directly.

Learn more

That is where identity-local signals and topology signals become two layers of the same blast radius: one tells you why the credential is dangerous, and the other tells you how far the damage can travel.

The Temporal cluster example

We built a small Temporal cluster to walk through the difference.

In that cluster, a Postgres credential was sitting in plain text on the temporal-server pod:

POSTGRES_USER=temporal
POSTGRES_PWD=temporal

It was the default credential that shipped with the environment and was never rotated.

Point a secret scanner at that setup and GitGuardian catches it immediately, correctly. It identifies the exposed credential and maps the secret's direct neighbors: the temporal-server pod that consumes the credential and the postgresql database it unlocks.

That is the identity-local view. It is accurate, and it is necessary. The credential is guessable, plaintext, and unrotated. It deserves a high-risk score before you know anything else about the environment.

But the blast radius does not stop at the pod and the database.

Go one hop further and order-worker can no longer advance a single order once temporal-server loses access to Postgres. temporal-ui also goes dark for anyone trying to observe or debug the incident. Neither service holds the credential, but both depend on the resource the credential unlocks.

A secret-centric view can show the exposed credential and its immediate relationships. The topology view shows the downstream services that fail because of it.

Why the identity score is still right

A scanner scores an identity by what the identity is and how risky it looks in context.

GitGuardian's NHI Governance walks AWS IAM, Entra, Okta, and Kubernetes. It finds the machine identities and secret managers behind them, then ranks those identities based on factors like admin rights, overprivilege, production exposure, and staleness.

That matters because machine identities now vastly outnumber human identities. By GitGuardian's own count, machine identities can outnumber humans by as much as 100 to 1. In a real environment, a few thousand machine identities can quickly become hundreds that are admin, stale, overprivileged, production-exposed, or tied to exposed secrets.

In the Temporal example, the Postgres credential earns its high score honestly. It is guessable, plaintext, and never rotated. All of that is true before you know anything about what depends on the database it opens.

The point is not that the identity-local score is wrong. It is that it is only one layer of the risk.

Where the two layers disagree

Most of the time, identity-local risk and topology risk point in the same direction. A plaintext admin credential is usually dangerous on its own and connected to something that matters.

The two layers split when a resource has a wide downstream blast radius but carries a weak secret signal, or no secret signal at all.

That creates a prioritization gap. A severity ranking has plenty to say about the exposed Postgres credential because there is a credential to score. It has far less to say about a load-bearing resource that does not have a leaked secret attached to it.

The only way to find that second kind of risk is to follow each identity out to the services leaning on what it reaches.

Walking the credential downstream

Anyshift keeps a versioned graph of cloud, Kubernetes, IaC, code, and the edges between them. For the Temporal cluster, we asked the graph to keep going past the secret's direct neighbors.

The direct path was straightforward: the plaintext Postgres credential sits on temporal-server, and temporal-server uses it to reach the postgresql database.

The downstream path is where the blast radius appears.

Graph of the Postgres credential blast radius: a plaintext credential on temporal-server unlocks postgresql-0, stalling order-worker and temporal-ui downstream.

order-worker runs the durable order-fulfillment workflows. If the state store is unavailable, orders customers already placed stop moving. temporal-ui is also affected, which means the interface an on-call team may use during the incident becomes unavailable too.

Neither order-worker nor temporal-ui touches the credential directly. A secret-centric view does not list either one as a direct neighbor. But both are part of the operational blast radius because they depend on the service and datastore the credential unlocks.

There is also a second-order problem. Rotating the leaked credential is the correct response. But in this setup, the value lives in a plaintext environment variable with nothing in front of it. Rotating it restarts temporal-server, which can then restart or disrupt every pod down the chain.

In other words, the fix has its own blast radius.

Example: Redis-cart and the checkout blast radius

The Temporal cluster is intentionally small. Order fulfillment is the durable back half of a store, and in our demo cluster that back half runs only two services deep.

The customer-facing front half is where a datastore compromise can fan out more widely.

To show that difference, we ran the same walk on a second cluster: the Online Boutique demo. This time, we followed the cart store downstream from redis-cart.

The result was more differentiated.

Service graph of the Online Boutique checkout blast radius: a redis-cart failure takes down four services, degrades four, and leaves two untouched.

checkoutservice calls cartservice first. When the cart goes down, checkout goes down with it, and customers cannot complete a purchase.

frontend is affected too, but it degrades differently. It talks to six other services on its own, so browsing, currency, ads, and recommendations can keep serving even while the cart page is broken.

recommendationservice and adservice never notice.

The result is not a single flat outage. It is four services down hard, four degraded, and two untouched. No single number on the credential carries that split.

That is the value of the topology layer. It shows not only that a resource matters, but how failure propagates across the application.

The low-severity identity with the widest blast radius

We then ran that ranking across every flagged identity in both clusters. First, we scored each one the way a secret or identity scanner would. Then we scored each one by what its resource actually reaches downstream.

Most of the rankings lined up.

The admin token, the bootstrap token, and the Postgres credential all rank high either way. They are risky as identities, and they are connected to important resources.

Then there is redis-cart.

Table ranking four identities by identity-local severity versus downstream blast radius, showing redis-cart lowest on severity but widest in reach.

A secret scanner has almost nothing to flag on it because the cart store ships without a password. There is no leaked credential to rotate. There is no secret sitting on the resource to score. From a secret-centric view, it belongs near the bottom of the queue.

But by downstream reach, redis-cart is the widest application-layer failure in either cluster. It drives the checkout cascade described above. The identity with nothing to steal opens the most services, and it is the one a severity score alone sends to the bottom of the queue.

That is where identity-local scoring and topology scoring disagree most clearly.

The honest limits of the graph

The graph only reaches as far as what it has ingested.

On the Temporal cluster, the blast radius stops at two services because two services are all that cluster has. If another workload reaches the database over a path the graph has not mapped, that workload is invisible to the topology view, just as it is invisible to the scanner.

That limitation matters. The graph is not guessing at the rest of the environment. It ranks based on what it has actually traced across cloud, Kubernetes, IaC, code, and dependency edges.

That makes the topology score useful, but not magical. Its accuracy depends on the completeness of the graph.

How the two scores work together

Neither score replaces the other.

The scanner puts real credentials and risky machine identities on the list. It ranks them by exposure, privilege, staleness, and other identity-local signals. For most of the queue, that is the right starting point.

The graph adds what the scanner cannot see on its own: how far a compromise travels once an identity reaches a resource, and which load-bearing resources matter even when no secret is present.

On the Temporal cluster, that means the plaintext Postgres credential is both a high-risk secret and a source of downstream service failure. On the Online Boutique cluster, it means redis-cart becomes visible as the widest application-layer blast radius even though there is no credential for a secret scanner to flag.

Together, the two layers give teams a better priority order: fix the credentials that are dangerous, and understand which identities and resources can break the most when they fail.

SRE Playbook: A Guide to Discover and Catalog Non-Human Identities (NHI)

Dwayne McDaniel — Tue, 28 Jul 2026 12:37:49 +0000

As a site reliability engineer in a global company, I'm running a modern (well, relatively modern, to be honest and modest) cloud-native stack: HashiCorp Vault as the secret manager, workloads on Kubernetes clusters in AWS (EKS), and development workflows automated through Jenkins (legacy) and GitLab CI. This setup is, quite likely, familiar to you — it's the normal playbook in the cloud-native era.

In theory, we have the right tools for both security and efficiency: After all, we have a state-of-the-art secret manager integrated with everything. But in reality, it's far from the truth. See if you resonate with the following scenarios:

Scenario A: A new colleague just joined the team.

Manager: "Your initial password to log in to your corporate account came to me via email, but since you can't log in to your mail account just yet, here, take a picture of my screen." (In some companies, taking a picture of a computer monitor would get you fired, I'm not kidding.)

Scenario B: A developer needs a temp password to access a database.

Dev: "Where is the newly created temporary password? Need it for debugging."
Ops: "In the Vault."
Dev: "I can't access Vault."
Ops: "No, you can't. It's not safe to open UI access to Vault. Corporate policy."
Dev: "Then how can I get the password?"
Ops: "Well... Technically, the password isn't in the Vault. There is a Jenkins pipeline that calls the Vault API to generate a temp password, then stores it in Jenkins secrets. You need to request access to the corresponding Jenkins pipeline, trigger it, then get the secrets from Jenkins."
Dev: "Why on earth do we store secrets in Jenkins when we have Vault, which we aren't allowed to use?"
Ops: "Corporate policy, just told you."

Scenario C: A new ops team member needs to update a certificate for a service running in production for the first time.

Ops: "Where is the old cert?"
Mentor: "In K8s as a secret."
Ops: "Where is the cluster?"
Mentor: "In AWS."
Ops: "How do I access that?"
Mentor: "You need to assume a specific role for that."
Ops "Which role?"
Mentor: "Let me check."
Ops: "Where is the new cert?"
Mentor: "Depends. Either in Vault or generated by Let's Encrypt."
Ops: "How is the process not automated?"
Mentor: "Been on the roadmap for 3 years."

You probably have already figured out where I'm going with these stories (oh, by the way, they are not made up), and you are right: the fragmented situation introduces new risks and operational challenges, even for global teams that have invested in best-in-class tools.

Today, let's have a closer look at it - from an SRE's standpoint.

1. Secret Managers Alone Aren't Enough

As our environment scales, so does the operational overhead and complexity.

Imagine this: secrets, users, and roles are being created for hundreds of developers. All of these are managed across multiple, disconnected places: Vault, Cloud IAM, Kubernetes clusters, Jenkins secrets, and GitLab secrets.

Just when you think this is already more than enough to handle, to make things worse, there comes more: besides username/password for humans, there are credentials, like API keys, roles, service accounts, certificates, and what have you, for apps and services and scripts and bots, used by thousands of machines.

Each system has its own way of handling credentials and permissions, and not everything is centrally visible (not to mention controlled).

Secret managers are a critical tool, but they are not a complete solution for all secret and identity challenges, for a number of reasons:

Not all credentials, roles, or secrets originate from or are managed by secret managers (for example, IAM roles are managed within cloud provider IAM systems), as we can see above in those real-world examples.
Some secrets, like database initial credentials or temporary access keys, are generated dynamically by cloud services and may not be automatically synced to secret managers.
CI systems often have their own secret storage mechanisms, and not all secret managers natively integrate or sync with these systems. This means additional silos.
Kubernetes Secrets: Although in an ideal world, K8s secrets should be synchronized from secret managers automatically, in the real world, they could be created from various sources. This means sensitive data may exist outside the view of our main secret management solution.
Non-Human Identities (NHIs): Service accounts, application identities, and other NHIs may be provisioned and managed in different systems, making it difficult to have a single source of truth.

Solely relying on a secret manager can (and will) create blind spots, because a more comprehensive approach is required for managing not only users and passwords, but also roles, certs, and all the other stuff across all systems.

2. An Introduction to Non-Human Identities (NHIs)

As shown in previous examples, there are scripts, bots, service accounts, and automation tools running in our stack. Each one needs a way to prove who it is — just like a human user — but instead of passwords, they use tokens, API keys, roles, certificates, or other types of secrets. These are Non-Human Identities (NHIs): identities for machines, apps, and services that need to talk to each other without human intervention, and chances are, they do not all live in the secrets manager.

NHIs are everywhere in modern DevOps and SRE workflows. They power CI/CD pipelines, connect microservices, sync data between clouds and datacenters, and keep our infra running. But unlike human users, NHIs don't have a face, a Slack handle, or a manager. They're spun up and torn down by code, often with little oversight, in multiple places. That's why it's important to understand where they live, what they can access, and how they're managed.

The reality is, for every engineer on the team, there are probably hundreds of NHIs quietly doing their jobs in the background. They're everywhere — across clouds, clusters, CI systems. Because they're so easy to create (and forget), they're often over-privileged but under-monitored without clear ownership.

This makes NHIs a goldmine for malicious actors. If a token or key leaks, it can open the door to sensitive systems — sometimes with more power than any single human user. And with so many NHIs scattered across the whole environment, it's easy to lose track of who (or what) has access to what. Oh, by the way, if you'd like to develop an incident response playbook for leaked secrets, read my previous blog.

If we're not actively managing NHI security — tracking where they live, enforcing least privilege, and cleaning up what we don't need — we're leaving our infra exposed. Getting proactive about NHI security isn't just a best practice; it's mandatory for keeping systems safe in a world where automation is king.

3. The Vicious Cycle of Toil and Risk

From an SRE perspective, the chaos of scattered NHIs, secrets, and roles is a significant source of toil and a direct threat to reliability. Here's why this is a critical problem that needs a playbook:

Lack of observability: Impossible to get a complete inventory of all identities and their permissions. This creates blind spots, increasing the risk of orphaned credentials and violating the principle of least privilege. Unused credentials accumulate, creating "zombie" identities that expand the attack surface.
Increased security toil: Manually tracking, rotating, and auditing secrets across multiple systems is repetitive and error-prone. This operational overhead detracts from engineering efforts that could improve site reliability. Without a central view, enforcing security practices like rotation or least privilege becomes a manual, best-effort, easy-to-forget process, making it impossible to define, measure, and meet security SLOs.
Degraded incident response: In the event of a breach, the time to resolution (TTR) is significantly higher. The lack of a clear inventory means we can't answer the most critical questions: "What is compromised? What can it access? How do we revoke it?" SREs must manually find and revoke credentials across disconnected systems, making a quick response nearly impossible.
Compliance: Proving compliance against corporate policies and security standards becomes a high-hanging fruit if we don't even have an overview.

When we can't oversee the lifecycle of NHIs and secrets, we enter a vicious cycle where toil increases, and reliability degrades. One of the core SRE principles is to automate away toil. Centralizing the management and observability of these NHIs is the first step to reducing risk.

4. The NHI Playbook: Establish a Central Plane for Observability

To address the fragmentation of NHIs and secrets, it's essential to establish a central platform that provides observability across all systems. An effective approach is to create a dashboard or an internal developer portal that serves as the source of truth. It should:

Aggregate data for full observability: Integrate with all sources — cloud IAMs, secret managers, CI/CD platforms, and Kubernetes — to collect and display every identity, secret, and role in one place.
Provide visibility: Offer a single plane for SREs and security teams to audit all credentials and permissions.

One option is to custom-build a portal tailored to our tools and environments. By using APIs provided by different tools and by using AI and MCP servers, we can quickly get started. For example, we can use or extend open-source developer portal frameworks (e.g., Backstage), or we can write scripts to collect and develop a web page to show them.

In this repository, I created some scripts to collect NHIs from different sources (AWS IAM, Vault, K8s, and GitLab CI).

For example, I can get all roles in IAM and see if they are assumed by any service.

iam_client = boto3.client("iam")
nhi_roles = []

paginator = iam_client.get_paginator("list_roles")
for page in paginator.paginate():
    for role in page["Roles"]:
        assume_role_policy = role.get("AssumeRolePolicyDocument", {})
        if assume_role_policy:
            statements = assume_role_policy.get("Statement", [])
            for statement in statements:
                principal = statement.get("Principal", {})
                if "Service" in principal:
                    nhi_roles.append(
                        {
                            "RoleName": role["RoleName"],
                            "Arn": role["Arn"],
                            "ServicePrincipal": principal["Service"],
                        }
                    )
                    # A role is considered an NHI if it can be assumed by any service.
                    # So we can break after finding the first service principal.
                    break

For another example, I can get variables from projects and see if secrets for machines/CI are there:

for project in projects:
    variables = project.variables.list()
    if variables:
        found_secrets = True
        print(f"\nProject: {project.name_with_namespace}")
        for var in variables:
            # Do not print var.value, as it is a secret
            print(
                f"- Variable Key: {var.key}, Scope: {var.environment_scope}"
            )

While this solution isn't trivial, centralizing visibility and control is a critical step toward risk reduction.

5. Why the DIY Solution Might Not Be For You

I wouldn't go so far as to say that building a custom tool for managing NHIs is "anti-pattern", but sometimes it is, because it often creates more toil than it solves. Here's why the DIY approach is a lost game from both a technical and operational standpoint:

More than just API calls: In the previous example, I provided four separate scripts for collecting NHI info from different sources. How about merging them into a microservice? Shall I build a Docker image and a Helm chart so that it can be deployed in a K8s cluster? Then it becomes a real service that needs to be version-controlled and lifecycle-managed. This is essentially committing to a permanent maintenance cycle.
More than just backend: Now the team is responsible for a full-stack product with a frontend page, and that's not trivial even if we use some open-source frameworks. And how about integrating it with the corporate SSO? Then a side project becomes a full-on engineering investment.
Engineering capacity: Every hour spent building and maintaining this internal tool is an hour not spent on other, more important stuff.
And how about the reliability of the platform itself?

The bottom line is that only you can decide if an in-house tool is the right choice for you. A DIY solution offers total control, but on the other hand, it could be a high-toil play.

6. A Managed Solution: NHI Governance

Instead of building and maintaining our own NHI aggregation platform, we can leverage a dedicated solution, like GitGuardian NHI Governance. It's designed to centralize the discovery, inventory, and management of Non-Human Identities (NHIs) and their secrets across the entire environment. Key features include:

Wide range of integrations: NHI Governance connects to a wide range of sources, including secret managers (Vault, AWS Secrets Manager, Azure Key Vault, etc.), cloud IAMs (AWS IAM, Azure Entra), CI/CD systems (GitLab CI), and K8s clusters.
Unified exploration map: A centralized, searchable inventory of all discovered NHIs, enriched with metadata such as source, environment, usage, and policy breaches.
Policy enforcement: GitGuardian applies security policies (informed by the OWASP Top 10 for NHIs) to detect risks like public/internal leaks, cross-environment or reused secrets, long-lived credentials, and more. Breaches are highlighted in the inventory and mapped visually for fast remediation.
Permission and blast radius analysis: Integrations with AWS IAM and Azure Entra provide deep context on permissions, roles, and the potential impact of compromised credentials, helping you prioritize remediation efforts.
Continuous tracking: Dashboards track breached policies over time, vault coverage, integration health, secret age distribution, and incident trends, providing actionable insights.
Secure and scalable: Integrations use secure auth methods (e.g., OIDC), require only read permissions, and never expose secret values. It's designed to scale with your environment and support multiple tenants or environments.

Apparently, choosing a platform like GitGuardian NHI governance means no complexity and overhead of a DIY solution. What's more, we could benefit from ongoing improvements, new integrations, and up-to-date security policies without additional engineering effort. This allows us to spend less time on data collection and more time on improving our security posture. But again, only you know which is better in your case, a custom-made solution or a managed one.

Summary

In complex cloud-native environments, SREs face a growing challenge: Non-Human Identities (NHIs) scattered across dozens of systems like Vault, AWS IAM, Kubernetes, CI systems, and more. The fragmentation creates blind spots, making it nearly impossible to maintain a complete inventory, not to mention enforce security policies or respond effectively to incidents.

While secret managers are essential, they are not the full picture. Building a custom internal developer portal to centralize this information is tempting, but could be costly; on the other hand, we can also choose a dedicated platform like GitGuardian NHI Governance.

Ready to move from fragmented visibility to centralized control? Stop chasing down scattered secrets and start proactively managing your NHI security today!

Confronting Vault Sprawl And The Risks It Brings

Dwayne McDaniel — Mon, 27 Jul 2026 13:01:49 +0000

Modern enterprises do not set out to create a maze of credentials, keys, and secrets stores. However, this is the reality most organizations find themselves in as they ship applications faster than ever, creating many new non-human identities (NHIs) in the process. Any issues around secrets management are typically solved by isolated teams working in silos, using whatever tools are available.

We can refer to the end result of these disjointed efforts around secrets management as "Vault Sprawl."

Vault sprawl is the uncontrolled growth of secret storage systems across an organization, driven by the rapid creation of new secrets needed to allow new workloads, scripts, bots, and now, agents to securely authenticate.

It shows up as multiple vault products and secret stores running at the same time, separate vault instances per team or environment, and the same credentials duplicated across tools because migrations are slow and integrations are uneven. Over time, it becomes difficult to answer basic questions like where a given secret lives, which copy is authoritative, and who can access it today.

Secrets Sprawl Sets The Stage For Vault Sprawl

Vault sprawl is directly related to the separate, perhaps riskier problem of secrets sprawl. Secret sprawl occurs when a credential leaks into plaintext across everyday engineering surfaces, then quietly spreads as it gets copied, reused, and forgotten.

That credential might be an API key, a token, or a database connection string that lands in source code or config. These sprawled secrets tend to keep showing up in new places because developers optimize for keeping systems running and unblocking deployments.

Unfortunately, we know this is a growing problem.

The 2025 State of Secrets Sprawl report found 23.77 million hard-coded credentials added to public GitHub repos in 2024 alone. This represents a 25% year-over-year increase. The same research found that this problem is at least eight times worse in private repositories that were scanned.

Vault Sprawl Is A Side Effect Of Good Intentions

When a problem shows up at that scale, teams respond the way teams always do under pressure: stop the bleeding in the places they can control. The obvious answer is to stop hardcoding secrets, but the question of how is often answered differently by each siloed team.

One group adopts the secret store that came with their cloud. Another relies on the built-in secret feature in their CI system. A platform team standardizes on a vault for Kubernetes. Each choice is defensible in isolation, but the organization pays the price for the lack of a single operating model.

Secret sprawl creates the initial urgency. Vault sprawl is the compounding side effect. The end state is duplicated credentials, fragmented access control, and unclear ownership, which makes rotations risky and incidents slower. It also makes governance harder because no one can state, with confidence, which systems contain the critical secrets and which copies are still live.

Developers Use Workarounds When Confused

Real evidence of a lack of a cohesive strategy across the whole enterprise shows up in a familiar place: leaked secrets. These are put there because a developer had a deadline and was not sure which vault held the right secret. Or because they deemed the "safe route" of requesting the correct access to be too slow. The State of Secrets report findings reinforce this, as we saw 5.1% of repositories using secrets managers still leaked secrets. Tooling does help, but it does not automatically create consistency across teams.

These secrets are not just in code, as secrets get copied into Jira tickets, Slack messages, documentation, and many other platforms around the software development cycle. Often, these are shared to help speed along a hotfix or to help the wider team work on an issue without getting proper access to the right vault. When secrets live in many places, this is, unfortunately, a predictable outcome.

Vault Sprawl Means Lack Of Governance

Governance is built on simple questions that need consistent answers:

Who owns that secret?
Which workload is allowed to use it?
Where is it stored?

Vault sprawl means there is no central source of truth, and getting to each answer becomes a negotiation across teams, tools, and ticket queues.

Vault sprawl means policies stop being global and become local. Each vault has its own access model, audit trail, and rotation workflow. That makes it hard to enforce least privilege, hard to prove environment separation, and hard to demonstrate consistent offboarding. Governance often becomes documentation-focused while operational reality continues to evolve at machine speeds.

Vault sprawl is a governance failure pattern that shows up as duplicated secrets, fragmented access, and unclear ownership. These issues directly map to the OWASP Non-Human Identities Top 10 for 2025, including:

Leaked Secrets – Exposed on public platforms or in plaintext inside internal systems
Cross-Environment Secrets Use – Shared secrets across environments
Reused Secrets – The same credential is used in multiple places
Duplicated Secrets – When the same secret exists in multiple vaults or locations
Long-Lived Secrets – Secrets that have gone unrotated for months or years, which are often forgotten but still exploitable

Vault sprawl is a governance problem because it turns identity and access into an organizational guessing game. This is the opposite of what auditors and regulators want to hear. They want clear answers to what state your environment was in when an unauthorized access event led to a material breach. They need to know that every effort was made to mitigate these risks.

The fix is not another vault or another policy doc. The solution is enterprise NHI governance that treats secrets storage, access, rotation, and lifecycle as one system with clear accountability across teams.

Enterprise NHI Governance Means Solving Vault (and Secrets) Sprawl

Fixing vault sprawl and getting to real governance means treating secrets and NHIs as shared enterprise infrastructure.

This starts with an inventory that covers the whole lifecycle of a secret, not just what lives in vaults. That means vaults, source code, CI/CD variables, build artifacts, and runtime environments. From that inventory, identify orphaned secrets, duplicated credentials, and stale access paths, then remove what is no longer needed.

This detection must be an ongoing effort but can also be shifted left, earlier in the workflow. Developers and DevOps teams need guardrails that catch secrets before they land in source control or get promoted into production. Pair tooling with clear standards so teams know where secrets belong and how to request access without slowing delivery.

Governance must also include lifecycle mapping and enriching secrets with metadata such as owner, intended workload, environment scope, and last rotation date. Use that to drive automated rotation, expiration, and renewal policies.

Finally, apply least privilege based on real usage. When you know which identities use each secret, you can tighten permissions, retire unused services, and reduce blast radius. For multi-cloud and M&A environments, consolidation matters, but a common control plane matters more. The goal is consistent policy and evidence, regardless of vault type.

How GitGuardian Can Help You End Vault Sprawl

Create One Source Of Truth Across Many Vaults

Vault sprawl gets dangerous when no one can answer, with confidence, where a secret lives, which copy is active, and which workload depends on it. GitGuardian tackles that by connecting to the vaults you already run and pulling back the signal you need for governance: inventory and metadata. This is designed to work across common secret managers like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and CyberArk Conjur, so teams can keep their operational preferences while leadership gets consistent visibility.

GitGuardian Connects to All Major Secret Managers

This also helps teams reduce operational overhead. Every additional vault adds licensing or consumption costs. But a bigger bill shows up as people's time as they work to solve secrets sprawl. By identifying patterns and existing resources across the organization's teams, they can build and maintain fewer vault integrations. Platform and security engineers can spend fewer cycles keeping the maze working and more time reducing other risks while accelerating delivery.

Turn "We Found A Leak" Into "This Is Now Governed"

Detection without follow-through is how secret sprawl turns into vault sprawl. GitGuardian's Push-to-Vault workflow is built for the last mile: moving an exposed secret from an incident into the right vault path, using a controlled process that avoids copy-paste remediation. The workflow is designed to help teams secure the secret, track it to ensure it remains under control, and reduce the backlog that often forms after high-volume leak events.

Push-to-Vault feature in the GitGuardian workspace

Exec teams want fewer fire drills while devs want fewer broken deployments. For certain types of secrets, GitGuardian provides incident response teams with a repeatable way to decide what to do next based on impact: vault or revoke. Noncritical and isolated secrets can easily be revoked from your GitGuardian workspace.

Critical secrets that are already managed follow an established rotation path. Unmanaged secrets that are not yet in production can be pushed into the vault, with clear guidance for developers to update the reference. This turns tribal knowledge into a shared playbook, which is what governance looks like under pressure.

NHI Governance Connects Secrets To The Identities That Use Them

Vault sprawl is ultimately an NHI sprawl problem. GitGuardian's NHI Governance is built to inventory non-human identities across the infrastructure, then use that inventory to drive security posture and lifecycle visibility. In practice, this helps teams map secrets to workloads, spot drift, and prove ownership, which is the foundation needed to align with the OWASP NHI risks you care about.

GitGuardian NHI Identities View showing breached policies

Progress has to be measurable, or NHI governance will remain a subject of debate rather than a coordinated program. GitGuardian NHI Governance helps you track movement over time using a performance dashboard that makes it easy to see which identities breached policies over time. This gives clear insight into if you are actually reducing risky patterns or just shifting them around.

In secret hygiene, vault coverage becomes a concrete metric, showing what percentage of secrets are stored in designated secret managers, plus an integration overview that highlights gaps in coverage across systems. Together, those views let you map progress from discovery to coverage, to fewer policy breaches, which is the practical definition of getting vault sprawl under governance.

GitGuardian's NHI Governance Analytics

Visibility Is The Foundation Of NHI Governance

Vault sprawl is a symptom of what every enterprise is going through: the massive explosion of non-human identities that make up modern infrastructure. Each leaked secret, appearing in plaintext, outside the vault, brings real risks. It is also evidence of a breakdown in NHI Governance strategy and execution.

Organizations need complete visibility to address NHI Governance. Without clear visibility into where secrets live, how they move through the delivery pipeline, and which workloads depend on them, governance becomes paperwork, and incident response becomes guesswork.

GitGuardian supports that visibility by combining secrets detection with inventory and NHI governance signals. It helps teams identify exposed credentials, understand where they are still in use, and reduce duplication across stores. Over time, that turns vault sprawl from an assumed cost of doing business into a measurable program with ownership, coverage, and lifecycle controls.

We would love to work with you to eliminate vault (and secrets) sprawl.

Shifting Security Left for AI Agents: Enforcing AI-Generated Code Security with GitGuardian MCP

Dwayne McDaniel — Fri, 26 Jun 2026 12:02:04 +0000

The rise of AI-powered coding agents promises to revolutionize software development, boosting productivity and accelerating iteration. Over the past year, AI in software development has started to evolve from locally embedded assistants to asynchronous cloud agents. However, this powerful new paradigm introduces a critical, industry-wide challenge: how do we ensure the code generated by these agents is secure by design?

The DevSecOps approach to code security is a great start. We can still utilize "security gates" like Pull Request (PR) checks and code reviews to help us identify when an agent has introduced a vulnerability. However, now that AI is able to iterate so quickly, these check-ins have become the new bottleneck. Every time an agent pauses to wait for a human to analyze scan results or request changes, it adds a significant amount of time to the development cycle.

The Industry Challenge: Securing AI-Generated Code

The fundamental challenge in securing code generated by AI agents stems from the training data that the underlying AI models were trained on. Humans are notoriously bad at writing vulnerability-free code, so LLMs have "learned" a lot from both bad and good examples. This means every line of code an agent suggests has a non-zero probability of introducing a known bad pattern or a vulnerability.

Developers can get instant vulnerability feedback via IDE plugins, but cloud coding agents like GitHub Copilot operate in isolated environments that are fundamentally incompatible with IDE plugins. This incompatibility makes it challenging to utilize state-of-the-art security tools early in the development cycle.

Another challenge with securing code was touched on in the introduction. The speed and autonomy of coding agents has completely changed the math on productivity. An agent can generate and commit dozens of complex PRs in the time a human developer would write a few functions. This volume of code overwhelms human developers with manual code reviews and security scan analyses from the CI/CD pipeline, turning them into a choke point.

The industry needs a solution that can integrate directly into the agent's workflow, identifying and correcting vulnerabilities at the moment the code is being generated or modified, without reliance on human analysis and feedback. GitGuardian MCP provides this capability by acting as an agent-native security tool directly available within the AI development environment.

Technical Implementation: Enforcing Security for Coding Agents with MCP

This section provides a step-by-step guide on how to integrate the GitGuardian MCP server directly into GitHub Copilot coding agent's configuration. This setup allows the agent to use the secret_scan tool to perform real-time security checks, ensuring code is secure before it is committed to a Pull Request branch and reviewed by humans.

If you just want to see the results, you can skip to the Demonstration section below.

1. Repository Setup

The first step is to establish an environment for the integration. In this example, we will set up a new empty repository in GitHub.

2. GitGuardian MCP Server Configuration

To integrate the MCP server, we need to add it to the agent's configuration and ensure the agent has the necessary permissions and network access.

We will add the GitGuardian MCP server to the Copilot coding agent configuration as shown below, referencing an environment secret for the personal access token variable (we will create this later).

{
  "mcpServers": {
    "GitGuardian": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/GitGuardian/ggmcp.git",
        "developer-mcp-server"
      ],
      "env": {
        "ENABLE_LOCAL_OAUTH": "false",
        "GITGUARDIAN_PERSONAL_ACCESS_TOKEN": "COPILOT_MCP_GITGUARDIAN_PERSONAL_ACCESS_TOKEN"
      },
      "tools": ["scan_secrets"]
    }
  }
}

Next, add https://api.gitguardian.com and https://dashboard.gitguardian.com to the Copilot coding agent internet access custom allowlist.

3. Service Account and Secret Management

To authenticate the agent's security scans, a dedicated GitGuardian service account with minimal permissions is required.

We can set do this in the GitGuardian settings. Create a new service account, and give it "scan" permissions.

Use the button at the bottom to create the service account and save the new service account's token for a later step.

4. Configuring the Environment Secret

The service account's token must be securely stored as an environment secret so that it's only accessible by the Copilot agent's MCP config.

Go to the GitHub repo's environment settings and navigate to the copilot environment or create one if it doesn't exist.

Add the environment secret we referenced earlier named COPILOT_MCP_GITGUARDIAN_PERSONAL_ACCESS_TOKEN, and paste the value of the service account token that was created in step 3.

5. Agent Instructions

The final piece of the setup is instructing the Copilot agent to use the new security tool as part of its standard workflow.

Create a Copilot instructions document that tells the agent to check all modified code with the secret_scan tool.

The GitGuardian MCP server is now set up and ready to be used by the Copilot coding agent.

Demonstration: MCP Security Tools in Action

To validate the MCP integration and Copilot's adherence to our new security rules, we can observe the agent's behavior during a typical development task.

1. Assign a task to Copilot

First, we will ask Copilot to generate code by creating an issue and assigning it to Copilot. In this example, we are asking for a boilerplate Flask API that supports authentication.

For demonstration purposes, we will explicitly ask Copilot to hardcode the secret key (this is a contrived example to force a finding, but hardcoded secrets may occur without explicit instructions).

2. Observe Copilot's behavior

Once assigned a task, Copilot will create a draft PR to track its work. Navigate to the PR and view the coding session to observe its activity in real time.

When the session kicks off, we can see the GitGuardian MCP server starting up.

As the agent implements the Flask API, we can see it has hardcoded the secret key.

Once Copilot is done making changes, it calls the secret_scan tool as instructed and finds the hardcoded secret key.

Because we explicitly asked Copilot to hardcode the secret to demonstrate this example, the agent only adds warnings instead of actually remediating the issue. In a real scenario, Copilot would not have conflicting instructions about how to handle the secret findings and would remediate the issue automatically.

Conclusion

In this blog post, we demonstrated how GitGuardian MCP can be used to shift security left in the absence of traditional security tools like IDE plugins. While hardcoded secrets are a prevalent and critical finding, the challenge of securing AI-generated code extends beyond secret exposure. This approach of providing agents with state-of-the-art security tools should be replicated to automate the detection and resolution of many issues.

Agents, like humans, aren't perfect, but we can secure AI-generated code. By embedding security directly into the AI agent's control plane and instructions, organizations can enforce security checks at the earliest possible stage, significantly accelerating the safety and productivity of agentic software development.