Shadow APIs often appear not through malice, but through routine delivery pressure: a service team ships an endpoint, another service starts calling it, and no gateway, inventory, or security control ever records that it exists.
What Makes an API “Shadow”
A shadow API is an endpoint that is reachable but absent from the organization’s approved API inventory, monitoring plan, documentation, or security review process.
It may be internal, external, temporary, deprecated, or accidentally exposed.
In microservice architectures, shadow APIs commonly include:
• Internal REST endpoints bypassing the API gateway
• gRPC methods missing from service catalogs
• Debug or admin routes left enabled after testing
• Older API versions still serving traffic
• Kubernetes services exposed through misconfigured ingress rules
• Lambda or cloud function URLs created outside standard deployment templates
• Partner endpoints documented in a ticket but never registered centrally
The key issue is not merely that the endpoint exists. The issue is that security, operations, and platform teams cannot reason about something they cannot see.
A public /api/v1/users/export endpoint with strong authentication, logging, schema validation, rate limits, and ownership metadata is manageable.
A private /internal/exportUsers endpoint accepting the same data but bypassing all controls is a material risk.
Why Microservices Create Shadow APIs
Microservices distribute ownership. That is useful for delivery speed, but it also fragments API control.
A monolith may expose a few route files and a single edge tier. A microservice estate may include hundreds of services, each with independent frameworks, deployment pipelines, sidecars, ingress objects, service discovery entries, and cloud permissions.
Shadow APIs emerge from several recurring patterns.
Direct Service-to-Service Calls
A team may expose an endpoint intended only for another internal service:
POST http://invoice-service.default.svc.cluster.local/recalculate
The endpoint works inside the cluster, so it never passes through the public API gateway.
Six months later, five services depend on it. No OpenAPI document exists. No one knows whether it accepts customer identifiers, payment data, or elevated operations.
Internal APIs still need governance. An attacker who reaches the cluster network through SSRF, compromised credentials, or workload takeover can use those routes laterally.
Temporary Endpoints That Become Permanent
A migration team adds:
POST /admin/backfill-customer-status
The route is meant for a two-week database correction. It accepts a list of customer IDs and updates account state. The sprint ends. The endpoint remains.
Temporary routes are especially risky because engineers often exempt them from normal controls to finish operational work. They may lack pagination, input validation, authorization checks, or rate limits.
Version Drift
An organization may publish /v3/orders but keep /v1/orders alive because a mobile app or partner integration still depends on it. Over time, the old version falls out of testing and monitoring.
Attackers like old APIs because they often contain older assumptions:
• Weak authorization rules
• Missing object-level checks
• More verbose error messages
• Older serializers
• Deprecated authentication mechanisms
• Business logic no longer reviewed by service owners
An endpoint does not become safe because it is undocumented. It becomes harder to defend.
Framework Defaults and Debug Routes
Modern frameworks often expose operational endpoints:
• Spring Boot Actuator: /actuator/env, /actuator/heapdump
• Express debug middleware
• Django admin panels
• Rails routes mounted for development
• Prometheus metrics endpoints
• Swagger UI and OpenAPI JSON documents
Some are safe when properly configured. Others leak environment variables, memory snapshots, build data, dependency versions, or internal URLs.
A common failure is enabling a broad actuator path during a performance incident and forgetting to lock it down afterward.
The Security Impact
Shadow APIs increase risk across three dimensions: exposure, privilege, and blind spots.
1. Broken Object-Level Authorization
OWASP API Security Top 10 repeatedly highlights broken object-level authorization. Shadow endpoints are prime candidates because they often skip shared authorization middleware.
Example:
GET /internal/customer/83922
Authorization: Bearer
If the endpoint checks only that the token is valid, not whether the caller can access customer 83922, it becomes a data exposure path.
Public endpoints may enforce tenant isolation, but internal endpoints sometimes trust that callers are “already inside.”
That assumption fails in Kubernetes clusters, flat VPCs, and overly broad service accounts.
2. Excessive Data Exposure
Shadow APIs may return fields never intended for consumers:
{
"id": "83922",
"email": "customer@example.com",
"passwordHash": "$2a$10$...",
"riskScore": 82,
"internalNotes": "Chargeback risk",
"ssnLast4": "1844"
}
Even if the route is not advertised, it may be discoverable through traffic inspection, JavaScript bundles, mobile app decompilation, logs, error messages, or DNS records.
3. Monitoring Gaps
If traffic bypasses the gateway, key controls disappear:
• Central request logging
• WAF rules
• API schema validation
• Bot detection
• Rate limiting
• Threat detection
• Standard authentication middleware
• Data loss monitoring
A shadow endpoint can receive malicious traffic for weeks without appearing in dashboards. The first visible sign may be fraud, account takeover, database load, or customer reports.
Where to Look for Shadow APIs
API discovery must pull from runtime, build-time, and configuration sources. No single data source is sufficient.
1. Gateway and Ingress Configuration
Start with what is officially exposed:
• NGINX ingress rules
• AWS API Gateway routes
• Azure API Management APIs
• Kong, Apigee, or Envoy configurations
• Istio VirtualService and Gateway resources
• Traefik routers
• Cloud load balancer listener rules
Compare those routes against service repositories and observed traffic. Any endpoint seen in traffic but absent from the gateway inventory deserves review.
Any endpoint defined in code but not reachable may still matter if a future ingress change exposes it.
2. Kubernetes Service Discovery
Kubernetes makes internal reachability easy. That convenience can hide sensitive APIs.
Useful commands include:
kubectl get svc -A
kubectl get ingress -A
kubectl get endpoints -A
kubectl get httproute -A
kubectl get virtualservice -A
Look for services with names such as admin, debug, internal, migration, backfill, tools, or ops. Naming is not proof, but it gives analysts a starting point.
Also inspect annotations. Cloud controller annotations can silently create external load balancers:
service.beta.kubernetes.io/aws-load-balancer-internal: "false"
A single boolean can turn an internal tool into an internet-facing service.
3. Runtime Traffic
Observed traffic is the strongest evidence that an endpoint matters.
Sources include:
• Envoy or service mesh access logs
• eBPF network telemetry
• VPC Flow Logs plus Layer 7 enrichment
• NGINX ingress logs
• Application logs
• API gateway access logs
• OpenTelemetry traces
• Packet captures in controlled environments
For HTTP APIs, collect method, host, path template, status code, caller identity, target service, authentication state, and response size. For gRPC, capture service and method names such as:
payments.AuthorizationService/CreateHold
Path normalization is critical. Without it, /users/123, /users/456, and /users/789 look like separate endpoints. A discovery system should collapse them into /users/{id} where possible.
4. Source Code and Route Extraction
Static analysis can reveal routes before deployment.
Examples:
Express:
router.post('/admin/reindex', requireAdmin, reindexHandler)
Spring:
@PostMapping("/customers/{id}/status")
FastAPI:
@app.get("/internal/risk/{customer_id}")
ASP.NET:
[HttpPost("accounts/{id}/freeze")]
The discovery process should extract method, path, file owner, repository, authentication middleware, and deployment target. Pairing this with runtime traffic shows whether a route is dead, active, internal-only, or newly exposed.
OpenAPI, Protobuf, and AsyncAPI Specifications
Contract files are useful, but they are often incomplete. Treat them as one evidence source, not the source of truth.
For REST, compare observed routes to OpenAPI paths. For gRPC, compare traffic and server reflection output to protobuf definitions. For event-driven APIs, inspect AsyncAPI specs, Kafka topic usage, schema registry subjects, and consumer groups.
A missing contract does not automatically mean the API is dangerous, but it does mean no one has formally described its behaviour.
Building a Shadow API Detection Program
The best programs combine automated discovery with ownership and remediation workflows.
1. Establish a Canonical API Inventory
Every endpoint should map to a record containing:
• Service name
• Owning team
• Repository
• Runtime environment
• Protocol: HTTP, gRPC, GraphQL, WebSocket
• Exposure: public, partner, internal, cluster-only
• Authentication method
• Authorization model
• Data classification
• Current version
• Deprecation date, if applicable
• Last observed traffic
• Contract location
• Logging and rate-limit status
This inventory should be generated as much as possible. Manual spreadsheets decay quickly.
A practical approach is to feed a central catalog from CI pipelines, Kubernetes admission controllers, gateway configs, and runtime telemetry.
Backstage can serve as a front end for ownership and documentation, but it needs automated inputs to stay accurate.
2. Detect Drift Continuously
Run drift checks on every deployment:
- Extract routes from the build artifact or source tree.
- Compare them with approved API contracts.
- Inspect Kubernetes and gateway changes.
- Flag new externally reachable paths.
- Require owner approval for sensitive methods such as POST, PUT, PATCH, and DELETE.
- Block deployments that expose admin routes without explicit policy exceptions.
This can be enforced through CI checks, Open Policy Agent, Conftest, Kyverno, or admission webhooks.
Example policy logic:
deny[msg] {
input.kind == "Ingress"
contains(input.spec.rules[].http.paths[].path, "/admin")
not input.metadata.annotations["security-approved"]
msg := "Admin path exposed without security approval"
}
Policy should prevent obvious mistakes without turning platform teams into ticket routers for every harmless change.
3. Classify Endpoints by Risk
Not all unknown APIs carry the same urgency. Risk scoring helps teams act.
High-risk indicators include:
• Internet exposure
• No authentication
• Admin verbs or privileged operations
• Access to personal, payment, health, or credentials data
• Deprecated versions receiving traffic
• Missing owner
• Large response bodies
• Abnormal error rates
• No rate limiting
• Use by unrecognized clients
• Write operations from outside the expected network segment
A newly discovered internal health endpoint is not the same as an unauthenticated /export route exposed through a load balancer. Triage should reflect that.
Hardening Shadow APIs After Discovery
Discovery is only useful if it changes system behaviour.
A. Remove or Retire
If an endpoint is unused, remove it. If it still receives traffic, identify callers through logs or traces and plan a migration. Set a removal date and monitor residual usage.
Deprecated APIs need explicit shutdown mechanics:
• Response headers announcing deprecation
• Client owner notifications
• Dashboards showing caller traffic
• Brownout windows
• Final removal tickets tied to release plans
Old endpoints survive because removal is more expensive than neglect. Make removal routine.
B. Put Internal APIs Under Zero-Trust Controls
Internal does not mean trusted. Service-to-service APIs should use strong workload identity, usually through mTLS with SPIFFE IDs, service mesh identity, or cloud-native workload identity.
Authorization should answer two questions:
- Who is calling?
- Is this caller allowed to perform this action on this object?
A policy such as “any pod in the namespace can call billing” is too broad for sensitive operations. Prefer explicit service identities:
allow inventory-service to call billing-service:GetInvoice
deny all other services
C. Standardize Middleware
Every service should use shared libraries or sidecar policies for:
• Authentication
• Authorization hooks
• Request validation
• Structured logging
• Correlation IDs
• Rate limiting
• Error handling
• Security headers where applicable
The goal is not uniform language or framework choice. The goal is consistent control behaviour across services.
D. Monitor Unknowns as First-Class Signals
Create alerts for:
• New path observed in production
• New external route
• Traffic to deprecated API versions
• Unauthenticated requests to non-health endpoints
• Internal endpoint called from unexpected workload
• Sensitive endpoint with sudden response size increase
• gRPC method observed without matching protobuf contract
These signals should flow to the owning team with enough context to act: sample timestamps, source workloads, destination service, path template, deployment version, and related pull requests.
A Practical 30-Day Plan
A focused first pass can produce results quickly.
Days 1–5: collect API gateway, ingress, service mesh, and Kubernetes service data. Build a rough inventory with ownership from repository metadata and deployment labels.
Days 6–10: enable or aggregate Layer 7 access logs for the highest-risk environments. Normalize routes and identify endpoints seen in traffic but absent from documented contracts.
Days 11–15: scan source repositories for route declarations. Compare code-defined routes with observed routes and published OpenAPI or protobuf specs.
Days 16–20: triage the top 25 unknown endpoints by exposure and data sensitivity. Remove dead admin routes. Add authentication to any unauthenticated sensitive path.
Days 21–25: add CI checks for newly introduced routes and ingress changes. Require contract updates for public and partner APIs.
Days 26–30: publish the inventory in a service catalog, assign owners, and create alerts for new production endpoints.
The first month should not aim for perfect coverage. It should convert invisible risk into named work owned by specific teams.
The Engineering Principle
Shadow APIs are a control-plane failure. They show that the organization can deploy network-reachable behaviour faster than it can inventory, govern, and observe it.
The fix is not a one-time scan. It is a feedback loop: extract routes before deployment, observe traffic during runtime, compare both against approved contracts, and make ownership visible.
A microservice architecture can tolerate rapid change if every new endpoint leaves a trace in the catalog, the logs, the policy engine, and the team’s review process. The next API shipped on Friday afternoon should be visible before Monday morning traffic reaches it.
Top comments (0)