The original content was shared on my Medium account; I'm sharing it here as well so it can reach a wider audience. here is the link
Should Internal Services Have Authentication?
As microservice architectures grow, security assumptions about the internal system start to be questioned. In many companies, the mindset is: “Let’s secure the services exposed to the outside; internal ones are already safe.” But this approach is not sustainable in the long run, neither for security nor operational management.
The Illusion of “Internal Services Are Already Safe”
For years, services running inside a private network were treated as trusted.
But here is the reality: When one internal service is compromised, it becomes a bridge to all other systems inside the network.
Network-level security(like VPC boundaries or security groups) alone is not enough.
In large environments where hundreds of services constantly talk to each other, relying only on the network layer basically means “trusting a firewall.” Modern security principles say the opposite:
No service should blindly trust another one.
The Hidden Cost of Living Without Internal Auth
Operating without authentication inside the network may look simple at first, everything can access everything.
But over time, questions like these become impossible to answer:
- Who is calling which endpoint?
- Who can access which data?
- Which service triggered this request?
Monitoring and auditing fall short, and during incidents or breaches, “Who made this call?” often stays unanswered.
Another hidden cost is uncontrolled access growth. Each new service gets access because it’s “internal,” and soon the system becomes a messy, implicit access network. In such an environment, applying security policy becomes nearly impossible.
Adding Auth to Every Service: Good in Theory, Painful in Practice
In theory, every microservice having its own authentication and authorization layer sounds correct. In practice, it becomes chaotic:
- Token verification
- Permission checks
- Policy management
- Handling error and fallback scenarios
Each service ends up reimplementing the same logic. This causes code duplication and inconsistent security standards across the company. One team may use OAuth, another JWT, another mTLS.
Eventually, every service follows its own path, and it becomes impossible to maintain a unified security posture.
The Alternative: Domain-Based Internal Gateway Layers
A more sustainable solution is to introduce internal API gateways for each domain. Each domain has its own gateway, and services inside that domain are only reachable through it.
Auth, authorization, rate limiting, logging, and tracing can all be handled in this gateway layer.
This approach reduces complexity:
- Services focus only on business logic.
- Security, observability, and access control become centralized.
- Domain boundaries become clearer.
- “Who can reach what?” becomes a definable and enforceable policy.
What This Model Brings
The biggest benefit is security. Each domain stays safe inside its own boundaries, and cross-domain access is controlled by explicit policies.
Tokens become domain-based, giving fine-grained control over inter-domain traffic.
Another major advantage is observability. Everything passing through a gateway is logged with identity information. This helps both performance analysis and security investigations.
Services also stay simpler, faster, and easier to manage since they don’t handle auth logic.
Counterarguments and the Reality
Some might think this model introduces performance overhead. But gateways already use mechanisms like caching, rate limiting, and circuit breakers to minimize this cost.
In modern infrastructure, the extra overhead is tiny compared to the value of centralized security and control.
From an organizational perspective, it doesn’t reduce team autonomy. Instead, domain-based gateways make responsibilities clearer and reduce total system complexity by removing security logic from services.
Conclusion: Internal Auth Gives You More Than Trust
Internal authentication doesn’t only provide security, it brings observability, governance, and sustainability.
In microservice architectures, secure communication can be achieved without sacrificing speed. Domain-based internal gateway design offers a healthier long-term structure for large organizations.
In the end, the real winners are those who build trust into their architecture, not their network.
Real-World Application: Current State, Problem, Solution, and Outcome
Let’s examine how this approach works in a real organization.
1) Internal-Auth Service: Central Identity, Distributed Trust
At the center of the system is an internal-auth service that issues short-lived JWT tokens to internal services.
A token contains:
- The name of the service requesting it
- Allowed domains and specific services/endpoints
- Token expiry(kept very short, like 10–15 minutes, since JWT invalidation is costly and complex)
- Optional metadata for monitoring/tracing
Tokens are signed using a private key.
Domain gateways know the corresponding public key and verify tokens locally, without contacting the auth service.
This reduces latency and prevents auth service overload.
_> Risk: Header Manipulation During JWT Verification
Since the JWT header is not encrypted, attackers may try to modify it. A common example is changing the alg field to “none” or switching algorithms. If the gateway trusts this header blindly, an unsigned or invalid token might be accepted.
To prevent this:
The algorithm must be enforced from configuration, not from the token.
Do not trust the JWT header; always verify using the expected algorithm and key.
Ensure library-level protections against alg-swapping attacks.
Without these controls, attackers might bypass signature validation completely._
2) Domain-Based Gateways: Controlled Boundaries, Enforced Policies
Each domain is protected by an internal gateway(e.g., KrakenD).
Goals:
- Services in other domains cannot access internal services directly.
- Inter-domain traffic goes through strict security rules.
- Authentication and authorization are handled at the gateway, not inside services.
Gateway evaluates incoming tokens:
- Verify signature with public key
- Check permissions for target service/endpoint
- Apply domain-level policies
- Route the request if allowed
This creates real domain boundaries and enables zero-trust inside the internal network.
3) Token Management in Services: Simplicity + Independence
Each internal service periodically requests a token from internal-auth using its own credentials.
Services:
- Cache the token(in-memory or shared storage like Redis for multi-instance setups)
- Refresh before expiry(e.g., at 90% TTL)
- Only include the token when calling another domain
- Handle token renewal internally
No service needs to trust another service, trust depends on tokens.
Architectural Challenges and Practical Solutions
Every security model brings risks. Here are the key risks in this design and how to address them.
Risk 1: Public Key Synchronization Issues
If gateways don’t receive updated public keys, token validation breaks.
Solution:
- Use a central config store(Consul, Vault, SSM, ConfigMap).
- Gateways fetch public keys on startup or periodically.
- Automate key rotation every 6–12 months(depending on risk).
Risk 2: Tokens Containing Too Many Permissions
Overpowered tokens create cross-domain vulnerabilities.
Solution:
- Apply “least privilege.”
- Auth service checks policies before issuing tokens.
- Use domain-specific tokens when needed.
- Periodically audit access lists and remove risky permissions.
Risk 3: Token Leakage or Theft
A stolen token allows an attacker to act as a service temporarily.
Solution:
- Keep TTL short(5–15 minutes).
- Store tokens only in-memory or protected caches.
- Use mTLS to add a second security layer.
- Tie tokens to IP/domain attributes if necessary.
Risk 4: Performance Load on Gateways
All traffic passing through gateways might create latency.
Solution:
Use caching and rate limiting.
Keep token validation lightweight.
Scale gateways horizontally when needed.
Recommendations for Long-Term Sustainability
1) Keep Domain Boundaries Strict
Services must not call other domains directly. All cross-domain communication must go through gateways.
2) Push Observability into the Gateway Layer
Gateways should log every request with identity metadata.
This answers:
- Which service called which service?
- Which token tried to access which domain?
- Where are unauthorized attempts happening?
3) Do Not Skip Key Rotation Automation
Key lifecycle management is critical. Both private and public keys must be rotated safely and automatically.
4) Manage Eventual Consistency Expectations
Policy updates, config changes, and key rotations propagate with slight delay, and that’s normal. Monitor these processes.
5) Provide a Simple SDK for Service Developers
Do not force each team to implement token refresh logic manually.
An internal SDK should handle:
- Fetching and renewing tokens
- Caching
- Retry and fallback logic
This prevents mistakes and speeds up development.
Final Outcome: A Secure, Scalable, and Manageable Internal Traffic Model
This architecture moves trust from individual services to well-defined domain boundaries. Zero-trust becomes truly applicable inside the system.
Centralized gateways deliver:
- Better governance
- Stronger security
- Improved observability
- Simpler services
With proper key management, short-lived tokens, and strict domain boundaries, this model scales effectively in large organizations.
What Architecture Should You Build?
In a structure with one public gateway, 4 domains, 12+ subdomains, 60+ internal services, and 4+ engineering teams, the redesigned architecture focuses on stronger isolation and reducing security risks.
Version-1: Simple Model
- Clients(mobile, web, etc.) reach internal services through a single public API Gateway.
- API Gateway is secured by OAuth/JWT/SSO.
- Internal services are not directly exposed to the internet.
- All internal services can freely communicate with each other.
This is secure enough for small environments, but lacks deeper segmentation.
Version-2: Improved Security and Isolation
To achieve stronger control, we redesign the architecture:
1. Client → API Gateway
Clients still access the system through the public API Gateway, the only externally exposed entry point.
2. Domain and Sub-domain Separation
- All services are grouped into 4 domains.
- Each domain can contain multiple sub-domains with several services.
- Domains are placed in different VPC/network segments for strong isolation.
3. Domain-Based Internal API Gateways
- Each domain has its own internal gateway.
- Internal communication is only allowed through these gateways.
- Services cannot talk to each other directly across domains.
- Internal Auth for Access Control
Cross-domain communication works only with proper authentication:
- A service requests a token from internal-auth using its own credentials.
- Internal-auth issues a short-lived JWT with explicit permissions.
- Every cross-domain request must include this token.
- Domain gateway verifies the token and checks authorization.
- Unauthorized requests are blocked at the gateway.
This allows:
- Centralized and manageable permissions
- Full domain-level segmentation
- Defined and safe communication paths
- Reduced impact of security breaches
Thank you for reading until the end. Before you go:
Top comments (0)