DEV Community

Cover image for Chapter 67 — Secure AI Networking & Service-to-Service Communication
Black Shadow Team ©
Black Shadow Team ©

Posted on

Chapter 67 — Secure AI Networking & Service-to-Service Communication

#ai

67.1 Introduction

A production AI platform is not a single application.

It is usually a distributed system containing:

  • frontend applications
  • API gateways
  • authentication services
  • application servers
  • AI orchestration services
  • model providers
  • background workers
  • databases
  • object storage
  • queues
  • caches
  • notification systems
  • observability services
  • external APIs

These components communicate continuously.

Therefore, network security becomes one of the most important parts of the overall AI security architecture.

A secure network design should answer:

  1. Who is communicating?
  2. What service is being contacted?
  3. Is the connection authenticated?
  4. Is the destination authorized?
  5. Is the traffic encrypted?
  6. Is the destination private or public?
  7. Can the service make arbitrary outbound connections?
  8. Can one tenant reach another tenant's resources?
  9. What happens if a service is compromised?
  10. Can suspicious traffic be detected?

The core principle is:

Network location should never be treated as proof of trust.

Being inside a private subnet does not automatically make a service trustworthy.


67.2 Traditional Network Trust vs Zero Trust

Traditional architectures often assume:

Internet
   ↓
Firewall
   ↓
Private Network
   ↓
Trusted Services
Enter fullscreen mode Exit fullscreen mode

This creates a large implicit trust boundary.

A zero-trust architecture instead assumes:

Service A
   ↓
Authenticate
   ↓
Authorize
   ↓
Encrypt
   ↓
Service B
Enter fullscreen mode Exit fullscreen mode

Every important service interaction receives explicit security treatment.


67.3 AI Application Network Model

A production AI platform may look like:

                         Internet
                            │
                            ▼
                     ┌─────────────┐
                     │ CDN / WAF   │
                     └──────┬──────┘
                            │
                            ▼
                     ┌─────────────┐
                     │ API Gateway │
                     └──────┬──────┘
                            │
                ┌───────────┴───────────┐
                ▼                       ▼
        Application Services       Auth Services
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
    Database  Queue    Cache
                │
                ▼
             Workers
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
    AI APIs  Storage   External APIs
Enter fullscreen mode Exit fullscreen mode

Each connection represents a potential security boundary.


67.4 TLS

Transport Layer Security protects network communications from interception and tampering.

A simplified flow is:

Service A
   │
   │ TLS
   ▼
Service B
Enter fullscreen mode Exit fullscreen mode

TLS should be considered for:

  • browser → API
  • API → authentication service
  • API → database
  • API → cache
  • worker → queue
  • application → AI provider
  • service → object storage
  • service → notification provider

The exact implementation depends on the infrastructure.


67.5 HTTPS

Public web traffic should use HTTPS.

The basic architecture is:

Browser
   │
 HTTPS
   ▼
Gateway
   │
 HTTPS / private secure channel
   ▼
Application
Enter fullscreen mode Exit fullscreen mode

HTTP should not be used for sensitive production communication merely because the traffic occurs inside a private network.


67.6 Certificate Validation

TLS is only useful when certificate validation is performed correctly.

Applications should validate:

  • certificate chain
  • hostname
  • expiration
  • trust configuration

Disabling certificate verification is dangerous.

Avoid patterns equivalent to:

verify_tls = false
Enter fullscreen mode Exit fullscreen mode

in production.


67.7 Mutual TLS

For sensitive service-to-service communication, mutual TLS can provide stronger service identity.

Ordinary TLS:

Client ────────► Server
        server
       authenticates
Enter fullscreen mode Exit fullscreen mode

mTLS:

Client ◄──────► Server
   │              │
   └─ authenticate ─┘
Enter fullscreen mode Exit fullscreen mode

Both sides present certificates.

This can help establish:

  • service identity
  • encrypted communication
  • stronger internal authentication

67.8 Service Identity

Every important service should have a distinct identity.

For example:

api-service
worker-service
billing-service
notification-service
media-service
ai-orchestrator
Enter fullscreen mode Exit fullscreen mode

Avoid using one universal credential for every internal component.

If one service is compromised, unique identities reduce the blast radius.


67.9 Service-to-Service Authorization

Authentication answers:

Who are you?

Authorization answers:

What are you allowed to do?

For example:

worker-service
   │
   ├── allowed → generation-service
   ├── allowed → object-storage
   └── denied  → billing-admin
Enter fullscreen mode Exit fullscreen mode

A successful network connection should not automatically grant unrestricted application access.


67.10 East-West Traffic

North-south traffic generally refers to:

Internet ↔ application
Enter fullscreen mode Exit fullscreen mode

East-west traffic refers to:

service ↔ service
Enter fullscreen mode Exit fullscreen mode

Modern AI systems can have enormous amounts of east-west traffic.

Therefore, internal traffic deserves serious security controls.

A compromised worker should not automatically be able to scan or communicate with every internal service.


67.11 Network Segmentation

A useful architecture separates services into logical zones.

Public Zone
 └── Gateway

Application Zone
 ├── API
 ├── Auth
 └── AI Orchestrator

Processing Zone
 ├── Workers
 └── Media Processing

Data Zone
 ├── Database
 ├── Cache
 └── Queue

Storage Zone
 └── Object Storage
Enter fullscreen mode Exit fullscreen mode

Network rules should explicitly define permitted communication paths.


67.12 Least-Privilege Networking

Instead of:

API → everything
Worker → everything
Enter fullscreen mode Exit fullscreen mode

prefer:

API
 ├── Database
 ├── Cache
 └── Queue

Worker
 ├── Queue
 ├── Database
 ├── Object Storage
 └── AI Provider
Enter fullscreen mode Exit fullscreen mode

The worker should not need unrestricted access to unrelated services.


67.13 Egress Control

One of the most important controls for AI systems is outbound traffic restriction.

A compromised AI worker might otherwise attempt:

Worker
  ├── Internet
  ├── Internal services
  ├── Metadata endpoints
  └── arbitrary external hosts
Enter fullscreen mode Exit fullscreen mode

A safer design is:

Worker
   │
   ▼
Egress Gateway
   │
   ├── Approved AI provider
   ├── Approved storage
   └── Approved APIs
Enter fullscreen mode Exit fullscreen mode

This dramatically reduces the impact of server-side request attacks.


67.14 SSRF

Server-Side Request Forgery occurs when an attacker influences a server into making a network request to an unintended destination.

AI applications are particularly relevant because they often process:

  • URLs
  • documents
  • web content
  • image URLs
  • API integrations
  • plugins
  • tool calls

A dangerous architecture might allow:

User
 ↓
AI Agent
 ↓
fetch arbitrary URL
Enter fullscreen mode Exit fullscreen mode

The AI system should never automatically receive unrestricted network access.


67.15 SSRF Defense

Use multiple layers:

User URL
   ↓
URL parser
   ↓
scheme validation
   ↓
hostname validation
   ↓
DNS resolution
   ↓
private/reserved IP rejection
   ↓
allowlist
   ↓
egress proxy
   ↓
request
Enter fullscreen mode Exit fullscreen mode

The important point is that URL validation should not rely on a single string comparison.


67.16 Private IP Protection

Outbound fetchers should carefully restrict access to private and special-purpose network ranges.

Examples include:

  • loopback
  • private network ranges
  • link-local addresses
  • cloud metadata endpoints
  • internal service addresses

An attacker should not be able to convert:

fetch URL
Enter fullscreen mode Exit fullscreen mode

into:

access internal infrastructure
Enter fullscreen mode Exit fullscreen mode

67.17 DNS Security

DNS is an important part of network security.

Applications should consider:

  • trusted DNS resolvers
  • DNS logging
  • DNS rebinding defenses
  • resolution validation
  • internal/private DNS separation

A particularly important issue is DNS rebinding.

A hostname might resolve to one address during validation and another address during the actual request.

Therefore, validation and connection handling should be designed together.


67.18 URL Allowlisting

Where practical, external integrations should use explicit allowlists.

For example:

Approved:
api.provider.example
storage.example
notifications.example
Enter fullscreen mode Exit fullscreen mode

rather than:

Any Internet destination
Enter fullscreen mode Exit fullscreen mode

This is especially important for:

  • AI tools
  • web fetchers
  • plugins
  • autonomous agents
  • document importers

67.19 AI Provider Connections

AI orchestration services often communicate with multiple providers.

For example:

AI Orchestrator
   │
   ├── Provider A
   ├── Provider B
   ├── Provider C
   └── Local Model
Enter fullscreen mode Exit fullscreen mode

Each provider connection should have:

  • dedicated credentials
  • TLS
  • timeout
  • rate limit
  • retry policy
  • request-size limit
  • output validation
  • monitoring

67.20 Provider Failover

Suppose Provider A becomes unavailable.

A failover architecture might be:

Request
   ↓
Provider A
   │
   └── failure
        ↓
     Policy Engine
        ↓
     Provider B
Enter fullscreen mode Exit fullscreen mode

However, failover should not accidentally transfer data to a provider that is not authorized to receive it.

Before failover, consider:

  • data classification
  • tenant policy
  • regional restrictions
  • model policy
  • provider approval
  • privacy requirements

67.21 Network Timeouts

Every outbound request should have bounded timeouts.

Without timeouts:

Worker
  ↓
external API
  ↓
hangs
  ↓
worker remains occupied
Enter fullscreen mode Exit fullscreen mode

Repeated failures can eventually exhaust worker capacity.

Use separate limits for:

  • connection timeout
  • request timeout
  • response timeout
  • total workflow timeout

67.22 Connection Pooling

Connection pools improve performance but must be bounded.

An unbounded pool can create:

too many requests
      ↓
too many connections
      ↓
resource exhaustion
Enter fullscreen mode Exit fullscreen mode

Configure maximum connection counts appropriate to:

  • service capacity
  • database capacity
  • provider limits
  • workload characteristics

67.23 Network Rate Limiting

Rate limiting can be applied at multiple levels:

Client
 ↓
Gateway
 ↓
Service
 ↓
Provider
Enter fullscreen mode Exit fullscreen mode

Examples:

  • requests per user
  • requests per tenant
  • requests per IP
  • connections per service
  • provider calls per minute
  • outbound bytes

Distributed rate-limit state can be implemented using the architecture discussed in Chapter 66.


67.24 API Gateway

The API gateway should provide a strong perimeter.

Typical responsibilities include:

  • TLS termination
  • authentication
  • request validation
  • rate limiting
  • routing
  • request size limits
  • WAF integration
  • observability
  • abuse controls

But authorization should still be enforced inside the application.

The gateway should not become the only security layer.


67.25 WAF

A Web Application Firewall can help identify and block common malicious traffic patterns.

Possible protections include:

  • injection patterns
  • malformed requests
  • excessive request rates
  • known malicious signatures
  • protocol violations

WAF rules should supplement application-level validation rather than replace it.


67.26 Internal APIs

Internal APIs should use authentication and authorization just like external APIs.

Avoid:

/internal/delete-user
Enter fullscreen mode Exit fullscreen mode

being protected only by the assumption:

Only internal services can access it.
Enter fullscreen mode Exit fullscreen mode

Instead:

Service identity
+
authorization policy
+
validated request
Enter fullscreen mode Exit fullscreen mode

should be required.


67.27 Service Mesh

A service mesh can centralize some service-to-service controls.

Conceptually:

Application A
    │
    ▼
Sidecar / Proxy
    │
    ▼
Network
    │
    ▼
Sidecar / Proxy
    │
    ▼
Application B
Enter fullscreen mode Exit fullscreen mode

Possible capabilities include:

  • mTLS
  • service identity
  • traffic policy
  • retries
  • observability
  • authorization

A service mesh can simplify large deployments but also adds operational complexity.


67.28 Kubernetes Networking

In Kubernetes, network policies can restrict which pods communicate.

Conceptually:

API Pod
  │
  ├── allowed → DB
  ├── allowed → Redis
  └── allowed → Queue

API Pod
  └── denied → unrelated workload
Enter fullscreen mode Exit fullscreen mode

Default-deny network policies are often a useful security baseline when operationally practical.


67.29 Namespace Isolation

Kubernetes namespaces can separate workloads logically.

Example:

production
 ├── api
 ├── workers
 └── ai

monitoring
 └── observability

security
 └── scanners
Enter fullscreen mode Exit fullscreen mode

Namespaces are useful organizational boundaries, but they should not be treated as complete security boundaries by themselves.


67.30 Container Network Security

Containers should have:

  • minimal network permissions
  • controlled DNS
  • restricted egress
  • restricted ingress
  • non-root execution where possible
  • limited capabilities

A compromised container should have as few paths as possible to reach sensitive infrastructure.


67.31 Database Network Access

Databases should not be publicly reachable.

Preferred:

Internet
   X
   │
Database

Application
   │
   ▼
Private Database
Enter fullscreen mode Exit fullscreen mode

Only authorized application services should have network access.

Database authorization still needs to be enforced at the database/application level.


67.32 Cache Network Access

Similarly:

Internet
   X
Redis
Enter fullscreen mode Exit fullscreen mode

Redis should reside in a private network.

Application services communicate through controlled network paths.


67.33 Object Storage Network Access

Object storage may use public cloud endpoints, but application access should still be controlled through:

  • IAM
  • signed URLs
  • bucket policies
  • private endpoints where available
  • encryption
  • logging

Network security is one layer of the storage security model.


67.34 Private Connectivity

Cloud platforms often provide private connectivity mechanisms.

Examples conceptually include:

Application Network
      │
      ▼
Private Endpoint
      │
      ▼
Managed Service
Enter fullscreen mode Exit fullscreen mode

This can reduce exposure to the public Internet.


67.35 Network Segmentation for Media Processing

Media processing is especially important because uploaded files are untrusted.

A strong architecture is:

Upload
  ↓
Quarantine Storage
  ↓
Scanner
  ↓
Isolated Processing Worker
  ↓
Validated Output
  ↓
Trusted Storage
Enter fullscreen mode Exit fullscreen mode

The processing worker should have minimal network access.

It may need:

read quarantine
write output
Enter fullscreen mode Exit fullscreen mode

but not:

access production database administration
access internal secrets
access unrelated services
Enter fullscreen mode Exit fullscreen mode

67.36 AI Agent Network Permissions

Autonomous agents introduce a special problem.

An agent might have tools such as:

web_fetch
email_send
file_read
database_query
api_call
Enter fullscreen mode Exit fullscreen mode

Each tool should have its own network policy.

For example:

Agent
 │
 ├── web_fetch → Internet proxy
 ├── file_read → approved storage
 ├── database → restricted API
 └── email → notification service
Enter fullscreen mode Exit fullscreen mode

Do not give the agent unrestricted network connectivity merely because it may need several tools.


67.37 Network Access as a Tool Permission

A useful security model is:

Tool
  ↓
Network destination
  ↓
Allowed method
  ↓
Allowed data class
  ↓
Allowed rate
Enter fullscreen mode Exit fullscreen mode

For example:

web_fetch
 ├── HTTPS only
 ├── approved domains
 ├── no private IPs
 ├── response size limit
 └── timeout
Enter fullscreen mode Exit fullscreen mode

This creates a much safer agent architecture.


67.38 Prompt Injection and Network Access

Prompt injection becomes significantly more dangerous when the AI agent has network privileges.

Consider:

Untrusted document
      ↓
Prompt injection
      ↓
AI agent
      ↓
network tool
Enter fullscreen mode Exit fullscreen mode

The malicious document may attempt to persuade the model to:

  • access unrelated URLs
  • transmit sensitive information
  • call unauthorized APIs

The defense is not merely "make the model smarter."

Instead:

Model decision
      ↓
Policy enforcement
      ↓
Network authorization
      ↓
Tool execution
Enter fullscreen mode Exit fullscreen mode

The network layer must independently enforce boundaries.


67.39 Data Exfiltration Controls

Outbound traffic should be monitored for unusual patterns.

Potential indicators:

  • sudden outbound volume
  • new destinations
  • unusual domains
  • repeated failed requests
  • large POST requests
  • unexpected protocols
  • unusual geographic destinations

Detection should combine:

network telemetry
+
application logs
+
identity information
+
request context
Enter fullscreen mode Exit fullscreen mode

67.40 DNS and Domain Monitoring

For systems with external network access, maintain visibility into:

  • requested domains
  • resolved IPs
  • service identity
  • destination port
  • request volume
  • response volume

This can help detect compromised workloads communicating with unexpected infrastructure.


67.41 Network Encryption vs Application Encryption

TLS protects data during transport.

Application-level encryption may protect sensitive data beyond the transport layer.

For highly sensitive information, consider whether encryption should occur:

Application
    ↓
encrypted payload
    ↓
TLS
    ↓
Network
Enter fullscreen mode Exit fullscreen mode

This provides defense in depth.


67.42 Certificate Rotation

Certificates should not live indefinitely.

A production system should support:

certificate issued
      ↓
deployed
      ↓
monitored
      ↓
rotated
      ↓
old certificate retired
Enter fullscreen mode Exit fullscreen mode

Automation is strongly preferable to manual certificate replacement.


67.43 Credential Rotation

The same principle applies to service credentials.

Credentials should be:

  • unique
  • short-lived where possible
  • scoped
  • rotated
  • revocable
  • stored securely

Avoid embedding service credentials in source code.


67.44 Network Observability

Important telemetry includes:

source service
destination service
source identity
destination identity
protocol
port
bytes
latency
result
authentication status
authorization result
Enter fullscreen mode Exit fullscreen mode

This helps reconstruct incidents.


67.45 Incident Response

Suppose a worker is compromised.

A strong architecture should allow:

Compromised Worker
       ↓
Detect
       ↓
Isolate network access
       ↓
Revoke identity
       ↓
Stop workloads
       ↓
Rotate credentials
       ↓
Investigate
       ↓
Recover
Enter fullscreen mode Exit fullscreen mode

Network isolation is one of the most powerful containment mechanisms.


67.46 Kill Switch

Critical AI infrastructure should have mechanisms to disable dangerous capabilities quickly.

For example:

Emergency Policy
      ↓
Disable external network access
      ↓
Disable autonomous tools
      ↓
Pause workers
      ↓
Preserve evidence
Enter fullscreen mode Exit fullscreen mode

This should be tested before an incident occurs.


67.47 Failure Containment

A secure distributed system assumes components can fail or become compromised.

Therefore:

Service A compromised
       ↓
Service B remains protected
       ↓
Database remains protected
       ↓
Secrets remain protected
       ↓
Incident contained
Enter fullscreen mode Exit fullscreen mode

This is the practical goal of segmentation and least privilege.


67.48 Secure Networking Checklist

[ ] Public services use HTTPS
[ ] TLS certificate validation is enabled
[ ] Sensitive internal connections are encrypted
[ ] Service identities are unique
[ ] Service-to-service authorization exists
[ ] Internal APIs require authentication
[ ] Network zones are defined
[ ] Database is private
[ ] Redis is private
[ ] Queue infrastructure is private
[ ] Worker networks are restricted
[ ] Egress is controlled
[ ] SSRF protections exist
[ ] Private IP access is blocked where appropriate
[ ] DNS security is considered
[ ] External domains are allowlisted where practical
[ ] AI provider connections are controlled
[ ] Timeouts exist
[ ] Connection pools are bounded
[ ] Rate limits exist
[ ] Network telemetry is collected
[ ] Suspicious outbound traffic can be detected
[ ] Credentials can be rotated
[ ] Certificates can be rotated
[ ] Compromised services can be isolated
[ ] Emergency network controls exist
Enter fullscreen mode Exit fullscreen mode

67.49 Reference Zero-Trust Architecture

A mature AI platform can use:

                         INTERNET
                            │
                            ▼
                     ┌─────────────┐
                     │ CDN / WAF   │
                     └──────┬──────┘
                            │
                           TLS
                            │
                            ▼
                     ┌─────────────┐
                     │ API Gateway │
                     └──────┬──────┘
                            │
                     Identity Check
                            │
                     Authorization
                            │
                            ▼
                  ┌──────────────────┐
                  │ Application Mesh │
                  └────────┬─────────┘
                           │
            ┌──────────────┼──────────────┐
            │              │              │
            ▼              ▼              ▼
          API           Workers       AI Service
            │              │              │
            │              │              │
            ▼              ▼              ▼
        Database        Queue        Egress Proxy
            │                             │
            ▼                             ▼
          Cache                     Approved APIs
            │
            ▼
       Object Storage
Enter fullscreen mode Exit fullscreen mode

Each major connection should have:

identity
+
encryption
+
authorization
+
network policy
+
monitoring
Enter fullscreen mode Exit fullscreen mode

67.50 Final Principle

Network security for AI systems is no longer simply:

"Put the servers behind a firewall."

A modern AI platform must assume that:

  • users can be malicious
  • files can be malicious
  • prompts can be malicious
  • AI outputs can be manipulated
  • workers can be compromised
  • internal services can fail
  • credentials can leak
  • agents can make incorrect decisions

Therefore, network controls must remain effective even when an application component behaves incorrectly.

The strongest architecture is:

Identity
   ↓
Authentication
   ↓
Authorization
   ↓
Encrypted Connection
   ↓
Network Segmentation
   ↓
Least-Privilege Access
   ↓
Egress Control
   ↓
Application Validation
   ↓
Monitoring
   ↓
Containment
Enter fullscreen mode Exit fullscreen mode

The key principle is:

Every network connection is an authorization decision, not merely a routing decision.

This becomes especially important when AI agents, external model providers, web tools, plugins and autonomous workflows are introduced.

The next logical chapter is:

Chapter 68 — Secure AI Secrets & Key Management: API Keys, Encryption Keys, KMS, Vault Architecture, Secret Rotation, Credential Isolation, Key Hierarchies, Envelope Encryption, HSMs, Provider Credentials & Emergency Revocation.

Top comments (0)