DEV Community

Cover image for CHAPTER 47 SECURE AI INFRASTRUCTURE & DEVSECOPS
Black Shadow Team ©
Black Shadow Team ©

Posted on

CHAPTER 47 SECURE AI INFRASTRUCTURE & DEVSECOPS

#ai

CHAPTER 47

SECURE AI INFRASTRUCTURE & DEVSECOPS

Cloud Architecture, Containers, Kubernetes, CI/CD, Supply-Chain Security, Runtime Protection, Disaster Recovery and Business Continuity

47.1 Introduction

A secure AI application cannot depend solely on secure prompts, secure APIs, or secure model behavior.

The infrastructure supporting the AI system is equally important.

A production AI platform may contain:

  • web applications,
  • API gateways,
  • authentication services,
  • databases,
  • object storage,
  • vector databases,
  • model gateways,
  • inference workers,
  • background workers,
  • document-processing pipelines,
  • monitoring systems,
  • logging systems,
  • container registries,
  • CI/CD systems,
  • secrets-management systems,
  • cloud infrastructure,
  • GPU or accelerator resources,
  • backup systems,
  • disaster-recovery environments.

Every additional component introduces another security boundary.

Therefore, AI infrastructure security should be treated as a continuous lifecycle rather than a single configuration task.

A useful model is:

Plan → Build → Verify → Deploy → Observe → Respond → Recover → Improve

This creates the foundation of a DevSecOps approach for AI systems.

47.2 What Is DevSecOps?

DevSecOps integrates security throughout the software-development and deployment lifecycle.

Traditional development can be represented as:

Development
    ↓
Testing
    ↓
Deployment
    ↓
Security Review
Enter fullscreen mode Exit fullscreen mode

A DevSecOps approach instead embeds security throughout:

Requirements
     ↓
Threat Modeling
     ↓
Secure Development
     ↓
Automated Security Testing
     ↓
Dependency Verification
     ↓
Build Verification
     ↓
Artifact Signing
     ↓
Deployment Controls
     ↓
Runtime Monitoring
     ↓
Incident Response
     ↓
Recovery
     ↓
Continuous Improvement
Enter fullscreen mode Exit fullscreen mode

For AI applications, additional AI-specific controls can be inserted into the lifecycle:

Model Evaluation
      ↓
Prompt/Policy Validation
      ↓
Model Artifact Verification
      ↓
Inference Security Testing
      ↓
AI Runtime Monitoring
Enter fullscreen mode Exit fullscreen mode

The objective is not to make security a final approval step.

The objective is to make security an automated and continuous property of the system.

47.3 Secure Cloud Architecture

A production AI platform should normally be divided into security zones.

A simplified architecture is:

                         INTERNET
                            │
                            ▼
                    ┌───────────────┐
                    │ CDN / WAF     │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │ API Gateway   │
                    └───────┬───────┘
                            │
              ┌─────────────┴─────────────┐
              │                           │
              ▼                           ▼
       ┌──────────────┐           ┌──────────────┐
       │ Web/API Tier │           │ Auth Service │
       └──────┬───────┘           └──────────────┘
              │
              ▼
       ┌──────────────┐
       │ Service Tier │
       └──────┬───────┘
              │
      ┌───────┼────────┐
      │       │        │
      ▼       ▼        ▼
   Database  Storage  AI Gateway
                         │
                ┌────────┼─────────┐
                │        │         │
                ▼        ▼         ▼
             Model    RAG       AI Worker
             Service  Service
Enter fullscreen mode Exit fullscreen mode

The central principle is:

Components should communicate only with the services they actually need.

A frontend should not directly access a production database.

An AI worker should not automatically have unrestricted access to object storage.

A background document-processing service should not automatically receive administrative credentials.

Infrastructure segmentation reduces the impact of compromised components.

47.4 Network Segmentation

Network segmentation divides infrastructure into logical security zones.

Example:

Public Zone
    │
    ├── CDN
    ├── WAF
    └── Load Balancer
          │
          ▼
Application Zone
    │
    ├── API
    ├── Web
    └── Authentication
          │
          ▼
Private Service Zone
    │
    ├── AI Gateway
    ├── Workers
    ├── RAG
    └── Processing Services
          │
          ▼
Data Zone
    │
    ├── PostgreSQL
    ├── Vector Database
    └── Object Storage
Enter fullscreen mode Exit fullscreen mode

The database should generally remain inaccessible directly from the public internet.

Likewise, internal AI services should generally be reachable through controlled service-to-service communication rather than arbitrary network access.

47.5 Zero-Trust Infrastructure

Zero-trust architecture assumes that network location alone does not establish trust.

A service should not receive access simply because:

“It is inside the private network.”

Instead, access should depend on:

  • identity,
  • authentication,
  • authorization,
  • service identity,
  • request context,
  • resource sensitivity,
  • policy,
  • logging,
  • continuous verification.

A useful conceptual model is:

Request
   │
   ▼
Authenticate
   │
   ▼
Identify Service/User
   │
   ▼
Evaluate Policy
   │
   ▼
Check Resource Permission
   │
   ▼
Allow / Deny
   │
   ▼
Audit
Enter fullscreen mode Exit fullscreen mode

This approach is particularly valuable for AI systems because AI workers often process sensitive information and interact with multiple internal services.

47.6 Infrastructure Identity

Every production service should have a distinct machine identity where practical.

For example:

api-service
ai-gateway
document-worker
embedding-worker
notification-service
backup-service
monitoring-service
Enter fullscreen mode Exit fullscreen mode

Avoid using one universal administrative credential.

Instead:

api-service
    ↓
API-specific permissions

document-worker
    ↓
Document-specific permissions

backup-service
    ↓
Backup-specific permissions
Enter fullscreen mode Exit fullscreen mode

This follows the principle of least privilege.

If one service is compromised, its credentials should not automatically provide control over the entire infrastructure.

47.7 Service Accounts

A service account represents an application or workload rather than a human.

Example conceptual permissions:

AI Gateway:
    read model configuration
    call approved inference services
    write inference metadata

Document Worker:
    read pending documents
    write processed chunks
    write processing status

Web API:
    read/write user-owned application data
    request AI jobs
Enter fullscreen mode Exit fullscreen mode

The permissions should be explicitly defined.

A useful policy representation is:

type ServicePermission =
  | "read:user_data"
  | "write:user_data"
  | "create:ai_job"
  | "read:document"
  | "write:document"
  | "invoke:model"
  | "read:model_config";
Enter fullscreen mode Exit fullscreen mode

47.8 Container Security

Containers provide application isolation and reproducible deployment environments.

A secure container lifecycle includes:

Source Code
    ↓
Dependency Installation
    ↓
Build
    ↓
Security Scan
    ↓
Minimal Image
    ↓
Image Signing
    ↓
Registry
    ↓
Deployment
    ↓
Runtime Monitoring
Enter fullscreen mode Exit fullscreen mode

Container images should be treated as software artifacts that require verification.

47.9 Minimal Container Images

Large container images often contain unnecessary packages and utilities.

A production image should contain only what the application requires.

Conceptually:

Bad:
Application
+ compiler
+ package manager
+ debugging tools
+ unnecessary utilities
+ development dependencies
+ source repository

Better:
Application
+ production dependencies
+ required runtime
Enter fullscreen mode Exit fullscreen mode

Reducing unnecessary software reduces the attack surface and simplifies vulnerability management.

47.10 Non-Root Containers

Where supported by the application, containers should avoid running the main process as root.

Conceptual Docker configuration:

FROM node:22-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev

COPY . .

RUN addgroup -S appgroup && adduser -S appuser -G appgroup

USER appuser

CMD ["node", "server.js"]
Enter fullscreen mode Exit fullscreen mode

The exact image and runtime should be selected according to the application's compatibility and security requirements.

47.11 Immutable Deployment Artifacts

Production environments should preferably deploy immutable artifacts.

Instead of:

Production
   ↓
Install latest dependencies
   ↓
Run application
Enter fullscreen mode Exit fullscreen mode

prefer:

Source
  ↓
Build
  ↓
Verified Artifact
  ↓
Registry
  ↓
Deployment
Enter fullscreen mode Exit fullscreen mode

The deployed artifact should correspond to a known build.

This improves:

  • reproducibility,
  • rollback,
  • auditability,
  • incident investigation,
  • supply-chain integrity.

47.12 Software Supply-Chain Security

Modern applications depend on external packages.

An AI application may depend on:

  • JavaScript packages,
  • Python packages,
  • operating-system packages,
  • model libraries,
  • SDKs,
  • database drivers,
  • cloud libraries,
  • container base images.

Each dependency introduces supply-chain risk.

A secure dependency lifecycle is:

Dependency Selection
       ↓
Version Pinning
       ↓
Vulnerability Scanning
       ↓
License Review
       ↓
Build Verification
       ↓
SBOM Generation
       ↓
Artifact Signing
       ↓
Deployment
Enter fullscreen mode Exit fullscreen mode

47.13 Dependency Pinning

Avoid uncontrolled dependency updates in production.

Instead of relying indefinitely on floating versions, use controlled dependency versions and lockfiles.

For example:

package.json
package-lock.json
Enter fullscreen mode Exit fullscreen mode

or equivalent mechanisms for other ecosystems.

Automated dependency updates can then be evaluated through CI rather than silently changing production software.

47.14 Software Bill of Materials

An SBOM, or Software Bill of Materials, describes software components contained within an application or artifact.

Conceptually:

Application
│
├── Framework
├── Database Driver
├── Authentication Library
├── AI SDK
├── HTTP Library
├── Cryptography Library
└── Operating-System Packages
Enter fullscreen mode Exit fullscreen mode

An SBOM helps organizations answer questions such as:

  • Which dependencies are present?
  • Which version is deployed?
  • Which applications use a vulnerable component?
  • Which artifact contains a particular package?

For large AI systems, this becomes increasingly important because the dependency graph can be complex.

47.15 Artifact Integrity

Production artifacts should have verifiable identity.

Conceptually:

Build
  │
  ▼
Artifact
  │
  ▼
Digest
  │
  ▼
Signature
  │
  ▼
Registry
  │
  ▼
Deployment Verification
Enter fullscreen mode Exit fullscreen mode

The deployment system can verify that the artifact being deployed corresponds to an approved build.

47.16 CI/CD Security

A secure CI/CD pipeline can be structured as:

Pull Request
     ↓
Lint
     ↓
Unit Tests
     ↓
Type Check
     ↓
Dependency Audit
     ↓
Secret Detection
     ↓
SAST
     ↓
Build
     ↓
Container Scan
     ↓
SBOM
     ↓
Artifact Signing
     ↓
Integration Tests
     ↓
Approval
     ↓
Deployment
Enter fullscreen mode Exit fullscreen mode

Not every control must execute at exactly the same stage.

The important principle is that deployment should depend on successful verification.

47.17 Secret Detection in CI

Secrets should never be intentionally committed to source control.

Examples include:

  • API keys,
  • database passwords,
  • signing keys,
  • service credentials,
  • cloud access credentials.

A CI system should detect suspicious secret patterns before code reaches production.

Example policy:

If secret detected:
    fail build
    prevent deployment
    notify responsible team
    rotate exposed credential if necessary
Enter fullscreen mode Exit fullscreen mode

Detection alone is insufficient if a real credential has already been exposed.

Credential rotation may also be required.

47.18 Secrets Management

Secrets should be separated from application source code.

Conceptual architecture:

Application
     │
     ▼
Identity
     │
     ▼
Secrets Manager
     │
     ├── Database Credential
     ├── AI Provider Credential
     ├── Signing Key
     └── Internal Service Credential
Enter fullscreen mode Exit fullscreen mode

Applications should receive only the secrets they require.

47.19 Secret Rotation

Secrets should have controlled lifecycle management.

A useful model is:

Create
  ↓
Activate
  ↓
Use
  ↓
Rotate
  ↓
Grace Period
  ↓
Revoke
  ↓
Destroy
Enter fullscreen mode Exit fullscreen mode

Rotation policies should consider:

  • credential type,
  • application behavior,
  • provider capabilities,
  • operational impact,
  • emergency revocation requirements.

A rotation system should also be tested.

A theoretical rotation interface might look like:

interface SecretRotationService {
  createVersion(secretName: string): Promise<string>;
  activateVersion(secretName: string, version: string): Promise<void>;
  revokeVersion(secretName: string, version: string): Promise<void>;
}
Enter fullscreen mode Exit fullscreen mode

47.20 Kubernetes Security

Kubernetes can provide powerful orchestration capabilities, but it also introduces substantial configuration complexity.

Important security areas include:

  • namespaces,
  • RBAC,
  • service accounts,
  • network policies,
  • pod security,
  • secret handling,
  • image policies,
  • admission controls,
  • resource limits,
  • logging,
  • monitoring.

A conceptual architecture:

Cluster
│
├── Ingress Namespace
│
├── Application Namespace
│
├── AI Namespace
│
├── Worker Namespace
│
├── Monitoring Namespace
│
└── Security/Operations Namespace
Enter fullscreen mode Exit fullscreen mode

Logical separation helps reduce accidental access between workloads.

47.21 Kubernetes RBAC

Role-Based Access Control should grant only the permissions required by a workload.

Conceptually:

Service Account
      ↓
Role
      ↓
Allowed Resources
      ↓
Allowed Operations
Enter fullscreen mode Exit fullscreen mode

For example:

document-worker
    ├── read documents
    ├── write processing status
    └── cannot administer cluster
Enter fullscreen mode Exit fullscreen mode

Avoid granting broad cluster-administrator privileges to ordinary application workloads.

47.22 Kubernetes Network Policies

Network policies can restrict which workloads communicate.

Conceptual policy:

API
 │
 ├──→ Auth Service
 ├──→ AI Gateway
 └──→ Database

Document Worker
 │
 ├──→ Object Storage
 ├──→ Embedding Service
 └──→ Vector Database
Enter fullscreen mode Exit fullscreen mode

Unnecessary paths should remain blocked.

This creates defense in depth even when application-level authorization fails.

47.23 Resource Limits

AI workloads can consume substantial computational resources.

Without appropriate limits, one workload can negatively affect other workloads.

Resource controls may include:

CPU limit
Memory limit
GPU allocation
Concurrent job limit
Queue depth
Request timeout
Maximum document size
Maximum generation duration
Enter fullscreen mode Exit fullscreen mode

This is both a reliability and security control.

47.24 AI Workload Isolation

AI workloads should be treated as potentially high-resource and high-risk workloads.

A secure architecture can separate:

General Application
       │
       ▼
AI Gateway
       │
       ▼
Controlled AI Queue
       │
       ▼
Isolated AI Worker
       │
       ▼
Model Runtime
Enter fullscreen mode Exit fullscreen mode

The worker should not automatically receive unrestricted access to:

  • databases,
  • internal networks,
  • user accounts,
  • cloud administration,
  • unrelated services.

47.25 Model Artifact Security

AI systems may download or store:

  • model weights,
  • tokenizers,
  • configuration files,
  • adapters,
  • embeddings,
  • preprocessing assets.

These artifacts should be treated as software supply-chain components.

A secure model-artifact lifecycle is:

Model Source
     ↓
Verification
     ↓
Security Evaluation
     ↓
Artifact Registry
     ↓
Integrity Check
     ↓
Controlled Deployment
     ↓
Runtime Monitoring
Enter fullscreen mode Exit fullscreen mode

A model file should not automatically be considered trustworthy simply because it has a familiar filename.

47.26 External AI Provider Isolation

If the application uses external AI providers, external calls should ideally pass through a controlled AI gateway.

Application
    │
    ▼
AI Gateway
    │
    ├── Policy
    ├── Privacy Check
    ├── Provider Selection
    ├── Rate Limit
    ├── Logging
    └── Cost Control
          │
          ▼
External Provider
Enter fullscreen mode Exit fullscreen mode

This provides a centralized security boundary.

The gateway can also prevent individual application components from directly storing provider credentials.

47.27 API Gateway Security

An API gateway can provide:

  • authentication,
  • authorization,
  • request validation,
  • rate limiting,
  • request-size limits,
  • timeout enforcement,
  • routing,
  • observability.

Example:

Client
  ↓
TLS
  ↓
WAF
  ↓
API Gateway
  ↓
Authentication
  ↓
Authorization
  ↓
Request Validation
  ↓
Application
Enter fullscreen mode Exit fullscreen mode

47.28 Rate Limiting

Rate limiting protects both availability and cost.

For an AI platform, controls can operate at multiple levels:

IP limit
User limit
Organization limit
API-key limit
Endpoint limit
AI-model limit
Daily quota
Concurrent-job limit
Enter fullscreen mode Exit fullscreen mode

A useful conceptual policy is:

request
  ↓
identify principal
  ↓
check quota
  ↓
check concurrency
  ↓
allow / reject
Enter fullscreen mode Exit fullscreen mode

AI workloads are particularly sensitive to uncontrolled usage because inference may be computationally expensive.

47.29 Runtime Security

Security does not end when deployment succeeds.

Runtime monitoring should observe:

  • unusual traffic,
  • authentication failures,
  • authorization failures,
  • unexpected resource consumption,
  • abnormal API usage,
  • unusual service-to-service communication,
  • repeated policy violations,
  • suspicious model behavior,
  • system health.

A useful monitoring pipeline:

Applications
    │
    ├── Logs
    ├── Metrics
    └── Traces
          │
          ▼
    Observability Platform
          │
          ▼
    Detection Rules
          │
          ▼
    Alerting
          │
          ▼
    Incident Response
Enter fullscreen mode Exit fullscreen mode

47.30 Security Logging

Security logs should answer:

Who did what, when, from where, against which resource, and what happened?

For example:

{
  "event": "ai_job_created",
  "actorId": "user-or-service-id",
  "resourceId": "job-id",
  "timestamp": "2026-01-01T00:00:00Z",
  "result": "accepted"
}
Enter fullscreen mode Exit fullscreen mode

Sensitive payloads should not automatically be written to logs.

Logs should themselves follow privacy and retention policies.

47.31 Observability

Three major observability categories are:

Logs
Metrics
Traces
Enter fullscreen mode Exit fullscreen mode

Logs

Provide event details.

Metrics

Provide numerical system behavior.

Examples:

request_count
error_rate
latency
queue_depth
cpu_usage
memory_usage
gpu_usage
Enter fullscreen mode Exit fullscreen mode

Traces

Show how a request moves through multiple services.

Example:

User Request
    ↓
API
    ↓
Auth
    ↓
AI Gateway
    ↓
Queue
    ↓
Worker
    ↓
Model
    ↓
Storage
Enter fullscreen mode Exit fullscreen mode

47.32 Health Checks

Production services should expose appropriate health information.

Two common concepts are:

Liveness
Readiness
Enter fullscreen mode Exit fullscreen mode

Liveness asks:

Is the process alive?

Readiness asks:

Is the service currently capable of receiving traffic?

Example:

GET /health/live
GET /health/ready
Enter fullscreen mode Exit fullscreen mode

A service may be alive but not ready because its database connection or required dependency is unavailable.

47.33 Secure Deployment Strategies

Several deployment strategies can reduce production risk.

Rolling deployment

Gradually replace old instances.

Blue/green deployment

Maintain two environments:

Blue  → Current
Green → New
Enter fullscreen mode Exit fullscreen mode

Traffic can move to the new environment after verification.

Canary deployment

Release the new version to a small percentage of traffic first.

95% → Old version
5%  → New version
Enter fullscreen mode Exit fullscreen mode

If the new version behaves correctly, the percentage can gradually increase.

These approaches improve rollback capability.

47.34 Rollback

Every production deployment should have a rollback strategy.

Conceptually:

Version N
   ↓
Deploy N+1
   ↓
Observe
   │
   ├── Healthy → Continue
   │
   └── Unhealthy
          ↓
       Rollback
          ↓
       Version N
Enter fullscreen mode Exit fullscreen mode

Rollback artifacts should remain available long enough to support realistic incident-response requirements.

47.35 Infrastructure as Code

Infrastructure should preferably be represented as code rather than manually configured wherever practical.

Examples of infrastructure-as-code concepts include:

Network
Database
Storage
Compute
IAM
Monitoring
Secrets references
Deployment configuration
Enter fullscreen mode Exit fullscreen mode

Advantages include:

  • reproducibility,
  • version control,
  • review,
  • auditing,
  • automated deployment,
  • easier disaster recovery.

47.36 Environment Separation

Production and development systems should be separated.

A typical structure:

Development
     │
     ▼
Testing
     │
     ▼
Staging
     │
     ▼
Production
Enter fullscreen mode Exit fullscreen mode

Production credentials should not be copied into development environments.

Production user data should not automatically be replicated into developer machines.

47.37 Disaster Recovery

Disaster recovery addresses major failures such as:

  • infrastructure outages,
  • accidental deletion,
  • database corruption,
  • storage failure,
  • deployment failures,
  • regional outages,
  • security incidents.

A basic disaster-recovery lifecycle:

Detect
  ↓
Contain
  ↓
Assess
  ↓
Restore
  ↓
Validate
  ↓
Resume
  ↓
Investigate
Enter fullscreen mode Exit fullscreen mode

47.38 Backup Strategy

Important data may include:

Database
Object storage
Application configuration
Infrastructure configuration
Audit records
Critical model artifacts
Enter fullscreen mode Exit fullscreen mode

Backups should be:

  • automated,
  • monitored,
  • access-controlled,
  • encrypted where appropriate,
  • tested through restoration exercises.

A backup that has never been restored should not automatically be considered reliable.

47.39 Recovery Point Objective

RPO represents how much data loss the organization can tolerate after a failure.

Example:

RPO = 15 minutes
Enter fullscreen mode Exit fullscreen mode

Conceptually, the recovery process should aim to restore the system to a state no more than approximately 15 minutes behind the desired recovery point.

The appropriate value depends on business requirements.

47.40 Recovery Time Objective

RTO represents the target amount of time required to restore service.

Example:

RTO = 1 hour
Enter fullscreen mode Exit fullscreen mode

Again, this is a business and system requirement rather than a universal technical value.

RPO and RTO should be explicitly defined for critical services.

47.41 Business Continuity

Business continuity extends beyond infrastructure recovery.

It asks:

How does the organization continue operating when one or more systems are unavailable?

An AI platform may require degraded-service modes.

For example:

Normal
  ↓
External AI unavailable
  ↓
Fallback model
  ↓
Reduced functionality
  ↓
Manual processing
Enter fullscreen mode Exit fullscreen mode

This is often more resilient than simply displaying a complete outage.

47.42 Graceful Degradation

AI systems should be designed to fail gracefully.

Example:

Primary AI Provider
       │
       X unavailable
       │
       ▼
Fallback Provider
       │
       X unavailable
       │
       ▼
Local/Reduced Capability
       │
       X unavailable
       │
       ▼
Safe Error Response
Enter fullscreen mode Exit fullscreen mode

A degraded system should never bypass security controls merely to remain operational.

47.43 Queue-Based Resilience

Long-running AI tasks should often be processed asynchronously.

Instead of:

Client
  ↓
Wait for 5-minute AI operation
Enter fullscreen mode Exit fullscreen mode

use:

Client
  ↓
Create Job
  ↓
Queue
  ↓
Worker
  ↓
AI Processing
  ↓
Result
Enter fullscreen mode Exit fullscreen mode

This improves:

  • reliability,
  • retry handling,
  • scalability,
  • resource management,
  • user experience.

47.44 Retry Policies

Retries should be carefully controlled.

A poorly designed retry system can amplify an outage.

For example:

Provider failure
      ↓
100 workers retry
      ↓
Provider receives 100 additional requests
      ↓
Failure becomes worse
Enter fullscreen mode Exit fullscreen mode

A safer strategy can include:

  • exponential backoff,
  • bounded retries,
  • jitter,
  • circuit breakers,
  • maximum queue duration.

47.45 Circuit Breakers

A circuit breaker temporarily stops requests to an unhealthy dependency.

Conceptually:

Healthy
   ↓
Failures increase
   ↓
Open circuit
   ↓
Stop unnecessary calls
   ↓
Wait
   ↓
Test dependency
   ↓
Recover
Enter fullscreen mode Exit fullscreen mode

This protects the rest of the system from cascading failure.

47.46 AI Cost Resilience

Infrastructure security also includes protection against uncontrolled AI spending.

Controls may include:

Per-user quota
Per-organization quota
Model-specific limits
Maximum token budget
Maximum image resolution
Maximum video duration
Concurrent generation limit
Daily spending threshold
Enter fullscreen mode Exit fullscreen mode

A request should be evaluated before expensive processing begins.

47.47 GPU and Accelerator Security

AI workloads may use GPUs or other accelerators.

Security considerations include:

  • workload isolation,
  • resource allocation,
  • driver management,
  • container compatibility,
  • monitoring,
  • memory/resource exhaustion controls,
  • secure artifact management.

The exact architecture depends heavily on whether inference occurs:

  • locally,
  • on dedicated servers,
  • in a cloud GPU environment,
  • through a managed AI provider.

47.48 Incident Response

A secure infrastructure program requires a documented incident-response process.

A high-level lifecycle is:

Preparation
    ↓
Detection
    ↓
Analysis
    ↓
Containment
    ↓
Eradication
    ↓
Recovery
    ↓
Lessons Learned
Enter fullscreen mode Exit fullscreen mode

The response process should identify:

  • responsible personnel,
  • escalation paths,
  • communication channels,
  • evidence-preservation requirements,
  • recovery procedures,
  • legal/compliance responsibilities where applicable.

47.49 Security Incident Example

Consider a hypothetical production credential exposure.

The safe response sequence is:

Detection
   ↓
Confirm exposure
   ↓
Restrict affected credential
   ↓
Rotate credential
   ↓
Review access logs
   ↓
Determine affected resources
   ↓
Contain additional access
   ↓
Restore secure configuration
   ↓
Document incident
   ↓
Improve preventive controls
Enter fullscreen mode Exit fullscreen mode

The objective is not merely to replace the credential.

The organization must also determine whether the credential was actually used.

47.50 Evidence Preservation

During incidents, relevant evidence may include:

  • authentication logs,
  • API logs,
  • deployment records,
  • cloud audit logs,
  • container metadata,
  • configuration changes,
  • security alerts.

Evidence handling should be controlled and privacy-aware.

Do not collect more sensitive information than necessary.

47.51 Security Testing of Infrastructure

Infrastructure testing should cover multiple layers:

Application
     ↓
API
     ↓
Container
     ↓
Kubernetes
     ↓
Network
     ↓
Identity
     ↓
Cloud Configuration
     ↓
Recovery
Enter fullscreen mode Exit fullscreen mode

Testing should be authorized and conducted in controlled environments.

The goal is to identify weaknesses before they become production incidents.

47.52 Production Security Checklist

A production AI infrastructure should evaluate at least:

Identity

  • [ ] MFA for privileged human accounts
  • [ ] least-privilege service identities
  • [ ] separate production credentials
  • [ ] credential rotation
  • [ ] privileged-access monitoring

Network

  • [ ] TLS
  • [ ] private data services
  • [ ] segmentation
  • [ ] restricted service-to-service traffic
  • [ ] controlled ingress

Containers

  • [ ] minimal images
  • [ ] non-root execution where practical
  • [ ] vulnerability scanning
  • [ ] signed artifacts
  • [ ] controlled registries

CI/CD

  • [ ] dependency scanning
  • [ ] secret detection
  • [ ] static analysis
  • [ ] automated tests
  • [ ] artifact verification
  • [ ] deployment approvals where appropriate

Kubernetes

  • [ ] RBAC
  • [ ] network policies
  • [ ] workload isolation
  • [ ] resource limits
  • [ ] controlled service accounts
  • [ ] admission/security controls

AI

  • [ ] model artifact verification
  • [ ] provider isolation
  • [ ] AI gateway
  • [ ] inference quotas
  • [ ] model monitoring
  • [ ] safe fallback behavior

Data

  • [ ] encrypted storage
  • [ ] access control
  • [ ] backup
  • [ ] retention policy
  • [ ] restoration testing

Resilience

  • [ ] documented RPO
  • [ ] documented RTO
  • [ ] tested disaster recovery
  • [ ] rollback procedure
  • [ ] graceful degradation
  • [ ] incident-response plan

47.53 Reference Secure AI Infrastructure

A complete conceptual architecture can therefore be represented as:

                         USERS
                           │
                           ▼
                    ┌─────────────┐
                    │ CDN / WAF   │
                    └──────┬──────┘
                           │
                           ▼
                    ┌─────────────┐
                    │ API Gateway │
                    └──────┬──────┘
                           │
                 ┌─────────┴─────────┐
                 │                   │
                 ▼                   ▼
          ┌────────────┐      ┌────────────┐
          │ Application│      │ Auth       │
          │ Services   │      │ Service    │
          └─────┬──────┘      └────────────┘
                │
                ▼
          ┌────────────┐
          │ AI Gateway │
          └─────┬──────┘
                │
        ┌───────┼────────┐
        │       │        │
        ▼       ▼        ▼
      Queue    RAG    Provider
        │       │       Gateway
        ▼       ▼
      Workers  Vector DB
        │
        ▼
   Isolated Model Runtime
        │
        ▼
   Controlled AI Providers


        ┌─────────────────────────┐
        │       DATA LAYER        │
        │                         │
        │ PostgreSQL              │
        │ Object Storage          │
        │ Vector Database         │
        │ Backup                  │
        └─────────────────────────┘


        ┌─────────────────────────┐
        │     SECURITY LAYER      │
        │                         │
        │ IAM                     │
        │ Secrets Manager         │
        │ Audit Logs              │
        │ Monitoring              │
        │ SIEM/Alerting           │
        └─────────────────────────┘


        ┌─────────────────────────┐
        │      DEVSECOPS          │
        │                         │
        │ Git                     │
        │ CI/CD                   │
        │ SAST                    │
        │ Dependency Scan         │
        │ Container Scan          │
        │ SBOM                    │
        │ Artifact Signing        │
        └─────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

47.54 Security Principles

The infrastructure architecture can be reduced to several principles:

Principle 1 — Least privilege

Give every identity only the access required.

Principle 2 — Assume breach

Design systems so that compromise of one component does not automatically compromise everything.

Principle 3 — Verify artifacts

Do not blindly trust software or model artifacts.

Principle 4 — Segment infrastructure

Separate public, application, AI, and data workloads where practical.

Principle 5 — Automate security

Move security checks into CI/CD.

Principle 6 — Monitor continuously

Production security requires continuous observation.

Principle 7 — Prepare for failure

Backups, rollback, failover, and disaster recovery must be designed before incidents happen.

Principle 8 — Test recovery

A recovery plan that exists only on paper is insufficient.

Principle 9 — Protect secrets

Credentials should be isolated, monitored, rotated, and revoked when necessary.

Principle 10 — Preserve safe degradation

Failure should not cause the application to bypass authorization or privacy controls.

47.55 Final Architecture Principle

The most important lesson of secure AI infrastructure is that security is not a single component.

It is a system property.

A secure AI platform should therefore combine:

Secure Identity
      +
Secure Network
      +
Secure Application
      +
Secure Containers
      +
Secure Supply Chain
      +
Secure AI Runtime
      +
Secure Data
      +
Secure Observability
      +
Secure Recovery
      =
Resilient AI Infrastructure
Enter fullscreen mode Exit fullscreen mode

The infrastructure layer provides the foundation upon which the application, AI model, data governance, privacy controls, and safety systems operate.

Without that foundation, even well-designed AI safety policies can be undermined by insecure deployment, excessive privileges, exposed credentials, compromised dependencies, weak isolation, or poor recovery capabilities.

The goal of DevSecOps is therefore not simply to prevent vulnerabilities.

The broader goal is to build a system that can:

resist failure, detect abnormal behavior, contain incidents, recover safely, and continuously improve.

END OF CHAPTER 47

Implementation snippets

  1. Secure service permission model

export type ServicePermission =
| "read:user_data"
| "write:user_data"
| "create:ai_job"
| "read:document"
| "write:document"
| "invoke:model"
| "read:model_config";

export interface ServiceIdentity {
serviceId: string;
permissions: ServicePermission[];
environment: "development" | "staging" | "production";
}

export function hasPermission(
identity: ServiceIdentity,
permission: ServicePermission
): boolean {
return identity.permissions.includes(permission);
}

  1. Basic readiness endpoint

export async function GET() {
const databaseReady = await checkDatabase();
const queueReady = await checkQueue();

if (!databaseReady || !queueReady) {
return Response.json(
{
status: "not_ready",
database: databaseReady,
queue: queueReady,
},
{ status: 503 }
);
}

return Response.json({
status: "ready",
});
}

  1. Controlled AI request policy

interface AIRequestPolicy {
maxConcurrentJobs: number;
maxInputSize: number;
allowedModels: string[];
}

export function validateAIRequest(
model: string,
inputSize: number,
policy: AIRequestPolicy
): boolean {
if (!policy.allowedModels.includes(model)) {
return false;
}

if (inputSize > policy.maxInputSize) {
return false;
}

return true;
}

  1. Conceptual CI security pipeline

steps:

  • checkout

  • install_dependencies

  • lint

  • type_check

  • unit_tests

  • dependency_audit

  • secret_detection

  • static_security_analysis

  • build

  • container_scan

  • generate_sbom

  • sign_artifact

  • integration_tests

  • deploy

  1. Simple retry policy

export async function retryWithBackoff(
operation: () => Promise,
maxAttempts = 3
): Promise {
let lastError: unknown;

for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return await operation();
} catch (error) {
lastError = error;

  if (attempt === maxAttempts) {
    break;
  }

  const delay = Math.min(1000 * 2 ** (attempt - 1), 10_000);
  await new Promise((resolve) => setTimeout(resolve, delay));
}
Enter fullscreen mode Exit fullscreen mode

}

throw lastError;
}

  1. Backup policy model

interface BackupPolicy {
database: boolean;
objectStorage: boolean;
configuration: boolean;
retentionDays: number;
restorationTestIntervalDays: number;
}

const productionBackupPolicy: BackupPolicy = {
database: true,
objectStorage: true,
configuration: true,
retentionDays: 30,
restorationTestIntervalDays: 30,
};

Top comments (0)