DEV Community

Cover image for Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability
Sushyam Nagallapati
Sushyam Nagallapati

Posted on

Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability

#kubernetes #powerplatform #devops #ai

Thanks for taking the time to read. If you’ve worked on AI platforms, cloud infrastructure, or platform engineering, I’d love to hear how your architecture differs in the comments.

In Part 4 of AI Infrastructure for Cloud Engineers, we looked at FinOps for AI and how GPU utilization, token consumption, model choice, and inference volume affect cost.

Read Part 4: FinOps for AI: Understanding GPU, Token, and Inference Costs

So far, we have looked at individual parts of AI infrastructure:

Kubernetes
GPUs
Scheduling
Model Serving
Observability
FinOps
Enter fullscreen mode Exit fullscreen mode

But production systems rarely operate as separate pieces.

The real challenge is bringing them together into a platform that developers can deploy to, operators can understand, security teams can govern, and businesses can afford to run.

That is what this final article is about.

What Does a Production AI Platform Look Like?

A simplified architecture might look like this:

Developer
    ↓
Git Repository
    ↓
CI Pipeline
    ↓
Container Registry
    ↓
GitOps Repository
    ↓
Kubernetes
    │
    ├── AI Applications
    ├── Model Servers
    ├── GPU Workloads
    ├── Vector Services
    └── AI Agents
    │
    ↓
Observability + Security + FinOps
Enter fullscreen mode Exit fullscreen mode

The model is only one component.

A production AI platform also needs:

  • Reproducible infrastructure
  • Controlled deployments
  • Secure identities and secrets
  • GPU scheduling
  • Autoscaling
  • Observability
  • Cost visibility
  • Rollback and recovery

The goal is not simply to make an AI application run.

The goal is to make it repeatable, secure, observable, scalable, and recoverable.

1. Start With Infrastructure as Code

Creating infrastructure manually might work for an experiment.

Production needs something reproducible.

Instead of engineers manually creating:

Kubernetes Cluster
GPU Node Pool
Network
Storage
Identity
Secrets Integration
Monitoring
Enter fullscreen mode Exit fullscreen mode

the infrastructure should be defined as code.

Conceptually:

Infrastructure Code
       ↓
Review
       ↓
Plan
       ↓
Apply
       ↓
Cloud Infrastructure
Enter fullscreen mode Exit fullscreen mode

This makes infrastructure:

  • Version controlled
  • Reviewable
  • Repeatable
  • Easier to recreate
  • Easier to audit

A typical repository might look like:

infrastructure/
├── network/
├── kubernetes/
├── gpu-nodes/
├── identity/
├── monitoring/
└── environments/
    ├── dev/
    ├── staging/
    └── production/
Enter fullscreen mode Exit fullscreen mode

The important principle is not the specific IaC tool.

It is that infrastructure changes follow the same engineering discipline as application changes.

2. Use Kubernetes as the Runtime Layer

Kubernetes becomes the runtime where the AI platform operates.

It might host:

AI API
Model Server
Embedding Service
Vector Search
AI Agents
Background Workers
GPU Workloads
Enter fullscreen mode Exit fullscreen mode

For example:

                    Kubernetes
                        │
        ┌───────────────┼───────────────┐
        ↓               ↓               ↓
   AI Services      Model Servers    AI Agents
        │               │               │
        └───────────────┼───────────────┘
                        ↓
                    GPU Pool
Enter fullscreen mode Exit fullscreen mode

This gives the platform a common way to manage:

Scheduling
Scaling
Networking
Health checks
Rollouts
Configuration
Resource allocation
Enter fullscreen mode Exit fullscreen mode

Kubernetes adoption for AI is already moving in this direction. CNCF's 2025 survey found that 66% of organizations hosting generative AI models use Kubernetes for at least some inference workloads.

The interesting part is that AI infrastructure begins to look less like a collection of individual servers and more like a shared platform.

3. Separate CI From Deployment

A useful pattern is to separate building software from deploying software.

The CI pipeline can handle:

Code
 ↓
Tests
 ↓
Security Scan
 ↓
Container Build
 ↓
Container Registry
Enter fullscreen mode Exit fullscreen mode

Deployment can then be handled separately through GitOps.

Container Registry
       ↓
Deployment Configuration
       ↓
Git Repository
       ↓
GitOps Controller
       ↓
Kubernetes
Enter fullscreen mode Exit fullscreen mode

Why separate them?

Because the CI system does not need broad credentials to modify production clusters directly.

Instead, production configuration lives in Git.

A change becomes something like:

Pull Request
     ↓
Review
     ↓
Merge
     ↓
GitOps Reconciliation
     ↓
Deployment
Enter fullscreen mode Exit fullscreen mode

Kubernetes itself recommends declarative, version-controlled configuration for production workloads, which also fits naturally with GitOps workflows.

4. Git Becomes the Desired State

Imagine production currently runs:

model-version: v12
replicas: 4
Enter fullscreen mode Exit fullscreen mode

A new release requires:

model-version: v13
replicas: 6
Enter fullscreen mode Exit fullscreen mode

Instead of manually changing the cluster, the team updates the configuration in Git.

model:
  version: v13

replicas: 6
Enter fullscreen mode Exit fullscreen mode

The GitOps controller compares:

Desired State in Git
        ↓
Actual State in Cluster
Enter fullscreen mode Exit fullscreen mode

and reconciles the difference.

That gives teams:

Change history
Code review
Rollback
Auditability
Environment consistency
Enter fullscreen mode Exit fullscreen mode

If something goes wrong, reverting the Git change can restore the previous desired configuration.

This becomes particularly useful for AI systems where changes may involve:

Model version
Prompt configuration
Resource limits
GPU requirements
Inference replicas
Routing policies
Enter fullscreen mode Exit fullscreen mode

5. Treat Security as a Platform Capability

Security should not be added after deployment.

It should exist throughout the platform.

A request path might look like:

User
 ↓
Authentication
 ↓
API Gateway
 ↓
AI Application
 ↓
Authorized Tool / Model
 ↓
Protected Resource
Enter fullscreen mode Exit fullscreen mode

Important controls include:

  • Workload identities
  • Role-based access
  • Network policies
  • Secrets management
  • Container scanning
  • Image provenance
  • Least privilege
  • Audit logging
  • Policy enforcement

The model should never become the security boundary.

If an AI agent requests access to a database, API, or production tool, the infrastructure still needs to verify whether that operation is allowed.

A useful principle is:

AI decides what it wants to do. The platform decides what it is allowed to do.

This becomes especially important as AI agents begin interacting directly with operational infrastructure.

6. Keep Secrets Outside the Application

AI applications may need credentials for:

Model providers
Databases
Vector stores
External APIs
Cloud services
MCP servers
Enter fullscreen mode Exit fullscreen mode

These should not appear inside:

Source code
Container images
Git repositories
Application logs
Enter fullscreen mode Exit fullscreen mode

Instead:

Secrets Manager
      ↓
Workload Identity
      ↓
Application
Enter fullscreen mode Exit fullscreen mode

Where possible, workload identity is preferable to long-lived static credentials.

If credentials are required, they should have:

  • Limited permissions
  • Defined ownership
  • Rotation
  • Expiration
  • Auditing

The same principle applies to development, staging, and production.

Each environment should have its own trust boundary.

7. Make Observability Part of the Platform

In Part 3, we looked at AI observability in detail.

At platform level, we want a common telemetry path.

Applications
GPU Nodes
Model Servers
AI Agents
     │
     ├── Metrics
     ├── Logs
     └── Traces
     │
     ↓
OpenTelemetry / Exporters
     ↓
Observability Platform
Enter fullscreen mode Exit fullscreen mode

OpenTelemetry provides Kubernetes tooling for collectors, operators, and workload instrumentation, making it useful as a common telemetry layer.

A production AI platform should let engineers move from:

User says AI is slow
Enter fullscreen mode Exit fullscreen mode

to:

Request ID
    ↓
API trace
    ↓
Model inference latency
    ↓
GPU saturation
    ↓
Growing queue depth
Enter fullscreen mode Exit fullscreen mode

without searching through five unrelated systems.

The platform should make diagnosis easier by default.

8. Connect Reliability With AI Metrics

Traditional SRE signals still matter:

Availability
Latency
Errors
Traffic
Enter fullscreen mode Exit fullscreen mode

AI adds another layer:

Time to first token
Tokens per second
Queue depth
GPU utilization
Model errors
Tool-call failures
Provider latency
Enter fullscreen mode Exit fullscreen mode

A service might look healthy at Kubernetes level:

Pods:      Healthy
CPU:       Normal
Memory:    Normal
Enter fullscreen mode Exit fullscreen mode

while users experience:

Queue:     Growing
TTFT:      Increasing
GPU:       Saturated
Enter fullscreen mode Exit fullscreen mode

Production readiness means connecting both views.

9. Add FinOps to the Architecture

Cost should not live in a completely separate dashboard owned only by finance.

The platform already knows:

GPU utilization
GPU hours
Requests
Tokens
Models
Tenants
Workloads
Enter fullscreen mode Exit fullscreen mode

Those signals can be connected to cost.

AI Workload
     ↓
Resource Usage
     ↓
Cost Allocation
     ↓
Team / Tenant / Product
Enter fullscreen mode Exit fullscreen mode

For example:

Workload: document-summary
Model: model-a
GPU Hours: 420
Requests: 180,000
Cost / Request: $0.018
Enter fullscreen mode Exit fullscreen mode

Now engineering teams can make better decisions about:

Scaling
Model choice
Prompt size
Caching
GPU capacity
Enter fullscreen mode Exit fullscreen mode

FinOps becomes part of platform engineering rather than something reviewed only when the cloud bill arrives.

10. Build a Safe Deployment Path

A production AI release should move through controlled stages.

Developer
    ↓
Pull Request
    ↓
Tests
    ↓
Security Checks
    ↓
Build Image
    ↓
Deploy to Staging
    ↓
Validation
    ↓
Production Approval
    ↓
GitOps Deployment
Enter fullscreen mode Exit fullscreen mode

Validation may include:

Unit tests
Integration tests
Model evaluations
Security tests
Smoke tests
Performance tests
Enter fullscreen mode Exit fullscreen mode

AI adds an important distinction.

The service may deploy successfully while the model behaves worse.

So release validation should consider both:

Infrastructure Health
        +
AI Quality
Enter fullscreen mode Exit fullscreen mode

A technically healthy deployment is not necessarily a successful AI release.

11. Design Rollback Before You Need It

Imagine model version v13 increases latency or produces worse results.

Production should not depend on someone remembering a long sequence of commands.

If configuration is version controlled:

v12
 ↓
v13
 ↓
Problem detected
 ↓
Revert
 ↓
v12
Enter fullscreen mode Exit fullscreen mode

Rollback becomes part of the deployment design.

The same applies to:

Container versions
Prompt configurations
Routing policies
Resource limits
Model versions
Enter fullscreen mode Exit fullscreen mode

Recovery should be tested before an incident occurs.

Putting Everything Together

Now the complete architecture starts to look like this:

                    Developer
                        ↓
                  Git Repository
                        ↓
                  CI / Validation
                        ↓
                 Container Registry
                        ↓
               GitOps Configuration
                        ↓
                GitOps Controller
                        ↓
                    Kubernetes
          ┌─────────────┼─────────────┐
          ↓             ↓             ↓
     AI Services   Model Servers   AI Agents
          │             │             │
          └─────────────┼─────────────┘
                        ↓
                  GPU Infrastructure
                        ↓
        ┌───────────────┼───────────────┐
        ↓               ↓               ↓
   Observability     Security        FinOps
Enter fullscreen mode Exit fullscreen mode

Supporting everything:

Infrastructure as Code
Identity
Secrets
Policies
Networking
Storage
Testing
Enter fullscreen mode Exit fullscreen mode

This is less about choosing one perfect tool.

It is about creating clear operational boundaries.

The Platform Should Reduce Developer Complexity

A developer building an AI feature should not need to understand every detail of:

GPU scheduling
Network policy
Secret rotation
Prometheus configuration
GitOps controllers
Cloud billing
Enter fullscreen mode Exit fullscreen mode

Ideally, the platform provides a supported path.

For example:

Developer defines:

Model
GPU requirement
Scaling policy
Environment
Enter fullscreen mode Exit fullscreen mode

The platform handles:

Infrastructure
Deployment
Security
Observability
Cost allocation
Enter fullscreen mode Exit fullscreen mode

This is where AI infrastructure starts overlapping with platform engineering.

Modern internal developer platforms increasingly combine Kubernetes, GitOps, observability, governance, security, and self-service into standardized workflows. AI agents are now beginning to become consumers of those same platforms alongside human developers.

A Practical Production Checklist

Before calling an AI platform production-ready, I would want clear answers to these questions:

  • Is infrastructure reproducible from code?
  • Are deployments declarative and version controlled?
  • Can every production change be audited?
  • Can a release be rolled back quickly?
  • Are secrets stored outside application code?
  • Does every workload follow least privilege?
  • Can GPU and inference health be monitored?
  • Can one request be traced across services?
  • Can costs be attributed to workloads or teams?
  • Are model quality and infrastructure health validated before release?

If several of these depend on manual knowledge, the platform still has operational risk.

What's Next?

This article completes the AI Infrastructure for Cloud Engineers series.

We started with:

Why Kubernetes?
Enter fullscreen mode Exit fullscreen mode

Then moved through:

GPU Scheduling
      ↓
Model Serving
      ↓
Observability
      ↓
FinOps
      ↓
Production Platform
Enter fullscreen mode Exit fullscreen mode

The biggest lesson for me is that production AI is not only a machine-learning problem.

It is also a:

Cloud problem
Distributed systems problem
Platform engineering problem
Security problem
SRE problem
FinOps problem
Enter fullscreen mode Exit fullscreen mode

And that is exactly why cloud engineers have an important role in the AI ecosystem.

Final Thoughts

A model can be impressive in a notebook.

A production AI system needs much more.

It needs:

Repeatable infrastructure
Controlled delivery
Secure access
Reliable compute
Observability
Cost visibility
Recovery
Enter fullscreen mode Exit fullscreen mode

Kubernetes provides the runtime foundation.

Infrastructure as Code makes the environment reproducible.

GitOps makes deployment controlled and auditable.

Security defines what workloads are allowed to access.

Observability tells us what the system is doing.

FinOps tells us whether we are using those resources efficiently.

Together, those pieces turn an AI application into an operable production platform.

And for cloud, DevOps, SRE, and platform engineers, that may be one of the most interesting parts of the current AI shift.

Thanks for Reading

This article completes my AI Infrastructure for Cloud Engineers series:

  1. Why Kubernetes Is Becoming the Operating System for AI Infrastructure
  2. Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving
  3. Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory
  4. FinOps for AI: Understanding GPU, Token, and Inference Costs
  5. Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability

Thanks to everyone who has read, commented, or shared their experience throughout the series.

I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, platform engineering, and the infrastructure behind production AI systems.

LinkedIn: Connect with me on LinkedIn

If you were designing an AI platform from scratch today, which part would you standardize first: infrastructure, deployment, security, observability, or cost management?

Top comments (0)