Cloud in the AI Era

I have been working with public cloud services of the major cloud providers for a decade, and for a while I've been asking myself – what has changed from an engineering, architecture, or cybersecurity point of view when organizations began to embrace generative AI for building modern applications.

In this blog post, I will review various domains (or pillars) and try to figure out if the changes are that drastic, or simply a minor change.

Physical Infrastructure

The most fundamental component in any architecture (whether cloud-native or AI) is the infrastructure, i.e., compute (to run workloads), storage (to store data temporarily or persistently), and network (for connectivity between components).

We still use standard compute, object storage, and VPCs. However, hardware selection and backplane throughput now dictate performance limits.

• Compute (Accelerators and Slicing):
  
  Shift from CPU vCPUs to GPU clusters, TPUs, and dynamic multi-tenant GPU slicing (MIG) for training and inference.

  • AWS: Amazon EC2 (P5/P4d, Trn1/Inf2), AWS Trainium / Inferentia.

• Storage (High-Throughput Pipelines):
  
  Object storage must feed massive data throughput directly to GPUs without causing memory starvation.

  • AWS: Amazon S3 Express One Zone, FSx for Lustre.

• Networking (Ultra-Low Latency Fabrics):
  
  East-West cluster inter-node bandwidth dwarfs traditional internet egress. Requires direct RoCE v2 / InfiniBand fabrics.

  • AWS: Elastic Fabric Adapter (EFA), AWS UltraClusters.

What to Offload to Neo-Clouds

Neo-clouds specialize strictly in high-density GPU infrastructure without the overhead or costs of traditional hyperscalers' managed services.

• Why Offload: Hyperscalers often face GPU quota limitations and charge heavy premiums for high-end accelerators (e.g., NVIDIA H100/H200/GB200 clusters). Neo-clouds offer bare-metal or near-bare-metal GPU clusters with specialized InfiniBand fabrics at 30%–60% lower compute rates.

• What Remains in Public Cloud: Web apps, API gateways, microservices, and general orchestration engines.

What to Offload to On-Premises Environments

On-premises deployment (using platforms like NVIDIA DGX SuperPODs, Dell/HPE GPU servers, and local Kubernetes clusters) makes sense for baseline capacity, extreme latency, and strict data sovereignty.

• Why Offload: Industrial automation, smart manufacturing, and medical imaging require real-time inference latency (<5ms) that cannot tolerate round-trip travel to a public cloud region. Small Language Models (SLMs) run directly on local edge hardware or private racks.

Bottom line: The shift of cloud to AI workloads is a minor change (evolutionary) from an architecture or engineering point of view. We may have to make some adjustments in terms of selecting the hardware, but it is not considered a major change.

Sustainability and Energy Efficiency

Mature organizations and the hyper-scale cloud providers are already prioritizing sustainability and energy efficiency when building or modernizing their data centers around the world. This is not something we can ignore – both in terms of power consumption requirements and the environmental impact.

AI training and high-volume inference generate significant power spikes, requiring explicit region selection and model right-sizing to hit corporate ESG targets.

• Training vs. Inference Carbon Impact:
  
  While model training causes massive upfront carbon spikes, continuous high-throughput inference consumes more power long-term.

  • AWS: AWS Sustainability Console.

• Model Efficiency and Footprint Reduction:
  
  Swapping 70B+ parameter models for Small Language Models (SLMs) to cut energy consumption per query by up to 90%.

  • AWS: Amazon Bedrock (deploying fine-tuned SLMs like Anthropic Claude Haiku or Llama 3 Small).

What to Offload to On-Premises Environments

To hit corporate sustainability targets under heavy AI demand, architects must actively route compute to regions offering clean energy and high thermal efficiency.

• Why Offload: Specialized providers locate GPU data centers directly at stranded energy sources (like flared gas or remote renewables), allowing enterprises to run heavy training runs with lower carbon impact.

Bottom line: The shift of cloud to AI workloads is a major paradigm shift (revolutionary) from an architecture or engineering point of view. AI has turned power grid capacity and thermal limits into non-negotiable architectural constraints rather than routine background operational details.

FinOps and Cost Optimization

With the high cost of the public cloud, and now with the demand for expensive hardware and token costs, any organization designing a modern application must embed cost as part of any architecture decision.

The unit of cost is no longer fixed hourly vCPUs; it's token generation rates, context lengths, and idle GPU capacity. Unmonitored recursive loops can exhaust monthly budgets in hours.

• Token-Based Unit Economics:
  
  FinOps teams must map cost per token, cost per prompt, and context-window payload lengths directly to application features.

  • AWS: AWS Cost Explorer (filtered by Amazon Bedrock token usage metrics), AWS CloudWatch spend metrics.

• GPU Idle Waste and Serverless Scaling:
  
  Dedicated GPU nodes sitting idle cost thousands per month. Scale-to-zero serverless inference endpoints are vital for intermittent traffic.

  • AWS: AWS Lambda (with Graviton / GPU inference endpoints), Amazon SageMaker Serverless Inference.

What to Offload to Neo-Clouds

Moving terabyte-scale datasets between environments to feed hungry GPU pipelines makes legacy cloud egress fees an immediate architectural liability.

• Why Offload: Many neo-clouds offer zero-egress or minimal data transfer fees, preventing the severe bandwidth penalties associated with shifting massive dataset checkpoints into public clouds.

What to Offload to On-Premises Environments

When inference workloads run continuously around the clock, public cloud pay-per-token pricing models trigger a cost inversion where owning on-premises hardware becomes vastly cheaper than renting cloud GPUs.

• Why Offload: Cloud AI pricing scales with usage (per-token or per-instance-hour); on-premises hardware amortizes over time. If an enterprise runs steady 24/7 baseline inference workloads, local GPU hardware often achieves a lower total cost of ownership (TCO) within 12–18 months.

Bottom line: The shift of cloud to AI workloads is a major paradigm shift (revolutionary) from an architecture or engineering point of view. AI shifts cloud economics from predictable infrastructure hours to volatile, context-dependent token generation, turning unmonitored execution loops into instant financial liabilities.

Reliability and Resiliency

Any mature architecture must take reliability and resiliency under consideration. No production workload would be built without embedding those principles as part of an architecture design.

The core availability patterns (multi-AZ, retry mechanisms) are unchanged, but we now apply them to stateful training jobs and external LLM rate limits.

• Distributed Checkpointing:

  Auto-saving model weights during multi-day training jobs to prevent single-node hardware crashes from losing days of compute time.

  • AWS: Amazon SageMaker HyperPod (automated node repair and fast checkpoint restore).

• API Fallbacks and Circuit Breakers:
  
  Routing traffic automatically when a primary LLM hits rate limits, high latency, or availability degradation.

  • AWS: Amazon Bedrock Cross-Region Inference, AWS AppSync / API Gateway circuit-breaker patterns.

Bottom line: The shift of cloud to AI workloads is considered a minor change (Incremental/Evolutionary). The underlying architectural patterns like redundant regions, retries, and circuit breakers remain identical; we are simply applying those same foundational rules to training checkpoints and LLM rate limits.

Data and Database Layer

As with any other production workload, the data layer must be taken into architectural consideration. Relational and NoSQL platforms remain standard, but native high-dimensional vector search and low-latency RAG pipelines are now fundamental requirements.

• Hybrid and Vector Search Engines:
  
  Storing and querying high-dimensional embeddings alongside traditional relational data for Retrieval-Augmented Generation (RAG).

  • AWS: Amazon Aurora PostgreSQL (with pgvector), Amazon OpenSearch Serverless (Vector Engine), Amazon DocumentDB (Vector search).

• Real-time Context Streaming:
  
  Streaming live business state into prompt context windows to avoid serving outdated data from batch ETL runs.

  • AWS: Amazon Managed Streaming for Apache Kafka (MSK), Amazon Kinesis Data Streams.

What to Offload to On-Premises Environments

For highly regulated industries handling sensitive records, keeping data storage and Retrieval-Augmented Generation context pipelines on-premises is essential to guarantee absolute data sovereignty and eliminate cloud privacy risks.

• Why Offload: Keeping vector stores, embeddings, and fine-tuning datasets inside private on-premises databases keeps sensitive data entirely within the internal security perimeter.

Bottom line: The shift of cloud to AI workloads is considered a mixed shift (Evolutionary Storage, Major Vector Shift). Core storage engines and relational databases simply added features to support vector embeddings, but the real-time context streaming required for RAG pipelines completely transformed how data feeds into applications.

Operational Excellence and Observability

Mature organizations look far beyond deployment, monitoring, and maintenance, and embed operational excellence and observability. CPU/Memory charts are insufficient. Platform engineers must now monitor token generation speeds, prompt drift, and output accuracy.

• AI Performance and Token Telemetry:

  Tracking operational metrics like Time to First Token (TTFT), tokens per second, and GPU VRAM allocation.

  • AWS: Amazon CloudWatch Container Insights (GPU metrics), Amazon Bedrock Model Invocation Logs.

• Quality, Drift, and Hallucination Evals:
  
  Programmatically auditing live model responses for toxicity, hallucination rates, and semantic drift against ground truth datasets.

  • AWS: Amazon Bedrock Model Evaluation.

Bottom line: The shift of cloud to AI workloads is considered a major paradigm shift (Revolutionary). Observability has evolved from tracking deterministic system health metrics like CPU and memory to measuring non-deterministic application behaviors like token speeds, hallucination rates, and model drift.

Security, Identity and Application Governance

We cannot consider any workload containing customer or business data without thinking about the various security or regulatory topics in every aspect of the workload. It is considered the highest operational risk layer. Security teams shift from securing static HTTP endpoints to governing autonomous, multi-step AI agents and mitigating prompt injection attacks.

• Data Boundary and Prompt Guardrails:
  
  Intercepting prompts and completions to prevent PII exfiltration, prompt injection, and intellectual property leaks into LLMs.

  • AWS: Guardrails for Amazon Bedrock, AWS KMS (model and vector encryption).

• Agentic IAM and Delegated Authority:
  
  Authenticating autonomous agents taking actions across microservices with limited, short-lived, delegated permissions.

  • AWS: AI agent identity and IAM, AWS Cognito (Agent context scopes).

What to Offload to On-Premises Environments

Running autonomous AI agents and identity guardrails on-premises gives security teams total control over credential delegation, prompt filtering, and governance boundaries without exposing internal logic to third-party cloud endpoints.

• Why Offload: Regulated industries processing sensitive customer PII or proprietary IP cannot allow prompt payloads or agent execution chains to traverse public networks. Running identity controls, content safety guardrails, and open-weight models on-premises ensures that autonomous agent actions and credential delegation remain entirely within the internal security perimeter.

Bottom line: The shift of cloud to AI workloads is considered a major paradigm shift (Revolutionary). Security teams must shift from defending static code perimeters to governing non-deterministic outputs, autonomous multi-step agents, and prompt payloads entering model context windows.

Summary

In this blog post, I reviewed many aspects common to any architect and how they shifted in the context of cloud services when organizations began to develop, build, and maintain AI workloads.

Due to cost, hardware availability, and data sovereignty, organizations should consider designing hybrid architectures that include the use of neo-clouds or on-premises, and this blog post highlights reasons to offload some of the workloads from the public cloud.

As technology evolves, organizations should keep questioning their existing architectures and make architecture and technology decisions accordingly – no "all-or-nothing" decision.

About the Author

Eyal Estrin is a cloud and information security architect with more than 25 years in the industry. He is the author of Cloud Security Handbook and Security for Cloud Native Applications.

The views expressed are his own.