DEV Community: Manohar Halappa

Architecting for AI-Native Platforms: RAG, LLM Orchestration, and Agentic Patterns

Manohar Halappa — Sun, 20 Sep 2026 08:12:06 +0000

AI adoption in an enterprise SaaS platform is rarely about adding an LLM API and calling it done.

The difficult part is integrating AI into an existing platform without weakening the properties that made the platform trustworthy in the first place.

A mature SaaS platform already has:

Domain models
Authentication and authorization
Tenant isolation
Data governance
Audit trails
APIs
Event-driven workflows
Operational controls
Reliability mechanisms

AI should not create a parallel architecture that bypasses these capabilities.

It should inherit them.

That's the architectural principle I use when thinking about evolving a mature SaaS platform toward AI-native capabilities.

AI Should Be an Augmentation Layer, Not a Parallel Platform

The first architectural decision is where AI belongs.

A tempting approach looks like this:

Existing Platform
       │
       └──────► AI Platform
                    │
                    ├── Own data
                    ├── Own permissions
                    ├── Own workflows
                    └── Own state

This creates a dangerous divergence.

Now there are effectively two systems that understand the business.

The better model is:

                  ┌─────────────────────┐
                  │   AI Capabilities   │
                  │                     │
                  │ RAG / LLM / Agents  │
                  └──────────┬──────────┘
                             │
                       Platform APIs
                             │
                  ┌──────────▼──────────┐
                  │  Canonical Domain   │
                  │       Model         │
                  └──────────┬──────────┘
                             │
                  ┌──────────▼──────────┐
                  │   Core Platform     │
                  │                     │
                  │ Auth / Tenancy /    │
                  │ Data / Audit / APIs │
                  └─────────────────────┘

The core platform remains the source of truth.

AI becomes another consumer and orchestrator of platform capabilities.

This distinction becomes increasingly important as AI moves from simply generating answers to taking actions.

Pattern 1: RAG — Give the Model the Right Context

Large language models are powerful, but they don't automatically know your organization's current data.

For enterprise applications, the challenge is therefore often less:

"Which model should we use?"

and more:

"How do we reliably provide the right context to the model?"

That's where Retrieval-Augmented Generation (RAG) becomes useful.

A simplified RAG pipeline looks like:

Documents / Domain Data
          │
          ▼
       Chunking
          │
          ▼
      Embedding
          │
          ▼
      Vector Index
          │
          │
      ┌───▼────┐
      │ Query  │
      └───┬────┘
          │
          ▼
      Retrieval
          │
          ▼
   Relevant Context
          │
          ▼
        LLM
          │
          ▼
       Response

The model isn't expected to remember everything.

The application retrieves relevant information and supplies it as context.

The Vector Pipeline Is the Foundation

If you're building multiple AI features, one of the first reusable platform capabilities should be the vector pipeline:

Ingest
  ↓
Normalize
  ↓
Chunk
  ↓
Embed
  ↓
Index
  ↓
Retrieve
  ↓
Rerank / Filter
  ↓
Generate

The specific vector technology can change.

For example, depending on the architecture and requirements, this could involve:

Amazon OpenSearch
Pinecone
Another vector-capable datastore

The important architectural decision is to avoid coupling every AI feature directly to the indexing implementation.

Instead:

                    AI Features
                 /      |       \
                /       |        \
             Search   Assistant   Agent
                \       |        /
                 \      |       /
                  ▼     ▼      ▼
                Retrieval API
                     │
                     ▼
               Vector Pipeline
                     │
                     ▼
                Domain Data

Once retrieval becomes a platform capability, multiple AI features can reuse it.

RAG Is More Than Semantic Search

One common mistake is to think of RAG as:

Question
   ↓
Vector search
   ↓
Top 5 documents
   ↓
LLM

Production systems usually need more controls.

The retrieval layer may need to consider:

Tenant
User permissions
Document type
Data freshness
Metadata
Access policies
Relevance
Source authority
Temporal constraints

For example:

User Query
    │
    ▼
Authorization Context
    │
    ▼
Tenant / Scope Filter
    │
    ▼
Semantic Retrieval
    │
    ▼
Metadata / Permission Filtering
    │
    ▼
Relevant Context
    │
    ▼
LLM

This is critical.

Retrieving information that the user isn't authorized to access is still a security vulnerability—even if the LLM never intentionally exposes it.

RAG Needs Freshness and Provenance

Enterprise data changes.

A vector index can therefore become stale.

Consider:

Source updated
     │
     ▼
Database = current
     │
     └──────► Vector index = old

The system now has two versions of reality.

That's why a production RAG architecture should think about:

Incremental indexing
Deletes
Updates
Re-indexing
Document versioning
Embedding version changes
Source timestamps
Provenance

A useful principle is:

The vector index is a derived representation, not the source of truth.

That makes lifecycle management much clearer.

Pattern 2: LLM Orchestration

Once AI workflows become more sophisticated, a single model invocation isn't enough.

A real enterprise workflow might look like:

Request
   │
   ▼
Authorize
   │
   ▼
Retrieve
   │
   ▼
Enrich
   │
   ▼
Generate
   │
   ▼
Validate
   │
   ▼
Persist
   │
   ▼
Audit

This is where orchestration becomes important.

For AWS-based architectures, workflow services such as Step Functions can provide explicit state management around multi-step operations.

The key architectural idea is:

Don't hide a distributed workflow inside one giant prompt or Lambda function.

Model the workflow explicitly.

Make AI Workflows Observable

Traditional distributed systems already taught us that asynchronous workflows need state.

AI workflows need the same discipline.

Instead of:

AI Request → ??? → Response

think:

                    AI Workflow
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
       Retrieve        Generate       Validate
          │              │              │
          └──────────────┼──────────────┘
                         ▼
                       Store
                         │
                         ▼
                       Audit

Each stage should have enough metadata to understand:

What was requested?
Which tenant initiated it?
Which data sources were consulted?
Which model/version was used?
What workflow executed?
Which tools were invoked?
What failed?
What was retried?
What was ultimately returned?

AI systems need observability at both the application and model layers.

Model Selection Should Be an Architecture Decision

Not every request needs the largest or most expensive model.

A mature AI platform can route workloads according to their requirements.

For example:

                     Request
                        │
                        ▼
                  Classify Task
                        │
            ┌───────────┼───────────┐
            ▼           ▼           ▼
          Simple      Complex      Batch
            │           │           │
            ▼           ▼           ▼
        Fast/cheap   Capable LLM  Offline
          model        model       inference

This introduces another important architectural metric:

Cost per successful business outcome

rather than simply:

Cost per LLM request.

The cheapest model isn't useful if it produces an answer that requires repeated retries or human correction.

Where SageMaker Fits

Not every AI workload is an LLM workflow.

Traditional machine-learning workloads still matter.

For use cases involving:

Model training
Feature engineering
Batch inference
Model evaluation
MLOps
Model deployment

a platform such as Amazon SageMaker can provide a different execution model.

A useful architectural separation is:

                   AI Platform
                       │
          ┌────────────┴────────────┐
          │                         │
          ▼                         ▼
   Generative AI               Predictive ML
          │                         │
   LLM / RAG / Agents       Training / Inference
          │                         │
          ▼                         ▼
   Bedrock / LLM stack          SageMaker

The goal isn't to force every AI capability through the same technology.

Pattern 3: Agentic Workflows

Agents introduce a fundamentally different capability.

A traditional application does this:

User
 ↓
API
 ↓
Business Logic
 ↓
Result

An agent can potentially do:

User
 ↓
Agent
 ↓
Decide next action
 ↓
Call tool
 ↓
Observe result
 ↓
Decide next action
 ↓
Call another tool
 ↓
Return result

That introduces a new architectural concern:

bounded autonomy.

Agents Need Explicit Boundaries

An agent should not automatically receive unrestricted access to the platform.

Instead, define:

Scope

What business problem is the agent allowed to solve?

Tools

Which APIs or actions can it invoke?

Permissions

What can it read?

What can it modify?

Limits

How many actions can it perform?

How much can it spend?

Approval

Which actions require human confirmation?

A useful mental model is:

                 ┌───────────────┐
                 │     Agent     │
                 └───────┬───────┘
                         │
                  Policy / IAM
                         │
            ┌────────────┼────────────┐
            ▼            ▼            ▼
        Tool A        Tool B        Tool C
        Read          Read          Write

The agent doesn't get "platform access."

It gets specific capabilities.

Human-in-the-Loop Is a Control, Not a Failure

For high-impact actions, autonomy shouldn't necessarily mean zero human involvement.

Consider:

Agent proposes action
        │
        ▼
Policy evaluation
        │
        ├── Low risk ──→ Execute
        │
        └── High risk ─→ Human approval
                              │
                              ▼
                           Execute

The important question isn't:

"Can the agent do this automatically?"

It's:

"What level of autonomy is appropriate for this action?"

This is the same risk-based thinking we already use in distributed systems and security architecture.

Audit Every Agent Action

Once an AI system can take actions, logging the final response isn't enough.

You need to understand the chain of execution.

For example:

Request
  ↓
Agent decision
  ↓
Tool selected
  ↓
Tool parameters
  ↓
Authorization check
  ↓
Tool result
  ↓
Next decision
  ↓
Final action

The exact implementation will depend on the platform and privacy requirements, but the architectural principle is straightforward:

An action taken by an agent should be as auditable as an action taken by a human or traditional service.

This becomes particularly important for regulated or enterprise environments.

Security Must Flow Through the AI Stack

One of the most dangerous architectural mistakes is treating AI as a separate security domain.

Your existing platform may already have:

Authentication
RBAC
ABAC
Tenant isolation
IAM
API authorization
Audit logging

The AI layer should inherit these controls.

Consider a multi-tenant SaaS application:

                 User
                   │
                   ▼
             Authentication
                   │
                   ▼
            Tenant Context
                   │
                   ▼
           Authorization
                   │
             ┌─────┴─────┐
             ▼           ▼
           RAG          Agent
             │           │
             ▼           ▼
          Retrieval     Tools
             │           │
             └─────┬─────┘
                   ▼
              Domain APIs

The AI system should not create a backdoor around the authorization model.

This is especially important for RAG.

Tenant isolation must exist in retrieval itself, not merely in the user interface.

Guardrails Are Architecture

AI guardrails shouldn't be an afterthought added after the first production incident.

They belong in the architecture.

Examples include:

Input controls

Prompt validation
Input size limits
Abuse detection
Sensitive-data handling

Retrieval controls

Tenant filtering
Authorization checks
Source validation
Freshness requirements

Output controls

Schema validation
Content validation
Confidence or quality checks
Business-rule validation

Operational controls

Token budgets
Rate limits
Model routing
Timeout limits
Retry limits

A useful pipeline looks like:

Input
  ↓
Validate
  ↓
Authorize
  ↓
Retrieve
  ↓
Generate
  ↓
Validate Output
  ↓
Business Rules
  ↓
Persist / Act
  ↓
Audit

The LLM is one component inside the workflow.

It shouldn't become the workflow itself.

Don't Let the LLM Become the Source of Truth

This is perhaps the most important design principle.

An LLM should generally reason over authoritative data, not replace it.

For example:

                    ┌───────────────┐
                    │ Source of     │
                    │ Truth         │
                    └───────┬───────┘
                            │
                            ▼
                       AI Context
                            │
                            ▼
                           LLM
                            │
                            ▼
                    Proposed Answer /
                         Action
                            │
                            ▼
                    Domain Validation
                            │
                            ▼
                       Platform

The model generates a result.

The platform determines whether that result is valid.

This distinction becomes critical when AI starts taking actions rather than simply answering questions.

What I Tell Engineering Teams

I use a simple rule when reviewing AI architecture:

The AI layer should inherit the trust properties of the core platform.

If your platform has strong:

Multitenancy
Authorization
Auditability
Observability
Idempotency
Resilience

then AI should inherit those properties.

If those properties are weak, introducing AI doesn't hide the weakness.

It can amplify it.

An AI system that can access ten times more data or execute ten times more actions can turn a small authorization mistake into a much larger incident.

A Practical AI-Native Evolution Path

You don't need to build an autonomous agent platform on day one.

A pragmatic evolution can look like this:

Phase 1
Canonical data + APIs
        │
        ▼
Phase 2
RAG / Retrieval
        │
        ▼
Phase 3
LLM-powered workflows
        │
        ▼
Phase 4
Tool-enabled assistants
        │
        ▼
Phase 5
Bounded agentic workflows
        │
        ▼
Phase 6
Selective autonomous actions

Each phase builds on the previous one.

This is important because the hardest part of AI adoption isn't usually the model.

It's building the platform capabilities around the model.

The AI Platform Capabilities I'd Build First

Before investing heavily in autonomous agents, establish the foundations.

1. Canonical data access

AI should consume well-defined domain APIs and data products.

2. Retrieval platform

Build reusable ingestion, chunking, embedding, indexing, retrieval, and authorization capabilities.

3. Model gateway

Centralize model access where practical so applications don't each implement their own:

Authentication
Model selection
Rate limiting
Cost tracking
Observability
Safety controls

4. Workflow orchestration

Use explicit workflows for multi-step AI operations.

5. Tool layer

Expose controlled business capabilities as tools rather than giving agents unrestricted database or infrastructure access.

6. Evaluation

Build repeatable evaluation datasets and quality metrics.

An AI feature isn't production-ready simply because it works for ten manually tested prompts.

You need to know:

Does it work?
How often does it fail?
When does it fail?
Which tenants/data types are affected?
Did a model or prompt change make it worse?

Metrics That Matter

Traditional application metrics aren't enough.

An AI-native platform should track several dimensions.

Quality

Retrieval precision / relevance
Groundedness
Task success rate
Validation failure rate
Human correction rate

Performance

End-to-end latency
Retrieval latency
Model latency
Token usage

Reliability

Workflow failures
Tool failures
Retry rates
Timeout rates

Security

Authorization failures
Cross-tenant retrieval attempts
Policy violations
Blocked tool calls

Economics

Cost per request
Cost per successful task
Cost per tenant
Cost by model
Cost by workflow

The goal is to optimize business outcomes, not simply model metrics.

What Changes When AI Becomes Agentic?

RAG primarily changes how applications retrieve information.

LLM orchestration changes how applications coordinate AI-powered workflows.

Agents change how applications take actions.

That means the risk profile evolves:

RAG
 │
 └── Information risk

LLM workflows
 │
 └── Information + workflow risk

Agents
 │
 └── Information + workflow + action risk

The more autonomy you introduce, the stronger your controls need to become.

The Real Lesson

AI-native architecture isn't about putting an LLM at the center of everything.

It's about creating a platform where AI capabilities can evolve without bypassing the engineering disciplines that already protect the business.

The architecture I want looks like:

                         AI Applications
                               │
                ┌──────────────┼──────────────┐
                ▼              ▼              ▼
               RAG          Workflows       Agents
                │              │              │
                └──────────────┼──────────────┘
                               ▼
                         AI Platform
                               │
                 ┌─────────────┼─────────────┐
                 ▼             ▼             ▼
             Retrieval      Models         Tools
                 │             │             │
                 └─────────────┼─────────────┘
                               ▼
                        Core Platform
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
          Identity         Domain Data       Audit
              │                │                │
              └────────────────┼────────────────┘
                               ▼
                        Source of Truth

The goal isn't to make the platform "AI-powered."

The goal is to make AI a first-class capability of the platform without making it a special exception to the platform's rules.

That's the difference between adding AI features and building an AI-native platform.

AI should inherit your platform's trust model—not replace it.

From Monolith to Serverless: How We Cut Infrastructure Costs by 30–35%:

Manohar Halappa — Sun, 20 Sep 2026 08:04:06 +0000

From Monolith to Serverless: How We Cut Infrastructure Costs by 30–35%

Modernizing a legacy monolith is rarely a matter of replacing servers with Lambda.

The difficult part is deciding what should become serverless, what should remain where it is, and in what order to make the change without disrupting the business.

In two modernization programs, I found that the biggest gains came from treating modernization as an architecture and economics problem, not simply a technology migration.

The result was a 30–35% reduction in infrastructure costs for the workloads we migrated.

The interesting part wasn't Lambda itself.

It was changing the system from:

Always running
      ↓
Wait for work
      ↓
Process work
      ↓
Wait again

to:

Event arrives
      ↓
Compute starts
      ↓
Process
      ↓
Compute stops

The fundamental shift was from paying for capacity to paying for useful work where the workload characteristics made that model appropriate.

Start With Boundaries, Not Code

One of the easiest mistakes in modernization is starting with:

"Which class should we extract first?"

That's usually the wrong question.

Start with:

"Which business capability has a clear boundary and a workload that benefits from independent scaling?"

Before touching the code, map the existing system.

Look at:

Request volume
Traffic patterns
CPU and memory utilization
Processing duration
Dependency relationships
Database access patterns
Failure characteristics
Scaling behavior
Deployment frequency
Business criticality

The resulting workload map often looks something like this:

                       Workload Characteristics

              ┌───────────────────────────────────┐
              │ High volume + predictable traffic  │
              │                                   │
              │ Existing service may be suitable │
              │ for optimized container compute  │
              └───────────────────────────────────┘

              ┌───────────────────────────────────┐
              │ Spiky + stateless + short-lived   │
              │                                   │
              │ → Lambda / API Gateway           │
              └───────────────────────────────────┘

              ┌───────────────────────────────────┐
              │ Async + long-running              │
              │                                   │
              │ → SQS + Step Functions + Lambda  │
              └───────────────────────────────────┘

              ┌───────────────────────────────────┐
              │ Large batch / data processing     │
              │                                   │
              │ → S3 + event-driven processing   │
              └───────────────────────────────────┘

This exercise prevents one of the most common modernization mistakes:

Forcing every workload into the same architecture.

Serverless is a tool, not an ideology.

The Extraction Strategy

Rather than rewriting the monolith, we extracted specific capabilities behind well-defined interfaces.

A simplified evolution looked like this:

                    BEFORE

              ┌─────────────────┐
              │                 │
              │    MONOLITH     │
              │                 │
              │  API            │
              │  Business Logic │
              │  Batch Jobs     │
              │  Integrations   │
              │                 │
              └───────┬─────────┘
                      │
                   Database

Over time:

                    AFTER

                ┌──────────────┐
                │ API Gateway  │
                └──────┬───────┘
                       │
                  ┌────▼─────┐
                  │  Lambda  │
                  └────┬─────┘
                       │
                       ▼
                ┌─────────────┐
                │  Services   │
                └─────────────┘

 Events ──→ SQS ──→ Lambda ──→ DynamoDB/S3

 Events ──→ Step Functions
                 │
          ┌──────┼──────┐
          ▼      ▼      ▼
       Lambda  Lambda  Lambda

The monolith didn't disappear overnight.

Instead, capabilities were progressively moved behind explicit boundaries.

That reduced both technical and organizational risk.

1. Read-Heavy and Spiky Workloads

The first candidates were workloads with:

Highly variable traffic
Short execution times
Stateless processing
Clear API boundaries
Significant idle periods

These workloads are natural candidates for:

API Gateway → Lambda → downstream service

The economics can be attractive because capacity scales with demand rather than requiring permanently provisioned application servers.

Consider a simplified workload:

Traffic

       ▲
       │             ████
       │             ████
       │       ███   ████
       │       ███   ████
       │  ██   ███   ████
       │  ██   ███   ████
       └──────────────────────→ time

With always-on infrastructure, you provision for the peak:

Provisioned capacity ─────────────────────
Actual workload        ▂▂▃▂▂▆▃▂▂▂▂▇▂

That gap represents potentially unused capacity.

With serverless:

Provisioned capacity
Actual execution      ▂▂▃▂▂▆▃▂▂▂▂▇▂

The infrastructure model more closely follows actual execution.

But there is an important qualification:

Serverless isn't automatically cheaper.

For continuously high utilization, long-running workloads, or workloads with substantial execution duration, other compute models can be more economical.

The architecture decision should therefore be based on workload economics, not technology preference.

2. Replace Polling With Events

One of the biggest opportunities wasn't moving existing code to Lambda.

It was eliminating unnecessary work.

A common legacy pattern looked like:

┌──────────────┐
│ Always-on VM │
└──────┬───────┘
       │
       ▼
   Poll queue
       │
       ├── No work → sleep
       │
       └── Work → process

The system was effectively paying for compute even when there was nothing to process.

We moved appropriate workflows toward:

Now compute existed primarily when there was work to perform.

This wasn't just a cost optimization.

It also reduced coupling between producers and consumers and made scaling more explicit.

3. Long-Running Workflows: Step Functions

Not every workflow belongs inside a single Lambda invocation.

Some operations involve multiple stages:

Fetch
  ↓
Validate
  ↓
Transform
  ↓
Persist
  ↓
Notify

Putting all of that into one function creates problems around:

Timeout limits
Retry behavior
Partial failure
Observability
Recovery
Operational visibility

Instead, we modeled long-running workflows explicitly.

For example:

             ┌────────────┐
             │   Start    │
             └─────┬──────┘
                   ▼
             ┌────────────┐
             │   Fetch    │
             └─────┬──────┘
                   ▼
             ┌────────────┐
             │  Validate  │
             └─────┬──────┘
                   ▼
             ┌────────────┐
             │ Transform  │
             └─────┬──────┘
                   ▼
             ┌────────────┐
             │   Persist  │
             └─────┬──────┘
                   ▼
             ┌────────────┐
             │  Complete  │
             └────────────┘

The workflow engine became responsible for orchestration, while individual functions remained focused on individual tasks.

This made failures easier to reason about.

If Transform fails, the workflow knows exactly where it failed and what retry policy applies.

4. Bulk Data Processing

Large data operations required a different approach.

Instead of treating a massive batch as one application process, we used object storage and event-driven processing where appropriate.

A typical pattern was:

                 ┌─────────────┐
                 │     S3      │
                 │ Raw Dataset │
                 └──────┬──────┘
                        │
                     Event
                        │
                        ▼
                 ┌─────────────┐
                 │ Processing  │
                 │   Workers   │
                 └──────┬──────┘
                        │
                        ▼
                 ┌─────────────┐
                 │  Persisted  │
                 │    Data     │
                 └─────────────┘

This creates another useful separation:

Storage doesn't need to be coupled to compute.

Data can exist independently of the processing lifecycle.

That makes retries, replay, and recovery significantly easier to design.

5. The Cost Reduction Was a Consequence, Not the Architecture

This is an important distinction.

We didn't say:

"Let's move everything to Lambda and save money."

We asked:

"Where are we paying for capacity that isn't being used?"

Then we targeted those workloads.

The cost model changed from:

Always-on infrastructure
+
Provisioned capacity
+
Idle periods
+
Operational overhead

toward:

Actual execution
+
Event-driven scaling
+
Managed orchestration

For the workloads we migrated, this contributed to a 30–35% infrastructure cost reduction.

But the number should not be interpreted as a universal serverless savings percentage.

The actual economics depend on:

Invocation frequency
Execution duration
Memory allocation
Concurrency
Data transfer
Storage
Database costs
Observability costs
Existing infrastructure utilization
Provisioned versus on-demand capacity

That's why cost per business transaction was more useful than looking only at the monthly infrastructure bill.

6. Guardrails Before Migration

Moving from a monolith to distributed services increases the number of things that can fail.

We therefore treated guardrails as part of the architecture.

Rollback-first deployments

Every deployment needed a safe rollback path.

The question wasn't:

"Can we deploy this?"

It was:

"What happens if this deployment is wrong?"

A deployment isn't production-ready until the recovery path is understood.

Infrastructure as Code

We standardized infrastructure using tools such as:

Terraform
AWS CDK
GitHub Actions

The goal wasn't simply automation.

It was repeatability.

A service shouldn't require an individual engineer to remember a collection of undocumented console steps.

Instead:

Code
  ↓
Pull Request
  ↓
Validation
  ↓
Infrastructure Plan
  ↓
Deployment

The infrastructure becomes part of the software lifecycle.

7. Resilience Patterns Belong in the Platform

Once the monolith becomes a collection of distributed services, failure modes multiply.

We standardized common resilience patterns:

Timeouts

Never allow a downstream dependency to block indefinitely.

Retries

Retry transient failures, but use bounded retries and appropriate backoff.

Dead-letter queues

Failed asynchronous messages should have somewhere explicit to go.

Idempotency

A retry should not accidentally create duplicate business operations.

Observability

Every workflow needs enough telemetry to answer:

What happened?
Where did it fail?
Why did it fail?
What was retried?
What is the current state?

These shouldn't be reinvented independently by every team.

Where practical, they belong in shared platform capabilities and golden paths.

The Part We Almost Got Wrong

One of the biggest lessons from modernization is that distributed architecture introduces its own tax.

After extracting services, you now have:

More deployments
More network calls
More failure boundaries
More observability requirements
More IAM policies
More infrastructure components
More operational states

A badly designed microservices architecture can cost more—both financially and organizationally—than the monolith it replaced.

So the target shouldn't be:

"Maximum number of microservices."

It should be:

"The minimum number of independently scalable and independently deployable boundaries that make business sense."

That's a very different goal.

What We Measured

Cost was only one dimension.

For every migrated workload, we tracked a combination of financial, operational, and performance metrics.

Cost

Infrastructure cost per month
Cost per transaction
Cost per million requests
Cost per processed workload

Performance

p50 / p95 / p99 latency
Cold-start impact where relevant
Queue processing latency
Workflow duration

Reliability

Error rate
Retry rate
Dead-letter volume
Timeout rate
Failed workflow executions

Delivery

Deployment frequency
Deployment duration
Rollback frequency
Mean time to recovery

Looking at these metrics together prevents a common optimization mistake:

Saving money by making the system slower or less reliable isn't necessarily an improvement.

The Real Lesson

The most important lesson wasn't:

"Serverless is cheaper."

It was:

"Don't pay for infrastructure when your architecture doesn't need it."

The architecture evolved from:

Servers
   ↓
Processes
   ↓
Polling
   ↓
Work

toward:

Events
   ↓
Managed orchestration
   ↓
On-demand compute
   ↓
Work

The infrastructure became more closely aligned with actual business activity.

And that is where the savings came from.

A Practical Modernization Checklist

If you're starting a similar journey, I'd ask these questions before extracting the first service:

Workload

Is the workload stateless?
Is traffic spiky?
Does it have long idle periods?
Does it have a clear business boundary?
Can it scale independently?

Data

Does it have a clear data ownership model?
Can reads and writes be isolated?
What happens during partial failure?
Is the operation idempotent?

Reliability

What happens when a dependency times out?
What happens when a message is processed twice?
Where do permanently failed messages go?
How is recovery performed?

Operations

Can the service be deployed independently?
Can it be rolled back safely?
Can operators understand its state without reading application logs?
Is the infrastructure reproducible?

Economics

What is the current utilization?
What are we paying for when the system is idle?
What is the expected cost at current and peak volume?
What additional costs will serverless introduce?

Final Thought

Modernization isn't about making a monolith disappear.

It's about changing the economics, reliability, and delivery model of the system one boundary at a time.

Serverless was valuable because it allowed us to align compute more closely with actual work.

Event-driven architecture was valuable because it allowed the system to react instead of constantly polling.

Infrastructure as code was valuable because it made the new architecture repeatable.

And the 30–35% cost reduction was ultimately a consequence of those architectural decisions—not the reason to make them.

The best modernization strategy isn't "move to serverless."

It's:

Understand the workload. Find the boundary. Remove unnecessary always-on capacity. Make failure recoverable. Measure the result. Then repeat.

Scaling ETL to 25M+ Records Across 120+ School Districts: An Architecture Story

Manohar Halappa — Sun, 20 Sep 2026 07:44:54 +0000

When you're moving a few thousand records, ETL is mostly a data-processing problem.

When you're moving 25+ million records across 120+ school districts, it becomes something else entirely:

A distributed-systems reliability problem where you need to prove that the data moved correctly.

In an education platform, this distinction matters. Student, enrollment, attendance, and course data isn't just another dataset. A partial or silently corrupted load can affect downstream analytics, interventions, reporting, and operational decisions.

This is the story of the architecture and reliability principles we used to approach that problem.

The Problem: Moving Millions of Records Without Losing Trust

Our platform needed to ingest data from 120+ school districts, each with its own data volume, timing characteristics, and potential failure modes.

Across a typical sync cycle, the pipeline could process 25M+ records spanning areas such as:

Students
Enrollments
Attendance
Courses
Sections
Staff and related relationships

The challenge wasn't simply:

"Can we process 25 million records?"

The harder questions were:

Did we receive everything the source intended to send?
Did every accepted record get processed?
What happened when a job failed halfway through?
Could we safely retry?
How do we detect partial loads?
Can we explain exactly what happened to a district's data days or weeks later?

That led us to four primary requirements.

1. Scale

The pipeline needed to handle 25M+ records per sync cycle, while accommodating significant differences between districts.

One district might have a relatively small dataset.

Another could generate millions of records.

That meant designing around variable workloads, rather than assuming every sync would behave the same way.

2. Correctness

Successful execution isn't the same thing as successful data ingestion.

A job can return 200 OK, complete without throwing an exception, and still produce an incomplete dataset.

For example:

Source:  1,250,000 records
Target:  1,247,831 records

From an infrastructure perspective, the job might appear healthy.

From a data perspective, something went wrong.

We therefore treated reconciliation as a first-class part of the pipeline, rather than something performed manually after an incident.

3. Auditability

Education data requires a strong operational audit trail.

We needed to answer questions such as:

When was this district synchronized?

How many records did we receive?

How many were validated?

How many were successfully processed?

Were any records rejected?

Did the job retry?

Did reconciliation pass?

If something failed, where did it fail?

This pushed us toward designing observability and auditability into the pipeline rather than bolting them on afterward.

4. Reliability

At this scale, failures are inevitable.

Networks fail.

External SIS systems become unavailable.

Workers restart.

Individual batches fail.

Dependencies time out.

A downstream service can become temporarily unavailable.

The architecture therefore had to assume that partial failure is normal.

The goal wasn't to eliminate failure.

It was to make failure safe, detectable, recoverable, and explainable.

The Architecture

At a high level, the pipeline looked like this:

                ┌─────────────────────┐
                │   School Districts  │
                │      / SIS          │
                └──────────┬──────────┘
                           │
                           ▼
                ┌─────────────────────┐
                │      Ingestion      │
                │ Scheduled / Batch   │
                │      Processing     │
                └──────────┬──────────┘
                           │
                           ▼
                ┌─────────────────────┐
                │     Validation      │
                │ Schema + Integrity  │
                │       Checks        │
                └──────────┬──────────┘
                           │
                           ▼
                ┌─────────────────────┐
                │     Processing      │
                │ Transform + Load    │
                │    Idempotently     │
                └──────────┬──────────┘
                           │
                           ▼
                ┌─────────────────────┐
                │   Reconciliation    │
                │ Source vs. Target   │
                │       Counts        │
                └──────────┬──────────┘
                           │
                           ▼
                ┌─────────────────────┐
                │   Observability &   │
                │       Audit         │
                └─────────────────────┘

The important part isn't any individual technology.

It's the control points between the stages.

1. Idempotent Ingestion

The first principle we adopted was simple:

Assume every operation can be retried.

A network timeout doesn't necessarily mean that the server didn't process the request.

A worker can crash after writing data but before acknowledging completion.

A scheduler can trigger the same operation more than once.

If the pipeline isn't idempotent, retries can turn transient failures into permanent data corruption.

We therefore designed operations around stable identifiers and idempotency keys.

Conceptually:

record + operation + source version
                │
                ▼
        deterministic identity
                │
                ▼
        idempotent processing

A retry should result in:

First attempt  → write
Second attempt → recognize existing operation
Third attempt  → same final state

rather than:

First attempt  → write
Second attempt → duplicate write

This became one of the most important design principles in the system.

2. Validate Before Processing

We didn't want malformed data to travel deep into the pipeline before being discovered.

Validation happened as early as practical.

Typical validation categories included:

Schema validation
Required-field validation
Referential integrity
Data-type validation
Source-specific business rules
Duplicate detection

This created a useful separation:

Raw input
   │
   ▼
Validation
   │
   ├── Invalid → rejected / dead-letter path
   │
   ▼
Valid records
   │
   ▼
Processing

That separation also improved troubleshooting.

Instead of asking:

"Why did this record disappear?"

we could ask:

"Was this record rejected during validation, or was it accepted and subsequently failed during processing?"

That distinction is extremely valuable in production.

3. Process in Batches, Not One Giant Transaction

With tens of millions of records, treating an entire sync as one atomic operation is usually impractical.

Instead, we divided workloads into manageable units.

Conceptually:

25M records
    │
    ├── Batch 1
    ├── Batch 2
    ├── Batch 3
    ├── ...
    └── Batch N

This provides several advantages:

Failure isolation

If one batch fails, we don't necessarily need to restart the entire sync.

Retryability

Failed batches can be retried independently.

Parallelism

Where appropriate, independent batches can be processed concurrently.

Operational visibility

Instead of:

Sync = FAILED

we can reason about:

1,248 batches completed
12 batches retried
2 batches failed

That is much more actionable.

4. Reconciliation: The Most Important Control

One of the biggest lessons from operating high-volume data pipelines is this:

A successful job does not prove a successful data load.

That's why reconciliation became an explicit stage.

At the end of a load, we compare source and target measurements.

For example:

                    Source       Target
Students            1,250,000    1,250,000
Enrollments         3,840,000    3,840,000
Attendance          8,910,000    8,909,997
Courses             1,120,000    1,120,000

Even a small mismatch matters.

The goal isn't necessarily to prevent every mismatch.

The goal is to ensure that a mismatch cannot silently pass through the system.

A simplified control flow looks like:

                 Sync Complete
                       │
                       ▼
              Compare source/target
                       │
              ┌────────┴────────┐
              │                 │
           Match             Mismatch
              │                 │
              ▼                 ▼
          Success            Alert
                                │
                                ▼
                         Investigate /
                         retry / repair

This changed the operational question from:

"Did the job finish?"

to:

"Can we prove the expected data arrived?"

That is a much stronger guarantee.

5. Dead-Letter Paths Are Part of the Design

Not every record that enters the pipeline will successfully complete.

Instead of allowing problematic records to disappear into logs, we need a deliberate failure path.

                  Processing
                      │
             ┌────────┴────────┐
             │                 │
          Success             Failure
             │                 │
             ▼                 ▼
          Target          Dead-letter
                             │
                             ▼
                       Investigation

The dead-letter path provides two important properties:

The primary pipeline can continue processing valid data.
Failed records remain visible and recoverable.

This is particularly important when one malformed record shouldn't block millions of valid records.

6. Observability Must Follow the Data

Traditional application monitoring often focuses on infrastructure metrics:

CPU
Memory
Latency
Error rates
Availability

Those are important, but they aren't enough for data pipelines.

We also needed data-level telemetry.

For each sync, useful metrics included concepts such as:

records_received
records_validated
records_processed
records_rejected
records_failed
records_retried
records_loaded
reconciliation_status

This lets operators move from:

"Something failed."

to:

"District X received 2.4M records.
2.39M were processed successfully.
4,812 were rejected during validation.
127 failed during transformation.
Reconciliation detected a 127-record mismatch."

That's the difference between monitoring infrastructure and observing a data pipeline.

7. Design for Partial Failure

Distributed systems fail partially.

A pipeline processing millions of records should assume that:

Some batches succeed.
Some batches fail.
Some requests time out.
Some retries succeed.
Some retries fail again.
External systems can disappear temporarily.

This leads to an important architectural principle:

Don't design around the assumption that the entire operation succeeds or fails together.

Instead, explicitly model intermediate states.

For example:

QUEUED
   ↓
PROCESSING
   ↓
VALIDATED
   ↓
LOADED
   ↓
RECONCILED

And failure states:

PROCESSING
    ↓
FAILED
    ↓
RETRYING
    ↓
PROCESSING

This state-oriented approach makes both automation and operational debugging significantly easier.

What We Learned

1. Idempotency is non-negotiable

Retries are inevitable.

Without idempotency:

Retry + partial success = duplicate or inconsistent data

With idempotency:

Retry + partial success = recoverable operation

Design for retries from the beginning rather than adding idempotency after the first production incident.

2. Reconciliation catches what application monitoring misses

A pipeline can have:

No infrastructure alarms
No application exceptions
Successful job completion

…and still have incorrect data.

Count-based reconciliation and other data-quality controls provide a second line of defense.

3. Batch-level controls are as important as record-level controls

Record-level validation answers:

"Is this record valid?"

Batch-level reconciliation answers:

"Did we process everything we expected to process?"

You need both.

4. Failure should be observable, not invisible

A rejected record isn't necessarily a disaster.

A rejected record that nobody knows about is.

Dead-letter queues, explicit states, metrics, and alerts turn hidden failures into manageable operational work.

5. Auditability should be designed, not added later

If you wait until an audit or production incident to answer:

"What happened to this data?"

you'll discover that the information you need may no longer exist.

Capture the evidence while the pipeline is running.

The Bigger Lesson

Scaling ETL isn't primarily about making the pipeline faster.

At 25M+ records, throughput is only one dimension of the problem.

The more important questions become:

Can we retry safely?
Can we detect partial failure?
Can we reconcile source and target?
Can we explain what happened?
Can we recover without reprocessing everything?
Can we prove the final state?

That changes how you design the system.

The pipeline isn't simply:

Extract → Transform → Load

It becomes:

Extract
   ↓
Validate
   ↓
Process
   ↓
Verify
   ↓
Reconcile
   ↓
Audit

And that's probably the biggest lesson we learned:

Modern ETL isn't just about moving data. It's about being able to prove that the data moved correctly.

When you're processing tens of millions of records across hundreds of data sources, trust becomes a system feature.