DEV Community: NARESH

When Your Backend Starts Building a Workflow Engine

NARESH — Sun, 08 Nov 2026 10:10:31 +0000

When Your Backend Starts Building a Workflow Engine

How ordinary queues, databases, and scheduled jobs gradually become custom orchestration infrastructure

When a worker crashes halfway through a business operation, who is responsible for making sure the remaining work gets completed?

That question became more interesting to me as I spent more time working with backend systems. I began to realize that successfully executing an operation is only part of the challenge. We also need to understand what happens when something fails halfway through.

A database update might succeed, but the service could crash before publishing the next event. A payment might go through, but its response could get lost. A worker might disappear after completing some steps, leaving the rest unfinished.

These scenarios gradually changed how I approached backend architecture. Instead of thinking only about whether an operation would succeed, I started asking how the system would recover without losing progress or repeating something that had already happened.

We already have familiar ways to handle these situations. Databases preserve state, queues distribute background work, and scheduled jobs help recover unfinished operations. For many applications, these are perfectly reasonable solutions.

But as I explored workflows involving multiple services, retries, and longer waiting periods, I became curious about something else: what happens when tracking execution progress and recovering from failures becomes a significant responsibility of the system itself?

That question led me to explore durable workflow execution and the engineering decisions behind systems like Uber Cadence, Temporal, and AWS Step Functions.

What interested me wasn't learning another infrastructure tool. It was understanding why engineers built these systems, what problems their designs solve, and what trade-offs they introduce.

More importantly, when does adopting a dedicated workflow engine make more sense than continuing with the databases, queues, and recovery mechanisms we already have?

That's the architectural question I want to explore in this article.

How Ordinary Backend Systems Evolve

Imagine we're building a simple order-processing system. When a customer places an order, the backend reserves inventory, processes the payment, creates a shipment, and sends a confirmation.

Initially, we might handle this through a series of service calls. For a small application, there's nothing wrong with that approach. The logic is straightforward, the execution flow is easy to follow, and we don't need additional infrastructure to coordinate every step.

But eventually, some operations take longer than expected. Payment providers might respond slowly, shipment creation could depend on another service, and we don't want customers waiting until every background operation finishes.

So we introduce a message queue.

Now the API can accept the order, and a background worker can process it asynchronously. We've separated the customer's request from the actual processing, which is a reasonable improvement.

Then another requirement appears: we need to know how far an order has progressed, even if the worker crashes.

We add a status field to our database.

ORDER_CREATED
    ↓
INVENTORY_RESERVED
    ↓
PAYMENT_COMPLETED
    ↓
SHIPMENT_CREATED
    ↓
COMPLETED

If a worker fails, another worker can inspect the stored state and determine what needs to happen next. Of course, we still need safeguards to prevent duplicate processing or repeating operations that may have already succeeded.

So far, we're still using familiar backend components. We have a database for persistent state, a queue for distributing work, and workers to execute operations.

But failures introduce new questions.

What if the database records a successful payment, but the process crashes before publishing the shipment task? What if two workers pick up the same order? What if the payment provider is temporarily unavailable, and we need to retry after several minutes?

We start adding recovery mechanisms. A retry counter, a timestamp for the next attempt, perhaps a scheduled job to find unfinished orders. As multiple workers become involved, we may also need leases to prevent them from processing the same operation concurrently.

A lease is simply temporary ownership of work. If one worker disappears, another can take over after that ownership expires.

Over time, our architecture might look something like this:

For many applications, this architecture is already a reliable and maintainable solution.

The interesting change happens when the requirements keep growing.

An order might need to wait for manual approval. A failed shipment might require a refund and an inventory release. Some operations may need to resume after several days, while others must survive deployments or temporary outages.

Our code is no longer responsible only for performing business operations. It is also responsible for tracking execution progress, deciding what runs next, scheduling retries, recovering interrupted work, and coordinating actions across services.

We may eventually introduce a dedicated coordinator to manage all of this.

And at that point, an interesting question emerges:

Are we still building business logic, or have we started building the infrastructure needed to execute that business logic reliably?

That distinction is where the architectural discussion becomes interesting. Not because our existing approach has suddenly become wrong, but because coordination itself is becoming a responsibility we need to design, operate, and maintain.

When Coordination Becomes Infrastructure

At this stage, we already have a system that can persist its state, process background jobs, and recover from failures. So why would we need anything more?

The answer isn't simply that our workflow has more steps or involves more services. A well-designed database state machine with a message queue can handle surprisingly complex operations.

The real question is how much responsibility we're taking on to keep those operations moving correctly.

Consider a payment that fails because the provider is temporarily unavailable. We can schedule a retry. That's straightforward. But what if the payment actually succeeded and the response was lost? Now we need idempotency or reconciliation to determine what happened.

Suppose the payment succeeds, but the worker crashes before scheduling shipment creation. We need a reliable way to ensure that the next task isn't lost. A transactional outbox can solve that problem by recording the state change and the intent to publish work in the same database transaction.

These are established solutions, and they work well.

Now imagine the same workflow needs to wait three days for approval, coordinate several external services, handle cancellation halfway through, and compensate for previously completed operations when something fails permanently.

We need to know not just the current business state, but which operations completed, which are still running, which can safely retry, and what should happen after a failure.

And this information must remain reliable even when workers crash, services restart, or the application is deployed again.

None of these requirements individually demands a workflow engine. The challenge is their combination.

Where does the boundary lie?

There's no fixed number of services, workflow steps, or retries that tells us when to introduce durable orchestration.

I think a more useful way to evaluate the architecture is to ask:

Recovery: Can we reliably resume interrupted operations without complicated manual intervention?

Coordination: Are we repeatedly implementing timers, retries, worker ownership, state transitions, and failure handling?

Visibility: Can we understand where a business process is stuck and what needs to happen next?

Maintenance: Is more engineering effort going into maintaining execution machinery than implementing the actual business workflow?

These questions help us identify when coordination is becoming a system of its own.

And there's an important trade-off here. Introducing a workflow engine doesn't eliminate complexity. It moves much of the generic execution and recovery machinery into a dedicated platform, while adding its own infrastructure, operational costs, and programming constraints.

For a straightforward payment lifecycle, PostgreSQL, a queue, and a reconciliation job may remain the better choice. For a multi-day onboarding process involving several external systems, approvals, timeouts, and compensating actions, maintaining our own coordination infrastructure may become harder to justify.

The point isn't to replace simpler tools as soon as workflows become complicated. It's to recognize when we're repeatedly building infrastructure that a dedicated workflow engine already provides.

This is the architectural problem that systems such as Uber Cadence, Temporal, and AWS Step Functions were designed to address.

Their common idea is surprisingly simple: the lifetime of a business process shouldn't depend on the lifetime of the worker executing it.

The interesting part is how they make that possible.

How Durable Execution Actually Works

The core idea behind durable execution is to separate the progress of a business operation from the process executing it.

Instead of depending on a worker's memory to remember everything, systems like Temporal and Uber Cadence maintain a durable execution history. This history records important events, such as when an activity was scheduled, when it completed, what result it returned, and when a timer was created.

Let's return to our order-processing example.

Suppose inventory has been reserved and payment has been completed. Before the workflow can schedule shipment creation, the worker crashes.

In an ordinary application, we would need our own recovery logic to determine the next step. With Temporal, the workflow's execution history has already recorded the completed activities, assuming their results reached the engine successfully.

Now another worker can take over.

### Workflow: order-123

1. Workflow started
2. ReserveInventory scheduled
3. ReserveInventory completed
4. ChargePayment scheduled
5. ChargePayment completed

### Worker A crashes

6. Worker B receives a workflow task
7. Previously recorded results are replayed
8. Shipment creation is scheduled

The interesting part is how Worker B knows where to continue.

Temporal doesn't restore the original JVM thread or recover its memory from a snapshot. Instead, it uses deterministic replay.

The replacement worker runs the workflow code against the recorded execution history. When it encounters an activity that has already completed, the SDK uses the stored result rather than executing that activity again. As the code progresses, it reconstructs the workflow's in-memory state until it reaches the point where new work is required.

This is why Temporal distinguishes between workflows and activities.

A workflow contains the coordination logic: which operation should run, what should happen after it completes, and which conditions determine the next step. Activities perform the actual work, such as calling a payment provider, updating a database, or invoking another service.

That separation makes replay possible. Workflow decisions must remain compatible with the recorded history, while external side effects happen through activities whose outcomes can be persisted.

It also introduces a constraint. If workflow code makes decisions using uncontrolled randomness, system time, or direct external calls, replay might follow a different path. Temporal provides replay-safe mechanisms for these situations, but developers must still design workflow code with determinism in mind.

The same architecture supports long-running waits. A workflow waiting three days for approval doesn't need a worker thread sleeping for three days. The engine records the timer or waiting state and schedules more work when the appropriate event occurs.

But there's an important limitation.

Durable execution does not guarantee that an external operation happens exactly once.

If a payment succeeds but its completion never reaches the workflow engine, the activity may be retried. The payment operation still needs idempotency, and some failures may require reconciliation or compensation.

What the Engine Handles, and What We Still Own

This distinction is important. The engine can preserve execution progress, retry failed activities, and coordinate recovery. But it cannot determine whether an external payment was successfully processed unless that outcome is reported or reconciled. Nor can it decide what a correct business compensation should be. Those responsibilities still belong to the application.

This is the trade-off behind the abstraction. A workflow engine takes responsibility for durable coordination, scheduling, and recovery, but it doesn't eliminate the need to design correct business operations.

And making that infrastructure reliable introduces its own engineering challenges. Recording history is only one part of the problem. The engine must also ensure that state changes and the tasks they create cannot become inconsistent.

That is where the architecture of real workflow platforms becomes particularly interesting.

What Production Engineering Teaches Us

Understanding how durable execution works is one thing. Seeing the trade-offs engineers faced while building and adopting these systems makes the architecture much more interesting.

Two examples stood out to me while exploring Uber Cadence and Temporal.

Uber Cadence: When Orchestration Adds Latency

In 2019, Uber published an engineering case study about redesigning its customer-support ticket-routing system using Cadence.

The original routing logic had grown complicated over time. Matching tickets to support agents involved language, location, issue type, priority, and several business-specific rules. As those requirements increased, the logic became scattered across different parts of the codebase, making it harder to test, modify, and debug.

Uber moved toward a workflow-based architecture, separating the routing process into smaller activities. This made the control flow easier to understand and allowed individual business operations to evolve more independently.

But this introduced another problem.

Cadence normally distributes activities through task queues, allowing workers to execute them independently. That flexibility comes with communication and scheduling overhead. Uber reported activity-to-activity latencies of around 100 milliseconds, which mattered for a system expected to assign tickets quickly.

So the team made a deliberate trade-off.

They used local activities for the core routing steps. A local activity is a short-running operation executed inside the same worker process as the workflow, rather than being dispatched through the normal activity task queue.

This avoids the additional scheduling round trip and reduces latency. However, because local activities run in the workflow worker process, they lose the flexibility of being deployed and scaled independently.

For Uber, that trade-off made sense. The core routing operations were latency-sensitive, while extensible scoring modules still benefited from independently hosted activities. So the team used local execution where speed mattered and retained distributed execution where flexibility was important.

The team reported more than a 33% improvement in routing-workflow p99 latency compared with its previous implementation. However, starting the workflow introduced additional latency, so the end-to-end latency observed by dependent services remained roughly comparable to the original system.

That's an important distinction. The improvement wasn't free, and the workflow engine didn't automatically make everything faster.

The architectural lesson is that abstraction boundaries have a cost. Distribution improves flexibility and isolation, but for latency-sensitive operations, keeping selected work local may be the better decision.

Source: Uber Engineering, Improving the User Experience with Uber's Customer Obsession Ticket Routing Workflow and Orchestration Engine (2019)

Temporal: Persisting State Isn't Enough if the Next Task Can Disappear

Temporal's architecture reveals a different kind of challenge.

Suppose a workflow completes payment and decides to schedule shipment creation.

The workflow engine must durably record that decision and ensure the shipment task eventually reaches a worker. If recording the workflow's progress and dispatching the task happen as two unrelated operations, a crash between them can create an inconsistency.

The engine might record that shipment creation was scheduled, but the task itself might never reach a worker.

This is similar to the dual-write problem we encounter when building event-driven applications with databases and message brokers.

Temporal addresses this through a mechanism closely related to the transactional outbox pattern.

When a workflow schedules an activity, Temporal persists the workflow's updated execution state and a transfer task together in the same persistence transaction.

A transfer task is an internal, durable record indicating that work needs to be dispatched. It is not yet the activity task delivered to an application worker.

A background processor reads these transfer tasks and forwards the work to Temporal's Matching service, the component responsible for connecting pending tasks with available workers.

If the transfer fails, the durable record remains available for another attempt. The transfer may happen more than once, so the engine also needs to handle duplicate dispatch safely.

The key idea is that the workflow's updated state and the intent to schedule its next task are committed together. A failure during task delivery no longer means that the intent to perform the work has disappeared.

This is the same principle behind a transactional outbox: persist the state change and the record of future work atomically, then deliver that work asynchronously.

I find this particularly interesting because it shows that a workflow engine has to solve the same distributed-systems problems that application developers face.

A system cannot provide reliable workflow recovery if it can lose the tasks required to continue execution.

Durable execution is therefore more than storing workflow history. Underneath the abstraction, the engine must coordinate state transitions, task scheduling, timers, and recovery while preserving consistency across failures.

Sources: Temporal, Designing a Workflow Engine from First Principles (2021) and Temporal's Workflow Lifecycle Architecture.

What These Decisions Tell Us

Uber's example shows that introducing orchestration can improve how complex logic is organized while creating new performance trade-offs. Temporal's design shows how much consistency machinery is required underneath an abstraction that appears simple to application developers.

Neither system makes distributed failures disappear. Instead, they make particular failure and coordination problems manageable through infrastructure designed to handle them.

That is the value of durable workflow execution, but it also raises the final architectural question: when does adopting that infrastructure become more valuable than continuing to maintain our own coordination mechanisms?

When Not to Introduce a Workflow Engine

Consider a payment service that stores transaction state in PostgreSQL, processes asynchronous events through Kafka, and uses a scheduled reconciliation job to recover uncertain payments.

If this architecture already handles failures reliably, provides enough visibility, and remains manageable for the team, what exactly would introducing Temporal improve?

A workflow engine could provide built-in execution history, durable timers, and recovery mechanisms. But it would also introduce another infrastructure dependency with its own operational requirements and failure boundaries.

Self-hosting means maintaining the workflow services and their persistence infrastructure. A managed offering reduces that burden but introduces additional costs and platform dependencies. Code-first engines like Temporal also require developers to understand deterministic replay, workflow versioning, and how code changes affect executions that are still running.

And importantly, none of this removes the need for application-level idempotency, reconciliation, or correct compensation logic.

Start with the Problem, Not the Infrastructure

A single background job may only need a queue and a worker. A straightforward business lifecycle might work well with a database state machine. A scheduled recovery job could be sufficient for handling delayed or incomplete operations.

There's no reason to replace these approaches simply because a more sophisticated abstraction exists.

On the other hand, consider an onboarding process involving multiple external services, human approvals, long waiting periods, and compensating actions. If the team is repeatedly implementing and maintaining recovery logic for these operations, a workflow engine becomes a much more compelling option.

The same consideration is becoming relevant for long-running AI agents, where execution can involve external tools, unreliable dependencies, and human intervention. But even there, the presence of multiple steps doesn't automatically justify a durable execution platform.

The decision ultimately comes down to a trade-off: Is maintaining our own coordination and recovery infrastructure becoming more costly and difficult than adopting a dedicated workflow engine?

The Bigger Architectural Lesson

What I found most interesting while exploring durable execution wasn't a particular Temporal feature or Cadence implementation detail. It was understanding why this infrastructure exists and the engineering decisions required to make it reliable.

Uber's experience shows how orchestration can improve modularity while introducing performance trade-offs. Temporal's architecture reveals how much work happens underneath a seemingly simple promise: that execution can continue despite failures.

Both are reminders that abstractions don't make complexity disappear. They move it to a different layer, where it can be managed more consistently.

For me, that's the real takeaway from studying durable workflow execution.

The goal isn't to end up using a workflow engine. It's to understand who owns the progress of a business operation when something fails, and to choose the simplest architecture that can recover it reliably.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️

Think Beyond LLMs: Choosing the Right Model for the Right Job

NARESH — Thu, 29 Oct 2026 07:29:49 +0000

An experiment-backed journey through specialized AI models, practical benchmarking, and intelligent system design.

Just because an LLM can perform a task doesn't necessarily mean it's the right model for the job.

We're already familiar with model routing, where simpler requests go to smaller, faster LLMs while more complex problems are handled by capable models. It's a practical way to balance cost, latency, and performance. But as I started exploring specialized AI models over the past few weeks, one question kept coming back to me: Why are we limiting our choices to LLMs? If all we need is to classify a support ticket, extract information from an invoice, or make a specific decision, could another approach handle it more efficiently?

That curiosity led me toward task-first AI design, where we choose the right capability based on what the application needs, whether that's a specialized model, an LLM, retrieval, or ordinary backend logic.

Over the past two weeks, I've been experimenting with Jev, ScaleDown's Constrained Decoding Models (CDMs), and an LLM hosted through NVIDIA NIM. I built a benchmarking lab and ran 900 inference requests across 100 support tickets, comparing accuracy, latency, reliability, and cost. Some results were surprising, especially when I looked beyond whether a model returned the correct answer.

In this article, I'll share what I learned from those experiments and how they're shaping my approach to future AI projects. We'll also walk through the design of an intelligent customer-support system to explore where these different capabilities belong and whether combining them actually makes engineering sense.

Beyond General-Purpose LLMs: Meet Jev and CDMs

Let's take a simple customer-support message: "I was charged twice for my subscription." Our application needs to identify the issue and route it to the right team. The expected output is just billing.

A general-purpose LLM can handle this easily. We provide the message, define the available categories, and ask it to return the correct one. Under the hood, the model generates its response token by token. We can also constrain its output to a predefined JSON schema. But when the application only needs a category, it's worth exploring models designed specifically for that kind of work.

Jev: Built Around Decisions

Jev, developed by TypeSafe AI, takes a different approach to interacting with AI. Instead of asking it to generate a response, we provide some context and ask a specific question. It returns a structured decision that our application can use directly.

Jev supports three main types of judgments: Choice, for selecting among predefined options; Score, for evaluating something on an ordered scale; and Noul, for yes-or-no probability judgments. For our support ticket, we could use Choice to select between billing, technical, account, and general, along with the model's probability distribution.

This makes Jev interesting for routing, prioritization, and other bounded decisions. Of course, a structured decision can still be wrong, and its confidence score shouldn't be treated as guaranteed accuracy.

ScaleDown CDMs: Models for Specific Tasks

ScaleDown takes specialization further with its Constrained Decoding Models (CDMs) and task-specific APIs. Its Classify model maps inputs to predefined labels using descriptions or rubrics, while Extract identifies information from documents and Summarize condenses longer content.

Consider an invoice-processing workflow. Rather than asking a general-purpose LLM to read an invoice, identify its fields, classify the document, and summarize it in a single prompt, we could evaluate specialized operations for each requirement.

There's an important overlap here. Jev and ScaleDown Classify can both categorize support tickets, and general-purpose LLMs can produce constrained structured outputs too. Their interfaces and capabilities alone don't tell us which will be faster, cheaper, or more accurate.

That overlap is exactly what made me want to test them against the same workload.

I Tested All Three. The Results Challenged My Assumptions.

Before adopting these models in my projects, I wanted to understand how they performed under the same workload. A specialized model might sound promising on paper, but choosing one for a real application requires more than API documentation.

So I built a small benchmarking lab using Python and FastAPI to compare Jev, ScaleDown CDM, and NVIDIA NIM's Llama 3.2 11B Vision Instruct.

How I Set Up the Experiment

I created 100 synthetic customer-support tickets across four categories: billing, technical, account, and general inquiries. The dataset included straightforward requests, ambiguous cases, and adversarial inputs.

All three models received the same classification objective and category definitions, adapted to their respective APIs. I tracked successful predictions separately from failed requests, since a timeout tells us something different from an incorrect classification.

My Model Comparison Lab comparing three inference approaches on support-ticket classification. The dashboard's adjudicated accuracy excludes flagged ambiguous cases, so its percentages differ from the all-request results discussed below.

What Did the Results Tell Me?

The first result surprised me. All three models achieved 95% accuracy when their requests completed successfully. Jev and CDM each classified 285 of 300 requests correctly, while NVIDIA correctly classified 210 of its 221 completed requests.

The bigger difference appeared in latency and availability. Jev recorded a median response time of 316 ms, followed by CDM at 562 ms and NVIDIA at 764 ms. Jev and CDM completed all 300 requests, while NVIDIA encountered 79 timeouts, reducing its end-to-end correct-response rate to 70%.

There's an important limitation here. NVIDIA was running through a free hosted evaluation endpoint. During execution, I also reduced its timeout from 75 to 15 seconds and its retry allowance from two retries to one. Failed requests were tracked separately from completed-response latency. This makes the reliability and latency comparison exploratory rather than a controlled measurement of the models' inherent performance.

One adversarial case also caught my attention. A billing ticket contained an instruction to ignore the classification rules and return general. NVIDIA followed that instruction, while Jev and CDM correctly identified the billing issue. It's an interesting failure case, although it doesn't establish that specialized models are inherently more secure.

What About Cost?

Pricing was another factor I wanted to understand. Using the published rates in October 2026, Jev cost $0.042 per million input tokens, while ScaleDown Classify cost $0.05.

In my experiment, they consumed an average of 448.1 and 375.4 reported input tokens per classification, respectively. Using:

Estimated cost = (input tokens / 1,000,000) × price per million tokens

Both worked out to approximately $1.88 per 100,000 classifications. These are projected inference costs using the measured averages, not actual billing statements. NVIDIA's hosted evaluation endpoint was free, so I couldn't make an equivalent production-cost comparison.

What I Took Away

The experiment showed me that specialized models can be promising for bounded tasks, but accuracy alone isn't enough to justify adopting them. Interestingly, all three models also returned valid single-label outputs on their successful requests.

What mattered was how these capabilities behaved under the tested conditions and whether their advantages would translate into a larger application.

That led me to another question: If I were designing an AI system today, where would I actually use each of these models?

To explore that, let's design a customer-support system from the ground up.

Designing a Customer Support System Without Relying on LLMs for Everything

Let's put what I've learned into a practical system-design exercise.

Suppose we're building an AI-powered customer-support system for a subscription platform. A customer submits this request:

"I was charged twice for my subscription this month. I've attached my invoice. Can you check whether I'm eligible for a refund and explain what happens next?"

Our system needs to understand the request, extract relevant information, verify transactions, check the applicable refund policy, and respond to the customer. We also need to handle missing information and prevent unauthorized financial actions.

We could build the initial version using a general-purpose LLM with tool access and structured outputs. That might work perfectly well. But what if we assigned each responsibility based on what it actually requires?

Breaking Down the Responsibilities

1. Understanding the customer's intent

First, we need to identify whether the request concerns billing, technical support, account access, or something else. A decision-oriented model such as Jev, or a classifier such as ScaleDown CDM, could handle this.

We wouldn't necessarily use both. Our benchmark suggests either could be a reasonable candidate, depending on the workload. The classification result determines where the ticket goes, not whether the customer deserves a refund.

2. Extracting information from the invoice

Next, we need details such as the invoice number, billing date, subscription amount, and transaction reference.

A specialized extraction model could return these fields in a structured format. However, extracted values still need validation against the source document and payment records. A confidently extracted amount doesn't mean the transaction has been verified.

3. Retrieving the right information

Once we understand the request, the application needs access to the customer's transaction history and the relevant refund policy.

Conventional database queries and APIs can retrieve account information, while document retrieval can help locate applicable policy documents. We also need to consider policy versions, subscription plans, and regional conditions to avoid making decisions using outdated information.

4. Making the actual business decision

This is where I'd be careful about giving AI too much authority.

Refund eligibility should depend on verified payment records, authorization checks, and explicit business rules. A model may help flag suspicious activity or recommend human review, but its prediction should remain one signal rather than the final authority.

If the system cannot verify a transaction or encounters conflicting information, it should request additional evidence or escalate the case instead of guessing.

5. Generating the customer response

Once the outcome has been established, a general-purpose LLM becomes useful again. It can turn verified findings into a clear explanation, reference the relevant policy, and guide the customer through the next steps.

The LLM handles communication, while the backend remains responsible for the decision and any authorized transaction.

Putting the Architecture Together

A proposed architecture informed by my classification experiments. Invoice extraction, policy retrieval, and the complete end-to-end workflow have not been benchmarked together.

Some operations could happen in parallel. For example, intent classification and invoice extraction can begin independently after input validation, avoiding unnecessary sequential processing.

We would also need fallback behavior. If extraction fails, the customer could be asked for missing details. If classification is uncertain, the ticket could enter a review queue. A failed model call shouldn't automatically prevent the customer from receiving assistance.

This design looks promising, but adopting every component would be premature without evaluating the complete workflow. Each additional model must justify the latency, operational dependencies, and cost it introduces.

That raises the next question: When does adding more models actually make the system worse?

When Adding More Models Makes Your Architecture Worse

Looking at the proposed architecture, there's an obvious temptation to give every task its own specialized model. Classification goes to Jev, extraction to CDM, and response generation to an LLM. It looks clean on a diagram, but would I actually deploy all of them?

Probably not without testing the complete workflow first.

Every additional model introduces another API dependency, network request, failure scenario, and operational cost. A classifier might respond in 300 milliseconds, but that doesn't mean the entire application becomes faster. If several inference calls depend on one another, their latencies accumulate. Even when we execute independent tasks in parallel, the workflow may still be delayed by the slowest required operation.

The simpler alternative deserves a fair comparison. For a small support application, a single LLM with structured outputs, a database, and well-defined business rules might already meet our requirements. Adding specialized models would make sense only if they measurably improved something important, such as classification accuracy, extraction reliability, latency, or cost.

There are also failure scenarios to consider. What happens if the extraction model is unavailable? Can we fall back to another model, or should we ask the customer for the missing information? If the classifier is uncertain, can we route the request for human review? These decisions affect the reliability of the complete application more than the performance of any individual model.

I'd also avoid sending the same request through multiple models just because they're available. Running Jev and CDM for the same classification adds little value unless their combined results improve a measured outcome.

My approach would be to start with the simplest architecture that satisfies the requirements, establish a baseline, and introduce specialization where the evidence justifies it.

The benchmark helped me understand individual model behavior. A production decision would require another level of evaluation: measuring the complete workflow under realistic traffic, including failures and recovery.

Ultimately, the number of models in an application tells us very little about how well it is engineered. What matters is whether each component contributes enough value to justify the complexity it brings.

A Practical Framework for Choosing the Right AI Model

After these experiments, I started reconsidering how I approach model selection. Rather than choosing an LLM first and designing the workflow around it, I'd begin by understanding what each task actually requires.

Here are four questions I want to carry into my future projects.

1. What kind of output does the task need?

Can ordinary code handle it reliably? Does it require a bounded decision, structured extraction, or open-ended generation? The output requirements should guide which approaches are worth considering.

2. What happens if the result is wrong?

Misclassifying a support ticket might delay resolution, while an incorrect financial decision could have serious consequences. The impact of failure determines the validation, safeguards, and human involvement required.

3. How do I know the model is good enough?

I'd evaluate it against representative inputs and measure accuracy, latency, reliability, and cost. More importantly, I'd examine where it fails, not just how often it succeeds.

4. Is the additional complexity worth it?

I'd compare the proposed approach with a simpler baseline. If a specialized model offers a meaningful improvement for the workload, it earns its place. Otherwise, I'd avoid introducing another dependency.

Final Thoughts

When I started exploring Jev and CDMs, I wanted to understand whether specialized models could outperform general-purpose LLMs. The experiments gave me something more valuable: a better way to question my architectural choices.

Going forward, I want to apply the same engineering discipline to AI models that I would to databases, caching, or messaging systems. Understand the workload, test the alternatives, and defend the trade-offs.

The next time I design an AI system, I want to explain not only what each model does, but why it deserves a place in the architecture.

That's what thinking beyond LLMs means to me.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️

Git at Agent Scale: What Anthropic's Scion Really Teaches Us

NARESH — Sun, 18 Oct 2026 18:23:07 +0000

For the past month, one question has stayed with me: AI coding agents are making it faster and easier to produce code, but can the infrastructure surrounding it keep up? Every change still needs to pass through source control, testing, review, and eventually deployment.

A few days ago, Anthropic published an engineering article explaining how they migrated their monorepo from Git to an internally developed version-control system called Scion. What caught my attention was the architectural reasoning behind that decision.

Since then, I've seen discussions suggesting that Git is becoming obsolete and Scion represents the future of source control. But I think there's a more interesting lesson here. Git's architecture hasn't suddenly become ineffective. Anthropic's workload has evolved to a point where some of Git's design choices have become increasingly expensive.

In this article, I want to explore what pushed Anthropic toward this decision, how Scion addresses those challenges, and what other engineering teams can learn from the trade-offs.

Because as AI changes how quickly we produce software, understanding the infrastructure supporting that process becomes just as important.

Understanding What Git Was Designed For

Before exploring Anthropic's decision, it's worth understanding what made Git so successful in the first place.

Git was designed as a distributed version-control system. A normal full clone doesn't just download the latest source files. It also brings the repository's history, allowing developers to create commits, inspect previous versions, manage branches, and merge changes locally without constantly communicating with a remote server.

One of Git's strengths lies in how efficiently it stores different versions of a project.

How Git Stores Source Code

Git uses a content-addressed object model built around three important objects:

Blob: Stores file contents without the filename.

Tree: Represents a directory, containing filenames and references to blobs or other trees.

Commit: References a root tree representing a project snapshot, along with parent commits and metadata.

Each object has an identifier derived from its contents and object type. This allows Git to reuse unchanged objects across multiple commits.

Consider a Java project containing README.md, Main.java, and PaymentService.java. We create an initial commit, then modify only PaymentService.java and commit again.

Here's what happens internally:

Git's object model: both commits can reference the same unchanged file blobs, while modified content receives a new object.

The second commit creates a new blob for PaymentService.java, along with the necessary updated tree objects and a new commit object. Meanwhile, README.md and Main.java continue referencing their existing blobs.

Each commit represents a complete logical snapshot, but Git doesn't physically duplicate every unchanged file.

We can inspect this behavior directly:

git cat-file -p HEAD
git ls-tree -r HEAD~1
git ls-tree -r HEAD

The first command displays the latest commit's internal information. The other two show file objects referenced by the previous and current commits. Comparing their blob IDs reveals which contents changed and which were reused.

How This Connects to Git's Distributed Architecture

When we run git add, Git stores staged file contents as objects and updates its index. Running git commit creates the necessary tree and commit objects, then updates the current branch reference. These operations happen locally.

When synchronizing with a remote, Git uses its transfer protocols to exchange the required objects, often through compressed packfiles that reduce storage and network overhead.

This architecture gives developers substantial independence while keeping repository storage efficient.

However, that independence comes with a trade-off. Clients maintain repository data locally and perform many source-control operations themselves. For most teams, this works remarkably well. But when millions of short-lived automated environments repeatedly interact with an enormous, rapidly changing monorepo, those costs can accumulate.

That's where Anthropic's workload becomes interesting.

Where Git Started Struggling at Anthropic

Anthropic's scaling challenge becomes clearer when we look at the numbers.

In May 2026, Anthropic was merging around 2,500 code changes per day. By September, that had grown to over 30,000. Its monorepo contained nearly 100 GB of main-branch history, with a compressed working copy exceeding 25 GB.

But the number that caught my attention was the approximately 25 million code checkouts happening every day for CI alone, excluding developer environments, agent sandboxes, and research workloads.

Consider what happens when an AI coding agent starts a task. It needs source files, possibly some repository history, and a workspace to make and validate changes. Preparing that environment can involve transferring Git objects, processing compressed packfiles, materializing files on disk, and maintaining local repository state.

Now multiply those operations across millions of automated checkouts.

Anthropic estimated that naively fetching its repository for every CI checkout would generate over 600 PB of Git traffic daily. This was a hypothetical estimate, not actual measured network usage, but it illustrates why efficient repository access had become so important.

Why Existing Git Optimizations Weren't Enough

What makes this case interesting is that Anthropic had already used the optimizations engineers would typically consider.

Git provides several mechanisms to reduce repository overhead:

Shallow clones reduce the amount of commit history downloaded.

Sparse checkouts limit which files are materialized locally.

Partial clones allow certain objects to be fetched only when needed.

Anthropic also maintained internal Git mirrors to distribute traffic and used prebuilt repository snapshots, called seeds, stored in blob storage. New environments could start from these snapshots instead of fetching everything from scratch.

These optimizations helped, but their operational costs grew alongside the workload. At its peak, Anthropic operated several hundred Git mirrors, putting increasing pressure on storage and compute resources. Meanwhile, seeds fell further behind the latest commits between refreshes, and history-dependent operations such as diff and blame required additional care with partial clones.

The underlying challenge was the accumulated cost of preparing and maintaining repository state across an enormous number of clients.

That led to an interesting architectural question. Most of Anthropic's clients were already-connected agents and automated jobs, many requiring only a few files, a diff, or some repository history.

If these clients are already online, do they really need to maintain extensive local repository state and perform so many source-control operations independently?

That question became central to Scion's design.

How Scion Approaches the Same Problem Differently

Anthropic's approach was to reconsider where source-control operations should happen. Since most of its agents and automated jobs are already connected to shared infrastructure, maintaining extensive repository data on every client offers fewer benefits.

Scion addresses this through three architectural decisions: moving computation to servers, recording changes incrementally, and distributing repository data through shared caches.

Moving Source Control Operations to the Server

In Git, operations such as commits, merges, and history queries can run locally. Scion shifts these responsibilities to shared servers, while clients manage working files and maintain a bounded local object cache.

Consider an AI agent modifying a Java service. It may need only a few directories from an enormous monorepo. Scion supports sparse and on-demand file retrieval, avoiding the need to maintain extensive repository history locally. Some automated workflows can even request diffs or perform writes through APIs without creating a working copy.

This reduces duplicated work across clients, but also makes them more dependent on shared services.

A Different Way to Represent Changes

Scion also changes how repository updates are recorded.

While a Git commit references a root tree representing a project snapshot, Scion uses changesets, entries in a write-ahead log (WAL) that record files changed relative to their first parent, with content-addressed references to compressed file contents stored separately.

Changesets are created server-side without requiring a shared branch reference to move. References such as main are updated separately, reducing the coordination required for ordinary changeset creation.

But there's a trade-off. A changeset records what changed; it cannot efficiently answer every history-related question on its own.

To handle operations such as file history, blame, and commit ancestry, Scion builds additional derived data structures. These can be computed ahead of time or generated when needed, allowing the underlying change records and query indexes to evolve independently.

Scaling Through Distributed Caching

The next challenge is making repository objects available to millions of workloads without repeatedly accessing authoritative storage.

Scion uses a layered caching architecture:

Scion's checkout architecture: node-local and regional caches serve requested objects, reducing repeated access to authoritative servers.

When a client requests an object, the node-local scion-daemon checks its cache. Missing objects are fetched through regional scion-cache servers, which contact authoritative Scion servers when necessary.

Because objects are content-addressed and immutable, cached copies can be safely reused. Scion also requests specific trees and blobs rather than relying on Git-style, client-specific packfile transfers. Combined with cache locality and shared object reuse, this reduces repeated network and processing work.

Anthropic reports cache hit rates above 99% at each tier, a 200× smaller CPU and memory infrastructure footprint compared with its previous Git mirroring system, and CI and research checkouts that became roughly two orders of magnitude faster. These are results measured in Anthropic's environment, rather than general benchmarks against Git.

What makes the architecture interesting is how these decisions complement each other. Lightweight clients reduce local overhead, incremental changesets make writes inexpensive, and shared caches reduce repeated data retrieval.

However, moving more responsibilities to shared infrastructure introduces another challenge: what happens when those services or caching layers fail?

The Trade-offs Behind Scion's Design

Scion reduces repeated work across clients, but moving source-control operations to shared servers also introduces new dependencies.

With a complete local Git repository, developers can continue committing changes, inspecting history, and managing branches even when the remote server is unavailable. Scion trades some of that independence for lightweight clients that depend on shared services. For Anthropic's largely online workloads, this may be worthwhile, but it makes service availability and failure isolation particularly important.

What Happens When a Regional Cache Fails?

Consider thousands of CI workers retrieving source files through a regional cache. Under normal conditions, most requests are served from cached objects, with relatively few reaching the authoritative servers.

Now suppose that regional cache becomes unavailable. If every worker bypasses it and directly contacts the authoritative servers, the sudden traffic increase could overwhelm them, turning a regional failure into a system-wide outage.

Anthropic addresses this risk by strictly enforcing its caching hierarchy to isolate failure domains. However, the article doesn't describe its complete recovery strategy.

If I were designing a similar system, I'd consider admission control to limit upstream traffic, request coalescing to avoid duplicate fetches for the same object, and bounded retries with exponential backoff and jitter. Healthy local caches could continue serving available objects, while excess requests would be delayed or rejected to protect the authoritative service.

These are additional design considerations, rather than confirmed details of Scion's implementation.

There's also the broader operational cost. Anthropic had to build supporting systems for code review, search, and CI, introducing responsibilities around migration, maintenance, integrations, and long-term ownership.

This raises an architectural question worth considering: when do the performance and scalability benefits justify the complexity of operating an entirely different source-control system?

The answer becomes clearer when we compare organizations with different workloads.

Should Every Company Rethink Git? Three Engineering Scenarios

After studying Anthropic's decision, I started wondering how the same architectural choices would apply to companies operating at different scales. Would a startup benefit from Scion? What about a growing product company with increasingly expensive CI pipelines?

Let's consider three hypothetical scenarios.

Company A: A Small Engineering Team

A startup with 20 developers working across a few backend services probably has little reason to replace Git. Its local workflows, established tooling, and manageable repository sizes make Git a practical choice. Building a custom source-control system would introduce complexity without solving a significant bottleneck.

Company B: A Growing Product Company

Consider a company with 500 engineers, a large monorepo, and thousands of daily CI executions. Checkout latency and infrastructure costs are increasing.

I'd first investigate shallow and partial clones, sparse checkouts, shared caches, and improvements to CI workflows. Measuring network usage, CPU consumption, and checkout latency would help determine whether these optimizations are sufficient before considering a larger architectural change.

Company C: An Agent-Heavy Engineering Organization

Now consider an organization running thousands of coding agents and millions of repository operations against an enormous monorepo.

If most workloads are short-lived, continuously connected, and require only portions of the repository, maintaining extensive local state becomes increasingly expensive. A Scion-like architecture with shared source-control APIs and distributed object caches may be worth evaluating.

Even at this scale, the decision depends on whether the performance gains justify migration costs and long-term operational complexity.

Choosing Based on the Workload

An approach to source-control decisions: measure the bottleneck, optimize the existing system, and consider architectural changes when those improvements are no longer sufficient.

This is how I'd approach the decision: understand the workload, identify measurable bottlenecks, and choose the least disruptive architecture that meets the requirements. Anthropic's solution makes sense for its particular constraints, but similar benefits may be achievable with much smaller changes elsewhere.

What I Took Away From Anthropic's Decision

What stayed with me after reading about Scion was the reasoning behind Anthropic's decision. They explored the optimizations available within Git, identified where those approaches were becoming difficult to scale, and eventually reconsidered the architecture itself.

This also made me think about what happens after source control becomes faster. AI agents can generate thousands of changes, but those changes still need to be tested, reviewed, integrated, and deployed safely. Improving one part of the development pipeline can expose bottlenecks elsewhere. At some point, CI capacity, merge coordination, or validation may become the next constraint.

I'm also curious to see how Git and its ecosystem evolve. Google and Meta have already explored source-control architectures for enormous repositories, and Anthropic's experience highlights another demanding workload. As AI-assisted development becomes more common, I wouldn't be surprised to see improvements in server-side APIs, object retrieval, and caching within the broader Git ecosystem. Perhaps many teams will benefit from these ideas without ever needing to replace Git.

For me, the biggest lesson is to question the assumptions behind our infrastructure as workloads evolve. Sometimes the right decision is a targeted optimization. Other times, the scale of the problem justifies rethinking the architecture entirely.

And that's the question I'll carry into future system design decisions: if our workload grows tenfold, which parts of our architecture will struggle first, and are we prepared for that?

For a deeper look at Scion's implementation and Anthropic's reported performance improvements, I'd recommend their original engineering article: Redesigning Source Control at Anthropic.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️

Designing a Production RAG Retrieval System for 500 Million Vectors

NARESH — Sun, 04 Oct 2026 18:42:10 +0000

Designing a Production RAG Retrieval System for 500 Million Vectors

Assume there is a fictional company called Atlas.

Atlas started as a fairly typical enterprise RAG product. Companies uploaded internal documents, Atlas converted them into embeddings, stored them in a vector index, and used retrieval to provide relevant context to an LLM. With around 10 million vectors, the architecture was simple enough to operate and fast enough for the traffic it was receiving.

Then the product started growing.

Larger customers brought millions of documents with them. Existing customers began indexing more of their internal knowledge. Uploads were happening throughout the day while search traffic was increasing at the same time. Atlas also had to keep tenant data isolated, respect document permissions, handle updates and deletions, and make newly uploaded information searchable without making users wait for a large indexing job to finish.

The team estimated that the retrieval layer could eventually reach around 500 million vectors.

They could have responded by choosing a larger vector database and hoping it scaled. That would avoid the more important question.

What would the system itself need to look like at that scale?

In this article, I want to design Atlas the way I would approach a real system design problem. We will start with the architecture that works today, push it until something fails, and introduce a new component only when that failure gives us a reason to.

By the end, those individual decisions will come together into a complete production retrieval architecture for Atlas.

Before Designing Anything, Lock the Requirements

Before changing the architecture, Atlas needs to be clear about what it is actually designing for.

For this system, I would start with a few working assumptions:

the corpus can grow to roughly 500 million vectors
documents are uploaded throughout the day
search traffic is much higher than write traffic
each request belongs to a tenant and may include metadata or ACL filters
newly uploaded documents should become searchable quickly
updates and deletes must not corrupt older versions
search should continue even if an individual node fails
retrieval quality matters, so lowering latency by aggressively sacrificing Recall@K is not acceptable

There are also two very different workloads hiding inside the same product.

The first is the write path:

Document
→ processed
→ embedded
→ stored
→ becomes searchable

The second is the read path:

Query
→ find relevant vectors
→ fetch the corresponding chunks
→ provide them to the LLM

Keeping those paths separate in our head is useful because they optimize for different things. Writes care about durability, freshness, and background indexing. Reads care about latency, recall, filtering, and avoiding unnecessary work.

With those requirements in place, we can now look at Atlas's current 10-million-vector setup and ask the first useful system-design question:

What breaks if we try to stretch the same architecture all the way to 500 million vectors?

What Atlas Looks Like Today

Before redesigning Atlas, it helps to see the version that already works.

At its current scale, Atlas does not need a very complicated retrieval architecture. A customer uploads documents, the system chunks them, generates embeddings, stores the vectors with metadata, and uses those vectors during retrieval to provide context to the LLM.

That is good enough when the corpus is still manageable, the working set fits comfortably within the available infrastructure, and the rate of change is not too high.

At this stage, Atlas can afford a relatively straightforward setup. Search runs against one logical retrieval layer, ingestion is continuous but still manageable, and the operational overhead stays low. There is no strong reason to introduce a router, shard management, tiered storage, or a more elaborate write path before the workload demands it.

That matters because this is how a lot of systems actually begin. They do not start with a full distributed retrieval architecture. They start with the simplest design that works, and they evolve only when a clear bottleneck appears.

For Atlas, the first bottleneck shows up when the current setup is stretched toward 500 million vectors.

Failure #1: One Search Unit Stops Making Sense

The current Atlas design is fine while the corpus is still around 10 million vectors. There is one logical vector store, the write path feeds into it, and the read path searches it directly.

The first serious problem appears when Atlas starts pushing toward 500 million vectors.

A 768-dimensional float32 vector takes about 3 KB. At 500 million vectors, the raw vector data alone is roughly 1.5 TB. Once we include the ANN structure, metadata, replicas, caches, and spare capacity, keeping the entire search system on one machine becomes difficult to justify.

So the first redesign is about capacity.

Atlas splits the corpus into shards and adds a router in front of them.

                 ┌→ Shard 1
Query → Router ──┼→ Shard 2
                 ├→ Shard 3
                 └→ ...

Each shard owns only part of the corpus and builds its own local search index. As the dataset grows, Atlas can add more shards instead of continuously scaling one machine vertically.

This also gives us a natural place to think about storage differently. The hottest search state can remain close to memory, while larger vector data and index structures can move to cheaper storage such as SSD.

At this point, Atlas has solved the first scaling limit: the corpus no longer has to fit inside one search unit.

But this redesign creates the next problem.

Those shards are optimized for search, while customers are still uploading documents all day. If every new vector forces Atlas to modify the optimized index immediately, the write path starts interfering with the read path.

Failures 1 and 2: Capacity Stops Scaling, and Writes Start Fighting With Reads

Atlas works well enough with its current architecture while the corpus is still around 10 million vectors. There is one logical retrieval layer, the write path feeds into it, and queries search it directly.

The first problem appears when Atlas starts moving toward 500 million vectors.

A 768-dimensional float32 vector takes about 3 KB. At that scale, the raw vectors alone are already around 1.5 TB. Once metadata, ANN structures, replicas, and spare capacity are added, keeping the entire retrieval system as one search unit becomes difficult to justify.

So the first redesign is about capacity. Atlas stops treating the corpus as one large index and starts partitioning it across shards, with a router in front to direct traffic to the right places. That gives the system a way to grow horizontally instead of relying on one increasingly expensive machine.

That solves the first bottleneck, but it exposes the next one.

Atlas is not indexing a static corpus. Customers are uploading documents throughout the day, which means the system is constantly producing new chunks, new embeddings, and new vector records. If every incoming write tries to update the fully optimized search structures immediately, ingestion starts competing with query traffic for CPU, memory, and I/O.

At that point, Atlas needs to separate accepting a write from fully optimizing it for search.

The write path becomes more structured. A document is chunked, embedded, and written with metadata and version information into a durable log. From there, the new vectors land in a mutable segment, where they can become part of the searchable corpus quickly. The heavier index construction work happens later, in the background, when that data is turned into an optimized segment.

This gives Atlas a much healthier shape.

Sharding solves the capacity problem. The mutable path solves the write-contention problem. Atlas no longer has to keep the entire corpus on one machine, and it no longer has to rebuild or heavily mutate its read-optimized structures for every upload.

That leaves the next question.

If recent data is living in mutable segments while older data is already in optimized segments, how should the read path search both without sacrificing freshness or wasting work?

Failures 3 and 4: Fresh Data Gets Missed, and Global Search Becomes Wasteful

Atlas now has shards, mutable segments for recent writes, and optimized segments for older data.

The next problem is freshness.

If a query searches only the optimized ANN segments, a document uploaded a few seconds ago may not appear yet. The write succeeded, but the background indexing step has not finished.

So each shard has to search both places:

Shard Query
→ Mutable Segment
→ Optimized Segment
→ Merge Local Candidates

That keeps recent writes visible without forcing Atlas to wait for full ANN optimization.

Once this works, another inefficiency becomes obvious.

Atlas is a multitenant product. A query from one customer should not search every vector in the platform. The request already carries useful context such as tenant ID, document permissions, and metadata filters, so Atlas can use that information before doing expensive retrieval.

The read path now becomes:

Query
→ Authentication + Tenant Context
→ Query Router
→ Relevant Shards
→ Mutable + Optimized Search
→ Local Candidates

For a small tenant, that may mean touching only a limited part of the cluster. A larger tenant can span several shards, but the router still avoids sending the request to places that cannot contain valid results.

ACLs and metadata filters are applied as part of that search strategy, not as an afterthought after scanning the entire corpus.

With these two changes, Atlas solves two different problems at once.

Searching mutable and optimized data preserves freshness. Tenant-aware routing and filtering reduce unnecessary fanout.

But once a query starts touching several shards, Atlas gets a new problem: every shard has its own idea of the best results. Someone now has to combine them into one global ranking.

Failure #5: Distributed Search Creates a New Bottleneck

Once Atlas starts routing a query to multiple shards, each shard returns its own local candidates.

That creates a new problem. The best result inside Shard A is only the best result inside Shard A. Atlas still needs to compare candidates from every participating shard before it can produce the final top results.

So the read path gets one more component:

The aggregator collects candidates from each shard, compares them using a common scoring strategy, and builds the global result set.

This works, but distributed search introduces a latency problem that did not exist before.

If a query touches ten shards, the response can be delayed by the slowest one. A single overloaded shard can push the p99 latency of the whole request higher even when the rest of the cluster is healthy.

Hot tenants make this worse. One customer may generate enough traffic to overload the shard group holding most of its data while the rest of the cluster remains underused.

Atlas now needs to think about routing and capacity together. Large tenants may need to be split across more shards, frequently accessed data may benefit from caching, and replicas can help absorb additional read traffic.

The important point is that sharding solves the storage problem, but it creates a coordination problem.

By this stage, Atlas is no longer running several independent vector indexes. It is running one distributed retrieval system that has to produce a single answer from many local searches.

What Happens When Something Actually Fails?

So far, Atlas has been designed for scale. Now it needs to survive failure.

Suppose a shard accepts new vectors and crashes before those vectors are fully indexed. The WAL becomes the recovery point. A replacement process can replay the accepted operations, rebuild the mutable state, and continue indexing without asking the customer to upload the document again.

Search availability needs a separate answer. If one shard disappears, Atlas should not depend on that single copy of the data. Replicas allow another node to continue serving reads while the failed shard recovers.

There is also a less dramatic failure mode that matters just as much: indexing falls behind.

If documents arrive faster than Atlas can process them, the mutable segments and indexing queue keep growing. Allowing that backlog to grow without control eventually hurts search latency too.

So Atlas needs backpressure.

Ingestion rate > indexing capacity
→ queue grows
→ slow or limit new ingestion
→ protect search traffic

This is an important shift in the design. Reliability is not only about recovering from a crashed server. It is also about making sure one overloaded part of the system does not drag the rest of the retrieval path down with it.

At this point, Atlas has enough pieces to assemble the complete production architecture.

Putting It Together: The Atlas Architecture

Atlas now looks very different from the system we started with.

The original design had one simple retrieval layer. The production version has evolved into three cooperating parts: ingestion, retrieval, and distributed storage.

The ingestion side is responsible for turning continuously changing documents into searchable state:

Upload
→ Chunking
→ Embedding
→ WAL
→ Mutable Segment
→ Background Indexing
→ Optimized Segment

The retrieval side begins with the user's identity and tenant context, routes the query only to the shards that matter, searches both recent and optimized data, and combines the results:

Query
→ Query Embedding
→ Auth + Tenant Context
→ Shard Router
→ Shard-local Search
→ Candidate Aggregator
→ Global Top-K
→ Reranking
→ ACL Validation
→ Fetch Chunks
→ LLM

Underneath both paths sits the distributed storage layer.

Each shard owns part of the corpus, replicas provide redundancy, and different parts of the search state can live on different storage tiers depending on how frequently they are accessed.

The important part is that none of these components were added just to make the architecture look more sophisticated. Every one of them exists because the simpler version eventually hit a specific limit.

That is the architecture Atlas would now take into production.

Walk One Request Through the Final System

At this point, the architecture is easier to understand by following two real flows through it.

First, take a document upload.

A customer uploads policy.pdf. Atlas chunks the document, generates embeddings for those chunks, attaches metadata and version information, and writes the operation into the WAL. From there, the chunks enter a mutable segment so they can become part of the searchable corpus quickly. Later, the background index builder converts that data into an optimized segment and folds it into the longer-lived search structure.

Now take a search request.

A user asks, "What is our refund policy?" Atlas generates the query embedding, attaches the tenant and access context, and sends the request through the shard router. Only the relevant shards are searched. Inside each shard, Atlas looks at both the mutable segment and the optimized segment, then produces local candidates. Those candidates are sent to the candidate aggregator, which forms the global top results. The reranker refines that result set, ACL validation makes sure the returned chunks are actually visible to the user, and the final chunks are passed to the LLM.

What matters here is that the architecture is no longer one large vector index with a query attached to it.

Atlas is coordinating ingestion, recent searchable state, distributed candidate generation, global ranking, and final authorization checks in one retrieval flow.

That is the real shift from the original 10-million-vector design. Search is still the center of the system, but it is no longer the whole system.

Conclusion

Atlas did not reach its final architecture by starting with shards, WALs, replicas, mutable segments, candidate aggregators, and tiered storage.

It started with a simple retrieval system that worked at 10 million vectors.

Then the workload changed.

The corpus became too large for one search unit, so Atlas introduced sharding. Continuous ingestion started interfering with read-optimized structures, so it separated mutable state from optimized state. Fresh data had to remain visible, so queries searched both. Multitenancy made global search wasteful, so routing became tenant-aware. Distributed search produced local answers, so Atlas added candidate aggregation and global ranking. Reliability then pushed the design toward WAL recovery, replicas, and backpressure.

The useful lesson is not the final diagram itself.

It is the sequence of decisions that produced it.

Good system design usually works this way. Start with the simplest architecture that satisfies the current workload. When a real constraint appears, understand what is failing and why. Then add the smallest architectural change that fixes that specific problem.

For Atlas, the final system looks complex only when viewed all at once.

Taken step by step, every component has a reason to exist.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️

Scaling Vector Search to 500 Million Vectors: The Architecture Behind It

NARESH — Thu, 01 Oct 2026 18:58:09 +0000

A vector search system can feel surprisingly simple at the beginning. A document is split into chunks, each chunk becomes an embedding, and a query is converted into the same vector space so we can search for the nearest matches.

That model works well until the scale becomes large enough that the search algorithm is no longer the only thing that matters.

For this article, I want to take a hypothetical production system with around 500 million vectors, each with 768 dimensions. The system serves a large number of search requests while documents are continuously uploaded, updated, and deleted. It also has to support multiple tenants, metadata filters, access-control rules, and fresh data without letting query latency grow out of control.

At that point, the interesting questions start changing.

How much of the index can realistically stay in memory? How do new vectors become searchable while the existing index is serving traffic? What happens when an ACL filter removes most of the candidates returned by ANN? How do we delete or update vectors without constantly rebuilding the index? And once the dataset no longer fits comfortably on one machine, how do we distribute the search without destroying tail latency?

Those are the questions I want to explore here.

The similarity calculation itself remains familiar. The real engineering work begins with everything we have to build around it.

Start With Constraints, Not a Vector Database

The first architectural decision should not be whether to use HNSW, DiskANN, Pinecone, Qdrant, Milvus, or anything else. Before choosing any of them, I would write down the workload the system actually has to support.

For our 500-million-vector system, a few numbers matter immediately: the embedding dimension, expected read QPS, ingestion rate, p99 latency target, acceptable Recall@K, and how quickly a newly uploaded document needs to appear in search.

Then come the constraints that are easier to underestimate. How selective are the metadata and ACL filters? Are tenants roughly the same size, or can one customer own millions of vectors? How often are documents updated or deleted? Do searches need read-after-write consistency, or is a few seconds of freshness lag acceptable?

These answers change the architecture considerably.

A read-heavy corpus that changes once a day has very different requirements from a RAG platform receiving thousands of document updates every second. A system where almost every query searches the entire corpus behaves differently from one where every query is restricted to a small tenant and a strict ACL.

The storage budget matters too. Keeping a large index entirely in memory may produce excellent latency, but that design only makes sense if its cost is acceptable.

So before thinking about a particular vector database, I would reduce the problem to a set of measurable requirements:

corpus size and vector dimensions
read and write throughput
latency and recall targets
freshness requirements
filtering and tenancy model
update and deletion frequency
durability and consistency expectations
infrastructure budget

Once those constraints are clear, the next question becomes much easier: what part of our current search approach breaks first?

The First Thing That Breaks: Exact Search

At smaller scale, the most straightforward search strategy is also the most accurate one. Take the query vector, compare it against every stored vector, sort by distance, and return the nearest results.

There is nothing wrong with this approach. In fact, it gives us a useful ground truth because we know we are not skipping any candidates.

The problem is the amount of work.

With 500 million vectors and 768 dimensions stored as float32, the raw vector data alone is roughly:

500,000,000 × 768 × 4 bytes
≈ 1.536 TB

That is before adding metadata, index structures, replicas, caches, or operational headroom.

More importantly, an exact query would need to compare against all 500 million vectors. Doing that for one request is expensive. Doing it continuously under high QPS quickly becomes unrealistic.

This is where approximate nearest-neighbor search becomes necessary.

The useful way to think about ANN is not as a smarter similarity function. The distance calculation is still the same. The optimization comes from avoiding most of the corpus.

HNSW, for example, navigates through a graph of nearby vectors instead of scanning everything. IVF narrows the search to a few promising regions before comparing individual vectors.

Both are answering the same architectural question:

How much of the dataset can we safely avoid touching for each query?

That introduces the first real trade-off in the system.

Search fewer candidates and latency improves, but the probability of missing a true nearest neighbor increases. Search more candidates and recall improves, but query cost rises again.

So the system is no longer optimizing only for speed. It has to choose a point on the recall-versus-latency curve that matches the product requirement.

For a production system, that choice should be measured, not guessed. If our target is Recall@10, we can compare ANN results against exact search on a representative evaluation set and decide how much approximation we are actually willing to accept.

Filtering Is Where ANN Assumptions Start to Break

At this point, the search itself is faster, but production queries rarely ask for "the nearest vectors in the entire database."

They usually look more like this:

Find the nearest vectors
where tenant_id = 42
and document_type = "policy"
and language = "en"
and the user has permission to read them

That changes the problem.

Suppose ANN retrieves the top 100 candidates from the full corpus, but after applying tenant and ACL filters, only two of them are actually valid. The most relevant document for that user might exist somewhere deeper in the index, but because it never entered those first 100 candidates, the filter cannot recover it.

Increasing the candidate count can help, but it is not a complete solution. If the filter is extremely selective, we may end up searching thousands of irrelevant vectors just to find a handful of valid ones.

There are a few ways to handle this.

For a very small filtered population, it can actually be cheaper to apply the filter first and run an exact search over the remaining vectors. For larger populations, the ANN structure itself may need to understand the filter so that search spends more time exploring valid candidates.

This means the query planner needs some awareness of cardinality.

Very selective filter
→ filter first
→ search a small candidate set

Broad filter
→ ANN first or filter-aware ANN
→ return candidates efficiently

ACLs make this even more important. A result that is semantically perfect but not authorized is still unusable, so access control cannot be treated as an afterthought.

This was one of the parts of vector search that surprised me the most. ANN performance is not determined only by the index. The shape of the filters can completely change how that index behaves.

The Write Path Nobody Explains

Once documents are being uploaded continuously, vector search stops being only a read problem. The write path starts to matter just as much.

A new document may go through something like this:

Upload
→ Chunk
→ Generate embeddings
→ Attach metadata + version
→ Write to a durable log
→ Add to a mutable searchable segment
→ Build ANN structures in the background
→ Seal into an optimized segment

The durable log, usually a WAL or something serving the same purpose, is the safety net.

Before the system starts modifying indexes and in-memory state, it records the accepted operation somewhere durable. If the process crashes halfway through, that operation can be replayed after restart instead of being lost.

That leads to an important distinction:

Durable ≠ Searchable ≠ Fully Indexed

A write can already be safe from a crash while still waiting to appear in search. It can also become searchable before it has been added to the expensive ANN structure.

That second case is especially useful.

Suppose the existing 500 million vectors are already stored in optimized segments. A few thousand new vectors arrive. Rebuilding the large ANN index for every upload would make no sense.

Instead, those new vectors can first live in a smaller mutable segment. Because that segment is still small, the system may search it directly or with a lightweight index.

In the background, the database can batch enough new data, build the optimized ANN representation, and eventually seal it into another immutable segment.

Over time, multiple smaller segments may accumulate. Compaction then combines or rebuilds them so that query fanout, deleted data, and storage overhead do not keep growing forever.

The interesting part is that ingestion does not need to stop while this happens. New writes continue landing in the mutable path while older data remains available through optimized segments.

That is the basic pattern that lets a vector database stay writable without constantly rebuilding the entire search structure.

The Read Path Has to Understand Freshness

The write path creates two different search surfaces: recent mutable data and older optimized segments. The read path has to search both.

A production query might look like this:

Query
→ Generate query embedding
→ Apply tenant / shard routing
→ Search mutable + optimized segments
→ Merge candidates
→ Re-score promising results
→ Apply final ACL checks
→ Fetch chunks
→ Send context to the LLM

The important part is the concurrent search across recent and optimized data.

If the system searches only the optimized ANN segments, a document uploaded a few seconds ago may be completely invisible until background indexing finishes. Searching the mutable segment as well gives us better freshness without forcing every write to wait for ANN construction.

The results from those paths are then merged into one candidate set.

This is also where approximate search and quantization become easier to reason about. The first stage does not need to return the final ten results directly. Its job is to find a reasonably good pool of candidates quickly.

For example:

500M vectors
→ ANN search
→ 100 promising candidates
→ full-precision scoring
→ top 10

The second stage can afford to be more expensive because it is working on 100 vectors rather than 500 million.

There is one important limitation, though. If ANN fails to include the true nearest vector in that candidate set, the later re-ranking stage cannot recover it. That is why Recall@K remains an important metric even when we add a more accurate re-ranking step.

Freshness therefore becomes part of the read architecture itself. The query is not just asking, "Which vectors are nearest?" It is also asking, "Which parts of the index represent the latest searchable state?"

When RAM Stops Being the Database

By now, ANN has reduced how many vectors we examine per query. But there is another problem hiding underneath the search algorithm: where do we keep all of this data?

At 500 million vectors with 768 float32 dimensions, the raw vectors alone take roughly 1.536 TB. Once we add graph links, metadata, replicas, and operational headroom, keeping everything in memory becomes expensive very quickly.

This is where the architecture starts separating the representation used for search from the representation used for final scoring.

Instead of keeping every vector in full precision on the hottest storage tier, we can keep a compressed representation for candidate generation.

Compressed representation
→ find promising candidates quickly
→ fetch higher-fidelity vectors
→ re-score accurately

Scalar quantization and Product Quantization both reduce the amount of data that has to remain close to the search engine. The important architectural idea is not the compression technique itself. It is that the first stage only needs enough information to narrow 500 million vectors down to a much smaller candidate set.

That also changes how we think about storage.

Frequently accessed search structures may stay in RAM. Larger vector or index data can move to NVMe or SSD. Durable data can live even further down in object storage, with caches keeping the active working set closer to compute.

RAM
→ fastest, most expensive

NVMe / SSD
→ larger capacity, slower access

Object storage
→ cheapest durable layer, highest access cost

Disk-aware approaches such as DiskANN come from this same constraint. Once the dataset becomes too expensive to keep entirely in memory, the search algorithm has to be designed around the storage medium as well.

At this scale, retrieval performance is no longer only about the ANN algorithm. It is also about deciding which representation deserves the fastest and most expensive storage.

Updates, Deletes, and Model Changes Are All Migrations

A production vector index is rarely static. Documents change, permissions change, chunks disappear, and eventually the embedding model itself changes.

That creates a different class of problem from search.

For updates, I would avoid thinking in terms of modifying one vector in place. A document edit can change the chunk boundaries, the embedding, and sometimes the metadata. In practice, it is often safer to treat the new state as another version and make sure stale writes cannot overwrite a newer one.

Deletes have a similar pattern. The fastest way to remove a result from search is usually a logical delete first.

Mark as deleted
→ stop returning it
→ clean it up physically later

That physical cleanup may happen through compaction, vacuuming, or rebuilding affected index structures in the background. Rebuilding a 500-million-vector index every time one document is deleted would be far too expensive.

The biggest migration happens when the embedding model changes.

Suppose V1 uses one embedding model and V2 uses another. Even if both produce vectors with the same number of dimensions, their coordinate spaces may not be compatible. Mixing them in one index can make similarity scores meaningless.

A safer migration looks like this:

V1 index continues serving traffic

New writes
→ V1 embedding
→ V2 embedding

Existing documents
→ backfill into V2

Evaluate V2
→ switch reads
→ keep V1 for rollback
→ retire V1 later

This is essentially a blue-green migration for the retrieval layer.

The pattern is the same across all three cases: keep the current system correct, build the new state in parallel, and remove the old state only when the replacement is ready.

When One Machine Stops Being Enough

Even after reducing memory usage and moving colder data to cheaper storage, there is a point where one machine is no longer enough.

At that stage, the vector space has to be partitioned across multiple shards.

A query now looks more like this:

Query
→ Router
→ Search relevant shards
→ Each shard returns local candidates
→ Merge candidates
→ Produce global top-K

The local search on each shard is familiar. The harder part is deciding how much of the cluster each query should touch.

If every request fans out to hundreds of shards, the system may have plenty of total compute but still suffer from poor tail latency. The query only finishes when the slowest participating shard responds.

That makes routing important.

For a multitenant system, tenant-aware routing can avoid searching data that could never belong to the user. Large tenants may still need to be split across several shards, while smaller tenants can share capacity.

Replication solves a different problem. Sharding gives us capacity. Replication gives us additional copies for availability and read scaling.

Hot tenants are another practical concern. A cluster may look healthy on average while one shard is overloaded because a single customer is generating most of the traffic.

At this point, vector search has become a distributed systems problem. The ANN algorithm still runs inside each shard, but overall performance now depends just as much on routing, fanout, replication, load distribution, and the global merge.

How an Architect Actually Chooses the System

By this point, the question is no longer "Which vector database is the best?" The better question is "What does this workload actually require?"

I would start with a small set of numbers:

How many vectors are we storing, and at what dimension?
What are the read QPS and write throughput?
What p99 latency can the product tolerate?
What Recall@K do we need?
How quickly must new data become searchable?
How selective are tenant, metadata, and ACL filters?
How frequently do documents change or get deleted?
How much of the working set can we afford to keep in RAM?
What durability and consistency guarantees matter?

Those answers narrow the design space quickly.

A smaller, read-heavy workload may work perfectly well with a RAM-first HNSW index. A much larger corpus may need quantization and SSD-aware search. If the dataset is huge but only a small portion is hot at any moment, separating compute from durable object storage may make more economic sense.

The same applies to database choice. A team already running PostgreSQL may decide that vector support inside the existing database is enough for its scale. Another system with hundreds of millions of vectors, heavy filtering, and strict latency targets may justify a dedicated retrieval engine.

The important part is to benchmark using the actual workload.

That means testing realistic filters, ingestion rates, update patterns, concurrency, and recall targets, not just running an ANN benchmark on a clean static dataset.

A vector database should be the result of the architecture decision, not the starting point.

Conclusion

At small scale, vector search looks like a nearest-neighbor problem.

At 500 million vectors, that description is no longer enough.

The search algorithm still matters, but it now sits inside a much larger system. We have to think about selective filters, mutable and immutable data, write durability, freshness, candidate generation, storage hierarchy, updates, deletes, model migrations, sharding, routing, replication, and cost.

What I find interesting is that none of this changes the basic goal. We are still trying to find the vectors closest to a query.

What changes is how much engineering is required to make that operation practical.

ANN helps us avoid touching the entire corpus. Quantization reduces how much data has to stay on expensive storage. Mutable segments keep recent writes searchable while optimized indexes are built in the background. Compaction cleans up old state. Sharding lets the system grow beyond one machine. Routing prevents every query from becoming an expensive cluster-wide operation.

Each decision appears because the previous design eventually hits a physical or operational limit.

That is probably the most useful way to think about vector search architecture at scale. Start with the workload, identify what breaks first, and introduce complexity only when the constraint actually demands it.

The similarity equation stays almost the same.

The infrastructure around it is what keeps evolving.

In the next part, we'll take these constraints and design the system from the ground up starting with the workload, then building the query path, storage, indexing, write path, sharding, and scaling strategy around the problems we uncovered here.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️

I Thought Idempotency Was Enough: Redesigning a Payment Service for Real-World Failures

NARESH — Sun, 20 Sep 2026 18:50:34 +0000

I thought the hardest part of building a reliable payment service was making sure the same payment never ran twice.

For one of my microservices projects, I built the payment flow around an internal wallet. It had the pieces I cared about at the time: idempotency, pessimistic locking for concurrent payments, transactional wallet and ledger updates, asynchronous communication, and basic failure handling. Within a system I controlled end to end, the design felt solid.

Then I started looking at what changes when the money movement leaves that boundary.

One failure scenario completely changed how I looked at the architecture: the payment provider successfully charges the customer, but my service crashes before recording the success. My database still says the payment is unfinished, while the provider has already moved the money.

That is where I realized duplicate prevention was only one part of payment reliability. A payment can continue beyond a single request, finish asynchronously, and leave two systems with different versions of what happened.

So I went back to the design with a different question: if I had to build this payment service again for failures that cross system boundaries, what would I change?

This is that redesign.

The Payment Service I Started With

Before introducing any external payment provider, I built the service around an internal wallet. When an order reaches the payment service, it checks the customer's balance, deducts the amount safely, records the transaction, and publishes the result so the rest of the order flow can continue.

The flow looks roughly like this:

Each part of this design exists for a specific failure mode.

The payment request arrives asynchronously through Kafka, so duplicate delivery is something the service has to expect rather than treat as an exceptional case. Before executing the payment, the service checks whether that logical operation has already been processed.

For concurrent payments, the wallet is read using a pessimistic write lock. If two orders for the same customer arrive almost together, only one transaction can modify the wallet balance at a time. The second payment sees the balance left by the first instead of making a decision from stale data.

The actual debit and ledger entry then happen inside the same database transaction. If writing the ledger fails after the balance has been modified, the entire transaction rolls back instead of leaving the wallet and transaction history out of sync.

A successful movement leaves an auditable record such as:

Customer: C101
Order:    O123
Type:     DEBIT
Amount:   ₹500
Status:   SUCCESS

Once the transaction commits, the payment result is published back into the asynchronous order flow.

For the system I was experimenting with, these guarantees covered the important risks: duplicate processing, concurrent balance updates, partial database writes, and reliable communication between services.

But this architecture has one very useful property that is easy to overlook: the money movement still happens entirely inside infrastructure I control. The wallet, ledger, locking, and transaction boundary all belong to the same system.

The moment an external payment provider enters that flow, that assumption disappears.

Where the First Assumption Breaks

The original design works because the important financial state stays inside one system. The wallet balance, ledger entry, and payment result can all be coordinated within the same database transaction.

I intentionally kept the first version that way. The goal was to learn microservices and distributed-system behaviour, not to integrate a real payment provider.

The failure model changes the moment the payment crosses that boundary.

Imagine my payment service sends a ₹500 charge to an external provider. The provider completes the payment successfully, but my service crashes before saving the result.

Now both systems hold different versions of the same payment:

Payment provider:    SUCCEEDED
My payment service:  PROCESSING

From my database, the payment still looks unfinished. From the provider's side, the money has already moved.

This is where the guarantees from the first design stop helping. A local transaction can roll back changes inside my database, but it cannot undo an operation that has already completed in another system. The pessimistic wallet lock does not help either; there is no longer a shared row to lock. The uncertainty now exists across a network boundary.

The obvious reaction is to retry the payment.

But before doing that, I need to answer a much more important question:

Did the first attempt actually fail, or did I simply lose the response?

That question becomes the starting point for the redesign. Every external payment operation needs an identity that remains stable even when the request is retried, the connection fails, or my service restarts.

The First Fix: Give Every External Payment a Stable Identity

Once the payment crosses into a third-party provider, retries become unavoidable. A timeout does not tell me whether the provider never received the request, started processing it, or completed the payment and lost the response on the way back.

So the first thing I would add to the redesigned service is a stable idempotency key for every external payment operation.

For example:

Payment: PAY-1042
Amount: ₹500
Provider: Stripe
Idempotency-Key: pay_1042_attempt_1

If the connection drops, I should not generate a new key and create another operation. I retry the same request with the same key.

Stripe's behaviour is a useful reference here. Once endpoint execution begins, Stripe associates the idempotency key with the result of that request. A retry with the same key and the same parameters returns the previous status code and response body instead of executing the operation again. If the same key is reused with different parameters, Stripe rejects it.

Even a 500 is interesting here. Once execution has started, a server error does not necessarily mean that no side effect occurred. Replaying the stored result is safer than blindly assuming the payment never happened and executing it again.

The recovery path now looks much safer:

Request 1

PAY-1042 + ₹500 + K1
        ↓
Provider processes payment
        ↓
Response is lost

Request 2

PAY-1042 + ₹500 + K1
        ↓
Provider recognizes the operation
        ↓
Previous result is returned

This protects a different boundary from the idempotency I already had inside my own system. Local duplicate protection stops the same payment command from being processed twice by my services. Provider-side idempotency protects the call between my payment service and the system actually moving the money.

I need both because duplication can happen on either side of that boundary.

There is also an important limit to what an idempotency key can do. Stripe can remove idempotency records after they have been retained for at least 24 hours. If that key is later reused after it has been pruned, it can be treated as a new request.

That means an idempotency key is useful for making retries safe, but it cannot become the permanent identity of the payment.

A payment can live much longer than a retry window. The customer might switch payment methods, complete authentication later, or remain in a processing state while the provider finishes the transaction.

So the redesigned service needs something more durable than a request key.

A Payment Needs a Lifecycle, Not Just a Request

The next gap appears when the payment itself lasts longer than the API call that started it.

My original flow is simple: receive the payment request, validate it, process it, and return success or failure. That works well for an internal wallet because the operation is usually immediate and fully controlled by the payment service.

External payments are not always that clean. A card can be declined and replaced with another payment method. The customer may need to complete 3D Secure authentication. Some payment methods can remain in a processing state before the provider knows the final result.

At that point, I need to separate the payment itself from the individual attempts used to complete it.

Stripe's PaymentIntent model gave me a useful way to think about this. Instead of treating every attempt as a new payment, one persistent object represents the payment for an order while different attempts and state changes happen underneath it.

I would bring the same idea into the redesigned service with a durable Payment record:

Payment: PAY-1042
Order:   O-501
Amount:  ₹500
Status:  PROCESSING

PAY-1042 remains the same even if the first card fails, the customer switches payment methods, authentication is required, or the provider takes time to finish processing.

Its state can move through stages such as REQUIRES_ACTION, PROCESSING, FAILED, or SUCCEEDED without creating a new payment every time something changes.

This creates an important separation in the design: the idempotency key identifies a particular operation or retry, while the Payment record represents the longer-lived financial process those operations belong to.

And once the payment can live beyond the original request, the service also needs a way to learn about changes that happen later.

Some Payments Finish After the Request Is Over

Once a payment has its own lifecycle, the next question is how my service learns about changes that happen after the original request has already finished.

A payment might remain in PROCESSING, wait for bank confirmation, require authentication, or complete a few seconds later. Keeping the original HTTP request open until all of that finishes is not a reliable option, so the provider needs another way to notify my system.

That is where webhooks fit in.

What stood out to me was how familiar this problem looked. A webhook behaves a lot like an event consumed from Kafka: delivery can fail, the same event can arrive more than once, and events may not arrive in the exact order they were created.

So I would treat the webhook handler as another idempotent consumer. When the provider reports that PAY-1042 has succeeded, the service first verifies the event, checks whether that event has already been processed, and only then applies the corresponding payment-state transition.

If the same event arrives again, it should become a harmless duplicate rather than trigger the order flow twice.

This gives the redesigned service a reliable asynchronous path for normal payment updates. But webhooks still do not guarantee that both systems will always stay in sync.

If my database continues to say PROCESSING while the provider already considers the payment successful, I need a way to determine which state reflects what actually happened.

That is where reconciliation becomes necessary.

When Retrying Is No Longer the Right Answer

Idempotency, a persistent payment record, and webhooks reduce a lot of uncertainty, but they still cannot guarantee that my system and the provider will always agree.

Suppose PAY-1042 has been sitting in my database as PROCESSING. My service may have crashed after the provider completed the payment, or a webhook may have failed before the local update was committed.

Meanwhile, the provider already considers the payment successful:

My payment service
PAY-1042 = PROCESSING

Payment provider
PAY-1042 = SUCCEEDED

At this point, retrying the charge is the wrong recovery strategy. The customer may already have paid.

This is where reconciliation becomes part of the design.

Reconciliation compares independently recorded versions of the same financial transaction, detects when they disagree, and brings the inconsistent state back in line with what actually happened.

For PAY-1042, the reconciliation process can query the provider using the stored provider payment reference, compare that state with my local record, and repair the mismatch:

Before reconciliation

Local      → PROCESSING
Provider   → SUCCEEDED

After reconciliation

Local      → SUCCEEDED
Provider   → SUCCEEDED

The important distinction for me was this: retries deal with uncertain execution; reconciliation deals with uncertain truth.

If I am still trying to safely execute an operation, retry logic helps. If the operation may already have happened and two systems disagree about the result, I need to establish the actual payment state before doing anything else.

That is why the redesigned service needs to retain durable payment and provider references long after the original API request is over.

PaymentIntent, Authorization, and Charge Are Different Things

One distinction I would keep explicit in the redesigned model is the difference between the payment lifecycle and the actual movement of money.

A PaymentIntent represents the payment I am trying to complete for an order. It can survive failed attempts, authentication steps, payment-method changes, and different status transitions while still representing the same logical payment.

A Charge is closer to a concrete attempt to move money. It records details such as the amount, payment method, outcome, captured amount, and refunded amount.

This becomes important when a payment is not settled immediately.

For some flows, the first step is authorization. The provider confirms that the payment method can cover the amount and places a temporary hold on those funds. The merchant can then capture the authorized amount later.

A hotel is a simple example. It might authorize ₹10,000 at check-in to reserve the funds, then capture the final bill when the stay ends. If the authorization is never captured and eventually expires, the hold is released.

In the redesigned service, I would therefore keep these concepts separate:

Payment
PAY-1042
→ the complete payment lifecycle

Authorization
→ reserves funds for a possible later capture

Charge
→ a concrete payment attempt and its financial result

Capture
→ completes an authorized payment

This separation also makes later operations easier to reason about. If a captured payment has to be returned, a refund should be recorded as another financial operation with its own history. It is not equivalent to rolling back the original transaction and pretending it never happened.

With these boundaries clear, the individual pieces of the redesign are in place. The next step is to connect them into one V2 architecture and see how the entire payment flow behaves under failure.

V2: Designing Around the Failure Boundary

By this point, the redesign is no longer about adding one more safety check to the original payment service. The boundary itself has changed.

In the first version, I controlled the wallet, the ledger, and the database transaction that connected them. In V2, part of the payment can happen in a system I do not control, complete after my request has ended, or succeed while my own database still says PROCESSING.

I still would not solve that by turning the payment service into a collection of new microservices. The design can remain fairly small. The payment service owns the lifecycle, a durable database stores the financial state, a provider adapter handles external calls and retry policy, webhooks bring asynchronous updates back into the system, and reconciliation handles payments whose state remains uncertain.

The diagram intentionally has fewer boxes than the number of failure cases we have discussed. Inbox and Outbox are persistence patterns inside the payment boundary, not independent services. Exponential backoff and jitter belong inside the provider adapter. Authorization, capture, retries, refunds, and individual attempts are behaviours around the payment lifecycle rather than separate architectural components.

Running one payment through V2

Suppose an order for ₹500 reaches the payment service through Kafka.

The first job is to establish which logical payment this command belongs to. If Kafka delivers the same command twice, the second delivery should resolve to the same operation rather than create another payment.

The service can create a durable record such as PAY-1042, linked to the order, amount, payment method, current state, and eventually the provider's payment reference.

That record is important because it outlives the message that created it.

When the service is ready to contact the provider, the provider adapter sends the request using a stable idempotency key. If the call times out, the retry uses the same key. It does not create another operation simply because the response was unclear.

Retrying also needs a policy.

If the failure looks transient, such as a connection timeout or temporary provider unavailability, the next attempt should not happen immediately in a tight loop. I would use exponential backoff so the delay grows after repeated failures, then add jitter so many clients recovering from the same incident do not retry at exactly the same moment.

The more important decision, though, happens before the retry.

A timeout, an invalid request, and a declined card are three different failures. Retrying all of them would waste resources and, in some cases, make the behaviour worse. Transient infrastructure failures may deserve another attempt. A permanent validation error or card decline should instead move the payment into the appropriate business state.

This is also why an ambiguous 500 deserves care. If the provider already started executing the request, an internal error does not automatically prove that nothing happened. With providers that support result replay for idempotent operations, the safer recovery path is to retry the same operation identity and recover the previous result rather than create a new payment attempt.

The key only solves that short-term execution problem. It is not where I would keep the long-term identity of the payment. The durable Payment record remains responsible for that, including the provider payment ID once one exists.

The payment can keep moving after my request returns

Now suppose the provider accepts the payment but returns PROCESSING.

That does not mean the operation failed. It means the final outcome is not known yet.

The same PAY-1042 can continue through the lifecycle while the customer completes authentication, the provider waits for confirmation, or an authorized payment waits to be captured.

The useful property of this model is that the payment identity remains stable while the execution underneath it changes.

A failed card attempt does not necessarily end PAY-1042. Another payment method can be attached and tried. A payment that requires authentication can wait for the customer to complete it. An authorized payment can remain in REQUIRES_CAPTURE until the later capture operation finishes it.

Each external mutation can still have its own operation identity, but all of them belong to the same payment lifecycle.

I would keep refunds outside this state machine. Once a successful payment has already moved money and later needs to be reversed, the refund becomes another financial operation. The ledger should preserve both events instead of rewriting history as though the original payment never happened.

That gives the payment model a useful separation: the lifecycle tells me what is happening to the payment, while the ledger tells me what financial movements actually occurred.

Webhooks close the asynchronous path

Once the original request is over, the provider still needs a way to tell my system that something changed.

That is the job of the webhook path.

Suppose the provider later reports that PAY-1042 succeeded. Before applying that transition, the webhook handler verifies the event and records its event ID in the Inbox.

If the same provider event arrives again, the existing Inbox entry tells the service that the event has already been handled. The second delivery becomes a harmless duplicate instead of running the business transition twice.

This is effectively the external version of a problem I already had with Kafka. Both boundaries can deliver something more than once, so both consumers have to tolerate repetition.

When a valid new event changes PAY-1042 from PROCESSING to SUCCEEDED, I would commit that state change together with any required ledger update and an Outbox record such as PaymentSucceeded.

That outbox record protects another small but dangerous window.

Without it, this sequence is possible:

Payment DB commit succeeds
Payment becomes SUCCEEDED

service crashes

Kafka event never gets published

The payment is correct locally, but the rest of the system never learns about it.

By writing the outbox event in the same local transaction as the payment update, the event survives the crash. A publisher or CDC process can send it to Kafka later.

So the local transaction still matters in V2. Its job has simply become more precise: it cannot make the provider and my database one transaction, but it can keep my own payment state, ledger, inbox/outbox records, and local events consistent with each other.

When the normal path still leaves uncertainty

There is one remaining case that none of these mechanisms completely removes.

Imagine PAY-1042 has remained PROCESSING longer than expected. No useful webhook has repaired it, but the provider now shows the payment as SUCCEEDED.

Local payment:     PAY-1042 = PROCESSING
Provider payment:  SUCCEEDED

This is where the reconciliation job earns its place in the architecture.

It does not need to run on every payment request. I would keep it off the hot path and use it to inspect payments that have remained uncertain beyond an expected window.

For each one, it can query the provider using the stored provider payment reference and feed the result back through the same payment lifecycle.

If the provider confirms that the ₹500 payment already succeeded, V2 repairs PAY-1042 and continues the order flow from there. It does not create another charge simply because the local record was stale.

This also gives me a cleaner recovery hierarchy.

Normal retries handle transient request failures. Webhooks carry the expected asynchronous state changes. Reconciliation handles the smaller set of cases where the local system and provider still drift apart.

The important part is that all three paths eventually converge on the same durable Payment record rather than inventing separate versions of the transaction.

By the end of the redesign, the architecture is still relatively small. The additional reliability mostly comes from making the boundaries explicit: one identity for the payment lifecycle, stable identities for external operations, disciplined retries at the provider boundary, idempotent event consumption, reliable event publication, and a recovery path for state disagreement.

That is the main difference from my first design. V1 worked because the critical financial operation stayed inside a boundary I controlled. V2 is designed for the moment that assumption is no longer true.

Conclusion

When I first built this payment service, I was focused on the failures inside a system I controlled: duplicate execution, concurrent balance updates, transaction consistency, and reliable event flow. For that boundary, the design made sense.

What changed my thinking was moving the payment outside that boundary. Once an external provider is involved, the difficult part is no longer just preventing the same request from running twice. A payment can succeed while my service crashes, remain in progress after the original request ends, or leave my database and the provider holding different versions of what happened.

The V2 architecture grew from those failure cases rather than from adding infrastructure for its own sake. Each new piece exists because a specific uncertainty needed a recovery path.

The biggest lesson I took from this redesign is simple: reliable payment systems are ultimately about preserving the correct financial state, even when requests fail, services restart, events repeat, and the truth is temporarily split across multiple systems.

That is how I would approach the payment service if I were designing it again today.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️

After Two Months of Codex, Here’s What I Realized About AI Coding

NARESH — Tue, 08 Sep 2026 17:55:51 +0000

Two months ago, I thought I was spending my time figuring out how to use Codex better.

I was wrong.

I was actually figuring out what I should expect from an AI coding agent in the first place.

Over the last couple of months, I have spent a lot of time experimenting with Codex and other agentic coding tools. I tried different models, different harnesses, different workflows, and different ways of splitting work across multiple agents.

At the same time, I kept seeing people build setups that looked almost unlimited.

One person would have a $200 subscription here, another $200 subscription there, multiple coding agents running in parallel, several models from different providers, and a workflow that looked incredibly sophisticated.

My first reaction was simple: I want that too.

Then I remembered something important.

I have roughly $20 a month that I am comfortable spending on these tools.

That constraint completely changed how I looked at AI coding.

I couldn’t just pick the most expensive model, run five agents at the same time, and call it a workflow. I had to ask a much more practical question:

What is the cheapest setup that can reliably get my work done?

That question led me somewhere more interesting.

I started noticing that a simple CRUD task, a repetitive refactor, or writing unit tests doesn’t require the same level of reasoning as debugging an issue that crosses twenty microservices, Kubernetes, a database, and several layers of infrastructure.

I also realized that the model is only part of the equation. The harness, the context it receives, the tools it can use, the permissions it has, and the way the work is verified can all change the outcome.

Perhaps the biggest lesson was that I was spending too much time trying to copy workflows designed around someone else’s constraints.

Today, when I see a new model, a new coding agent, or an impressive multi-agent setup, I don’t immediately ask, “Is this better?”

I ask:

“Better for what?”

This article is about what I learned from two months of using Codex, experimenting with the surrounding AI coding ecosystem, and trying to make all of it work within real constraints.

Some lessons are about models and cost. Others are about harnesses, orchestration, permissions, and engineering judgment.

And some of them have very little to do with AI at all.

They are lessons I probably should have learned from software architecture years ago.

The question I was asking was wrong

For a while, whenever I tried a new coding agent, I evaluated it with one question:

Which one is the best?

That sounds reasonable until you start using these tools for very different kinds of engineering work.

A model that feels incredible while debugging a distributed system might be completely unnecessary for creating a few API endpoints. And a model that works perfectly well for repetitive implementation can become frustrating when the task requires understanding a large, unfamiliar codebase.

I started seeing this difference in my own work.

If I need to create a few CRUD endpoints, add some unit tests, rename a set of files, or make a straightforward refactor, I don’t need an agent to spend a huge amount of effort reasoning about the entire system. The task is relatively constrained. I already know what needs to happen.

Now take a different kind of problem.

Suppose a request is timing out somewhere in a system with twenty microservices. The request passes through an API gateway, several backend services, a message queue, a database, Kubernetes, and distributed tracing.

Now the agent has to investigate the system, follow the request across boundaries, inspect logs and configuration, form hypotheses, test them, and verify that the fix actually works.

That is a completely different problem.

The difference isn’t just the number of lines of code. It is the amount of uncertainty involved.

For the first task, I mostly need execution.

For the second, I need investigation, reasoning, context, tool usage, and verification.

That distinction changed the way I started looking at models.

Instead of asking, “Which model should I use?”, I started asking:

“What does this task actually require?”

If the answer is mostly repetitive execution, a cheaper model can often be enough.

If the task involves an unfamiliar repository, ambiguous requirements, multiple systems, or a difficult debugging problem, stronger reasoning becomes much more valuable.

This also changed how I think about the model comparisons we see online. A benchmark can tell you something useful about a model or an agent, but it cannot tell you which one is the best choice for every task you have.

The best model for writing a simple test is probably not the same model I’d want investigating why production requests are intermittently failing.

And once I started thinking this way, another problem became obvious:

The cheapest model isn’t necessarily the cheapest solution.

The cheapest model isn’t always the cheapest solution

Once I started thinking about models based on the task, I ran into another problem.

I initially thought choosing the cheaper model would automatically mean I was saving money.

That isn’t always true.

Suppose a cheaper model takes ten or fifteen turns to understand a problem, keeps making the wrong changes, and still needs me to step in and fix things. A stronger model might solve the same problem in two or three turns.

The price of the model is lower in the first case. The cost of getting the work done may not be.

That made me think less about cost per request and more about cost per successfully completed task.

That cost includes more than the model itself. It includes iterations, context consumed, the time I spend supervising the agent, and the cost of fixing things when it gets something wrong.

Artificial Analysis is useful here because it makes some of these trade-offs easier to see. Its coding-agent benchmarks compare performance, token usage, cost, and execution time, and it also lets you compare different agent harnesses. Its published cost metric is based on pay-per-token API pricing, so it is better viewed as a way to understand relative efficiency than as a direct comparison of consumer subscription prices.

And the interesting part is that there isn’t one tool that wins at everything.

One system can be cheaper, another can be faster, and another can perform better on a particular type of task. That’s why I don’t think a leaderboard should answer the question of what you should use.

Don’t look at a leaderboard and ask, “Which one should I buy?”

Look at it and ask, “Which of these characteristics actually matter for the work I do?”

If I mostly write straightforward backend code, I may care more about cost and speed.

If I regularly work on unfamiliar repositories or difficult debugging problems, I may be willing to spend more for stronger reasoning.

If I need an agent to work inside a particular environment, the harness and tool access may matter more than the model’s headline score.

This is also where having a smaller budget can teach you something valuable.

If I can afford only a $20 subscription today, I have to learn how to make good decisions within that constraint. I can use stronger models when the problem actually requires them and cheaper models when they don’t.

The goal isn’t to stay on the cheapest tools forever.

If one day I can comfortably afford a $200 plan, or a company gives me substantial credits, I want to already know how to use that extra capability well.

Having access to a more powerful model doesn’t automatically make me a better engineer.

Knowing when to use it, when not to use it, and how to structure the work around it is the skill that compounds.

If I learn that discipline with a $20 budget today, I can carry the same thinking into a $200 budget tomorrow.

The tools can get more powerful.

The decision-making should get better too.

The model is only part of the agent

The more I experimented with coding agents, the more I realized I was giving too much credit to the model.

I would try a new model, see how it performed, and immediately think, this model is better or this model is worse.

But the model doesn’t work in isolation.

A coding agent needs to understand the repository, read files, search for code, run commands, inspect errors, modify files, and verify whether its changes actually work. How those capabilities are exposed to the model can have a huge impact on the result.

That surrounding layer is what I think of as the harness.

Codex, OpenCode, Pi, Cursor, and other agentic coding tools can provide different harnesses around models. They can differ in how they manage context, expose tools, handle permissions, run commands, recover from failures, and verify changes.

So I stopped looking at an agent and asking only:

“Which model is underneath?”

I started asking:

“What can this agent actually do?”

Consider two agents using models with similar capabilities.

One can inspect the repository, search through the codebase, run tests, execute shell commands, inspect the output, and continue working based on what it discovers.

The other has limited context and fewer tools.

Even with comparable models, I would expect very different results.

For a small change, that difference might barely matter. The agent already has enough information to make the change.

For a complicated debugging task, the ability to investigate becomes much more important. The agent needs to form a hypothesis, gather evidence, change something, run the system again, and determine whether the evidence supports the hypothesis.

That is why I now think about an AI coding setup as a combination of:

model capability + harness + context + tools + permissions + verification

The model still matters a lot. But treating it as the entire system misses a large part of what determines whether an agent can actually get useful work done.

You can have a very capable model inside a poor workflow and get mediocre results.

You can also have a reasonably capable model inside a well-designed workflow and get surprisingly useful results.

Once I started seeing agents this way, another question naturally followed:

If the setup matters this much, how much of someone else’s AI workflow should I actually copy?

I stopped copying other people’s AI workflows

For almost three months, I have been exploring different ways people are using AI for software engineering.

I have watched videos, read blogs, followed experiments, and looked at workflows from engineers building with these tools across different parts of the world. Some of them are genuinely impressive. They have figured out ways to run agents in parallel, delegate tasks, review each other’s work, and keep large projects moving with very little manual intervention.

And almost every time I saw one of these setups, I had the same thought:

Why can’t I do this too?

If someone is running ten or fifteen agents at the same time, why shouldn’t I?

If someone has multiple subscriptions and access to several expensive models, why shouldn’t I build the same setup?

Then I realized I was looking at the workflow without looking at the problem that created it.

Someone might be running fifteen agents because they have a large workload, a tight deadline, and enough credits to make parallelism worthwhile. They may have reached that workflow because doing everything sequentially had become the bottleneck.

That doesn’t mean I have the same bottleneck.

If I have enough time but limited credits, running fifteen agents to build an application doesn’t necessarily make sense. I can use one agent, work through the tasks sequentially, and trade time for compute.

It might take longer.

That’s okay.

Time and compute are both resources. If you have plenty of compute but very little time, parallelism can make sense. If you have limited compute but enough time, sequential work can be the better trade-off.

AI coding tools make this easy to forget because they constantly encourage us to use more. More agents. More models. More context. More parallel tasks.

But the fact that you can run fifteen agents doesn’t mean you should.

What changed my thinking was asking a different question whenever I saw an impressive workflow:

What problem forced this person to build this workflow?

Maybe they introduced multiple agents because their tasks were highly parallelizable. Maybe they needed faster feedback. Maybe they were working across several repositories. Maybe their project was large enough that one agent could no longer handle everything efficiently.

Once you understand the problem, you can design your own solution.

You might end up with the same workflow.

Or you might discover that you need only one agent and a completely different process.

That is the part I think is easy to miss when learning from people who are already operating at a much larger scale. Their solution can be excellent and still be completely wrong for your situation.

So I stopped copying workflows.

I started studying the reasoning behind them.

And that has become one of the most useful habits I’ve developed while working with AI coding agents:

Don’t copy someone’s solution. Understand the problem that led them to that solution, then solve that problem in your own way.

Because when your constraints change, the right workflow should change with them.

Don’t build tomorrow’s architecture today

There is another lesson from software engineering that kept coming back to me while I was experimenting with AI coding agents.

I used to over-engineer things.

If I was building a new application, my first instinct was often to think about API gateways, multiple microservices, message queues, orchestration layers, and all the infrastructure I might need when the system became large.

The problem was that I was designing for a future that didn’t exist yet.

A project might have a few users and a handful of APIs, but I was already designing it as if millions of requests were coming tomorrow.

Over time, I started to appreciate a simpler approach:

Build the simplest architecture that solves the problem you actually have today. Add complexity when the requirements justify it.

For many applications, that might mean starting with a modular monolith and extracting services later when there is a real reason to do so.

The same thinking applies surprisingly well to AI coding workflows.

When I started exploring agentic coding, I made a similar mistake.

I saw people running multiple agents in parallel, with one agent planning, another implementing, another reviewing, and another validating. It looked powerful, so I started thinking that a serious AI coding workflow should probably look like that.

But again, I was solving a problem I didn’t have.

If I am building a small feature that one agent can complete reliably, adding an orchestrator and four specialized agents doesn’t make the system better. It adds coordination, context passing, failure modes, and more things I need to supervise.

The workflow architecture becomes a problem of its own.

Parallel agents become useful when there is actually parallel work to do.

If I have several independent tasks that can safely run at the same time, parallelism can reduce the total time. If different repositories need to be investigated independently, multiple agents may make sense. If a large project has enough independent workstreams, orchestration can become valuable.

But if the work is inherently sequential, adding more agents doesn’t magically make it parallel.

This is the same engineering instinct I now try to apply to both software architecture and AI workflows:

Don’t build tomorrow’s architecture today.

Start with what the problem requires.

Then add complexity when the problem earns it.

I no longer see multi-agent systems as the natural next step after using a single coding agent.

I see them as an architectural choice.

And architectural choices should come from requirements, not from how impressive the final setup looks.

That sounds like a small distinction.

For me, it changed the way I build with AI.

More capability requires more control

There is another part of AI coding that took me a little longer to appreciate.

The more capable an agent becomes, the more access you naturally want to give it.

At first, that might mean reading files and editing code. Then you give it permission to run tests, install dependencies, use the terminal, inspect logs, interact with a database, or access cloud infrastructure.

And once an agent can do all of that, it can investigate problems that would otherwise take a lot of manual work.

But there is a trade-off.

More capability also means a larger blast radius when something goes wrong.

Suppose an agent is debugging a database issue. If it can only read the schema and run safe queries, its ability to investigate is limited, but the damage it can cause is limited too.

Now give that same agent unrestricted database access.

It may be able to solve the problem much faster. But if it misunderstands the situation and decides that dropping and recreating a table is the easiest way to fix the schema, the mistake is no longer just a bad code change.

The same applies to cloud infrastructure.

An agent with access to Kubernetes, deployment configuration, cloud resources, and production logs can investigate problems that a locally restricted agent simply cannot see.

But giving an agent unrestricted control over all of those systems just because it makes debugging easier is a very different trade-off.

This changed how I think about permissions.

I used to think of permissions mainly as a way to unblock the agent.

Now I think of them as part of the architecture.

The question isn’t only:

“What does the agent need access to?”

It is also:

“What is the worst thing this agent could do with that access?”

That leads to a much more practical approach.

Give the agent enough access to do its job, but keep destructive operations behind appropriate boundaries. Separate read access from write access where possible. Use isolated environments when experimenting. Require human approval for actions that can cause irreversible changes.

And most importantly, verify what the agent actually did.

An agent can produce a correct-looking answer while making incorrect assumptions along the way. A successful command does not necessarily mean a successful fix.

For me, this is one of the biggest differences between experimenting with an AI coding agent and actually using one for serious engineering work.

When the agent only writes code, I can review the diff.

When the agent can also operate the environment, I need to think about the consequences of every permission I give it.

The goal isn’t to make agents powerless.

It is to give them capabilities proportional to the trust we place in them.

Once an AI agent moves from “write this code” to “go investigate and fix this system,” permissions, isolation, and verification become just as important as the model’s reasoning ability.

The more autonomous the workflow becomes, the more deliberately the surrounding harness needs to define its boundaries.

That is the lesson I took away:

The best AI workflow isn’t the one that gives the agent the most freedom. It’s the one that gives it enough freedom to be useful while keeping its actions within a boundary you can trust.

What my AI coding setup looks like today

After experimenting with all of this, my AI coding setup is much less complicated than I initially thought it needed to be.

I don’t have one model or coding agent that I expect to handle everything.

I choose based on the task.

For straightforward implementation work, I am comfortable using a cheaper model or a tool that gives me enough capability at a lower cost. CRUD APIs, unit tests, repetitive changes, and well-defined tasks usually don’t justify using the strongest model available.

When the problem becomes harder, I want stronger reasoning and a better environment around the agent. That might mean using Codex for deeper investigation, switching models when another one fits the problem better, or changing the harness because its tooling makes more sense for the task.

I have also become much more comfortable with sequential workflows.

I don’t need five agents running just because my tooling supports five agents. If one agent can finish the task reliably, that is often the better engineering decision.

When there is genuinely independent work, parallel agents become useful. At that point, the extra compute is buying something valuable: time.

So today, I don’t really have a single “AI coding stack.”

I have a set of tools I can reach for depending on the problem, the constraints, and what I am trying to optimize.

And I am comfortable changing that setup.

If a better option appears tomorrow, I am willing to switch. These tools are changing too quickly for me to build my workflow or identity around one particular product.

What matters more is understanding why I am using a tool, what capability I am paying for, and whether that capability is actually improving the outcome.

The tools will keep changing.

The models will keep changing.

My decision-making process should survive those changes.

The tools I use

I have tried enough tools at this point to realize that I don’t need all of them.

My choices are mostly driven by three things: capability, cost, and how well the tool fits the way I work.

I don’t see these tools as direct replacements for one another.

OpenCode is useful when I want flexibility around models and providers. Cursor is interesting for its integrated development experience. Other options can make sense when they are already available to me through an existing subscription or bundled offer.

Codex is the tool I currently come back to most for deeper engineering work. That is a personal choice based on how I work today, not a claim that it is the best tool for everyone.

And that distinction matters.

I am not trying to build a stack that looks impressive. I want a setup that gives me the capabilities I need without paying for capabilities I don’t use.

The exact tools will probably change.

The way I choose them shouldn’t.

What two months of Codex taught me about AI coding

I have been using AI for software development for around one and a half years, so these last two months weren’t about discovering AI coding for the first time.

They were about understanding how I want to use it.

After experimenting with Codex, Claude Code, different models, harnesses, and agentic workflows, I don’t think there is a single tool everyone should use.

But if you ask me a much more specific question, “If I have around $20 to spend every month, what would you personally choose?”

My answer is Codex.

That’s my personal preference based on how I work today.

I have used both Codex and Claude Code, and I genuinely like Claude Code. The interactive coding experience and surrounding capabilities make it a very good product.

But for my workflow, Codex currently feels more efficient.

That doesn’t mean I think Codex will always be the answer. These products are changing too quickly for me to make that claim.

It is also why I would be careful about locking yourself into yearly plans for AI coding tools. Models, pricing, usage limits, and new products can change quickly. I would rather keep the flexibility to switch when the trade-offs change.

But after everything I’ve experimented with, the lesson that matters most to me isn’t Codex versus Claude Code.

It is this:

Don’t optimize for their workflow. Optimize for your workflow.

You will see engineers running ten agents in parallel.

You will see people using several expensive models.

You will see elaborate orchestration systems and workflows that look incredibly sophisticated.

Learn from them.

Understand why they built that workflow.

Then ask yourself whether you have the same problem.

If you don’t, don’t copy the solution.

Build your own.

Maybe you need one agent. Maybe you need several. Maybe you need a powerful model for difficult debugging and a cheaper one for repetitive work. Maybe your constraints lead you to a completely different setup.

That’s where I think the real engineering skill is.

Not knowing which tool is currently winning.

Not having the most expensive subscription.

Not running the most agents.

Knowing how to look at a problem, understand your constraints, and design the workflow that makes sense for you.

The tools will change.

The models will change.

The prices will change.

Your ability to make good decisions should outlast all of them.

Don’t optimize for their workflow. Optimize for your workflow.

📖 Blog by Naresh B. A.
👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML
🌐 Portfolio: (Naresh B A)
📫 Let’s connect on (LinkedIn) | GitHub: (Naresh B A)

Thanks for reading. This is my personal engineering perspective, and I’d genuinely be interested in hearing where you agree or disagree.❤️

Why Instagram Reels Feels Different From YouTube Shorts: An Engineering Look at Recommendation Systems

NARESH — Sun, 30 Aug 2026 18:22:22 +0000

The observation that started this rabbit hole

Have you ever watched a few Instagram Reels about something completely random, then opened the app later and wondered, “How did my feed become this specific?”

That kept happening to me.

If I watched a few anime Reels, Instagram would quickly start surrounding me with related content. Not necessarily the exact same video. I might watch a clip of Goku turning Super Saiyan, then get another Dragon Ball scene, an edit from a different creator, a related fight, or something from the same anime universe.

YouTube Shorts felt different.

When I repeatedly watched Shorts from a particular creator, that creator seemed to keep returning. Sometimes I would even encounter very similar clips uploaded by different channels. It felt less like my feed was chasing the topic I cared about at that moment and more like YouTube was leaning on what it already knew I enjoyed.

At first, my conclusion was simple:

Maybe Instagram just has a better recommendation system.

But “better” is a dangerous word in engineering.

Both companies operate recommendation systems at enormous scale, backed by sophisticated retrieval, ranking, machine learning, and infrastructure. So I started looking through Meta’s engineering publications, Google research papers, and YouTube’s own documentation.

What I found was more interesting than one algorithm being better than another.

The two systems share surprisingly similar foundations. The difference seems to emerge from the signals they trust, how quickly those signals matter, what they optimize for, and several engineering decisions hidden behind the feed.

That is what this article is about.

Before comparing them, one architectural constraint matters

At the scale of Instagram and YouTube, recommendation starts with a fairly brutal engineering constraint.

There may be millions, or even billions, of pieces of content that could theoretically be shown to a user. Running the most powerful ranking model against every one of them whenever someone opens the app would be an extremely expensive way to build the system.

You could throw more compute at the problem, but the user is waiting for the next Reel or Short now. Recommendation systems live under tight latency budgets, so spending heavy neural network inference on obviously irrelevant content makes little sense.

This is why both systems are built as a funnel.

Start with multiple candidate sources

The system does not usually search for recommendations from one giant homogeneous pool.

Different retrieval sources may represent different kinds of opportunities:

content related to your recent interests
creators or channels you have interacted with
historically relevant topics
trending or fresh content
exploration candidates the system wants to test

Each source is trying to answer a broad question:

Is there any reasonable chance this user might care about this item?

At this point, recall matters more than perfect ordering. Throwing away a great candidate too early means no downstream model can recover it.

Retrieval: reduce the search space cheaply

Retrieval is the first major reduction.

Embedding-based systems are especially useful here. In a simplified Two-Tower design, one neural network produces a representation of the user from signals such as watch history, recent interactions, or context. Another produces representations for media items using information about the content and its behavior.

The resulting vectors make it possible to search for items that are mathematically close to the user’s current representation.

The important optimization is that the system does not deeply evaluate every video. It performs a much cheaper search over a representation space and reduces an enormous corpus to perhaps thousands of plausible candidates.

Think of retrieval as:

Find everything that might deserve a closer look.

Merge: turn several retrieval paths into one candidate set

Those candidates may have arrived from several sources.

Before ranking them seriously, the system needs to combine them.

This stage can remove duplicates, balance candidate sources, and stop one retrieval strategy from completely dominating the pool.

For example, if every retrieval source independently discovers the same highly popular video, carrying several copies of it further down the pipeline wastes compute.

By the end of this stage, the system has one cleaner candidate set ready for ranking.

First-stage ranking: be selective without being expensive

Now the system knows these candidates are at least plausible.

But there may still be thousands of them.

Running the largest ranking model on all of them is still expensive, so a lightweight ranker acts as another filter.

It uses relatively inexpensive features and fast scoring to remove weaker candidates.

This model does not need to make the final decision. Its responsibility is closer to:

Which of these candidates are worth spending serious computation on?

This can reduce thousands of candidates to hundreds.

Second-stage ranking: spend compute where it matters

Only now does it become economically reasonable to use a much heavier model.

Because the candidate set is small, the ranker can consider richer information about both the user and the content.

Instead of predicting a single notion of relevance, it may estimate several outcomes:

Will the user watch?
Will they finish?
Will they like or share it?
Will they skip quickly?
Will they find it satisfying?

These are multi-objective decisions.

A video that has a high chance of being clicked or watched is not necessarily the best recommendation if it also has a high probability of being skipped, disliked, or producing poor long-term satisfaction.

This is where ranking becomes much more than sorting videos by similarity.

Re-ranking: the highest score does not automatically win

Even after the heavy ranker produces its scores, the system may still modify the final ordering.

Suppose the top ten predictions all come from the same creator, contain nearly identical clips, or repeatedly cover the same topic.

A mathematically accurate ranking could still produce a terrible feed.

Re-ranking gives the product another layer of control over things such as diversity, freshness, safety, originality, policy, and exploration.

Only after this stage do we get the final personalized feed.

Then the loop begins again.

You watch, skip, like, share, search, or leave. Those actions become new evidence that can influence future requests.

So the architecture is really a sequence of increasingly expensive questions:

Retrieval
Could this be relevant?

First-stage ranking
Is this candidate worth deeper evaluation?

Second-stage ranking
How valuable is this recommendation likely to be?

Re-ranking
Should it actually appear in the final feed?

This funnel is the common foundation underneath both Instagram Reels and YouTube Shorts.

The interesting difference starts after that foundation.

Both companies can build sophisticated retrieval and ranking models. What changes the experience is what signals they feed into those models, how quickly those signals change, and what the system ultimately considers a successful recommendation.

The first surprise: the architectures are more similar than they look

Before looking at the research, I expected Instagram and YouTube to have noticeably different recommendation architectures.

That was not really the case.

Meta’s published recommendation systems and Google’s YouTube research both describe variations of the same fundamental pattern we just saw: retrieve a broad set of plausible content, progressively rank it with more expensive models, then apply additional logic before deciding what reaches the user.

The models have evolved significantly over the years, and the exact production systems behind Reels and Shorts are not publicly exposed. Still, at the system-design level, both companies are solving the same constraints:

Huge content corpus
        ↓
Find plausible candidates quickly
        ↓
Spend more compute on fewer candidates
        ↓
Predict several possible user reactions
        ↓
Construct the final feed

That changed the question for me.

If both platforms have access to sophisticated retrieval, embeddings, neural ranking, massive infrastructure, and years of interaction data, then Instagram feeling different cannot be explained by saying its model is simply more advanced.

The difference has to emerge somewhere else.

Three areas started to matter much more:

Signals: What does each platform know about the user?

Freshness: How quickly can a new interaction influence the next recommendation?

Objective: What behavior is the system ultimately trying to optimize?

Those choices can make two systems built on very similar foundations produce noticeably different feeds.

And this is where Instagram starts to become particularly interesting.

Instagram’s interesting advantage is not simply better AI

The first useful clue came from how Meta handles candidate retrieval.

In its published architecture for Instagram Explore, Meta describes using multiple retrieval sources at the same time. Some are pre-generated and capture longer-term interests. Others operate in real time and are designed to capture recent interactions.

That distinction matters.

Suppose most of my Instagram history says I care about software engineering, football, and technology. Then, during one evening, I watch several Dragon Ball clips almost completely, like one of them, and save another.

Instagram does not need to retrain a neural network specifically for me.

Fresh interaction signals can change the information used during the next recommendation request. Meta also describes retrieving content directly from a user’s interaction history by finding items similar to things they recently liked, saved, or shared.

Conceptually, the system can do something like:

Recently watched Dragon Ball Reel
             ↓
     Content representation
             ↓
 Find related candidate content
             ↓
 Add those candidates to the pool
             ↓
           Ranking

This mechanism is publicly documented for Explore, so I would not claim that Instagram Reels uses the exact same implementation. But it shows that Meta’s recommendation infrastructure is explicitly designed to combine persistent interests with much fresher behavioral signals.

There is another detail that may explain something I had noticed for a long time.

When I watched anime content on Instagram, I often received related clips from different creators, but rarely the exact same clip reposted again and again.

That is not entirely accidental.

Instagram says that when it detects multiple identical pieces of recommended content, using audio and visual matching, it tries to recommend the original instead of the repost. This policy applies to Reels, Explore, and recommended Feed content. Materially changed versions, such as reactions, remixes, memes, or new voiceovers, can still remain eligible.

That creates an interesting product effect.

If I watch Goku turning Super Saiyan, the system does not necessarily have to keep showing five copies of that transformation. It can move around the same interest space:

Goku transforms
      ↓
Goku vs Frieza
      ↓
another Dragon Ball edit
      ↓
related anime content

From the user’s perspective, this can feel like the system understood the topic rather than merely memorizing the video.

That distinction is important.

Perceived recommendation quality does not come only from having a more powerful ranking model. Candidate freshness, duplicate handling, source diversity, and how broadly the system explores an interest can change the experience before the final ranking model even gets a chance to make its decision.

YouTube has a different kind of memory

YouTube becomes interesting for almost the opposite reason.

Its recommendation system is not limited to what you just watched in Shorts. YouTube publicly documents that recommendations can use watch history, search history, channels you enjoy, subscriptions, likes, dislikes, and satisfaction signals. It also learns across formats such as Shorts, long-form video, live streams, and posts.

That creates a much deeper historical profile.

If I watch several Shorts from one creator, YouTube has a strong reason to keep that creator in the candidate pool. The system is not only learning that I liked a topic. It is also learning that I seem to trust or enjoy a particular channel.

Conceptually:

Short from Creator A
      ↓
High watch / engagement
      ↓
Creator affinity increases
      ↓
More content from Creator A
becomes competitive in retrieval and ranking

This matches something I noticed personally. Once I start watching Shorts from a specific channel, that channel tends to keep resurfacing much more consistently.

But YouTube has another advantage that Instagram does not have in quite the same form.

A user might watch a 20-second Short about transformers, search for attention mechanisms, spend 40 minutes on a technical lecture, subscribe to an ML channel, and later watch another related Short.

Those interactions happen at very different depths, but they all contribute evidence about the same person.

A long-form video can reveal something a short interaction cannot: sustained intent.

Watching three 15-second clips about a topic might mean curiosity. Spending an hour on the same subject is a much stronger signal.

YouTube also explicitly talks about optimizing for long-term viewer satisfaction, not just immediate watch time. Signals such as survey responses, dislikes, and “Not interested” feedback help distinguish content that merely captures attention from content users actually value.

This makes YouTube’s recommender feel less like a system that only asks:

What should I show you next?

and more like one that also has to consider:

What kind of content relationship am I building with you over time?

That does not mean YouTube is less capable of reacting quickly. It means it has a much larger body of historical evidence competing with the latest few interactions.

And that leads to the question that changed this entire comparison.

The question I originally asked was slightly wrong

At the beginning, I was asking a simple question:

Why does Instagram’s recommendation system feel better than YouTube Shorts?

After looking at the architecture, that question started to feel incomplete.

What I was really noticing was not necessarily better recommendation quality. I was noticing how visibly the feed reacted to my current interest.

Those are different things.

A recommender can optimize for several goals at once:

immediate relevance
long-term satisfaction
diversity
creator discovery
freshness
retention
recommendation stability

A system that reacts aggressively to the last five things I watched may feel incredibly intelligent during that session.

It can also be wrong.

If I suddenly watch several Formula 1 clips tonight, should the platform immediately rebuild my feed around F1? Or should it treat that behavior as temporary curiosity while preserving the interests it has learned over months?

There is no universally correct answer.

That is a product decision expressed through ranking objectives, feature freshness, candidate sources, and weighting between recent and historical behavior.

So the more useful question became:

Why does Instagram make changes in my immediate interests feel more visible, while YouTube often feels more anchored to my broader viewing history?

That question gives us something much more interesting to investigate than simply declaring one algorithm better than the other.

The infrastructure detail I did not expect: move expensive work away from peak traffic

One line in Meta’s Explore engineering write-up caught my attention:

Applying a much heavier MTML model during peak hours could be tricky.

Their solution was surprisingly practical. For some users, Meta precomputes recommendations during off-peak hours so that expensive inference does not always have to happen while the user is waiting.

This is a classic production-engineering trade-off.

The most accurate model may also be the most computationally expensive one. If millions of users arrive at roughly the same time, running that model synchronously for every request can create unnecessary latency and infrastructure cost.

Instead of asking:

How do we make the heavy model infinitely faster?

the architecture can ask:

Which parts of this computation can safely happen before the request arrives?

Conceptually, the design looks like this:

Meta publicly confirms the precomputation strategy, but does not expose enough detail for us to claim exactly whether every cached object is a candidate set, ranking score, final list, or some combination of them.

That uncertainty does not reduce the engineering lesson.

This pattern appears everywhere in large systems. Search engines build indexes before queries arrive. Databases use materialized views. Recommendation systems precompute embeddings and candidate information. Applications warm caches before predictable traffic spikes.

The principle is the same:

Do not spend request-time compute on work that does not require request-time information.

Of course, recommendation introduces another problem: freshness.

A result calculated earlier may not know that the user suddenly spent the last ten minutes watching Formula 1.

So precomputation cannot blindly replace real-time personalization. The system has to decide which information is stable enough to prepare ahead of time and which signals must remain fresh.

That tension between accuracy, freshness, cost, and latency is one of the most interesting system-design decisions hiding behind what feels like an instant swipe.

The creepy recommendation problem: did Instagram hear me?

Almost everyone has had some version of this experience.

You talk to a friend about a show, a product, or a random topic, then a few hours later something closely related appears in your feed. The immediate reaction is obvious: the app must have heard me.

For ordinary personal chats on WhatsApp, that explanation does not fit how the system is publicly described. Message content is end-to-end encrypted, so Meta does not have a normal path where it reads a private conversation, extracts a keyword, and feeds that directly into Instagram ranking.

The more interesting explanation is that recommendation systems often do not need that information.

If you and your friend are talking about One Piece, several other signals may already point in the same direction. The topic may be trending, you may have interacted with anime before, your friend may be engaging with related content inside Instagram, or more creators may suddenly be publishing similar clips.

Instagram also has social context that YouTube does not have in quite the same form. Activity from people connected to you can create legitimate recommendation paths without requiring access to a private conversation.

So what feels like the app listening may actually be several weak signals lining up at the same time.

And from an engineering perspective, that is probably the more interesting explanation: a recommender can feel disturbingly specific without ever needing the signal you assumed it was using.

What this comparison teaches us about designing a recommender

Looking at Instagram and YouTube changed the way I think about recommendation systems.

The model matters, but several decisions around the model may matter even more.

Start with the objective, not the architecture

Before choosing embeddings, ANN indexes, or ranking models, decide what a successful recommendation actually means.

Is the goal:

immediate engagement?
completion rate?
long-term satisfaction?
discovery?
creator diversity?
retention?

Two systems can use almost identical architectures and still behave very differently because they optimize different outcomes.

A technically impressive ranker trained against the wrong objective will still produce the wrong product.

Model intent at more than one timescale

A user is not one static vector.

There is a difference between:

What I usually like
What I have cared about recently
What I want right now

If the system ignores recent behavior, recommendations become stale.

If it reacts too aggressively, watching five Formula 1 clips could convince the system that Formula 1 has suddenly become your entire personality.

Good personalization needs some balance between persistent preference and temporary intent.

Candidate quality can matter as much as ranking quality

It is tempting to focus all the attention on the final neural ranker.

But if retrieval sends poor candidates downstream, the ranker cannot magically recover content that was never retrieved.

The Instagram duplicate-content example makes this especially clear.

Removing repeated copies, mixing candidate sources, preserving freshness, and exploring nearby interests can change the perceived quality of the feed before expensive ranking even begins.

Spend compute progressively

The recommendation funnel is also a useful general system-design pattern.

Use cheap operations while the search space is large. Spend expensive computation only after the candidate set becomes small enough to justify it.

And whenever possible, move stable work outside the request path.

That might mean precomputed embeddings, cached candidate information, or other offline computation that reduces what has to happen while the user is waiting.

Fast adaptation is a trade-off, not automatically a feature

This may be the most important lesson.

A system that changes immediately can feel remarkably intelligent.

It can also overfit to temporary behavior.

A system that changes slowly may feel less responsive, but it may preserve a better understanding of long-term interests.

So when building a recommender, the question is not just:

How quickly can the model learn what the user is doing?

It is also:

How quickly should the product allow that behavior to change what the user sees?

That decision sits somewhere between machine learning, system design, and product design.

And it may matter more than choosing the newest recommendation model.

So, was my original observation correct?

Partly.

Instagram really did feel more responsive to whatever I cared about in the moment, while YouTube Shorts often felt more anchored to creators, viewing history, and longer-term interests.

What changed after the research was my explanation for it.

I started with the assumption that Instagram simply had the better recommendation system. The evidence does not support that conclusion.

Both platforms use sophisticated multi-stage recommenders. The difference appears to come from how they combine recent behavior, historical interests, content relationships, and the objectives their ranking systems optimize.

Some of my original assumptions held up:

But one important claim remains unproven: there is no public benchmark showing that Instagram Reels adapts faster than YouTube Shorts.

What we can say is more useful.

Instagram has documented mechanisms that can make changing interests highly visible, including fresh candidate sources, interest-based discovery, and duplicate handling. YouTube brings a broader history into the decision, including channels, search behavior, long-form consumption, and satisfaction signals.

So the difference I felt was probably not:

Instagram understands me better.

A better interpretation is:

The two systems appear to make different trade-offs between what matters to me right now and what has mattered to me over time.

That is a much more interesting engineering difference than declaring one algorithm the winner.

Blog by Naresh B. A.

Backend & AI Systems Engineer | Distributed Systems · Production ML

Portfolio: Naresh B A

Let’s connect on LinkedIn | GitHub: Naresh B A

Thanks for reading. This is my personal engineering perspective, and I’d genuinely be interested in hearing where you agree or disagree. ❤️

Inside a Production LLM Memory System

NARESH — Mon, 17 Aug 2026 16:48:51 +0000

From observations to durable knowledge, a practical architecture for making AI systems remember what matters.

You can explain what an AI application should remember.

Building a system that actually remembers it reliably is a very different problem.

In the previous article, LLM Memory Is Not Chat History: How to Design Memory That Improves Future Decisions, I focused on the conceptual side of memory: what deserves to become memory, how memories evolve, why retrieval is a decision rather than simple similarity search, and why forgetting is part of the lifecycle.

This article moves one layer deeper.

Suppose you're building an AI coding assistant. During a pull request review, it discovers that a pagination bug was caused by non-deterministic ordering. That observation may eventually become useful knowledge for future reviews.

But now the engineering questions begin.

Where should that observation be stored?

Should the application wait for an LLM to interpret it?

What happens if the same lesson already exists?

How do we decide whether it is a duplicate, a refinement, or a new revision?

How do we retrieve it later without returning an outdated version?

And how do we do all of this without turning a relatively simple AI application into a distributed system?

For the architecture we're building here, you don't need Kafka, a fleet of microservices, or a dedicated memory platform. A well-designed modular monolith can take you surprisingly far.

That's the architecture we'll build in this article.

We'll design a practical memory module with a canonical store, semantic retrieval, caching, asynchronous memory formation, identity resolution, and lifecycle management. The goal isn't to design the largest possible system. It's to design the smallest architecture that can behave like a serious production memory system and still remain simple enough to build, operate, and evolve.

If the previous article was about what memory means, this one is about what memory looks like when you actually build it.

How the Pieces Fit Together

The architecture is split into three complementary paths: write, read, and lifecycle.

The write path starts when the application produces an observation that may be useful later. The memory module persists it and creates a durable memory formation job, allowing the user-facing request to continue without waiting for memory processing. A background worker then extracts the relevant information, optionally uses an LLM for unstructured content, and produces a candidate memory. Identity resolution and admission determine whether that candidate should create, reinforce, revise, or supersede existing knowledge.

The read path works in the opposite direction. When a future request needs memory, the system first uses known structural signals to narrow the search, then combines semantic retrieval with revision and lifecycle filtering before ranking the candidates. Redis accelerates frequently accessed memories, while PostgreSQL remains the canonical source of truth.

The lifecycle path runs independently in the background, maintaining memories through reinforcement, consolidation, expiration, supersession, and deletion.

This separation keeps the user-facing path simple while allowing memory to form, evolve, and be retrieved independently.

That last sentence is important because it captures the architecture without repeating the detailed sections that follow.

Start With One Source of Memory Truth

The first version of a memory system should be deliberately boring.

Not because memory is simple, but because its semantics are already complicated. Adding infrastructure complexity before the workload requires it only makes the system harder to reason about.

For a modular monolith, I would start with one canonical datastore:

PostgreSQL with pgvector for semantic retrieval.

The important part isn't the database itself. It's having one place that answers:

What memories exist, which version is current, what evidence supports them, and whether they are still valid?

Redis can accelerate reads, and pgvector can help find semantically related memories, but neither should become the authority on what the application actually believes.

A useful mental model is:

An observation is something the application encountered.

A candidate is information that appears worth preserving but hasn't yet been admitted as durable memory.

A memory family represents one evolving piece of knowledge.

A memory revision represents a specific version of that knowledge.

Consider pagination:

Revision 1

Use offset pagination.

Revision 2

Use cursor pagination for large datasets.

Revision 3

Cursor pagination must use deterministic ordering with a stable secondary key.

These aren't three unrelated memories. They are revisions of the same evolving idea.

That makes immutable revisions useful. Instead of overwriting the previous value, each meaningful change creates a new revision while the memory family points to the current one.

PostgreSQL is a good starting point because it can keep the structured parts of this model together:

Memory families and revisions
Evidence and provenance
Lifecycle state
Temporal validity
Background memory jobs
Embeddings

With pgvector, semantic retrieval can live alongside that canonical metadata without introducing another datastore immediately.

This isn't a claim that PostgreSQL is the best database for every memory system. It's a deliberate starting trade-off.

A dedicated vector database may become worthwhile when corpus size, query volume, latency, or indexing requirements justify the additional operational complexity. Until then, keeping the system within one storage boundary makes it easier to reason about and operate.

The principle is simple:

Start with the smallest storage architecture that can preserve the semantics your memory system requires.

Let the workload earn the complexity.

And once we have a clear source of truth, the next question becomes more interesting:

How does information actually get into this system without slowing down the application that produced it?

That is where the write path begins.

The Write Path: Learning Without Blocking the Application

Once we have a canonical memory store, the next question is: how does information actually become memory?

The first design I'd avoid is making memory formation part of the user-facing request:

Interaction → LLM extraction → embedding → storage → response

That adds latency and makes the main application depend on an expensive, failure-prone background concern.

Instead, separate observing something from deciding what it means.

The important decision is to persist the observation first.

In a modular monolith, a durable memory_jobs table is enough to create this separation:

BEGIN
    INSERT observation
    INSERT memory_job
COMMIT

The request can continue while a background worker processes the memory job.

Because the formation job is durable, failed processing can be retried without losing the original observation or blocking the user-facing request. If an LLM provider is temporarily unavailable, memory formation can simply wait until the job can be processed again.

Deterministic Extraction First

The worker should not call an LLM simply because one is available.

If structured events already contain the information we need, extract it directly.

For example:

repository = payments-api
component = pagination
issue = unstable-ordering

There is no reason to spend an LLM call rediscovering those fields.

LLM-assisted extraction becomes useful when valuable information is buried inside unstructured conversations, reviews, or tool output.

Even then, the model produces a candidate, not canonical truth.

Identity Before Insertion

Once we have a candidate, we need to determine how it relates to existing knowledge.

It might be:

- NEW
- DUPLICATE
- REINFORCEMENT
- REFINEMENT
- REVISION
- SUPERSESSION
- CONTRADICTION

For example, if the existing memory says:

Use cursor pagination for large datasets.

and a new review discovers:

Cursor pagination requires deterministic ordering with a stable secondary key.

we shouldn't blindly create another unrelated memory. The new information may represent a refinement or revision of the existing memory family.

This is why the write path is more than:

message → embedding → vector database

The embedding helps us find related knowledge.

It does not decide what the application should remember.

The overall principle is simple:

Persist experience quickly. Interpret it asynchronously. Admit knowledge deliberately.

Once memory can be formed without blocking the application, we need the opposite path: how do we retrieve the right memory when a future request actually needs it?

The Read Path: Finding the Right Memory

Writing memory is only half the problem.

When a future request arrives, the system has to answer a harder question:

Which memories should influence this decision?

The first instinct is often to send the query directly to a vector index and take the nearest results.

That works as a candidate-generation mechanism.

It shouldn't be the final retrieval strategy.

A better read path starts with what the application already knows.

If the request is about a particular repository, component, environment, or entity, those signals should narrow the search before semantic retrieval begins.

For example:

repository = payments-api
component = pagination
environment = production
memory_type = engineering_rule

Semantic retrieval can then expand recall within that relevant space.

This matters because two memories can be semantically similar while only one actually belongs to the current problem.

Redis Is an Accelerator, Not the Truth

Redis can sit in front of the canonical store for frequently accessed memories.

Conceptually:

Request-local cache
        ↓
Redis
        ↓
PostgreSQL + pgvector

But the cache should never become the authority on memory correctness.

Revision-oriented keys make this easier:

memory:{family_id}:revision:{revision}
memory:{family_id}:current

When a new revision becomes current, the pointer changes. Older immutable revisions don't need to be overwritten.

This also gives the application a way to detect stale cached data rather than assuming the cache is always current.

And there is an important distinction from the previous article:

Cache freshness and memory freshness are different problems.

Redis can contain the latest revision while that knowledge is already temporally obsolete.

Retrieval Is a Ranking Decision

After candidate memories are found, the system still needs to decide which ones deserve influence.

Useful signals can include:

Semantic relevance
Exact scope match
Temporal applicability
Evidence quality
Importance
Reinforcement
Redundancy

There is no universal weighting formula that works for every application.

The right ranking strategy depends on what the memory system is trying to optimize.

The important architectural boundary is that semantic similarity discovers candidates; the retrieval layer decides which candidates are actually useful.

That distinction keeps the memory module focused on returning relevant knowledge.

It also keeps context assembly outside this module. The memory system returns memory records. The application decides how those records should be used in the final model interaction.

Retrieval isn't a search problem. It's a decision problem.

And even a well-designed retrieval path isn't enough by itself. Memories continue to change after they're created, which means the system also needs a way to maintain them over time.

Memory Has to Maintain Itself

A memory system shouldn't stop working after a memory is created.

As new observations arrive, existing knowledge may become stronger, more precise, outdated, or completely wrong. That means memory needs a maintenance path just as much as it needs a write path and a read path.

The same background worker can handle these lifecycle operations:

The important part is that these operations should not require the application request to coordinate them synchronously.

A new incident might reinforce an existing memory today. A month later, several incidents might reveal a broader pattern worth consolidating. Eventually, an older memory may become irrelevant or be replaced by a newer revision.

These are background maintenance activities.

Consolidation Is Where Experience Becomes Knowledge

Suppose the application encounters several pagination failures:

Incident 1 → unstable ordering
Incident 2 → NULL ordering inconsistency
Incident 3 → incomplete cursor state

Individually, these are observations.

Together, they may support a more reusable lesson:

A cursor should encode the complete ordering state used by the database.

That derived memory is more useful than simply storing three increasingly similar incident summaries.

But consolidation should remain traceable. The derived memory should retain references to the experiences that support it. If later evidence contradicts the conclusion, the system should be able to understand where that knowledge came from.

This is also why recursively summarizing existing summaries is risky. Over time, information can drift further away from the original evidence.

Forgetting and Deletion Are Different

Lifecycle maintenance also needs a clear boundary between forgetting and deletion.

Forgetting means the memory should no longer influence normal retrieval.

Deletion means the underlying content should actually be removed or anonymized according to the application's retention policy.

For this architecture, deletion can remain deliberately simple:

Mark unavailable
      ↓
Invalidate Redis
      ↓
Remove vector representation
      ↓
Delete / anonymize canonical content

There is no need for a distributed deletion coordinator at this stage.

The broader principle is the same one we've followed throughout the architecture:

Keep expensive lifecycle work asynchronous, keep canonical state authoritative, and make every transition explicit.

At this point, the architecture is complete enough to build. But that raises an equally important question: when does this simple architecture stop being enough?

Where This Architecture Stops Being Enough

The architecture we've built is intentionally simple.

A modular monolith, PostgreSQL + pgvector, Redis, and a background worker can take a memory system surprisingly far. But it won't be the right architecture forever.

The important question isn't:

"When should I use microservices?"

It's:

"What constraint is the current architecture no longer handling well?"

For example, PostgreSQL may become the bottleneck if the memory corpus and retrieval workload grow beyond what the current setup can comfortably handle.

Semantic search may eventually require a specialized vector system because latency, throughput, or indexing requirements have changed.

Background memory jobs may grow large enough to compete with the application's primary workload, making independent worker scaling necessary.

Tenant isolation may become more demanding, or different teams may need independent ownership and deployment boundaries.

Multi-region requirements can introduce another class of constraints around data placement, availability, and consistency.

These are legitimate reasons to evolve the architecture.

But notice what they have in common.

None of them says:

"The system is production, therefore it needs microservices."

They are measurable constraints.

A natural evolution might eventually look like:

Modular Monolith
      ↓
Separate Worker
      ↓
Specialized Retrieval
      ↓
Independent Memory Service
      ↓
Distributed Architecture

The exact path will depend on the workload.

Some applications may never need to move beyond the first stage. Others may gradually extract individual components as scaling or ownership boundaries emerge.

That's why I would avoid designing the distributed version upfront.

Every new service introduces another network boundary, deployment surface, failure mode, and operational dependency. If the current architecture can satisfy the requirements with simpler infrastructure, that simplicity is an advantage.

The goal isn't to build the most distributed memory system possible.

It's to build the simplest system that satisfies the requirements, and let evidence justify the next layer of complexity.

Architecture should become more distributed because measurements and ownership boundaries demand it, not because "production-grade" means drawing more boxes.

That principle is useful far beyond memory systems. Good architecture isn't about predicting every future problem.

It's about leaving enough structure that the next problem can be solved without rebuilding everything.

Conclusion

Building memory for an AI application doesn't require starting with a distributed platform.

A modular monolith with a clear memory boundary, PostgreSQL as the source of truth, pgvector for semantic retrieval, Redis for acceleration, and a durable background worker can provide a surprisingly strong foundation.

The important part isn't the number of components.

It's the decisions behind them.

Observations should be persisted before they're interpreted. Expensive memory formation should happen asynchronously. Candidates should be resolved against existing knowledge before becoming durable memories. Retrieval should combine structure with semantics instead of trusting similarity alone. And memory should continue evolving after it is created.

Part 1 asked a conceptual question:

What should an AI application remember?

This article answered the architectural one:

How can we build a system that remembers it reliably?

The answer doesn't begin with a vector database.

It begins with a clear memory model, a canonical source of truth, and enough structure to let knowledge evolve without turning every application into a distributed system.

And when the workload eventually demands more, the architecture can evolve with it.

Start simple. Make the semantics strong. Let the workload earn the complexity.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️

LLM Memory Is Not Chat History: How to Design Memory That Improves Future Decisions

NARESH — Tue, 04 Aug 2026 18:04:35 +0000

Most production AI systems don't fail because the model forgets.

They fail because the application remembers the wrong things.

Suppose you're building an AI coding assistant. Last week, it reviewed a pull request and discovered that a pagination bug was caused by non-deterministic ordering. Today, another pull request modifies the same pagination logic.

Should the assistant remember that previous incident?

At first glance, the answer seems obvious.

"Of course. Just store the conversation."

But that raises a much more interesting question.

Should the application remember the entire discussion, the final conclusion, the evidence that led to it, or only the lesson that remained true after the bug was fixed?

Those are completely different kinds of information, and treating them as the same thing is where many AI applications begin to break down.

This is one of the biggest misconceptions in discussions around LLM memory. Memory is often treated as nothing more than conversation history stored in a database or a vector index. If a previous message looks similar to the current request, retrieve it and append it to the prompt.

That approach works well for prototypes.

It rarely works for production systems.

Real applications accumulate experience over time. Some information becomes more trustworthy as new evidence confirms it. Some knowledge becomes obsolete. Some observations are corrected. Others are replaced entirely. And much of what an application observes should never influence future decisions in the first place.

In other words, storing information is easy.

Deciding what deserves to be remembered is the hard part.

The moment you start thinking about memory this way, you realize it is no longer a storage problem.

It's a knowledge management problem.

A production memory system has to answer questions like:

Is this an observation or an established fact?
Does this strengthen an existing memory or replace it?
Is it still valid?
Should it influence the current decision?

Notice that none of these questions are about embeddings, vector databases, or context windows.

They're about deciding how past experience should shape future decisions.

That's the mental model we'll build throughout this article.

Rather than focusing on where memories are stored, we'll explore what deserves to become memory, how knowledge evolves over time, and what it takes to build a production memory system that improves future decisions.

Because a production memory system is not defined by where information is stored.

It is defined by how experience is transformed, evolved, retrieved, and eventually forgotten.

The Mistake: Treating Chat History as Memory

When people first add memory to an LLM application, the implementation often looks surprisingly simple.

Store the conversation. When a new request arrives, retrieve the previous messages and append them to the prompt.

For simple chatbots, that approach is perfectly reasonable. If a user asks, "What did I ask five minutes ago?", the conversation itself is the answer.

The problem begins when an application is expected to learn from experience instead of merely recalling it.

Let's return to our AI coding assistant.

During a pull request review, it identifies that a pagination bug was caused by non-deterministic ordering. The discussion spans dozens of messages as the developer investigates different hypotheses, tests multiple fixes, and eventually reaches the correct conclusion.

A week later, another pull request modifies the same pagination logic.

Does the application need the entire conversation again?

Probably not.

Most of those messages captured the investigation, not the lesson. They included incorrect assumptions, abandoned ideas, and intermediate reasoning that no longer adds value.

What actually matters is the knowledge that survived the investigation.

For example:

Cursor pagination must use deterministic ordering, including a secondary key when the primary sort field is not unique.

That single piece of knowledge is far more valuable than replaying every message that led to it.

This is the fundamental distinction many memory implementations miss.

Chat history records what happened. Memory preserves what should influence future decisions.

Those goals overlap, but they are not the same.

Once you recognize that difference, another realization follows naturally. A production memory system shouldn't ask, "Which previous messages look similar to this request?"

Instead, it should ask, "What knowledge from past experience is still relevant and trustworthy enough to influence this decision?"

That shift in thinking changes how you design the entire system. Instead of accumulating conversations indefinitely, you begin treating memory as curated knowledge that evolves over time.

The rest of this article explores what that evolution looks like and why it matters when building reliable AI applications.

Memory, Context, and Execution State

One reason LLM memory feels confusing is that three different concepts are often treated as if they mean the same thing: memory, context, and execution state.

They all carry information across requests in some form, but they solve completely different problems.

Memory is information preserved because it may improve future decisions.

An AI coding assistant might remember that a particular repository always uses cursor pagination, or that a team prefers concise pull request comments. These are pieces of knowledge that remain useful beyond a single interaction.

Context is the information assembled for the model's current decision.

It may include the user's latest request, relevant documentation, tool outputs, system instructions, and a small number of retrieved memories. Once the request is complete, that context disappears. A new request will likely receive a different context, even if it uses the same underlying memory.

Execution state tracks the progress of an ongoing task.

Suppose an agent is reviewing a pull request and pauses while waiting for a security scan to finish. It needs to remember which files were already analyzed, which tool was called, and where execution should resume. That information keeps the workflow running correctly, but it isn't knowledge that should influence future pull request reviews.

A useful way to think about these concepts is to ask a different question for each one:

Memory answers, "What have we learned?"

Context answers, "What does the model need right now?"

Execution state answers, "Where are we in this workflow?"

The distinction matters because they have different lifecycles.

A workflow's execution state disappears when the task finishes. Context is rebuilt every time the model is invoked. Memory, however, can survive for weeks, months, or even years, continuously evolving as the application gains more experience.

This also reveals an important relationship between them.

Memory does not go directly into the prompt.

Instead, it first passes through a retrieval process that decides what is relevant. Only the selected memories become part of the context for the current request.

In other words, an application may store thousands of memories, but only a handful should influence any single decision.

Memory persists across time.

Context exists for a decision being made now.

Execution state exists only until the current work is complete.

What Actually Counts as Memory?

A production AI application observes an enormous amount of information during its lifetime. Every conversation, tool call, document retrieval, workflow execution, and model response produces new data.

Most of it should never become memory.

Consider an AI coding assistant reviewing a pull request.

During the review, it reads the codebase, analyzes static analysis results, calls external tools, evaluates multiple hypotheses, and generates several intermediate conclusions before arriving at the correct recommendation.

All of that information exists during the review.

Very little of it deserves to survive after the review is complete.

A common mistake is assuming that if something was useful once, it should be stored forever. In reality, production memory should be highly selective. Otherwise, the application slowly accumulates noise instead of knowledge.

A useful way to think about memory is this:

Memory is information intentionally preserved because it has the potential to improve future decisions.

That definition immediately excludes a surprising amount of information.

A temporary tool result is not memory.

An intermediate reasoning step is not memory.

A discarded hypothesis is not memory.

A conversation transcript is not automatically memory.

These pieces of information may help solve the current problem, but once that problem is finished, they usually have little value on their own.

What deserves to persist is the knowledge that remains useful beyond the original interaction.

Returning to our pagination example, the investigation itself is not the memory.

The durable lesson is.

Cursor pagination should use deterministic ordering, including a secondary key whenever the primary sort field is not guaranteed to be unique.

That single lesson is reusable. It can improve future code reviews without replaying the entire investigation that produced it.

This distinction is subtle, but it fundamentally changes how a memory system is designed.

Instead of asking, "What information can we store?", we begin asking a much harder question:

"What information is worth remembering?"

Everything that follows in this article builds on that idea. Before an application can retrieve, revise, reinforce, or forget a memory, it must first decide whether that information deserves to become memory at all.

From Observation to Durable Memory

One of the biggest misconceptions in LLM memory is that every interaction should become a memory.

In reality, production systems shouldn't promote information directly from a conversation into long-term memory. Doing so quickly fills the system with duplicated facts, temporary observations, incorrect conclusions, and outdated knowledge.

Memory is something that should be earned, not automatically created.

Every piece of information begins as an observation.

An observation is simply something the application has seen, heard, or produced. It might come from a user conversation, a tool result, a code review, a retrieved document, or even another AI model. At this stage, the application makes no assumptions about whether the information is useful or even correct.

For example, during a pull request review, the assistant may observe:

"The pagination query returns duplicate records because the ordering is unstable."

That statement is valuable, but it is still only an observation. It hasn't been verified, it hasn't been reused, and it certainly hasn't earned a permanent place in the application's memory.

The next step is deciding whether the observation is worth preserving.

Perhaps the same issue appears repeatedly across different repositories. Maybe another engineer confirms the root cause. Or perhaps automated tests validate the conclusion. At this point, the application has enough evidence to treat it as a candidate memory.

Candidate memories are information that appears useful but still require validation before influencing future decisions.

Only after sufficient evidence exists should the application promote it to durable memory.

For our coding assistant, the durable memory is no longer the original bug report. Instead, it becomes the reusable lesson:

Cursor pagination should always use deterministic ordering, including a secondary key whenever the primary sort field is not guaranteed to be unique.

Notice what changed.

The application didn't remember the conversation.

It remembered the knowledge extracted from the conversation.

This distinction is what separates a production memory system from a conversation archive. Instead of accumulating messages indefinitely, the application continuously filters experience, preserving only information that has demonstrated long-term value.

Thinking about memory as a lifecycle rather than a storage operation also prepares us for the next challenge. Once a memory exists, it doesn't remain static. New evidence may strengthen it, refine it, replace it, or invalidate it entirely.

That's where designing memory becomes significantly more difficult than simply storing information.

Memory Identity Is Harder Than Storage

Once an application decides that something deserves to become memory, another challenge immediately appears.

Has the application learned something new, or has it learned something it already knows?

At first, this sounds like a duplicate detection problem.

In reality, it's much more complicated.

Imagine our AI coding assistant already has this memory:

Use cursor pagination for large datasets.

A few weeks later, another code review uncovers a subtle issue. Cursor pagination still produced duplicate records because the ordering wasn't deterministic.

Now the application receives a new piece of information:

Cursor pagination should include a deterministic secondary key whenever the primary sort field is not unique.

Should it create a brand new memory?

Should it overwrite the existing one?

Or should it treat the new information as a refinement of what it already knows?

This is the problem of memory identity.

Before storing anything, the application needs to understand how the incoming knowledge relates to existing memories. Broadly, a new observation can fall into one of several categories:

A completely new memory.
A duplicate of an existing memory.
Additional evidence supporting an existing memory.
A refinement that adds more detail.
A correction that fixes an earlier understanding.
A contradiction that requires further validation.
A replacement that becomes the new authoritative version.

These distinctions matter because not all updates should be handled the same way.

A duplicate shouldn't increase the size of the memory store.

Additional evidence should strengthen confidence without changing the underlying knowledge.

A refinement should make the memory more precise.

A correction should preserve history while updating the current understanding.

And a replacement should allow future decisions to rely on the newer knowledge without pretending the older version never existed.

Consider how a single memory might evolve over time.

Version 1

Use offset pagination.

Version 2

Use cursor pagination for large datasets.

Version 3

Cursor pagination should use deterministic ordering, including a secondary key when necessary.

These aren't three unrelated memories.

They're three stages in the evolution of the same idea.

A production memory system should recognize that relationship. Otherwise, it gradually fills with fragmented knowledge, forcing retrieval to choose between competing versions of what is fundamentally the same memory.

This naturally introduces another important concept: supersession.

When a memory is superseded, the older version isn't necessarily wrong. It reflects what was believed at a particular point in time. The newer version simply becomes the best available understanding based on current evidence.

Thinking this way changes the role of memory completely.

Instead of acting like a notebook that endlessly collects facts, it begins to resemble a knowledge base that continuously refines its understanding of the world.

How Memories Evolve

Unlike traditional databases, memory isn't static.

Every new experience has the potential to change what the application knows. Sometimes that change increases confidence. Sometimes it adds missing detail. Sometimes it completely replaces an older understanding.

A production memory system should support all of these possibilities instead of treating every update as a new record.

The simplest form of evolution is reinforcement.

Suppose the AI coding assistant repeatedly encounters the same issue across different repositories. Every independent incident confirms that cursor pagination requires deterministic ordering.

The underlying memory doesn't change.

Its confidence does.

As evidence accumulates, the application becomes more certain that this knowledge should influence future decisions.

Not every update strengthens a memory, however.

Sometimes new information reveals that the existing memory is incomplete.

Perhaps another investigation discovers that deterministic ordering alone isn't enough. The cursor must also encode the complete ordering state used by the database to guarantee consistent pagination.

In this case, the memory isn't discarded.

It is revised to become more accurate and complete.

The most significant evolution occurs when new evidence fundamentally changes the application's understanding.

Imagine the team eventually replaces cursor pagination with keyset pagination across every service.

The previous memory wasn't incorrect when it was created. It reflected the best engineering decision at that time.

Today, however, it should no longer guide future decisions.

This is supersession.

Supersession doesn't erase history. Instead, it establishes a new authoritative version while preserving the older one for historical reference.

That distinction is important because engineering knowledge evolves continuously. Best practices change. Frameworks introduce new capabilities. Architectural decisions that were once appropriate may later become technical debt.

A memory system that never updates eventually becomes a liability rather than an asset.

Viewed together, reinforcement, revision, and supersession describe how an application learns over time.

It gains confidence in proven knowledge.

It improves incomplete knowledge.

And it replaces outdated knowledge when better evidence becomes available.

Memory is no longer a collection of stored facts.

It becomes a continuously evolving representation of what the application currently believes to be true.

Why Freshness Is More Than Cache Invalidation

When engineers hear the word freshness, they often think about caches.

Has Redis been updated?

Was the cache invalidated?

Are we reading the latest value from the database?

Those are important questions, but they're only part of the story.

A production memory system isn't concerned only with whether the stored copy is the latest. It also needs to determine whether the knowledge itself is still valid.

Consider this memory:

"The deployment freeze ends on Friday."

Suppose the application correctly updates every cache whenever the canonical store changes. Redis contains the latest revision, every service is synchronized, and no stale cache exists anywhere.

On Saturday, however, that memory is still wrong.

Nothing failed technically.

The knowledge simply expired.

This illustrates an important distinction that traditional cache invalidation doesn't solve.

Memory freshness has multiple dimensions.

Canonical Freshness

Is this the latest stored revision?

This is the type of freshness most systems already understand. Every component should agree on which version of the memory is current.

Temporal Freshness

Is the memory still true today?

Some knowledge naturally expires over time, even if nobody modifies it. Deployment schedules, temporary workarounds, maintenance windows, and feature flags all have a limited lifespan.

Evidential Freshness

How recently was this memory verified?

A recommendation that has been confirmed repeatedly over the last month deserves more confidence than one that hasn't been validated for two years.

Retrieval Freshness

Even if a memory is accurate, should it influence the current decision?

A coding guideline for a frontend application is unlikely to help when reviewing a database migration. The memory is still correct, but it isn't relevant.

Thinking about freshness this way changes how we evaluate memory quality.

Instead of asking, "Is this memory up to date?", we begin asking, "Is this the right memory to use right now?"

That's a much harder question.

It also explains why production memory systems cannot rely solely on storage infrastructure. A perfectly synchronized cache may still return knowledge that is obsolete, irrelevant, or no longer trustworthy.

A cache can be fresh while the memory inside it is obsolete.

Recognizing that distinction is essential because retrieval isn't just about finding similar memories. It's about selecting memories that are both relevant and still deserving of influence.

That's exactly what we'll explore next.

Retrieval Is a Decision, Not a Search

Once an application has accumulated a collection of memories, the next challenge is deciding which ones should influence the current decision.

A common approach is straightforward.

Convert the current request into an embedding, perform a similarity search, retrieve the closest memories, and append them to the prompt.

Semantic search is a valuable tool.

It just isn't the entire retrieval strategy.

Consider an AI coding assistant reviewing a pull request that modifies pagination logic.

Suppose it retrieves these two memories:

Memory A: Cursor pagination should use deterministic ordering, including a secondary key when necessary.

Memory B: The team prefers concise pull request comments.

Both memories are correct.

Both may even be semantically related to the current request.

But only one helps determine whether the pagination implementation is correct.

A production memory system shouldn't ask:

"Which memories are most similar?"

Instead, it should ask:

"Which memories deserve to influence this decision?"

That distinction changes how retrieval is designed.

Semantic similarity becomes one signal among many rather than the only ranking criterion.

Before performing a semantic search, an application often already knows important facts about the request.

It may know the repository being reviewed, the component being modified, the programming language, the environment, or even the type of decision the model is about to make.

These structural signals can dramatically reduce the search space before semantic retrieval expands recall.

After potential memories are identified, additional factors determine which ones should actually be used.

A production retrieval strategy should consider questions such as:

Does this memory belong to the same entity or component?
Is this the latest active version?
Is it still valid?
How strong is the supporting evidence?
Has it been useful in similar situations before?
Does it add new information, or simply repeat another retrieved memory?

Notice that similarity isn't even the most important question.

The objective isn't to retrieve the closest memories.

The objective is to retrieve the most useful ones.

That's an important difference because the quality of an AI application's decisions depends less on how much it remembers and more on whether it remembers the right things at the right time.

Retrieval, therefore, isn't simply a search problem.

It's a ranking problem.

And ultimately, it's a decision problem.

Memory Interference: When Remembering More Makes Decisions Worse

It's tempting to think that more memory always leads to better decisions.

In practice, the opposite is often true.

As a memory system grows, the challenge shifts from remembering information to preventing the wrong information from influencing the model.

This is known as memory interference.

Memory interference occurs when retrieved memories compete with, contradict, or dilute one another, reducing the quality of the final decision.

Imagine our AI coding assistant retrieves the following memories during a pull request review:

A memory describing an old pagination strategy that has since been replaced.
Three nearly identical memories created from duplicate incidents.
A frontend coding guideline that happens to share similar terminology.
The correct and current pagination recommendation.

Individually, none of these memories are necessarily incorrect.

Together, they create noise.

Instead of helping the model focus on the most relevant engineering knowledge, they compete for attention inside the limited context window.

Interference can appear in several forms:

Duplicate interference, where repeated memories crowd out more useful information.

Temporal interference, where outdated knowledge competes with its newer replacement.

Scope interference, where knowledge from one project, environment, or domain leaks into another.

Contradictory interference, where competing memories remain active without a clear resolution.

The goal of a production memory system is not simply to retrieve relevant memories.

It is to retrieve a coherent set of memories that work together.

This is why retrieval quality should be evaluated as a collection rather than as individual results.

A memory that is useful on its own may become harmful when combined with outdated, duplicate, or contradictory information.

Designing a reliable memory system therefore requires more than storing, updating, and retrieving knowledge.

It also requires continuously reducing interference so that every retrieved memory has a clear purpose.

In many cases, improving memory quality isn't about remembering more.

It's about remembering less, but remembering better.

Consolidation: Turning Experience into Knowledge

Not every lesson is learned from a single experience.

Some of the most valuable knowledge emerges only after an application observes the same pattern repeatedly.

Imagine our AI coding assistant encounters three different incidents over several weeks.

In the first, cursor pagination returns duplicate records because the ordering isn't deterministic.

In the second, a database handles NULL values differently, causing inconsistent results.

In the third, the cursor fails because it doesn't encode every field used in the database's ordering.

Viewed individually, these appear to be unrelated problems.

Taken together, they reveal a broader engineering principle.

A cursor should encode the complete ordering state used by the database.

That statement isn't copied from any single incident.

It's derived from all of them.

This process is called consolidation.

Consolidation identifies recurring patterns across multiple experiences and transforms them into reusable knowledge. Instead of remembering every incident independently, the application learns the underlying principle that explains them.

This is fundamentally different from summarization.

A summary makes existing information shorter while preserving the original meaning.

Consolidation produces new knowledge by combining evidence from multiple observations.

That's an important distinction because a shorter memory isn't necessarily a better one. Compressing three incidents into a paragraph still leaves the application remembering three incidents.

A consolidated memory captures the lesson those incidents collectively teach.

One practical consideration is preserving traceability.

Even after creating a consolidated memory, the application should retain links to the supporting observations. If new evidence contradicts the conclusion later, those original experiences provide the context needed to validate, revise, or even revoke the derived knowledge.

Over time, this allows the memory system to grow in understanding rather than simply in size.

Instead of accumulating isolated experiences, it continuously transforms experience into knowledge.

That's what makes long-term memory valuable.

Managing the Memory Lifecycle

Creating a memory is only the beginning.

As an application learns and the world around it changes, memories need continuous maintenance. Some remain valuable for years. Others become outdated within days.

Without active lifecycle management, a memory system gradually loses reliability, not because it forgets too much, but because it remembers things it no longer should.

One way this happens is through decay.

Not every memory deserves the same level of influence forever. If a memory hasn't been validated or used for a long time, its confidence can gradually decrease. The memory still exists, but it becomes less likely to influence future decisions until new evidence reinforces it.

Some memories have a known lifespan from the moment they are created.

A deployment freeze, a temporary workaround, or a maintenance window all become invalid after a specific point in time.

This is expiration.

Unlike decay, expiration is predictable. The application already knows when the memory should stop influencing decisions.

Sometimes a memory must be withdrawn immediately.

Perhaps a security recommendation is discovered to be incorrect, or a previously trusted procedure is found to introduce data corruption.

In these situations, the memory should no longer participate in retrieval, even if it remains stored for auditing or historical analysis.

This is revocation.

Another important distinction is forgetting.

Forgetting doesn't necessarily mean removing data from storage.

It means the memory is intentionally excluded from influencing future decisions. The underlying record may still exist for compliance, debugging, or historical reference, but operationally, the application behaves as though that memory no longer exists.

Finally, there is deletion.

Deletion is a storage operation. The memory is physically removed, and future retrievals cannot return it.

While forgetting and deletion often occur together, they solve different problems.

Forgetting changes behavior.

Deletion changes persistence.

Recognizing that distinction gives applications much more flexibility. A system can stop using a memory immediately while preserving its history for auditing, or permanently remove it when retention policies or user requests require it.

A mature memory system isn't defined by how much information it stores.

It's defined by how responsibly it manages the entire lifecycle of that information.

Framework Memory vs Application Memory

If you've worked with LLM frameworks, you've probably come across features with names like conversation memory, summary memory, checkpointers, or vector store memory.

These are incredibly useful building blocks.

They just aren't a complete memory system.

A conversation buffer preserves previous messages.

A summary memory compresses long conversations into a shorter representation.

A checkpointer restores the execution state of a workflow after an interruption.

A vector store retrieves semantically similar information.

Each of these solves a specific technical problem.

None of them decides what your application should remember.

That decision belongs to your application.

Only your application understands whether a new observation should become durable memory, whether it refines an existing memory, whether it contradicts previous knowledge, whether it has expired, or whether it should be forgotten altogether.

In other words, frameworks provide the infrastructure.

Your application defines the memory semantics.

This distinction is easy to overlook because most frameworks intentionally stay generic. They expose flexible primitives that work across many different use cases, from customer support bots to coding assistants to research agents.

The responsibility for designing a reliable memory model, however, cannot be generalized. It depends entirely on the behavior your application is expected to learn, retain, and apply over time.

That's why choosing a framework is only a small part of building long-term memory.

The real challenge is deciding what deserves to influence future decisions.

Conclusion

When people talk about LLM memory, the conversation often starts with chat history, vector databases, or context windows.

Those are important pieces of the puzzle, but they don't define a production memory system.

A production memory system isn't built by storing more information. It's built by making better decisions about information. It decides what deserves to become memory, how memories evolve as new evidence arrives, when they should influence future decisions, and when they should quietly fade away.

One point worth emphasizing is that memory and context are not the same thing.

Memory represents the knowledge distilled from past interactions, observations, and evidence. Context is the information assembled for a specific model invocation. Memory may contribute to context, but it is only one possible input. Understanding context engineering deserves its own discussion, so this article intentionally focused on memory as an independent system.

As AI applications become more capable, designing memory will become less about storage and more about knowledge management. The systems that perform well won't be the ones that remember everything. They'll be the ones that remember the right things, evolve their knowledge over time, and forget what no longer matters.

In this article, we've built the conceptual foundation for what a production memory system should look like. In the next article, we'll move from concepts to architecture how a production memory service handles observation ingestion, identity resolution, retrieval scoring, and the lifecycle events that govern what gets forgotten. The components, the data flows, and the trade-offs I ran into while building this at scale.

Because in the end, a production memory system is not defined by where information is stored.

It is defined by how experience is transformed, evolved, retrieved, and eventually forgotten.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️

Graph Engineering for AI Coding Agents: Beyond Prompt Loops

NARESH — Thu, 30 Jul 2026 18:52:28 +0000

How context, contracts, deterministic validation, and targeted repair shaped ContractGraph.

Over the past week, I've been experimenting with something that's been getting a lot of attention recently: Graph Engineering.

When I first started reading about it, almost every explanation focused on graphs, nodes, and DAGs. Those are important pieces of the puzzle, but I couldn't shake the feeling that they weren't answering the question I actually cared about.

Why do we suddenly need graph engineering for AI coding agents?

After spending months using tools like Claude Code and, more recently, Codex, I noticed something interesting. Most coding tasks don't fail because the model can't write code. They fail because long-running workflows gradually lose structure. Context becomes inconsistent, requirements get forgotten, outputs become difficult to verify, and when something goes wrong, the easiest solution is often to start the entire task again.

That observation eventually led me to build a small research framework called ContractGraph.

I didn't build it to compete with existing frameworks, and I didn't build it around a particular model. My goal was much simpler: could a workflow be designed so that every step had a clear objective, a well-defined contract, and a deterministic way to decide whether it was actually complete?

The result was a contract-driven graph that orchestrated a research workflow through isolated tasks, explicit validation, and targeted repair instead of treating the entire workflow as one giant prompt.

Although my first implementation focused on research agents, the underlying ideas are much broader. The same approach could be applied to coding agents, document pipelines, evaluation systems, or any workflow where multiple AI-driven steps need to produce reliable, verifiable results.

This article isn't a tutorial on building graph frameworks, nor is it a comparison between Claude Code, Codex, LangGraph, or any other tool. It's a walkthrough of the ideas that led me to build ContractGraph, what worked, what broke, and why I now think graph engineering is less about drawing DAGs and more about designing reliable AI workflows.

A Quick Note on Graph Engineering

Graph engineering is the practice of breaking a complex AI workflow into smaller execution units and connecting them through explicit rules.

A graph-engineered system usually contains:

Multiple execution units + explicit routing + shared state + verification + durable coordination

Each node handles one responsibility, edges decide what happens next, shared state carries information across the workflow, and verification checks whether the system should continue, retry, repair, or stop.

The Two Questions That Kept Bothering Me

When I started exploring graph engineering, I expected to spend most of my time thinking about nodes, edges, and execution flow. Surprisingly, those weren't the hardest parts.

Two questions kept coming back.

What information should an AI agent be allowed to treat as truth?

And once it finishes a task,

How do I know the output actually satisfies what I asked for?

The more I experimented with long-running AI workflows, the more I realized these questions mattered far more than the graph itself. If an agent starts with the wrong information or finishes without a reliable way to verify its work, the graph is simply moving mistakes from one node to another.

Those two questions eventually became the foundation of everything I built.

I called them Context and Contract. One defines what an agent is allowed to believe. The other defines what it must produce.

Everything else in ContractGraph was built around those two ideas.

Context and Contracts: The Two Building Blocks

The more I worked with AI workflows, the more obvious one thing became: the graph itself was not the hardest part.

The real challenge was making sure every node started with the right information and produced something that could be verified before the workflow moved forward.

That led me to two ideas.

The first was Context.

Instead of giving every agent access to everything, each node should receive only the information it is allowed to trust. That could include user requirements, project constraints, verified documents, or accepted outputs from earlier nodes.

The second was Contract.

Every node should have a clear definition of what it must produce. If the result does not satisfy that contract, the workflow should not continue as though nothing happened. It should validate the output, repair the failed step, or stop.

In simple terms:

Context defines what an agent is allowed to believe.

Contracts define what an agent must deliver.

Once those two boundaries are explicit, the graph becomes much easier to reason about. Every node has a clear purpose, a limited source of truth, a measurable output, and a deterministic check before handing control to the next step.

From an Idea to a Graph

Once Context and Contracts were separated, the next question became:

How do these two ideas turn into an executable workflow?

I didn't want to hardcode different graphs for research, coding, document generation, or any other task. The goal was to compile a workflow dynamically from the problem itself rather than from predefined execution paths.

That process begins with two simple steps.

The first constructs the Context.

Rather than treating every file or document as equally trustworthy, the workflow builds a canonical context: a validated source of truth that every node can rely on during execution. Depending on the task, that context might be derived from project documentation, user requirements, source code, design documents, or other trusted inputs.

The second constructs the Contract.

Instead of describing the task in natural language alone, it generates a structured contract that defines what every stage must produce, how nodes depend on one another, and what must be validated before execution can continue.

With those two building blocks in place, the workflow is compiled.

The initialization stage doesn't execute anything yet. Its job is to plan. It produces the execution graph, prepares scoped context packets for each node, generates reusable manifests and shared skills, and determines the execution order directly from the contracts instead of relying on hardcoded logic.

Execution then becomes surprisingly simple.

Each node receives only its scoped context, performs its assigned work, passes through deterministic validation, persists the accepted result, and hands control to the next node. If validation fails, only the affected node is repaired before execution resumes, avoiding a full restart of the workflow.

The result is a graph whose structure is determined by the context and contracts supplied at the beginning of the workflow rather than by a fixed implementation. The same compilation pipeline can therefore assemble workflows for research, coding, documentation, or entirely different problems without changing the underlying runtime.

The diagram below illustrates the execution lifecycle I experimented with while building ContractGraph.

Putting ContractGraph to the Test

Designing the framework was only half the challenge. The more important question was whether the architecture would continue to hold once the workflow became large enough to expose its weaknesses.

My first experiments used a much larger workflow running in Claude Code. It consisted of dynamically generated execution nodes, nested planning stages, and long-running research tasks. The overall quality was excellent, but one limitation quickly became apparent: completing the workflow required a significant amount of model computation.

That naturally led to a question:

Could the same graph architecture still produce reliable workflows on a much smaller, lower-cost model?

To explore that, I designed a second experiment around a study-plan generation workflow. Instead of relying on a frontier model, I executed the compiled graph through OpenCode using Big Pickle. The underlying execution model remained the same. The workflow was compiled dynamically, generated specialized execution nodes, validated every stage deterministically, repaired failures when necessary, and assembled the final artifact through the same graph pipeline.

The objective wasn't to compare models or argue that one was better than another. It was to understand how much of the workflow's reliability came from the execution architecture itself, rather than from the capabilities of the underlying model.

The results surprised me.

The biggest challenges weren't model failures.

They were workflow failures.

And that's where graph engineering became interesting.

Where It Broke

The experiment didn't convince me that graph engineering prevents failures.

It convinced me that it makes failures visible.

During execution, I encountered all kinds of issues: incomplete metadata, missing evidence, unresolved references, duplicated content, invalid lifecycle states, incorrect trace information, and even simple encoding problems. None of these were caused by the graph itself. They were exactly the kinds of failures you'd expect to see in long-running AI workflows.

The difference was that every failure had a precise location.

Instead of treating the entire workflow as a single failed execution, I could identify the exact node that violated its contract, repair only that node, run deterministic validation again, and continue execution from that point onward. The rest of the workflow remained intact because every successful stage had already been verified and persisted.

That fundamentally changed how I thought about graph engineering.

I stopped seeing it as a way to make models smarter.

I started seeing it as a way to make failures local, observable, and recoverable.

The graph didn't eliminate mistakes.

It gave every mistake an address.

Why Deterministic Validation Matters

One thing became clear during the experiment.

An AI model can confidently tell you that a task is complete. That doesn't mean it actually is.

Large language models are excellent at reasoning, summarizing, and making decisions, but they're not a reliable way to verify whether a workflow has satisfied its own requirements.

That's why every node in ContractGraph ends with deterministic validation.

Instead of asking another model whether the output "looks good," the workflow checks objective conditions: Is the required schema complete? Are all dependencies satisfied? Is every required artifact present? Are references resolved? Has the node actually produced everything its contract promised?

These aren't subjective questions. They're either true or false.

That separation turned out to be one of the most valuable ideas in the framework.

I let the model focus on reasoning, while deterministic validation focused on verification.

In other words:

A model can argue that its work is complete. A deterministic validator can prove that a required field is missing.

Why I Still Think It's Worth It

Long before I started experimenting with graph engineering, I had already been treating AI coding agents as a small team rather than a single assistant. It wasn't unusual for me to have Claude Code, Codex, and OpenCode working on different parts of the same repository at the same time.

That introduced a problem I hadn't anticipated.

Different agents could eventually need to modify the same files or depend on the same implementation. Without coordination, one agent could unintentionally overwrite another's work or invalidate assumptions made earlier in the workflow.

My solution was surprisingly simple. Every task had ownership. If an agent was responsible for a particular part of the codebase, other agents weren't allowed to modify those files until ownership was released. Combined with validation after every significant step, that prevented many conflicts before they reached the repository.

The trade-off was obvious.

This style of development consumes significantly more tokens than asking a single model to generate a feature in one conversation. Every validation, retry, repair, and coordination step adds additional work.

But the outcome felt different.

Instead of repeatedly fixing regressions introduced by previous fixes, the workflow became much more predictable. When I shipped features using this approach, they required remarkably little rework because every stage had already been verified before the next one began.

I don't think graph engineering is something every prompt should use. For small changes, it's unnecessary overhead.

But for larger features, long-running implementations, or workflows involving multiple agents, I think the additional cost buys something much more valuable than cheaper inference.

It buys confidence.

The Actual Cost of Graph Engineering

If this article has made graph engineering sound like a silver bullet, it isn't.

Every capability comes with additional work. Context preparation, contract generation, graph compilation, deterministic validation, targeted repair, and final aggregation all require extra model calls and tokens. Compared to a single prompt, the workflow is undeniably more expensive.

I experienced that firsthand.

My earliest experiments used frontier models and produced excellent results, but they also consumed a substantial amount of model computation. That became one of the main reasons I wanted to explore whether the same architecture could run on smaller, lower-cost models.

What I learned was simple:

Graph engineering doesn't reduce compute. It deliberately reallocates it.

Instead of spending computation recovering from regressions, restarting failed workflows, or repeatedly fixing downstream mistakes, the system spends it upfront on planning, verification, and failure isolation.

That changes the optimization target.

Graph engineering isn't designed to produce the cheapest execution.

It's designed to produce the most predictable execution.

During one of my OpenCode experiments, the workflow processed approximately:

These numbers shouldn't be interpreted as "graph engineering is cheap." If anything, they demonstrate the opposite. Graph engineering intentionally spends additional computation to gain something more valuable: failure isolation, auditability, resumability, and confidence.

Whether that trade-off is worthwhile depends entirely on the problem you're solving.

When Graph Engineering Is Worth It

After spending time building and experimenting with ContractGraph, I don't think graph engineering is something that every AI workflow needs.

If you're asking an AI to generate a SQL query, write a small function, or make a few UI changes, a well-written prompt is usually enough. Adding contracts, validation layers, and execution graphs would only make the workflow more expensive and more complicated.

Where I think graph engineering starts to shine is when the cost of failure becomes higher than the cost of additional computation.

That includes long-running research tasks, large feature implementations, multi-agent workflows, security-sensitive systems, enterprise automation, or any process where the output needs to be verifiable and reproducible.

The more steps a workflow contains, the more valuable explicit coordination becomes.

For me, the biggest takeaway wasn't that graph engineering makes AI smarter.

It makes complex AI workflows easier to understand, validate, recover, and evolve.

I also don't think we've reached the end of this idea.

There are still many areas I haven't explored, such as dependency-aware execution, smarter scheduling, resource optimization, partial graph recompilation, and more advanced coordination strategies between agents. Those are experiments for another day.

For now, one thing has become clear to me:

Use prompts to solve tasks. Use graph engineering to build systems.

Final Thoughts

When I started exploring graph engineering, I thought I was going to learn about graphs.

Instead, I ended up thinking much more about boundaries, verification, and coordination.

Building ContractGraph changed the way I think about AI workflows. The graph itself wasn't the interesting part. The interesting part was giving every step a clear purpose, limiting what an agent is allowed to believe, defining what it must produce, and proving that the work is actually complete before moving forward.

I don't think this is the only way to build AI systems, and I certainly don't think every workflow needs this level of structure. But for complex, long-running, or multi-agent workflows, I believe graph engineering offers a practical way to make AI systems more predictable and easier to evolve.

If there's one idea I'd like you to take away from this article, it's this:

Prompt engineering tells an agent what you want. Context defines what it's allowed to believe. Contracts define what it must produce. Graph engineering determines how the system proves that the work is complete.

ContractGraph is still evolving, so I'm not publishing the repository just yet. Before I open-source it, I want to simplify the architecture and make sure the ideas are presented clearly rather than tied to my own experiments.

If you're working on graph engineering, agentic workflows, or reliable AI systems and would like to discuss the architecture, validation pipeline, or some of the implementation details, feel free to reach out. I'm always happy to exchange ideas with people exploring this space.

Once the framework reaches a point where I think it's genuinely useful to others, I'll publish a sanitized version of the repository along with the supporting documentation.

This article is not a complete guide to graph engineering. For a broader introduction, refer to this guide:

https://www.aibuilderclub.com/blog/graph-engineering-guide-2026

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️

I Switched from Claude Code to Codex. Here's What Surprised Me.

NARESH — Sat, 25 Jul 2026 06:54:28 +0000

TL;DR

After using Claude Code for nearly six months and Codex for the past few weeks, I've realized they aren't competing by solving different problems. They're solving the same problem with two very different philosophies.

Claude Code gives you a highly collaborative coding experience. Codex focuses on getting the work done with minimal interaction. Neither approach is objectively better, but one will probably fit your workflow better than the other.

This article isn't about benchmarks or declaring a winner. It's about the day-to-day experience of living with both tools and the small behavioral differences that only become obvious after using them on real projects.

For nearly six months, Claude Code was my default way of building software.

During that time, I went well beyond writing prompts. I built custom skills, experimented with sub-agents, refined different workflows, and spent enough time inside the terminal that AI-assisted development simply became part of my daily routine. By the time I looked at Codex, I already had a workflow I was genuinely happy with, so I wasn't searching for a replacement.

A few weeks ago, I subscribed to Codex out of curiosity. I wanted to see how another mature coding agent approached the same job, using the same projects and the same way of working that I'd already developed with Claude Code.

What caught me off guard wasn't the quality of the code. Both tools are more than capable of producing production-ready implementations.

The real difference was the experience.

Claude Code always felt like another engineer sitting beside me, constantly sharing what it was doing and inviting me into the process. Codex took almost the opposite approach. I'd hand it a task, let it work, and a while later it would return with the implementation. The interaction was quieter, but the outcome was often the same.

That shift completely changed the way I think about AI coding tools.

This isn't a benchmark, a feature comparison, or an attempt to decide which tool is objectively better. It's a reflection on what changed after spending months with Claude Code, then intentionally switching to Codex. If there's one thing I learned from using both, it's that the biggest difference isn't the models themselves. It's the way they change how you build software.

Claude Code Feels Like You're Building Together

The first thing that stood out to me about Claude Code wasn't the quality of the code it generated. It was how much of the process it let me see.

Every task feels collaborative. As it works, it continuously shows which files it's reading, what it's trying to understand, and the changes it's planning to make. Instead of waiting for a finished result, you're involved from the beginning. If something looks off, it's easy to step in, adjust the direction, and keep moving without waiting for the entire task to finish.

That completely changes the experience. It feels less like delegating work to an AI and more like pairing with another engineer who's talking through the implementation as it unfolds.

The longer I used Claude Code, the more I appreciated everything around the model. Skills, sub-agents, and project-specific instructions aren't features you fully appreciate on day one. They become valuable over time because they gradually shape Claude Code around the way you work. After a few months, it stopped feeling like a generic coding assistant and started feeling like an environment that understood my project's conventions.

The terminal experience is another reason I kept coming back to it. I rarely use the IDE extension because the terminal fits my workflow much better. Running multiple sessions, assigning different tasks to different agents, and coordinating them from the command line feels natural to me.

There is one trade-off I noticed, though.

Over several months of daily use, Claude Code consistently exhausted my usage limits sooner than Codex. Before anyone blames context management, I should mention that I follow a fairly disciplined workflow. I regularly compact conversations before they grow too large, split bigger problems into smaller tasks, and use separate sessions whenever it makes sense. More importantly, I followed the same workflow when I started using Codex, so I wasn't comparing two different ways of working.

That makes me think the difference comes from how the two tools are designed rather than how I use them.

Claude Code invests more in keeping you involved throughout the implementation. That constant communication is one of the reasons I enjoyed using it so much, even if it also meant reaching my usage limits sooner in my own workflow.

If I had to describe Claude Code in a single sentence, I'd say this:

Claude Code optimizes for the experience of building software, not just delivering the final implementation.

Codex Feels Like You've Delegated the Work

The biggest surprise after switching to Codex wasn't a new feature. It was how little it asked for my attention.

With Claude Code, I had become used to following the implementation as it happened. Codex works differently. I describe the task, it acknowledges it, and then it gets to work. There aren't constant progress updates or a running commentary explaining every decision. Most of the time, it simply disappears and returns when the work is finished.

That took a few days to get used to.

At first, the silence made me feel like less was happening. I was so accustomed to seeing every step that the lack of updates almost felt strange. But after using it for a while, I realized the implementation quality hadn't changed. The only thing that had changed was how much of the process I was seeing.

That naturally changed the way I used it.

Instead of collaborating throughout the implementation, I started treating Codex like an engineer I could delegate work to. I focused on writing a clear task, let it work independently, and came back to review the result once it was done.

One behavior stood out almost immediately.

Whenever Claude Code hit its usage limit, my session usually paused until the limit reset. Codex handled the situation differently. If a task was already in progress, it would usually finish the implementation, wait for any running commands to complete, leave behind a detailed summary of everything it had done, and only then tell me I'd reached my limit. It sounds like a small detail, but it made interrupted sessions feel much less disruptive.

I also noticed a consistent difference in how far my usage budget went.

Using exactly the same workflow I described earlier, I was able to complete noticeably more work before hitting my limits. I don't know what happens internally, so I won't speculate about the reason, but it was consistent enough that I noticed it within the first couple of weeks.

That's what ultimately changed my perspective.

Claude Code feels like a tool that's designed to keep you involved throughout the journey. Codex feels like a tool that's designed to let you move on to something else while the work gets done.

If I had to describe Codex in one sentence, I'd say this:

Codex optimizes for finishing the task with as little interruption as possible.

Three Differences That Changed My Perspective

After using both tools, I realized I wasn't really comparing models anymore. I was comparing two very different approaches to AI-assisted software development.

The quality of the code was never the deciding factor for me. The real difference was how each tool expected me to work with it.

1. Collaboration vs Delegation

This was the biggest shift.

Claude Code is built around collaboration. It keeps you involved, shares its progress, and makes implementation feel like a conversation. You're constantly aware of what it's doing, which makes it easy to change direction before it goes too far.

Codex takes the opposite approach. It assumes you've already defined the problem clearly, so instead of keeping you updated, it focuses on finishing the task and reporting back once there's something worth reviewing.

Neither approach is better. They simply optimize for different kinds of developers. Some days I want to stay involved throughout the implementation. Other days I'd rather hand off the work and spend my time thinking about the next problem.

2. Attention vs Throughput

The second difference was something I didn't appreciate until I had spent a few weeks with Codex.

Claude Code does a great job of keeping your attention on the implementation. That makes the experience feel collaborative, but it also means you're interacting with it far more often.

Codex demands much less attention. I found myself assigning a task, switching to something else, and coming back later to review the result instead of following every intermediate step.

That small difference changed my workflow more than I expected. Instead of constantly context-switching between my own work and the AI's progress, I could treat implementation as something happening in parallel.

3. How They Spend the Budget

One pattern kept showing up throughout my testing.

Using the same workflow, I consistently managed to complete more work in Codex before reaching my usage limits. Claude Code reached those limits noticeably sooner.

I don't know why that happens internally, so I'm not going to speculate. It's simply what I observed after using both tools on real projects.

To me, it reflects the priorities of each product.

Claude Code seems willing to spend more resources creating a richer, more interactive experience. Codex appears more focused on maximizing how much work gets done before the session ends.

Neither choice is right or wrong. They're optimizing for different definitions of a good developer experience.

By the end of those few weeks, I stopped asking myself which tool was better.

The more useful question became much simpler.

Do I want to collaborate on this task, or do I want to delegate it?

Once I started thinking about it that way, choosing between Claude Code and Codex became much easier.

So, Which One Is the Better Fit?

After spending time with both, I don't think the answer comes down to which model writes better code.

It comes down to what you expect from an AI coding assistant.

These days, I usually have a clear picture of what I want to build before I even open my editor. At that point, I don't need the AI to explain every decision it makes. I'd rather hand over the implementation, work on something else, and come back once it's finished. That's why I've been reaching for Codex more often. It fits the way I like to work today.

If your workflow is similar, I think you'll probably enjoy Codex as well.

On the other hand, there are plenty of situations where I'd still recommend Claude Code.

When I'm exploring a new codebase, designing a new architecture, or working through an unfamiliar problem, that continuous feedback becomes genuinely useful. Being able to see what the agent is reading, how it's approaching the problem, and where it's heading makes it much easier to course-correct before small mistakes become larger ones.

One thing I don't think gets enough credit is Claude Code's harness.

The model is only one part of the experience. The terminal workflow, skills, sub-agents, project instructions, and the overall developer experience all work together to create an environment that feels polished and intentional. Even after trying Codex, I still think Claude Code has the stronger harness.

That's actually why one of my favourite setups today is Claude Code running OpenAI models.

For me, it combines the best parts of both worlds. I get the workflow and tooling that Claude Code has refined over time, along with the execution style that I've personally enjoyed with OpenAI's models.

If there's one piece of advice I'd give, it's this:

Don't become loyal to a single model.

This space changes far too quickly. New models appear every few months, existing ones improve, and yesterday's recommendation can become outdated surprisingly fast. Instead of trying to find the one perfect tool, spend time understanding how each one fits into your workflow.

The more comfortable you become switching between them, the more value you'll get from all of them.

One Piece of Advice Before You Choose

If you're just getting started with AI coding tools, there's one mistake I'd recommend avoiding.

Don't spend weeks trying to save $17 or $20 by forcing yourself to use a setup that doesn't fit your machine or your workflow.

I say that because I did exactly that.

Before I ever paid for Claude Code, I spent a lot of time experimenting with Ollama and local models. My laptop wasn't powerful enough to run the larger models comfortably, so I mostly relied on smaller ones for tasks like generating unit tests and trying out different agent workflows.

It was a useful learning experience, but it also made one thing very clear.

I didn't want to build real projects that way.

Once your development environment starts growing, the cost of running everything locally becomes much more obvious. Your IDE, Docker containers, databases, browsers, and the language model are all competing for the same CPU, memory, and GPU resources. On smaller projects that's manageable. On larger projects, it quickly becomes frustrating.

The model itself was only part of the problem.

The overall experience never matched what I got from Claude Code or Codex. The frontier models were consistently better at understanding larger codebases, handling longer tasks, and producing reliable implementations with much less effort on my side.

That's when I stopped thinking of a subscription as an expense and started thinking of it as a productivity investment.

If you're building software regularly, ₹2,000 a month is often worth paying if it saves you hours every week.

I'd also be careful about believing every "100% free Claude Code alternative" video you see online.

Most of them prove that something is possible, not that it's enjoyable to use every day.

There's nothing wrong with experimenting using local or open models. In fact, I think it's one of the best ways to learn how agentic coding works. But when you're working on projects that actually matter, I still prefer mature coding agents paired with mature frontier models.

The difference isn't just the model.

It's everything around it.

Permission prompts before destructive commands, better task orchestration, more predictable behaviour, and a polished harness all become surprisingly important once you start relying on these tools every day. They're the kinds of details you rarely notice until they're missing.

If you're a student or you're just exploring AI coding for the first time, there's nothing wrong with using the free options to learn the basics. But once you start building projects seriously, I'd recommend investing in a good coding agent instead of spending weeks trying to recreate the same experience for free.

In my experience, you'll learn far more by building software than by constantly trying to optimize away the subscription cost.

Final Thoughts

After spending months with Claude Code and then switching to Codex, I don't think I've found the perfect AI coding tool.

What I found was that the workflow matters far more than I originally expected.

Two tools can produce equally good code and still feel completely different to use. That isn't because one model is smarter than the other. It's because they're designed with different ideas of what a great developer experience looks like.

If you're thinking about trying one of these tools, don't subscribe just because someone says it's the best. Spend some time understanding how it fits into your own workflow. That's exactly what I did before paying for Codex. I read articles, watched demos, compared real-world experiences, and only subscribed once I felt it solved a problem I actually had.

You're spending your own money, so make sure you're getting value in return.

I'd also avoid becoming too attached to any single model or platform. This space moves incredibly fast. New models are released every few months, existing ones improve constantly, and the tool that works best for you today might not be the one you use six months from now.

The real skill isn't picking the "right" AI coding assistant.

It's learning how to work effectively with whichever one you're using.

That's a skill that will stay valuable long after today's models have been replaced.

If you've used Claude Code, Codex, Cursor, or any other coding agent, I'd love to hear what your experience has been. One of the most interesting things about this space is that two developers can use the same tool every day and still come away with completely different perspectives.

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️