DEV Community: Kainat Saricioglu

Your API Was Built for Humans. Now an AI Agent Is Calling It.

Kainat Saricioglu — Wed, 23 Sep 2026 16:06:58 +0000

For years we designed APIs around one assumption: a human is somewhere on the other side of the request. A user clicks a button, the frontend sends a request, the API validates it, the backend does the work, and a response comes back.

Then AI agents showed up. Now the caller might not be a browser. It might be an agent deciding which endpoint to call, what arguments to send, whether to call something else afterwards, and whether the result is good enough to continue.

Your API can still be perfectly correct. Authentication still works, the endpoints still follow REST conventions, and nothing is broken. But an API that is comfortable for humans is not automatically a good API for agents, and the gap shows up in places backend engineers already care about: contracts, permissions, retries, errors, and tracing.

Quick definitions: An AI agent is a program built around a language model that decides which actions to take to complete a task, instead of following a fixed script. A tool is one action you expose to that agent, such as "cancel an order." MCP (Model Context Protocol) is an open standard that describes how an AI client discovers and calls those tools, so every integration doesn't have to invent its own mechanism.

The difference isn't just "who is calling"

In a traditional application, the frontend already knows almost everything. It knows what the endpoint does, which fields are required, which values are valid, and what should happen after a successful response. The API can afford to be terse, because a developer has already read the docs and encoded that knowledge in the client.

flowchart LR
    U([User]) --> FE[Frontend]
    FE -->|POST /orders| API[Backend API]
    API --> DB[(Database)]

An agent starts with far less. Before it can call anything, it has to discover which tools exist, pick one, construct the arguments, interpret the response, and decide what to do next.

flowchart LR
    U([User]) --> AG[AI agent]
    AG --> D[Discover tools]
    D --> S[Choose a tool]
    S --> P[Construct arguments]
    P -->|API call| API[Backend API]
    API --> R[Interpret response]
    R --> N[Decide next step]
    N --> S

The agent isn't just executing a workflow. It's participating in one. So your API now has to communicate more than "here is an endpoint." It has to communicate what can be done, what is allowed, what input is expected, and what the operation actually means.

1. Make operations explicit

Take an endpoint like POST /orders/action. A frontend developer reads the docs once and moves on. An agent has to guess: does "action" mean create, cancel, approve, retry, or update?

An agent-facing interface works better when each operation is named and described on its own:

{
  "name": "cancel_order",
  "description": "Cancel an existing order that has not yet been shipped.",
  "parameters": {
    "orderId": { "type": "string", "description": "The order's identifier." },
    "reason":  { "type": "string", "description": "Why the customer is cancelling." }
  }
}

The description is no longer just documentation. It becomes part of the model's decision-making context, which means vague wording leads directly to wrong tool choices. It's worth describing side effects too: an agent should be able to tell from the description alone that refund_payment moves real money.

2. Responses are part of the conversation

Traditional responses are written for code that already knows the domain:

{ "id": 4821, "status": 3, "code": "PENDING", "x": true }

A frontend developer knows that status 3 means pending and that x is the cancel flag. An agent doesn't, and it will guess. Compare that with a response that carries its own meaning:

{
  "orderId": "4821",
  "status": "pending",
  "canCancel": true,
  "message": "The order has not been shipped and can still be cancelled."
}

This doesn't mean stuffing paragraphs into every payload. It means the contract should expose meaning rather than internal implementation details. A useful test: if you deleted the frontend entirely, could a caller understand this response without guessing? If not, the contract is leaning on knowledge that lives outside the API.

3. Don't turn every API into an AI API

I wouldn't rewrite existing REST APIs just because agents exist. Your system still serves browsers, mobile apps, internal services, scheduled jobs, and partner integrations, and those consumers have different needs.

The more interesting question is what an agent-facing layer looks like on top of what you already have. Often it's a thin adapter that exposes a small set of well-described tools and calls the same services underneath:

flowchart TD
    AG([AI agent]) --> T[Agent tool layer / MCP server]
    FE([Browser / mobile]) --> API[Existing REST API]
    T --> SVC[Application services]
    API --> SVC
    SVC --> DB[(Database)]

The business logic doesn't change. What changes is that one more consumer gets an interface designed for how it actually works. This is also why MCP is interesting: it gives that layer a standard shape for describing and invoking tools.

4. "The agent has my token" is not a security architecture

This is where things get serious. Suppose your API already exposes GET /customers/{id}, POST /payments, POST /refunds, and DELETE /users/{id}. A human user may be allowed to do all of that. Now an agent is doing it on the user's behalf, and the question becomes: who is actually authorized here, and for what?

The easy shortcut is to hand the agent one powerful access token and let it call anything. It works, right up until a prompt injection, a misread instruction, or a wrong tool choice turns into a deleted customer. (Prompt injection is when text the model reads, such as the contents of an uploaded document, contains instructions that hijack what the agent does next.)

The MCP authorization specification is strict about this, and the reasoning applies well beyond MCP. A protected server acts as an OAuth 2.1 resource server, meaning it must validate that a token was issued for itself and reject anything else. Passing a client's token straight through to an upstream API is explicitly forbidden, because the upstream service can no longer tell who is really asking.

So the architecture should not be "user token → agent → whatever API the agent wants." It should have a boundary where tokens are validated and permissions are narrow:

Agent                          Agent
 ├── customer.read     instead   └── admin.*
 ├── order.read          of
 └── order.cancel

Least privilege, meaning each caller gets only the permissions it needs, matters more than ever once software is choosing which operations to invoke. In ASP.NET Core, that boundary is ordinary policy-based authorization applied per tool:

app.MapPost("/agent-tools/cancel-order", async (
        CancelOrderRequest request,
        IOrderService orders,
        CancellationToken ct) =>
    {
        var result = await orders.CancelAsync(request.OrderId, request.Reason, ct);
        return result.ToHttpResult();
    })
    .RequireAuthorization("order.cancel");  // this tool, this scope, nothing wider

5. Reads and writes are not the same kind of tool

get_customer and delete_customer are both "tools," but their consequences aren't remotely comparable. It helps to separate them deliberately: reads such as get_order or search_products on one side, writes such as refund_payment or delete_customer on the other.

For consequential writes, the agent should confirm with the person before acting: "I found the order and it can still be cancelled. Would you like me to cancel it?" Only after a yes does cancel_order run.

The key distinction is that authorization and confirmation are different things. Having permission to perform an operation doesn't mean the operation should happen without asking, especially when it's irreversible or moves money.

6. Idempotency stops being optional

Agents retry. Networks fail. Tools time out. And an agent often cannot tell whether an operation actually succeeded.

Picture a payment that succeeds on the server, followed by a network timeout on the way back. All the agent sees is a timeout, so it tries again, and now the customer has paid twice.

Idempotency means an operation can be repeated without changing the result beyond the first time. The usual mechanism is an idempotency key: the caller sends a unique value with the request, and the server remembers the outcome for that key.

POST /agent-tools/create-payment
Idempotency-Key: 9c4d7e2a-1f0b-4c88-9a41-2f6e0c3d5b7a

On the server, the first request does the work and stores its result; a repeat of the same key returns the stored result instead of charging again:

public async Task<PaymentResult> CreatePaymentAsync(
    string idempotencyKey, PaymentRequest request, CancellationToken ct)
{
    var existing = await store.FindByKeyAsync(idempotencyKey, ct);
    if (existing is not null)
        return existing.Result;          // already processed: return, don't re-charge

    var result = await payments.ChargeAsync(request, ct);

    await store.SaveAsync(idempotencyKey, result, ct);
    return result;
}

In production you'd also handle two requests arriving with the same key at once, usually by storing the key first with a unique constraint so the second caller waits or gets the first one's result. The point is that the agent doesn't need to understand any of this. The API protects itself against repeated execution.

7. Errors should tell the caller what to do next

400 Bad Request is technically valid and practically useless to an agent. It can't tell whether to fix the arguments, retry later, or stop and ask the user. So it may well do the worst thing: retry the same call in a loop.

An error that carries a decision inside it works much better:

{
  "error": "ORDER_CANNOT_BE_CANCELLED",
  "message": "Order 4821 has already been shipped.",
  "retryable": false,
  "nextAction": "Offer the customer a return instead."
}

Now the caller knows what happened, whether retrying is pointless, and what a sensible next step looks like. This is one of those changes that helps humans just as much, which is a good sign you're improving the API rather than decorating it for AI.

8. Tracing has to cover the whole workflow

Classic request logging tells you that POST /orders returned 200 in 183 ms. That's no longer the interesting unit of work. A single user request might look like this:

user request → model call → search_customers → get_customer
             → search_orders → cancel_order → model call → answer

If that takes 40 seconds, where did the time go? The model, a slow endpoint, a retry, or a loop where the agent kept calling the same tool? You can't tell from per-request logs, which is why distributed tracing, meaning one connected trace that follows a request across every service and call it touches, becomes the thing that saves you.

OpenTelemetry's GenAI semantic conventions (a shared vocabulary for naming these spans) define spans for agent runs, model calls, and tool executions, along with attributes for the model and token usage. They're still evolving, so names may change, but the shape is already useful. In .NET, a tool span is just an Activity:

private static readonly ActivitySource Source = new("MyApp.AgentTools");

using var activity = Source.StartActivity("execute_tool", ActivityKind.Internal);
activity?.SetTag("gen_ai.tool.name", "cancel_order");
activity?.SetTag("app.order.id", request.OrderId);

With that in place, you can follow one user request from the agent's decision, through the tool call, into the API and database, and back out again.

The checklist

If I were designing an agent-facing interface today, this is what I'd hold it to:

Explicit operations. cancel_order, not POST /orders/action.
Described side effects. The agent should know a tool moves money before it calls it.
Narrow permissions. Per-tool scopes, never a shared admin token.
Safe retries. Idempotency keys on every state-changing operation.
Meaningful responses. No decoding of internal status codes.
Actionable errors. Say whether a retry helps, and what to try instead.
End-to-end traces. Cover the agent run, not just the HTTP request.
Human confirmation for anything irreversible or financially significant.

What this means for backend engineers

I don't think agents make REST APIs go away. What's changed is that APIs have a new kind of consumer, one that reads descriptions, picks tools, builds parameters, reacts to errors, retries, and chains operations together.

That pulls API design into AI system design. And the skills it needs are ones backend engineers already have: clear contracts, least privilege, idempotency, useful errors, and good tracing. We were supposed to be doing all of this anyway. The difference is that a human developer could paper over a vague contract by reading the code, and an agent can't.

The endpoint hasn't changed. The caller has.

Have you exposed part of your system to an AI agent yet? I'd be interested to hear what you had to change first. Let me know in the comments. 👇

RAG Is Not a Vector Database Problem. It’s a Data Problem.

Kainat Saricioglu — Wed, 16 Sep 2026 10:00:04 +0000

I built a RAG application, and the more I worked on it, the clearer one thing became: the vector database was not the difficult part.

It's easy to focus on embeddings, similarity search, pgvector, and which LLM to use. But when a RAG application gives a wrong answer, the cause is usually much earlier in the pipeline. The document may have been parsed incorrectly. A chunk may have lost important context. Metadata may be missing. Or we may simply be sending the wrong pieces of information to the model.

That changed how I think about RAG. I now see it less as an "AI problem" and more as a data pipeline problem, and that view makes it much easier to reason about and debug.

Quick definitions: RAG (Retrieval-Augmented Generation) means searching your own documents for relevant passages and giving them to an LLM along with the user's question, so it answers from your data. An embedding is a list of numbers that represents the meaning of a piece of text, so texts with similar meanings get similar numbers. A chunk is a small piece of a document that gets its own embedding.

The RAG pipeline is longer than it looks

RAG is usually explained as a short chain: question, embedding, vector database, LLM, answer. That makes it sound almost trivial. In a real application, the pipeline looks more like this:

flowchart TD
    A[Documents] --> B[Parsing]
    B --> C[Cleaning]
    C --> D[Chunking]
    D --> E[Metadata]
    E --> F[Embedding]
    F --> G[(Vector store)]
    G --> H[Retrieval]
    H --> I[Filtering / Reranking]
    I --> J[Context construction]
    J --> K[LLM]
    K --> L[Answer]

Every arrow is a place where information can be lost. And once information is lost, no vector database can bring it back.

The vector database gets blamed for problems it didn't create

Imagine you upload a 200-page company policy and ask: "What is the maximum amount an employee can claim for business travel?" The app answers: "Employees can claim up to $500."

The actual policy says: "Employees can claim up to $500 per trip, excluding accommodation."

Nothing crashed. The vector search worked, and the LLM worked. The answer even looks plausible. But it's incomplete, and the cause could be as simple as the chunker splitting that sentence in two:

Chunk 12: "...Employees can claim up to $500 per trip,"
Chunk 13: "excluding accommodation. Meal expenses are..."

If chunk 12 is retrieved and chunk 13 isn't, the model never sees the full rule, and it can't answer correctly with information it never received. That leads to the mental model I now use:

RAG quality is limited by the quality of the data that reaches retrieval.

1. Parsing is already a RAG problem

Before you create any embeddings, you have to extract text from the document. With real-world files, and especially PDFs, that's harder than it sounds. A table that looks like this on screen:

Benefit	Limit
Travel	$500
Accommodation	$1,000
Meals	$100

can come out of a PDF text extractor like this:

Benefit Limit Travel Accommodation Meals $500 $1,000 $100

A human instantly sees which amount belongs to which benefit. The extracted text no longer says that. You can embed it with an excellent model, store it in Postgres, and send it to the newest LLM, and you'll still get unreliable answers, because the damage happened before the vector database was involved.

The practical lesson: look at your extracted text before blaming anything later in the pipeline. For table-heavy documents, it's worth converting tables into a form that keeps each row together, such as Travel: $500.

2. Chunking is not just "split every 500 characters"

A document isn't just a stream of characters. It has headings, paragraphs, lists, tables, and sentences that refer back to earlier sentences. Consider this section:

Refund Policy
Customers may request a refund within 30 days.
For enterprise customers, this period is extended to 60 days.

A naive chunker might put each sentence in a separate chunk. Now a user asks: "How long do enterprise customers have to request a refund?" The second chunk is the best match, but on its own it says "this period" without saying which period, and it doesn't even mention refunds.

A common fix is chunk overlap, where neighboring chunks share a few sentences so text cut at a boundary also appears whole in the next chunk. Overlap helps, but it doesn't guarantee that a chunk carries its full meaning. Two further approaches often help more:

Split on the document's structure, meaning headings and paragraphs, rather than a fixed character count.
Add the section heading to each chunk's text before embedding it, so "this period" arrives together with "Refund Policy".

3. More chunks don't automatically mean better retrieval

When retrieval is weak, it's tempting to just fetch more chunks. In RAG this setting is often called top K, meaning "return the K most similar chunks." Raising K sometimes helps, but it can also hurt.

If you retrieve 20 chunks and only 4 are relevant, the model now has to reason through old policy versions, duplicates, and loosely related sections. That costs more tokens, adds latency, and increases the chance of conflicting or outdated information in the answer.

So the goal of retrieval isn't to fetch as much as possible. It's to fetch the smallest set of information that is sufficient to answer the question. A reranker, which is a second model that re-scores the retrieved chunks by how well they actually answer the question, can help you retrieve broadly and then keep only the best few.

4. Metadata can be as important as embeddings

Each chunk has an embedding, but you usually know much more about it: its department, document type, year, region, and version. Throwing that information away is a mistake.

If a user asks about "the 2026 travel policy," you don't want a 2022 policy ranking highly just because its wording is similar. Metadata lets you combine semantic similarity with ordinary filters and business rules. With pgvector, both fit in one SQL query:

SELECT c.content, d.title, c.chunk_index
FROM document_chunks c
JOIN documents d ON d.id = c.document_id
WHERE d.tenant_id = $1              -- only this customer's data
  AND d.document_type = 'Policy'
  AND d.is_current_version = true   -- skip outdated versions
ORDER BY c.embedding <=> $2         -- <=> is cosine distance: smaller = more similar
LIMIT 5;

One pgvector detail is worth knowing here. With an HNSW index (an index that finds approximately nearest vectors quickly), Postgres finds nearby vectors first and applies the WHERE filter afterwards, so a strict filter can return fewer rows than your LIMIT. pgvector 0.8 and later can keep scanning until enough rows match:

SET hnsw.iterative_scan = relaxed_order;

This is the point where RAG starts to feel very familiar to backend engineers. It's not just AI anymore. It's data modeling.

5. Your schema matters

A chunk is not the document. Its embedding is only one property of it: an indexable representation of part of the document. A schema along these lines keeps the rest of the information that retrieval depends on:

CREATE TABLE documents (
    id                  UUID PRIMARY KEY,
    tenant_id           UUID NOT NULL,
    title               TEXT NOT NULL,
    document_type       TEXT NOT NULL,
    version             INT  NOT NULL,
    is_current_version  BOOLEAN NOT NULL,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE document_chunks (
    id           BIGSERIAL PRIMARY KEY,
    document_id  UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    chunk_index  INT  NOT NULL,
    section      TEXT,
    content      TEXT NOT NULL,
    embedding    vector(1024) NOT NULL
);

The original content, the relationships, the version, and the permissions all still matter, and they belong in your schema just as much as the vector does.

6. Access control matters even more in enterprise RAG

Imagine one system holding HR, finance, customer, and engineering documents. A user asks: "What is our salary adjustment policy?" Semantic search may find exactly the right document, but should this user be allowed to see it? That's a separate question, and similarity search doesn't answer it.

Retrieved chunks are application data, and they need the same protection as any other data. Authentication, authorization, tenancy (keeping each customer's data separate), auditing, and data retention all still apply. The safest place to enforce them is inside the retrieval query itself, as in the tenant_id filter above, so unauthorized content never reaches the model at all. AI doesn't remove these backend responsibilities. It makes them more important.

7. Measure retrieval quality instead of guessing

The easiest mistake is testing RAG by hand. You ask one question, the answer looks good, and you conclude that RAG works. That tells you very little.

A better approach is a small evaluation set: a list of questions where you already know which chunk should be found. Even 20 to 30 cases are enough to start. Then you can measure the hit rate, which is the share of questions where the expected chunk appears in the top K results:

public record EvalCase(string Question, long ExpectedChunkId);

public class RetrievalEvaluator(IRetriever retriever)
{
    public async Task<double> HitRateAsync(IReadOnlyList<EvalCase> cases, int topK = 5)
    {
        var hits = 0;

        foreach (var evalCase in cases)
        {
            var results = await retriever.SearchAsync(evalCase.Question, topK);

            if (results.Any(r => r.ChunkId == evalCase.ExpectedChunkId))
                hits++;
        }

        return (double)hits / cases.Count;
    }
}

Now, when you change the chunk size or the embedding model, you get a number instead of a feeling. It also changes the debugging conversation from "the LLM gave a bad answer" to "did retrieval return the right evidence?", which is a much more useful engineering question.

8. Debug RAG from the bottom up

When an answer is wrong, don't start by swapping the LLM. Walk backwards through the pipeline instead:

The source document: was the information actually there?
Extraction: was the text extracted correctly?
Chunking: did the relevant information stay together?
Metadata: were the document, version, and tenant attached correctly?
Embedding: was the chunk embedded at all?
Retrieval: was the right chunk returned?
Context construction: did anything get cut off, removed, or reordered?
Prompt: did we clearly tell the model how to use the context?
The LLM: only now is it time to suspect the model.

This works much better when you can see what the model actually received, so log it on every request:

logger.LogInformation(
    "RAG query {QueryId} retrieved chunks {ChunkIds} with distances {Distances}",
    queryId,
    results.Select(r => r.ChunkId),
    results.Select(r => r.Distance));

Without this, it's easy to keep changing the embedding model, then the chunk size, then the LLM, then the prompt, without ever knowing which change fixed the problem.

The backend engineer's view of RAG

RAG is usually presented as an AI architecture, but much of it is really a data architecture. All the familiar backend concerns are still here: ingestion, validation, data modeling, indexing, caching, authorization, versioning, observability, performance, cost, and testing. What's new is that they now sit next to embeddings, semantic retrieval, context construction, and LLM calls. The interesting engineering happens where those two worlds meet.

So, should you use a vector database?

Yes. Whether it's a dedicated product or pgvector inside Postgres, you need a way to search by meaning. But I wouldn't start a RAG project by asking "Which vector database should I use?" I'd start by asking:

"What does my data look like, and what does a correct retrieval result look like?"

From there, work backwards: which questions the system must answer, what evidence each answer needs, where that evidence lives, how documents should be split, which metadata is required, and only then which storage technology fits. The technology should follow the retrieval requirements, not the other way around.

One final thought

In a RAG system, the LLM is often the easiest part to replace. You can switch models, vector databases, embedding providers, and frameworks. But if the data pipeline is poor, every combination will keep producing poor results.

A powerful model can't use information that was never indexed correctly. A vector database can't recover context that chunking destroyed. And an embedding model can't fix a badly extracted document.

So the next time your RAG application gives a strange answer, don't start with "Which LLM should we use?" Start with:

"What data did the model actually see?"

That question will usually take you much closer to the real problem, because RAG isn't just about giving an LLM access to your data. It's about building a reliable pipeline that gets the right data to the model at the right time. And that's a backend engineering problem.

Have you debugged a RAG system that gave confidently wrong answers? I'd love to hear where the problem turned out to be. Share it in the comments. 👇

Your Database Said "Success." Your Message Broker Said "Try Again."

Kainat Saricioglu — Mon, 17 Aug 2026 17:20:39 +0000

This is where distributed systems get interesting.

If you've worked with backend systems long enough, you've probably written code that looks roughly like this:

await using var transaction = await db.Database.BeginTransactionAsync();

order.Status = OrderStatus.Paid;

db.Orders.Update(order);

await db.SaveChangesAsync();

await messageBus.PublishAsync(
    new PaymentCompleted(order.Id)
);

await transaction.CommitAsync();

At first glance, it looks reasonable.

The order is updated.

The event is published.

The transaction commits.

Done.

Except there is a problem hiding in the middle.

What happens if PublishAsync() fails?

What happens if RabbitMQ is temporarily unavailable?

What happens if the network connection drops after the broker accepted the message but before your application receives the response?

What happens if the application crashes at exactly the wrong millisecond?

And most importantly:

What does your system believe happened?

This is one of those backend problems that doesn't show up in a happy-path demo.

It shows up at 3:17 AM in production.

The problem with "just use a transaction"

Let's simplify the system.

Imagine an Order Service.

It has:

              ┌──────────────┐
              │  Order API   │
              └──────┬───────┘
                     │
                     ▼
              ┌──────────────┐
              │ Order Service│
              └──────┬───────┘
                     │
             ┌───────┴────────┐
             ▼                ▼
      ┌────────────┐   ┌──────────────┐
      │ PostgreSQL │   │   RabbitMQ   │
      └────────────┘   └──────────────┘

A request comes in:

POST /orders/123/pay

The service needs to do two things:

Update the database.
Tell other services that the payment succeeded.

For example:

Order.Status = Paid
        +
PaymentCompleted event

And this is where the trouble starts.

Your database and your message broker are two different systems.

A database transaction can guarantee atomicity inside the database.

RabbitMQ doesn't magically become part of that transaction.

So this is not really:

BEGIN
    UPDATE DATABASE
    PUBLISH MESSAGE
COMMIT

It is more like:

DATABASE                    MESSAGE BROKER

   │                              │
   │ UPDATE                       │
   │─────────────────────────────>│
   │                              │
   │ COMMIT                       │
   │                              │
   │                              │
   │        ??? PUBLISH ???       │
   │                              │

There is a gap.

And that gap is where distributed systems become difficult.

Failure scenario #1: Database first

Let's say we do the sensible-looking thing:

await db.SaveChangesAsync();

await messageBus.PublishAsync(message);

Suppose the database succeeds.

Then:

Database
──────────────
Order 123
Status = Paid

Everything looks good.

But immediately afterward:

RabbitMQ
──────────────
PaymentCompleted
       ❌

Maybe RabbitMQ is temporarily unavailable.

Maybe DNS failed.

Maybe the pod restarted.

Maybe the process crashed.

Maybe there was a network timeout.

Now your database says:

"Payment completed."

But the rest of your system never heard about it.

The Inventory Service doesn't know.

The Notification Service doesn't know.

The Analytics Service doesn't know.

Whatever depends on PaymentCompleted doesn't know.

And if you simply retry the entire HTTP request, you could create another problem.

Failure scenario #2: Message first

So perhaps we reverse the order:

await messageBus.PublishAsync(message);

await db.SaveChangesAsync();

Now imagine the message is successfully published.

Then the database transaction fails.

Maybe there is a deadlock.

Maybe a constraint violation occurs.

Maybe the database connection disappears.

Now we have the opposite situation:

RabbitMQ
──────────────
PaymentCompleted
       ✅

Database
──────────────
Order.Status = Pending
       ❌

The Inventory Service receives:

PaymentCompleted

But the Order Service says:

Actually... no.

Now we're inconsistent in the other direction.

"Can we just use distributed transactions?"

This is where someone usually says:

"Why not use a distributed transaction?"

In theory, we could try to coordinate the database and broker through a distributed transaction protocol.

In practice, this introduces another set of problems.

Distributed transactions can be complex, expensive, operationally awkward, and tightly couple infrastructure components.

The classic transactional outbox pattern exists largely because we want the database update and the intent to publish an event to become atomic without requiring a two-phase commit across the database and broker.

And this is where I think a very simple idea becomes extremely powerful.

The Outbox Pattern

Instead of immediately publishing the message, we save the message inside the same database transaction as the business change.

Something like:

                    ┌─────────────────────────┐
                    │       Transaction       │
                    │                         │
                    │  UPDATE Orders          │
                    │          +              │
                    │  INSERT OutboxMessage   │
                    │                         │
                    └────────────┬────────────┘
                                 │
                              COMMIT
                                 │
                    ┌────────────▼────────────┐
                    │      Database            │
                    │                          │
                    │ Orders                   │
                    │ OutboxMessages           │
                    └────────────┬─────────────┘
                                 │
                                 │
                         Background Worker
                                 │
                                 ▼
                         ┌──────────────┐
                         │   RabbitMQ   │
                         └──────────────┘

Now the important part:

The application isn't trying to atomically update two systems anymore.

It only has to atomically update one: the database.

A simple Outbox table

For example:

CREATE TABLE OutboxMessages
(
    Id              UUID PRIMARY KEY,
    Type            VARCHAR(200) NOT NULL,
    Payload         JSONB NOT NULL,
    OccurredAt      TIMESTAMP NOT NULL,
    ProcessedAt     TIMESTAMP NULL,
    RetryCount      INT NOT NULL DEFAULT 0
);

Now our application transaction becomes:

await using var transaction =
    await db.Database.BeginTransactionAsync();

order.Status = OrderStatus.Paid;

db.Orders.Update(order);

var message = new OutboxMessage
{
    Id = Guid.NewGuid(),
    Type = nameof(PaymentCompleted),
    Payload = JsonSerializer.Serialize(
        new PaymentCompleted(order.Id)
    ),
    OccurredAt = DateTime.UtcNow
};

db.OutboxMessages.Add(message);

await db.SaveChangesAsync();

await transaction.CommitAsync();

Notice something important.

We're not talking to RabbitMQ inside the transaction anymore.

We're just changing the database.

Either both changes succeed:

Orders
   +
OutboxMessages

Or neither does.

That's the part we can make truly atomic.

The transactional outbox pattern specifically works by storing the outgoing message in the same database transaction and having a separate relay publish it to the broker.

But now we have another problem

The outbox worker has to read those messages and publish them.

Something like:

while (!stoppingToken.IsCancellationRequested)
{
    var messages = await db.OutboxMessages
        .Where(x => x.ProcessedAt == null)
        .OrderBy(x => x.OccurredAt)
        .Take(100)
        .ToListAsync();

    foreach (var message in messages)
    {
        await messageBus.PublishAsync(
            message.Type,
            message.Payload
        );

        message.ProcessedAt = DateTime.UtcNow;
    }

    await db.SaveChangesAsync();
}

Looks good.

Until we consider this:

1. Worker reads message
2. Worker publishes message
3. RabbitMQ accepts message
4. Worker crashes
5. Worker never marks message as processed
6. Worker restarts
7. Worker publishes message AGAIN

Congratulations.

We've solved one consistency problem and created another.

Duplicate messages.

And this is one of the most important lessons in distributed systems:

"Exactly once" is usually much harder than it sounds.

The outbox relay itself can publish a message more than once if it crashes after publishing but before recording that it was published. The standard pattern therefore expects consumers to be able to process duplicate messages safely.

Which brings us to my favorite word in distributed systems:

Idempotency

An operation is idempotent when performing it multiple times has the same effect as performing it once.

For example:

Set status = Paid

is naturally easier to make idempotent than:

balance += 100

because:

Set Paid
Set Paid
Set Paid

still results in:

Paid

But:

+100
+100
+100

doesn't.

Make the consumer idempotent

Suppose our PaymentCompleted event reaches the Notification Service.

We could create an inbox/processed-message table:

CREATE TABLE ProcessedMessages
(
    MessageId UUID PRIMARY KEY,
    ProcessedAt TIMESTAMP NOT NULL
);

Then:

await using var transaction =
    await db.Database.BeginTransactionAsync();

var alreadyProcessed =
    await db.ProcessedMessages
        .AnyAsync(x => x.MessageId == message.Id);

if (alreadyProcessed)
{
    return;
}

await notificationService.SendPaymentConfirmationAsync(
    message.OrderId
);

db.ProcessedMessages.Add(
    new ProcessedMessage
    {
        MessageId = message.Id,
        ProcessedAt = DateTime.UtcNow
    });

await db.SaveChangesAsync();

await transaction.CommitAsync();

The MessageId should also be protected by a database-level unique constraint.

Why?

Because this is not enough:

if (!exists)
{
    insert();
}

Two instances can execute the check concurrently:

Instance A                 Instance B

    │                          │
    │── Does it exist? ───────>│
    │                          │
    │<────── No ───────────────│
    │                          │
    │                          │
    │── Does it exist? ───────>│
    │                          │
    │<────── No ───────────────│
    │                          │
    ▼                          ▼
  INSERT                     INSERT

Now both think they're the first.

That's why the database should enforce the invariant.

For example:

ALTER TABLE ProcessedMessages
ADD CONSTRAINT PK_ProcessedMessages
PRIMARY KEY (MessageId);

PostgreSQL's unique constraints are enforced through unique indexes, making the database itself responsible for preventing duplicate keys.

This is a pattern I really like:

Application logic decides what should happen.
The database enforces what must never happen.

But wait... there's another race condition

Let's make the system more realistic.

We have multiple instances:

                    ┌───────────────┐
                    │ Load Balancer │
                    └───────┬───────┘
                            │
                 ┌──────────┼──────────┐
                 ▼          ▼          ▼
             API Pod 1  API Pod 2  API Pod 3
                 │          │          │
                 └──────────┼──────────┘
                            ▼
                        Database

Now imagine all three workers poll the outbox at the same time.

They might all see:

Message #123
ProcessedAt = NULL

What stops them from all publishing it?

This is where things get interesting.

One approach is to atomically claim rows.

For example, conceptually:

SELECT *
FROM OutboxMessages
WHERE ProcessedAt IS NULL
ORDER BY OccurredAt
FOR UPDATE SKIP LOCKED
LIMIT 100;

The exact implementation depends on your database, workload and broker semantics, but the underlying idea is important:

Reading a message and claiming responsibility for it are not necessarily the same operation.

And once you start thinking about multiple instances, retries and crashes, the design becomes much more than:

"Let's add RabbitMQ."

What about retries?

The worker will fail.

That's normal.

A production system should expect failure.

So instead of:

try
{
    await Publish(message);
}
catch
{
    // ¯\_(ツ)_/¯
}

we need an actual retry strategy.

For example:

Attempt 1
   ↓
failure
   ↓
wait 1 second
   ↓
Attempt 2
   ↓
failure
   ↓
wait 5 seconds
   ↓
Attempt 3
   ↓
failure
   ↓
wait 30 seconds
   ↓
Attempt 4
   ↓
failure
   ↓
Dead Letter / Failed State

I generally prefer exponential backoff rather than hammering a dependency that is already unhealthy.

Something like:

var delay = TimeSpan.FromSeconds(
    Math.Pow(2, retryCount)
);

But retrying forever is not a strategy.

Eventually you need to know:

"This message is failing too many times. What do we do with it?"

That's where a dead-letter mechanism, failed state, alerting and operational tooling become important.

A message stuck for 30 seconds might be normal.

A message stuck for 30 minutes might be a problem.

A message stuck for 3 days is probably an incident.

Observability is part of the architecture

This is another thing I think backend developers sometimes underestimate.

Having an outbox is not enough.

You need to be able to answer:

How many messages are waiting?

How old is the oldest message?

How many have failed?

How many retries happened?

Which message is failing?

Which service consumed it?

How long did it take?

Are duplicates increasing?

Is the broker unavailable?

For example, one metric I'd absolutely want is:

outbox_oldest_message_age_seconds

Because:

Queue size = 10

doesn't tell me much.

Those 10 messages might have been created 100 milliseconds ago.

But:

Queue size = 10
Oldest message = 47 minutes old

is a very different story.

There is no "perfect" outbox implementation

And this is probably the most important point.

The outbox pattern isn't magic.

It introduces its own costs.

You now have:

Another table
Another background process
Retry logic
Duplicate handling
Cleanup/retention
Monitoring
Potential ordering problems
Additional database load
Operational complexity

The polling-publisher approach, for example, is straightforward and works with SQL databases, but ordering and efficient publishing become concerns. Transaction-log-based approaches can reduce some polling concerns but introduce database-specific infrastructure and their own duplicate-publishing considerations.

So I don't think the answer should be:

"Always use the Outbox Pattern."

The better question is:

"What consistency guarantee does this particular business operation actually require?"

That's the architectural question.

Payment systems make this especially interesting

Imagine this:

Customer
   │
   ▼
Payment API
   │
   ├──────────────► Payment Provider
   │
   ▼
Database
   │
   ▼
Outbox
   │
   ▼
Message Broker
   │
   ├────────► Order Service
   │
   ├────────► Notification Service
   │
   └────────► Analytics

Now think about retries.

The customer clicks:

Pay

The request times out.

They click again.

The first payment might have succeeded even though the client never received the response.

Now you potentially have:

HTTP retry
     +
Payment retry
     +
Message retry
     +
Consumer retry

Every layer can independently retry.

And suddenly:

Idempotency isn't an optimization.

It's a business requirement.

You might need an idempotency key such as:

Idempotency-Key: 4f1a7c...

and persist that key with the operation.

Now your system can distinguish:

New request

from:

Same request being retried

This is one reason I find payment and identity systems particularly interesting: the cost of "doing the same thing twice" can be much higher than simply returning a duplicate record.

The architecture I would start with

For a typical .NET microservice, I'd be comfortable starting with something like:

                         ┌───────────────┐
                         │     Client    │
                         └───────┬───────┘
                                 │
                                 ▼
                         ┌───────────────┐
                         │ ASP.NET Core  │
                         │      API      │
                         └───────┬───────┘
                                 │
                       ┌─────────▼─────────┐
                       │   DB Transaction  │
                       │                   │
                       │ Business Data     │
                       │        +          │
                       │ Outbox Message    │
                       └─────────┬─────────┘
                                 │
                              COMMIT
                                 │
                                 ▼
                         ┌───────────────┐
                         │ Outbox Worker │
                         └───────┬───────┘
                                 │
                           retry/backoff
                                 │
                                 ▼
                         ┌───────────────┐
                         │   RabbitMQ    │
                         └───────┬───────┘
                                 │
                ┌────────────────┼────────────────┐
                ▼                ▼                ▼
          Order Service    Notification       Analytics
                              Service
                │                │
                └───────┬────────┘
                        ▼
                 Idempotent Consumer

And I'd explicitly design for these failure modes:

Failure	Expected behavior
Database unavailable	Request fails; nothing is committed
Business transaction fails	No event exists
Broker unavailable	Event remains in outbox
Worker crashes before publish	Event is retried
Worker crashes after publish	Duplicate is possible
Consumer receives duplicate	Duplicate is safely ignored
Consumer fails	Message is retried
Message permanently fails	Dead-letter/failed state + alert
Multiple workers process same event	Database constraint protects invariant

That's a much more realistic definition of "reliable" than:

"It works when everything works."

The part I like most about backend engineering

This is why I still find backend development so interesting.

The difficult part usually isn't writing:

await db.SaveChangesAsync();

The difficult part is asking:

What happens if the next line never executes?

And then:

What if it executes but the response is lost?

And:

What if the application crashes immediately afterward?

And:

What if two instances do it simultaneously?

And:

What if the message arrives twice?

And:

What if the database succeeds but the broker doesn't?

And eventually:

Can the system recover without someone manually fixing the data at 3 AM?

That's where architecture stops being a collection of boxes and arrows.

It becomes a set of guarantees.

My current rule of thumb

When designing distributed backend systems, I try to think in terms of failure boundaries, not just components.

Instead of asking:

"Should we use RabbitMQ?"

I'd ask:

"What happens if RabbitMQ disappears for 10 minutes?"

Instead of:

"Should this be a microservice?"

I'd ask:

"What happens when this service is unavailable?"

Instead of:

"Should we add retries?"

I'd ask:

"What happens when the operation succeeds but the response is lost and we retry it?"

Instead of:

"Can we process this asynchronously?"

I'd ask:

"What consistency guarantee does the business actually need?"

Those questions usually lead to much better architecture decisions.

One final thought

I think one of the biggest misconceptions about backend engineering is that reliability comes from adding more infrastructure.

More services.

More queues.

More replicas.

More caching.

More Kubernetes.

More distributed components.

Sometimes it does.

But sometimes the best reliability improvement is much simpler:

Make the important invariant explicit.

Then decide where that invariant should be enforced.

Sometimes that's the application.

Sometimes that's the database.

Sometimes it's both.

And sometimes the correct answer is to avoid distributing the operation in the first place.

The more distributed our systems become, the more valuable these fundamentals become.

Because eventually every distributed system has the same question waiting for us:

"What happens when things fail between step A and step B?"

That's the question I'm increasingly interested in.

And I suspect it's one of the questions that separates code that works from software that can actually survive production.

What would you choose?

Suppose you have:

Database update
      +
Event publication

and you cannot use a distributed transaction.

Would you choose:

A) Transactional Outbox
B) Direct publish + retry
C) Event sourcing
D) Something else

And more importantly:

What failure scenario would drive your decision?

I'd genuinely like to hear how other backend engineers approach this.

I'm planning to write more about the practical side of backend engineering—especially .NET, distributed systems, microservices, databases, identity/security, and the kinds of problems that only become obvious when software meets production.

#BackendDevelopment #DotNet #CSharp #Microservices #DistributedSystems #SoftwareArchitecture #RabbitMQ #PostgreSQL #Database #SystemDesign