My 3-layer agent session locking fix stopped it from quietly wrecking customer records

I thought one lock would fix my agent bugs.

It didn’t.

What finally stopped my workflow from quietly corrupting customer records was a 3-layer defense:

1. serialize work per customer, ticket, or document
2. attach idempotency keys to every side-effecting POST
3. persist state in order with version checks

I learned this the ugly way at 2:13 a.m. when an n8n workflow reopened a closed Zendesk ticket, reassigned the account owner in HubSpot, and then did both again when the webhook retried.

By the time I checked PostgreSQL, the audit trail made it look like the customer had requested three different changes in under a minute.

The stack looked pretty normal:

• n8n orchestration
• Redis feeding multiple workers
• PostgreSQL as the durable store
• an OpenAI-compatible SDK talking to a pooled model backend
• tools that could mutate HubSpot and Zendesk

My original fix was a per-customer lock.

That lock-only design was not just incomplete. For production agents touching CRMs, tickets, billing systems, or internal records, it was reckless.

The short version

If your agent can write to HubSpot, Zendesk, Salesforce, Stripe, or your own internal APIs, a lock alone is not enough.

You need all three:

Layer	What it prevents
Per-entity lock	Two workers mutating the same record at once
Idempotency key	Retries creating duplicate external side effects
Ordered persistence	Older state snapshots overwriting newer ones

Miss one of these and you can still corrupt data while all your dashboards stay green.

Failure mode #1: the lock did nothing against retries

I used a PostgreSQL advisory lock keyed by customer ID.

That serialized work inside my app, which felt safe.

But the dangerous part wasn’t lock acquisition.
The dangerous part was the outbound POST.

Here’s what happened:

• worker A acquired the advisory lock for customer 4812
• the agent called HubSpot to update owner and lifecycle stage
• HubSpot processed the request
• the HTTP response timed out or got dropped
• n8n marked the step as failed
• the workflow retried
• the same mutation got sent again

The lock worked exactly as designed.

It prevented overlap inside my system.
It did absolutely nothing to stop duplicate side effects outside my system.

That’s the trap.

If your agent can POST to a write API without idempotency, you are gambling with customer data.

What I changed

I started generating an idempotency key at the workflow boundary in n8n, then carrying it through every mutating tool call.

For APIs with native idempotency support, use it.
For APIs without it, build your own dedupe layer.

Example: generate an idempotency key once per workflow operation

import crypto from "node:crypto";

export function buildIdempotencyKey({
  workflowId,
  customerId,
  operation,
  intentPayload,
}) {
  const intentHash = crypto
    .createHash("sha256")
    .update(JSON.stringify(intentPayload))
    .digest("hex")
    .slice(0, 16);

  return `${workflowId}:${customerId}:${operation}:${intentHash}`;
}

Example: store it before sending the external request

create table if not exists mutation_dedupe (
  idempotency_key text primary key,
  customer_id bigint not null,
  operation text not null,
  status text not null,
  response_json jsonb,
  created_at timestamptz not null default now()
);

async function reserveMutation(db, key, customerId, operation) {
  const result = await db.query(
    `insert into mutation_dedupe (idempotency_key, customer_id, operation, status)
     values ($1, $2, $3, 'pending')
     on conflict (idempotency_key) do nothing`,
    [key, customerId, operation]
  );

  return result.rowCount === 1;
}

If reserveMutation() returns false, I know this exact mutation was already attempted and I can skip or replay from stored state.

Idempotency belongs above the model layer

My first mistake was treating idempotency like a tool wrapper problem.

That was too low in the stack.

The right place is the orchestration layer:

• n8n
• Temporal
• Make
• Zapier
• custom job runners

Why?

Because the model can decide to call the same tool twice with slightly different phrasing.
Or call two different tools that mutate the same downstream record.

If idempotency only exists inside a helper around fetch(), you’ve already lost the bigger picture.

In my setup:

• PostgreSQL held durable truth
• Redis handled queues and short-lived coordination
• the OpenAI-compatible client was just the model interface

That split matters.

Redis is great for ephemeral coordination.
It is not where I want the source of truth for whether a customer mutation already happened.

Failure mode #2: parallel tool calls created conflicting writes

Fixing retries exposed the next bug.

Nothing failed.
That’s what made it nastier.

The model decided to update HubSpot and reopen a Zendesk ticket in the same turn.
Both actions looked valid in isolation.
Together, they stomped on each other because they were based on slightly different snapshots of the same customer state.

One branch saw:

{ "owner": "enterprise_queue" }

The other branch saw:

{ "owner": "account_executive" }

Both wrote back confidently.

This is where people get fooled by throughput.
Parallel tool use looks great in demos and can be fine for read-heavy workflows.
For mutation-heavy workflows, it’s often a footgun.

If your agent is reading and writing the same customer, ticket, or document across multiple tools, slower serialized execution is usually better than fast conflicting writes.

What I changed

I split tools into two classes:

• read-only
• mutating

Then I allowed concurrency only for read-only calls.
Mutating calls had to run one at a time per business entity.

Example: simple tool policy

type ToolPolicy = {
  name: string;
  mode: "read" | "write";
  entityScope: "customer" | "ticket" | "document";
};

const toolPolicies: ToolPolicy[] = [
  { name: "getHubSpotContact", mode: "read", entityScope: "customer" },
  { name: "searchZendeskTickets", mode: "read", entityScope: "customer" },
  { name: "updateHubSpotOwner", mode: "write", entityScope: "customer" },
  { name: "reopenZendeskTicket", mode: "write", entityScope: "ticket" },
];

Example: serialize writes per entity

async function runToolCalls(toolCalls, runReadOnly, runSerializedWrite) {
  const readCalls = toolCalls.filter(t => t.mode === "read");
  const writeCalls = toolCalls.filter(t => t.mode === "write");

  const readResults = await Promise.all(readCalls.map(runReadOnly));

  const writeResults = [];
  for (const call of writeCalls) {
    writeResults.push(await runSerializedWrite(call));
  }

  return [...readResults, ...writeResults];
}

The lock still mattered here, but in a different way than I originally thought.

• the lock protected the entity across workers
• the no-parallel-mutations rule protected the entity inside a single run

You need both.

Failure mode #3: state was persisted in the wrong order

Once retries and parallel writes were under control, I still had corruption.

This one was subtle.

The external systems were correct.
My internal state was not.

The problem was that I was saving agent state after side effects in whatever order responses came back.

That meant this could happen:

1. tool call 1 updated HubSpot
2. tool call 2 updated Zendesk
3. state snapshot B got written to PostgreSQL
4. delayed callback from tool call 1 wrote older snapshot A afterward

So HubSpot and Zendesk were right, but my local record regressed.

Then the next workflow run loaded stale state and “fixed” the customer back into the wrong configuration.

This is the bug people miss because every API call returned 200 OK.

What I changed

I added a monotonic version per customer record and rejected stale writes.

Example schema

create table if not exists customer_state (
  customer_id bigint primary key,
  version bigint not null,
  state_json jsonb not null,
  updated_at timestamptz not null default now()
);

Example optimistic write

update customer_state
set version = $3,
    state_json = $4,
    updated_at = now()
where customer_id = $1
  and version = $2;

Where:

• $2 = expected current version
• $3 = next version

If the update count is 0, the worker is stale and has to rehydrate from the latest state.

Example worker logic

async function persistNextState(db, customerId, expectedVersion, nextState) {
  const nextVersion = expectedVersion + 1;

  const result = await db.query(
    `update customer_state
     set version = $3,
         state_json = $4,
         updated_at = now()
     where customer_id = $1
       and version = $2`,
    [customerId, expectedVersion, nextVersion, nextState]
  );

  if (result.rowCount === 0) {
    throw new Error("stale_state_write");
  }

  return nextVersion;
}

Boring fix. Completely necessary.

The architecture that finally stopped the damage

After a few painful iterations, this is the minimum setup I trust for always-on agents:

• PostgreSQL advisory locks to serialize per-customer mutations across workers
• idempotency keys created at the n8n workflow boundary
• idempotency attached to every side-effecting POST
• Redis used for queueing and ephemeral coordination, not correctness
• serialized mutating tool calls, even when the model wants concurrency
• ordered state persistence with version checks in PostgreSQL

Example: PostgreSQL advisory lock per customer

select pg_advisory_xact_lock($1);

And in Node:

await db.query("begin");
await db.query("select pg_advisory_xact_lock($1)", [customerId]);

// read current state
// decide next mutation
// reserve idempotency key
// perform external call
// persist next state version

await db.query("commit");

That’s the pattern.
Not “just add a lock.”

Why this gets worse as agents scale

These bugs get much worse the moment you stop treating LLM usage like a toy workflow.

The failure modes multiply when you have:

• multiple n8n workers
• webhook retries
• Zapier or Make triggers firing at the same time
• OpenAI-compatible clients fanning requests across different models
• customer records getting touched 24/7

That’s the real transition point.

Once compute is cheap and predictable enough to keep agents always on, reliability bugs become the real bill.

That’s one reason I care a lot about infrastructure choices for agent systems.
If you’re still paying per token, teams tend to under-run automations and avoid stress-testing them.
When compute is flat-rate and predictable, you can actually run the volume needed to find the ugly race conditions before your customers do.

That’s also why services like Standard Compute are interesting for this class of workload: if you’re building agents on top of an OpenAI-compatible API and want them running continuously, predictable cost matters a lot more than people admit. The moment agents become infrastructure instead of demos, surprise billing and token micromanagement get in the way of testing, retries, and 24/7 execution.

Practical checklist

If your agent mutates business data, this is the checklist I’d use:

• lock per business entity, not just per workflow run
• generate idempotency keys at the orchestration layer
• persist idempotency records before external writes
• treat Redis as coordination, not truth
• disable parallel mutations against the same entity
• version your durable state
• reject stale writes
• rehydrate on version mismatch
• log mutation intent separately from model output
• test with retries, dropped responses, and duplicate webhooks

Quick chaos test ideas

You do not need a giant platform team to test this.

A few ugly scripts go a long way.

Replay the same webhook twice

curl -X POST http://localhost:5678/webhook/customer-update \
  -H 'Content-Type: application/json' \
  -d '{"customerId":4812,"action":"reassign_owner"}'

curl -X POST http://localhost:5678/webhook/customer-update \
  -H 'Content-Type: application/json' \
  -d '{"customerId":4812,"action":"reassign_owner"}'

Simulate concurrent workers

seq 1 10 | xargs -n1 -P10 -I{} curl -X POST http://localhost:5678/webhook/customer-update \
  -H 'Content-Type: application/json' \
  -d '{"customerId":4812,"action":"reassign_owner"}'

Inject timeout after downstream success

If you control the tool wrapper, add a flag that sleeps or throws after the external API returns success but before your app records completion.

That’s the exact gap where duplicate writes are born.

My actual takeaway

My original mistake was thinking session locking was the whole pattern.

It isn’t.
It’s one layer.

If your agents touch CRMs, tickets, billing systems, or internal records, the minimum sane setup is:

• lock
• idempotency
• ordered persistence

Anything less is how you end up quietly corrupting customer data while your dashboards still look healthy.

If you’re building always-on agents in n8n, Make, Zapier, or custom workers, I’d start there before chasing fancier model behavior.

The expensive bug usually isn’t the prompt.
It’s the duplicate write you didn’t notice until a customer asks why their account changed three times overnight.