DEV Community: Auto Respond

Two clocks in lead response automation: transport latency and safe send timing

Auto Respond — Sun, 16 Aug 2026 02:07:26 +0000

A lead-response worker can be fast and still send at the wrong time.

That sounds contradictory until you split time into two clocks:

The transport clock measures how long an event spends moving through ingestion, processing, and delivery.
The policy clock decides when a reply is allowed to leave.

Many systems measure the first and bury the second inside a delay or cron job. That makes incidents hard to explain. A message can leave the queue in 80 ms and still be early, late, duplicated, or unsafe.

This article sketches an architecture that treats both clocks as first-class data. The examples use inbound Yelp and Thumbtack workflows, but the model applies to any channel with webhooks, retries, account rules, and an outbound API.

Disclosure: this post comes from the Auto-Respond team. Auto-Respond supports lead-response workflows for Yelp and Thumbtack, among other channels. AI assisted with editing this draft, and the team reviewed the technical content and product statements before publication.

Clock one: transport latency

The transport clock answers an engineering question:

How long did the system take to turn a received event into an acknowledged outbound request?

A useful trace has more than one timestamp:

type TransportTrace = {
  sourceEventAt: string;        // timestamp supplied by the channel
  ingressAcceptedAt: string;    // our edge accepted the webhook
  jobClaimedAt: string;         // a worker claimed the job
  decisionReadyAt: string;      // reply or action was ready
  sendStartedAt: string;
  providerAcknowledgedAt?: string;
  visibleAt?: string;           // only when the channel exposes it
};

These timestamps let you separate source delivery, queue wait, decision time, outbound request time, and acknowledgment. Do not call the last one "visible reply latency" unless the channel exposes a read-back signal that proves the reply appeared in the conversation.

Use UTC wall-clock timestamps for correlation across services. Use a monotonic clock for durations inside one process. Wall clocks can jump during time synchronization, while a monotonic timer only moves forward.

Clock two: safe send timing

The policy clock answers an operational question:

Is this message allowed to leave now?

Represent that decision explicitly:

type SendPolicy = {
  notBefore: string;
  expiresAt?: string;
  reason:
    | "immediate-first-response"
    | "business-rule-delay"
    | "rate-limit-backoff"
    | "manual-review"
    | "escalation-window";
  policyVersion: string;
};

The worker should never infer notBefore from "job age" or from when it happened to claim the job. Store the boundary when the policy decision is made. If a later rule change matters, create a new policy decision with a new version and keep the old one in the audit trail.

This separation matters for channel-specific behavior. Yelp and Thumbtack can feed the same normalized lead model while still having different reply rules, account settings, API behavior, or retry semantics. Sharing a queue does not mean sharing every send policy.

An acknowledgment is not permission

A provider acknowledgment proves that an API accepted a request. It does not prove three other things:

that the customer can see the reply;
that the reply was allowed by the latest business rule;
that another worker did not already send the same logical message.

Keep the states separate:

received
  -> eligible
  -> scheduled
  -> send_started
  -> provider_acknowledged
  -> visible_confirmed (when available)

A terminal failure, cancellation, or escalation can branch from any state where sending remains possible. Avoid a generic done state. It hides whether the system actually sent, merely received an acknowledgment, or handed the conversation to a person.

Claim only when both clocks agree

A simple worker loop can enforce the boundary:

async function claimSendable(now: Date) {
  return db.messageJobs.findOneAndUpdate(
    {
      state: "scheduled",
      notBefore: { $lte: now },
      $or: [
        { expiresAt: null },
        { expiresAt: { $gt: now } }
      ]
    },
    {
      $set: {
        state: "send_started",
        claimedAt: now
      },
      $inc: { attempt: 1 }
    },
    {
      sort: { notBefore: 1 },
      returnDocument: "after"
    }
  );
}

The database transition must be atomic. Reading a ready job and updating it in two separate operations leaves room for two workers to claim it.

Also re-check current conversation state after the claim. A lead may have replied while the job waited. The stored schedule says when sending became possible, not that sending is still correct.

Idempotency protects the boundary

Webhooks retry. Queue workers crash. Outbound requests time out after the channel has accepted them. Those are ordinary conditions, so the design needs one stable idempotency key per logical send.

A practical key can combine:

source account + conversation + source event + message purpose + policy version

Store the key before calling the outbound provider. Then use a unique constraint so concurrent workers cannot create two send records.

A timeout after the request is the awkward case. Do not immediately issue a second send. First use the channel's supported read-back or request-status mechanism, when available. If the outcome cannot be resolved automatically, mark it delivery_unknown and route it to review. Guessing is how a fast retry becomes a duplicate customer message.

Delays need reasons, not magic numbers

A value like delayMs: 300000 is difficult to operate. Five minutes for what reason? Which rule produced it? Does a later inbound message cancel it? Can a human override it?

Persist the reason and the evaluated inputs:

{
  "not_before": "2026-08-16T08:35:00.000Z",
  "reason": "business-rule-delay",
  "policy_version": "yelp-v12",
  "evaluated_at": "2026-08-16T08:30:00.000Z",
  "cancel_on_inbound": true
}

Do not put customer names, phone numbers, full messages, or addresses into timing logs. A source event identifier can be hashed if operators only need correlation.

Escalation has its own deadline

Some conversations should leave automation instead of waiting for another send attempt. Model escalation as a deadline, not as a note attached to a failed job.

Useful fields include:

escalateAt;
escalationReason;
automationPausedAt;
assignedQueue;
resolvedAt.

Then define the ordering rule. For example, if escalateAt is due, pause the automated send before notifying the review queue. That ordering prevents a message from leaving during the handoff.

The same rule should be idempotent. Replaying an escalation event must not create several assignments or several notifications.

Metrics that expose the real failure

Track the two clocks independently.

For transport:

source-to-ingress delay;
queue wait;
decision duration;
outbound request duration;
acknowledgment rate;
unknown-delivery rate.

For policy timing:

sends before notBefore (the target is zero);
jobs sent after expiresAt;
policy cancellations honored;
duplicate logical sends;
escalations completed before their deadline;
time spent waiting for review.

A single "response time" chart cannot tell you whether the system was slow or correctly waiting. It also cannot show an early send. Pair latency percentiles with policy-violation counters and report sample counts.

Test both clocks with a fake clock

Timing code becomes easier to test when business logic receives a clock instead of calling new Date() everywhere.

interface Clock {
  now(): Date;
}

function isSendable(job: Job, clock: Clock) {
  const now = clock.now();

  if (job.state !== "scheduled") return false;
  if (now < job.notBefore) return false;
  if (job.expiresAt && now >= job.expiresAt) return false;

  return true;
}

Table-driven tests should cover the exact boundary, one millisecond before it, expiry, cancellation after a new inbound event, duplicate claims, an ambiguous provider timeout, and escalation winning a race with send.

The tests should also replay the same webhook and the same queue job. "Exactly once" is usually an application guarantee built from idempotent effects, not a promise made by the transport.

The design rule

Transport latency and safe send timing describe different truths. One says how fast the machinery moved. The other says whether it was correct to move at all.

Keep both clocks in the event model, version the policy decision, make claims atomic, and treat provider acknowledgment as its own state. That gives operators enough evidence to explain a late reply without hiding a policy wait, and enough control to stop an early or duplicate send.

For a channel-specific view of the conversation boundary, the Yelp auto responder guide describes the supported workflow without treating every channel as identical.

Designing an append-only human takeover audit log for AI voice workflows

Auto Respond — Sun, 09 Aug 2026 22:19:36 +0000

I work with the Auto-Respond Team. I used AI assistance while editing this article, then checked the code and technical claims before publishing. The design below is a tested pattern, not a claim about a released product.

A voice workflow can switch from an automated agent to a human operator in a few hundred milliseconds. Later, somebody will ask a deceptively simple question: what happened?

A normal application log rarely gives a reliable answer. One service says the takeover was requested. Another says a worker claimed it. A third overwrote the session row after reading an old version. If the only surviving record is the final state, the path that produced it is gone.

The fix I keep coming back to is an append-only event stream. It keeps the decision trail without keeping the conversation, rejects stale writers, and rebuilds cleanly after a crash.

Write down the invariants first

The storage design should make these statements true:

Every event has one stable identity.
Sequence numbers increase by exactly one within a voice session.
An actor is a company role or service role, never a person's name.
A decision reason cannot be edited after it is written.
Audio, transcripts, phone numbers, and free-form operator notes stay outside the audit stream.
Replaying the same ordered events produces the same state.

Test that last property. If replay changes with the clock, a network call, or whatever order a loose query returns, the log cannot be trusted as state.

Use a narrow event envelope

Here is a TypeScript shape for the append boundary:

type Actor =
  | { kind: "service"; ref: "voice-router" | "policy-engine" | "dialer" }
  | { kind: "company_role"; ref: "support-agent" | "shift-supervisor" }
  | { kind: "system"; ref: "recovery-worker" };

type TakeoverEventType =
  | "session.started"
  | "takeover.requested"
  | "takeover.claimed"
  | "takeover.released"
  | "automation.resumed"
  | "session.ended"
  | "content.redacted";

type DecisionReason =
  | "caller_requested_human"
  | "low_model_confidence"
  | "policy_requires_human"
  | "operator_claimed"
  | "operator_unavailable"
  | "connection_lost"
  | "session_completed";

type AuditEvent = {
  eventId: string;
  sessionId: string;
  seq: number;
  type: TakeoverEventType;
  occurredAt: string;
  recordedAt: string;
  actor: Actor;
  reason: DecisionReason;
  policyVersion: string;
  correlationId: string;
  contentRef?: string;
  contentDigest?: string;
  attributes: Record<string, string | number | boolean>;
};

The actor field deliberately avoids email addresses, display names, and employee IDs. A separate access-controlled system may map an internal operator assignment to a person when there is a legitimate need. The general audit feed only needs to prove which company role or service made the transition.

Keep reason as a controlled value. Free-form text is tempting, but it mixes sensitive content with operational evidence and makes analysis inconsistent. If a new reason appears, add it to the contract and version the policy.

Enforce append-only storage in SQL

The table needs more than a convention that says "please do not update rows."

CREATE TABLE voice_takeover_events (
  session_id UUID NOT NULL,
  seq BIGINT NOT NULL CHECK (seq > 0),
  event_id UUID NOT NULL,
  event_type TEXT NOT NULL,
  occurred_at TIMESTAMPTZ NOT NULL,
  recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  actor_kind TEXT NOT NULL,
  actor_ref TEXT NOT NULL,
  reason_code TEXT NOT NULL,
  policy_version TEXT NOT NULL,
  correlation_id UUID NOT NULL,
  content_ref TEXT,
  content_digest TEXT,
  attributes JSONB NOT NULL DEFAULT '{}',
  PRIMARY KEY (session_id, seq),
  UNIQUE (event_id)
);

CREATE TABLE voice_takeover_heads (
  session_id UUID PRIMARY KEY,
  last_seq BIGINT NOT NULL,
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

REVOKE UPDATE, DELETE ON voice_takeover_events FROM voice_app;
GRANT SELECT, INSERT ON voice_takeover_events TO voice_app;

The composite primary key prevents two events from occupying the same sequence. The unique event ID makes a retried append recognizable. Database permissions stop the application role from rewriting history.

For stricter deployments, add a database trigger that rejects UPDATE and DELETE regardless of the caller. Retention can still be handled through partition expiry under a separate administrative role. Append-only does not mean keep every event forever.

Guard the sequence in one transaction

The head row is a compare-and-set boundary. A writer must provide the sequence it believes is current.

type AppendInput = Omit<AuditEvent, "seq" | "recordedAt"> & {
  expectedSeq: number;
};

async function appendTakeoverEvent(input: AppendInput): Promise<AuditEvent> {
  return db.transaction(async (tx) => {
    const previous = await tx.events.findByEventId(input.eventId);
    if (previous) return previous;

    const head = await tx.heads.lockOrCreate(input.sessionId);

    if (head.lastSeq !== input.expectedSeq) {
      throw new Error(
        "STALE_SESSION_VERSION expected=" +
          input.expectedSeq +
          " actual=" +
          head.lastSeq,
      );
    }

    const event: AuditEvent = {
      ...input,
      seq: head.lastSeq + 1,
      recordedAt: new Date().toISOString(),
    };

    await tx.events.insert(event);
    await tx.heads.advance(input.sessionId, head.lastSeq, event.seq);

    return event;
  });
}

The lockOrCreate function should take a row lock. The advance function should update only when last_seq still equals the value just read. The event insert and head update belong in the same transaction.

This closes two awkward failure paths. Concurrent takeover claims cannot both win. A retry with the same eventId gets back the event already written. A retry with a new ID but an old expected sequence fails visibly instead of inventing a second history.

Keep decision evidence immutable

Suppose the policy engine requests a human because confidence falls below a threshold. Store the reason code, policy version, and non-sensitive values used by the decision:

{
  "reason": "low_model_confidence",
  "policyVersion": "voice-handoff-2026-08",
  "attributes": {
    "confidenceBucket": "below_0_55",
    "attempt": 1
  }
}

Do not later replace the reason with "caller_requested_human" because it looks cleaner in a report. If new evidence changes the interpretation, append a new reviewed annotation event. History should show that an interpretation changed.

I also avoid storing raw model prompts and chain-of-thought text. The audit question is about inputs, policy, transition, and actor. Internal reasoning text is neither required nor a safe substitute for structured decision evidence.

Draw a hard redaction boundary

The event stream should contain references, not conversation content.

A contentRef can point to an encrypted object governed by a shorter retention policy. A contentDigest can prove which object was evaluated without exposing it. The digest is only useful if the input space is not guessable, so compute it with a keyed hash or include a secret salt managed outside the event table.

Never put these values in attributes:

caller phone numbers or email addresses;
transcript fragments;
raw audio locations;
operator free-form notes;
provider tokens or webhook payloads.

If a deletion request covers the referenced content, delete or redact that object according to policy and append a content.redacted event. The operational history survives, while the sensitive material does not.

Rebuild state with a pure reducer

The replay function should have no database or network access:

type SessionState = {
  mode: "automated" | "human" | "ended";
  ownerRole?: string;
  lastSeq: number;
  redactedRefs: Set<string>;
};

function reduceEvent(state: SessionState, event: AuditEvent): SessionState {
  if (event.seq !== state.lastSeq + 1) {
    throw new Error("SEQUENCE_GAP");
  }

  switch (event.type) {
    case "session.started":
      return { ...state, mode: "automated", lastSeq: event.seq };

    case "takeover.claimed":
      return {
        ...state,
        mode: "human",
        ownerRole: event.actor.ref,
        lastSeq: event.seq,
      };

    case "takeover.released":
    case "automation.resumed":
      return {
        ...state,
        mode: "automated",
        ownerRole: undefined,
        lastSeq: event.seq,
      };

    case "session.ended":
      return { ...state, mode: "ended", lastSeq: event.seq };

    case "content.redacted": {
      const refs = new Set(state.redactedRefs);
      if (event.contentRef) refs.add(event.contentRef);
      return { ...state, redactedRefs: refs, lastSeq: event.seq };
    }

    default:
      return { ...state, lastSeq: event.seq };
  }
}

Sort by seq, not timestamps. Two services can have clock skew, and recordedAt may be later for a retried delivery. Sequence defines the session order. Timestamps remain useful evidence, but they do not decide replay.

Test the ugly races

A useful integration test pauses two operators after they both read the same head. Release them together and assert that only one claim is appended. The loser should receive STALE_SESSION_VERSION, reload the state, and stop.

Also test these cases:

the same event ID arrives five times;
an insert succeeds but the transaction rolls back before the head advances;
sequence 8 is missing during replay;
content is redacted after the session ends;
an operator role releases control while a recovery worker attempts to resume automation.

The pass condition is stronger than "the final row looks right." The event list must have no gaps, no duplicate IDs, one winning claim, and the same reconstructed state on every replay.

The same boundary appears in AI answering service workflows where automation and a human may share one conversation. If the handoff matters, a mutable flag is the wrong artifact. Record the transition.

Idempotent lead ingestion: deduplicating webhook retries without losing follow-ups

Auto Respond — Sat, 08 Aug 2026 19:24:20 +0000

I work with the Auto-Respond Team. I used AI assistance to edit this article, then checked the code and technical claims before publishing. The examples use fake adapters and fake identifiers. No customer data is involved.

Webhook providers retry. Your load balancer retries. A worker can crash after it writes to the database but before it acknowledges the message. If ingestion treats each delivery as a new lead, one real inquiry can produce two conversations, two timers, and a very confused customer.

The tempting fix is a seen flag. That works until the flag is written in a different transaction from the jobs it is meant to protect.

A safer design gives every provider event a stable identity, records that identity atomically with its downstream work, and lets each worker retry a bounded number of times. This article walks through that design in TypeScript.

Start with the delivery contract

An inbound delivery is not the same thing as a lead.

One lead may produce a creation event, an update, and a reply. Each event may arrive more than once. Deduplicating only on leadId would throw away legitimate updates. Deduplicating on a request timestamp would do the opposite: every retry would look new.

I model the boundary like this:

type ProviderEvent = {
  provider: "yelp" | "thumbtack" | "other";
  accountId: string;
  eventId?: string;
  leadId: string;
  kind: "lead.created" | "lead.updated" | "lead.replied";
  occurredAt: string;
  payload: unknown;
};

Use the provider's immutable event ID when it exists. Scope it by provider and connected account, because IDs from different tenants may overlap.

When a provider does not supply an event ID, build a fallback from documented stable fields. Do not include a delivery timestamp, request ID, or retry counter. Those values change on redelivery.

import { createHash } from "node:crypto";

export function idempotencyKey(event: ProviderEvent): string {
  const identity = event.eventId
    ? ["provider-event", event.eventId]
    : ["lead-event", event.leadId, event.kind];

  const scoped = [
    event.provider,
    event.accountId,
    ...identity,
  ];

  return createHash("sha256")
    .update(JSON.stringify(scoped))
    .digest("hex");
}

This function is deterministic, but that alone does not make the system safe. The database still needs to enforce uniqueness.

A fallback key is only as good as the provider contract behind it. If lead.updated can happen several times with the same leadId, add a stable version or change identifier. If the provider exposes neither, ask for a contract change or accept a short dedupe window with an explicit collision risk. Do not quietly pretend the ambiguity is solved.

Let the database win the race

Two application instances can receive the same retry at the same time. A "select, then insert" sequence has a race between those statements.

Put the invariant in the database:

CREATE TABLE webhook_receipts (
  id UUID PRIMARY KEY,
  provider TEXT NOT NULL,
  account_id TEXT NOT NULL,
  idempotency_key TEXT NOT NULL,
  event_kind TEXT NOT NULL,
  external_lead_id TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  payload_digest TEXT NOT NULL,
  UNIQUE (provider, account_id, idempotency_key)
);

CREATE TABLE delivery_jobs (
  id UUID PRIMARY KEY,
  receipt_id UUID NOT NULL REFERENCES webhook_receipts(id),
  job_key TEXT NOT NULL UNIQUE,
  job_type TEXT NOT NULL,
  status TEXT NOT NULL,
  attempt_count INTEGER NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMPTZ NOT NULL,
  last_error_code TEXT
);

The unique constraint decides which request created the receipt. Every concurrent retry gets the existing row.

The receipt and its jobs must be created in one transaction. Otherwise a process can write "seen" and crash before it schedules the work. The next delivery then looks like a duplicate and the lead is never processed.

type IngestResult =
  | { state: "created"; receiptId: string }
  | { state: "duplicate"; receiptId: string };

async function ingest(rawBody: Buffer, signature: string): Promise<IngestResult> {
  verifyProviderSignature(rawBody, signature);

  const event = decodeProviderEvent(rawBody);
  const key = idempotencyKey(event);
  const digest = createHash("sha256").update(rawBody).digest("hex");

  return db.transaction(async (tx) => {
    const receipt = await tx.receipts.insertOrGet({
      provider: event.provider,
      accountId: event.accountId,
      idempotencyKey: key,
      eventKind: event.kind,
      externalLeadId: event.leadId,
      payloadDigest: digest,
    });

    if (!receipt.created) {
      return { state: "duplicate", receiptId: receipt.id };
    }

    await tx.jobs.insert({
      receiptId: receipt.id,
      jobKey: `${receipt.id}:initial-processing:v1`,
      jobType: "initial-processing",
      status: "queued",
      nextAttemptAt: new Date(),
    });

    return { state: "created", receiptId: receipt.id };
  });
}

insertOrGet needs to be an atomic upsert backed by the unique constraint. It is not shorthand for a read followed by a write.

Validate the provider signature against the raw request bytes before trusting fields from the payload. If persistence fails, return a retryable server error. Return success only after the transaction commits. A duplicate whose original transaction committed can receive the same success response as the first delivery.

This pattern shows up in marketplace lead systems, including the ingestion side of a Yelp auto responder. The important part is not the channel name. It is treating retries as a normal delivery state instead of an exceptional one.

Make jobs idempotent too

The webhook receipt protects ingestion. It does not protect a worker that times out after calling another service.

Give each logical side effect its own key. A job retry may run several times, but it should not create several logical operations.

For example:

type JobIdentity = {
  receiptId: string;
  operation: "initial-response" | "crm-upsert" | "audit-export";
  version: number;
};

function jobKey(value: JobIdentity): string {
  return [
    value.receiptId,
    value.operation,
    `v${value.version}`,
  ].join(":");
}

Pass that key to downstream APIs that support idempotency. For systems that do not, store a local operation record before the call and reconcile uncertain outcomes instead of immediately repeating the side effect.

An HTTP timeout means "the client did not receive a response." It does not prove the server did nothing. That distinction matters for outbound messages and CRM writes.

If a worker crashes after a successful call but before marking the job complete, the recovery path should check the stored operation state or query the downstream system by a stable external reference. Blind retries are easy to code and hard to clean up.

Bound retries and preserve the job

Retries need an end. I use a short schedule for transient failures, with jitter added by the worker:

const RETRY_DELAYS_MS = [
  1_000,
  5_000,
  30_000,
  120_000,
] as const;

function nextAttempt(attemptCount: number, now = Date.now()): Date | null {
  const base = RETRY_DELAYS_MS[attemptCount];
  if (base === undefined) return null;

  const jitter = Math.floor(Math.random() * Math.min(1_000, base / 4));
  return new Date(now + base + jitter);
}

When the schedule is exhausted, move the job to a terminal review state. Keep the receipt and the job. Deleting either erases the evidence needed to diagnose the miss or replay it safely.

Not every failure deserves a retry. Invalid signatures, unsupported event kinds, and schema errors should fail without entering the queue. Rate limits and temporary upstream errors usually can retry. Authentication failures should pause the integration and surface a credential problem instead of hammering the endpoint.

Do not let retries delay the webhook acknowledgement. The request handler persists the receipt and queue entry, then returns. Workers own the slower work.

Store enough audit evidence, not the whole lead

An audit trail should answer:

Which provider event did we receive?
Which idempotency key won?
Which jobs were created?
How many attempts ran, and what state are they in now?
Was a duplicate acknowledged without creating more work?

You usually do not need the full message body to answer those questions. Store the external identifiers, timestamps, event kind, payload digest, operation versions, and sanitized error codes. Keep raw payload retention short and access controlled if the business requires it at all.

A digest is useful for spotting a provider that reuses one event ID with different payloads. That should be an alert, not a second insert. Record both digests in a security-conscious diagnostic path and investigate the contract violation.

Test the failure boundaries

The happy path is one test. The useful tests force each boundary:

describe("webhook ingestion", () => {
  it("creates one receipt and one job for repeated delivery", async () => {
    await Promise.all([
      postFixture("evt-123"),
      postFixture("evt-123"),
      postFixture("evt-123"),
    ]);

    expect(await countReceipts("evt-123")).toBe(1);
    expect(await countJobs("evt-123")).toBe(1);
  });

  it("keeps distinct updates for the same lead", async () => {
    await postFixture("evt-created", { kind: "lead.created", leadId: "lead-7" });
    await postFixture("evt-updated", { kind: "lead.updated", leadId: "lead-7" });

    expect(await countReceiptsForLead("lead-7")).toBe(2);
  });

  it("does not lose work when the transaction rolls back", async () => {
    failNextJobInsert();

    await expect(postFixture("evt-rollback")).rejects.toThrow();
    await postFixture("evt-rollback");

    expect(await countReceipts("evt-rollback")).toBe(1);
    expect(await countJobs("evt-rollback")).toBe(1);
  });
});

Also test a worker timeout after the downstream side effect succeeds. That is where local idempotency often looks correct while the external system still gets duplicates.

The final property is simple to state: one provider event creates one durable receipt, and every required operation remains recoverable until it reaches a known terminal state. The implementation takes more than a seen flag, but it gives retries somewhere safe to land.

Table-driven tests for AI quote guardrails

Auto Respond — Thu, 30 Jul 2026 19:08:06 +0000

I work with the Auto-Respond Team. This article describes a general testing pattern we use around AI lead handling. The code uses fake services and fake leads. No customer data is involved.

A model can write a convincing quote that is completely wrong.

That is the uncomfortable part of using generated text in a sales workflow. The wording may look polished while the number came from nowhere, the service needs an inspection, or the customer asked for work outside the configured scope.

I do not think a larger prompt fixes this. Price rules belong in ordinary code, where they can be reviewed and tested.

Put the decision before the prose

The model should not decide whether a quote is allowed. Give that job to a small deterministic function. The model can turn an approved decision into natural language after the policy check passes.

Here is a deliberately small TypeScript shape:

export type PriceRule = {
  service: string;
  mode: "fixed" | "range" | "inspection";
  min?: number;
  max?: number;
  currency: "USD";
  policyVersion: string;
};

export type QuoteRequest = {
  service?: string;
  requestedAmount?: number;
  details: string[];
};

export type QuoteDecision =
  | { action: "quote"; min: number; max: number; reason: string }
  | { action: "ask"; question: string; reason: string }
  | { action: "handoff"; reason: string };

The useful constraint is in the return type. There is no free-form price field that a caller can populate from model output.

Keep the policy function boring

Boring is a compliment here.

export function decideQuote(
  request: QuoteRequest,
  rules: PriceRule[],
): QuoteDecision {
  if (!request.service) {
    return {
      action: "ask",
      question: "Which service do you need?",
      reason: "service-missing",
    };
  }

  const rule = rules.find(
    (candidate) => candidate.service === request.service,
  );

  if (!rule) {
    return {
      action: "handoff",
      reason: "service-not-configured",
    };
  }

  if (rule.mode === "inspection") {
    return {
      action: "handoff",
      reason: "inspection-required",
    };
  }

  if (rule.min === undefined || rule.max === undefined) {
    return {
      action: "handoff",
      reason: "invalid-price-rule",
    };
  }

  if (
    request.requestedAmount !== undefined &&
    (request.requestedAmount < rule.min ||
      request.requestedAmount > rule.max)
  ) {
    return {
      action: "handoff",
      reason: "requested-amount-outside-configured-range",
    };
  }

  return {
    action: "quote",
    min: rule.min,
    max: rule.max,
    reason: `approved-by-${rule.policyVersion}`,
  };
}

This function does not try to sound human. It answers one question: may this workflow present a configured price or range?

Test the table, not the prompt

Prompt snapshots are useful, but they are a weak price-control test. A wording change can make a snapshot noisy without changing policy. A table gives each business rule a name and an expected decision.

import { describe, expect, it } from "vitest";
import { decideQuote, type PriceRule } from "./quote-policy";

const rules: PriceRule[] = [
  {
    service: "drain-cleaning",
    mode: "range",
    min: 120,
    max: 220,
    currency: "USD",
    policyVersion: "pricing-2026-07",
  },
  {
    service: "main-line-repair",
    mode: "inspection",
    currency: "USD",
    policyVersion: "pricing-2026-07",
  },
];

const cases = [
  {
    name: "quotes a configured range",
    request: {
      service: "drain-cleaning",
      details: ["kitchen sink is slow"],
    },
    expected: "quote",
  },
  {
    name: "asks when the service is missing",
    request: { details: ["please send a price"] },
    expected: "ask",
  },
  {
    name: "hands off an unknown service",
    request: {
      service: "roof-replacement",
      details: ["two-story house"],
    },
    expected: "handoff",
  },
  {
    name: "hands off work that needs an inspection",
    request: {
      service: "main-line-repair",
      details: ["yard is wet near the cleanout"],
    },
    expected: "handoff",
  },
  {
    name: "rejects a requested number outside the configured range",
    request: {
      service: "drain-cleaning",
      requestedAmount: 25,
      details: ["another company said it would be cheap"],
    },
    expected: "handoff",
  },
] as const;

describe("quote policy", () => {
  it.each(cases)("$name", ({ request, expected }) => {
    expect(decideQuote(request, rules).action).toBe(expected);
  });
});

The fifth case is easy to miss. A lead may mention a price, ask for a discount, or paste a number from another site. That number is context, not an approved rule.

Treat model output as untrusted input

If a model extracts service or requestedAmount, validate the result before calling decideQuote.

A schema validator helps with shape. It does not prove that the extracted value is true. Keep the original lead text available for review, and prefer a handoff when the extraction is uncertain.

I also store four pieces of audit data with the decision:

the normalized service identifier
the price-policy version
the decision reason
the final allowed action

Do not log the whole conversation by default. Tests rarely need a real phone number, address, or customer name. Synthetic fixtures are easier to share and much safer to debug.

Test the sentence after the number is approved

Once the decision returns quote, the model can draft the customer-facing response. The draft still needs a final assertion.

For a range of $120 to $220, extract every currency amount from the generated message and fail the response if any amount falls outside the approved range. Also fail if the message turns a range into a guaranteed final price.

That gives you two separate tests:

Was a quote allowed?
Did the generated sentence stay inside the allowed decision?

The split is worth the extra code. When a test fails, you know whether the problem is policy selection or wording.

The practical boundary

In Auto-Respond's AI answering workflow, the safe product boundary is that AI may quote configured prices or ranges based on business rules. It must not invent or guarantee a final price. The same boundary works in a custom system: deterministic policy first, generated prose second.

If your current test says only expect(message).toContain("$"), replace it with a decision table. You will catch the failures that polished demo transcripts hide.

Designing a Stop-on-Reply State Machine for Queued Follow-Ups

Auto Respond — Thu, 30 Jul 2026 02:51:45 +0000

Lead-follow-up systems usually fail at the boundary between two valid events: a message is queued, then the customer replies before the worker sends it. A settings flag called "stop on reply" is not enough. The stop has to survive retries, duplicate webhooks, delayed jobs, and two workers racing each other.

This article shows a small TypeScript state machine and the tests I would require before allowing it near a production conversation.

Disclosure: I work with the Auto-Respond Team. This is an engineering pattern from our own testing, not an independent product review.

Start with an explicit conversation state

Do not infer the state from whichever table a worker happens to read. Give the conversation one authoritative state and restrict transitions.

type ConversationState =
  | "active"
  | "replied"
  | "booked"
  | "opted_out"
  | "paused"
  | "closed";

type StopReason = Exclude<ConversationState, "active">;

interface Conversation {
  id: string;
  state: ConversationState;
  version: number;
  stoppedAt?: string;
  stopReason?: StopReason;
}

The version matters. It gives the database a compare-and-set boundary, so a worker cannot save an old active snapshot over a newer replied state.

Treat inbound events as commands, not observations

A webhook saying that a customer replied is not merely an analytics event. It is a command to transition the conversation and invalidate future work.

Give every inbound event a stable provider ID. If the provider does not supply one, derive an idempotency key from the conversation ID, event type, and provider timestamp, then keep the raw payload hash for investigation.

interface StopEvent {
  eventId: string;
  conversationId: string;
  reason: StopReason;
  occurredAt: string;
}

async function applyStop(event: StopEvent): Promise<void> {
  await db.transaction(async (tx) => {
    const firstSeen = await tx.eventInbox.insertIfAbsent({
      id: event.eventId,
      receivedAt: new Date().toISOString(),
    });

    if (!firstSeen) return;

    const conversation = await tx.conversations.getForUpdate(
      event.conversationId,
    );

    if (!conversation || conversation.state !== "active") return;

    await tx.conversations.updateWhereVersion(
      conversation.id,
      conversation.version,
      {
        state: event.reason,
        version: conversation.version + 1,
        stoppedAt: event.occurredAt,
        stopReason: event.reason,
      },
    );

    await tx.messageOutbox.cancelFutureForConversation(
      conversation.id,
      event.occurredAt,
    );
  });
}

The inbox insert, state transition, and outbox cancellation belong in one transaction. Splitting them creates a failure window where the conversation says replied but a future message is still sendable.

Make the send worker prove permission twice

Cancelling future rows is necessary, but it is not sufficient. A worker may already have claimed a row before the reply arrived.

Use a two-check send path:

Claim one queued message with a short lease.
Read the authoritative conversation state immediately before delivery.
Atomically mark the row sending only if the conversation is still active.
Call the provider.
Record the provider result with the same attempt ID.

async function deliver(row: OutboxMessage): Promise<void> {
  const attemptId = crypto.randomUUID();

  const permitted = await db.transaction(async (tx) => {
    const conversation = await tx.conversations.getForUpdate(
      row.conversationId,
    );

    if (!conversation || conversation.state !== "active") {
      await tx.messageOutbox.cancel(row.id, "conversation_stopped");
      return false;
    }

    return tx.messageOutbox.markSendingIfQueued(row.id, attemptId);
  });

  if (!permitted) return;

  const result = await provider.send({
    to: row.destination,
    body: row.body,
    idempotencyKey: row.id,
  });

  await db.messageOutbox.recordProviderResult(row.id, attemptId, result);
}

The second check closes the most common race. It does not make a remote provider call transactional, so the UI and audit log must be honest about the final boundary.

Define the handoff boundary

Once a provider has accepted a message, cancellation may no longer be possible. Model that fact instead of presenting a perfect-looking toggle.

Useful outbox states are queued, leased, sending, accepted, delivered, cancelled, and failed. A stop event should cancel queued and expired leased rows. It should mark an in-flight sending row as stop_requested and capture whether the provider accepted it before or after the stop timestamp.

That gives support staff a concrete answer when someone asks why a message still arrived. "Automation was off" is not an audit trail.

Test the races deliberately

A happy-path unit test is almost worthless here. Add barriers so the tests can pause a worker at each boundary.

At minimum, cover these cases:

reply before the worker claims the row;
reply after claim but before the permission check;
reply after permission check but before the provider accepts;
duplicate reply webhook;
delivery worker retry after a timeout;
booking and reply arriving in either order;
opt-out arriving while two follow-ups are queued;
stale worker attempting to overwrite a newer conversation version.

A useful integration test freezes the worker after claim, applies the reply, releases the worker, then waits beyond the original scheduled time. The pass condition is not just "row cancelled." It is no provider acceptance and no retry capable of recreating the row.

it("does not send when a reply wins the claim race", async () => {
  const gate = createBarrier();
  const queued = await seedQueuedFollowUp();

  worker.beforePermissionCheck = () => gate.wait();
  const delivery = worker.runOnce();

  await gate.reached;
  await applyStop(replyEventFor(queued.conversationId));
  gate.release();

  await delivery;
  expect(provider.accepted).toHaveLength(0);
  expect(await outbox.state(queued.id)).toBe("cancelled");
});

Keep evidence that survives the dashboard

For each stop test, store the configuration version, conversation ID, provider event ID, queued message IDs, state transitions, timestamps, worker attempt IDs, and provider responses. Do not put real customer data in the test packet.

A dashboard can then show four separate facts:

when the inbound stop event occurred;
when the conversation state changed;
which queued rows were cancelled;
whether any provider accepted a message after that point.

This is the difference between a feature claim and a reproducible result.

For context, our current vendor-supplied Yelp workflow facts and stop conditions are published in the Yelp auto responder guide. Apply the same race tests to that workflow or any competing implementation before trusting the setting.

The durable design rule is simple: stopping is a state transition, not a filter in the UI. Put it in the transaction, recheck it at delivery, and test the races on purpose.

How to Measure First-Response Latency for Marketplace Leads

Auto Respond — Wed, 29 Jul 2026 05:09:17 +0000

If a lead-response system claims to answer quickly, the first question should be: quickly from which timestamp to which timestamp?

A dashboard that measures only application processing time can look healthy while the customer waits through webhook delivery, provider queues, retries, and channel rendering. The useful metric is the delay the lead actually experiences.

Disclosure: we build Auto-Respond, which handles lead replies for home-service businesses. The measurement method below is vendor-neutral and applies whether you build the system or buy one.

Define two latency metrics, not one

Track an internal engineering metric and an end-to-end operator metric:

Internal latency: from the moment your endpoint accepts the lead event to the moment the reply provider acknowledges the outbound message.
Observed first-response latency: from the marketplace's lead-event timestamp to the moment a reply becomes visible in the marketplace conversation.

The first metric helps engineers find slow code. The second tells an operator what the customer experienced. Do not substitute the first for the second.

A useful event model is:

t0_source_created: timestamp attached by the marketplace to the new lead or message
t1_ingress_accepted: webhook accepted by your edge
t2_normalized: source payload converted into your common lead schema
t3_reply_ready: reply decision and text completed
t4_send_acknowledged: outbound channel API accepts the send
t5_reply_visible: reply appears in the source conversation, when the platform exposes a read-back event or API

Then calculate:

internal_latency = t4 - t1

observed_latency = t5 - t0

If the source does not expose t5, label t4 - t0 as send-ack latency, not customer-visible latency. Precision in the label matters.

Use a monotonic clock inside the service

Wall clocks can jump because of time synchronization. Record UTC timestamps for cross-system correlation, but use a monotonic clock for durations inside one process.

You also need to account for source-clock skew. A negative t1 - t0 is usually not miraculous performance; it is a clock problem. Keep skewed samples, flag them, and exclude them from percentile calculations until you understand the cause.

Log one trace per lead event

A compact structured record is enough:

{
  "trace_id": "01J...",
  "source": "marketplace_a",
  "source_event_id_hash": "sha256:...",
  "received_at": "2026-07-29T04:00:00.412Z",
  "normalize_ms": 18,
  "decision_ms": 241,
  "send_ms": 96,
  "internal_total_ms": 355,
  "send_attempts": 1,
  "outcome": "sent",
  "duplicate": false
}

Hash or replace personal identifiers. You rarely need a customer's name, phone number, address, or full message in a latency log. Keeping those fields out reduces privacy risk and makes the dataset easier to share internally.

Count failures in the latency report

A percentile calculated only from successful replies hides the cases that matter most. Report these beside p50, p95, and p99 latency:

percentage of eligible leads with no reply
outbound API failure rate
retry rate
duplicate-reply rate
source events received outside the platform's reply window
percentage of samples with missing or unreliable timestamps

A fast system that silently drops eligible leads is not a fast system. It is an unreliable one with flattering charts.

Measure percentiles by source

Do not pool every marketplace into a single number. Webhook behavior, API limits, and delivery acknowledgement vary by source.

For each source, publish at least:

Metric	Why it matters
p50	Normal experience
p95	What happens on a busy or slow path
p99	Tail failures that operators notice
max	Useful for incident review, not as a stable KPI
no-reply rate	Prevents success-only sampling
sample count	Stops tiny samples looking conclusive

Split by hour of day as well. A 24/7 responder should not have a clean daytime chart and an unmeasured overnight queue.

Separate transport time from decision time

When p95 increases, one total duration is not actionable. Break it into stages:

source-to-ingress delay
queue wait
payload normalization
reply decision or generation
outbound API call
acknowledgement-to-visible delay

Track the slowest contributing stage for each trace. That makes the incident question concrete: did the source deliver late, did our queue back up, or did the outbound channel slow down?

Make retries idempotent

Marketplaces retry webhooks. Workers retry jobs. Providers sometimes return ambiguous timeouts. Your latency test should include repeated delivery of the same source event ID.

The expected result is one conversation reply and a second trace marked as a duplicate. A duplicated answer should be treated as a failed test even if both sends were fast.

A practical idempotency key is a stable combination of source, account, conversation, and source event ID. Store it before the expensive reply path, not after the send.

Test the whole path with canary leads

A synthetic test that calls your internal function is useful, but it is not an end-to-end measurement. Maintain a small canary matrix for each supported source:

Create or use a permitted test lead.
Record the source's event timestamp.
Observe ingress, processing, send acknowledgement, and visible reply.
Verify that exactly one reply appears.
Verify that later automation stops when the test lead replies or books.

Run canaries after deploys and on a schedule that respects each platform's test and messaging policies. Never create fake public reviews or send messages to real prospects for monitoring.

Define “first response” before setting an SLO

A transport acknowledgement is not a meaningful reply. Decide what qualifies before you announce a target.

For example, a valid first response might need to acknowledge the requested service, ask one relevant next question, and avoid promising unsupported availability or pricing. A generic “we received your request” can be tracked separately as an acknowledgement.

At Auto-Respond, the operational target for supported Yelp, Thumbtack, Facebook, Instagram, and Google Local Services Ads text leads is a first reply in under 2 seconds. We also schedule day 1, 3, and 7 follow-ups that stop when the lead replies or books. For one channel-specific example, our Yelp auto responder guide explains the conversation flow and boundaries.

That product target is not a reason to omit the tail. The same report still needs p95, p99, missed replies, retries, and duplicates.

A simple weekly review

A useful weekly latency review fits on one page:

volume and eligibility by source
p50/p95/p99 observed latency by source
no-reply, retry, and duplicate rates
five slowest traces with the contributing stage
comparison with the previous complete week
one owner and due date for each regression

The stoplight should be based on a predeclared service-level objective and error budget, not on whether the latest graph “looks fine.” If the visible-reply timestamp is unavailable, state that limitation prominently.

The main lesson is simple: instrument from the marketplace event to the visible conversation, count failures alongside latency, and keep source-specific tails visible. Otherwise you are measuring the fastest part of the system and calling it the customer experience.

Speed-to-lead is an engineering problem: winning the job in under 2 seconds

Auto Respond — Sun, 26 Jul 2026 02:08:26 +0000

Disclosure: I run Auto-Respond, an AI lead-response tool for home-service businesses. This post is about the problem, not a pitch. The product link sits at the bottom.

Most "lead management" advice treats speed as a nice-to-have. For local service businesses it is close to the whole game, and it turns out to be an interesting systems problem.

The number that matters

Dr. James Oldroyd's lead-response research at MIT found that waiting even 5 to 30 minutes to respond makes you roughly 100x less likely to ever reach a lead. Harvard Business Review's audit of thousands of US companies found the median first-response time was measured in hours, and a large share never responded at all. The business that answers first usually books the job.

So the target is not "respond quickly." The target is: first meaningful reply in under 2 seconds, 24/7, with no human in the loop for the first touch.

Why it is harder than a webhook plus an autoresponder

A canned "Thanks, we'll be in touch" hits the 2-second bar and loses the job anyway, because the lead moves on to whoever actually answered their question. A real first reply has to:

Normalize very different inputs. A Yelp "Request a Quote", a Thumbtack lead, a Facebook Messenger message, a Facebook Lead Ad, an Instagram DM, and a Google Local Services Ads lead all arrive in different shapes, over different APIs, with different rate limits and reply windows.
Understand intent, not just keywords. Job type, location, and urgency decide whether this is a booking or a tire-kicker.
Ask the right next question and keep the thread going until the lead books or opts out.
Recover missed phone calls. A contractor on a roof cannot pick up. A missed call should trigger an instant AI voice callback so the lead is still answered.
Follow up on a schedule (day 1, 3, 7) and stop automatically the moment the customer replies, so you are persistent without being spam.
Hand off cleanly to a human and the CRM once the lead is qualified.

Each of those is a small thing. Doing all of them within a couple of seconds, across six channels, reliably, is the actual product.

Lessons from building this

A few things that were not obvious going in:

The latency budget is spent before you write a word. Channel webhook, intent parse, reply generation, channel send. If the customer should see a reply in under 2 seconds, each hop gets a tiny slice, so you design for the slow channel, not the fast one.
Idempotency matters more than you expect. Lead platforms retry webhooks. Answer the same lead twice and you look broken.
"Stop" logic is a feature. Persistent follow-up only works if it reliably detects a reply or booking and then goes quiet. Getting that wrong is worse than not following up at all.
Voice is a different failure surface. Text you can queue. A missed-call callback has to happen now or it is pointless.

If you just want it to work

If you run a home-service business and do not want to build any of this, that is what Auto-Respond does: it answers Yelp (including Request a Quote), Thumbtack, Facebook Messenger, Facebook Lead Ads, Instagram, and Google Local Services Ads leads in under 2 seconds, qualifies the job, books it, follows up on day 1, 3 and 7, calls back missed calls, and syncs to your CRM. Setup takes a couple of minutes.

Vendor-neutral buyer's guide to picking an AI answering service: Best AI Answering Service in 2026 - How to Choose

Happy to answer questions about the speed-to-lead side in the comments.