A lead-response worker can be fast and still send at the wrong time.
That sounds contradictory until you split time into two clocks:
- The transport clock measures how long an event spends moving through ingestion, processing, and delivery.
- The policy clock decides when a reply is allowed to leave.
Many systems measure the first and bury the second inside a delay or cron job. That makes incidents hard to explain. A message can leave the queue in 80 ms and still be early, late, duplicated, or unsafe.
This article sketches an architecture that treats both clocks as first-class data. The examples use inbound Yelp and Thumbtack workflows, but the model applies to any channel with webhooks, retries, account rules, and an outbound API.
Disclosure: this post comes from the Auto-Respond team. Auto-Respond supports lead-response workflows for Yelp and Thumbtack, among other channels. AI assisted with editing this draft, and the team reviewed the technical content and product statements before publication.
Clock one: transport latency
The transport clock answers an engineering question:
How long did the system take to turn a received event into an acknowledged outbound request?
A useful trace has more than one timestamp:
type TransportTrace = {
sourceEventAt: string; // timestamp supplied by the channel
ingressAcceptedAt: string; // our edge accepted the webhook
jobClaimedAt: string; // a worker claimed the job
decisionReadyAt: string; // reply or action was ready
sendStartedAt: string;
providerAcknowledgedAt?: string;
visibleAt?: string; // only when the channel exposes it
};
These timestamps let you separate source delivery, queue wait, decision time, outbound request time, and acknowledgment. Do not call the last one "visible reply latency" unless the channel exposes a read-back signal that proves the reply appeared in the conversation.
Use UTC wall-clock timestamps for correlation across services. Use a monotonic clock for durations inside one process. Wall clocks can jump during time synchronization, while a monotonic timer only moves forward.
Clock two: safe send timing
The policy clock answers an operational question:
Is this message allowed to leave now?
Represent that decision explicitly:
type SendPolicy = {
notBefore: string;
expiresAt?: string;
reason:
| "immediate-first-response"
| "business-rule-delay"
| "rate-limit-backoff"
| "manual-review"
| "escalation-window";
policyVersion: string;
};
The worker should never infer notBefore from "job age" or from when it happened to claim the job. Store the boundary when the policy decision is made. If a later rule change matters, create a new policy decision with a new version and keep the old one in the audit trail.
This separation matters for channel-specific behavior. Yelp and Thumbtack can feed the same normalized lead model while still having different reply rules, account settings, API behavior, or retry semantics. Sharing a queue does not mean sharing every send policy.
An acknowledgment is not permission
A provider acknowledgment proves that an API accepted a request. It does not prove three other things:
- that the customer can see the reply;
- that the reply was allowed by the latest business rule;
- that another worker did not already send the same logical message.
Keep the states separate:
received
-> eligible
-> scheduled
-> send_started
-> provider_acknowledged
-> visible_confirmed (when available)
A terminal failure, cancellation, or escalation can branch from any state where sending remains possible. Avoid a generic done state. It hides whether the system actually sent, merely received an acknowledgment, or handed the conversation to a person.
Claim only when both clocks agree
A simple worker loop can enforce the boundary:
async function claimSendable(now: Date) {
return db.messageJobs.findOneAndUpdate(
{
state: "scheduled",
notBefore: { $lte: now },
$or: [
{ expiresAt: null },
{ expiresAt: { $gt: now } }
]
},
{
$set: {
state: "send_started",
claimedAt: now
},
$inc: { attempt: 1 }
},
{
sort: { notBefore: 1 },
returnDocument: "after"
}
);
}
The database transition must be atomic. Reading a ready job and updating it in two separate operations leaves room for two workers to claim it.
Also re-check current conversation state after the claim. A lead may have replied while the job waited. The stored schedule says when sending became possible, not that sending is still correct.
Idempotency protects the boundary
Webhooks retry. Queue workers crash. Outbound requests time out after the channel has accepted them. Those are ordinary conditions, so the design needs one stable idempotency key per logical send.
A practical key can combine:
source account + conversation + source event + message purpose + policy version
Store the key before calling the outbound provider. Then use a unique constraint so concurrent workers cannot create two send records.
A timeout after the request is the awkward case. Do not immediately issue a second send. First use the channel's supported read-back or request-status mechanism, when available. If the outcome cannot be resolved automatically, mark it delivery_unknown and route it to review. Guessing is how a fast retry becomes a duplicate customer message.
Delays need reasons, not magic numbers
A value like delayMs: 300000 is difficult to operate. Five minutes for what reason? Which rule produced it? Does a later inbound message cancel it? Can a human override it?
Persist the reason and the evaluated inputs:
{
"not_before": "2026-08-16T08:35:00.000Z",
"reason": "business-rule-delay",
"policy_version": "yelp-v12",
"evaluated_at": "2026-08-16T08:30:00.000Z",
"cancel_on_inbound": true
}
Do not put customer names, phone numbers, full messages, or addresses into timing logs. A source event identifier can be hashed if operators only need correlation.
Escalation has its own deadline
Some conversations should leave automation instead of waiting for another send attempt. Model escalation as a deadline, not as a note attached to a failed job.
Useful fields include:
- escalateAt;
- escalationReason;
- automationPausedAt;
- assignedQueue;
- resolvedAt.
Then define the ordering rule. For example, if escalateAt is due, pause the automated send before notifying the review queue. That ordering prevents a message from leaving during the handoff.
The same rule should be idempotent. Replaying an escalation event must not create several assignments or several notifications.
Metrics that expose the real failure
Track the two clocks independently.
For transport:
- source-to-ingress delay;
- queue wait;
- decision duration;
- outbound request duration;
- acknowledgment rate;
- unknown-delivery rate.
For policy timing:
- sends before notBefore (the target is zero);
- jobs sent after expiresAt;
- policy cancellations honored;
- duplicate logical sends;
- escalations completed before their deadline;
- time spent waiting for review.
A single "response time" chart cannot tell you whether the system was slow or correctly waiting. It also cannot show an early send. Pair latency percentiles with policy-violation counters and report sample counts.
Test both clocks with a fake clock
Timing code becomes easier to test when business logic receives a clock instead of calling new Date() everywhere.
interface Clock {
now(): Date;
}
function isSendable(job: Job, clock: Clock) {
const now = clock.now();
if (job.state !== "scheduled") return false;
if (now < job.notBefore) return false;
if (job.expiresAt && now >= job.expiresAt) return false;
return true;
}
Table-driven tests should cover the exact boundary, one millisecond before it, expiry, cancellation after a new inbound event, duplicate claims, an ambiguous provider timeout, and escalation winning a race with send.
The tests should also replay the same webhook and the same queue job. "Exactly once" is usually an application guarantee built from idempotent effects, not a promise made by the transport.
The design rule
Transport latency and safe send timing describe different truths. One says how fast the machinery moved. The other says whether it was correct to move at all.
Keep both clocks in the event model, version the policy decision, make claims atomic, and treat provider acknowledgment as its own state. That gives operators enough evidence to explain a late reply without hiding a policy wait, and enough control to stop an early or duplicate send.
For a channel-specific view of the conversation boundary, the Yelp auto responder guide describes the supported workflow without treating every channel as identical.
Top comments (0)