If a lead-response system claims to answer quickly, the first question should be: quickly from which timestamp to which timestamp?
A dashboard that measures only application processing time can look healthy while the customer waits through webhook delivery, provider queues, retries, and channel rendering. The useful metric is the delay the lead actually experiences.
Disclosure: we build Auto-Respond, which handles lead replies for home-service businesses. The measurement method below is vendor-neutral and applies whether you build the system or buy one.
Define two latency metrics, not one
Track an internal engineering metric and an end-to-end operator metric:
- Internal latency: from the moment your endpoint accepts the lead event to the moment the reply provider acknowledges the outbound message.
- Observed first-response latency: from the marketplace's lead-event timestamp to the moment a reply becomes visible in the marketplace conversation.
The first metric helps engineers find slow code. The second tells an operator what the customer experienced. Do not substitute the first for the second.
A useful event model is:
-
t0_source_created: timestamp attached by the marketplace to the new lead or message -
t1_ingress_accepted: webhook accepted by your edge -
t2_normalized: source payload converted into your common lead schema -
t3_reply_ready: reply decision and text completed -
t4_send_acknowledged: outbound channel API accepts the send -
t5_reply_visible: reply appears in the source conversation, when the platform exposes a read-back event or API
Then calculate:
internal_latency = t4 - t1
observed_latency = t5 - t0
If the source does not expose t5, label t4 - t0 as send-ack latency, not customer-visible latency. Precision in the label matters.
Use a monotonic clock inside the service
Wall clocks can jump because of time synchronization. Record UTC timestamps for cross-system correlation, but use a monotonic clock for durations inside one process.
You also need to account for source-clock skew. A negative t1 - t0 is usually not miraculous performance; it is a clock problem. Keep skewed samples, flag them, and exclude them from percentile calculations until you understand the cause.
Log one trace per lead event
A compact structured record is enough:
{
"trace_id": "01J...",
"source": "marketplace_a",
"source_event_id_hash": "sha256:...",
"received_at": "2026-07-29T04:00:00.412Z",
"normalize_ms": 18,
"decision_ms": 241,
"send_ms": 96,
"internal_total_ms": 355,
"send_attempts": 1,
"outcome": "sent",
"duplicate": false
}
Hash or replace personal identifiers. You rarely need a customer's name, phone number, address, or full message in a latency log. Keeping those fields out reduces privacy risk and makes the dataset easier to share internally.
Count failures in the latency report
A percentile calculated only from successful replies hides the cases that matter most. Report these beside p50, p95, and p99 latency:
- percentage of eligible leads with no reply
- outbound API failure rate
- retry rate
- duplicate-reply rate
- source events received outside the platform's reply window
- percentage of samples with missing or unreliable timestamps
A fast system that silently drops eligible leads is not a fast system. It is an unreliable one with flattering charts.
Measure percentiles by source
Do not pool every marketplace into a single number. Webhook behavior, API limits, and delivery acknowledgement vary by source.
For each source, publish at least:
| Metric | Why it matters |
|---|---|
| p50 | Normal experience |
| p95 | What happens on a busy or slow path |
| p99 | Tail failures that operators notice |
| max | Useful for incident review, not as a stable KPI |
| no-reply rate | Prevents success-only sampling |
| sample count | Stops tiny samples looking conclusive |
Split by hour of day as well. A 24/7 responder should not have a clean daytime chart and an unmeasured overnight queue.
Separate transport time from decision time
When p95 increases, one total duration is not actionable. Break it into stages:
- source-to-ingress delay
- queue wait
- payload normalization
- reply decision or generation
- outbound API call
- acknowledgement-to-visible delay
Track the slowest contributing stage for each trace. That makes the incident question concrete: did the source deliver late, did our queue back up, or did the outbound channel slow down?
Make retries idempotent
Marketplaces retry webhooks. Workers retry jobs. Providers sometimes return ambiguous timeouts. Your latency test should include repeated delivery of the same source event ID.
The expected result is one conversation reply and a second trace marked as a duplicate. A duplicated answer should be treated as a failed test even if both sends were fast.
A practical idempotency key is a stable combination of source, account, conversation, and source event ID. Store it before the expensive reply path, not after the send.
Test the whole path with canary leads
A synthetic test that calls your internal function is useful, but it is not an end-to-end measurement. Maintain a small canary matrix for each supported source:
- Create or use a permitted test lead.
- Record the source's event timestamp.
- Observe ingress, processing, send acknowledgement, and visible reply.
- Verify that exactly one reply appears.
- Verify that later automation stops when the test lead replies or books.
Run canaries after deploys and on a schedule that respects each platform's test and messaging policies. Never create fake public reviews or send messages to real prospects for monitoring.
Define “first response” before setting an SLO
A transport acknowledgement is not a meaningful reply. Decide what qualifies before you announce a target.
For example, a valid first response might need to acknowledge the requested service, ask one relevant next question, and avoid promising unsupported availability or pricing. A generic “we received your request” can be tracked separately as an acknowledgement.
At Auto-Respond, the operational target for supported Yelp, Thumbtack, Facebook, Instagram, and Google Local Services Ads text leads is a first reply in under 2 seconds. We also schedule day 1, 3, and 7 follow-ups that stop when the lead replies or books. For one channel-specific example, our Yelp auto responder guide explains the conversation flow and boundaries.
That product target is not a reason to omit the tail. The same report still needs p95, p99, missed replies, retries, and duplicates.
A simple weekly review
A useful weekly latency review fits on one page:
- volume and eligibility by source
- p50/p95/p99 observed latency by source
- no-reply, retry, and duplicate rates
- five slowest traces with the contributing stage
- comparison with the previous complete week
- one owner and due date for each regression
The stoplight should be based on a predeclared service-level objective and error budget, not on whether the latest graph “looks fine.” If the visible-reply timestamp is unavailable, state that limitation prominently.
The main lesson is simple: instrument from the marketplace event to the visible conversation, count failures alongside latency, and keep source-specific tails visible. Otherwise you are measuring the fastest part of the system and calling it the customer experience.
Top comments (0)