My agent's p50 was 29s. Its p95 was 182s. That ratio decided the product.

I have 67 timed turns against a live LLM agent, captured in a single batch run and written to disk:

	seconds
p50	29.2
p95	182.0
max	210.3
n	67 completed turns

The median says background job, that's fine. The p95 says you may never put a human in front of this.

Those are not two performance notes. They are a product spec — and I found that out the expensive way, by designing the product first and measuring second.

• Live: porchlight.edycu.dev
• Poke the real agent yourself: try.porchlight.edycu.dev
• Code: github.com/edycutjong/porchlight

The product, briefly, because the latency only means something against it

A member cancels, and the reason vanishes — nobody writes it down. Months later the creator fixes the exact thing that drove people away, and the people who left for that reason are never told. The state of the art is a "we miss you" blast to everyone.

Porchlight puts an agent — Minds by Animoca Brands — on the critical path in three places: a short warm exit interview that files a structured return-condition in the member's own words; condition matching, which asks whether this announcement genuinely resolves that person's reason for leaving; and a win-back draft that quotes the member back to themselves.

The middle one is the step that has to be an agent, and I wanted to prove that rather than assert it.

So I shipped the dumb version alongside it

Every "AI-powered" claim should ship its control. Mine is twenty lines, it lives in the repo, and it runs on the same inputs on every demo run:

// src/keywordBaseline.ts — the "dumb tool" strawman
const STOP = new Set(['the','and','are','was','were','you','your','for','that','this','with',
  'have','has','had','not','but','now','all','its',"it's",'been','back','big','news','just',
  'about','from','they','them','our','out','get','got','weekly','more'])

const tokens = (s: string): string[] =>
  (s.toLowerCase().match(/[a-z][a-z'-]{3,}/g) ?? []).filter((w) => !STOP.has(w))

/** True iff the parting quote and the announcement share at least one salient keyword. */
export function keywordResolves(changeText: string, verbatimQuote: string): boolean {
  const a = new Set(tokens(changeText))
  return tokens(verbatimQuote).some((w) => a.has(w))
}

Here is a real pair from the seed data. A member left saying:

"the long chatty sit-downs with guests were the whole reason i was here, now it is quick clips"

and the creator later announced:

"Big news — the deep-dive interviews are back, weekly."

Same event. After stopwords, the announcement contributes {deep-dive, interviews} and the quote contributes {long, chatty, sit-downs, guests, whole, reason, here, quick, clips}. The intersection is empty, so keywordResolves returns false — and no amount of stopword tuning will ever link "clips" to "deep-dive". The agent resolves it, and explains why.

Across 54 captured judgements the agent resolved 14 departures, 11 of which the keyword baseline scores 0.00 on — while refusing 35 non-matching pairs at ≥0.90 confidence. The recall is the pitch; the precision is what makes it safe to actually send. If you are emailing real people who already left once, a false positive is worse than a miss.

Fine. The agent is load-bearing. Now the bill.

The bill

Sorted, those 67 samples look like this: fastest 10.3s, a long fat body between 15s and 50s, a handful in the 60–115s range, then five clustered at ~182s, then one at 210.3s — that last one being a 180s client timeout followed by a successful retry.

That is not a distribution you can hide behind a spinner.

The architecture I had sketched before measuring: visitor clicks announce a change, the server fans out across every open departure, results render. With 18 departures that is 18 turns. At p50 that's about nine minutes. At p95 it's closer to an hour. And even a single turn — the best case in the whole design — is a coin flip between ten seconds and three minutes.

What got cut

Three decisions, all downstream of that one ratio.

1. No synchronous fan-out, ever. The public demo replays verdicts captured ahead of time by a separate npm run precompute pass, which writes them to src/liveCache.json with a capturedAt stamp on each one. Every verdict a visitor sees is real agent output; none of it is computed while they wait. The UI says when it was captured, because a replay that pretends to be live is a lie.

2. The one genuinely live path is bounded and rationed.

/** Longest a visitor is asked to wait on a live turn before we give up on it. */
const WEB_DEADLINE_MS = 100_000

function withDeadline<T>(work: Promise<T>, ms: number): Promise<T> {
  return Promise.race([
    work,
    new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error(`no reply within ${Math.round(ms / 1000)}s`)), ms).unref(),
    ),
  ])
}

100 seconds is not a round number I liked; it is p50 with real headroom and deliberately below the 182s p95. It gives up on the slow tail on purpose rather than holding a browser open for three minutes. Some requests do fail, and the error message says exactly that — that this is a real call to a real agent and sometimes it is slow. Paired with 3 live calls per IP per 15 minutes, and scoped to one member the visitor picks rather than a fan-out.

3. No keyword fallback in the deployed app. This is the decision I'd defend hardest. When the agent is slow or unreachable, the tempting move is to fall back to the cheap path — you always have one, because you built it as the control. But the cheap path is the exact mechanism the product exists to beat. Falling back to it means quietly shipping the strawman under the good name, and nobody would ever know. With no credentials the service returns 503 and says why.

Two other things the SDK taught me, both non-obvious

A stale reply is a silent correctness bug. Send-then-wait reads like it should just work:

const before = await c.getLatestHistoryFingerprint(alias).catch(() => undefined)
await c.sendMessage({ alias, messageText: text })
const outcome = await c.waitForReply({
  alias, timeoutMs: CONFIG.replyTimeoutMs,
  afterFingerprint: before,      // captured BEFORE the send
  sentMessageText: text,
})

Drop afterFingerprint and you can be handed the previous turn's reply. It does not throw. It returns a completely plausible answer to a question you did not ask. In a system whose entire job is per-member judgement, that is a wrong email to a real person, and it is the worst class of bug to debug because nothing anywhere looks broken.

No JSON mode means scraping prose. There is no schema/response-format option, so structured output means asking for JSON in the prompt, then going and finding it — indexOf('{'), lastIndexOf('}'), JSON.parse the slice, hand it to Zod. It works, and it is brittle by construction: the reply can preface the JSON with commentary, fence it, or emit two objects. (Replies also arrive as HTML, which is its own small adventure in stripping tags after decoding entities rather than before.)

The number I had to retract

Same agent, same prompt, same member quote, run weeks apart. A departure that said:

"you stopped doing the long-form lore videos i subscribed for"

judged against "the deep-dive interviews are back, weekly" resolved true in an early run and false, confidence 0.60 in the full capture, with this rationale:

"The announcement restores long-form content but specifically as deep-dive interviews, not the lore videos the member subscribed for; the subject-matter mismatch means the member's core interest in lore is likely still unmet."

The second answer is better than the first. That is not the point. The point is that I had a recovered-revenue figure in my README that read like a constant, and it is not one — it is a snapshot of a single run. There is no temperature or seed exposed, so I cannot opt into determinism even where I'd want it.

So I rewrote the README to say the figures are per-run, and published the traces in both directions. If you derive a metric from a batch of LLM judgements, you have measured that run. Say so in the same sentence as the number, or someone will eventually try to reproduce it and conclude you made it up.

Limitations, honestly

• The membership platform is mocked. Patreon's API is restricted, so cancel/rejoin runs against a storefront I built, not real billing.
• The sandbox's curated verdicts are a replay, dated in the UI. Only the "put it on the spot" path is live.
• 54 judgements is a small sample. The precision/recall shape is indicative, not a benchmark.
• ~3% of turns failed with a bare untyped fetch failed and succeeded on retry. With no error code or retryable flag, every failure has to be treated as retryable — which is wrong for 4xx-class problems.
• Agents and their skills are console-only — you can't provision one programmatically, so the full loop can't run in CI and onboarding needs a human.
• No external users yet. This has never met a real creator's churn.

On rigor rather than as the story: 58 tests, 100% line/branch/function coverage, Playwright E2E, and a CI stage that fails the build if the deployed app comes up in mock mode.

The one thing I'd carry to the next project

Measure the tail before you draw the architecture. The p50 is a comfort; the p95 is the constraint. Mine bought a precompute cache, a 100-second deadline, a rate limit, and a refusal to ever fall back to the dumb path — and in retrospect those four decisions are most of the engineering.

Everything the SDK taught me, latency data included, is in FEEDBACK.md. To reproduce the numbers: npm run precompute writes per-call timings straight into src/liveCache.json.

Code: github.com/edycutjong/porchlight · Live: porchlight.edycu.dev · Try the real agent: try.porchlight.edycu.dev

The most interesting thing you can do in the sandbox is try to fool it — describe a fix that shouldn't win someone back, and see whether it stays quiet. If it holds up, a star helps.