DEV Community

Qasim
Qasim

Posted on

A/B test email subject lines with an agent — no ESP required

#ai #email #api #productivity

Subject-line A/B testing is the kind of thing you usually rent. You sign up for an ESP, drop your contacts into their list, pick "split test" from a dropdown, and let their black box decide a winner — on their definition of "winner," with their attribution, on their schedule. That's fine if email is your whole product. It's a lot of ceremony if you have an agent that already sends mail and you just want to know which of two subjects gets more humans to reply.

Here's the pivot: the experiment is just two sends and a counter. If your agent owns a mailbox, it can run the split itself — pick a variant per recipient, send from its own address, watch replies land back in its own threads, and tally the result. No ESP, no list import, no separate analytics product. The whole thing lives inside the same mailbox the agent already uses.

I work on the Nylas CLI, so the terminal commands below are the exact ones I reach for, and I'll show the matching HTTP call for each one so you can wire it into whatever your agent is actually built on. The data plane is the same either way.

What you actually get

An Agent Account is a real, sending-and-receiving mailbox that your code owns outright — no human, no OAuth handshake, no token to refresh. Under the hood it's just a Nylas grant with a grant_id, which is the important part: every grant-scoped endpoint you already know (Messages, Threads, Folders, Webhooks) works against it unchanged. There's nothing new to learn on the data plane. You send with POST /v3/grants/{grant_id}/messages/send, you read threads with GET /v3/grants/{grant_id}/threads/{thread_id}, and replies arrive as message.created webhooks. That's the entire surface area this experiment needs.

What Nylas does not do is run the experiment for you. There's no "split test" endpoint, no built-in significance math, no variant tracking. And that's correct — an experiment is your business logic, not the mail provider's. Nylas moves the email and reports the events; the random assignment, the sample-size discipline, and the winner call all live in your app. I'll be honest about that line throughout, because the parts you own are the parts that make the result trustworthy.

Why this beats reaching for an ESP

  • No data export. Your recipients, your sends, your replies all stay in your system. You're not syncing a contact list into a third party just to test two strings.
  • Reply rate, not open rate. Most ESP split tests optimize opens or clicks. With an agent mailbox you can attribute on replies — a human typed something back — which is a far stronger signal for outreach, support, or sales follow-ups.
  • One code path, every provider. A reply from Gmail, Outlook, Yahoo, or an IMAP mailbox lands as the same message.created event on the same thread shape. You write the attribution once.
  • The math is yours, so you can trust it. You control the split, the minimum sample, and when you're allowed to declare a winner. No vendor deciding "statistical significance" behind a curtain.

The tradeoff: you're writing the experiment harness yourself. If you only ever test subject lines for newsletter blasts and never look at replies, a marketing ESP is genuinely the better tool. This is for the case where an agent is the sender and replies are the outcome you care about.

Before you begin

You'll need a Nylas project and API key, plus one Agent Account to send from. If you've never set one up, the Agent Accounts quickstart walks the whole flow; the supported-endpoints reference lists exactly what a grant can do.

Provision the mailbox. The API auto-creates a default workspace and policy, so a single call is enough:

curl --request POST \
  --url 'https://api.us.nylas.com/v3/connect/custom' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "provider": "nylas",
    "name": "Outreach Bot",
    "settings": { "email": "outreach@yourcompany.com" }
  }'
Enter fullscreen mode Exit fullscreen mode

Same thing from the CLI:

nylas agent account create outreach@yourcompany.com --name "Outreach Bot"
Enter fullscreen mode Exit fullscreen mode

The email has to live on a domain you've registered with Nylas (a custom domain, or a *.nylas.email trial subdomain). Both calls return a grant_id — that's the handle every command below uses. One caveat worth front-loading: a brand-new sending domain warms over roughly four weeks, so don't read a low reply rate in week one as a verdict on your subject line. It might just be your domain reputation.

A note on what's not available, because it shapes the design: Agent Accounts don't support custom metadata or templates. You can't tag a send with key1: variant_a and filter on it later, and you can't render a Nylas-hosted template. Both are fine — they just mean the variant mapping lives in your database, and if a subject's matching body needs HTML, you render it yourself and pass it in the body field. The mailbox stays dumb; your app stays smart.

Step 1: assign a variant per recipient

This is your code, not Nylas's. For each recipient, flip a fair coin and record the assignment before you send. Keeping it deterministic and stored up front is what makes the experiment honest — you decide the split in advance, not after you've peeked at results.

import secrets

SUBJECTS = {
    "A": "Quick question about your trial",
    "B": "Are you stuck on anything?",
}

def assign_variant() -> str:
    # secrets gives a fair, unbiased coin — don't use a time-seeded PRNG here
    return "A" if secrets.randbelow(2) == 0 else "B"
Enter fullscreen mode Exit fullscreen mode

A 50/50 split keeps the two cohorts the same size, which keeps the later comparison simple. The point of randomizing per recipient rather than splitting your list down the middle is to avoid accidentally correlating a variant with some property of the list order — newest signups, say, or a particular acquisition source.

Write the assignment to your own store keyed by recipient. You'll fill in the thread_id in the next step once the send returns it:

-- your DB, your schema
INSERT INTO experiment_sends (recipient, variant, thread_id, replied)
VALUES ('prospect@example.com', 'A', NULL, false);
Enter fullscreen mode Exit fullscreen mode

Step 2: send each variant from the agent mailbox

Now send. The subject comes from the variant; everything else is identical so you're testing the one thing you mean to test. If you omit the from field, Nylas defaults it to the Agent Account's own address — exactly what you want here.

curl --request POST \
  --url 'https://api.us.nylas.com/v3/grants/<NYLAS_GRANT_ID>/messages/send' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "to": [{ "email": "prospect@example.com" }],
    "subject": "Quick question about your trial",
    "body": "Hi there — checking in to see how the trial is going."
  }'
Enter fullscreen mode Exit fullscreen mode

The CLI equivalent, which is the one I actually run when I'm poking at this by hand:

nylas email send <NYLAS_GRANT_ID> \
  --to prospect@example.com \
  --subject "Quick question about your trial" \
  --body "Hi there — checking in to see how the trial is going."
Enter fullscreen mode Exit fullscreen mode

The send response carries a thread_id. Record it against the variant you assigned. This is the load-bearing line of the whole experiment: the thread_id is how a future reply gets credited back to variant A or B. There's no metadata tag doing this for you — the link lives only in your DB.

UPDATE experiment_sends
SET thread_id = 'thread_abc123'
WHERE recipient = 'prospect@example.com';
Enter fullscreen mode Exit fullscreen mode

If your body is HTML, render it yourself and pass the markup straight into body — remember, templates aren't supported on Agent Accounts, so there's no server-side rendering step. One subject string, one matching body, one recorded thread, per recipient. Loop that across your list and the experiment is live.

Step 3: attribute replies back to the winning variant

Replies come in as the standard message.created webhook. Webhooks are application-scoped, not grant-scoped, so you subscribe once at the app level and every grant's events arrive at the same endpoint, each payload carrying a grant_id you filter on.

curl --request POST \
  --url 'https://api.us.nylas.com/v3/webhooks' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "trigger_types": ["message.created"],
    "webhook_url": "https://your-app.example.com/nylas/replies",
    "description": "Subject-line experiment reply attribution"
  }'
Enter fullscreen mode Exit fullscreen mode

From the CLI:

nylas webhook create \
  --url https://your-app.example.com/nylas/replies \
  --triggers message.created \
  --description "Subject-line experiment reply attribution"
Enter fullscreen mode Exit fullscreen mode

When the webhook fires, the attribution logic is short but has three guards you can't skip:

  1. Dedupe on the notification id. Nylas guarantees at-least-once delivery — the same event can arrive up to three times. The top-level notification id is constant across all retries of one event, so that's your dedup key. If you've seen it, drop it.
  2. Ignore the agent's own sends. message.created also fires when your agent sends mail. Check the from address: if it's the Agent Account itself, it's not a reply, skip it. Only an inbound message from someone else counts.
  3. Match the thread_id to your DB. Pull thread_id from the event, look it up in experiment_sends, and credit that row's variant. A reply to a thread you don't have on file isn't part of the experiment. 
def handle_webhook(notification):
    if seen_before(notification["id"]):          # guard 1: dedup on notification id
        return

    msg = notification["data"]["object"]
    sender = msg["from"][0]["email"]
    if sender == AGENT_EMAIL:                     # guard 2: skip the agent's own sends
        return

    thread_id = msg["thread_id"]
    row = db.lookup_send(thread_id)               # guard 3: match to a tracked send
    if row and not row.replied:
        db.mark_replied(thread_id)                # credit row.variant with a reply
Enter fullscreen mode Exit fullscreen mode

One thing the brief is right to flag: don't trust the webhook payload for the message body. The docs disagree on whether the body is inline — the agent-accounts guide says fetch it, the general webhook doc says it's inline unless the message exceeds ~1 MB. They agree on the action, so do that: when you need the actual reply text, fetch the full message with GET /v3/grants/{grant_id}/messages/{message_id}, and branch on the message.created.truncated trigger name for the oversized case. For pure reply attribution you don't even need the body — the thread_id and from are enough.

If you'd rather verify a thread by hand while debugging, the CLI reads it directly:

nylas email threads show <thread-id> <NYLAS_GRANT_ID>
Enter fullscreen mode Exit fullscreen mode

A webhook is the live path, but keep a periodic reconciliation poll too — list recent threads and re-check any tracked send that hasn't been marked replied. The same poll covers you if a grant is offline long enough to miss events. (You'll want a real send timestamp for windowing that poll; date at the shell is fine for the lower bound while you're testing locally.)

Step 4: call the winner — honestly

Here's where most home-grown A/B tests go wrong, so this is the part to slow down on. Reply rate is a low-volume signal. If variant A is 3 replies out of 40 and variant B is 5 out of 40, that gap is almost certainly noise. Two or three replies either way will flip it. Declaring a winner there is worse than not testing at all, because you'll act on a result that isn't real.

A few rules I'd hold yourself to, all of which are your app's job, not Nylas's:

  • Set a minimum sample per variant before you start, and don't look until you hit it. Peeking at results and stopping the moment one variant pulls ahead is the classic way to manufacture a fake winner. Pick the number first, then run to it.
  • For reply rates in the single-digit-percent range, "enough" is bigger than you think — typically hundreds of sends per variant, not dozens, before a few points of difference means anything.
  • Run a real significance test on the two proportions (a two-proportion z-test or a chi-square is plenty) rather than eyeballing the percentages. If you're not comfortable picking a test, the safe default is: bigger sample, larger observed gap, more skepticism. 
# your stats — Nylas has no opinion on significance
a_rate = a_replies / a_sent
b_rate = b_replies / b_sent
winner = "A" if a_rate > b_rate else "B"
# but only report it once both a_sent and b_sent clear your pre-set minimum,
# and only if a significance test on the two proportions clears your threshold.
Enter fullscreen mode Exit fullscreen mode

The discipline is the product here. Nylas delivers the inbound reply events carrying a thread_id, at least once per event — so your app does the dedup on the notification id and your app matches each thread_id back to its variant; Nylas neither dedupes nor attributes for you. Whether you turn those attributed replies into a trustworthy decision is then entirely about the sample-size and significance rules you enforce before peeking.

Guardrails worth knowing

  • Free-plan send cap: 200 messages per account per day. A subject test that needs hundreds of sends per variant may take several days to fill — that's fine, just account for it in your "don't peek early" window.
  • Domain warm-up: the ~4-week reputation ramp on a new domain can depress replies independently of your subject. Don't start the experiment that matters on a cold domain.
  • Inbound retention: the free plan keeps inbox mail for 30 days. Reconcile and persist your reply tallies before that window closes if you're running a long test.
  • Variant mapping is yours to protect. Since there's no metadata tag, a lost or corrupted thread_id mapping means an un-attributable reply. Treat the experiment_sends table as the source of truth and write the thread_id in the same transaction as the send.

What's next

Two subjects, a fair coin, and a thread you already own. That's a real subject-line experiment — no ESP, no list import, no black box deciding the winner for you. The only thing standing between you and a trustworthy result is the sample-size discipline, and that part was always going to be your job.

AI-answer pages for agents

When this post is published, link AI agents and crawlers to the retrieval-ready version on cli.nylas.com:

Top comments (0)