Shadow-test a new AI email agent on live threads with Nylas

You have an email agent that works in a test inbox. It classifies a support request, pulls the relevant context, and drafts a plausible reply. The risky next step is turning it loose on [e-mail]: a new prompt can misunderstand a frustrated customer, a new model can change its format, and a harmless-looking tool change can make the agent write twice.

There is a useful stage between “it passed the fixture” and “it can send customer email”: shadow mode. The candidate agent receives the same live inbound messages as production, reads the same canonical thread context, and records what it would have done. It never sends, moves, labels, or creates a draft in the customer mailbox. A person or the existing production workflow still owns the outcome.

This post builds that boundary around a Nylas Agent Account. Nylas delivers the event and exposes the message/thread data; your application persists the candidate version, proposed action, and comparison result. That division matters: shadow mode is an application rollout feature, not an email-provider switch.

I work on the Nylas CLI, so I use it to inspect the live plumbing and send test events. The implementation below is deliberately database-shaped rather than tied to a queue or model provider.

What shadow mode is and is not

A shadow run consumes real traffic but produces no customer-visible side effect. For each inbound message, it can return a structured proposal such as:

{
  "action": "reply",
  "reason": "The customer asked for a password-reset link.",
  "body": "Hi Dana, here is a fresh password-reset link…",
  "confidence": 0.91
}

That record is useful only when you compare it with an outcome you trust: the production agent's action, a human-approved reply, a support label, or an explicit reviewer verdict. A candidate that produces polished prose but would have answered a billing question instead of escalating it has not passed.

Shadow mode is not a privacy exemption. The model sees live customer content, so run it only where you already have authority to process that content, minimize retention, and exclude attachments and tools unless the experiment specifically needs them. “No send” is an important safety property; it is not the whole threat model.

Start from the existing webhook pipeline

This is a spoke, not a second webhook tutorial. The durable ingest pattern is in Build a webhook-driven email pipeline for your AI agent: verify the raw-body signature, acknowledge quickly, persist a deduplicated job, and do model work in a worker.

Subscribe to message.created as that guide shows. Nylas delivers webhooks at least once, so a redelivery must not create another shadow run. The top-level notification id is the delivery key; the inner data.object.id is the message key. Keep both, because they answer different questions.

Persist the shadow decision outside the mailbox

Do not create a Nylas draft for the candidate reply. Drafts are visible mailbox state, which means a human can mistake an experiment for a proposed response and send it. Shadow output belongs in your own database beside the rollout decision.

This small schema is enough to start:

create table shadow_events (
  notification_id text primary key,
  grant_id text not null,
  message_id text not null,
  thread_id text,
  received_at timestamptz not null default now()
);

create table shadow_runs (
  notification_id text not null references shadow_events,
  candidate_version text not null,
  proposed_action jsonb,
  input_hash text,
  input_snapshot jsonb,
  status text not null default 'pending',
  outcome text not null default 'pending',
  reviewer_verdict text,
  actual_message_id text,
  started_at timestamptz,
  created_at timestamptz not null default now(),
  primary key (notification_id, candidate_version)
);

In the webhook transaction, insert the event and its pending run before enqueueing it. ON CONFLICT DO NOTHING makes a redelivery a no-op:

with event as (
  insert into shadow_events (notification_id, grant_id, message_id, thread_id)
  values ($1, $2, $3, $4)
  on conflict (notification_id) do update
    set notification_id = excluded.notification_id
  returning notification_id
)
insert into shadow_runs (notification_id, candidate_version)
select notification_id, $5 from event
on conflict do nothing
returning notification_id;

Enqueue only when that statement returns a row. The first table makes webhook ingest idempotent. The second lets a deliberate backfill run another candidate version without pretending that prompt version A and model version B are the same experiment. This example retains an encrypted, access-controlled input snapshot for 30 days so a reviewer can replay a candidate; delete it after that window and retain only the hash and labels. After deletion, replay is intentionally impossible.

Fetch canonical context, then run the candidate

The webhook tells you what changed. Fetch the message before you ask the candidate to reason about it, and use thread_id to assemble the conversation context you actually need. The Nylas Message API returns a message body; the Messages list supports filtering by thread_id.

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages/<MESSAGE_ID>" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages?thread_id=<THREAD_ID>" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

nylas email read <message-id>
nylas email threads show <thread-id>

Filter out messages the Agent Account sent itself before creating a run. message.created covers sent and received messages, and an experiment that scores its own output as fresh inbound mail is not measuring customer traffic.

The worker below is illustrative pseudocode. First claim the already-created run; only the worker that changes pending to running may call the candidate. The important part is the absence of messages/send and draft creation:

async function shadowMessage(job, candidate) {
  const claimed = await db.query(`
    update shadow_runs
    set status = 'running', started_at = now()
    where notification_id = $1 and candidate_version = $2 and status = 'pending'
    returning notification_id
  `, [job.notificationId, candidate.version]);
  if (!claimed.rowCount) return;

  const message = await nylas.messages.get(job.grantId, job.messageId);
  if (message.from[0]?.email === AGENT_ADDRESS) {
    await db.query(`update shadow_runs set status = 'ignored'
                    where notification_id = $1 and candidate_version = $2`,
      [job.notificationId, candidate.version]);
    return;
  }

  const messages = await nylas.messages.list({
    grantId: job.grantId,
    threadId: job.threadId,
  });
  const proposal = await candidate.propose({
    message,
    thread: messages.data,
    tools: [],              // shadow mode cannot cause external side effects
    attachments: [],        // add only after a separate attachment review
  });

  await db.query(`
    update shadow_runs
    set status = 'complete', input_hash = $3, input_snapshot = $4,
        proposed_action = $5
    where notification_id = $1 and candidate_version = $2
  `, [
    job.notificationId,
    candidate.version,
    hash(message, messages.data),
    JSON.stringify(redactForEvaluation(message, messages.data)),
    JSON.stringify(proposal),
  ]);
}

On a worker failure, mark the run failed and deliberately reset it to pending before retrying; alert on stale running claims. Do not let a second delivery run beside the first one.

The candidate should return a constrained object, not free-form internal reasoning. For a support agent, reply, escalate, ignore, and request_information are usually enough. Put the proposed body in the object only when a reviewer needs to evaluate wording; otherwise store an action and a compact rationale.

Compare a proposal to reality

The comparison should match the decision you intend to automate. If the rollout target is routing, grade routing; do not let an eloquent draft hide a wrong category. If the target is reply generation, compare the candidate draft to a human-approved reply and give reviewers a way to mark it safe, wrong, incomplete, or unsafe.

Useful initial checks are deliberately boring:

Check	What it catches
Action agreement	Candidate replied when a human escalated, or vice versa.
Unsafe-action rate	Candidate proposed sending, disclosing, or changing something outside its allowed scope.
Duplicate-run rate	Redelivered notifications or worker retries created more than one candidate result.
Time to proposal	A candidate that needs minutes is not ready for a real-time reply loop.
Reviewer override reason	The missing policy, tool, or context that the next version needs.

Do not compare raw generated text with string equality. Two safe replies can use different wording. Start with a reviewer label, then add task-specific checks. For example, confirm that an escalation preserved the thread ID, that a refund case avoided a promise, or that a support reply cited the current policy.

Make the outcome link explicit. When the existing workflow sends or a reviewer decides, write that decision against the same notification and candidate version:

update shadow_runs
set outcome = $3, actual_message_id = $4, reviewer_verdict = $5
where notification_id = $1 and candidate_version = $2;

For a human reply, $4 is the sent message ID; for a routing-only experiment it can be null and $3 is the final queue or escalation action. That gives the review UI a concrete pair: the candidate proposal beside what actually happened.

Promote in stages

Shadow mode earns trust gradually. A practical rollout has three stages:

1. Shadow: record proposals only. No send, no draft, no mailbox mutation.
2. Advisory: write the proposal to your review UI or create a deliberate approval draft after a human asks for one. That draft is an approval artifact, not shadow output; a reviewer owns the send.
3. Limited autonomy: enable a narrow action for a low-risk category, retain the event and proposal records, and keep an immediate kill switch.

The category boundary should be narrow enough to explain in one sentence. “Reset-password questions that match a known account and need no account change” is a rollout boundary. “Support email” is not.

Test the boundary before live traffic

Nylas can send a test webhook to an endpoint, and the CLI can display a mock payload. Test that the ingest path accepts one notification, records one shadow_events row, and never calls the send or drafts path.

nylas webhook test payload message.created
nylas webhook test send https://agent.example.com/webhooks/nylas

Then replay the same payload. Assuming your existing append-only audit log records attempted Nylas operations, this is the smallest database check for the boundary:

select
  (select count(*) from shadow_runs
   where notification_id = '<NOTIFICATION_ID>' and candidate_version = '<VERSION>') = 1
    as one_candidate_run,
  not exists (
    select 1 from audit_log
    where notification_id = '<NOTIFICATION_ID>'
      and operation in ('messages.send', 'drafts.create')
  ) as no_mailbox_mutation;

Both values must be true. This catches duplicate evaluation before it becomes accidental duplicate behavior when you turn the candidate on.

When shadow mode is worth it

For a brand-new mailbox with no business risk, an ephemeral Agent Account is the faster test. For a production workflow where the difference between a good and bad reply matters, shadow mode is the bridge you want. It lets you observe a candidate on the messy mix of real threads, reply styles, missing information, and edge cases that fixtures never reproduce without making the customer your canary.

The durable rule is simple: Nylas carries the message and thread; your application records the experiment; the candidate proposes; a human or the existing workflow acts. Only after those records show that the candidate handles a deliberately narrow category should it earn the right to send.

Where to go next:

• Build a webhook-driven email pipeline for your AI agent: the durable production ingest pattern
• Agent Account supported endpoints: current Agent Account API and webhook support
• Using webhooks with Nylas: webhook setup, retries, mock payloads, and test delivery
• Verify webhook signatures: raw-body HMAC verification