DEV Community: Qasim Muhammad

Email Triage Taxonomies for LLM Classification

Qasim Muhammad — Tue, 16 Jun 2026 21:40:16 +0000

The most important design decision in an email classifier isn't the model — it's the label set, and here's the one I keep coming back to:

You triage email into one of four categories:

URGENT  — production incidents, executive requests; reply within 1 hour
ACTION  — code reviews, meeting follow-ups; reply same day
FYI     — informational, no response needed
NOISE   — newsletters, marketing, automated notifications

From:    {sender}
Subject: {subject}
Snippet: {snippet}

Return ONLY the category name. Nothing else.

That's the working prompt from the Nylas email triage recipe, and almost every line encodes a taxonomy-design lesson worth unpacking. Most people building email agents obsess over model choice and prompt phrasing. The recipe's quiet thesis is that the label set itself does the heavy lifting — get the taxonomy right and a cheap model classifies well; get it wrong and no model saves you.

Why four is the magic number

The recipe states it flatly: four is the right number. Three loses fidelity — everything important collapses into one overloaded bucket and you've built a binary classifier with extra steps. Five and the model starts confusing categories, because the boundaries between adjacent labels get too thin to express in a definition.

Notice what makes these four work. They aren't topics — they're response obligations. URGENT means "reply within the hour," ACTION means "reply today," FYI means "no response needed," NOISE means "archive." Each label maps to exactly one behavior. That's the test I'd apply to any email taxonomy: if two labels lead to the same action, merge them; if one label leads to two different actions depending on content, split it.

The same principle shows up in the sales context. The Agent Accounts overview describes an outreach agent classifying replies as interested / not now / unsubscribe — three labels, because the workflow has exactly three branches: book the meeting, schedule a follow-up, stop emailing. The taxonomy is the decision tree, flattened.

The taxonomy is the dispatch table

The recipe makes this literal. Its entire action loop is a for over unread messages with one branch per label:

for msg in fetch_unread():
    cat = classify(msg)
    if cat in ("URGENT", "ACTION"):
        draft_reply(msg)
    elif cat == "NOISE":
        archive(msg)
    # FYI: do nothing

Two details here do more work than they appear to. First, the agent never sends — URGENT and ACTION produce drafts a human reviews, because the cost of a wrong send (wrong person, wrong tone) is far higher than the friction of one extra click. Second, the loop is idempotent by construction: it only pulls --unread messages, so anything already triaged falls out of the next run without a dedup table. The taxonomy didn't just classify the mail; it shaped the control flow into something you can run from cron unattended.

Drafting also runs at different settings than classification — temperature=0.7 with a "three sentences max" instruction, versus the classifier's temperature=0. Deterministic decisions, natural prose. Same pipeline, two different jobs, and the recipe is blunt that the sentence cap is load-bearing: without it you get drafts that read like a politely overcompensating intern.

Definitions are examples, not adjectives

Look again at the prompt. URGENT isn't defined as "very important and time-sensitive" — it's "production incidents, executive requests." Concrete instances, not abstract qualities. LLMs pattern-match far better against examples than against adjectives, and ambiguous adjectives are where classifiers drift: one model's "important" is another's "routine."

The deadline annotations ("reply within 1 hour," "reply same day") double as tie-breakers. When a message sits between two buckets, the model can ask the implicit question — does this need an answer in an hour or a day? — which is a much sharper discriminator than topical similarity.

Constrain the output like you mean it

Taxonomy design extends to the response format. The recipe runs classification at temperature=0 with max_tokens=10: deterministic output, one category name, no room for an explanatory paragraph. And it still validates — the code checks the response against the four valid strings and falls back to FYI on anything unrecognized, because LLMs occasionally invent a category. An unrecognized label defaulting to "leave it alone" is the safe failure; defaulting to NOISE would silently archive real mail.

Input is constrained just as aggressively: sender, subject, and a 200-character snippet — never the full body. That's enough for over 90% accuracy on this task, and it keeps costs almost ignorable. The recipe's math: GPT-4o-mini runs about $0.15 per million input tokens, a snippet plus prompt is roughly 150 tokens, so 100 emails cost around $0.002. Drafting uses the pricier GPT-4o, but only on the URGENT and ACTION subset — typically under 20% of the inbox — so a heavy 200-message day still costs about a nickel. Cheap classification is what makes the whole pattern viable as a cron job running every fifteen minutes rather than a precious resource you ration. And for mail that can't leave your infrastructure, the recipe's privacy mode swaps in a local Ollama endpoint: Llama 3.1 classifies nearly as well as GPT-4o-mini on this task, though drafting quality drops unless you're running a 70B+ parameter model.

The counterargument: rigid buckets lose information

The pushback I hear: a fixed taxonomy throws away nuance — why not let the model return free-form tags, or scores along multiple axes? Honestly, sometimes that's right. If you're building analytics over a support inbox, richer structure (category plus urgency plus confidence, as the multi-day support patterns do) earns its complexity, since downstream consumers can aggregate it.

But for an agent that has to act, free-form output is a liability. Every distinct output the model can produce is a code path you have to handle, and "handle" means test. Four labels means four branches you can reason about, load-test, and audit. Forty emergent tags means a routing layer that's effectively another model call. The recipe's discipline — closed vocabulary, validated output, one action per label — is what makes the agent's behavior predictable enough to run unattended against a real mailbox. (Agent Accounts, where you'd give that agent its own inbox to triage, are in beta — taxonomy patterns apply to any mailbox.)

One refinement worth planning for: taxonomies are per-inbox, not universal. The recipe notes that engineering inboxes hit URGENT differently than sales inboxes, so the category definitions — not the category count — are where you customize.

Here's an exercise that takes twenty minutes: pull the last 50 messages from whatever inbox your agent will manage and hand-label them with the four buckets above. Wherever you hesitate, write down why — those hesitations are your category boundaries telling you they need sharper example lists. What labels did your inbox force you to add, and what did they map to in terms of action?

Designing Agent Email Addresses That Humans Trust

Qasim Muhammad — Tue, 16 Jun 2026 21:40:13 +0000

You've built the agent, the reply loop works, the demo lands — and then someone asks the question you didn't budget time for: "so what address does it send from?" Suddenly you're staring at noreply-svc-prod2@yourcompany.com realizing that the first thing every recipient sees isn't your prompt engineering. It's the From line.

An agent's email address is an interface. Humans parse it before the subject, mail servers judge it before the body, and spam filters score it before any human sees it at all. Once your agent has a real mailbox — which is what Nylas Agent Accounts (currently in beta) provide — address design becomes a product decision with three layers: the local part, the domain, and the disclosure question.

The local part: role beats persona beats hash

Take three candidate addresses for the same scheduling agent:

scheduling@ — a role. Tells recipients what the mailbox does and implies that emailing it is how you use the service.
jane.ai@ — a persona. Friendlier in a sidebar avatar, but it invites recipients to treat the sender as a colleague, with all the expectations that carries.
bot-7f3a@ — an artifact. Screams "auto-generated," gets mentally filed next to spam, and gives a human nothing to anchor on.

The docs consistently model the first pattern — sales-agent@, support@, scheduling@ appear throughout the Agent Accounts overview — and I think that's right for a reason deeper than convention. A role address makes an honest promise about capability: scheduling@ claims it can schedule, nothing more. A persona address makes an implicit promise of general competence that current agents can't keep. When jane.ai@ fails to understand a simple request, it reads as a person being obtuse; when scheduling@ fails, it reads as a tool hitting its limits. Same failure, different trust damage.

The overview's framing is worth internalizing: an agent identity should be like any other user in your organization — reachable, persistent, accountable. Address design is how that accountability becomes visible.

Claiming the address is one API call

Once a domain is registered and verified, the address itself costs nothing but the decision. Creating the account with your chosen alias is a single request — "provider": "nylas", no OAuth, no refresh token:

curl --request POST \
  --url "https://api.us.nylas.com/v3/connect/custom" \
  --header "Authorization: Bearer <NYLAS_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "provider": "nylas",
    "settings": {
      "email": "scheduling@agents.yourcompany.com"
    }
  }'

The response carries a grant_id that drives the mailbox from then on. The same creation works from the CLI — nylas agent account create scheduling@agents.yourcompany.com — or from the Dashboard, where you pick a registered domain and an alias and the account is live immediately.

That cheapness cuts both ways. It means you can prototype three naming patterns in an afternoon and reply-test them on real recipients. It also means nothing stops you from minting bot-7f3a@ because a UUID was lying around. Remember that the address is the one part of the system whose history you can't transplant: months of threads, recipients' contact lists, allowlists in customer systems, accumulated sender reputation. Provisioning is disposable; identity isn't. Choose the local part like you'll keep it.

The domain: a subdomain is cheap insurance

The provisioning docs recommend a dedicated subdomain for production — agents.yourcompany.com — and the reasoning is reputation isolation: sender reputation accrues per domain, so your agents' sending behavior never contaminates the domain your humans and your marketing depend on. If an experiment misbehaves, the blast radius is the subdomain.

Registering a domain is a one-time event per organization: add it from the Dashboard or API, pick the data center region (US or EU), publish the MX and TXT records, and verification happens automatically once DNS propagates. After that, you mint addresses under it freely. A few patterns from the docs scale this up:

Per-customer domains in multi-tenant apps: scheduling@customer-a.com, scheduling@customer-b.com, each on the customer's own verified domain with its own send quota — 200 messages per account per day on the free plan — and its own sender reputation. One application can manage an unlimited number of registered domains, so this doesn't fragment your infrastructure.
Environment separation: agents.staging.yourcompany.com vs agents.yourcompany.com, so test traffic never touches production reputation.
Reputation sharding: high-volume senders split across sales-a.yourcompany.com, sales-b.yourcompany.com so one domain's deliverability issue doesn't spread.

One caveat on subdomains: don't let isolation drift into obfuscation. agents.yourcompany.com signals "automated, but officially ours." A lookalike domain that hides the relationship signals something much worse.

(For prototyping, trial addresses use the format alias@<your-application>.nylas.email — fine for testing, but the format itself tells recipients nothing about you, which is one more reason to move to your own domain before anything customer-facing.)

The disclosure question

Should the address admit the sender is an agent? My position: the system should disclose, and the address is the cheapest place to do it. Role addresses on an agents subdomain disclose structurally — no recipient who glances at scheduling@agents.acme.com will mistake it for a person, and nobody feels deceived later.

The counterargument deserves a fair hearing: persona addresses measurably improve open and reply rates in outreach, and plenty of teams ship alex@ agents for exactly that reason. Short-term, they're probably right. But that uplift is borrowed against the moment a recipient realizes they've been having a heartfelt back-and-forth with a language model that signed off as Alex. The reply-rate gain is yours; the trust debt lands on your whole domain — the same domain all your other mail comes from. That trade looks worse every quarter as recipients get better at spotting synthetic correspondence.

There's a middle path that keeps most of the warmth: role address, human-adjacent display name. "Sam (Acme Scheduling Assistant)" <scheduling@agents.acme.com> reads friendly in the inbox list and honest on inspection.

Before your next agent ships, write down its From line — display name, local part, domain — and show it to someone outside the team with one question: "who do you think sent this, and what happens if you reply?" If their answer doesn't match your architecture, fix the address before you fix the prompt. Which pattern have you shipped, and did recipients ever call it out?

From Chatbot to Mailbox: Persistent Agent Memory in Threads

Qasim Muhammad — Tue, 16 Jun 2026 21:39:59 +0000

Day 1, 4:02 p.m.: a customer asks your agent a billing question and gets an answer. Day 6, 9:30 a.m.: they reply "actually, that didn't work." If your agent lives in a chat widget, that second message starts from zero — the session died with the tab, the context is gone, and the customer gets to repeat themselves. If your agent lives in a mailbox, the reply arrives inside the conversation, with the full history attached by the protocol itself.

That's the argument in one before/after: chat sessions evaporate; email threads persist. And for agents that work across days rather than minutes, the thread is the most underrated memory substrate available.

The protocol already built your memory layer

Email threading runs on three headers, as the threading docs lay out. Every message carries a globally unique Message-ID. A reply adds In-Reply-To (the ID it's answering) and References (the full chain of IDs, oldest to newest). By the time a thread is five messages deep, References holds five Message-IDs in order — a complete, tamper-evident record of the conversation's shape, maintained by every mail client on earth.

Compare that to what we hand-roll for chatbots: session stores, conversation tables, context windows we serialize and rehydrate. Email gives you the equivalent for free, federated across organizations, and — this is the part I find most compelling — human-auditable. Anyone with mailbox access can read exactly what the agent's memory contains, because the memory is the correspondence itself. No vector store inspection tools required.

With Nylas Agent Accounts (in beta), the agent owns the mailbox where this accrues, and you never parse headers by hand. The Threads API groups messages by their header chain; each thread object gives you ordered message_ids, participants, and activity timestamps. When a reply fires message.created, the payload includes a thread_id — fetch the thread, walk its messages, and the agent has its full conversational past before deciding anything. Tip from the docs: fields=include_basic_headers fetches just the three threading headers when you need them raw, skipping a header payload that's often larger than the message body.

Don't reconstruct memory from subject lines

One tempting shortcut deserves a warning. Plenty of implementations match replies by subject: if it starts with Re: and contains the original subject, it must be a reply. The threading docs list exactly how that breaks. Recipients edit subjects — "Q3 budget review" comes back as "Re: Q3 budget review — updated numbers attached." Two prospects receive the same "Following up on your demo request," and a reply to either matches both. A forwarded thread keeps its subject while losing its conversational context entirely. Headers reference specific Message-IDs, not human-editable text; match on them first, and treat subject matching as a last-resort fallback for ancient mail clients.

The write side is symmetric and just as automatic: pass reply_to_message_id on the send and Nylas populates In-Reply-To and References for you, so the reply threads correctly in every recipient's client. Better still, the memory works across access paths. If the agent sends through the API and a human supervisor later replies from Apple Mail over IMAP, everything stays in one thread, because grouping follows the header chain rather than the send mechanism. One transcript, multiple writers.

Threads remember what was said — not what you were doing

Now the honest limitation, which the docs are upfront about: the thread is episodic memory, not working memory. It knows the words exchanged. It doesn't know which task the agent was on, which workflow step, which ticket. That mapping lives in your application:

// On outbound: bind the thread to internal state.
threadState.set(sentMessage.threadId, {
  taskId: currentTask.id,
  step: "awaiting_reply",
});

// On inbound webhook: restore context, or treat as new.
const context = threadState.get(inboundMessage.threadId);
if (context) await resumeTask(context.taskId, inboundMessage);
else await triageNewMessage(inboundMessage);

In production that map belongs in Postgres or Redis, not memory — conversations span days, and an in-memory map doesn't survive a deploy. So the architecture is two layers: the thread holds the durable transcript, your store holds a thin pointer from thread_id to agent state. The heavy content lives in the mailbox; you persist only the index.

Dormancy is a feature you have to design for

Persistence cuts both ways: threads come back from the dead. The multi-day support agent recipe treats revival as a first-class case with concrete policies worth stealing:

Reclassify on every reply. A thread that opened as a "general" question can become a billing dispute by message two. The recipe re-runs classification on the full transcript each turn and only auto-replies above a 0.85 confidence threshold.
Cap the loop. After 6 turns, escalate to a human — an agent still going back and forth at turn seven isn't converging.
Treat long silence as a state change. If a thread has been quiet for more than 168 hours and the customer suddenly returns, the recipe escalates rather than letting the agent resume as if nothing happened. Context that old deserves human eyes.
Watch the escalation rate. The recipe's operational rule: if more than 40–50% of tickets end up with a human, the agent isn't pulling its weight — tune the knowledge base or narrow the categories it handles rather than lowering the confidence bar.

That last one captures the design mindset: a chatbot architecture asks "is the session alive?" A mailbox architecture asks "what does this silence mean?" — a genuinely richer question.

Where the chat people have a point

The fair counterargument: email is slow and threads are noisy. Latency is measured in minutes to days, quoted text and signatures pollute the transcript you feed the model, and a CC'd third party can wander into the "memory" mid-conversation. For interactive flows — debugging a config live, navigating a UI — chat's immediacy wins, and nothing here argues otherwise.

But most agent work that matters commercially isn't interactive. Support, scheduling, procurement, follow-ups — these are inherently asynchronous, multi-day processes, and forcing them into session-shaped memory is why so many "AI assistants" forget you between Tuesday and Friday. Match the memory model to the conversation's natural tempo.

A concrete way to test the idea: take one workflow where your agent currently loses context between sessions, give it a mailbox, and store nothing yourself except the thread_id → state mapping. Run it for two weeks. My bet is the surprising part won't be the persistence — it'll be how much easier debugging becomes when you can read your agent's memory in an email client.

Scheduling Without a Human Calendar

Qasim Muhammad — Tue, 16 Jun 2026 21:39:56 +0000

A human assistant borrows the boss's calendar; a scheduling bot owns its own. That one difference dissolves most of what makes calendar automation miserable.

The borrowed-calendar model is how nearly every scheduling tool works today: connect to a person's Google or Microsoft account via OAuth, request calendar scopes, and act on their behalf. It works, but the seams show everywhere. The human's calendar fills with bookings the bot manages. Delegation permissions vary by provider and admin policy. Tokens expire when the person changes their password or leaves the company. And the bot has no address of its own — every invite, every confirmation email, appears to come from a person who didn't write it.

Nylas Agent Accounts (currently in beta) invert this. Each account is a real mailbox and a real calendar, provisioned automatically, owned by your application. From a participant's perspective there's nothing special about it — it's just another attendee on the invite.

What "owning availability" actually changes

When the bot's calendar is its own, availability stops being a permissions question and becomes a query. The agent calls the free/busy endpoint against its own primary calendar, gets back busy blocks for a time window, and proposes open slots. No delegation, no scopes negotiation, no "the admin needs to approve calendar sharing for service accounts."

The scheduling-agent tutorial wires the whole loop: a meeting request lands at scheduling@agents.yourcompany.com, a message.created webhook fires, an LLM parses duration and timezone, the agent checks its own free/busy and replies with 3 candidate slots. When the human picks one, the agent creates the event:

curl --request POST \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/events?calendar_id=primary&notify_participants=true" \
  --header "Authorization: Bearer <NYLAS_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "title": "Product demo",
    "when": { "start_time": 1744387200, "end_time": 1744390800 },
    "participants": [{ "email": "alice@example.com" }]
  }'

With notify_participants=true, an invitation goes out from the agent's own address as a standard ICS request. Google Calendar, Microsoft 365, and Apple Calendar all render it as a normal invite, because under the hood it is one — plain iCalendar, the same protocol humans have used for decades.

RSVPs flow back without polling

The return path is the part borrowed-calendar setups handle worst, and here it's automatic. When Alice clicks Yes in Gmail, Google sends the response to the agent's mailbox, the platform parses it, updates the event's participants[].status, and fires an event.updated webhook. The agent learns who accepted without reading a single email.

The same machinery runs in reverse when the agent is the invitee. Someone adds the agent's address to their meeting; the invitation hits the mailbox, a matching event appears on the agent's calendar with its status as noreply, and event.created fires. The agent responds through the send-rsvp endpoint with one of 3 statuses — yes, no, or maybe — and the organizer sees it accepted like any other attendee. A plain reply email won't update anyone's calendar; the endpoint exists because RSVP is calendar protocol, not prose.

Changes propagate the same way: PUT /events/{id} updates the time or title on every participant's calendar, wherever they're looking at it, and DELETE /events/{id} removes it everywhere.

The small print that saves you a debugging session

A few mechanics from the calendar docs that aren't obvious until they bite:

Two webhooks fire for one invite. Because the agent is always a mailbox too, an inbound invitation triggers event.created and message.created for the invite email itself. Pick one to drive your logic and deliberately ignore the other, or you'll process every meeting twice.
notify_participants is a sharp tool. Every create, update, and delete with notify_participants=true sends real email. Pass false for bulk backfill or pre-staged events the agent will announce later — but not when cancelling, because deleting an event without notification leaves the meeting sitting on everyone's calendar.
The agent has no time zone. A human calendar carries a default; an Agent Account doesn't. Pass timezone explicitly on create, or stick to epoch start_time/end_time values.
One agent can run several calendars. The primary is provisioned automatically and can't be deleted while other calendars exist, and you can create more up to your plan's cap — a sales-calls calendar and an internal calendar on the same agent keeps concerns separate.

The honest limits

Two things the dedicated-identity model doesn't fix, and one operational note.

Negotiation isn't an endpoint. Counter-proposing a time isn't first-class today — the documented pattern is to RSVP no or maybe and reply with an alternative by email, letting the organizer recreate the event. If your flow is negotiation-heavy (propose slots to many people, collect picks, book the winner), the docs point you at Scheduler, which is built for round-trips and works with Agent Accounts. The Events API shines when the agent already knows the time.

Parsing is still the hard part. Owning the calendar removes the permissions problem, not the language problem. "Sometime late next week, ideally afternoon, I'm in Lisbon" still needs an LLM, and the tutorial is candid that wrong intent extraction creates real calendar chaos — it recommends human confirmation for first-time senders and high-value meetings.

Quotas apply. Every proposal and confirmation email counts against the account's send cap — 200 messages per account per day on the free plan — and for a busy agent it's worth setting an explicit policy before launch rather than discovering the ceiling in production.

The counterargument I'd actually respect: if your bot schedules on behalf of a specific person — managing a real executive's actual calendar — then borrowing that calendar is correct, because the availability that matters is theirs. Dedicated identity wins when the bot is the service itself: demo bookings, interview coordination, support callbacks. Pick the model that matches whose time is being scheduled.

If you've got an Agent Account already, here's a 15-minute experiment: create an event with yourself as participant and notify_participants=true, accept it from your phone, and watch the event.updated webhook arrive with your status change. Once you've seen the loop close end to end, the architecture mostly designs itself.

Mailboxes as Cattle: Ephemeral Email Infrastructure

Qasim Muhammad — Tue, 16 Jun 2026 21:39:53 +0000

When was the last time you deleted an email account on purpose?

For most teams the answer is never, and that tells you something. We treat mailboxes the way we treated servers in 2008: hand-built, carefully named, kept alive indefinitely because recreating one is painful. They're pets. Meanwhile every other piece of our infrastructure — compute, queues, databases — became cattle: numbered, provisioned by code, destroyed without sentiment when the job ends.

Email is finally catching up. With Nylas Agent Accounts (in beta), a mailbox is created with one call and destroyed with another, and that symmetry is the whole point.

The full lifecycle in two commands

Provisioning, from the CLI:

nylas agent account create signup-agent@agents.yourdomain.com

Or via POST /v3/connect/custom with "provider": "nylas" — no OAuth, no refresh token, just an address on a domain you've registered. Teardown is equally unceremonious:

nylas agent account delete signup-agent@agents.yourdomain.com --yes

(The API equivalent is a DELETE on the grant.) The signup automation recipe treats this as a loop: provision a fresh inbox, point a third-party signup form at it, catch the verification email through a message.created webhook, follow the confirmation link, delete the grant. No human inbox involved at any step, and nothing left behind.

The middle of that loop is about twenty lines of webhook handler, and the recipe's version filters hard before acting — right grant, right sender, right URL shape:

const { grant_id, id: messageId, from } = event.data.object;
if (grant_id !== AGENT_GRANT_ID) return;

const sender = from[0]?.email ?? "";
if (!sender.endsWith("@saas-you-care-about.example.com")) return;

const match =
  /https:\/\/saas-you-care-about\.example\.com\/confirm\?token=[^"\s<]+/.exec(
    message.body,
  );
if (match) await fetch(match[0]);

Nylas fires message.created within a second or two of mail arriving, so the whole signup round-trip typically finishes before a human would have found the email.

The one-time pet in this story is the domain, not the mailbox. You register a domain once per organization — picking a US or EU data center region — publish the MX and TXT records, and then mint as many addresses under it as your plan allows. Trial *.nylas.email subdomains skip even that for prototyping. See provisioning and domains for the flow.

What cattle-class mailboxes are good for

Per-run test inboxes. E2E tests that assert on real email delivery have always been awkward — shared test accounts accumulate state, and yesterday's run pollutes today's assertions. A fresh mailbox per CI run resets the world. The signup recipe's advice applies directly: if you're running a large test matrix, provision multiple grants rather than reusing one, because quotas are per account — 200 messages per account per day on the free plan.

Per-workflow identities. A research agent that needs a developer account on a data source, a QA bot that registers for a SaaS every run, a purchasing agent that needs a marketplace profile. Each gets an address, does its job, and gets reaped.

Per-environment separation. The provisioning docs describe running agents.staging.yourcompany.com and agents.yourcompany.com in the same application, so staging traffic never touches the production domain's reputation.

Per-customer pools. Sender-reputation isolation by splitting volume across sales-a.yourcompany.com, sales-b.yourcompany.com, and so on — a deliverability problem on one domain doesn't contaminate the rest.

Cattle still need fences

Disposable doesn't mean unmanaged — arguably the opposite. Two practices from the docs are worth treating as non-negotiable.

First, reap your herd. The recipe is blunt: don't ship per-run agents without teardown, because inactive grants accumulate. Delete on completion or failure — the failure path is where orphans come from. If your CI provisions a mailbox, the cleanup step belongs in a finally block, not at the happy-path end.

Second, fence the inbox. A disposable address that leaks into a spam list keeps receiving junk until you delete it. Pair an allowlist of expected sender domains with a block rule for everything else, so the inbox only accepts mail from the service you're actually testing against. And don't trust the first message that arrives — some services send a "Welcome" email before the verification email, so match the sender and the expected URL pattern before your code clicks anything.

Third, fence the herd, not the animals. Per-account configuration defeats the point of cattle. Workspaces fix that: pass a workspace_id when you create the grant — or let auto_group place accounts into a workspace by matching their domain — and attach a policy there once. Every disposable mailbox inherits the same send quota, retention window, and block rules with zero per-account setup, and moving an account later is a single PATCH on the grant.

One more option if humans occasionally need to peek inside a longer-lived account: set an app_password at creation (18–40 printable ASCII characters, with an uppercase letter, a lowercase letter, and a digit) and connect a normal IMAP client. Nylas stores it as a bcrypt hash, so you can't retrieve it later — only reset it. Skip it for true cattle; protocol access stays disabled by default.

The counterargument: identity is the one thing you can't dispose

Here's where the metaphor honestly breaks down. A container's identity doesn't matter; an email address's identity is the product in some workflows. An address that customers know, that's whitelisted in their systems, that has months of threading history — that's a pet, and it should be. Reputation also accrues over time: a brand-new address sending cold outreach behaves differently from an established one.

So the pets-vs-cattle question for email isn't "which model is right" but "which mailboxes are which." My rule of thumb: if a human would notice the address changing, it's a pet. If only your code ever reads it, it's cattle — and keeping cattle alive out of habit is just unmanaged inventory.

Audit your own herd this week: list every automation-owned mailbox your team has, and ask of each one, "if this were deleted tonight, what breaks?" If the answer is "nothing," you've found your first candidate for a teardown script. How many pets are you feeding that should've been cattle?

The 5-Minute Mailbox

Qasim Muhammad — Tue, 16 Jun 2026 21:38:34 +0000

The email mailbox just became an API resource, and that matters far more than the setup time it saves.

For most of software history, a real email address — one that sends, receives, and threads — was an artifact of IT process. Someone created it in an admin console, someone else configured the client, and your application got access through OAuth consent screens and refresh tokens borrowed from a human. Compare that to how you get a database, a queue, or a TLS cert today: one API call, one ID back, done.

Nylas Agent Accounts (currently in beta) close that gap. The quickstart goes from API key to a sending-and-receiving mailbox in under 5 minutes, and the provisioning step is a single request:

curl --request POST \
  --url "https://api.us.nylas.com/v3/connect/custom" \
  --header "Authorization: Bearer <NYLAS_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "provider": "nylas",
    "settings": {
      "email": "test@your-application.nylas.email"
    }
  }'

No refresh token, no OAuth dance — unlike OAuth providers, the "nylas" provider needs only an email address on a registered domain. The response contains a grant_id, and that one ID drives everything else: messages, drafts, threads, folders, attachments, calendar, webhooks. There are actually three ways to create an account — this API call, the Dashboard, or a single CLI command (nylas agent account create) — but they all end at the same place: a live mailbox.

Why 5 minutes is a threshold, not a convenience

Provisioning time isn't a linear cost. There's a threshold below which a resource changes category — from "thing you request" to "thing your code creates." Virtual machines crossed it with cloud APIs and we got autoscaling. TLS certs crossed it with ACME and we got HTTPS-by-default. Mailboxes crossing it means email addresses stop being scarce, pre-planned identities and start being something a program allocates when it needs one.

What does that look like in practice?

System mailboxes without ceremony. A support@ or scheduling@ address your app owns end-to-end — no consent screen, no integration that breaks when an employee offboards. The docs' framing is exact: a mailbox your application owns, not borrows.

Ephemeral inboxes. Spin up an address for a test run, a workflow, or a single customer interaction, then delete the grant. A concrete version: your CI pipeline creates e2e-run-4821@yourapp.nylas.email, your signup flow sends its verification email there, the test asserts on the real delivery — not a mock — and teardown is one delete on the grant. When creation costs one HTTP call, teardown becomes a reasonable habit instead of a cleanup chore you defer forever.

Identity for agents. An AI agent with its own address can send, receive replies in-thread, and even RSVP to calendar invites — every account ships with a primary calendar that speaks standard iCalendar, so Google Calendar, Microsoft 365, and Apple Calendar treat it as a normal participant. The send-rsvp endpoint means the agent's "yes" shows up like anyone else's.

The receive side is the half that's actually new

Plenty of services let you send email programmatically in minutes. The unusual part here is that the mailbox receives, and inbound mail fires the standard message.created webhook — identical in shape to the same event for a Gmail or Outlook grant. Register a webhook once:

nylas webhook create \
  --url https://yourapp.example.com/webhooks/nylas \
  --triggers message.created

…and replies land in your handler seconds after they arrive. If you'd rather not run webhook infrastructure yet, polling GET /v3/grants/{grant_id}/messages works too. Either way, the loop closes: your code can ask a question over email and react to the answer.

The send half deserves a sentence as well. Outbound goes through the same POST /v3/grants/{grant_id}/messages/send endpoint used for any connected grant, and the recipient sees a normal message from the agent's own address — no "sent via" branding, no relay footer. Outbound messages are capped at 40 MB total. The whole point of the design is that an Agent Account is just another grant: the message.created payload is identical in shape to the one a Gmail or Outlook grant produces, and you branch on the grant's provider field ("nylas") only if you need to tell them apart. Everything you've already built — pagination, attachment downloads, thread fetches — works unchanged.

The honest caveats

Instant shouldn't mean careless, and there are two qualifiers worth stating plainly.

First, the 5-minute path uses a *.nylas.email trial subdomain — instant because there's no DNS involved. For production you'll want your own domain, which means publishing two kinds of DNS records (MX for inbound routing, TXT for ownership and SPF/DKIM) and waiting for propagation before verification completes. That's a one-time cost per domain, not per mailbox, but it's real. The docs recommend a dedicated subdomain like agents.yourcompany.com so your agents' sender reputation stays isolated from your primary domain — see provisioning and domains.

Second, cheap creation makes governance more important, not less. The defaults are sane — the free plan caps sending at 200 messages per account per day, with 3 GB of storage per organization and 30-day inbox retention — but defaults aren't a strategy. Every account you create without an explicit workspace_id lands in your application's default workspace, so attaching a policy and rules there governs all of your unassigned mailboxes in one move. The policies and rules system exists for exactly that, and it's worth configuring before your mailbox count gets interesting.

You could also argue most apps don't need a receiving mailbox at all — transactional send-only covers password resets fine. True. But the moment your product wants a conversation rather than a notification — support, scheduling, anything an agent does — the send-only model runs out, and historically the next step was a painful jump to "go provision accounts with an email admin." Now the next step is a POST request.

Try the threshold test yourself: start a timer, follow the quickstart, and stop when you've sent yourself an email from the new address and seen the reply come back through a webhook. If you beat 5 minutes, ask the more interesting question: what would you build if mailboxes were as disposable as containers?

Rate-Limit Your Own Agent Before Someone Else Does

Qasim Muhammad — Tue, 16 Jun 2026 21:38:17 +0000

0.1%. That's the complaint rate that puts an email-sending account under review on Nylas Agent Accounts — one spam report per thousand sends. At 0.5%, sending is paused outright. For bounces, the review threshold is 5% and the pause kicks in at 10%. These aren't suggestions; they're enforced by the platform, and a pause doesn't clear itself on a timer — you have to contact support with evidence of a fix.

Here's my position: those numbers shouldn't be your rate limit. They should be your last line of defense, behind a stricter limit you set yourself. Rate-limit your own agent before someone else does it for you.

An LLM loop has no natural stopping point

Traditional email code sends when a human or a cron job tells it to. An autonomous agent sends when a model decides to, and models inside feedback loops make weird decisions. A reply triggers a webhook, the webhook triggers a reply, and a benign bug becomes a thousand sends before lunch. Nothing in the model's reasoning says "this is my 400th message this hour, that seems off." That awareness has to live in infrastructure.

Agent Accounts (in beta) bake the infrastructure in through policies. A policy bundles daily send quotas, storage caps, retention windows, and spam settings, and applies to every account in a workspace. Without one, an account runs at your billing plan's maximums — 200 messages per account per day on the free plan — which is exactly what you don't want for an experiment that might loop. Every limit on a policy is optional; omit one and it defaults to the plan maximum, ask for more than the plan allows and the API rejects it.

Quotas are a statement about expected behavior

The useful mental shift: a self-imposed quota isn't throttling, it's an assertion. "This support agent should never need more than 150 sends a day. If it asks for number 151, something upstream is wrong." That's the same logic as a circuit breaker in a service mesh — you're not limiting capacity, you're encoding an expectation so violations become visible instead of expensive.

Policies let you encode different expectations per agent archetype. A prototype gets a tight quota; a production sales agent gets a higher one. The docs explicitly suggest separate workspaces per archetype, because a triage agent and an outreach agent have completely different send profiles.

Outbound rules go a step further than volume — they constrain direction. A rule with trigger: "outbound" evaluates before the message reaches the provider, and a block action rejects the send with a 403:

curl --request POST \
  --url "https://api.us.nylas.com/v3/rules" \
  --header "Authorization: Bearer <NYLAS_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "Block outbound to example.net",
    "trigger": "outbound",
    "match": {
      "conditions": [
        { "field": "recipient.domain", "operator": "is", "value": "example.net" }
      ]
    },
    "actions": [{ "type": "block" }]
  }'

The recipient.* fields match any recipient, including BCC and SMTP envelope recipients — so an agent can't smuggle a send past the rule by hiding the address. You can also match outbound.type (compose vs reply) to, say, let an agent reply freely but block it from starting brand-new threads.

Watch the same telemetry the platform watches

The bounce and complaint rates that trigger pauses are computed from events you can subscribe to: message.transactional.delivered, message.transactional.bounced, message.transactional.complaint, and message.transactional.rejected — four webhook triggers that are your only real-time window into those rates. The docs' advice is blunt: wire them up and pause your own outbound logic when bounces or complaints climb. You'll see the problem in your own telemetry before the platform tells you about it, and "we paused ourselves" is a much better incident report than "we got paused."

It also helps to know what's actually being counted. Bounce rate only counts hard bounces — addresses that don't exist — divided by a recent representative send volume; soft bounces from full mailboxes or greylisting don't touch it, and healthy is under 2%. Complaint rate counts recipients clicking Mark this email as spam or dragging your mail to junk, measured only across domains that send complaint feedback. That's why 0.1% is so easy to hit at low volume: a handful of annoyed recipients in a 2,000-send week puts the account under review.

The error responses are worth knowing too. A reputation pause surfaces as a 400 on send; a per-account or per-domain rate limit returns 429 (back off and retry); an abuse restriction returns 403 with send blocked by abuse restriction. That last one can be scoped to a single sender address, a domain and its subdomains, a grant, or the entire application — and an application-level restriction stops every Agent Account under the app, not just the one that misbehaved. If your agent treats all send failures as retryable, it will hammer a paused account and learn nothing.

Circuit breakers need receipts

Two details make the rule layer trustworthy enough to bet on. First, evaluation fails closed: if a block rule can't be evaluated because of a transient infrastructure error — say, a list lookup failure during in_list matching — Nylas blocks the message rather than letting it through. The failure is surfaced as retryable: an API send returns 503 instead of 403, and inbound SMTP answers with a 451 tempfail so the sending server retries instead of bouncing. A safety mechanism that silently disables itself under load isn't a safety mechanism.

Second, every evaluation writes an audit record. GET /v3/grants/{grant_id}/rule-evaluations lists, most recent first, which rules matched, what actions were applied, and the normalized sender and recipient data that was considered. When a block happened because evaluation errored rather than matched, the record carries blocked_by_evaluation_error: true. So when your agent's send comes back 403 at 2 a.m., "why was this blocked?" is one API call, not an archaeology project. A circuit breaker without observability is just a mystery outage.

The counterargument: limits break legitimate bursts

The honest objection is that real workloads spike. A support agent during an outage might legitimately need 5x its normal volume, and a hard quota turns your safety net into an availability incident. That's true — if the quota is a dead end.

So don't make it a dead end. Make hitting the quota an escalation path: alert a human, queue the overflow, require an approval to raise the cap. The failure mode of a too-tight quota is a Slack ping and an hour of delayed email. The failure mode of no quota is a 10% bounce rate, a platform-level pause that requires a support ticket to lift, and a sender reputation you rebuild over weeks. Those aren't symmetric risks.

There's also a softer dial worth knowing: policies expose spam_sensitivity from 0.1 to 5.0 for inbound filtering. Inbound hygiene matters for outbound health, because agents that reply to junk generate complaints.

Concrete next step: before your agent's next deploy, create one policy with a daily quota at roughly 2x the agent's observed peak, attach it to the workspace, and subscribe to the four message.transactional.* triggers. Then deliberately make your agent hit the quota in staging and check that your alerting fires. If it doesn't, you've found the gap while it's still cheap.

Common Pitfalls Building Email Agents (and Fixes)

Qasim Muhammad — Tue, 16 Jun 2026 21:18:04 +0000

A team ships their first email agent on a Thursday. Demo went great, handler's deployed, webhook's registered. Friday morning the on-call wakes up to an inbox where the agent has been enthusiastically replying to its own replies all night, a customer who received the same answer three times, and a thread in Gmail that's shattered into five separate conversations. None of it was an exotic failure — every one of these is a known pitfall with a known fix, documented in the Nylas Agent Accounts cookbook (the product's in beta; the mistakes are timeless).

Here are the nine I'd check before any launch.

1. The agent replies to itself

The message.created webhook fires for outbound messages too — when your agent sends a reply via the API, that sent message triggers the same event as inbound mail. Skip this check and you've built a perpetual motion machine: reply, webhook, reply.

Fix: filter the agent's own address at the very top of the handler, before any other logic.

const sender = msg.from?.[0]?.email;
if (sender === AGENT_EMAIL) return;

2. No webhook deduplication

Delivery is at-least-once. If your endpoint doesn't return 200 fast enough, or the network hiccups, the same message.created notification arrives again — and a naive handler replies twice. The dedup recipe calls this the most common source of duplicates.

Fix: an atomic check-and-set on the message ID before processing — INSERT ... ON CONFLICT DO NOTHING in Postgres, SET id 1 NX EX 86400 in Redis. Give records a TTL of 24–48 hours so late redeliveries still get caught without the table growing forever.

3. Dedup without locking

Two concurrent workers (Lambda instances, ECS tasks) can race past the check-and-set in the same millisecond and both generate a reply. Dedup catches the same event delivered twice; it can't catch the same event processed simultaneously.

Fix: a per-thread lock with a 30-second TTL so a crashed worker self-releases — and a double-check inside the lock that inspects the thread's latest message and bails if the agent already replied. You need dedup and locking; they cover different failure modes.

4. Trusting the webhook payload for the message body

The webhook carries summary fields — subject, from, snippet — not the full body. Worse, if a body exceeds roughly 1 MB, the event type becomes message.created.truncated and the body is omitted entirely. Agents that parse the payload directly work in testing and fail on real-world mail.

Fix: always fetch the full message from the API using the ID in the payload, as the reply-handling recipe does, and handle the truncated event type explicitly.

5. Replies that don't thread

Send a "reply" as a fresh message and it lands as a disconnected email in the recipient's client — no quoted context, no conversation grouping. Multiply by a few turns and the customer is hunting through five fragments of one discussion.

Fix: pass reply_to_message_id on every reply. That makes the platform set the In-Reply-To and References headers so the message threads correctly in Gmail, Outlook, and the agent's own mailbox. Match incoming replies by thread_id, never by subject line — subjects get edited, and two different threads can share one.

await nylas.messages.send({
  identifier: AGENT_GRANT_ID,
  requestBody: {
    replyToMessageId: msg.id,
    to: [{ email: sender }],
    body: replyBody,
  },
});

6. Replying instantly to every message

Humans send corrections. A recipient fires off a reply, spots a mistake, and sends a follow-up fifteen seconds later — and your agent has already answered the first message, so now it answers the second too, and the conversation forks.

Fix: a 30–60 second cooldown before responding in active threads, batching consecutive inbound messages into one considered reply.

7. No outbound circuit breaker

Even with dedup, locking, and self-filtering, a logic bug can still produce a reply storm — and an autonomous sender fails at machine speed. This is the safety net the dedup recipe says not to ship without.

Fix: a per-thread send budget. If the agent has sent 3 or more messages on one thread within 5 minutes, stop sending and escalate to a human. A rate limit triggering is a page; a runaway agent is an apology tour.

8. Letting junk wake the agent

Spam, bounce-backs, and out-of-office auto-replies all fire message.created. If every one of them reaches your LLM, you're paying inference costs to reason about garbage — and risking the agent answering it.

Fix: push filtering below your application using rules. A block rule rejects known-bad senders at the SMTP level so your code never sees the message; assign_to_folder routes automated notifications away from the inbox so your handler can skip folders the agent shouldn't answer. Rules run in priority order (0–1000, lower first), so put specific matches before broad contains rules — the first matching block is terminal.

9. Treating a blocked send as a retryable error

If your workspace has outbound rules, a send matching a block rule returns 403 — and no retry will ever deliver it, because the rule rejected it before the provider was involved. An agent with generic retry logic will hammer that send forever and report a flaky network.

Fix: treat 403 on send as terminal. Log it, then query GET /v3/grants/{grant_id}/rule-evaluations to see exactly which rule matched and what data was evaluated — that endpoint is the fastest answer to "why didn't this send?"

There's one nuance worth encoding in your error handler. Rule evaluation fails closed: if a block rule can't be evaluated because of a transient infrastructure problem (say, a list lookup failure during in_list matching), the send is blocked anyway — but it comes back as a 503, not a 403, and the audit record carries blocked_by_evaluation_error: true. So the rule is simple: retry 503, never retry 403. Conflating the two is how agents either give up on deliverable mail or hammer undeliverable mail.

The pattern across all nine: email agents fail at the seams between at-least-once infrastructure and autonomous action, not in the LLM prompt. The fixes are boring — a filter, a lock, a cap, a rule — and that's the point. Boring is what you want standing between a language model and a real human's inbox.

Turn this into a pre-launch checklist: nine items, and your load test should specifically exercise #2 and #3 by firing duplicate webhooks from concurrent connections. Which of these has bitten you in production — and was the fix on this list?

Idempotency Lessons From an Email Agent

Qasim Muhammad — Tue, 16 Jun 2026 17:17:54 +0000

A customer emails your support agent at 9:14 a.m. At 9:15 they get a helpful reply. At 9:16 they get the same reply again, word for word. Nothing crashed. No exception was thrown. Your agent just did exactly what it was told — twice.

I think email agents are the best teacher of idempotency I've seen in years, because the failure mode is so visceral. A duplicate database row is invisible. A duplicate email lands in a human's inbox and makes your product look broken. Building a reply loop on Nylas Agent Accounts (currently in beta) forced me to internalize lessons that apply to any event-driven system, not just email.

Lesson 1: at-least-once is the honest contract

The instinct is to blame the platform: "why did I get the same webhook twice?" But at-least-once delivery is the only honest guarantee a webhook system can make. Per the duplicate-reply docs, if your endpoint doesn't return 200 fast enough, or a transient network blip eats the response, the message.created notification gets delivered again. The alternative — exactly-once — would mean the platform silently drops events whenever it's unsure, and a dropped event is worse than a repeated one.

So duplicates aren't a bug to report. They're a contract to design for. The fix is an atomic check-and-set keyed on the message ID:

const alreadyProcessed = await db.processedMessages.setIfAbsent(messageId, {
  receivedAt: Date.now(),
});
if (alreadyProcessed) return;
await handleMessage(event.data.object);

The atomicity matters more than the storage. In Postgres that's INSERT ... ON CONFLICT DO NOTHING; in Redis it's SET messageId 1 NX EX 86400. A read-then-write sequence reintroduces the race you're trying to close. And give the records a TTL — 24 to 48 hours covers redeliveries without growing the table forever. After that window, a webhook for the same message ID is almost certainly a bug in your own system, not a redelivery, and you want it to surface.

There's a quieter corollary: acknowledge before you act. The docs' example handler calls res.status(200).end() as its first line and only then starts processing. Every second your endpoint spends on an LLM call before responding is a second in which the platform may decide the delivery failed and queue a retry. You can't eliminate redeliveries, but you can stop manufacturing them.

Lesson 2: dedup and locking solve different problems

Here's the part most people miss. Deduplication catches the same event delivered twice. It does nothing about the same event processed twice concurrently. If your handler runs on Lambda or multiple worker processes, two instances can blow past the check-and-set within the same millisecond window.

The docs recommend a per-thread lock with a 30-second TTL, so a crashed worker releases automatically. And inside the lock, a double-check against ground truth: fetch the thread, look at latestDraftOrMessage, and bail if the from address is the agent's own. Between the webhook arriving and your lock being acquired, another worker may have finished the whole job — the thread itself is the only record that can't lie about it.

That layered structure — dedup, then lock, then verify state — generalizes. Idempotency isn't one mechanism. It's a stack of cheap checks, each catching what the previous one can't.

Lesson 3: the best coordination is no coordination

The thorniest duplicates don't come from infrastructure at all. They come from two actors watching the same inbox — two agents, or an agent and a human, both deciding the same message needs a reply. You can't dedup your way out of that; it's not a duplicate event, it's a coordination problem.

The cleanest fix is architectural: one agent, one inbox. Agent Accounts make that nearly free, since each agent gets its own address and its own webhook stream — sales-agent@, support-agent@, scheduling@, each filtering on its own grant_id. No overlap means no conflict to resolve. When humans need visibility, they get read-only IMAP access instead of becoming a second writer.

This is the distributed-systems lesson in miniature: partitioning beats locking whenever you can afford it.

Lesson 4: assume your logic is the next bug

Even with all three layers, you can still build a reply storm. Outbound sends fire message.created too. If your handler forgets to skip the agent's own messages, the agent replies to itself, which triggers another webhook, forever. The first guard is two lines at the top of every handler:

// First check in every handler — skip messages from the agent itself.
const sender = msg.from?.[0]?.email;
if (sender === AGENT_EMAIL) return;

The second guard is a per-thread send budget: more than 3 sends within 5 minutes means something's wrong, so stop and escalate to a human instead of sending.

That's idempotency's underrated cousin — a circuit breaker for when your correct code does something incorrect at volume. Dedup protects you from the platform. The rate limit protects you from yourself.

Lesson 5: the cheapest event to handle is the one that never fires

One more layer sits below all of this. Agent Accounts support server-side rules that sort inbound mail before your webhook handler ever sees it — route automated notifications to a folder the agent doesn't reply in, block spam at the SMTP layer, archive what needs no response.

curl --request POST \
  --url "https://api.us.nylas.com/v3/rules" \
  --header "Authorization: Bearer <NYLAS_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "match": [{ "field": "from.domain", "operator": "equals", "value": "noreply.example.com" }],
    "actions": [{ "action": "assign_to_folder", "value": "notifications" }],
    "description": "Route automated notifications to a separate folder"
  }'

Your handler then checks which folder a message landed in and skips folders the agent shouldn't touch. Every message you filter out declaratively is a message your idempotency stack never has to be correct about. Shrinking the input space is the idempotency strategy nobody writes blog posts about, because it looks like configuration instead of engineering.

The counterargument worth taking seriously

"This is a lot of machinery for sending email." Fair. If your agent handles ten messages a day from a single-threaded process, a dedup table alone will carry you a long way, and the lock may be premature. The docs themselves note that synthetic concurrent load testing is the only way to surface the race — which implies a single-threaded deployment won't hit it.

But the cost asymmetry should drive the decision. The whole stack is maybe forty lines of code. A double reply to a customer is a trust incident you can't un-send. I'd rather carry the forty lines.

One more habit worth stealing: log every skip. When a message is dropped because it's a duplicate or another worker holds the lock, write that down. Silent idempotency is correct but undebuggable.

If you're building a reply loop, read the prevention recipe end to end, then write a load test that fires the same webhook payload at your handler from five concurrent connections. If exactly one reply goes out, you've earned the right to ship. What's the worst duplicate-action bug you've shipped — and which layer would have caught it?

Observability for Email Agents

Qasim Muhammad — Tue, 16 Jun 2026 17:17:50 +0000

You can't watch an email agent work, but everything it did yesterday is one API call away:

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages?limit=50" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

That's the strange, underrated property of building agents on email. Most autonomous systems need observability bolted on — tracing, structured logs, replay tooling. An agent that lives in a mailbox gets three observability primitives for free, because the medium is the record. Here's how to use each one, drawn from how Agent Account mailboxes (currently in beta) actually behave.

The event stream: webhooks

Every inbound message fires message.created — typically within seconds of the SMTP handoff — with a payload that includes the thread_id your agent needs to reconstruct conversation state. That's your input-side event stream, no instrumentation required.

One wrinkle to handle: when a message body exceeds roughly 1 MB, the trigger becomes message.created.truncated and the body is omitted from the payload — fetch the full message by ID in that case, or your agent will silently reason over nothing.

The output side is richer than most people expect, because the platform owns the SMTP path end-to-end and reports back on every send:

Trigger	What it tells you
`message.send_success`	The recipient's server accepted the message
`message.send_failed`	The send died first — outbound rule block, policy limit, or deliverability gate
`message.bounce_detected`	The remote server bounced it, hard or soft

Pipe those three into whatever metrics system you already run and you have per-message delivery telemetry for an autonomous sender. A climbing send_failed count is your earliest signal that something upstream — a rule, a quota, a reputation problem — is throttling the agent.

Two refinements. If your workflow is batch rather than real-time, you don't need webhooks at all for the input side — GET /messages with received_after polls fine; webhooks earn their keep in near-real-time agent loops. And aggregate the outbound triggers per domain, not just per grant: sender reputation is shared across every Agent Account on a given domain, so one misbehaving agent's bounce rate quietly degrades its siblings' deliverability. Fleet observability is a domain-level concern wearing per-account clothes.

State you can read: folders

An agent mailbox comes with six system folders — inbox, sent, drafts, trash, junk, archive — and they double as a state machine you can inspect. junk shows you what spam filtering and mark_as_spam rules decided to divert; if real customer mail is landing there, you'll see it by listing one folder. Custom folders extend the pattern: rules that route invoices or VIP senders into named folders turn "what kind of mail is the agent getting?" into a folder-counts query.

The drafts folder earns special mention in human-in-the-loop designs. If your agent proposes replies as drafts and a reviewer approves them, the drafts folder is your approval queue — its count is your queue depth, and a draft that's been sitting there for hours is a stalled approval you can detect with a folder listing.

The governance layer is observable too. Every rule that fires on inbound mail is logged as a rule evaluation you can fetch afterward — so "why did the agent never see that message?" has a queryable answer (a block rule rejected it at the SMTP layer) instead of a shrug.

The audit log: sent mail

Here's the primitive that ordinary agent architectures genuinely lack. Every action this agent takes in the world is an email, and every email it sends is preserved in sent — addressed, timestamped, threaded to its context. The audit log can't drift from reality because it is reality.

Threading makes the log legible. Replies group by standard RFC 5322 headers, so reviewing an incident means fetching one thread and reading the whole exchange in order — what came in, what the agent said, what came back.

And because IMAP access exposes the identical mailbox the API sees, a non-engineer can audit the agent by opening it in Outlook or Apple Mail. There's no separation between protocol traffic and API traffic: one mailbox, one record, two ways to read it. Try giving your compliance team that kind of access to your LLM's tool-call logs.

A Tuesday-morning incident, walked through

Here's how the three primitives compose under pressure. Tuesday, 9:40 a.m.: your dashboard shows the support agent's reply rate dropped to zero overnight, but inbound volume looks normal. Where do you look?

First, the event stream. message.created events are still arriving, so mail is landing — input is healthy. But message.send_failed started climbing at 11 p.m. The agent has been drafting replies and failing to deliver them for ten hours.

Second, the governance record. A send that fails before it reaches the recipient is typically an outbound rule block or a policy limit, and rule evaluations are logged per grant:

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/rule-evaluations" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

The evaluations show which rule matched and what action it took. In this story, a recently enabled outbound rule is matching more broadly than intended.

Third, the mailbox itself. Fetch the affected threads and read them in order — what came in, what the agent tried to say, where it stopped. Total diagnostic surface: one webhook chart, one API call, one folder read. No log aggregator, no trace sampling, and the postmortem writes itself from artifacts that can't disagree with each other.

The blind spots

Honest limits, so you don't design around capabilities that aren't there. Native open and click tracking — message.opened, message.link_clicked — isn't emitted for messages sent through the API on these accounts, so "did a human read it?" is not an observable event; delivery signals are where your visibility ends. And a send_success only means the recipient's server accepted the message — recipient-side filtering afterward is invisible to you, as it is to every sender on earth.

There's also a subtler gap: the webhook stream tells you what happened, not why the agent chose it. Mailbox observability covers actions; you still own logging the reasoning (prompts, classifications, decisions) that produced each send.

Wire the outbound three first

If you instrument nothing else this sprint, subscribe to message.send_success, message.send_failed, and message.bounce_detected and chart them. Input observability fails loud — the agent stops responding and someone notices. Output observability fails quiet: the agent keeps cheerfully sending into a rising failure rate, and the webhook stream is how you find out in minutes instead of weeks.

What's on your email-agent dashboard today — and if the answer is "nothing yet," which of the three primitives would you wire up first?

Human-in-the-Loop Design for Email Agents

Qasim Muhammad — Tue, 16 Jun 2026 17:17:47 +0000

A refund request lands in your support agent's queue. The knowledge-base match comes back at 0.91 confidence — comfortably above your drafting threshold, with a clean article on refund policy attached. The agent should still not send that reply. If that sentence sounds wrong to you, this post is the argument for why it's right.

Autonomy is a dial, not a switch

Most teams frame human-in-the-loop as a binary: either the agent sends email on its own or a human approves everything. The framing fails in both directions. Full-auto on everything means the one bad reply ends up screenshotted in a board deck; full-review on everything means you've built an expensive draft generator that saves nobody time.

The better model is a dial set per message type, not per agent. The same support agent can run fully autonomous on order-status lookups, draft-and-approve on billing questions, and hands-off-escalate on anything with legal weight. The email support agent recipe implements exactly this, with two independent gates deciding where the dial sits for each message.

Gate one: confidence

The first gate is mechanical — how sure is the knowledge-base match? The recipe's thresholds:

Confidence	Action
≥ 0.85	Draft directly from the matched article
0.60 – 0.85	Draft conservatively, cite the source article inline so the reviewer can verify
< 0.60	Don't draft — flag for manual review with the best-guess article attached

The middle tier is the clever part. Citation-required drafts let reviewers calibrate their scrutiny: trust the high-confidence pile, check the cited ones, write the low-confidence ones themselves. That's what keeps review from becoming a second full-time job.

Gate two: risk — and it overrides

The second gate is about consequences, and it doesn't care what gate one said:

Low — password resets, FAQ-shaped questions → draft, human approves.
Medium — refunds, account changes, anything touching billing → draft, human approves with extra scrutiny.
High — legal threats, regulatory matters, fraud reports → no draft at all; escalate immediately to a person with full context.

This is why the 0.91-confidence refund reply doesn't go out: refunds are medium-risk regardless of match quality. Confidence measures "do we know the answer?"; risk measures "what happens if we're wrong?" — orthogonal questions, and conflating them is how an agent ends up committing your company to something. There's a widely cited airline-chatbot ruling about precisely that failure: the bot promised a refund policy that didn't exist, and the company was held to it.

The choke point

In code, the whole policy is small:

def handle(msg):
    question = extract_question(msg)
    article, conf = kb.search(question)

    if classify_risk(msg) == "high":
        escalate_to_human(msg, reason="high-risk topic")
        return
    if conf < 0.60:
        flag_for_review(msg, article)
        return

    draft = generate_draft(msg, article, cite_inline=(conf < 0.85))
    queue_for_approval(msg, draft, article)

Notice what's absent: a send call. queue_for_approval is the choke point — in production it drops drafts into Slack or a review tool, never directly into the outbox. The recipe states the load-bearing constraint outright: always show the draft before sending, never auto-send. Every other rule can be tuned; remove that one and you no longer have a gate, you have a delay.

If you're giving the agent its own mailbox to run this from, Agent Accounts — currently in beta — are the natural home for a support@ identity the agent owns end-to-end. They also give the choke point a native implementation: the Drafts API supports full CRUD, so the agent can create a draft in its own mailbox, a reviewer can amend it, and approval sends the existing draft — the send action on a draft behaves exactly like a regular send. The pending reply lives where the conversation lives, instead of in a screenshot pasted into Slack.

Declaring the policy instead of coding it

The same gates work as configuration. If the agent runs on a skill-file platform, the recipe shows the whole policy as a SKILL.md rather than code:

# Support agent

## Reply style
- Replies are under 120 words.
- Cite KB articles inline: [KB-1234](https://kb.example.com/1234).
- Match the tone of the inbound message.

## Drafting rules
- Always show the draft before sending. Never auto-send.
- If confidence < 0.6, do not draft — flag for human.
- Refunds, account changes, legal threats: never draft. Escalate.

## Polling
- Check the support inbox every 10 minutes.
- Process at most 5 tickets per cycle while the agent is in shakedown.

Notice the policy reads like an onboarding doc for a junior hire — which is the right intuition. The drafting rules section is the dial from earlier, written in plain English, and "always show the draft before sending" appears here too because it survives every refactor: it's a property of the system, not of any particular implementation.

Loosening the dial, slowly

Human-in-the-loop isn't a permanent tax; it's how you earn the data to automate more. The recipe's shakedown numbers: process 5 tickets per cycle while tuning the matcher and risk classifier, bump to 20 once the false-positive rate is acceptable. Polling every 5–15 minutes is plenty for support latency — and simpler than webhook fan-out on a shared inbox.

Two practices make the loosening defensible. Log everything — every classification, KB lookup, and approval decision — so when you propose moving a message type from draft-and-approve to full-auto, you argue from a thousand logged approvals, not vibes. And track what the agent can't match: those tickets are your map of missing KB articles.

The counterargument deserves its hearing: review fatigue is real, and a human rubber-stamping 200 drafts a day is barely a gate. Two answers. First, tier aggressively — automate the genuinely safe tiers so human attention concentrates where it changes outcomes, and batch the repetitive stuff: when three "where's my receipt?" tickets arrive in a row, that's one KB article, one draft template, one reviewer pass, not three. Second, remember what the gate is actually defending against. The recipe puts it bluntly: even at 99% accuracy, the 1% that makes legal commitments destroys trust faster than the 99% builds it. Fatigue is a workload-design problem with workload-design fixes. The 1% is not.

Map your own agent's message types into the three risk tiers this week, and set the dial per tier. Which message type are you currently over-reviewing — and which one, honestly, shouldn't be on full-auto?

The Lifecycle of an Agent Identity: Provision to Teardown

Qasim Muhammad — Tue, 16 Jun 2026 17:17:43 +0000

Every infrastructure team has a graveyard. Service accounts nobody remembers creating. API keys that outlived their project by three years. Credentials that still work and absolutely shouldn't. Email identities for AI agents are about to join that graveyard in large numbers — unless you treat them like infrastructure with a lifecycle: provisioned deliberately, auditable while alive, destroyed on purpose.

Here's that lifecycle, stage by stage, using Agent Accounts (currently in beta) as the concrete machinery.

Birth: a domain, then an address

An agent identity starts before the agent does, with a domain decision. Two options: a *.nylas.email trial domain that works instantly with zero DNS setup, or your own domain — registered once per organization, verified by publishing the MX record (inbound routing) and TXT records (ownership proof plus SPF/DKIM for outbound) at your DNS provider. Verification happens automatically once records propagate; the domain status flips to verified and it's ready to host accounts.

The recommended production pattern is a dedicated subdomain like agents.yourcompany.com, so the fleet's sender reputation stays isolated from your primary domain. Reputation is a domain-level asset; don't let an experimental agent spend it.

With a domain ready, birth is one call:

nylas agent account create sales-agent@agents.yourcompany.com

Prefer raw API? POST /v3/connect/custom with "provider": "nylas" does the same job — unlike OAuth providers, no refresh token is involved, just an address on a registered domain:

{
  "request_id": "5967ca40-a2d8-4ee0-a0e0-6f18ace39a90",
  "data": {
    "id": "b1c2d3e4-5678-4abc-9def-0123456789ab",
    "provider": "nylas",
    "grant_status": "valid",
    "email": "sales-agent@agents.yourcompany.com",
    "created_at": 1742932766
  }
}

The response includes the grant_id (data.id) — the identity's handle for everything that follows. Two birth-time decisions worth making consciously rather than by default: placement (pass a workspace_id so the account inherits that workspace's policy limits and rules; omit it and the account is auto-grouped by domain or dropped into the application default) and protocol access (an optional app_password — 18–40 printable ASCII characters with at least one uppercase, one lowercase, and one digit — enables IMAP/SMTP so humans can open the mailbox in Outlook or Apple Mail; it's bcrypt-hashed on write and can never be retrieved, only reset).

Work: one ID, the whole surface

While the agent lives, the grant_id is its entire interface. Messages, threads, drafts, folders, attachments — all the existing endpoints work against /v3/grants/{grant_id}/... exactly as they do for a human-connected account. The mailbox arrives with six system folders (inbox, sent, drafts, trash, junk, archive); custom folders can be added beside them.

The working identity also emits a steady event stream. Inbound mail runs the workspace's rules at the SMTP stage — block rejects a message before it's ever stored, mark_as_spam routes it to junk, assign_to_folder files it — and whatever survives fires message.created, identical in payload shape to the same webhook on any connected grant. Outbound, deliverability signals come back as message.send_success, message.send_failed, and message.bounce_detected, so every send the identity makes has a verdict you can record. An identity that's alive is an identity that's telling you things; silence on those triggers is itself a signal worth alerting on.

The fleet-management view matters as much as the single-account view. The CLI covers it without a dashboard visit:

nylas agent account list --json   # inventory
nylas agent status                # connector readiness
nylas agent policy list           # what governs whom

That inventory command is your defense against the graveyard. If agent account list returns something nobody can explain, you've found a zombie.

Audit: the mailbox is its own record

Here's where email identities beat most service credentials: the audit trail is built into the artifact. Every message the agent ever sent sits in its sent folder; every conversation is a thread you can fetch and read. The provisioning docs' verification step — send a test message in, then list the mailbox — doubles as the ongoing health check:

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages?limit=5" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

And because IMAP access shows the identical mailbox the API sees, a human reviewer can audit an agent by literally opening its mail. No log pipeline required for the first pass. The governance layer keeps its own books too: every rule that fires on inbound mail is logged as a rule evaluation you can query later, so "why did this message end up in junk?" has a recorded answer instead of a shrug.

One honest caveat before you lean your compliance story on this: the mailbox-as-audit-log has a retention horizon. On the free plan, inbox mail is retained for 30 days and spam for 7. If your audit requirements are measured in years, configure retention through the workspace policy or run an export job — don't discover the horizon during an investigation.

Mid-life changes don't require rebirth, either: move an account to a different workspace — different policy, different rules — with PATCH /v3/grants/{grant_id} and a new workspace_id. Governance evolves; the identity persists.

Death: delete on purpose

The stage everyone skips. An agent identity should die when its work ends — project shipped, test run finished, customer churned:

nylas agent account delete sales-agent@agents.yourcompany.com --yes

The discipline that makes this painless is pairing: whatever process creates an account owns its deletion. The docs' environment-separation pattern helps here too — staging agents live on agents.staging.yourcompany.com, so a sweep of stale staging identities can't touch production, and per-customer domains mean offboarding a tenant cleanly removes their agents with them.

Ephemeral identities aren't a workaround; they're the design. A mailbox that exists for one CI run and is gone an hour later never joins the graveyard.

Script both ends before you need either

Concrete next step: before your next agent ships, write the teardown script in the same PR as the provisioning script — create and delete, side by side, tested together. Then run nylas agent account list on whatever you have today.

Be honest: how many identities in that list could you explain right now?