DEV Community: Pykero

Should Your WhatsApp AI Sales Agent Handle Voice Notes, and What Does It Cost?

Pykero — Sat, 07 Nov 2026 09:06:10 +0000

Yes, if your customers actually send them. In many markets, especially across the Gulf, North Africa and Latin America, a large share of inbound WhatsApp sales messages arrive as voice notes, and an agent that only reads text is silently ignoring those leads or dumping them on a human. Adding speech-to-text to an existing WhatsApp agent is a bounded project: expect one to three weeks of engineering on top of a working text agent, with recurring cost driven mostly by how many minutes of audio you process and which languages you need to understand well.

Why voice notes break text-only agents

A WhatsApp AI sales agent is usually built around a simple loop: receive a message via webhook, reason about it with an LLM, call a few tools (catalogue, CRM, booking), reply. Voice notes break the first step. The webhook does not deliver words; it delivers a media ID pointing to an audio file.

According to the WhatsApp Cloud API media documentation, audio arrives in formats such as OGG with the Opus codec (the native voice note format), AAC, MP4, MPEG and AMR. Your system has to download that file with a short-lived URL, convert or pass it to a speech model, and only then does the rest of your pipeline have anything to work with.

If you built your agent without this step, one of three things is happening today:

Voice notes are dropped. The agent replies with something generic or nothing at all, and the lead goes cold.
Voice notes are routed to a human. This works until volume grows, and it quietly destroys the response-time advantage that justified the agent.
Customers adapt by typing. Some do. The ones who prefer to talk simply go to a competitor who answers.

Before committing budget, measure it. Count inbound messages by type for two or three weeks. If voice notes are under five percent of volume, a graceful fallback ("Could you type that out for me?") may be enough. If they are a quarter of your inbound, you are leaving real revenue in an audio file nobody listens to.

What you are actually building

The architecture that works best is boring on purpose: transcribe first, then feed the transcript into the exact same text agent you already have.

Step 1: Media ingestion

A small service receives the webhook, detects an audio message, fetches the media through the short-lived URL before it expires, and stores the raw file. Store it, not just the transcript. Keep the original OGG/Opus file alongside the sender, timestamp and media ID, and later attach the transcript and the agent's reply to the same record. That one record does three jobs. When a customer complains that the agent quoted the wrong item, you can replay the audio against the transcript and see whether the speech model misheard the product name or the agent misread a correct transcript. When a compliance team asks what was actually said, you have the source. And when you reach the point of comparing speech models, those stored notes are the fifty-to-hundred-recording evaluation set you would otherwise have to go back and collect. Without the audio you can only guess which of the two layers broke.

Step 2: Speech-to-text

This is the real decision point. You have three broad options:

A cloud speech API. Fastest to integrate, priced per minute of audio, good for English and major languages. Quality on dialects and noisy recordings varies widely between providers.
A self-hosted open model. Whisper and its faster derivatives run well on a modest GPU and cost you infrastructure rather than per-minute fees. Worth it when volume is high or when the audio cannot leave your region. We compared the two paths in detail in self-hosted speech-to-text vs cloud API.
A specialised or fine-tuned model. Necessary when the general models fail on your audience. Gulf Arabic, medical vocabulary, product names and heavy code-switching are the typical triggers. See what Arabic speech recognition really costs for the specific challenges there.

Step 3: The existing agent, unchanged

The transcript becomes the user message. Everything downstream, your prompts, tool calls, guardrails, escalation rules and evals, keeps working. This is the main reason to resist the temptation of a model that "listens and answers" in one go. It feels elegant, but it hides the transcript, couples your speech quality to your reasoning quality, and makes every bug twice as hard to find.

One useful addition: pass a short note into the prompt that the message was a voice note and may contain transcription errors. The LLM then handles a misheard product name more gracefully, asking a clarifying question instead of confidently quoting the wrong item.

Step 4: Replying, in text first

The Cloud API allows you to send audio back, so a voice reply is possible. Start with text anyway. Text replies are cheaper, can be skimmed, are easier for a human supervisor to review, and avoid the extra latency of text-to-speech. Add voice output later only if customer feedback demands it, and then only for short confirmations rather than full product explanations.

What drives the cost

The budget splits into a one-off build and a recurring run cost. The build is comparatively predictable. The run cost depends on your volume and language.

Build cost drivers

Whether a text agent already exists. If it does, voice is an increment. If not, you are really buying a full agent, and our breakdown of WhatsApp sales agent cost is the right starting point.
Language and audio quality. English voice notes in quiet rooms are a solved problem. Dialectal Arabic recorded on a motorbike is not. The second case needs an evaluation set of real recordings, time to compare models, and possibly fine-tuning.
Compliance requirements. Healthcare and government buyers often cannot send customer audio to a foreign cloud. That pushes you toward self-hosting, which adds infrastructure setup and a few more days of work.
Review tooling. You will want a simple screen where a human can play the audio, see the transcript, and see the agent's reply. Skip it and your team will debug from log files for months.

Recurring cost drivers

Minutes of audio per month. This is the only number that matters for the speech line item, and you should derive it from your own two-to-three-week count rather than guess. If voice notes are a quarter of your inbound and a typical one runs about 40 seconds, every thousand voice notes is roughly 670 minutes of audio. A cloud provider bills those minutes directly, so the cost grows in a straight line with volume. A self-hosted GPU bills by the hour whether it is transcribing or sitting idle, so its cost is flat and only wins once there are enough minutes to keep it busy.
Slightly longer prompts. A rambling 40-second voice note transcribes into several sentences where the typed version would be one line asking whether the blue one comes in medium. LLM token spend rises a little per conversation as a result. Unless your customers record very long notes, it is a rounding error next to the speech minutes.
Model upkeep. If you fine-tuned for a dialect, someone has to re-run your fifty-to-hundred-note evaluation set and re-score it with a native speaker whenever the base model or vendor changes. That is a recurring cost, not a one-off, and it is not zero.

A practical rule: take your measured monthly minutes, get a per-minute price from one cloud API and the hourly price of the smallest GPU instance that runs Whisper comfortably, and compare the two against that single number. At low volume the API wins easily, because a GPU idle for most of the day is pure waste. At high volume, or when a healthcare or government buyer forbids audio leaving the region, self-hosting wins regardless of the arithmetic.

Keep the pipeline simple, it pays twice

There is a strong pull, once you have audio, to build an elaborate chain: transcribe, then summarise, then classify intent, then extract entities, then draft a reply. Each extra step adds latency, cost and a new place for errors to compound.

We learned this on our own outreach engine, which scrapes each prospect's website and drafts a tailored email with a local model. Our first version chained several specialised calls together. The version that won on both cost and quality simply asked one model to pull out the relevant facts and write the draft in a single pass. The same lesson applies here: once you have a transcript, one well-structured call that understands the request and decides the next action usually beats a multi-step chain. We wrote more about this trade-off in single call vs agent chains.

The one step you should not collapse is transcription itself. Keep it separate and keep the transcript. Everything else can be a single call until you have evidence it should not be.

When it is not worth it

Voice note support is a poor investment when:

Voice notes are rare in your actual inbound traffic and a polite typed fallback is accepted by customers.
Your product requires precise identifiers such as serial numbers or dosages, and your audience's language is poorly served by current speech models. A misheard digit is worse than asking for text.
You have not fixed the text agent yet. If the text agent escalates half of its conversations, voice will escalate even more. Fix the foundation first.

A sensible rollout

Measure the share of voice notes for two to three weeks.
Collect fifty to a hundred real voice notes (with consent and your privacy policy in order) as an evaluation set.
Run two or three speech models over that set and have a native speaker score the transcripts. Pick the winner on your data, not on a benchmark.
Ship transcription into the existing text agent, with the "this was a voice note" hint in the prompt.
Review a sample of audio, transcript and reply every week for the first month.
Only then decide on voice replies, fine-tuning, or self-hosting.

If your WhatsApp inbox is full of audio your agent cannot hear, that is a measurable leak in your sales funnel and a fairly contained project to fix. If you want a second opinion on your volume, language and the cheapest path to a working pipeline, let's talk.

Originally published on the Pykero blog.

AI Abandoned Cart Recovery: Should You Build an Agent or Buy a Plugin?

Pykero — Thu, 05 Nov 2026 09:06:08 +0000

Buy an abandoned-cart plugin when your shoppers forget and a reminder or coupon brings them back. Build a custom AI recovery agent when your carts die because a question went unanswered at checkout: shipping to their country, whether the part fits their model, how returns work for sale items. The plugin sends a message. The agent sends an answer. Those are different products, and only one of them protects your margin.

Why most recovery flows underperform

The standard recovery stack is a timer and a template. Shopper abandons a checkout, a tool waits an hour, sends "You left something behind", waits a day, sends a 10% code. It is cheap, it is automatic, and for a share of carts it works.

The problem is the assumption baked into it: that every abandonment is a lapse of attention. Talk to your support inbox and you will find a different story. People leave because they could not find the delivery time to their city, because the size chart contradicted the reviews, because they wanted to know if the warranty covered them abroad, because the checkout showed a tax line they did not expect. None of those people are waiting for a coupon. They are waiting for an answer, and if they do not get it within the hour they buy elsewhere or forget.

A discount sent to a shopper with an unanswered question does two bad things. It fails to convert them, because the blocker is still there. And it trains your repeat buyers to abandon on purpose and wait for the code.

What an AI recovery agent actually does

Strip the hype and the agent is a small, well-scoped system:

Trigger on the abandoned checkout event from your platform. Shopify, for example, records abandoned checkouts and lets you act on them from the admin or programmatically (Shopify docs).
Read context: line items, shipping country, cart value, whether the shopper is a returning customer, and any on-site chat or search they did before leaving.
Infer the likely blocker and open with a question or a direct answer rather than a generic nudge. "Still deciding between the 42 and the 43? Both are in stock, and exchanges are free within 30 days" beats "Complete your order".
Answer follow-ups from your real catalog, shipping matrix, and policy documents, with retrieval grounded in your own data so it cannot invent a return window. If you have not seen how that grounding works, start with RAG explained for founders.
Hand off to a human the moment the conversation touches money, complaints, or anything outside the knowledge base.
Log everything so you can see which blockers actually cost you sales.

That last bullet is undersold. After a month, the agent's transcripts are the best voice-of-customer research you own. If 30 conversations in a row ask about delivery to the same region, the fix is a shipping-page change, not more messaging.

The CTA matters more than the model

Here is a lesson from our own tooling that transfers directly. We run a cold-outreach engine for ourselves, and for a long time every email ended with a calendar link. Replies were thin. When we swapped the booking link for a soft ask, essentially "reply yes if this is relevant and we will send details", reply rates rose and stayed up. Interested people were not refusing the call. They were refusing the work of scheduling one from a cold message.

Recovery messages have the same friction problem. "Click here to complete your checkout" asks the shopper to re-enter a flow they already left, often on a different device. "Reply with your city and I will confirm the delivery date" asks for five seconds and gives them a reason to keep talking. Design the first message as the start of a conversation, not a redirect. We covered the general pattern in AI sales agent CTA design, and it applies here verbatim.

When buying is the right call

Be honest about your abandonment profile before you commission anything. Buy the plugin if:

Your catalog is simple. Few SKUs, no compatibility or sizing questions, standard shipping to one or two countries. There is nothing for an agent to answer.
Discounts are already your lever. If your margin tolerates a code and your customers respond to one, the plugin is doing its job.
You have no support history to learn from. Without a record of pre-purchase questions, you are guessing at the blockers. Run the cheap tool first and read the replies.
Volume is low. Under a few hundred abandoned carts a month, a human replying from the support inbox with a good macro set will beat any automation on quality and cost.

Most plugins also now bolt on an "AI" layer that rewrites the subject line or personalises the product grid. That is not the same thing as answering a question, but it is cheap, and if it moves your numbers, take it.

When building is the right call

Build when the plugin's model of the problem is wrong for your store:

Carts die from questions. Technical products, fit-sensitive apparel, cross-border shipping, regulated goods, anything with a long consideration cycle.
Discounting hurts. Thin margins, premium positioning, or a brand promise around never going on sale.
Your customers live on WhatsApp. In the Gulf and much of the Middle East and North Africa, a WhatsApp reply gets read in minutes while an email waits until Monday. Account for the platform's conversation-based pricing and template approval process (WhatsApp Business Platform pricing) and read our breakdown of what an AI WhatsApp sales agent costs before committing.
You already run a headless or custom storefront. The webhook and data plumbing is yours anyway, and a generic Shopify app may not even install cleanly.

What it costs, honestly

Think in three buckets, and measure each against the flow you already run: the one-hour reminder and the 10% code.

Build. A first version that covers trigger, context reading, grounded answers on one channel, human handoff, and a transcript dashboard is a small custom feature project, measured in weeks rather than months. The expensive part is not the LLM call. It is making your shipping rules, return policy, and product attributes machine-readable and keeping them that way. For the agent to say "both the 42 and the 43 are in stock, and exchanges are free within 30 days", stock by variant, size equivalence, and the current return window all have to be readable from a system, not from a spreadsheet and three people's heads. If they are not, budget time to fix that first, because that is where most of the project's weeks go.

Run. Messaging fees dominate on WhatsApp, almost nothing on email, and on WhatsApp every opening message is a template that has to pass approval before it can send. LLM spend per conversation is small if you keep prompts tight and cache the catalog context. For predictable spend, cap conversations per shopper so one stuck cart cannot become an open-ended thread, and apply the techniques in LLM cost optimization. Compare the total to the margin you currently hand out in 10% codes, not to zero.

Maintain. Policies change, products change, and the agent will confidently repeat last season's return window unless a change to the policy page invalidates its knowledge base. Treat the agent as a product with an owner, not a one-time install. The minimum is a monthly read of handoffs and misses, and that same review is where you catch patterns like 30 shoppers in a row asking about delivery to one region, which is a shipping-page fix rather than an agent fix.

A sensible path

Pull three months of pre-purchase support questions and abandoned checkouts. Tag the questions. If fewer than a fifth of abandonments plausibly map to a question, buy the plugin and stop reading.
If questions dominate, ship the agent on email first, one channel, one language, hard handoff rules. Measure recovered revenue against the discount flow you are replacing, not against zero.
Add WhatsApp once the answers are proven and the opt-in flow exists on the storefront.
Feed the blocker log back into the site. The best recovery agent slowly makes itself less necessary.

If your carts are dying from unanswered questions and you want a grounded estimate for your catalog and channels, let's talk.

Originally published on the Pykero blog.

Is Web Scraping Legal for Your AI Product? What Founders Should Weigh Before Building

Pykero — Tue, 03 Nov 2026 09:05:28 +0000

Web scraping is not illegal by default, but it is not a free lunch either. The legal exposure of an AI product built on scraped data depends on three things: whether the data is public or behind a login, whether it contains personal data, and whether you republish it or only use it internally. Get those three right in the design and most of the risk disappears before a lawyer ever sees the product.

This is a founder's guide, not legal advice. The point is to help you ask the right questions before you spend a build budget on a feature that compliance may force you to rip out.

Where the risk actually sits

Founders tend to worry about the wrong thing. The fear is "hacking" style liability. In practice, scraping disputes land in four buckets.

1. Contract: the site's terms of service

Most commercial sites ban automated access in their terms. Whether those terms bind you depends heavily on whether you agreed to them. Browse-wrap terms (a footer link nobody clicks) are weaker than click-wrap terms you accepted when creating an account. The practical rule: scraping behind a login you created means you agreed to the terms, and you are now in breach-of-contract territory. Scraping public pages without an account is a much weaker claim against you.

2. Computer misuse laws

Statutes like the US Computer Fraud and Abuse Act (18 U.S.C. § 1030) are about unauthorised access. Recent US case law has narrowed this for publicly available pages, but circumventing authentication, IP bans or technical blocks puts you back in scope. If you have to defeat a block to keep scraping, treat that as a signal to stop, not an engineering challenge.

3. Personal data: GDPR and its cousins

This is the bucket that bites AI products hardest. If you scrape names, emails, job titles or anything that identifies a person, you are processing personal data and need a lawful basis under GDPR Article 6, plus transparency and deletion obligations. "It was public" is not a lawful basis on its own. Scraping personal data at scale and then enriching it with an LLM makes this worse, because you are now profiling.

4. Copyright and database rights

Factual data (prices, opening hours, product specs) is weakly protected. Articles, images, descriptions and curated datasets are not. In the EU, the Database Directive 96/9/EC adds a separate right over substantial extraction from structured collections. The 2019 Digital Single Market Directive carved out a text-and-data-mining exception for commercial use, but rights holders can opt out through machine-readable means. That is why robots.txt and similar signals matter more in Europe than their informal status suggests.

The risk ladder, from safe to stop

Use this to place your feature before scoping it:

Low risk: public pages, no login, factual or company-level data, used internally to generate your own output (a summary, a score, a draft). Rate-limited, respects robots.txt.
Medium risk: public pages but you store and display excerpts of the original content to users, or you build a searchable index of a competitor's catalogue.
High risk: personal data at scale (profiles, contact details), data behind any authentication, or republishing copyrighted content as your product.
Stop: bypassing blocks, scraping health or financial data about individuals, or scraping platforms whose terms you accepted and which explicitly forbid it.

If your feature is "medium" or above, that is not an automatic no. It means the legal review and the data design need to happen before the sprint, and the cost estimate in your scraping pipeline budget should include them.

Design choices that lower exposure

Legal risk in scraping is mostly an architecture decision. A few patterns move a feature down the ladder without killing it.

Extract facts, drop the source

The biggest lever is retention. If you fetch a page, extract the handful of facts you need and discard the raw HTML, you hold almost no copyrightable material and a minimal footprint of anything personal. If you store full page copies "in case we need them later", you have built a copy of someone else's site.

We run this pattern in our own outreach engine. It fetches each prospect's public site with a self-hosted Firecrawl instance and passes the content to a local LLM in a single call that both extracts what the company does and drafts one tailored email. The raw page content is thrown away after the call. We originally tried a multi-step chain (crawl, summarise, classify, draft) and the single-call version was both cheaper and produced better emails, which is a useful reminder that simpler pipelines often win on compliance too, because there are fewer places where data lingers.

Scrape the company, not the people

For sales and market-intelligence use cases, almost everything you need is company-level: what they sell, who they sell to, what stack they advertise, what they recently launched. Scope your extraction prompt to that and explicitly instruct the model to ignore personal names and contact details. In our outreach engine the extraction prompt asks for exactly one thing, what the company does, and the only person-level field anywhere in the pipeline is the single address the email goes to. That one address still needs a lawful basis, but there is no shadow database of individuals to defend, because the pipeline never had a place to put one.

Respect the signals, log that you did

Honour robots.txt, identify your crawler with a real user agent and contact URL, rate-limit per domain, and keep a log of which pages were fetched and when. None of this is strictly required everywhere, but when a site owner complains, "we honoured your robots.txt and crawled at one request every few seconds" is a very different conversation from "we rotated residential proxies to get around your block".

Prefer official data when it exists

Many sites you would scrape also publish feeds, sitemaps, APIs or structured data. An API with terms you accept is slower to integrate but gives you a contract you can point to. If a data vendor already licenses what you need, compare their price to the cost of building and defending a scraper. That is a genuine build-vs-buy decision, not an engineering preference.

Self-host or use a scraping API?

If scraped data is a core input to your product, self-hosting an open-source crawler such as Firecrawl is usually the right call. The reason we self-host it for our outreach engine is not cost, it is that every answer to the review questions above becomes ours to give: the rate limit is set in our config, robots.txt handling is our code, the fetched page lives in memory on our box for one LLM call and is then gone, and nothing leaves our infrastructure because the model is local too. That is a short, checkable story to tell a data-protection officer. For a low-volume or experimental feature, a hosted scraping API gets you to a signal faster, and you can migrate once the feature proves itself.

Either way, check what the provider does with the content it fetches for you. Some hosted services cache pages or use them to improve their own systems, which makes your retention policy only as good as theirs. If you have promised "we discard the raw page after extraction" and your provider keeps a copy, that promise is already broken.

Questions to settle before you brief an agency

What exactly do we extract, and what do we delete immediately after?
Does any of it identify a person? If so, what is our lawful basis and deletion path?
Will users ever see the original content, or only our derived output?
Is anything behind a login or a block we would need to circumvent?
Which jurisdictions are the sites and our users in?
Is there an API or licensed source that makes this moot?

Bring answers to these to your first scoping call and the estimate you get back will be far more accurate. If the feature touches outreach, read our notes on cold email deliverability as well, because a technically clean scraper paired with a spammy send pipeline still ends badly.

The short version

Scraping public, factual, company-level data and turning it into your own output is a well-trodden, defensible path. Scraping people, scraping behind logins, or republishing what you scraped is where products get pulled. Decide which side of that line your feature sits on before you build, design the pipeline to keep as little as possible, and treat blocks as a stop sign.

If you are weighing an AI product that depends on external web data and want a second opinion on the architecture and the risk ladder, let's talk.

Originally published on the Pykero blog.

How to Choose Your First AI Project

Pykero — Sun, 01 Nov 2026 09:05:37 +0000

Your first AI project should be a narrow, repetitive, text-heavy task that a human already does slowly or inconsistently, where the inputs already exist and the output can be checked by someone on your team. It should ship in weeks, not quarters, and it should have one metric you can read six weeks after launch. Everything else about "AI strategy" follows from what that first result teaches you.

Most founders we talk to get this backwards. They start with the most visible idea (a chatbot on the homepage, an "AI copilot" for the whole product) and end up with a demo that impresses in a meeting and never reaches production. The filter below is what we use to steer them toward something that actually ships.

The four-question filter

Run every candidate through these in order. A project that fails any one of them is a second or third project, not a first.

1. Does a human do this today, and do they complain about it?

The best first projects are already happening manually. Someone on your team reads inbound leads and sorts them. Someone summarizes support tickets for the weekly review. Someone copies fields from PDFs into a CRM. Someone writes near-identical follow-up emails forty times a week.

If nobody is doing the task today, you are not automating a workflow, you are inventing a product. That is a legitimate thing to do, but it is a product bet with product risk, and it should not be your first AI project.

The complaint matters as much as the task. A workflow people hate gives you a motivated internal user who will test the output, report failures, and defend the project when it is imperfect in week two.

2. Do the inputs already exist in a usable form?

Text in emails, documents, support tickets, call transcripts, and web pages is usable. Data scattered across six spreadsheets with inconsistent column names is not, and "we just need to clean the data first" is the most common cause of death for a first AI initiative.

Pick a task where the model can read what already exists. If the honest answer is that the data needs a migration project before any model can touch it, choose a different first project and put the migration on the roadmap separately.

3. Can a non-engineer check whether the output is right?

Your first project needs a human who can look at the output and say "yes" or "no" in under a minute. A drafted email, a classification label, an extracted set of fields, a one-paragraph summary: all checkable. A multi-step agent that negotiates with vendors or edits your database autonomously is not checkable by glancing at it, and you have no evidence yet about how your model behaves on your data.

This is also why we tell founders to stay internal for the first project. When the output is wrong, the cost is a few minutes of a colleague's time, not a customer's trust. Once you have a few weeks of real error rates, the decision to go customer-facing becomes a decision based on data. We cover what that transition takes in moving an AI agent from pilot to production.

4. Is there one metric you can read in six weeks?

Hours saved per week. Reply rate. Percentage of tickets routed correctly. Days from inbound lead to first response. One number, measured before and after.

If the only honest success criterion is "the team feels more productive", the project cannot be evaluated and cannot justify a second one. For the finance conversation behind this, see our guide on calculating the ROI of an AI agent.

What this filter rules out, and why that is fine

The homepage chatbot. Fails question three and question four at once. Nobody on your team can check every answer it gives a visitor, the metric is unclear, and the downside of a wrong answer is public. A fine third project, once you have internal error rates to point at.
The internal knowledge assistant over "all our documents". Fails question two. "All our documents" means six formats, three permission models, and nobody who can say whether an answer was right. Shrink it to one document type and one team, and it passes.
The full-workflow agent. Anything described as "an agent that handles the whole process end to end" fails question three. In our own case, the end-to-end version would have been "an agent that runs outreach." We did not build that. We built the one step a human did most often and liked least, the research-and-draft, and left sending, follow-up, and CRM updates to people. The whole-process version is a second or third project, assembled from steps that have each been checked.
Anything that starts with building infrastructure. Vector databases, data lakes, orchestration platforms. These are tools you may need later. Buying or building them before you have a working task is how budgets vanish, and it is almost always a sign that the task itself has not been pinned down.

A worked example from our own operations

Our own first internal AI project was not glamorous. We send outreach to prospective clients, and the slow part was research: someone had to open each company's site, understand what they do, and write an email that referenced it. That checked every box. A human did it and disliked it. The input, a public website, already existed. A colleague could read the draft and judge it in thirty seconds. The metric was reply rate.

We built a small engine that crawls each prospect's site with a self-hosted crawler, hands the content to a locally hosted model, and produces one tailored email per company. The first architecture we tried was a chain: one step to extract facts about the company, another to pick an angle, another to draft. It worked, but when we collapsed it into a single prompt that extracted the relevant facts and drafted the email in one pass, the drafts got better and the cost per email dropped. That finding shaped how we build client projects now, and we wrote it up in single call vs agent chains.

The point is not the specific architecture. The point is that a narrow, checkable, measurable first project taught us something general in a few weeks, which is exactly what a first project is for.

What it should cost and how long it should take

A first project that passes the filter is typically a few weeks of agency work: a short discovery to pin down the task and the metric, a build, and a couple of weeks of supervised use with real inputs. If the quote you receive runs beyond six to eight weeks, or requires standing up new infrastructure before anything is usable, the scope has drifted. Push back on scope before you push back on price.

Model costs for an internal task at this scale are usually a rounding error next to the build. Our outreach engine runs on a locally hosted model and a self-hosted crawler, so the per-email cost is effectively hardware we already own. The architecture change mattered more than the model choice: collapsing three model calls into one cut the cost per email, and it did so while improving the drafts. The exception is high-volume document processing, where model choice does matter; even there, a small or self-hosted model is often enough once the task is narrow, and you can test that during the build rather than deciding up front.

Also budget for the boring part: logging every input and output from day one. We could only compare the chain against the single prompt because every crawled page and every draft had been stored. Without that log you cannot measure error rates, you cannot compare two approaches, and you cannot answer the "did it work" question in week six.

How to run the six weeks

Week zero: write down the task, the current baseline for your metric, and who will check outputs daily.
Weeks one to three: build. Insist on seeing real inputs, not synthetic examples, by the end of week one.
Weeks four to six: run it alongside the human. The human still does the task; the AI output is compared against theirs. Count agreements and failures.
End of week six: read the metric. Decide to expand, keep as is, or stop. Any of the three is a good outcome, because you now have evidence.

If it worked, the second project is usually obvious: the next step in the same workflow, or the same task for a neighboring team. If it did not, you have learned where your data or your domain breaks a model, for the price of a few weeks rather than a quarter.

If you have a candidate in mind and want a second opinion on whether it passes the filter, let's talk.

Originally published on the Pykero blog.

Build vs Buy Internal Tools: When Founders Should Build Their Own

Pykero — Sat, 31 Oct 2026 09:07:42 +0000

Build an internal tool when it touches your core data or core workflow, when per-seat SaaS pricing scales faster than your headcount justifies, or when the off-the-shelf version forces your team to work around it every day. Buy when the tool is a commodity, when your need is standard, and when the vendor's uptime and security are things you would rather rent than own. AI-assisted development has moved the line toward building, but it has not moved the line on maintenance, and that is where most founders get the math wrong.

Why the question is back on the table

For a decade the default advice to founders was simple: never build what you can buy. Engineering time was scarce, SaaS was cheap, and every hour spent on an internal tool was an hour not spent on the product.

That advice was written for a world where a "small" internal tool took a team a quarter. It no longer does. Developers now routinely rebuild a scoped version of a commercial product, a browser screen recorder, a photo search tool, a lightweight CRM, in a weekend with an LLM doing most of the typing. The cost of the first version has collapsed.

But the first version was never the expensive part. Owning the thing is. Our own outreach engine, described below, took far less time to get working than it has taken to keep working: scrapers break, model versions change, and one engineer still spends part of every month on it. The first version was cheap. The second year is the real price. So the question has changed from "can we afford to build this" to "can we afford to keep this alive", and that needs a different framework.

The four questions that decide it

1. Does the tool sit on your core data?

If the tool reads or writes the data that makes your business valuable, customer records, usage events, clinical notes, pricing logic, then buying means handing your most important asset to a vendor's data model and export policy. You will spend months fighting their schema and their API limits. Tools on core data lean strongly toward build.

If the tool holds data you would not miss, meeting recordings, design mockups, standup notes, buy it and move on.

2. How does the SaaS price scale?

Per-seat pricing is the trap. It looks trivial at five people and becomes a line item the finance lead asks about at forty. Do the three-year arithmetic before you sign.

An illustrative example: a tool at $30 per seat per month for 40 people is $14,400 a year, or $43,200 over three years. That is before the inevitable tier upgrade when you need SSO or an audit log. A focused internal tool built in two to three weeks and maintained a day or two a month usually lands well under that over the same period.

Usage-based pricing is friendlier. If you only pay for what you use and usage tracks revenue, buying is often fine even at scale.

3. Who fixes it when it breaks?

This is the question founders skip. An internal tool has dependencies that rot, a third-party API that changes its auth flow, a database that needs a migration, and a security patch that lands on a Friday. If nobody owns it, it quietly becomes the kind of debt we describe in technical debt management: working, unloved, and terrifying to touch.

Rule: an internal tool with no named owner is a liability, not an asset. If you cannot name the person who will maintain it next year, buy.

4. Will you keep changing it?

If your requirements for the tool shift weekly, because you are still learning your own process, SaaS will fight you at every step. Custom fields, Zapier glue, and workaround spreadsheets pile up. Building wins when the tool is a moving target.

If the process is stable and standard, invoicing, expense reports, calendar booking, the vendor has already solved it better than you will.

Our own example: the outreach engine

We run our own outbound at the agency, and this decision came up for us directly. The off-the-shelf personalization tools either produced generic emails or charged per enrichment credit in a way that scaled badly with the volume we wanted.

So we built it. A self-hosted Firecrawl instance scrapes each prospect's site, a local LLM reads the result, and the engine writes one tailored email per company. It runs on hardware we already had, so the marginal cost per prospect is close to zero.

The part that mattered for build versus buy was what happened after the first version. Our initial pipeline chained several steps: extract facts about the company, plan the angle, then draft. When we collapsed that into a single call that extracts and drafts together, it came out ahead on both cost and quality. We could only learn that because we owned the tool and could change the prompt architecture in an afternoon. With a SaaS product that experiment is not available to you. We cover the general pattern in single call vs agent chains, and the specific case in AI cold email agent: build vs buy.

The honest cost of that decision is that one engineer spends part of every month keeping it healthy, because scrapers break and model versions change. We consider that worth it. You should decide whether your equivalent is.

What a good "build" looks like

If the four questions point toward building, keep it disciplined:

Scope to one workflow. Not "an internal CRM", but "the view the sales lead needs every Monday morning". You can widen it later.
Use a boring stack. For us that is Next.js, PostgreSQL or Supabase, and TypeScript. Sometimes it is a script and a cron job. The goal is that any engineer on the team can fix it.
Time-box the first version. One to three weeks. If it does not fit, the tool is bigger than you think.
Name an owner and budget their time. Treat it as a small product with a backlog, not a side project.
Instrument it. A tool nobody knows is broken is worse than no tool. Log runs, alert on failures.

When buying is clearly right

Some categories are not close:

Authentication, payments, transactional email. The vendor's security and deliverability are the product. Do not own these.
Anything with a compliance surface. If the tool will be in scope for SOC 2, HIPAA, or a data-residency audit, you want a vendor's attestation, not your own.
Commodity collaboration. Video calls, docs, chat. Your version will be worse and nobody will thank you.
Anything you need this week that is not core. Buy it, set a calendar reminder for the renewal date, and re-run this framework then.

A rule you can apply in one meeting

Score the four questions. If two or more say build, build a time-boxed first version for one workflow with a named owner. If fewer than two do, buy, and revisit at the next renewal.

Run our outreach engine through it and you can see the scoring in practice. Core data: partly, since prospect research feeds our pipeline but we would survive losing it. Pricing: the per-enrichment-credit model scaled badly with the volume we wanted, so build. Owner: one engineer, named, with part of each month budgeted, so build is allowed. Changing it: we rewrote the prompt architecture from a multi-step chain to a single call once we had data, so build. Three of four said build, and the decision has held.

Write the decision and the date down, because the right answer changes as the company grows and the honest thing is to re-check it rather than defend it.

Where an agency fits

Most founders do not lack the judgment to make this call. They lack an engineer with two free weeks. That is the gap we fill: a scoped internal tool, built on a stack your team already knows, handed over with an owner's manual rather than a dependency. If you are staring at a SaaS renewal and wondering whether to build instead, let's talk.

Originally published on the Pykero blog.

Self-Hosted Speech-to-Text vs Cloud API: What It Really Costs

Pykero — Thu, 29 Oct 2026 09:05:42 +0000

Cloud speech-to-text costs you per minute of audio and nothing else, so it wins when volume is low, spiky, or unknown. Self-hosted speech-to-text costs you a GPU, an engineer who understands inference, and an on-call rota, so it wins when volume is steady, audio cannot leave your jurisdiction, or you need accuracy the cloud providers do not deliver for your language. Most founders should start on an API behind a thin abstraction and move only when one of those three triggers fires.

The two bills look nothing alike

The reason this decision goes wrong is that the two options are priced in different units, and people compare the wrong ones.

A cloud API bill is linear and visible. Every minute of audio has a price, usually with a discount tier at high volume. Diarisation, timestamps, and custom vocabulary often cost extra. There is no fixed cost, no idle cost, and no capacity planning. Your finance team can read the invoice.

A self-hosted bill is step-shaped and partly hidden. The visible part is the GPU: a rented instance or a card you own. Open models like Whisper and the newer transcription models from NVIDIA and others can process an hour of audio in a fraction of real time on a single modern GPU, which is why "one GPU, no API bill" headlines keep appearing. The hidden part is everything around the card:

An engineer who can serve the model, batch requests, and handle out-of-memory failures
Monitoring, autoscaling or a queue, and someone to answer when it stops at 3am
Model updates, regression testing, and re-benchmarking every quarter
Storage and retention for audio that now lives on your infrastructure

If you only compare GPU rent to API minutes, self-hosting always looks cheaper. If you add one quarter of a senior engineer's time, it often does not, until the volume gets large.

A useful way to frame the comparison

Ask one question: does the workload keep a GPU busy most of the day, every day? A call-centre recording pipeline, a telehealth platform transcribing every consultation, or a media archive doing a backlog of thousands of hours will. A SaaS feature that transcribes a few meetings per customer per week will not. Idle GPU time is the single biggest reason self-hosted projects end up more expensive than the API they replaced.

We learned a version of this on our own outreach engine, which is not speech but shares the economics. We scrape each prospect's website with a self-hosted crawler and run a local model to write one tailored email per company. That setup pays for itself because the job runs in a steady batch every day, latency does not matter, and we control the whole pipeline. The other lesson from that project was that a single model call that extracts the facts and drafts the email beat a chain of smaller calls on both cost and quality. The same discipline applies to transcription pipelines: fewer, well-designed steps are cheaper to run and easier to debug than an elaborate chain. We wrote more about that in single call vs agent chains.

When self-hosting wins regardless of price

Three situations make the cost comparison secondary.

1. The audio cannot leave your region or your tenant

Clinical consultations, legal recordings, government calls, and financial advice all carry data-residency and confidentiality obligations. Under the GDPR a cloud transcription provider is a processor you must contract with and audit. Under HIPAA you need a business associate agreement. In Saudi Arabia and the UAE, regulators increasingly expect health and government data to stay in-country, which several transcription APIs still cannot guarantee. We covered the regional detail in AI data residency in Saudi Arabia and the UAE.

Self-hosting removes the third party from the audio path. It does not remove the work: encryption at rest, access logs, retention and deletion, and a clear answer to "who can listen to this file" are now your responsibilities. Compliance moves in-house rather than disappearing.

2. The cloud providers are weak on your language or domain

General-purpose APIs are excellent on clean English and good on major European languages. They degrade on dialectal Arabic, code-switched speech, noisy clinical environments, and specialised vocabulary such as drug names. If your product lives in one of those niches, an open model you can fine-tune on your own recordings will often beat the API, and fine-tuning is only practical when you host the model yourself. See what Arabic speech recognition really costs for the dialect-specific picture.

Public comparisons such as the Open ASR Leaderboard are a reasonable starting point, but they measure English benchmarks. Nothing replaces running fifty of your own recordings through each candidate and counting the errors that matter to your users.

3. Volume is high and predictable

Once you are transcribing thousands of hours a month on a steady schedule, per-minute pricing starts to dwarf the fixed cost of a GPU and an engineer. This is the only one of the three triggers that is purely financial, and it is the one founders most often claim prematurely. The test is the same one from the framing above: pull last month's audio hours by day and by hour, and ask whether a single GPU would have been busy for most of that time. A telehealth platform recording every consultation during clinic hours passes. A product whose transcription load lives in a Monday-morning spike and a quarterly projection does not, however large the projected total. Our outreach engine qualified because the batch ran every day at the same size. Check the actual usage curve, not the projected one.

When the API wins, and why that is usually the answer for v1

You have not validated the feature yet. Paying per minute for an experiment is cheaper than paying for a GPU that sits idle while you find out nobody uses transcription. The idle GPU is the quiet killer here: it costs the same whether it processes ten hours or none.
Volume is spiky. A product that transcribes nothing overnight and everything at 9am needs either an oversized GPU or a queue that users will notice. APIs absorb spikes for you.
You need extras on day one. Diarisation, word-level timestamps, language detection, and streaming are solved problems in the mature APIs. Reproducing them on an open model is weeks of work.
Your team is small. Inference serving is its own skill. If nobody on the team has run a GPU service in production, the quarter of a senior engineer from the cost section becomes closer to a full one while they learn, and the 3am page lands on a founder. That is the moment the API bill you were trying to avoid starts to look cheap.

The architecture that keeps both doors open

Whichever you choose, do one thing from the first commit: put transcription behind a single interface in your codebase. One function that takes audio and options and returns a transcript with timestamps and a confidence field. Every provider adapter, cloud or self-hosted, implements that interface.

This buys you three things:

A switch that is a config change. Moving from API to self-hosted, or running both and routing by customer region, becomes a deployment, not a rewrite.
A fair benchmark. You can run the same test set through every adapter and compare word error rate, latency, and cost on identical inputs.
A fallback. When the self-hosted service is down or the queue is deep, overflow can go to the API. We described the same pattern for language models in multi-LLM provider failover.

Keep the interface boring. Audio in, structured transcript out, provider named in the metadata so you can trace every result back to the model that produced it.

A decision checklist for the next planning meeting

Measure real volume in audio hours per month, and look at the shape of the curve, not just the total.
Write down the residency constraint as a sentence a lawyer would sign off on, or confirm there is none.
Benchmark on your own audio. Fifty representative recordings, two or three candidates, errors counted by a human who knows the domain.
Cost both options fully. API: minutes times rate plus extras. Self-hosted: GPU plus a realistic fraction of an engineer plus on-call.
Start on the API unless residency or accuracy forces your hand, and build the abstraction so the move is cheap later.

If you are weighing this for a healthcare, government, or Arabic-language product, we have done the benchmarking and the hosting on both sides and can give you a straight answer for your volume and your constraints. Let's talk.

Originally published on the Pykero blog.

n8n vs custom code for AI automation: what to buy, what to build

Pykero — Sun, 25 Oct 2026 09:05:38 +0000

Use a workflow engine like n8n when the automation is internal, low volume, and a failed run is an annoyance rather than a lost customer. Write custom code when the flow touches customers, runs thousands of times a day, needs real tests, or contains an LLM prompt you will keep tuning. The expensive mistake is not picking the wrong one. It is picking one and refusing to move when the signals say you have outgrown it.

Why this question comes up now

n8n has become the default answer when a founder says "we should automate this with AI". It is source-available under n8n's own Sustainable Use License, you can self-host it, and it ships nodes for most SaaS tools plus generic LLM and HTTP nodes. A non-engineer can build something that works in an afternoon. That is genuinely valuable, and it is also exactly why teams end up with a dozen fragile workflows nobody dares to touch.

The decision is less about n8n versus code and more about which parts of your automation deserve engineering discipline. We covered the conceptual split between agents and workflows in AI agents vs workflows. This post is the practical, money-on-the-table version.

What n8n is actually good at

Be honest about the strengths, because they are real:

Speed to a first working version. Auth to Gmail, HubSpot, Slack, Notion, and Postgres is a dropdown, not a week of OAuth plumbing.
Visibility for non-engineers. Ops and sales people can read a canvas. They cannot read a NestJS service.
Cheap experiments. You can test whether an LLM step adds value before anyone writes a line of production code.
Internal glue. "When a deal closes, create a folder, post in Slack, draft an onboarding email" is the sweet spot. Volume is low, failure is visible, and a human is nearby.

If your automation lives entirely in that list, stop reading and go build it in n8n. Hiring an agency for this is overkill.

Where it starts to hurt

The problems arrive gradually, and each one alone looks tolerable.

Prompts become untestable

An LLM prompt inside a node is a string in a database. There is no diff, no review, no eval suite. When output quality drifts, the only debugging tool is re-running the workflow and squinting at the result. Compare that with how the prompt in our own outreach engine lives: a version-controlled file, a fixed set of prospect websites as fixtures, and a strict output schema every drafted email has to satisfy. When we swapped the call to action from a calendar link to "reply YES", that was a one-line diff someone could review and roll back. When extraction accuracy drops, a test fails in CI instead of a salesperson noticing a week later. If you are serious about reliability, read deterministic guardrails for AI features. Almost none of that is ergonomic inside a visual editor.

Error handling is where visual tools lose

A real pipeline needs: retries with backoff, idempotency so a retry does not send the email twice, dead-letter handling for inputs the LLM cannot parse, and alerting when a scheduled run silently does not happen. Take the outreach engine again. A Firecrawl crawl can time out on a slow prospect site, the local LLM can return something that fails the output schema, and a retried run must not send a second cold email to the same company. Each of these is possible in n8n. Each one is also a tangle of extra nodes and branches that makes the canvas unreadable, which defeats the one advantage the canvas had. In code, these are a few well-understood patterns: a retry wrapper around the crawl, a schema check that routes failures to a review queue, and a per-company sent flag checked before anything goes out.

Volume changes the economics

At ten runs a day, nobody cares how a workflow is executed. At ten thousand, you care about concurrency, queue depth, and how execution logs are stored and pruned. n8n can be scaled in queue mode, but at that point you are operating infrastructure anyway, and you have taken on the ops cost of a workflow engine without getting the testability of code. You might as well own the code.

Ownership and lock-in

n8n's license is permissive for internal use but restricts offering it as a hosted service to others. If the automation is part of what you sell, check the license with someone who reads licenses, and assume the logic should live in code you fully own. The same goes for anything with compliance exposure: audit logs, data residency, and access control are easier to prove when the flow is a service you control end to end.

A concrete example from our own stack

We run our outreach engine on custom code, not a workflow tool, and the reason is instructive. The pipeline crawls each prospect's website with a self-hosted Firecrawl instance, hands the content to a local LLM, and produces one tailored email per company. The first instinct was a multi-step chain: one call to extract facts, another to pick an angle, another to draft. We tried it and also tried a single call that extracts the facts and drafts the email together, with a strict output schema. The single-call version won on both cost and quality. We go deeper on why in single call vs agent chains.

The relevant point here: discovering that took dozens of controlled runs against the same fixture set, comparing outputs side by side, and changing one variable at a time. That is a test harness, and it is trivial in a TypeScript repo. It is miserable in a drag-and-drop canvas. Had we built the engine in n8n, we would probably still be running the slower, more expensive chain because the cost of experimenting would have been too high.

The second lesson from the same tool: the biggest lift in reply rate came from a copy change, not a model change. Moving the call to action from a calendar link to a soft "reply YES" consistently got more responses from cold prospects. That change took one line in a version-controlled prompt. Being able to make and measure it cheaply is the whole argument for code when the flow matters to revenue.

A decision framework you can apply in ten minutes

Score each automation on these five questions. Three or more "yes" answers means build it in code.

Does a customer see the output directly? Emails, WhatsApp replies, quotes, support answers.
Will it run more than a few hundred times a day within a year?
Does it contain an LLM step whose quality you will keep tuning? Extraction, classification, drafting.
Does a failed or duplicated run cost money or trust? Double-charging, double-sending, missed leads.
Does it touch regulated or sensitive data? Health, finance, government, personal data of EU or Gulf residents.

Zero or one "yes": n8n, built by whoever owns the process. Two: n8n for the orchestration, but keep prompts and business rules in a small code service that n8n calls over HTTP. This hybrid is underrated. It keeps the readable canvas for ops while giving engineering a tested, versioned core. Three or more: code, with n8n at most as a trigger source.

What the hybrid looks like in practice

A small NestJS or Next.js API with one endpoint per capability: extract-company-facts, draft-email, qualify-lead.
Prompts, Zod schemas for structured output, and fixtures live in the repo and run in CI.
n8n handles the boring edges: cron triggers, pulling from the CRM, posting to Slack.
When volume or reliability requirements grow, you replace the n8n orchestration with a queue. Nothing about the logic changes.

This also makes the hiring question easier. You can get the n8n wiring done by an ops person or a freelancer. You bring in engineers for the service, which is the part that needs tests and a maintenance plan. For what that ongoing cost realistically looks like, see the real maintenance cost of an AI agent.

What to ask whoever you hire

"Which parts of this would you deliberately keep out of n8n, and why?" A good answer names prompts, retries, and idempotency.
"How will we test a prompt change before it goes live?" If the answer is "run the workflow and check", keep looking.
"What happens when the LLM returns garbage at 3am?" You want to hear dead-letter queue, alert, and a human review step.
"Who owns the prompts and schemas, and where do they live?" The answer should be "you, in your repo".

The short version

n8n is a great prototyping tool and a fine home for internal glue. It is a poor home for the AI logic that drives revenue, because that logic needs experiments, tests, and version history, and visual tools make all three expensive. Prototype in n8n, prove the value, then move the core into code you own. If you want help drawing that line for your specific stack, let's talk.

Originally published on the Pykero blog.

Why Your AI Agent Sent That Message Twice, and What to Require Before Launch

Pykero — Fri, 23 Oct 2026 09:05:15 +0000

Your AI agent sent the message twice because something underneath it retried, not because the model "decided" to. Webhooks, queues, workers and LLM API clients all default to at-least-once delivery, and an agent that calls the outside world without an idempotency key will eventually double-send, double-book or double-charge. The fix is architectural and cheap compared to the damage, so it belongs in the requirements, not in a post-launch patch.

Retries are the default, not the exception

Founders tend to picture an agent as a single conversation: a customer writes, the model thinks, a reply goes out. In production it is a chain of hops, and nearly every hop will repeat itself under stress.

The inbound webhook. Messaging platforms resend a notification when your endpoint is slow to respond or returns an error. If your handler already started working before it timed out, you now have two copies of the same event in flight.
The queue. Standard queues such as Amazon SQS are explicitly documented as at-least-once: a message can be delivered more than once, and your consumer is expected to cope (AWS docs). Most managed agent runtimes sit on something equivalent.
The worker. A crash, a deploy or an out-of-memory kill mid-task means the job runs again from the start. Everything it already did, it does again.
The model call. Every serious LLM SDK retries on rate limits and transient errors. If the first attempt actually succeeded server-side but the response was lost, you pay twice and, worse, your code may act on both.
The tool call. The agent calls your send-message or create-order function. That function talks to a third-party API with its own timeouts. A timeout is not a failure; it is an unknown.

None of these is exotic. They are the normal behaviour of normal infrastructure. An agent that is "99% reliable" in a demo is being tested at a load where none of this fires. In month two, with a few thousand conversations a day, all of it fires.

Why the LLM makes this harder than a classic job

A conventional worker that retries at least behaves the same way the second time. An agent does not. If a worker re-runs a planning step, the model may pick a slightly different action: the first attempt sent "Your order ships Tuesday", the retry sends "Your order ships tomorrow" because the context window now includes a half-written state. The customer sees two confident, contradictory messages from the same business.

This is why prompt-level fixes fail. Telling the model "never send a message twice" assumes the model knows the first attempt happened. It often does not, because the failure occurred after the model returned and before your code recorded the result. Deduplication has to live below the model, at the boundary where your system touches the world. This is the same principle behind the deterministic guardrails we cover in reliability for AI features: the model proposes, the code decides whether that action is still allowed.

What exactly-once actually means for an agent

You cannot get exactly-once delivery from the network. You can get exactly-once effect by making every external action idempotent. Three pieces do the work.

An idempotency key per intended action

Before the agent sends anything, your code derives a key from what the action is, not from the attempt. For a reply, that is typically the inbound message ID plus the conversation step. For an order, the cart ID plus the customer. Two retries of the same intent produce the same key; two genuinely different actions do not. Stripe's API is the reference design here: the client supplies the key, the server stores the first result and returns it on repeats (Stripe docs). Your agent's send function should behave the same way, whether the downstream API supports keys natively or you enforce it yourself.

Reserve, then act, then record

The failure that bites hardest is "sent but not recorded". The agent calls the messaging API, the API accepts the message, your process dies before writing to the database, and the retry sees no record and sends again. The fix is to reserve first: insert the key into an actions table with status pending before calling out. A unique constraint on the key means a concurrent or repeated attempt fails at the insert and stops (PostgreSQL constraints). After the external call succeeds, update the row to sent with the provider's message ID. If you find a pending row older than your timeout, that is a reconciliation case, not a resend case: query the provider by key, or hold for a human.

Dedupe inbound events at the edge

Webhook handlers should acknowledge fast and store the event ID in a dedupe table with the same unique-constraint trick. A duplicate delivery hits the constraint and is dropped before any model is invoked. This also saves money: every duplicate you drop at the edge is an LLM call you never pay for, which matters at the volumes we discuss in what a WhatsApp sales agent costs.

What we learned on our own outreach engine

Our cold-outreach system scrapes each prospect's site with a self-hosted crawler and a local model, then drafts one tailored email per company. Early on we assumed the hard problem was writing quality, and it partly was. The pattern that won on both cost and quality was a single model call that extracts facts and drafts the email together, rather than a multi-step chain. An underrated benefit showed up later: a single call has one output to make idempotent, so the "one email per company" promise could be enforced with one reservation row keyed on the company domain. Each extra step in a chain is another place where a retry can double-fire, so collapsing the chain simplified the reliability story as much as it simplified the bill. We wrote up the drafting side in single call vs agent chains.

What to require before an agent touches a real customer

Put these in the statement of work or the vendor contract. Each is a yes or no question with a demonstrable answer.

Every outbound action has a named idempotency key and the derivation is documented per action type.
Reservation before send. Show the table, the unique constraint and the state machine. Ask what happens to a row stuck in pending.
Inbound dedupe at the webhook, with the retention window stated. Platforms can redeliver hours later.
Retry-safe model calls. The SDK retries are fine, but the surrounding code must not treat "no response" as "nothing happened".
A chaos test in acceptance. Kill the worker mid-send, replay the same webhook three times, force a timeout on the messaging API. Zero duplicates reach the test number. If the agency cannot run this test, they have not built for it.
Reconciliation, not resend, as the default when state is unknown. A human-visible queue for unknowns beats an automatic retry every time.
Observability on duplicates. A counter for constraint hits is your early warning that the infrastructure is retrying more than expected, long before a customer complains. Pair it with the alerting from monitoring missed agent runs.

The cost conversation

Making an agent retry-safe is not a large line item. For a typical sales or support agent with three or four action types, expect a few days of focused engineering, most of it in designing keys and the reservation flow, plus a day for the chaos tests. The expensive version is the one where you skip it: refunded double charges, a messaging number flagged for spam-like behaviour, and the slow erosion that happens when customers learn your agent is not quite trustworthy. If a proposal for an agent that sends real messages does not mention idempotency anywhere, that is the question to ask before signing.

If you are scoping an agent that will act on behalf of your business and want the reliability requirements written down before the build starts, let's talk.

Originally published on the Pykero blog.

Your Agency Is Writing Your Code With AI. Here Is What to Require Before You Sign

Pykero — Wed, 21 Oct 2026 09:04:59 +0000

Yes, your agency is almost certainly using AI to write part of your code, and that is fine. What is not fine is AI-drafted code that no human read carefully and no test exercises, merged on a Friday because the demo was due. The fix is not a tool ban. It is a short list of controls you put in the contract and the repository before the first sprint.

Why this matters more than it did two years ago

A developer with a coding assistant can produce a working-looking feature in an afternoon that used to take three days. That is real. The problem is that review, testing and understanding did not get faster at the same rate. If an agency keeps the same review habits it had before and triples its output, you get the same bug rate per line applied to three times as many lines.

Technical debt used to accumulate at human speed. Now it accumulates at machine speed, and it looks clean on the surface. The first version of our own outreach engine, described in section 4, was a five-step chain that read beautifully: every step well named, every function confidently commented, nothing a reviewer would stop on. It also had five places to fail silently, and the clean surface was exactly why nobody questioned the four extra steps. AI-written code is well formatted and well named, which is precisely what makes a weak reviewer skim it. We wrote about the general discipline in technical debt management. This post is about the specific version that shows up when a vendor is writing the code.

The five things to require

1. A test gate that blocks merges, not a testing promise

"We write tests" is a sentence. A gate is a configuration. Ask for both of these before work starts:

Branch protection on main so nothing merges while the pipeline is red. GitHub documents this under protected branches, and every major host has an equivalent.
A coverage threshold enforced in CI, not reported in a slide. On a TypeScript stack this is a few lines in a Vitest or Jest config. The number matters less than the fact that it fails the build when it drops.

Then ask one question that separates agencies: "Can you show me a pull request from last month that was blocked by a failing test?" If the answer is "our tests never fail", the tests are not testing anything.

2. A named human who reads every AI-drafted diff

Assistants produce code. They do not take responsibility for it. Your contract should name who does, and the repo should prove it: every pull request has a reviewer who is not the author, and the review has comments, not just a green checkmark.

Watch for two anti-patterns:

Self-merged pull requests. One person prompts, one person approves, same person. That is zero review.
Review by AI only. AI review bots catch style and obvious bugs. They do not know that your pricing rule has a VAT exception for Saudi customers. The human who knows the domain must read the diff.

3. A dependency and secrets policy

AI assistants love adding packages. They will reach for a library to pad a string. Each one is a supply-chain surface and a license question. Require:

A lockfile committed and a rule that new dependencies are called out in the pull request description.
No secrets, customer data or proprietary code pasted into tools that train on inputs. Most enterprise tiers of the major assistants offer a no-training setting; the contract should say the agency uses one.
Clear assignment of all deliverables to you, however they were produced. Good agencies already do this. The AI clause just closes the "but a tool wrote it" loophole.

4. Fewer lines, not more

The cheapest code to test is the code that was never written. AI makes it painless to generate a six-step pipeline where one step would do, and every extra step is a place for a silent failure.

We hit this in our own outreach engine. It scrapes a prospect's site with a self-hosted crawler and uses a local model to draft one tailored email per company. The first version was a chain: extract facts, summarise, pick an angle, draft, polish. It was slower, pricier and, more importantly, it had five places to go wrong and five things to test. Collapsing it into a single call that extracts the facts and drafts in one pass beat the chain on both cost and output quality, and it halved the surface we had to cover with tests. We wrote the general case up in single call vs agent chains.

Ask your agency how they decide when generated code is too much code. If they do not have an answer, you will pay for the extra lines twice: once to write and once to maintain.

5. A codebase you could hand to someone else

The real test of AI-written code is whether a new engineer, with or without an assistant, can read it in a week. Require:

Architecture decisions written down in the repo, in a short docs folder or ADR file, not in a chat history. Write down the why, not just the what. If your agency collapses a five-step chain into one call, as we did, the reason belongs next to the code, or the next engineer with an assistant will rebuild the chain, because a chain is what assistants reach for.
Consistent patterns across features. AI assistants drift. Feature A uses one data access style, feature B another, because they were prompted on different days. Linting and a small set of templates fix most of this.
Integration tests for the flows that make you money: signup, checkout, the webhook from your payment provider. Unit tests on helpers are cheap filler. The Saudi VAT exception from section 2 is the kind of rule that only an end-to-end test on a real checkout will catch, and covering the three paths like that with a tool like Playwright is what protects revenue.

What this does to the quote

Expect fair quotes to drop on boilerplate: CRUD screens, forms, admin tables, standard integrations. Expect them to stay roughly flat on the parts that were never typing-limited: the data model, the edge cases in your business rules, performance, and the review itself.

An agency whose quote dropped 60 percent across the board since adopting AI tools has either cut review or is about to. One whose quote did not move at all is probably not passing the savings through. The honest middle is a lower number on the predictable work and a line item for testing and review that did not shrink. If you are weighing engagement models, fixed price vs time and materials covers how to structure that.

Questions to ask on the first call

Which AI coding tools do you use, and on what tier?
Show me your branch protection and CI config on a recent project.
Who reviews AI-drafted code, and can I see a pull request with real review comments?
How do you decide when generated code is too much code?
What happens to my code and data inside the tools you use?

Clear, specific answers mean the agency has thought about this. Vague reassurance means you are the first client to ask. Pair these with the broader software project red flags and you will screen out most of the risk in a single conversation.

The bottom line

AI coding tools are a speed multiplier. Multiply a disciplined team and you get a disciplined team that ships faster. Multiply a sloppy one and you get sloppy at scale, with a codebase that looks great until the first real incident. Your job as the buyer is not to pick the tool. It is to require the gate, the reviewer, the policy and the readable repo, and then check that they exist.

If you want a build where the tests are in the pipeline and the reviewer is named before the first commit, let's talk.

Originally published on the Pykero blog.

Small LLM vs Frontier Model: Which One Does Your AI Feature Actually Need?

Pykero — Mon, 19 Oct 2026 09:03:13 +0000

Most AI features inside a SaaS product do not need a frontier model. If the task is bounded (classify a ticket, extract fields from a document, draft an email from facts you already have), a small model with a strict prompt and an output validator is usually good enough and costs a fraction per call. Reserve the frontier model for open-ended reasoning, long multi-step tool use, and anywhere a wrong answer costs real money.

The question is not "which model is smartest"

Founders tend to ask us which model is best. That is the wrong question, because the best model on a leaderboard is rarely the best model for a feature that will run fifty thousand times a month inside your product. The right question is: what is the cheapest model that passes my eval set at the accuracy I need, with the guardrails I can afford to build?

Take the ticket-classification example from the top of this post and make it concrete. At fifty thousand tickets a month, the leaderboard winner and a distilled "flash" model will both tag the obvious tickets the same way. The difference shows up on the ambiguous ones, and a validator that checks the tag against your allowed list and retries on a miss catches a good share of what the small model gets wrong. The only way to know whether the remainder is acceptable is to run the fifty to a few hundred real tickets in your eval set through both models and compare, with the per-call price next to each result.

Framed that way, the decision stops being a bet on vendor marketing and becomes an engineering measurement. The small-model category has moved fast; the recent wave of "flash" and distilled releases from several labs means a model that would have been state of the art two years ago now costs very little per token and runs on a single GPU. Whether that is enough for you depends on four things.

Four questions that decide it

1. Is the task bounded?

A bounded task has a known input shape, a known output shape, and a small set of acceptable answers. "Pull the invoice number, due date and total from this PDF" is bounded. "Read this customer's last six emails and decide what to do" is not.

Small models are strong on bounded tasks because the prompt can carry most of the intelligence: the schema, the rules, three worked examples. They degrade on unbounded tasks because they need to improvise, and improvisation is exactly what gets worse as parameter count drops.

2. Can you check the output cheaply?

If a deterministic check can catch most bad outputs, you can run a smaller model and let the validator do the heavy lifting. JSON schema validation, regex on dates and amounts, a lookup against your own database, a length cap: these cost nothing per call and turn a model's occasional mistake into a retry instead of a bug a customer sees. We wrote about this pattern in deterministic guardrails for AI feature reliability.

If the only way to check the output is for a human to read it, the model has to be right the first time more often, and that pushes you up the price list.

3. What does a wrong answer cost?

A mis-tagged support ticket costs an agent ten seconds. A wrong dosage summary in a healthcare tool, or a wrong refund decision in an e-commerce agent, costs a lot more. Price the failure, not the call. Where the failure is cheap, optimise for cost per call. Where the failure is expensive, the frontier model's per-token premium is trivial next to the risk, and you should also add a human review step regardless of model.

4. Does the input fit, and does it need memory?

Small models typically have shorter usable context and get lost sooner in long documents. If your feature needs to hold a 200-page contract in view, or carry state across a twenty-step workflow, you are either chunking aggressively (more engineering) or paying for a model that can hold it. Multi-step agentic work with tools is the clearest case for frontier models, and it is worth reading agents versus workflows before you assume you need the agent at all.

What a small model is actually good at: our outreach engine

Our own outbound tool is a concrete example. It crawls each prospect's website with a self-hosted Firecrawl instance, hands the page text to a local model, and writes one tailored email per company. The task is bounded: the input is a few pages of text, the output is a short email following our template and tone, and a validator rejects anything that is too long, mentions a product we do not offer, or fails to cite a fact from the site.

What surprised us was that a single call doing "extract the facts and draft the email" beat a multi-step chain (extract, then plan, then write) on both cost and quality, even with the smaller model. Fewer hops meant fewer places for the model to drift from the source text. We covered the general case in single call versus agent chains. The point for a buyer: a well-constrained small model plus a validator was the right tool, and a frontier model would have added cost without changing the reply rate.

When you should pay for the frontier model

Be honest about the cases where cheap is expensive. The outreach engine above gets away with a small model for one reason: there is a single hop, page text in and email out, with a validator standing behind it. Remove any of those conditions and the picture changes.

Open-ended reasoning over messy input. Triage that needs judgement, not rules. Anything where the instructions say "use your discretion". Our validator can reject an email that cites no fact from the site; it cannot judge whether the model chose the right fact, and when that judgement is the whole feature, you want the stronger model.
Multi-step tool use. Agents that call several APIs, interpret results, and decide the next step. The moment our task became "crawl the site, decide which pages matter, check the CRM, then write", the small model would be carrying state across hops, which is exactly where it drops the thread.
Low-volume, high-value calls. If the feature runs a few hundred times a month and each result is reviewed by a senior person, the cost difference is noise. Buy accuracy.
Code generation and complex transformation. The quality gap is still wide here.
The first version, before you have evals. It is often cheaper to launch on a strong model, collect real traffic, build an eval set of fifty to a few hundred real examples from it, and then downgrade where the evals say you can.

That last one matters. Starting on a frontier model and moving down is a measured decision backed by an eval set. Starting on a small model and discovering in production that it is not good enough is a rewrite of your prompts and guardrails under pressure, with no eval set to tell you when you are done.

How the choice changes your quote

Founders expect the small model to make the project cheaper to build. It mostly does not. The build cost is in the prompt design, the validator, the eval set, the retry and fallback logic, and the integration into your product. That work is the same whichever model sits behind it. If anything, a small model needs a thicker validation layer, which is slightly more engineering, not less.

What changes is the running cost. Per-call prices between a frontier model and a small hosted model differ by a large multiple, and over a year of real traffic that difference can exceed the original build budget. Ask for a projection at three volume levels before you sign; our guide to LLM cost optimisation lists the levers that matter beyond model choice, such as caching and prompt length.

Self-hosting the small model is a separate decision. Tools like vLLM and Ollama make it technically easy, but it only pays off at steady volume or when data residency forces your hand. At typical early-stage volumes, a cheap hosted API model wins once you count the engineering time.

The hedge: build the switch, not the bet

Whatever you decide, do not let it be baked into the code. Insist on three things in the brief:

Model behind a single config value. Changing from a frontier model to a small one, or between providers, should be a deployment, not a sprint.
An eval set shipped with the feature. Fifty to a few hundred real examples with expected outputs, run before every model change. Without it, "is the cheaper model good enough?" is a guess. We argue this belongs in the contract in evals in AI vendor contracts.
A fallback path. When the small model fails validation twice, escalate that single call to the stronger model rather than failing the user. This gives you small-model economics with frontier-model worst-case quality.

Done this way, the model question becomes a dial you turn as traffic and evidence come in, instead of a decision you make once with no data.

If you are scoping an AI feature and want a straight answer on which model tier it needs and what it will cost to run, let's talk.

Originally published on the Pykero blog.

How to Brief an AI Agency So the Quote Is Actually Accurate

Pykero — Sat, 17 Oct 2026 09:03:47 +0000

An accurate quote for a custom AI project comes from a brief that answers five questions: what business outcome you want, what data you actually have, what a wrong answer costs you, which systems the agent must read from and write to, and what constraints are non-negotiable. Everything else, including the model, the framework, and whether it is "an agent" or "a workflow", is the agency's job to propose. Founders who send a two-page problem statement get a number. Founders who send a feature list or a vague "we want AI in our product" get a range so wide it is useless, or a lowball that gets renegotiated in month two.

Why most AI briefs produce bad quotes

A software agency prices risk. Every unknown in your brief gets converted into either a padded estimate or an optimistic one that falls apart when the real data shows up. AI projects have more unknowns than ordinary software because the hard part is rarely the code. The hard part is the data, the edge cases, and deciding how good is good enough.

The three most common brief failures we see:

Solution instead of problem. "Build a RAG chatbot over our docs with GPT and a vector database." The agency now has to reverse-engineer what you actually need, and often the prescribed design is the expensive one. We have been on the receiving end of this ourselves: the multi-step chain we assumed for our own outreach engine lost to a single call on both cost and quality, and a brief that had prescribed the chain would have locked in the worse design. We cover why this matters in RAG vs fine-tuning for founders.
No definition of correct. "It should answer customer questions accurately." Accurately compared to what? Who checks? What happens when it is wrong? "Right nine times out of ten and flags the rest for a human" is a definition. "Accurate" is not.
Integrations mentioned in passing. "It should also update the CRM." That sentence can double a project. An agent that drafts a CRM note for a human to approve and one that changes deal stages on its own are different builds, and the brief usually does not say which one is meant. Authentication, rate limits, field mapping, and failure handling for a third-party system are real work.

Fix those three and the quote you get back will be close to what you end up paying.

The five sections your brief needs

1. The outcome, in business terms

One paragraph. Not "an AI assistant" but "reduce time-to-first-response on inbound support from hours to under two minutes, without increasing escalations to the human team." A measurable outcome lets the agency propose the cheapest design that hits it, instead of the most impressive one.

Include the volume. Fifty requests a day and fifty thousand a day are different systems with different cost structures. If you do not know, give your best guess and say it is a guess.

2. The data you actually have

This section decides more of the price than anything else. Be specific and be honest:

Where it lives. Postgres, a Google Drive full of PDFs, a legacy CRM with an export button, WhatsApp threads on someone's phone.
How much and how clean. Ten thousand well-structured records is a different job from two hundred scanned forms in three languages.
Who owns it and whether it contains personal data. If the project touches EU residents' data, say so. The agency has to design for GDPR from day one, not retrofit it.

Attach a redacted sample. Twenty rows or ten documents is enough. No serious agency can estimate extraction or retrieval work from a prose description of data, and any that claims to is guessing.

3. The cost of a wrong answer

This is the question founders skip and agencies most need answered. An AI feature that suggests blog tags can be wrong a quarter of the time with no consequence. An agent that quotes prices to customers or files insurance claims cannot.

State, in plain language:

What happens when the output is wrong. Embarrassment, a refund, a regulatory problem, a patient safety issue.
Who catches it today. A human reviewer, nobody, the customer.
What error rate you would accept at launch. Even a rough figure helps. "If it is right nine times out of ten and flags the rest for a human, that beats what we have."

This single section determines whether the agency designs a cheap system with a human in the loop or an expensive one with layered verification. Our piece on deterministic guardrails for AI features explains why the second kind costs more and when it is worth it.

4. Systems the agent must touch

List every system the AI must read from or write to, and for each one: do you have API access, who administers it, and is there a sandbox. Writing is more expensive than reading. An agent that drafts a CRM note for a human to approve is a smaller project than one that updates deal stages on its own.

If a system has no API, say that too. Browser automation and screen scraping are possible but they are a different line item, and they break more often.

5. Hard constraints

Things the agency cannot trade away:

Data residency. Must stay in the EU, in Saudi Arabia, on your own servers.
Budget per request or per month. A ceiling on inference cost shapes model choice immediately.
Latency. A WhatsApp reply can take four seconds. A voice agent cannot.
Timeline and what is driving it. A trade show date is a real constraint. "ASAP" is not.
Legal or disclosure requirements. If customers must be told they are talking to a machine, say so.

Keep this list short. Every item here removes design options, and some of those options were the cheap ones.

What to leave out

The architecture. Do not specify the model, the vector database, the orchestration framework, or how many steps the pipeline should have. When we built our own outreach engine, which scrapes each prospect's website with a self-hosted crawler and drafts one tailored email with a local model, our starting assumption was a multi-step chain: one call to extract facts, another to plan, another to write. The version that won on both cost and quality was a single call that extracted the facts and drafted the email together. Had a client prescribed the chain in their brief, we would have built the worse system and charged more for it. Let the agency propose, then challenge the proposal.

A long feature list. Features without priority get priced as all mandatory. If you must list them, mark three as launch-critical and the rest as later.

Competitor screenshots as the spec. Useful as reference, dangerous as requirements. You end up paying to replicate someone else's compromises.

What a good agency will do with your brief

Take the support example from section one. A brief that says "under two minutes to first response, current volume roughly five hundred tickets a day, right nine times out of ten with the rest flagged to a human, must draft but not send, EU data only" should get back, within a week or two, a proposal that reads like an answer to those exact lines: a recommended approach with the reasoning, a pilot scope (perhaps one ticket category, drafts only), a cost broken into build, inference at your stated volume, and ongoing maintenance, and a proposed way to measure the two-minute target and the one-in-ten error rate before you pay for the full build. If the response is a single number with no assumptions listed, the agency did not read the brief. If it is a thirty-page deck that never mentions your data sample, same conclusion.

You should also expect questions back. Good agencies push on the cost-of-wrong-answer section and on the integrations, because that is where the real estimation risk lives. Expect to be asked who reviews the flagged tenth today, and whether the CRM has a sandbox. Agencies that ask nothing are either very experienced in your exact problem or planning to discover the problems on your invoice.

Before signing, use how to vet an AI agency demo to pressure-test what they show you, and make sure the success metric from section one makes it into the contract. Agreeing on how the system will be evaluated before the build starts is the cheapest insurance you can buy.

A template you can copy

1. Outcome: what changes for the business, with a number and a volume.
2. Data: where it lives, how much, how clean, personal data yes/no. Sample attached.
3. Cost of wrong: what happens, who catches it today, acceptable error rate at launch.
4. Systems: each one, read or write, API access yes/no, sandbox yes/no.
5. Constraints: residency, cost ceiling, latency, hard dates, disclosure rules.
6. Not in scope for v1: the things you are consciously deferring.

Two pages, plus appendices. It takes a founder an afternoon to write and saves weeks of back-and-forth, or a project that quietly doubles in price.

If you have a brief half-written and want a second pair of eyes before sending it to anyone, or you would rather talk the problem through first, let's talk.

Originally published on the Pykero blog.

AI agent human oversight: what the EU AI Act actually requires you to build

Pykero — Tue, 13 Oct 2026 09:02:37 +0000

Human oversight under the EU AI Act means a named person can understand what your agent is doing, override or stop it, and prove afterwards that they could. For most agents that comes down to four things you have to build: a capability sheet, a decision boundary enforced in code, a stop control that really halts the system, and a log that records every consequential decision together with who made it. Article 14 only binds high-risk systems, but if you sell into healthcare, HR, lending or the public sector, assume you are in scope and design for it now rather than retrofit it under procurement pressure.

Who this actually applies to

Article 14 sits in Chapter III of Regulation (EU) 2024/1689, the chapter that sets obligations for high-risk AI systems. Whether your agent is high-risk is decided by two annexes. Annex I covers AI that is a safety component of a regulated product, which includes medical devices. Annex III lists use cases: biometrics, critical infrastructure, education, employment and worker management, access to essential private and public services, law enforcement, migration, and the administration of justice.

Translate that into agency terms:

Not high-risk: a WhatsApp agent that answers product questions and recommends a size. Transparency rules under Article 50 still apply, and we covered those in disclosure rules for AI sales agents.
High-risk: an agent that triages patient messages, screens job applicants, scores a loan file, or decides eligibility for a public benefit. Same stack, same model, completely different obligations.

Annex III obligations were scheduled to apply from 2 August 2026. The timeline has been the subject of proposed amendments, so check the current consolidated text before committing to a date in a contract. What has not waited for the legal deadline is procurement: hospital and government buyers already ask for oversight documentation in tenders, which is why this matters commercially today.

What Article 14 asks for, in plain language

The article requires that high-risk systems are designed so natural persons can effectively oversee them while in use. The person doing the oversight must be enabled to:

understand the system's capabilities and limitations, and monitor its operation for anomalies
stay aware of automation bias, the tendency to over-trust the output
correctly interpret the output, with tools to do so
decide not to use the system, or disregard, override or reverse its output
intervene in its operation, or interrupt it through a stop button or similar procedure

Two things are easy to miss. First, the obligation to build these measures in falls on the provider, which is you or your agency, while the obligation to assign competent people falls on the deployer. Second, the article never says a human must approve every action. The measures must be commensurate with the risk. That proportionality is your design space, and it is where most of the engineering decisions live.

The four things to build

1. A capability sheet

A one-page document, shipped with the product, that states what the agent is for, what it was tested on, what it is known to get wrong, and what it must never do. This sounds like paperwork, but it is also the spec for the next three items. If you cannot write down what the agent must never do, you cannot gate it.

2. A decision boundary enforced in code, not in the prompt

The most common mistake we see in agent demos is a system prompt that says "never send a message without approval" and nothing else. A prompt is a suggestion to a probabilistic model. It is not a permission system, and Anthropic's own reporting on unintended agent actions is a useful reminder that instructions get ignored under enough context pressure.

Split the agent's actions into two lists. Autonomous: fetch, classify, summarise, draft, retrieve. Gated: send to a patient, update a record, reject an application, issue a refund above a threshold. Then enforce the split in the tool layer. The agent gets a request_approval tool and never gets send_message at all. The approval service is ordinary deterministic code with ordinary tests. We go deeper on routing and fallbacks in escalation paths for AI agents, and the rest of the hardening list is in our AI agent security checklist.

3. A stop control that stops

Article 14 is explicit about interruption. A real stop control has three parts: a flag the agent checks before every tool call, cancellation of runs already in flight, and a drain of queued work so nothing resumes on its own when someone flips the flag back. Test it on a schedule, the way you test backups. If stopping the agent today means asking DevOps to scale a deployment to zero, you do not have a stop button, you have an outage procedure.

4. An oversight record that proves something

The log is the part a regulator, an auditor, or a court will actually read. For every consequential output it should let you answer five questions:

What did the agent see? Inputs, retrieved documents, model version, prompt version.
What did it propose? The draft, the classification, the recommended decision.
What did the human do? Approved, edited, rejected, or never looked.
Who was the human? An identity, not a shared service account.
When, and for how long? A decision approved in under a second is a rubber stamp, and the record should make that visible.

Store it append-only, with the same discipline we describe in court-ready architecture for healthcare AI.

Our own outreach engine is nowhere near high-risk, but it is built this way because the record is useful regardless of regulation. It scrapes each prospect's site with a self-hosted Firecrawl instance, and a local LLM extracts the relevant facts and writes one tailored email per company in a single call. That single-call pattern beat a multi-step chain on both cost and output quality, and it had a side effect we did not plan for: the facts the model relied on and the draft it produced come out of the same call, so the reviewer sees the evidence next to the conclusion, and there is one input and one output to store per email rather than a chain of intermediate states to reconstruct later.

Automation bias is the requirement people skip

The Act names automation bias directly, and it is the one measure that almost no agent product addresses. If the approve button is the default focus, the reviewer sees only the final answer, and the queue is long, your oversight is theatre and the log will show it.

Three design moves fix most of it:

Show the evidence, not just the answer. The retrieved passage, the extracted fact, the rule that fired.
Hide the suggestion for a sample of items. Let the reviewer decide cold on a fraction of cases and compare.
Track the override rate per reviewer. Zero overrides over weeks is a warning sign, not a success metric.

What this costs and how to scope it

Designed in from the first sprint, oversight is a modest slice of an agent build: the approval service, the flag, and the log schema are small, well-understood components. Bolted on after launch, it is a rewrite of the tool layer and a migration of every historical run you now wish you had recorded. When you scope a high-risk agent, put these four items in the statement of work explicitly rather than hoping they appear under "security".

If you are evaluating an agency for this kind of work, four questions separate a real implementation from a slide:

Show me the stop control, and show me the last time it was tested.
Show me one complete log entry for one real decision.
Which actions are gated, and where in the code is that enforced?
How do you detect that reviewers have stopped reviewing?

If you are building an agent for a hospital, an HR team, a lender or a public body and you want it to survive the procurement questionnaire and the audit that follows, let's talk.

Originally published on the Pykero blog.