DEV Community: Pykero

Small LLM vs Frontier Model: Which One Does Your AI Feature Actually Need?

Pykero — Mon, 19 Oct 2026 09:03:13 +0000

Most AI features inside a SaaS product do not need a frontier model. If the task is bounded (classify a ticket, extract fields from a document, draft an email from facts you already have), a small model with a strict prompt and an output validator is usually good enough and costs a fraction per call. Reserve the frontier model for open-ended reasoning, long multi-step tool use, and anywhere a wrong answer costs real money.

The question is not "which model is smartest"

Founders tend to ask us which model is best. That is the wrong question, because the best model on a leaderboard is rarely the best model for a feature that will run fifty thousand times a month inside your product. The right question is: what is the cheapest model that passes my eval set at the accuracy I need, with the guardrails I can afford to build?

Take the ticket-classification example from the top of this post and make it concrete. At fifty thousand tickets a month, the leaderboard winner and a distilled "flash" model will both tag the obvious tickets the same way. The difference shows up on the ambiguous ones, and a validator that checks the tag against your allowed list and retries on a miss catches a good share of what the small model gets wrong. The only way to know whether the remainder is acceptable is to run the fifty to a few hundred real tickets in your eval set through both models and compare, with the per-call price next to each result.

Framed that way, the decision stops being a bet on vendor marketing and becomes an engineering measurement. The small-model category has moved fast; the recent wave of "flash" and distilled releases from several labs means a model that would have been state of the art two years ago now costs very little per token and runs on a single GPU. Whether that is enough for you depends on four things.

Four questions that decide it

1. Is the task bounded?

A bounded task has a known input shape, a known output shape, and a small set of acceptable answers. "Pull the invoice number, due date and total from this PDF" is bounded. "Read this customer's last six emails and decide what to do" is not.

Small models are strong on bounded tasks because the prompt can carry most of the intelligence: the schema, the rules, three worked examples. They degrade on unbounded tasks because they need to improvise, and improvisation is exactly what gets worse as parameter count drops.

2. Can you check the output cheaply?

If a deterministic check can catch most bad outputs, you can run a smaller model and let the validator do the heavy lifting. JSON schema validation, regex on dates and amounts, a lookup against your own database, a length cap: these cost nothing per call and turn a model's occasional mistake into a retry instead of a bug a customer sees. We wrote about this pattern in deterministic guardrails for AI feature reliability.

If the only way to check the output is for a human to read it, the model has to be right the first time more often, and that pushes you up the price list.

3. What does a wrong answer cost?

A mis-tagged support ticket costs an agent ten seconds. A wrong dosage summary in a healthcare tool, or a wrong refund decision in an e-commerce agent, costs a lot more. Price the failure, not the call. Where the failure is cheap, optimise for cost per call. Where the failure is expensive, the frontier model's per-token premium is trivial next to the risk, and you should also add a human review step regardless of model.

4. Does the input fit, and does it need memory?

Small models typically have shorter usable context and get lost sooner in long documents. If your feature needs to hold a 200-page contract in view, or carry state across a twenty-step workflow, you are either chunking aggressively (more engineering) or paying for a model that can hold it. Multi-step agentic work with tools is the clearest case for frontier models, and it is worth reading agents versus workflows before you assume you need the agent at all.

What a small model is actually good at: our outreach engine

Our own outbound tool is a concrete example. It crawls each prospect's website with a self-hosted Firecrawl instance, hands the page text to a local model, and writes one tailored email per company. The task is bounded: the input is a few pages of text, the output is a short email following our template and tone, and a validator rejects anything that is too long, mentions a product we do not offer, or fails to cite a fact from the site.

What surprised us was that a single call doing "extract the facts and draft the email" beat a multi-step chain (extract, then plan, then write) on both cost and quality, even with the smaller model. Fewer hops meant fewer places for the model to drift from the source text. We covered the general case in single call versus agent chains. The point for a buyer: a well-constrained small model plus a validator was the right tool, and a frontier model would have added cost without changing the reply rate.

When you should pay for the frontier model

Be honest about the cases where cheap is expensive. The outreach engine above gets away with a small model for one reason: there is a single hop, page text in and email out, with a validator standing behind it. Remove any of those conditions and the picture changes.

Open-ended reasoning over messy input. Triage that needs judgement, not rules. Anything where the instructions say "use your discretion". Our validator can reject an email that cites no fact from the site; it cannot judge whether the model chose the right fact, and when that judgement is the whole feature, you want the stronger model.
Multi-step tool use. Agents that call several APIs, interpret results, and decide the next step. The moment our task became "crawl the site, decide which pages matter, check the CRM, then write", the small model would be carrying state across hops, which is exactly where it drops the thread.
Low-volume, high-value calls. If the feature runs a few hundred times a month and each result is reviewed by a senior person, the cost difference is noise. Buy accuracy.
Code generation and complex transformation. The quality gap is still wide here.
The first version, before you have evals. It is often cheaper to launch on a strong model, collect real traffic, build an eval set of fifty to a few hundred real examples from it, and then downgrade where the evals say you can.

That last one matters. Starting on a frontier model and moving down is a measured decision backed by an eval set. Starting on a small model and discovering in production that it is not good enough is a rewrite of your prompts and guardrails under pressure, with no eval set to tell you when you are done.

How the choice changes your quote

Founders expect the small model to make the project cheaper to build. It mostly does not. The build cost is in the prompt design, the validator, the eval set, the retry and fallback logic, and the integration into your product. That work is the same whichever model sits behind it. If anything, a small model needs a thicker validation layer, which is slightly more engineering, not less.

What changes is the running cost. Per-call prices between a frontier model and a small hosted model differ by a large multiple, and over a year of real traffic that difference can exceed the original build budget. Ask for a projection at three volume levels before you sign; our guide to LLM cost optimisation lists the levers that matter beyond model choice, such as caching and prompt length.

Self-hosting the small model is a separate decision. Tools like vLLM and Ollama make it technically easy, but it only pays off at steady volume or when data residency forces your hand. At typical early-stage volumes, a cheap hosted API model wins once you count the engineering time.

The hedge: build the switch, not the bet

Whatever you decide, do not let it be baked into the code. Insist on three things in the brief:

Model behind a single config value. Changing from a frontier model to a small one, or between providers, should be a deployment, not a sprint.
An eval set shipped with the feature. Fifty to a few hundred real examples with expected outputs, run before every model change. Without it, "is the cheaper model good enough?" is a guess. We argue this belongs in the contract in evals in AI vendor contracts.
A fallback path. When the small model fails validation twice, escalate that single call to the stronger model rather than failing the user. This gives you small-model economics with frontier-model worst-case quality.

Done this way, the model question becomes a dial you turn as traffic and evidence come in, instead of a decision you make once with no data.

If you are scoping an AI feature and want a straight answer on which model tier it needs and what it will cost to run, let's talk.

Originally published on the Pykero blog.

How to Brief an AI Agency So the Quote Is Actually Accurate

Pykero — Sat, 17 Oct 2026 09:03:47 +0000

An accurate quote for a custom AI project comes from a brief that answers five questions: what business outcome you want, what data you actually have, what a wrong answer costs you, which systems the agent must read from and write to, and what constraints are non-negotiable. Everything else, including the model, the framework, and whether it is "an agent" or "a workflow", is the agency's job to propose. Founders who send a two-page problem statement get a number. Founders who send a feature list or a vague "we want AI in our product" get a range so wide it is useless, or a lowball that gets renegotiated in month two.

Why most AI briefs produce bad quotes

A software agency prices risk. Every unknown in your brief gets converted into either a padded estimate or an optimistic one that falls apart when the real data shows up. AI projects have more unknowns than ordinary software because the hard part is rarely the code. The hard part is the data, the edge cases, and deciding how good is good enough.

The three most common brief failures we see:

Solution instead of problem. "Build a RAG chatbot over our docs with GPT and a vector database." The agency now has to reverse-engineer what you actually need, and often the prescribed design is the expensive one. We have been on the receiving end of this ourselves: the multi-step chain we assumed for our own outreach engine lost to a single call on both cost and quality, and a brief that had prescribed the chain would have locked in the worse design. We cover why this matters in RAG vs fine-tuning for founders.
No definition of correct. "It should answer customer questions accurately." Accurately compared to what? Who checks? What happens when it is wrong? "Right nine times out of ten and flags the rest for a human" is a definition. "Accurate" is not.
Integrations mentioned in passing. "It should also update the CRM." That sentence can double a project. An agent that drafts a CRM note for a human to approve and one that changes deal stages on its own are different builds, and the brief usually does not say which one is meant. Authentication, rate limits, field mapping, and failure handling for a third-party system are real work.

Fix those three and the quote you get back will be close to what you end up paying.

The five sections your brief needs

1. The outcome, in business terms

One paragraph. Not "an AI assistant" but "reduce time-to-first-response on inbound support from hours to under two minutes, without increasing escalations to the human team." A measurable outcome lets the agency propose the cheapest design that hits it, instead of the most impressive one.

Include the volume. Fifty requests a day and fifty thousand a day are different systems with different cost structures. If you do not know, give your best guess and say it is a guess.

2. The data you actually have

This section decides more of the price than anything else. Be specific and be honest:

Where it lives. Postgres, a Google Drive full of PDFs, a legacy CRM with an export button, WhatsApp threads on someone's phone.
How much and how clean. Ten thousand well-structured records is a different job from two hundred scanned forms in three languages.
Who owns it and whether it contains personal data. If the project touches EU residents' data, say so. The agency has to design for GDPR from day one, not retrofit it.

Attach a redacted sample. Twenty rows or ten documents is enough. No serious agency can estimate extraction or retrieval work from a prose description of data, and any that claims to is guessing.

3. The cost of a wrong answer

This is the question founders skip and agencies most need answered. An AI feature that suggests blog tags can be wrong a quarter of the time with no consequence. An agent that quotes prices to customers or files insurance claims cannot.

State, in plain language:

What happens when the output is wrong. Embarrassment, a refund, a regulatory problem, a patient safety issue.
Who catches it today. A human reviewer, nobody, the customer.
What error rate you would accept at launch. Even a rough figure helps. "If it is right nine times out of ten and flags the rest for a human, that beats what we have."

This single section determines whether the agency designs a cheap system with a human in the loop or an expensive one with layered verification. Our piece on deterministic guardrails for AI features explains why the second kind costs more and when it is worth it.

4. Systems the agent must touch

List every system the AI must read from or write to, and for each one: do you have API access, who administers it, and is there a sandbox. Writing is more expensive than reading. An agent that drafts a CRM note for a human to approve is a smaller project than one that updates deal stages on its own.

If a system has no API, say that too. Browser automation and screen scraping are possible but they are a different line item, and they break more often.

5. Hard constraints

Things the agency cannot trade away:

Data residency. Must stay in the EU, in Saudi Arabia, on your own servers.
Budget per request or per month. A ceiling on inference cost shapes model choice immediately.
Latency. A WhatsApp reply can take four seconds. A voice agent cannot.
Timeline and what is driving it. A trade show date is a real constraint. "ASAP" is not.
Legal or disclosure requirements. If customers must be told they are talking to a machine, say so.

Keep this list short. Every item here removes design options, and some of those options were the cheap ones.

What to leave out

The architecture. Do not specify the model, the vector database, the orchestration framework, or how many steps the pipeline should have. When we built our own outreach engine, which scrapes each prospect's website with a self-hosted crawler and drafts one tailored email with a local model, our starting assumption was a multi-step chain: one call to extract facts, another to plan, another to write. The version that won on both cost and quality was a single call that extracted the facts and drafted the email together. Had a client prescribed the chain in their brief, we would have built the worse system and charged more for it. Let the agency propose, then challenge the proposal.

A long feature list. Features without priority get priced as all mandatory. If you must list them, mark three as launch-critical and the rest as later.

Competitor screenshots as the spec. Useful as reference, dangerous as requirements. You end up paying to replicate someone else's compromises.

What a good agency will do with your brief

Take the support example from section one. A brief that says "under two minutes to first response, current volume roughly five hundred tickets a day, right nine times out of ten with the rest flagged to a human, must draft but not send, EU data only" should get back, within a week or two, a proposal that reads like an answer to those exact lines: a recommended approach with the reasoning, a pilot scope (perhaps one ticket category, drafts only), a cost broken into build, inference at your stated volume, and ongoing maintenance, and a proposed way to measure the two-minute target and the one-in-ten error rate before you pay for the full build. If the response is a single number with no assumptions listed, the agency did not read the brief. If it is a thirty-page deck that never mentions your data sample, same conclusion.

You should also expect questions back. Good agencies push on the cost-of-wrong-answer section and on the integrations, because that is where the real estimation risk lives. Expect to be asked who reviews the flagged tenth today, and whether the CRM has a sandbox. Agencies that ask nothing are either very experienced in your exact problem or planning to discover the problems on your invoice.

Before signing, use how to vet an AI agency demo to pressure-test what they show you, and make sure the success metric from section one makes it into the contract. Agreeing on how the system will be evaluated before the build starts is the cheapest insurance you can buy.

A template you can copy

1. Outcome: what changes for the business, with a number and a volume.
2. Data: where it lives, how much, how clean, personal data yes/no. Sample attached.
3. Cost of wrong: what happens, who catches it today, acceptable error rate at launch.
4. Systems: each one, read or write, API access yes/no, sandbox yes/no.
5. Constraints: residency, cost ceiling, latency, hard dates, disclosure rules.
6. Not in scope for v1: the things you are consciously deferring.

Two pages, plus appendices. It takes a founder an afternoon to write and saves weeks of back-and-forth, or a project that quietly doubles in price.

If you have a brief half-written and want a second pair of eyes before sending it to anyone, or you would rather talk the problem through first, let's talk.

Originally published on the Pykero blog.

AI agent human oversight: what the EU AI Act actually requires you to build

Pykero — Tue, 13 Oct 2026 09:02:37 +0000

Human oversight under the EU AI Act means a named person can understand what your agent is doing, override or stop it, and prove afterwards that they could. For most agents that comes down to four things you have to build: a capability sheet, a decision boundary enforced in code, a stop control that really halts the system, and a log that records every consequential decision together with who made it. Article 14 only binds high-risk systems, but if you sell into healthcare, HR, lending or the public sector, assume you are in scope and design for it now rather than retrofit it under procurement pressure.

Who this actually applies to

Article 14 sits in Chapter III of Regulation (EU) 2024/1689, the chapter that sets obligations for high-risk AI systems. Whether your agent is high-risk is decided by two annexes. Annex I covers AI that is a safety component of a regulated product, which includes medical devices. Annex III lists use cases: biometrics, critical infrastructure, education, employment and worker management, access to essential private and public services, law enforcement, migration, and the administration of justice.

Translate that into agency terms:

Not high-risk: a WhatsApp agent that answers product questions and recommends a size. Transparency rules under Article 50 still apply, and we covered those in disclosure rules for AI sales agents.
High-risk: an agent that triages patient messages, screens job applicants, scores a loan file, or decides eligibility for a public benefit. Same stack, same model, completely different obligations.

Annex III obligations were scheduled to apply from 2 August 2026. The timeline has been the subject of proposed amendments, so check the current consolidated text before committing to a date in a contract. What has not waited for the legal deadline is procurement: hospital and government buyers already ask for oversight documentation in tenders, which is why this matters commercially today.

What Article 14 asks for, in plain language

The article requires that high-risk systems are designed so natural persons can effectively oversee them while in use. The person doing the oversight must be enabled to:

understand the system's capabilities and limitations, and monitor its operation for anomalies
stay aware of automation bias, the tendency to over-trust the output
correctly interpret the output, with tools to do so
decide not to use the system, or disregard, override or reverse its output
intervene in its operation, or interrupt it through a stop button or similar procedure

Two things are easy to miss. First, the obligation to build these measures in falls on the provider, which is you or your agency, while the obligation to assign competent people falls on the deployer. Second, the article never says a human must approve every action. The measures must be commensurate with the risk. That proportionality is your design space, and it is where most of the engineering decisions live.

The four things to build

1. A capability sheet

A one-page document, shipped with the product, that states what the agent is for, what it was tested on, what it is known to get wrong, and what it must never do. This sounds like paperwork, but it is also the spec for the next three items. If you cannot write down what the agent must never do, you cannot gate it.

2. A decision boundary enforced in code, not in the prompt

The most common mistake we see in agent demos is a system prompt that says "never send a message without approval" and nothing else. A prompt is a suggestion to a probabilistic model. It is not a permission system, and Anthropic's own reporting on unintended agent actions is a useful reminder that instructions get ignored under enough context pressure.

Split the agent's actions into two lists. Autonomous: fetch, classify, summarise, draft, retrieve. Gated: send to a patient, update a record, reject an application, issue a refund above a threshold. Then enforce the split in the tool layer. The agent gets a request_approval tool and never gets send_message at all. The approval service is ordinary deterministic code with ordinary tests. We go deeper on routing and fallbacks in escalation paths for AI agents, and the rest of the hardening list is in our AI agent security checklist.

3. A stop control that stops

Article 14 is explicit about interruption. A real stop control has three parts: a flag the agent checks before every tool call, cancellation of runs already in flight, and a drain of queued work so nothing resumes on its own when someone flips the flag back. Test it on a schedule, the way you test backups. If stopping the agent today means asking DevOps to scale a deployment to zero, you do not have a stop button, you have an outage procedure.

4. An oversight record that proves something

The log is the part a regulator, an auditor, or a court will actually read. For every consequential output it should let you answer five questions:

What did the agent see? Inputs, retrieved documents, model version, prompt version.
What did it propose? The draft, the classification, the recommended decision.
What did the human do? Approved, edited, rejected, or never looked.
Who was the human? An identity, not a shared service account.
When, and for how long? A decision approved in under a second is a rubber stamp, and the record should make that visible.

Store it append-only, with the same discipline we describe in court-ready architecture for healthcare AI.

Our own outreach engine is nowhere near high-risk, but it is built this way because the record is useful regardless of regulation. It scrapes each prospect's site with a self-hosted Firecrawl instance, and a local LLM extracts the relevant facts and writes one tailored email per company in a single call. That single-call pattern beat a multi-step chain on both cost and output quality, and it had a side effect we did not plan for: the facts the model relied on and the draft it produced come out of the same call, so the reviewer sees the evidence next to the conclusion, and there is one input and one output to store per email rather than a chain of intermediate states to reconstruct later.

Automation bias is the requirement people skip

The Act names automation bias directly, and it is the one measure that almost no agent product addresses. If the approve button is the default focus, the reviewer sees only the final answer, and the queue is long, your oversight is theatre and the log will show it.

Three design moves fix most of it:

Show the evidence, not just the answer. The retrieved passage, the extracted fact, the rule that fired.
Hide the suggestion for a sample of items. Let the reviewer decide cold on a fraction of cases and compare.
Track the override rate per reviewer. Zero overrides over weeks is a warning sign, not a success metric.

What this costs and how to scope it

Designed in from the first sprint, oversight is a modest slice of an agent build: the approval service, the flag, and the log schema are small, well-understood components. Bolted on after launch, it is a rewrite of the tool layer and a migration of every historical run you now wish you had recorded. When you scope a high-risk agent, put these four items in the statement of work explicitly rather than hoping they appear under "security".

If you are evaluating an agency for this kind of work, four questions separate a real implementation from a slide:

Show me the stop control, and show me the last time it was tested.
Show me one complete log entry for one real decision.
Which actions are gated, and where in the code is that enforced?
How do you detect that reviewers have stopped reviewing?

If you are building an agent for a hospital, an HR team, a lender or a public body and you want it to survive the procurement questionnaire and the audit that follows, let's talk.

Originally published on the Pykero blog.

How Much Does It Cost to Integrate an AI Agent With Your CRM?

Pykero — Sun, 11 Oct 2026 09:02:32 +0000

Integrating an AI agent with your CRM costs anywhere from a few thousand dollars for a read-only agent that enriches and summarises records, up to the tens of thousands once the agent writes back to the CRM, triggers workflows, and needs to pass a security review. The CRM vendor matters less than you think. What moves the price is how much the agent is allowed to do, how clean your data is, and how many LLM calls it takes to process one record.

Why "connect it to HubSpot" is not a scope

Every founder brief we see says some version of "we want an AI agent connected to our CRM." That sentence describes four very different projects:

Read and summarise. The agent reads a contact, their emails, and their deal history, and produces a briefing before a call.
Read and enrich. The agent pulls external data (company website, LinkedIn, news) and fills in empty fields.
Read and recommend. The agent scores leads, suggests next actions, and leaves notes for the rep to approve.
Read and act. The agent updates deal stages, creates tasks, sends sequences, and books meetings without a human in the loop.

Each step down that list roughly doubles the engineering effort, because write access brings validation rules, idempotency, rollback, audit logs, and the question of who gets blamed when the agent moves a deal to "Closed Lost" by mistake. Decide which of the four you are buying before you ask for a quote. If you are still deciding whether you need an agent at all, AI agents vs workflows is the place to start.

The four cost drivers

1. Permissions and the principle of least privilege

Your CRM admin will ask, correctly, why an AI agent needs full access. The honest answer is that it almost never does. A well-scoped integration requests only the OAuth scopes it uses. HubSpot documents its granular scopes in the HubSpot API overview, and Salesforce lets you create a dedicated integration user with a permission set that exposes only specific objects and fields.

Scoping this properly costs time up front: someone has to map every field the agent touches and justify it. Skipping it costs more later, when the security review bounces the project or, worse, the agent has write access to objects nobody intended. Budget for the scoping conversation. It is usually a day or two of combined work between your admin and the engineering team, and it is the cheapest risk reduction in the whole project.

2. Data quality, the cost nobody budgets

The agent can only be as good as the records it reads. In practice, CRMs that have been running for a few years contain:

Duplicate contacts with different emails for the same person
Status stored in free-text notes instead of a picklist
Custom fields created by a rep who left in 2023 and never documented
Pipelines where "Qualified" means three different things to three teams

An agent that reads this will confidently produce wrong summaries. The fix is not a smarter prompt, it is a data cleanup pass and a written definition of what each field means. For a mid-sized HubSpot instance this is often a week of work. For a heavily customised Salesforce org it can be the single largest line item. Ask any vendor quoting you whether their price includes a data audit. If it does not, add it yourself.

3. LLM step count per record

The model bill is rarely the expensive part of a CRM integration, but the architecture that drives it is. A naive design chains five or six calls per record: fetch, classify, extract, decide, draft, verify. Each hop adds latency, a failure point, and tokens.

We learned this building our own outreach engine, which scrapes each prospect's website with a self-hosted crawler and a locally hosted model, then writes one tailored email per company. The first version was a multi-step chain. When we collapsed it into a single call that extracts the relevant facts and drafts the email in one pass, both the cost per company and the quality of the output improved. The multi-step version was losing context between hops and paying for it twice. The same pattern applies to CRM records: one well-structured call with a strict output schema usually beats a chain, and it makes the integration cheaper to run and easier to debug. We wrote up the trade-off in detail in single call vs agent chains.

For budgeting, ask the vendor two questions: how many model calls per record, and what happens to the monthly bill if your contact volume triples. If they cannot answer the second one, they have not thought about it.

4. Write-back, retries, and audit

The moment the agent writes to the CRM, you need:

Idempotency. If the job runs twice, the record should not be updated twice.
Rate limit handling. HubSpot and Salesforce both throttle API calls. HubSpot's limits are documented in its API usage details. Hitting them mid-batch without a retry strategy leaves half your records updated.
An audit trail. Every change the agent makes should be traceable to a run, a prompt version, and the input it saw. When a rep asks "why did the AI change this?", you need an answer in under a minute.
A human approval path for high-stakes actions, at least until the agent has earned trust.

This layer is where a no-code connector stops being enough. It is also where most of the cost in the "read and act" tier lives, and it is the same engineering that gets an agent from a demo to something your team relies on. See AI agent pilot to production for what that transition involves.

Typical ranges

Treat these as order-of-magnitude guides, not quotes. Every CRM org is different, and the data audit alone can swing the number. To make the ranges useful, here is where the money actually goes in each tier, using the time estimates from the sections above.

Read and summarise, standard objects, HubSpot or Pipedrive: low thousands of dollars, two to three weeks. The day or two of scoping with your admin, field mapping for one object, one well-structured model call per record with a strict output schema, and a briefing template the reps will actually read. No write access, so no idempotency or audit layer.
Read and enrich with external sources: mid to high thousands. Everything above, plus the crawler, deduplication against the duplicate contacts your CRM already has, and confidence scoring so the agent does not overwrite a correct phone number with a scraped wrong one. This is the tier where the week-long data audit stops being optional, because enrichment on top of dirty records multiplies the mess.
Read and recommend with notes and tasks: high thousands to low tens of thousands. Write access begins here. Add the approval UI, the audit log that traces each note to a run and prompt version, and rate limit handling so a batch of a few thousand contacts does not stall halfway through.
Read and act autonomously, Salesforce with custom objects: tens of thousands. Security review, integration user and permission set setup, validation rule handling for every custom object the agent touches, a staged rollout that starts in suggest mode, and the "Closed Lost by mistake" rollback path. On a heavily customised org the data audit can cost as much as the agent itself.

Add ongoing costs: model usage, hosting for the agent runtime, and maintenance when the CRM vendor changes an API or your team adds a new pipeline. A rough rule is to expect maintenance at a meaningful fraction of the build cost each year, and to expect the model bill to track your record volume, which is why the "what if volume triples" question matters.

What to cut if the budget is tight

In order of least regret:

Drop autonomous actions. Ship the agent in "suggest and a human clicks approve" mode. You keep most of the value and lose most of the risk.
Limit to one object. Deals or contacts, not both. Expand once the first one works.
Skip external enrichment at first. Your CRM already holds more signal than most teams use.
Do not skip the data audit. It is the one cut that comes back to bite you.

How to get a straight quote

Send prospective vendors a one-page brief that answers the questions this post has raised: which CRM, which objects (ideally one), which of the four tiers of autonomy, your current record volume, and whether an admin is available for the day or two of scoping. Then ask four things back:

How many model calls per record, and why that many. If the answer is five or six, ask whether it could be one. When we collapsed our own outreach chain into a single pass, cost and quality both improved, and a vendor who has done that work will have a view on it.
What the monthly run cost looks like at three times your current volume.
Whether the price includes a data audit, and roughly how long they expect it to take on your org.
Whether every write the agent makes is traceable to a run, a prompt version, and the input it saw.

A vendor who answers those four questions clearly is one you can work with. A vendor who answers "it depends" to all four is quoting you for the "connect it to HubSpot" sentence, not for a project.

If you want a second opinion on a quote you already have, or you would like us to scope the integration against your actual CRM, let's talk.

Originally published on the Pykero blog.

How to Vet an AI Agency Demo Before You Sign

Pykero — Fri, 09 Oct 2026 09:03:04 +0000

An AI agency demo proves exactly one thing: the system works on the inputs the agency chose to show you. It does not prove it works on your data, your edge cases, or your users at 2 a.m. Before you sign, you need to replace their inputs with yours, look at the failures instead of the successes, and get the test set written into the contract.

This matters more for AI projects than for ordinary software. A CRUD app either saves the record or it does not. An LLM-powered agent can look perfect on fifty curated examples and then confidently produce garbage on the fifty-first. A founder who cannot tell a rigged demo from a robust system is buying on faith.

Why AI demos are so easy to fake, even accidentally

Nobody has to be dishonest for a demo to mislead you. Three things happen naturally:

The examples are cherry-picked. The team built the prototype against a handful of inputs and those are the ones that work. They show you what they tested. If the demo set is ten clean, well-formatted records and your real inbox is support tickets in two languages with no punctuation, you are looking at a different product than the one you will be paying for.
Failures get filtered out before you see them. A recent post making the rounds described a benchmark that scored a model a perfect 1.00 because the harness silently skipped the three questions the model failed. The same thing happens in demos: a retry loop, a fallback to a canned answer, or a quietly dropped row hides the misses. A score of "98 percent" with no denominator and no list of the misses is the same trick in a nicer font.
The system is tuned for the demo, not for production. Low temperature, a huge context window stuffed with your exact documents, a human pre-cleaning the input. None of that survives contact with real traffic and real costs. The stuffed context in particular is a tell: it makes every answer look grounded while quietly multiplying the per-input token bill you will ask about later.

None of this is malicious. But the burden is on you to see past it.

Bring your own inputs

The single most useful thing you can do is show up with your own data. Not a description of your data, the actual records.

Pull 20 to 50 real examples from the workflow the agent will handle. Deliberately include the ugly ones:

A support ticket written in two languages with no punctuation
A scanned PDF where the table is slightly rotated
A lead form where the company field says "n/a" and the phone number is in the email field
Two records that look like duplicates but are not
A message that asks for something the agent should refuse or escalate

Then ask the agency to run these live, in front of you, and show you every output. Not an accuracy percentage. The raw outputs, side by side with the inputs.

If they ask for a week to "prepare the environment" first, that is useful information too. A system that needs a week of prep to handle 30 of your records is a system that will need a week of prep for every new customer segment you add.

Ask to see the failures

Flip the usual demo script. Instead of "show me it working", ask:

"Show me the last five inputs where it got the wrong answer." A serious team has these. They keep a failure log because that is how you improve an LLM system. A team with no failures to show has either not tested or is not telling you.
"What does the system do when the model is wrong?" The honest answer involves validation, confidence thresholds, and a route to a human. The worrying answer is "it rarely is". Every model is wrong some percentage of the time; the design question is what happens next. We covered the patterns in deterministic guardrails for AI features.
"How do you know it has not regressed after you change a prompt?" You are listening for the word evals: a fixed set of inputs with expected outputs that runs on every change. If they do not have one, you will be the eval set, in production.

Ask what is actually running

A demo hides architecture, and architecture is what you are paying for over the next two years. Ask them to walk through the request path for one input, start to finish:

What is the model, and is it a provider API or self-hosted?
How many model calls per input, and how long does the slowest one take?
What retrieval is happening, over what documents, refreshed how often?
What is deterministic code and what is left to the model?
What does one input cost in tokens, and what does that look like at your expected monthly volume?

Be especially alert to complexity theater. Agencies sometimes present a five-agent pipeline with planners, critics and orchestrators because it looks sophisticated on a slide. In our own outreach tooling, where we scrape each prospect's site and have a local model draft one tailored email, the version that extracted facts and wrote the draft in a single call beat the multi-step chain on both cost and output quality. Every extra hop was another place for the pipeline to lose context or hallucinate a bridge between steps. We wrote up the trade-offs in single call vs agent chains.

Simple is not automatically better, but a team that can explain why each component exists is a team that understands the system. A team that cannot is maintaining something they inherited from a tutorial.

Check the production story, not just the prototype

The gap between a demo and a deployed system is where most AI projects die. The good news is that the 30 records you brought and the request walkthrough you just sat through give you everything you need to make these questions concrete instead of hypothetical:

Who owns the prompts and the eval set after launch? You should. The eval set is your 30 records plus every failure they showed you, with agreed correct outputs. Make sure it is named in the deliverables, not described as "documentation".
What happens when the model provider deprecates the version you tested on? Providers retire models on a schedule. The answer you want is "we swap the model, re-run the acceptance set, and show you the diff". If re-running the set is not a one-command operation, ask why not.
How is cost capped? Take the per-input token cost from the walkthrough, multiply by your monthly volume, and ask what stops the bill from being ten times that. A runaway retry loop or a spam burst against an unprotected endpoint will find out for you. Rate limits and per-user budgets belong in the build, not in a later phase.
What is monitored? At minimum: latency, error rate, fallback rate, and some sampled quality review. Fallback rate is the one to watch. It is the production version of "what happens when the model is wrong", and if it climbs from 5 percent to 20 percent after a prompt change, that is your regression alarm. If their answer is "we'll add logging", they have not shipped one of these before.
What does month three look like? New edge cases arrive, the eval set grows, a prompt gets tuned, a model gets updated. Who re-runs the set each time, and what does that cost per month?

A team that has taken agents from pilot to production will answer these without hesitation, because they have been burned by each one. Our pilot to production guide lists the specific failure modes we see most often.

Put the test set in the contract

Here is the move that separates a hopeful engagement from a safe one. Take the inputs you brought to the demo, agree on what a correct output looks like for each, and write that into the statement of work as the acceptance test.

This does three things at once. It forces both sides to define "working" in concrete terms before money changes hands. It gives the agency a target they can actually hit, which good agencies appreciate. And it gives you a regression suite you will keep running for the life of the product.

Agree on a threshold, not perfection. Something like "at least 90 percent of the acceptance set correct, zero outputs in the unsafe category, and every incorrect output routed to review" is realistic for most document and sales workflows. A hundred percent is a sign nobody has thought about the failure path.

A short checklist for the meeting

Walk in with this and you will learn more in an hour than in three polished pitches:

Run my 30 real inputs live, show every output
Show me five recent failures and what you changed because of them
Walk me through one request end to end: models, calls, retrieval, cost
What happens when the model is wrong?
Who owns prompts, evals and monitoring after launch, and what does month three cost?
Will you put the acceptance set and threshold in the contract?

An agency that welcomes these questions is one you can work with. One that treats them as hostile is telling you how the project will go.

If you are evaluating AI partners right now and want a second opinion on a proposal, or want to see how we would run your 30 ugliest records, let's talk.

Originally published on the Pykero blog.

How Much Does a Custom AI Lead Qualification Agent Cost?

Pykero — Wed, 07 Oct 2026 09:02:25 +0000

A custom AI lead qualification agent costs between 15k and 60k USD to build and somewhere from a few hundred to a few thousand dollars a month to run. The wide range is not about the model. It comes from how many data sources the agent has to read, how deep the CRM integration goes, and how much of the decision you are willing to automate without a human looking at it.

What a lead qualification agent actually does

Strip the marketing away and the agent is a pipeline with four jobs:

Enrich: take a form submission, chat transcript or inbound email, find the company and the person, and gather facts about them.
Score: compare those facts to your ideal customer profile and produce a fit score with reasons.
Route: write the result to the CRM, assign an owner, and either trigger a reply, a meeting link, or a polite no.
Report: tell you what it decided and why, so you can audit it and tune it.

Each of those jobs has its own cost driver. If you want a sense of whether this should be an agent at all or a plain workflow with one LLM call inside it, read AI agents vs workflows first. Most lead qualification is closer to a workflow than people expect, and that is good news for the budget.

Build cost by scope

Narrow: one intake channel, one CRM, 15k to 25k USD

This is the version most founders should start with. Leads arrive from a single source, usually a website form or a chat widget. The agent looks up the company domain, scrapes the homepage and one or two key pages, pulls a few fields from an enrichment API, and asks the model for a fit score and a short justification. It writes everything to one CRM, say HubSpot through the HubSpot CRM API, and assigns the lead to a rep based on territory or segment.

Four to six weeks of work. The expensive part is not the model prompt, it is handling the messy edge cases: personal email addresses with no company, companies with no website, duplicate contacts already in the CRM, and leads that arrive twice.

Mid: multiple channels, custom rules, 25k to 45k USD

Here you add inbound email parsing, WhatsApp or LinkedIn intake, and a scoring rubric that is specific to your business rather than generic firmographics. Concretely, "specific" means the one-page rubric with good, bad and borderline examples described at the end of this post becomes the prompt, and the agent has to return the matching reason for each score, not just a number. You also usually want the agent to draft a first reply for the rep to approve. If you are already asking for a structured JSON object with score and reasons, adding the draft to that same object costs almost nothing, which is exactly the single-call pattern we describe below. Routing becomes stateful: round-robin, capacity-aware, or based on which rep owns an existing account, which means the duplicate-contact lookup from the narrow version now has to be right every time because a wrong match sends the lead to the wrong person.

Eight to ten weeks. Most of the extra cost is integration work and the review UI, not the AI.

Full: enrichment at scale, outbound actions, reporting, 45k to 60k USD and up

At this level the agent can also qualify lists you upload, not just inbound leads, and it takes outbound actions on its own within limits you set. You get a dashboard of decisions, override tracking, and weekly accuracy reports comparing the agent's score to what actually closed. Expect two to three months and a proper evaluation set before launch.

The single biggest lever: one call or a chain

The architecture choice that moves cost most is whether the enrichment and scoring step is one LLM call or a multi-step chain of calls.

We learned this on our own outreach engine, which scrapes each prospect's website with a self-hosted Firecrawl instance and a locally hosted model, then writes a tailored email per company. Our first design was a chain: one call to extract company facts, another to classify fit, another to draft. When we collapsed it to a single call that extracts the facts and produces the draft in one structured response, both the cost per prospect and the output quality improved. The chain was losing context between steps and paying for the same tokens three times.

For lead qualification the same pattern applies. Feed the scraped pages and enrichment fields into one call, ask for a JSON object with the score, the reasons, and the suggested next action, and validate the output with a schema. We wrote up the general trade-off in single call vs agent chains. Reach for a chain only when a step genuinely needs the result of a previous step to decide what to fetch next.

Monthly running cost

For a B2B company handling a few hundred to a few thousand leads a month, the running costs line up roughly like this:

Enrichment data: usually the largest line. Contact and company enrichment APIs charge per credit, and that scales linearly with volume. Scraping the prospect's own website with a tool like Firecrawl is far cheaper and often more useful than a third-party firmographic record.
LLM tokens: surprisingly small. A single structured call per lead with a few thousand tokens of context is cents, not dollars, on current mid-tier models. If you self-host a small model, it is effectively hosting cost only.
Hosting: a worker, a queue, a small Postgres database, and a review UI. Tens to low hundreds of dollars a month on a managed platform.
Maintenance: CRM APIs change, websites change, your ICP changes. Budget a retainer. We break this down in what an AI agent costs to maintain.

The honest summary: if your lead volume is under a thousand a month, you are mostly paying for engineering, not for inference.

What pushes the price up

Scoring rules nobody has written down. If sales "just knows" what a good lead looks like, the first two weeks of the project are an interview process, and the rubric will change three times. Write the rubric before you brief an agency.
CRM hygiene. Duplicates, free-text fields that should be enums, and custom objects with no documentation all cost integration hours.
Outbound actions. The moment the agent sends anything to a lead, you need approval flows, rate limits, suppression lists, and a way to stop it. See AI cold email agent: build vs buy for how those costs stack.
Compliance. If you qualify leads in the EU, automated decisions that significantly affect a person are regulated under Article 22 of the GDPR. Keeping a human on disqualifications is the simplest way to stay clear of that, and it also makes the system better.

Build vs buy, quickly

AI SDR and lead scoring SaaS tools are cheap to start and charge per seat, per lead, or per enrichment credit. If you have a standard ICP, one CRM, and modest volume, buy one and move on.

Build when at least two of these are true. First, your scoring logic is a real competitive advantage, meaning the one-page rubric contains things a generic firmographic model cannot see. Second, you need the agent to read sources the SaaS tools do not cover, such as the prospect's own website, which in our outreach engine turned out to be more useful than any purchased firmographic record. Third, your volume makes per-lead pricing painful, which is the case when the SaaS charges you per lead while your own single structured call costs cents. Fourth, your data cannot leave your infrastructure, in which case a self-hosted Firecrawl and a locally hosted model, the same setup we run ourselves, turn the inference line into plain hosting cost. In those cases the narrow build at 15k to 25k pays back within the first year or two and you own the logic afterward.

A sensible way to start

Write the scoring rubric as a one-page document with examples of good, bad and borderline leads.
Pull two hundred historical leads with known outcomes. That is your evaluation set.
Build the narrow version: one channel, one CRM, one structured LLM call, human review on every decision.
Measure agreement between the agent and your reviewer for three or four weeks.
Automate only the segments where agreement is high. Keep humans on the rest.

That sequence keeps the first invoice in the 15k to 25k range and gives you real numbers before you decide whether the full build is worth it.

If you want a scoped estimate for your own intake channels and CRM, let's talk.

Originally published on the Pykero blog.

How to Run an AI Agent Pilot That Actually Reaches Production

Pykero — Mon, 05 Oct 2026 09:02:49 +0000

An AI agent pilot reaches production when it is scoped around one business metric, runs on your real data with a real stop date, and leaves behind receipts: run logs, an eval set, and a measured cost per completed task. Pilots that stall are almost never killed by the model. They stall because nobody wrote down what "done" meant before the demo looked impressive.

We have built agents for sales, support, document processing and internal operations, and we have watched the same pattern repeat. The pilot works in the room. Then three months pass and it is still "being evaluated". This guide is about how to avoid that as the buyer.

Why most pilots never leave the pilot stage

Three failure modes cover most of the cases we see, and each one has a specific shape:

The pilot measured the wrong thing. Teams measure "did the agent produce a plausible answer" instead of "did the ticket close, did the lead reply, did the clinician accept the draft". We made this mistake with our own cold-email engine: weeks spent grading how well the agent wrote the body copy, while the number that mattered, replies, sat flat. Plausible output is cheap. Outcomes are the point.
The pilot ran on clean data. Ten hand-picked documents or a curated list of questions. Production is the other ninety percent: scanned PDFs, Arabic mixed with English in the same message, customers who ask three things at once. A pilot that never sees those cases cannot tell you its real failure rate, and its "accuracy" number is a number about the demo set, not about your business.
Nobody owned the go/no-go decision. The vendor wants to extend, the champion is busy, and the pilot drifts. The tell is a pilot with a start date and no end date. That is not a pilot. It is a subscription, and the "three months of evaluation" we keep seeing is what it looks like from the inside.

Each of these is fixable at the scoping stage, which is where the buyer has the most leverage. The rest of this guide is the scoping checklist.

Step 1: Pick one metric and one workflow

Resist the urge to pilot "an AI assistant for the team". Pick a single workflow with a measurable outcome that already has a baseline. Examples that pilot well:

First-response drafts for one support queue, measured by agent acceptance rate and time to first response.
Lead qualification over WhatsApp for one product line, measured by qualified-lead handoff rate.
Invoice or claim data extraction for one document type, measured by field-level accuracy against the current manual process.

The metric must be something you would still care about if there were no AI involved. If you cannot compute the baseline today, spend the first week of the pilot computing it, because without a baseline the pilot can never prove anything. Our guide to calculating AI agent ROI covers how to turn those metrics into a number the board understands.

A lesson from our own tooling: in the cold-email engine we run for ourselves, we spent weeks tuning how the agent wrote the body of each email, and the metric barely moved. What moved it was changing the ask at the end. Replacing a calendar link with a soft "reply YES" lifted reply rates consistently, because scheduling a meeting from a cold email is more friction than an interested stranger will tolerate. The agent was fine. We had been measuring and optimising the wrong part of the funnel. Define the outcome metric first and let it tell you where to look.

Step 2: Run on real data, with real volume

Agree in writing how the pilot gets access to production-like data. This is usually the slowest part, so start the security and legal conversation on day one, not week three. Practical options, in order of preference:

Shadow mode. The agent runs on live traffic but its output is logged, not sent. A human still does the job. You compare the two.
Assisted mode. The agent drafts, a human approves or edits before anything goes out. Edit distance becomes a free quality signal.
Scoped autonomous mode. The agent acts on its own for a narrow, low-risk slice, with clear escalation paths back to a person.

Shadow mode is underrated. It removes almost every objection from compliance, it generates the eval dataset for free, and it tells you the real failure rate before a single customer sees the agent. For healthcare and government buyers in the Gulf, it also sidesteps most of the data residency debate during the pilot, because nothing leaves the existing workflow.

Step 3: Define the receipts before you start

Every "done" in a pilot needs a receipt, something you can inspect after the fact. Before the build starts, agree on exactly what artefacts the pilot produces:

Run logs for every task. Inputs, tool calls, model outputs, latency, token usage. Not a dashboard screenshot. Structured logs you can query. OpenTelemetry is the obvious neutral format if you want them to outlive the vendor relationship.
An eval dataset. A few hundred real cases with the expected outcome, scored automatically. This is the asset that lets you upgrade models, switch providers, or change prompts later without guessing. If the pilot ends without one, you have bought a demo.
A cost per completed task. Model spend, tool calls, retries and human review time, divided by tasks actually completed. Not cost per API call.
A failure taxonomy. The ten most common ways it went wrong, with counts. This becomes the production backlog.

Put these in the statement of work as deliverables. We cover how to make them contractual in putting evals in AI vendor contracts. A vendor who pushes back on delivering logs and eval data is telling you something.

Step 4: Set the go/no-go criteria and date

Write down three numbers before the first line of code:

Ship threshold. The metric at which you commit to production, for example "agent drafts accepted without edits on 60 percent of tickets, zero policy violations in 500 runs".
Kill threshold. The level below which you stop, regardless of how promising the roadmap sounds.
Decision date. A calendar day on which the sponsor decides, with the data in front of them.

The space between ship and kill is where most pilots live, and it is where you need a rule. Ours is simple: one extension of fixed length, with a specific hypothesis about what will move the metric. If the second run does not clear the ship threshold, stop. Open-ended extensions are how a pilot quietly becomes a year of spend.

Step 5: Price the pilot so it is worth finishing

A free pilot has no deadline, because the vendor's incentive is to convert you, not to finish. A pilot priced at the full production rate makes the buyer nervous about committing. The structure that works in our experience is a fixed-price pilot that explicitly includes the eval harness and logging, followed by a separately scoped production phase. The buyer owns everything the pilot produces, whichever way the decision goes. The trade-offs between fixed and variable pricing are covered in fixed price vs time and materials.

Make sure the pilot also answers the architecture questions that affect production cost. In our own outreach engine, a pilot-scale comparison showed that a single model call which extracted the facts from a scraped page and drafted the email in one step beat a multi-step chain on both cost and output quality. Had we only tested the chain, we would have carried its cost into production. Use the pilot to run at least one such A/B on the core design, while the volume is small enough to make it cheap.

What production hardening actually adds

Buyers often ask why the production build costs multiples of the pilot when "it already works". The honest list, and where each item comes from if the pilot left proper receipts:

Retries, idempotency and dead-letter handling for every tool the agent calls. The failure taxonomy tells you which tools actually fail and how often, so you harden the top of the list first instead of wrapping everything.
Rate limits and budget caps so a loop cannot run up a bill overnight. The cost per completed task from the pilot is the number the cap is set against. Without it, the cap is a guess.
Permission scoping, so the agent only sees what the current user may see. In shadow mode the agent saw everything the human saw. Production has to narrow that, and the run logs show which data it actually used.
Monitoring for silent failures, not just errors. An agent that stops running still returns a 200. The eval set, replayed on a schedule, is the cheapest alarm for a model or prompt that has quietly degraded.
Human review queues and audit trails, which regulated buyers will need before sign-off. If the pilot ran in assisted mode, the edit-distance data already tells you how large that queue needs to be and which cases should land in it.

None of this is visible in a demo, and all of it is what separates a pilot that worked from a system you can put in front of customers. A pilot that produced real logs, a real eval set and a real failure taxonomy makes this list shorter and far more predictable to quote. A pilot that produced only a demo means the production team starts by rebuilding the pilot to find out what it should have measured.

A one-page pilot charter

If you take one thing from this article, write a single page before the kickoff with: the workflow, the outcome metric and its baseline, the data access mode, the four receipts, the ship and kill thresholds, the decision date, and who owns the decision. Share it with the vendor and the sponsor. A pilot run against that page either ships or teaches you something specific. Either result is worth paying for.

If you are planning an AI agent pilot and want a partner who will put those thresholds in the contract rather than argue about them later, let's talk.

Originally published on the Pykero blog.

How Much Does a Custom AI Web Scraping and Data Extraction Pipeline Cost?

Pykero — Sat, 03 Oct 2026 09:02:28 +0000

A custom AI web scraping and data extraction pipeline usually costs $8,000 to $25,000 to build when the target is well defined: one data model, a known set of source sites, and output that goes into a database or a CRM. Expect $40,000 or more when you need to defeat bot protection, handle hundreds of unpredictable layouts, or guarantee data quality that a human would otherwise sign off on. Running costs are typically a few hundred dollars a month, and the design decision that most affects that number is whether you extract in one LLM call or in a chain of them.

Below is how we price this kind of work, what pushes it up, and the mistakes that make founders pay twice.

What "AI scraping pipeline" actually means

Classic scraping is brittle by construction. You write CSS selectors against a page's HTML, and the day the site changes a class name your data goes stale or, worse, silently wrong. The AI version replaces the selector layer with a model that reads the page as text and returns the fields you asked for in a fixed schema.

A production pipeline has four parts, and each has its own cost profile:

Crawling and rendering. Fetching pages, executing JavaScript where needed, respecting rate limits and robots.txt. Tools like Firecrawl or Playwright do the heavy lifting here.
Extraction. Turning raw page content into structured records with an LLM, validated against a schema so bad outputs are rejected rather than stored.
Storage and dedup. Writing records to PostgreSQL or Supabase with proper keys, change detection, and history.
Orchestration and monitoring. Scheduling runs, retrying failures, and alerting you when a source quietly stops producing rows.

Founders usually budget for the first two and forget the last two. The last two are where a pipeline either becomes a reliable asset or a monthly fire.

Build cost by tier

These are ranges for an experienced team working with a modern stack. They assume you already know what fields you need and where they live.

Tier 1: focused extractor, $8k to $15k

One target data model (say, product listings or company profiles)
Five to fifty source sites with mostly static content
Scheduled runs, schema validation, writes to your database
Basic alerting when a run fails or yields far fewer records than usual

This takes two to three weeks. It is the right scope for a lead enrichment feed, a pricing monitor for your own category, or a content index for an internal search product.

Tier 2: multi-source with quality controls, $15k to $25k

Hundreds of sources with varied layouts, including JavaScript-heavy pages
Change detection so you only re-extract pages that actually changed
A review queue where low-confidence records go to a human before publishing
Per-source health dashboards and cost tracking

Three to five weeks. This is where most SaaS products that resell or display scraped data should land, because the review queue is what protects your reputation when a source changes its page structure.

Tier 3: adversarial or regulated, $40k and up

Sites with aggressive bot protection, logins, or session handling
Personal data in scope, which means retention rules, deletion flows, and audit logs
Output that feeds a decision (pricing, credit, hiring) and must be defensible
Proxy management, captcha budgets, and legal review as line items

Six weeks or longer, and the ongoing costs are materially higher. If you are in this tier, read our guide on data residency for Saudi Arabia and the UAE before picking infrastructure, because where the crawler runs and where the data lands both matter.

What drives the monthly bill

Once built, three things determine your running cost:

Page volume. Rendering JavaScript costs real compute. A self-hosted crawler on a modest VPS handles tens of thousands of pages a month; managed crawling APIs charge per page and get expensive fast at scale.
Tokens per page. A full product page can be several thousand tokens. Stripping navigation, footers, and scripts before the model sees the content cuts this by half or more, and a small model is often enough for extraction.
Number of LLM calls per record. This is the big one, and it is a design choice.

On that last point, we have direct experience. Our own outreach engine scrapes each prospect's website with a self-hosted Firecrawl instance and a local model, then writes a tailored email per company. We tested two designs: a multi-step chain that first extracted facts, then classified them, then drafted, versus a single call that extracts the facts and produces the draft together. The single call won on both cost and output quality. Fewer hops meant fewer places for context to get lost, and the token bill dropped because we were not re-sending the same page content three times. We wrote up the general argument in single call vs agent chains, and it applies directly to extraction pipelines.

With a single-call design and a self-hosted crawler, a Tier 1 or Tier 2 pipeline usually runs for a low three-figure monthly amount in compute and tokens. Add managed crawling, proxies, and a frontier model per page and the same volume can cost ten times that. If compliance or volume pushes you toward running the model yourself, our breakdown of self-hosting an LLM vs using an API covers the break-even.

Where founders overpay

Paying for a generic "AI agent" when they need a pipeline. Extraction is a batch job with a known schema. Our own outreach crawler is exactly that: fetch, strip, one extraction call, write. Nothing in it decides what to do next, and that is a large part of why it runs for a low three-figure monthly amount instead of ten times that. Agents add latency, cost, and failure modes you do not want in a nightly crawl.

Skipping schema validation. If the model returns a price as "contact us" and you store it as a string, your downstream code breaks a week later. Validate every record against a strict schema at extraction time and reject or quarantine anything that fails. In Tier 2 the same gate is what feeds the human review queue, so you are paying for the validation layer once and using it twice.

No change detection. Re-extracting every page every night is the most common reason an LLM bill surprises people. Most pages in a nightly crawl are identical to yesterday's, so hash the cleaned content and only call the model when the hash changes. This single step often matters more to the monthly bill than which model you pick.

Ignoring the robots and rate-limit layer. The Robots Exclusion Protocol is a standard, not a suggestion, and polite crawling is also what keeps you from getting your IP range blocked. Build this in from day one. Retrofitting it is painful.

Underestimating maintenance. Even an LLM-based pipeline needs someone to watch the dashboards. Sources die, layouts change enough to confuse the model, and legal terms shift. The "far fewer records than usual" alert in Tier 1 exists because the usual failure is silent: the run succeeds, the row count drops, and nobody notices until a customer does. Budget for it the same way you would for any production system; our post on AI agent maintenance cost gives a framework that translates well.

When you should not build this

Buy or use a hosted tool when:

You need raw page content, not structured fields, from a handful of sites. A crawler like Firecrawl alone gives you clean markdown, and you can skip the extraction layer entirely.
The data is available through an official API or a licensed dataset. An API does not change its class names on you, which is the whole problem this pipeline exists to solve.
The output is for one-off research rather than a recurring product need. Orchestration and monitoring are half the build, and they buy you nothing if you only run it once.

Build when the extracted data is part of your product, when quality failures cost you customers, or when the source set is large and messy enough that a hosted tool's generic extraction cannot keep up. The ten-page test below usually settles it: if a hosted tool gets your ten sample pages right out of the box, use it and move on. If it misses fields on even a few of them, you are building, and you already know which tier.

How we scope this

We start by asking for ten sample pages and the exact fields you want from each. From that we can usually tell within a day which tier you are in, whether a small model is enough, and whether any source will fight back. The first deliverable is a working extractor on those ten pages with validated output, so you see real records before committing to the full build.

If you have a data source you want turned into a reliable feed, let's talk.

Originally published on the Pykero blog.

Should You Build or Buy AI Meeting Transcription for Your Product?

Pykero — Thu, 01 Oct 2026 09:02:41 +0000

Buy the transcription itself. Build the part your users actually pay for: the summary, the extracted decisions and action items, and the push into their CRM or ticketing tool. The exceptions are real but narrow: audio in a language or vocabulary the hosted vendors mangle, a compliance rule that keeps audio on your own servers, or a minute volume so high that API fees eat your margin.

This guide walks through the three layers of a meeting-notes feature, where the money and risk sit in each, and how to decide without a six-month research project.

The feature is three layers, not one

Founders tend to say "we want AI meeting notes" as if it were one thing. It is three, and the build-vs-buy answer is different for each.

Capture. Getting the audio. Either a bot joins the call, you pull the recording after the meeting through the platform's API, or the user uploads a file.
Transcription. Turning audio into text with timestamps and, ideally, speaker labels.
Understanding. Turning the transcript into something useful: summary, decisions, follow-ups, structured fields, and the workflow that moves them somewhere.

Layer three is your product. Layers one and two are plumbing that a dozen vendors sell. Treat them that way until you have evidence otherwise.

Layer 1: capture, and why you should avoid bots if you can

A bot that joins Zoom, Meet, or Teams calls is the most visible way to capture audio. It is also the most expensive to maintain. You deal with waiting rooms, hosts who kick the bot, platform updates that change join flows, and consent prompts that vary by region.

Before committing to a bot, ask whether your users need notes during the call or after it. If after is fine, two cheaper routes exist:

Platform recordings. Zoom's cloud recordings and the Google Meet REST API let you fetch recordings and transcripts after the meeting with the user's permission. No bot, no join logic.
Upload. For sales teams that already record calls elsewhere, a drag-and-drop upload gets you to a working feature in days.

If you genuinely need live capture, vendors such as Recall.ai sell the bot layer as an API. Building your own with the Zoom Meeting SDK is possible, but it becomes a permanent maintenance line item. Budget for it as one.

Layer 2: transcription, and the three cases where building wins

For English and the major European languages, hosted speech APIs are good, cheap per minute, and improving every quarter. Deepgram and AssemblyAI both return word timestamps and speaker diarization in one call. Pick one, wrap it behind your own interface so you can swap later, and move on.

Build or self-host only when one of these is true:

1. Your audio is not what the vendors trained on

Gulf Arabic, code-switched Arabic and English, heavy medical or legal vocabulary, or call-centre audio with poor microphones all degrade hosted accuracy sharply. If your users are clinicians in Riyadh dictating in dialect, the vendor's English benchmark is irrelevant. This is where a fine-tuned open model earns its keep. We covered the economics of that path in what Arabic speech recognition costs, and the short version is: run a fifty-file evaluation on your real audio before you believe any vendor's accuracy claim.

2. Audio cannot leave your infrastructure

Healthcare and government buyers in Saudi Arabia and the UAE increasingly require that recordings stay in-country or on-premise. A hosted US API is then a non-starter regardless of quality. Open models like Whisper under the MIT licence, or the CPU-friendly whisper.cpp, can run inside your own VPC. Add pyannote for speaker diarization. Expect to own GPU capacity planning, model updates, and an evaluation harness. That is a real engineering cost, so only pay it when the contract requires it.

3. Volume makes the per-minute fee your biggest cost

If every seat records hours of calls per day, API fees stop being a rounding error. But do not model the crossover from a spreadsheet guess. The interface wrapper you put around the vendor in the first week is also your metering point: log minutes transcribed per customer per month from day one, so the crossover is a number you read off a dashboard rather than one you argue about.

Then price the self-hosted side honestly. The vendor's bill is one line. Yours is at least four: GPU rental, the engineer who owns capacity planning, model updates when a better open checkpoint ships, and the evaluation harness that tells you whether the new checkpoint is actually better on your audio. Teams that compare the vendor invoice against GPU rental alone always conclude they should self-host, and then discover the other three lines. Many products never reach the crossover. Some do within a year of launch. The metering tells you which one you are, and the four-line costing tells you whether to act on it.

Layer 3: understanding, where you should build

Here is where founders underinvest. The transcript is raw material. The summary, the extracted next steps, the "who owes whom what" table, and the one-click sync to HubSpot or a support queue are what the user screenshots and shares. No vendor knows your users' workflow, so this layer is yours.

Start with one structured call

The temptation is to build a chain: one call to clean the transcript, one to segment topics, one to summarize each topic, one to extract action items, one to format. Resist it until you have evidence a single call is failing.

We learned this in our own outreach engine, which scrapes a prospect's website with a self-hosted crawler and a local model, then writes one tailored email per company. We compared a multi-step pipeline against a single call that extracts the relevant facts and drafts the email in one go. The single call won on both cost and output quality. Every extra hop added latency, another place for the model to drift, and another prompt to maintain. The same pattern holds for meeting transcripts: ask for summary, decisions, action items with owners, and any CRM fields as one JSON object, validate the schema, and only split it when a specific field is measurably weak. We went deeper on the trade-off in single call vs agent chains.

Design the review step, not just the generation

Users will not trust auto-generated action items pushed straight into their CRM. Show the draft, let them tick or edit each item, then sync. This review UI is unglamorous and it is the difference between a feature people enable and one they turn off after a week. Keep the transcript timestamps so each extracted item links back to the moment it was said. That provenance is what makes a sales manager believe the summary.

Measure before you tune

Build a small evaluation set early: twenty to fifty real meetings with a human-written "ideal" summary and action list. Score each prompt change against it. Without this, every prompt tweak is a guess and every model upgrade is a gamble.

A decision table you can use this week

English-language SaaS, post-call notes, moderate volume: hosted transcription API, platform recordings or upload for capture, your own single-call understanding layer. Fastest path to revenue.
Live notes during calls: add a bot-as-a-service vendor. Avoid writing the bot yourself unless the bot is your product.
Arabic, medical, legal, or noisy audio: evaluate hosted vendors on your real audio first. If accuracy is unacceptable, fine-tune an open model and self-host it.
Regulated buyers requiring data residency: self-host transcription and the LLM in-region from day one. Retrofitting this later is painful.
Very high minute volume: start hosted, instrument cost per minute, and plan the migration to self-hosted once the crossover is in sight.

What this costs to get wrong

The expensive mistakes are symmetrical, and both come from skipping the cheap checks above.

The first team builds its own Zoom bot and self-hosted Whisper stack for an English-only sales product. Months go into waiting-room handling, hosts kicking the bot, and join flows that break on platform updates. None of it shows up in the product the customer sees, which is the summary and the HubSpot sync. A hosted API and platform recordings would have shipped the same user-facing feature in weeks.

The second team bolts a hosted US transcription API onto a product aimed at Saudi healthcare buyers. The demo works. Then procurement asks where the audio is stored, the answer is a US region, and the deal stalls while engineering rebuilds transcription and the LLM in-country under deadline pressure. The pipeline was wrapped behind an interface, so the swap is possible, but the diarization, the evaluation harness, and the GPU capacity planning all arrive at once instead of on a schedule.

Both are avoidable with the fifty-file evaluation on real audio and a frank conversation, before the architecture is chosen, about where your buyers' data must live.

If you are scoping a meeting-notes or call-intelligence feature and want a second opinion on which layers to buy and which to own, let's talk.

Originally published on the Pykero blog.

How Much Does a Custom AI Document Assistant Cost?

Pykero — Tue, 29 Sep 2026 09:02:30 +0000

A custom AI assistant that answers questions over your own documents typically takes a small team three to six weeks to build to a usable first version, and the cost is set mostly by how messy your documents are, how strict your access rules are, and whether the data can leave your servers. The language model is the cheapest and least risky line item. Everything around it is where budgets slip.

This guide breaks the cost into the parts you will actually be quoted for, so you can compare proposals and decide whether to build at all.

What you are really buying

"Chat with our documents" sounds like one feature. In practice it is four systems bolted together:

Ingestion: pulling files from SharePoint, Google Drive, a CRM, email, or a shared folder, and keeping them in sync.
Preparation: turning PDFs, scans, spreadsheets and slide decks into clean text with structure preserved.
Retrieval: indexing that text so the right passages come back for a question. This is the RAG layer we covered in RAG explained for founders.
Generation and UI: the model that writes the answer, plus the interface, citations, feedback buttons and logging.

Vendors who quote a single low number are usually pricing only the last part.

The cost drivers, in order of impact

1. Document quality and variety

A clean folder of Word files with sensible titles is a one week ingestion job. Ten years of scanned contracts, policies in six revisions, and tables that only make sense visually is a month. OCR, table extraction, deduplication and version handling are unglamorous engineering, and they are where most of the effort lands.

Ask any vendor how they handle a scanned PDF with a table in it. If the answer is vague, expect the assistant to confidently misquote your own pricing sheet.

2. Permissions

If everyone in the company may see everything, retrieval is simple. If the assistant must never show HR documents to engineers, or a client's file to another account manager, every query has to be filtered by who is asking. That means mirroring your source system's access control into the index and keeping it in sync when permissions change.

This is the single most underestimated item in document assistant projects. We wrote up the patterns in permission-aware RAG. Budget for it explicitly, because retrofitting it later means rebuilding the index.

3. Where it runs

Three realistic options, roughly in ascending build cost and descending running cost:

Cloud API for the model, your own database for the index. Fastest to ship. Documents are chunked and stored in something like PostgreSQL with pgvector, and the model is called per question. Per-query cost is small but never zero, and your text goes to the provider.
Cloud infrastructure you control, open-weight model. A GPU instance running a model through a server such as Ollama or vLLM. Data stays inside your account. You pay for the instance whether or not anyone is asking questions.
Fully on-premises. A box in your own rack or a private data centre. Highest setup cost, best answer to a regulator, and the only option for some healthcare and government buyers in the Gulf and the EU.

The trade-offs are laid out in detail in self-hosting an LLM vs API. The short version: if compliance forces the decision, the decision is cheap to make, and the cost is just the price of doing business in your sector.

4. Answer quality requirements

A tool for the sales team to find the latest case study can tolerate an occasional miss. A tool that nurses use to check a clinical protocol cannot. The stricter the bar, the more you spend on evaluation sets, citation enforcement, guardrails that refuse to answer when retrieval is weak, and human review loops.

The cost difference is not abstract. For the sales tool, a wrong answer means someone opens the wrong PDF and closes it. For the clinical tool, every answer needs a citation to the exact protocol revision, a refusal path when the retrieved passages do not actually contain the answer, and a clinician signing off on the evaluation set before launch. That review loop alone can add more engineering weeks than the entire sales pilot.

Insist on an evaluation set from day one: real questions from real staff, with the correct answer and the correct source document. Start with the ten questions your team asks most often, and grow it from the feedback buttons in the interface. It is the only honest way to know whether a change made the assistant better or worse, and it is what you should hold a vendor to.

Rough effort bands

Treat these as ranges for a scoped proposal, not a price list. Every serious quote should be preceded by a look at your actual documents.

Pilot, single source, no permissions, cloud API. One to two weeks of engineering. Good for proving the idea to a sceptical board.
Production for one department, permissions mirrored from one system, citations, feedback capture. Three to six weeks.
Company-wide, multiple sources, strict access control, self-hosted model, audit logs. Two to four months, plus ongoing maintenance.

Ongoing costs come in three flavours: model usage or GPU hosting, re-indexing as documents change, and someone owning the evaluation set and quality. The last one is the one companies forget, and it is why so many pilots quietly rot after launch.

Where the model itself fits

Founders often assume the choice of model is the expensive decision. It rarely is. Model costs keep falling, and switching providers behind a retrieval layer is a small job if the system was designed for it.

What matters more is how the model is used. In our own outreach engine, which scrapes each prospect's website with a self-hosted Firecrawl instance and a local model, we tried multi-step chains that first extracted facts, then planned, then drafted. A single call that extracts the facts and writes the email in one pass turned out to be both cheaper and better. The same lesson holds for document assistants: a well-prompted single call over well-retrieved passages beats an elaborate agent loop most of the time. We covered that in single call vs agent chains.

The practical consequence for your budget: spend on retrieval and document preparation, not on orchestration frameworks.

Build, buy, or wait

Buy if your documents already live in Microsoft 365 or Google Workspace, your access rules match those platforms, and you have no data residency constraint. The bundled assistants are good enough for many teams and the price is predictable. Run the same test on them you would run on a vendor: hand one your worst scanned PDF with a table and see whether it quotes the numbers correctly.

Build if any of these apply:

Documents are spread across systems that no single vendor indexes, such as a CRM, a shared drive and ten years of email.
Access rules are more complex than folder permissions, like the account-manager-per-client case above.
Data cannot leave your infrastructure, or your regulator wants to see exactly where it goes. This is the usual position for healthcare and government buyers in the Gulf and the EU.
You want the assistant embedded in your own product or workflow rather than in a separate chat window.

Wait if you cannot name ten questions your staff ask repeatedly that a document would answer. Without that list, you do not have a use case, you have a demo. Those same ten questions become the seed of your evaluation set, so writing them down costs nothing and tells you immediately whether the project is real.

Questions to ask any vendor

How will you handle our scanned PDFs and tables, and can you show me output from a sample?
How are permissions enforced at query time, and what happens when someone's access changes?
What does the evaluation set look like, and who maintains it after launch?
Which parts run on our infrastructure, and which parts call an external API?
What does re-indexing cost when we add a thousand documents a month?
If we switch models next year, what has to change?

A vendor who answers all six crisply is worth paying. One who redirects the conversation to which model they use is selling you the cheap part.

If you are weighing a document assistant for a compliance-heavy team and want a scoped estimate against your actual files rather than a slide, let's talk.

Originally published on the Pykero blog.

Must Your AI Sales Agent Tell Customers It's a Bot?

Pykero — Sun, 27 Sep 2026 09:02:49 +0000

Yes. If your AI sales agent talks to anyone in the EU, you are required to tell them they are talking to an AI, and that rule has been enforceable since 2 August 2026. If it runs on WhatsApp, Meta's messaging policy adds a second requirement: the customer must always be able to reach a human. Neither rule is hard to satisfy, and neither has to hurt your funnel, but both have to be designed in rather than bolted on.

What the law actually says

The relevant text is Article 50 of the EU AI Act, Regulation (EU) 2024/1689, which you can read in full on EUR-Lex. The transparency obligations are short and unusually plain for EU legislation:

Article 50(1): providers of AI systems intended to interact directly with natural persons must design them so that those people are informed they are interacting with an AI system, unless that is obvious to a reasonably well-informed, observant and circumspect person.
Article 50(2): providers of systems that generate synthetic text, audio, image or video must mark the output as artificially generated in a machine-readable format, where technically feasible.
Article 50(4): deployers who publish AI-generated text on matters of public interest must disclose it, unless a human editorially reviewed it and takes responsibility.

For a sales or support agent, 50(1) is the one that bites. The European Commission's own AI Act overview confirms the phased timeline: prohibited practices in February 2025, general-purpose model duties in August 2025, and most remaining obligations including Article 50 from 2 August 2026. The penalty ceiling for breaching these transparency duties, under Article 99, is the higher of 15 million euros or 3 percent of worldwide annual turnover.

Two points founders often miss:

Scope is about the user, not you. The Act applies to systems used by people located in the EU. A Dubai clinic with an Arabic-speaking WhatsApp agent still triggers it the moment a patient messages from Paris.
"Obvious" is a weak defence. A chat bubble that says "Hi, how can I help?" in a natural tone and answers in under a second is not obviously a machine to an ordinary customer. The regulator's test is the customer's perception, not yours.

WhatsApp has its own rule, and Meta enforces it faster than any regulator

If your agent runs on the WhatsApp Business Platform, read the WhatsApp Business Messaging Policy. It requires that businesses using automation clearly disclose that the person is interacting with an automated experience, and that they provide a clear way to request transfer to a human agent. Meta can throttle or shut a business number for violations, and appeals move at Meta's pace, not yours.

This matters more than the fine in practice. A regulator might take a year to notice you. A number ban takes your revenue channel offline on a Tuesday afternoon. If you are pricing a deployment, the escalation path is not optional scope; we covered how to build it in AI agent escalation paths, and it should already be in the estimate you got for what an AI WhatsApp sales agent costs.

Outside the EU

Other jurisdictions are converging on the same shape, and the one-line disclosure shown in the next section already covers most of them. California has required since 2019 that a bot used to sell goods or services identify itself, which a first message containing the words "AI assistant" does in its opening sentence. Several Gulf regulators publish AI ethics principles that name transparency toward end users, and public-sector and healthcare tenders in Saudi Arabia and the UAE increasingly ask for it explicitly. Go back to the Dubai clinic: one patient messaging from Paris pulls it under Article 50, a hospital tender at home asks for the same disclosure in writing, and a customer with a California number is covered by the 2019 bot law. Three rulebooks, one first message. Designing to the EU standard once is cheaper than tracking each market separately.

The design question: how to disclose without killing conversion

The fear is real: you built a sales agent because it converts, and now you are being told to open every conversation with a disclaimer. Here is what actually works.

Say it once, early, in plain words

One line at the start of the conversation is enough for Article 50(1). Keep it short and human:

Hi, I'm Layla, the AI assistant for Northwind Clinic. I can book appointments and answer questions. Type "human" any time to reach our team.

That single message satisfies both the AI Act and WhatsApp. It also does something useful for you: it sets expectations, so when the agent asks a qualifying question the customer does not feel interrogated by a stranger.

Do not fake a human name without the label

Giving the agent a name is fine. Giving it a name, a profile photo of a real-looking person, and no disclosure is exactly the pattern the law was written to stop. If you want a persona, pair it with the word "AI" or "assistant" in the first message and in the WhatsApp business profile description.

Make the handoff explicit, both ways

When the conversation moves to a person, say so: "Passing you to Omar from our team." When a human hands back to the agent after hours, say that too. Customers tolerate automation; they do not tolerate being unsure who they are talking to.

Keep the machine-readable marker in mind

If your agent sends generated documents, quotes or images, Article 50(2) expects a machine-readable marker such as metadata or a content credential. Most teams can satisfy this by keeping generated PDFs and images on a template pipeline that stamps provenance metadata. Ask your vendor how they handle it before you sign.

What we learned from our own outreach

We run our own outbound engine. It scrapes a prospect's website, uses a local model to pull out concrete facts about the company, and drafts one tailored email per prospect in a single call. The part relevant here is where the automation stops: a person reviews and sends, and a person answers every reply.

That boundary turned out to be commercially useful, not just compliant. We tested the call to action and found that asking for a simple "reply YES" consistently beat linking to a calendar. The reason is friction, but the effect on disclosure is the same: the moment someone replies, they are talking to a human, and we can say so honestly. We wrote more about the CTA finding in AI sales agent CTA design. If your funnel has a natural point where a human takes over, put the handoff there and be explicit about it. You get better replies and a cleaner compliance story at the same time.

A short checklist before you go live

First message discloses AI in the user's language, and it renders in the WhatsApp preview.
A human escape hatch exists, tested on a real phone, and it works outside business hours (even if the answer is "a person will reply at 9am").
Business profile description on WhatsApp mentions automated assistance.
Handoff messages are explicit in both directions.
Generated files carry provenance metadata, or you have a written reason why it is not technically feasible.
Logs record which turns were AI and which were human, so you can prove compliance if asked.
Contract with your vendor names who is the "provider" and who is the "deployer" under the Act, because the obligations differ.

The real risk is not the fine

The 15 million euro ceiling in Article 99 is the number everyone quotes, but it is not what should worry you. An AI agent that pretends to be a person, gets caught, and gets screenshotted is a brand problem before it is a legal one, and a WhatsApp number ban lands months before any regulator writes to you. Everything the rules ask for is already in the Layla message above: who the customer is dealing with, what it can do, and the one word to type to reach a person. Our own outreach passes the same test not because we wrote a policy but because the human boundary is real; the moment a prospect replies YES, a person is reading. Build the disclosure, the escape hatch and the AI-versus-human logs into the first sprint and you will never think about Article 50 again.

If you are scoping a WhatsApp or web sales agent for a market that includes the EU or the Gulf and want the disclosure, escalation and logging handled properly from day one, let's talk.

Originally published on the Pykero blog.

Should Your AI Agent Cite Its Sources?

Pykero — Sat, 26 Sep 2026 12:33:52 +0000

Yes, if a human will act on what the agent says without re-checking it. An agent that can show exactly which page, record, or document each claim came from is one you can audit, correct, and trust in front of a customer. An agent that cannot is a well-written guess, and the cost of that guess lands on your team, not on the model.

The question founders actually face is not whether provenance is nice. It is where it is worth paying for, how much it costs, and how to write it into a build so it cannot be quietly dropped when the deadline gets tight.

What "citing sources" really means in an agent

There are three very different things people mean by this, and vendors blur them on purpose.

Prompted citations. You tell the model "cite your sources" and it writes plausible references. These are text, not links to anything the system verified. They fail silently.
Grounded citations. The system retrieves specific passages, passes them to the model with stable IDs, and the model refers to those IDs. The system then maps IDs back to real documents. This is real provenance and is what RAG done properly gives you.
Verified citations. After generation, a check confirms that each cited passage actually supports the sentence it is attached to. This is the strongest form and the one you want in regulated or high-stakes work.

If a demo shows citations, ask which of the three you are looking at. If the answer is the first, the feature does not exist yet.

Where provenance pays for itself

Outputs a person sends or signs

Sales outreach, proposals, support replies, and anything with your company's name on it. When a rep hits send on an agent-drafted email that contains a fabricated detail about the prospect, you lose the lead and some reputation with it.

In our own outreach tool, the agent reads each prospect's website and drafts one tailored email. Early on we made a simple design decision: the model extracts the facts it plans to use and writes the email in the same call, and every fact stays tied to the page it came from. That let a reviewer glance at the draft, see which page justified the opening line, and reject drafts where the "fact" was a stretch. We covered why the single call beat the multi-step version on cost and quality in single call vs agent chains. The provenance came almost for free because we never let the source URL fall off the data as it moved through the pipeline.

Healthcare, legal, government, finance

Here provenance is not a feature, it is the product. A clinician will not act on a summary that says "the patient has a documented allergy" unless they can click through to the note. A procurement officer evaluating an AI vendor will ask how a given answer can be reconstructed after the fact. If you sell into these markets, put provenance next to access control in your architecture, because the two are linked: a citation is only safe if the reader was allowed to see the source, which is the whole point of permission-aware retrieval.

Anything you will be asked to defend later

Disputes, audits, and post-incident reviews all ask the same question: why did the system say this? A log of prompts and completions is not an answer. A log that says "this sentence was generated from passage 14 of document X, retrieved at this time, visible to this user" is.

Where it is not worth it

Be honest about this too, because provenance adds friction and you do not want it everywhere.

Creative and exploratory tools. Brainstorming names, rewriting tone, generating variants. Nobody needs a footnote on a headline suggestion, and there is no retrieved passage for it to point at anyway.
Internal assistants with a human in the loop by default. If the user is an expert who will always verify before acting, a short "here is what I looked at" list beats per-sentence citations. The reviewer in our outreach tool needed to know which page justified the opening line, not a reference stapled to every clause.
Code generation. Tests and review are the verification layer. Citations to documentation are pleasant but rarely change behavior.

The rule of thumb: the further the output travels from the person who requested it before someone checks it, the more provenance you need. A draft the rep reads before sending needs less than a support reply that goes out on its own.

What it costs to build

Provenance is one of the cheapest things to add early and one of the more annoying things to retrofit. The cost sits in three places.

Data plumbing. Every chunk you index needs a stable ID, a source reference, and ideally a character range. This is a schema decision made once. In the outreach tool it was a single field, the URL of the page each fact came from, carried on the record through every step and never dropped. Skipping it saves a day now and costs weeks later when you try to reconstruct which chunk produced which sentence.

Prompt and output format. The model has to be given IDs and asked to reference them in a structured way, and your parser has to handle the model getting it slightly wrong. Budget for a structured output format and a fallback for when the model cites an ID that was not in context. That case is the prompted-citation failure from earlier sneaking back in, and it must be flagged, not rendered as a link. Several providers now offer native citation modes that return source spans alongside text, which removes most of the parsing work. Check your provider's documentation for whether the model you are using supports it before building your own.

UI. Someone has to design how a citation looks and what happens when you click it. Underestimating this is the most common reason provenance features ship half-finished. A citation that opens the whole 40-page PDF at page one is technically present and practically useless. Deep link to the passage.

Token cost is real but secondary. Passing IDs and asking for structured references adds a modest overhead per call. If your budget is tight, the levers in LLM cost optimization matter far more than trimming citations.

How to write it into a vendor spec

Vague language here gets you the prompted version. Write acceptance criteria that can be measured.

Every factual claim in an output is linked to a retrieved passage, not to a document title. The link opens the passage, not the file.
A citation the model invents is treated as a defect. The system must reject or flag any reference to a source that was not in the retrieval set for that call.
Citation accuracy is a tested number. Take a sample of outputs, have a person check whether each cited passage supports the claim, and set a threshold. This belongs in the same document as your other evals in the vendor contract.
Provenance is logged, not just displayed. For any output, you can later retrieve the exact sources, their versions, and who was allowed to see them.
No citation, no claim. If the agent cannot find a source for something, it says so rather than filling the gap. This single rule removes most of the embarrassing failures.

If a vendor pushes back on any of these as "too heavy for an MVP," ask which of them they would be willing to have a customer discover was missing.

The quiet benefit: it makes the agent better

Teams that add provenance usually find their agents improve for a reason that has nothing to do with trust. Once every claim has to point at a passage, you can see which retrievals are weak, which documents are stale, and which prompts cause the model to overreach. Provenance turns "the agent was wrong" into "the agent was wrong because the retrieval returned the 2023 pricing page," which is a bug you can fix. Without it you are tuning in the dark.

If you are scoping an agent where the output leaves your building, treat provenance as part of the core build, not a phase two item. We can help you design the retrieval, the citation model, and the review flow so the thing you ship is checkable from day one. Let's talk.

Originally published on the Pykero blog.