DEV Community: Pykero

What AI Agent Maintenance Actually Costs After Launch

Pykero — Wed, 29 Jul 2026 09:01:59 +0000

Building an AI agent typically costs less than keeping it working. Budget 15 to 25 percent of your original build cost per year for maintenance, and expect the biggest expense to be things that fail silently rather than things that crash loudly.

Most founders scope an AI agent project like a normal feature: fixed build cost, ship date, done. Then three months post-launch, something quietly breaks, nobody notices for two weeks, and the "finished" project needs another sprint. That's not a scoping failure, it's a category difference. Traditional software fails loudly: a null pointer, a 500 error, a failed test. AI agents fail quietly, because the model doesn't know it's wrong. It just produces a confident, plausible, incorrect answer and moves on.

Why AI agents cost more to maintain than typical software

Silent failures are the real threat

We run our own outreach engine that scrapes each prospect's site with a self-hosted Firecrawl instance plus a local LLM, extracting facts and drafting one tailored email per company in a single call. The failure mode we hit most often wasn't the LLM hallucinating, it was upstream: a prospect's site redesigned its homepage, the scraper still returned a 200, and the agent kept writing emails referencing a service the company had quietly discontinued months earlier. Nothing crashed. Nothing alerted. The output just got worse.

That's the pattern to plan for. Your AI agent's inputs (scraped pages, third-party APIs, CRM records, PDFs) change without telling you, and the agent has no built-in way to know the ground shifted under it.

Model and API churn

The model provider behind your agent will deprecate versions, change rate limits, or adjust behavior on updates you didn't ask for. A prompt tuned carefully against one model's tendencies can behave differently against its replacement, even when the interface looks identical. We saw this on the local LLM behind our own outreach engine: swapping in a newer point release of the same model family produced tighter, less repetitive summaries of scraped page content, but it also started dropping the specific service names our personalization step depends on. That meant re-running our extraction prompts against a batch of past scrapes and comparing outputs line by line before we'd trust the new version in production. This isn't hypothetical, it's a scheduled event: every foundation model has a deprecation timeline, and "just swap the model string" undersells the retesting that responsible teams actually do first.

Data and workflow drift

Business rules change. Pricing tiers change. The support agent that was trained on your refund policy needs an update the day that policy changes, not whenever someone remembers to tell engineering. Static software degrades slowly through neglect; agent logic degrades the moment the business changes and nobody re-syncs it.

What a realistic maintenance budget actually covers

Monitoring and alerting: something needs to flag when output quality drops, not just when the process crashes. This is usually the most underbudgeted line item.
Data source health checks: scheduled validation that scrapers, APIs, and integrations are still returning what you expect, structurally, not just returning something.
Model version management: testing against new model releases before you're forced onto them, and keeping a rollback path.
Prompt and logic updates: a standing process for feeding business changes (pricing, policy, new products) back into the agent's instructions.
Fallback and escalation logic: a defined path for when confidence is low, so the agent hands off to a human instead of guessing.
On-call ownership: someone accountable when the agent misbehaves at 2am, even if the fix waits until morning.

None of this is exotic. It's the same discipline you'd apply to any production system with external dependencies, applied to a system whose failure mode is "confidently wrong" instead of "down."

A rule of thumb for budgeting

For a mid-complexity agent (a WhatsApp sales assistant, an internal support bot, a data-enrichment pipeline), plan on 15 to 25 percent of the initial build cost per year in ongoing maintenance, split roughly between monitoring/infrastructure and prompt/logic upkeep. Our own outreach engine sits comfortably inside that range: the recurring cost is mostly the hours spent re-checking scraper output after a site redesign and re-validating extraction prompts after a model swap, not new infrastructure spend. Agents touching money, health data, or compliance-sensitive decisions should be budgeted higher, because the cost of a silent failure is not "an annoying email," it's a real liability. If you're negotiating with a vendor, this is exactly the kind of commitment worth pinning down in the contract rather than assuming; see our breakdown of what to demand in evals as part of an AI vendor contract.

Build vs. buy changes the shape of the cost, not whether it exists

Choosing a no-code agent platform over a custom build doesn't remove maintenance, it relocates it. The vendor handles infrastructure and model upgrades, but you still own your prompts, your data connections, and your business logic, and now you're also exposed to the vendor's pricing changes and roadmap decisions. We've laid out that trade-off in more detail in custom AI agent vs. chatbot platform. Either way, run the numbers the same way you'd evaluate any recurring infrastructure cost, alongside model spend; our guide on LLM cost optimization covers the token-cost side of that math, and it pairs directly with the maintenance side covered here.

If you're scoping an AI agent and the proposal in front of you doesn't mention monitoring, model updates, or a maintenance retainer, ask why. It's either been priced in quietly (rare) or it hasn't been priced in at all (common, and expensive later).

Weighing whether an agent needs this level of upkeep at all, or whether a simpler workflow would do? Worth reading alongside AI agents vs. workflows before you commit to either the build or the budget. If you want a second opinion on what a realistic maintenance plan should look like for your use case, let's talk.

Originally published on the Pykero blog.

Permission-Aware RAG: What It Really Costs to Get Right

Pykero — Mon, 27 Jul 2026 09:02:03 +0000

Permission-aware RAG means your retrieval step filters by who's asking, not just what they asked — and that filter has to live inside the vector query itself, not as a second pass the LLM does after the fact. Most teams find this out the hard way: their RAG demo works great on a shared knowledge base, then someone asks it to enforce per-user document access and the whole retrieval layer needs to change.

Why "just add a permission check" doesn't work

The instinct is to keep your existing retrieval pipeline and bolt a check onto the end: retrieve the top-k chunks, then ask an LLM (or a rules engine) "should this user see this?" before passing context to the answer step. This fails in two ways:

It's non-deterministic. An LLM asked to police access control will occasionally get it wrong, and "occasionally wrong" is not an acceptable access-control policy, especially with health records, legal documents, or HR files in the mix.
The leak already happened. By the time you're filtering post-retrieval, the restricted content has already been pulled into a context window and sent somewhere. Logging, caching, or a stray debug trace can leak it before your filter ever runs.

The fix is to push permissions into the query, the same way you'd push a WHERE user_id = ? clause into a SQL query instead of fetching everything and filtering in application code. We've seen the same principle play out in our own tooling: in an internal outreach engine we built, a single call that filters and extracts in one pass consistently beat a multi-step chain that fetched broadly and reasoned about relevance afterward — cheaper, faster, and fewer places for something to go wrong. Permission-aware RAG is the security-critical version of that same lesson: filter at the source, don't filter after the fact.

What actually changes in your architecture

If you're running Postgres with pgvector or a managed vector store, permission-aware retrieval typically requires:

Permission metadata on every chunk. Not just a document_id — you need to know which roles, teams, or user IDs can see each chunk at index time. This usually means re-chunking and re-embedding your existing corpus, not a metadata patch.
Row-level security or an equivalent filter in the query path. Postgres has native support for this via row-level security policies, and Supabase builds directly on top of it for multi-tenant RLS. The similarity search and the access filter run as one query, so a user's vector search literally cannot return rows they're not permitted to see.
A permissions model that changes over time. Documents get shared, roles get revoked, teams get restructured. If your embeddings and your permissions live in different systems that sync on a schedule, there's a window where a fired employee's session can still retrieve documents they lost access to an hour ago. Real-time permission checks are non-negotiable once this goes near anything regulated.
Re-indexing strategy, not just retrieval logic. This is the part that blows up estimates. Adding ACLs to a RAG system that already has 50,000 indexed documents means you're not writing a new query — you're re-processing the entire corpus with permission tags attached, then validating that the tags are correct before you trust the system with sensitive content.

Where the cost actually goes

Teams budget for the model and the retrieval logic. On that same 50,000-document corpus, the real cost shows up in:

Data audit. Someone has to determine who's actually supposed to see what, and at 50,000 documents that's not a spreadsheet filled out in an afternoon — it's confirming access department by department, often against tribal knowledge that was never written down, before a single permission tag gets attached.
Re-embedding at scale. Re-chunking and re-embedding 50,000 documents isn't a job you queue up overnight and check on tomorrow — if the permission model changes what counts as one retrievable unit (splitting a shared folder into per-team sections, for instance), the chunk boundaries themselves change, so this is a re-processing pass over the whole corpus, not a metadata update layered on top of it.
Testing for negative cases. Testing that the system retrieves correctly is easy. Testing that it never retrieves something a user shouldn't see, across every role combination, is the part that takes real QA time — this is the same category of work covered in an AI agent security checklist, and it deserves the same rigor you'd apply to authentication code, not "we tried a few prompts and it seemed fine."
Audit logging. In healthcare or government contexts, you often need to prove, after the fact, exactly what a user retrieved and why — which means logging the filtered query, not just the final answer. This overlaps directly with what we've written about court-ready architecture for healthcare AI: the log has to be defensible, not just present.

When it's worth building vs. buying

If your access model is simple — one tenant, one shared knowledge base, everyone sees everything — don't build permission-aware retrieval at all. It's unnecessary complexity for a problem you don't have. Start with the basics in RAG explained for founders and move on.

Build it yourself when:

Your permission model is core to the product (multi-tenant SaaS, healthcare records, internal knowledge bases with confidential HR or legal content).
You need real-time revocation — access changes have to take effect immediately, not on the next sync.
You're in a regulated space where "we filtered it after retrieval" won't survive an audit.

Consider a managed layer (many enterprise search and RAG platforms now ship ACL-aware retrieval out of the box) when your permission model maps cleanly onto standard role/group structures and you don't need to customize the filtering logic itself. Whichever you choose, put the same scrutiny into the vendor's retrieval guarantees that you'd put into any AI vendor contract — see evals in AI vendor contracts for what to actually ask for.

The one-sentence version

Don't ask the LLM to enforce access control after retrieval — enforce it in the query that does the retrieving, and budget for re-indexing your corpus, not just writing new retrieval code.

If you're scoping a RAG system that needs real permission boundaries — healthcare records, multi-tenant SaaS, internal docs with confidential sections — let's talk.

Originally published on the Pykero blog.

What a HIPAA-Compliant AI Voice Agent Actually Costs

Pykero — Sat, 25 Jul 2026 09:01:25 +0000

A HIPAA-compliant AI voice agent for healthcare typically costs $40,000-$150,000 to build, depending on call complexity and EHR integration, plus $2,000-$15,000/month to operate. The build cost isn't dominated by the speech model — it's dominated by the compliance and data-retention layer wrapped around it.

Most cost estimates for "AI voice agents" quietly assume a sales or support use case, where a wrong transcription costs you an annoyed customer. In healthcare, a wrong transcription in a medication name or a dropped consent statement is a liability. That difference reshapes the budget.

Where the money actually goes

1. Speech recognition (10-20% of build cost)

This is the smallest line item, despite being the part founders worry about most. You have three options:

Managed API with a BAA (e.g., enterprise-tier Deepgram, Azure Speech, Google Healthcare API) — fastest to ship, but you're paying per-minute and locked into the vendor's accuracy on medical terminology.
Fine-tuned open-weight model — better accuracy on clinical vocabulary and accents, but adds MLOps overhead.
Self-hosted model — highest control over data residency, needed if your contracts or state law prohibit sending PHI to a third party.

If your patient population speaks Gulf Arabic or another dialect underserved by mainstream ASR, budget separately for this — see our breakdown on Arabic speech recognition costs for how accent and dialect coverage move accuracy and price independently of the base model choice.

2. Compliance infrastructure (30-40% of build cost)

This is where healthcare voice AI diverges hardest from a generic voice bot:

Business Associate Agreements with every vendor in the call path (ASR, LLM, telephony, storage)
Encryption at rest and in transit, with key management you can audit
Role-based access control on transcripts and recordings
Immutable audit logs of who accessed what patient data and when

The U.S. Department of Health and Human Services publishes the actual HIPAA Security Rule requirements — worth reading directly rather than trusting a vendor's compliance checklist, since "HIPAA-compliant" is not a certification anyone issues, it's a set of administrative, physical, and technical safeguards you're responsible for implementing.

3. Data retention and lifecycle policy (15-20%)

Retention isn't a settings toggle you flip once. You need:

Configurable retention windows per data type (call audio vs. transcript vs. structured extraction)
Automated deletion or archival on schedule
A defensible chain of custody if a recording ever needs to be produced for a malpractice claim or audit

If you're also building the surrounding patient record system, this overlaps with what we've written about court-ready architecture for healthcare AI — the same evidentiary standards that apply to clinical documentation apply to voice interaction logs.

4. Integration with EHR/EMR (20-30%)

The voice agent is worthless in isolation. Most of the real engineering effort goes into:

Pulling patient context before the call (so the agent isn't starting cold)
Writing structured summaries back into the EHR after the call
Handling the failure mode where the integration is down mid-call

5. Ongoing operations ($2K-$15K/month)

This scales with call volume and includes ASR/LLM usage, monitoring, and a human-in-the-loop review process for a sample of calls — which most healthcare compliance teams require regardless of how good your model claims to be.

The extraction pattern that actually saves money

A place teams overspend is call processing after the fact — running the transcript through multiple chained LLM calls (summarize, then extract entities, then classify, then draft the EHR note). In our own tooling, we found that a single well-structured call — extract facts and produce the structured output in one pass — consistently beat multi-step chains on both cost and accuracy, because each additional hop introduces a new place for the model to drop or hallucinate detail. The same principle applies directly to post-call processing of a patient conversation: one careful extraction call beats four cheap ones. We go deeper on why in single-call vs. agent chains.

Build vs. buy

Off-the-shelf healthcare voice AI platforms exist and can get you to a pilot fast, but almost none offer a BAA, configurable retention, or audit logging at the tier a small clinic can afford — those features show up once you're paying enterprise pricing. If you're evaluating vendors instead of building, put their claims through the same rigor you'd apply to an internal build; our guide to evals in AI vendor contracts has a checklist for pinning vendors to accuracy and compliance commitments in writing, not just in a sales deck.

For most clinics and health-tech startups, the deciding factor isn't cost — it's whether your patient population, call volume, and compliance obligations justify a custom build now, or whether a pilot on a managed platform buys you time to validate demand before you invest in the infrastructure above.

What to budget for a first version

Pilot (single use case, e.g., appointment reminders + rescheduling): $40K-$60K build, $2K-$4K/month
Mid-complexity (intake, triage questions, EHR write-back): $70K-$110K build, $5K-$8K/month
Full clinical documentation assistant: $120K-$150K+ build, $8K-$15K/month

These ranges assume you're not also building the EHR — if you are, add that scope separately, and check our voice agent latency checklist once you're in build, since a compliant-but-slow agent still fails the patient experience test.

If you're scoping a healthcare voice AI project and want a realistic estimate for your specific call volume and compliance requirements, let's talk.

Originally published on the Pykero blog.

Do You Need an MCP Server for Your Product?

Pykero — Thu, 23 Jul 2026 09:01:59 +0000

Build an MCP server only if you need external AI agents — Claude, ChatGPT, Cursor, or a customer's internal agent stack — to call your product directly and take real actions on a user's behalf. If you just want your own product to have an AI feature, you don't need MCP at all; you need a normal integration with an LLM API, which is a different and usually cheaper project.

Model Context Protocol (MCP) is the open standard Anthropic published for connecting AI agents to external tools and data sources (spec and docs). It's gained real traction fast — OpenAI, Google, and most agent frameworks now support it — which is why it's showing up in founder roadmaps this year. But "the industry adopted a standard" and "we should build one" are different questions, and we're seeing teams skip straight past the second one.

What an MCP server actually does

An MCP server is a thin adapter. It exposes a fixed set of your product's actions and data — "create an invoice," "look up a customer's order status," "schedule a shipment" — as callable tools, described in a schema an agent can read and reason about. The agent (running in Claude, a custom app, whatever) discovers your tools, decides which one to call, and gets structured results back.

Under the hood, it's still your existing API or database being called. MCP doesn't give you new capabilities — it gives external agents a standard, discoverable way to reach capabilities you already have. That's the whole value proposition: instead of building a bespoke integration for every agent platform your customers use, you build one server and every MCP-compatible agent can talk to it.

Who actually needs one right now

We ran into this exact decision with our own outreach engine — the tool we mention below that scrapes a prospect's site and drafts a cold email. A customer asked whether they could point their internal agent stack at it directly instead of going through our UI. That question is the entire test: someone with their own agent already running wants to call your product as a tool, not a website. Three situations where that's a real requirement, not a nice-to-have:

You sell to technical buyers who run their own agent stacks. If your customers are already wiring Claude or internal agents into their workflows and asking "can we connect to your API from our agent," an MCP server is the difference between a same-week yes and a custom integration quote.
Your product is itself an agent-orchestration layer and you need to expose sub-tools to a supervising agent — this overlaps heavily with the tradeoffs in agents vs. workflows: if a fixed workflow would do, you don't need agent-callable tools at all.
You're building a developer platform where "AI-native integration" is part of the pitch, not an afterthought.

If none of those describe you, an MCP server is a solution looking for a problem. Most SaaS products get more near-term value from a solid, versioned REST API with an OpenAPI spec — which almost every agent framework can already consume as tools without you writing an MCP layer at all.

The real cost of building one

For a product with five to ten core actions, building an MCP server is closer in scope to a focused API-integration project than a platform rewrite — usually a few weeks for one engineer, assuming the underlying logic already exists and is reasonably clean. Tool design is the part that actually eats the schedule, not the protocol itself — the same lesson we learned collapsing our outreach engine's two-step "extract facts, then draft email" flow into one call, below: deciding what to expose and at what granularity takes longer than writing the adapter code. The work breaks down into:

Tool design — deciding which actions to expose and at what granularity. This is the part teams underestimate.
Schema and validation — strict input/output schemas so the agent gets predictable errors instead of silent failures.
Auth and scoping — an agent acting on behalf of a user needs the same permission boundaries a human session would get, not broader ones.
Observability — logging every tool call with enough context to debug "why did the agent do that," because you will need it.

The cost isn't the protocol — it's the same discipline you'd want in any API you expose to a third party, plus the added surface area of something non-deterministic (an LLM) deciding when to call it.

The failure mode: too many tools, too granular

The most common mistake is treating MCP tool design like REST endpoint design — one tool per database table, one per CRUD operation. That gives an agent 40 narrow tools to choose between, which tanks both accuracy and latency, since the model has to reason about which of many similar-looking tools applies.

We hit a version of this problem building our own outreach engine, which scrapes a prospect's site and drafts a tailored cold email. Early versions split that into separate "extract facts" and "draft email" steps, calling the model twice. Collapsing it into a single call that did both, with the extraction as an intermediate output inside one prompt, beat the two-step chain on both cost and quality — fewer round trips meant less compounding error. The same principle applies to MCP tools: fewer, higher-level tools ("find_and_summarize_customer," not "get_customer" plus "get_orders" plus "get_notes") give the agent less surface area to get wrong. This is the same tradeoff we cover in single-call vs. agent chains — it applies just as much to tool design as to prompt design.

When to skip it entirely

Skip MCP if:

Your only goal is an in-product AI feature (chatbot, search, summarization) — that's a direct LLM API integration, and the cost considerations look more like LLM cost optimization than protocol design.
No customer or partner has actually asked for agent access — build the REST API well, and add an MCP layer later if demand shows up. It's a thin wrapper you can add without re-architecting anything.
You're choosing between a custom agent and an off-the-shelf chatbot platform for a single use case — that decision, covered in custom AI agent vs. chatbot platform, usually resolves before MCP even becomes relevant.

A pragmatic rollout path

Start by exposing your existing API as an OpenAPI spec, if it isn't already — most agent frameworks can consume that directly as tools with no MCP involved. If you see real demand from customers who specifically want MCP compatibility (increasingly common in enterprise procurement conversations), wrap three to five of your highest-value actions as an MCP server, instrument it well, and expand based on what agents actually call — not what seems complete on paper.

MCP is a real standard worth having on the roadmap if agent-driven integration is core to how your customers will use your product. It's not a checkbox to add to every SaaS product this quarter.

If you're trying to figure out whether your product needs one — or need help scoping and building it right — let's talk.

Originally published on the Pykero blog.

AI Cold Email Agents: Build vs. Buy for Founders

Pykero — Tue, 21 Jul 2026 09:02:14 +0000

Build a custom AI cold email agent when personalization quality is your actual bottleneck and you're prepared to own the pipeline. Buy an off-the-shelf tool like Instantly or Smartlead when your bottleneck is deliverability, warm-up, and sequencing — problems those platforms have already solved better than you will in a first pass. Most founders assume they need to build; most should start by buying and only build the one piece that's actually broken.

What "AI cold email agent" usually means in practice

Strip away the marketing and there are really three separate jobs:

Sending infrastructure — SPF/DKIM/DMARC, IP warm-up, inbox rotation, bounce handling.
Sequencing and tracking — follow-ups, opens, replies, unsubscribes.
Personalization — figuring out something true and specific about each prospect, then writing an email that uses it.

Off-the-shelf tools are excellent at (1) and (2). They're mediocre at (3), because true personalization means researching each prospect individually, and most platforms bolt on a generic "AI write this" button that runs the same prompt template for every lead. That's the gap a custom agent is actually solving.

The build option: what it costs and where it breaks

We run our own outreach engine for exactly this reason. It scrapes each prospect's site with a self-hosted Firecrawl instance, feeds the cleaned page content to a local LLM, and asks for two things in one shot: pull out three or four facts worth mentioning, and draft the email using them. No separate "research agent" that hands off to a "writing agent," no planning step, no critique loop — one call, one output.

That decision wasn't arbitrary. We tested the multi-step version first: a chain that scraped, then summarized, then planned an angle, then drafted, then self-reviewed. It was slower, cost more per lead in tokens, and — this was the surprising part — the output quality wasn't better. Extra steps gave the model more chances to drift from the source material or hallucinate a detail that sounded plausible but wasn't on the page. Collapsing it to a single call with the full scraped context in the prompt produced tighter, more accurate emails for less money. If you're deciding between a single call and an agent chain more generally, the tradeoffs are the same ones covered in single call vs. agent chains — this is that pattern applied to outreach specifically.

Building this yourself means owning:

Scraping reliability — sites change, block bots, or have nothing useful on the homepage.
Prompt and context engineering — deciding what to extract, how much page content fits in context, and how to fail gracefully when there's nothing good to extract.
Token cost at volume — a single call per lead is cheap, but cheap times 50,000 leads still needs a budget; see LLM cost optimization if you're scaling past a few thousand sends a month.
Sending infrastructure — an agent that writes a great email still needs somewhere reliable to send it from, which is usually the part worth buying rather than building.

Realistic cost for a first version: a few thousand dollars of build time for scrape-extract-draft, plus ongoing LLM API spend that's typically cents per lead if you stick to a single call. That number multiplies quickly if you add retries, multiple model calls for "quality," or a vector database to store scraped context you don't actually need to search later — resist adding those until you've measured that the simple version is actually underperforming.

The buy option: what you get and what you give up

Instantly, Smartlead, Apollo, and similar platforms give you deliverability infrastructure that took them years to tune, plus sequencing, tracking, and unsubscribe handling out of the box. Their built-in AI personalization is real but shallow — usually one prompt per lead using whatever fields you've imported (name, company, title), not a per-prospect research step. For volume outreach where the message doesn't need to reference something specific about the prospect, that's often good enough.

Where it falls short is exactly the case where personalization is your differentiator — for example, if your pitch depends on referencing something concrete on the prospect's own site, their tech stack, or a problem visible in their public-facing product. Generic tools won't do that research for you.

There's also a compliance layer that doesn't disappear just because AI is writing the copy: unsubscribe handling, sender identification, and (in the US) CAN-SPAM requirements apply the same way to AI-drafted emails as to human-written ones — see the FTC's compliance guide if you're building or buying and haven't checked this.

A hybrid that avoids most of the risk

The pragmatic middle ground, and what we'd recommend to most founders: keep an established platform for sending, warm-up, and sequencing, and build only the personalization step as a custom single-call agent that feeds its output into that platform via API or CSV. You get proven deliverability without rebuilding it, and you get the personalization depth that off-the-shelf AI features don't provide. This also caps your build scope to the one component where custom work actually pays off, rather than reinventing infrastructure that's already commoditized.

One more lever worth testing regardless of which path you pick: what you ask the prospect to do next. In our own tool, switching the call-to-action from "book a call" — a calendar link — to a soft "reply YES" consistently lifted reply rates. Scheduling friction kills otherwise-interested cold leads before they ever get to a call. It's a cheap test to run on top of whatever drafting approach you land on.

How to decide

If your current emails read as generic and that's costing you replies, build the personalization step — but start with one LLM call, not a chain. Our own test showed the multi-step scrape-summarize-plan-draft-review version cost more in tokens and produced worse output than a single call with the full scraped page in context, because every extra step was a chance to drift from the source or invent a detail that wasn't actually on the site.
If your problem is emails not landing in inboxes at all, buy sending infrastructure — Instantly and Smartlead have already spent years tuning warm-up and rotation, and no amount of AI-written copy fixes a deliverability problem.
If you're not sure which one is broken, instrument reply rate and inbox placement separately before writing any code. In our case the bottleneck turned out to be neither drafting nor deliverability — it was the ask itself. Swapping "book a call" for a soft "reply YES" moved reply rates more than any change to the personalization did.

If you're weighing whether this is worth building in-house versus scoping it out, let's talk.

Originally published on the Pykero blog.

RAG vs. Fine-Tuning: A Decision Framework for Founders

Pykero — Mon, 20 Jul 2026 09:01:56 +0000

Use RAG (retrieval-augmented generation) when your underlying data changes frequently or you need to show users where an answer came from. Use fine-tuning when the problem is the model's behavior itself — tone, output format, or a narrow task it keeps getting structurally wrong. Most teams we talk to reach for fine-tuning first because it sounds more "custom," then discover RAG would have solved 80% of the problem for a fraction of the cost and complexity.

What each one actually does

These solve different problems, and conflating them is where most founders lose weeks.

RAG retrieves relevant documents at query time and stuffs them into the prompt as context. The model's weights never change. You're extending what the model knows, not how it behaves.
Fine-tuning adjusts the model's weights on a training set of examples. You're changing how the model responds — its format, tone, or task-specific reasoning pattern — not giving it new facts.

The original RAG paper from Lewis et al. (2020) framed it exactly this way: pair a parametric model with a non-parametric memory you can update independently. That's still the core tradeoff a decade later — do you want the knowledge to live in the weights, or in a store you can edit without retraining?

When RAG wins

Pick RAG if any of these are true:

Your data changes weekly or daily. Pricing tables, inventory, policy documents, support tickets. Fine-tuning on a moving target means constant retraining.
You need citations. Healthcare, legal, and financial use cases usually require showing the source. RAG gives you a paper trail; fine-tuning bakes facts into an opaque weight matrix you can't audit.
You're pre-product-market-fit. RAG lets you swap the underlying model or adjust retrieval logic without touching a trained artifact. We cover the mechanics in our RAG primer and the storage layer in vector databases explained.
Budget is tight. No training run, no GPU rental, no MLOps pipeline to maintain. You pay per query, not per update cycle.

When fine-tuning wins

Fine-tuning earns its cost when the issue isn't missing knowledge, it's wrong behavior:

The model keeps ignoring your output format. You need strict JSON, a specific tone, or a domain-specific reasoning style (e.g., a triage workflow) that prompting alone won't hold consistently at scale.
You have thousands of labeled examples of correct behavior and the pattern is stable — not "new facts," but "do this task this exact way."
Latency and prompt size are a real constraint. Every RAG query pays a retrieval cost and a larger prompt. If you're doing millions of calls a day, a fine-tuned model with a short prompt can be meaningfully cheaper and faster than a long RAG context, once training is amortized. See our cost breakdown for how those numbers actually compare over time.
Your domain vocabulary or reasoning pattern is unusual enough that even good retrieval and prompting can't reliably steer a general-purpose model (e.g., a highly specialized clinical coding task).

OpenAI's fine-tuning documentation is a reasonable reference for what fine-tuning is actually good at: format adherence, tone, and narrow task consistency — not injecting new facts.

The hybrid case

In practice, the strongest production systems use both, but not on day one:

Ship with RAG and prompting. It's fast to iterate on and cheap to change.
Instrument everything — log every query, retrieval, and output.
Once you have real failure patterns (not hypothetical ones), fine-tune narrowly on the specific behavior that's broken.

Building a fine-tuning pipeline before you have production logs is designing in the dark. You're guessing at what "good" looks like instead of training on your actual failure modes.

What we've seen in practice

We run our own outreach tooling that scrapes a prospect's site and drafts a tailored cold email. Our first instinct was the "proper" architecture: an agent chain that extracts facts, then reasons about relevance, then drafts, then critiques its own draft. It was slower, more expensive, and produced worse emails than a single well-structured LLM call given the scraped page as context — essentially a RAG pattern with one document instead of a corpus. We only would have reached for fine-tuning if that single-call approach had failed to hold a consistent voice across hundreds of emails, and it didn't. The lesson generalizes: don't reach for the heavier tool before you've proven the lighter one is insufficient. The same instinct that overbuilds a multi-step agent chain when one call would do is the instinct that overbuilds a fine-tuning pipeline when retrieval would do.

A founder's checklist

Before committing budget, answer these:

How often does the underlying data change? Daily/weekly → RAG. Rarely/never → fine-tuning is viable.
Do you need to show sources? Yes → RAG, or at minimum RAG-plus-fine-tuning, never fine-tuning alone.
Is the failure mode "wrong facts" or "wrong format/behavior"? Wrong facts → RAG. Wrong behavior → fine-tuning.
Do you have labeled examples of correct behavior yet? No → you're not ready to fine-tune regardless of the use case.
What's your query volume? Low-to-medium → RAG's per-query overhead is fine. High and latency-sensitive → the amortized cost of fine-tuning starts to win.

If you're integrating either into an existing product rather than starting fresh, our guide to integrating AI into your product walks through the surrounding architecture decisions — auth, rate limiting, and fallback behavior — that matter as much as the model choice itself.

Neither approach is more "advanced" than the other — they solve different problems, and the fastest path to a working product is usually the cheaper one you can validate first. If you want a second opinion on which one fits your product, let's talk.

Originally published on the Pykero blog.

Single LLM Call vs. Multi-Step Agent Chains

Pykero — Sun, 19 Jul 2026 09:02:00 +0000

A single, well-designed LLM call almost always beats a multi-step agent chain on cost, latency, and reliability — unless the task genuinely requires the model to observe an intermediate result before deciding what to do next. Most "agent" features founders ask us to build don't need that. They need one good prompt with structured output.

Why chains are the default anyway

Multi-step chains are seductive because they mirror how a human would solve the problem: research, then draft, then review, then revise. Frameworks make it easy to wire up four LLM calls with a planner in front, and it demos well. We built exactly that shape ourselves, for our own cold outreach system: extract company facts, then summarize, then draft the email, then a separate "polish" pass — four hops because that's how a person would do it if you handed them the job one step at a time.

It's also where the shape breaks. Each hop is:

A new place to lose context. Our "summarize" step only saw the extraction step's output, not the original scraped page — so by the time the email got drafted, it was working from a lossy summary of a summary.
A new latency tax. Four sequential calls at 2-3 seconds each is 10+ seconds before a user sees anything, even before you add retries.
A new failure surface. In our pipeline, the "polish" step regularly rewrote details the extraction step had gotten right, because it had no visibility into why the draft said what it said — it just saw a draft and a mandate to improve it.

Anthropic's own guidance on this is blunt: start with the simplest solution, and only add agentic complexity — multi-step planning, tool loops — when a single call with good context genuinely can't do the job (Building Effective Agents). That's not a hedge, it's the pattern that holds up in production.

What we learned running our own outreach engine

We run our own cold outreach system: it scrapes a prospect's site with a self-hosted Firecrawl instance and a local LLM, then writes one tailored email per company. Our first version split this into stages — extract company facts, then summarize, then draft the email, then a separate "polish" pass. It was slower, cost more in tokens per lead, and the polish step regularly overwrote details the extraction step had gotten right, because it had no visibility into why the draft said what it said.

We collapsed it into a single call: extract the facts and draft the email in one prompt, with the extraction fields specified as structured output before the email body. Cost per lead dropped, and quality went up, because the model held the full context — the actual scraped page, not a lossy summary of it — the whole time it was writing. The lesson generalizes past outreach: if your "agent" is really just transform-input-to-output with no branching decision in the middle, splitting it into steps is pure overhead.

A simple test before you chain anything

Ask whether the task has a decision point that depends on information the model doesn't have yet. If yes, you likely need at least one loop — the model has to act, see what happened, then decide the next action. If no, every step's input is fully knowable before the model runs, and it belongs in one call with a good prompt, or a deterministic code pipeline with the LLM doing one clearly-scoped part.

Examples that genuinely need iteration:

An agent debugging code against a failing test suite (it must run the test to know what to fix next)
An agent navigating a website where the next click depends on what's on the current page
A support agent that needs to ask a clarifying question before it can act

Examples that don't, no matter how "agentic" they feel in the pitch deck:

Summarizing a document and extracting structured fields
Classifying and routing a support ticket
Drafting an email, a spec, or a status update from known inputs

We cover the broader version of this decision — when a task is a workflow versus a true agent — in our breakdown of agents vs. workflows. If you're choosing between building this in-house and buying a platform, custom agent vs. chatbot platform covers that adjacent decision.

The cost math founders actually care about

Chains cost more in three ways that compound — our own outreach pipeline paid all three before we collapsed it:

Token cost. Each call re-sends context (system prompt, prior outputs) that a single call would only pay for once. Our four-step version resent the scraped page's summary at every hop; the single-call version pays for the full page once.
Retry cost. If any step in the chain fails or returns malformed output, you either retry that step (adding latency) or retry the whole chain (adding cost). When our "polish" step returned something malformed, the fix was to rerun it — but by then the extraction and draft steps had already run and been paid for. A single call fails once, not four times.
Engineering cost. Our four-stage version needed orchestration code and state passing between extract, summarize, draft, and polish — code that disappeared entirely once we merged it into one prompt with structured output. That's ongoing maintenance surface, not just build cost.

If you're estimating what an AI feature will cost to run at volume, model both architectures against real sample inputs before committing — see our guide to LLM cost optimization for how to structure that estimate. We've also written about the diligence question that comes up once a vendor pitches you a multi-agent system: ask them to justify each hop, the way we describe in evals in AI vendor contracts.

Structured output does most of what "planning steps" claim to do

A lot of what multi-step chains are used for — extract this, then format that, then validate the other thing — is better solved with structured output: a JSON schema or function-calling spec the model fills in a single pass. Modern models are reliable at returning well-formed structured data when you specify the schema clearly, which removes the main reason teams reach for a "validation step" in the first place. Add a deterministic code-level check after the single call (does this field parse, is this value in range) instead of a second LLM call to check the first one's work.

Where we land

Default to one call. Give it the full context it needs, ask for structured output, and validate deterministically. Reach for a chain — or a real agent loop with tool use — only when the task has an actual decision point that depends on something the model can't know until it acts. That's a smaller category than most agent demos suggest, and it's the difference between an AI feature that's cheap and boring to run and one that quietly burns budget every time it re-reads its own output.

If you're scoping an AI feature and aren't sure which side of that line it falls on, let's talk.

Originally published on the Pykero blog.

Court-Ready Architecture: Why Audit Trails Matter for Healthcare AI

Pykero — Sat, 18 Jul 2026 09:02:17 +0000

Court-ready architecture is a system design discipline: every AI-driven decision, data access, and model output is logged immutably, tied to its inputs, and reconstructable months later without relying on memory or good faith. For founders building AI products in healthcare or govtech, that's the difference between handing an auditor a clean timeline and handing them a shrug. It costs more to build upfront, and it stops being optional the moment your product touches PHI, benefits eligibility, or any decision a patient, citizen, or regulator could contest.

What "court-ready" actually means

Most teams build logging for debugging: enough to figure out why a request failed last Tuesday. Court-ready logging is a different target — it assumes someone hostile, or at least skeptical, will ask "prove it" a year from now, about a specific decision, not a general system health question. Concretely: "why did the system flag this claim in March" has to be answerable even after the model has been redeployed twice since. That means:

Immutability. Logs can't be edited or deleted by anyone, including admins, without leaving evidence of the edit — an append-only audit_log table with insert-only permissions, not a row you can UPDATE.
Completeness. The record includes not just "what happened" but the exact inputs that produced it — the prompt version and retrieved context, not just the final output a human saw.
Chain of custody. You can trace a single decision from raw data in, through every transformation, to the output a human acted on, the same way a Postgres trigger writing to audit_log inside the same transaction as the mutation guarantees the log and the change can't drift apart.

This is a much higher bar than console.log and a rotating log file. It's closer to how event sourcing treats state — as an append-only sequence of facts rather than a mutable snapshot you overwrite.

The four things every court-ready system needs

1. Immutable audit logs

Every read and write of sensitive data gets an append-only record: who, what, when, from where. Postgres makes this cheap with a dedicated audit_log table and triggers, or a write-ahead pattern where the log write happens in the same transaction as the mutation — see our take on Postgres for startups for patterns that scale without needing a separate event store on day one.

2. Data provenance

If an AI model made a recommendation, you need to know which records fed it, which prompt version ran, and which model checkpoint produced the output — not just the final answer. This is exactly the gap most "AI agent" demos skip, and it's why evals in AI vendor contracts increasingly require vendors to expose this trail rather than a black box.

3. Access control history

Role-based access control tells you who can see data today. Court-ready systems also log who did see it, every time, forever. HIPAA's Security Rule is explicit about this under 45 CFR 164.312(b) — "audit controls" is a named, required safeguard, not a best practice.

4. Model versioning and decision reconstruction

If a model updates weekly, you need to know which version handled which request. Without that, you can't answer "why did the system flag this claim in March" once the model has since changed. NIST's AI Risk Management Framework treats this kind of traceability as a core governance function, not an engineering nice-to-have.

Why this matters more for AI than for traditional software

Traditional software is deterministic — the same input produces the same output, so a code review plus a test suite gives you confidence. AI systems don't offer that guarantee. A model's output depends on the exact prompt, the exact retrieved context, and the exact model version at that moment — the same three things item #2 above requires you to log. If you can't reproduce those three things, you can't explain the decision after the fact — you can only guess at what probably happened. In healthcare, that gap is a malpractice exposure. In govtech, it's a public records request you can't fulfill, or a procurement audit you fail. This is also why the CFR 164.312(b) "audit controls" requirement and the NIST traceability guidance cited above exist as named, separate obligations from access control itself — regulators have already priced in that AI-era software won't behave deterministically enough for access logs alone to answer "what happened."

This is also where AI agents raise the stakes compared to fixed workflows. A rules-based system has a finite set of paths, all of which are known in advance. An agent that dynamically decides which tools to call and in what order has effectively infinite paths — see AI agents vs. workflows for why that distinction changes your engineering risk profile. Every one of those dynamic decisions needs to be captured if you want to reconstruct it later.

The pattern that actually works: fewer hops, more logging surface

The instinct when building "auditable AI" is to add more infrastructure — a dedicated logging service, a lineage graph database, a compliance dashboard. In our own experience building an internal outreach tool that scrapes a prospect's site and drafts a tailored email, we found the opposite lesson: collapsing a multi-step chain (scrape → extract facts → plan → draft) into a single LLM call that does extraction and drafting together didn't just cut cost and improve quality — it also cut the number of places state could silently drift between steps. Every hop in a multi-step pipeline is a place where an intermediate result can go unlogged, get transformed without a record, or be overwritten before anyone captures it.

The same logic applies to audit trails: the simpler and more linear your data pipeline, the fewer places you need instrumented logging, and the easier it is to guarantee completeness. Before you build an elaborate lineage system, ask whether you can reduce the number of steps a piece of data passes through in the first place.

What this costs, realistically

Budget 15-25% more engineering time for the first version of a healthcare or govtech AI feature that needs to be court-ready — append-only schemas, request/response capture at the model boundary, and a retention policy that doesn't quietly purge the evidence you'll need in eighteen months. It's not free, and it's not optional once real patient or citizen data is involved. Security fundamentals like the ones in secure your web app are a prerequisite, not a substitute — access control without an audit trail tells you what's allowed, not what happened.

The teams that get burned are the ones who treat audit logging as a Phase 2 feature. You can't retroactively log a decision you already made. If your product is heading toward healthcare, government, insurance, or any domain where someone can contest an AI-driven outcome, build the audit trail into the architecture from the first sprint, not after the first incident.

If you're scoping an AI product for a regulated space and want a second opinion on what "court-ready" needs to look like for your specific data and decisions, let's talk.

Originally published on the Pykero blog.

AI Competitor Intelligence Agents: Build vs. Buy in 2026

Pykero — Thu, 16 Jul 2026 22:15:49 +0000

Buy a competitor-intelligence SaaS tool if you need broad pricing and feature tracking across many competitors with no engineering lift. Build a custom AI agent if you need signals specific to your sales motion — hiring patterns, support-ticket tone, positioning changes — that generic tools don't surface, and you're willing to own a small scraping pipeline. Most founders overbuild this: the winning architecture is usually simpler and cheaper than the multi-agent demo they saw on Hacker News.

When buying makes sense

Off-the-shelf competitive intelligence tools are good at the job they're built for: tracking pricing pages, changelogs, and review sites across dozens of competitors, with dashboards and alerts out of the box. If your need is "tell me when a competitor changes their pricing," buying is almost always cheaper than building. You skip proxy management, CAPTCHA handling, and the ongoing maintenance of scrapers that break every time a competitor redesigns their site.

Buy when:

You're tracking 10+ competitors and just need the basics (pricing, features, reviews)
Nobody on the team owns scraping infrastructure
The output feeds a dashboard, not a specific downstream workflow

When building your own makes sense

Custom builds earn their cost when the intelligence needs to plug directly into something you already do — feeding your sales team a per-prospect brief, flagging when a target account's tech stack changes, or watching a narrow set of 3-5 competitors for signals a generic tool doesn't track (job postings that reveal roadmap direction, support forum complaints, a sudden shift in landing-page messaging).

This is close to the same build-vs-buy calculus we lay out in custom AI agent vs. chatbot platform: buy for generic coverage, build when the value is in connecting the signal to a workflow only you have.

The architecture that actually works

The instinct with agent projects is to reach for a multi-step framework: one agent plans, another scrapes, another extracts, another writes. In practice this adds latency, cost, and failure points without adding accuracy. We saw this firsthand building our own outreach engine: it scrapes each prospect's site with a self-hosted scraper and a local LLM, and does the fact-extraction and email draft in a single call rather than a chain of agent steps. That single-call pattern beat the multi-step version on both cost and output quality, because there was less room for one agent's mistake to compound into the next one's.

The same pattern holds for competitor intelligence:

Scrape the target pages (pricing, changelog, careers, blog) on a schedule — a headless browser for JS-heavy sites, plain HTTP for static ones.
Extract and summarize in one call — feed the raw page content to an LLM with a single prompt that pulls out structured facts (price changes, new features, headcount growth) and drafts the summary, rather than splitting extraction and writing into separate agent turns.
Diff against the last run so you only surface what actually changed, not a fresh wall of text every time.
Route the diff to Slack, email, or a CRM field your sales team already checks.

This is the same "keep it to as few LLM calls as reasonably possible" principle we cover in LLM cost optimization — every extra agent hop is both a cost line item and a place for errors to creep in, a tradeoff we also walk through in AI agents vs. workflows.

If you do need actual tool-calling — say, the agent has to decide which pages to check based on what it finds — standardizing how your LLM talks to scraping and search tools is worth doing through Model Context Protocol rather than a bespoke integration per tool, since it keeps you from rewriting the tool-calling layer every time you swap models.

What it costs to build

The reason this stays cheap is architectural, not aspirational: it's the same self-hosted-scraper-plus-single-LLM-call shape as the outreach engine above, and that shape has no orchestration layer to build, debug, or pay tokens for. For a narrow, well-scoped version — 3-5 competitors, a handful of pages each, a weekly digest — that's one scrape job, one extraction prompt, and one diff step, which is why it lands in the low thousands of dollars of build time rather than the multi-agent-framework price tag founders often assume it needs. Costs rise with:

Scale: monitoring 20+ competitors means proxy rotation and rate-limit handling
JS rendering: sites built on heavy client-side frameworks need a real headless browser, not simple HTTP fetches
Freshness: near-real-time monitoring costs more to run than a weekly batch job
Structured output: pulling clean, consistent fields (not just a paragraph summary) takes more prompt iteration and testing

None of this requires a large team. It's closer to a two-to-four-week build for one engineer than a multi-quarter platform project — which is exactly why it's worth scoping tightly before buying a $500/month SaaS subscription you'll outgrow or underuse.

A decision checklist

Do you need coverage or a specific signal? Coverage → buy. Specific signal tied to your workflow → build.
How many competitors? Under 5, deeply → build. Dozens, shallowly → buy.
Who maintains it? If no one owns broken scrapers six months from now, buy.
Does the output feed a workflow or a dashboard? Workflow-integrated → build. Dashboard-only → buy.
What's your tolerance for wrong answers? If this feeds a sales rep's talking points, put real effort into evaluating output accuracy before you trust it — the same discipline we recommend in evals in AI vendor contracts applies just as much to a system you built in-house.

If none of the SaaS tools track what actually matters to your deals, or you're already scraping and drafting content elsewhere in the business, a lean custom agent is a reasonable one-to-two-sprint investment — just resist the urge to make it more architecturally complex than a scrape-then-extract pipeline actually requires.

Weighing whether this is worth building for your team? Let's talk.

Originally published on the Pykero blog.

Why Your AI Sales Agent's Call-to-Action Is Killing Replies

Pykero — Wed, 15 Jul 2026 23:49:58 +0000

The calendar-link CTA is quietly killing your AI sales agent's reply rate. Asking a cold lead to "book a call" forces them to leave the conversation and commit to a time slot before they've even decided they're interested — replace it with a low-friction ask like "reply YES" and replies go up, because the CTA finally matches the medium.

This sounds like a small copywriting tweak. It isn't. It's a mismatch between how conversational AI agents are supposed to work and how most teams still write their final message.

Why founders default to "book a call"

Most outbound AI agents — whether they run over email, SMS, or WhatsApp — get built by people who came from traditional sales-ops tooling, where the calendar link is the default endpoint of every sequence. It's measurable, it plugs into a CRM, and it feels like progress: a booked meeting is a concrete outcome you can report on.

The problem is that a calendar link asks for two commitments at once: "yes, I'm interested" and "yes, I'll block 30 minutes on a specific day." For a warm inbound lead who already wants to talk to you, that's fine. For a cold lead who just received a message from an AI agent they've never interacted with, it's too much, too fast.

The friction hiding inside a calendar link

Every additional step between "read the message" and "respond" loses people. A calendar link isn't one step — it's several:

Click the link (often on a phone, sometimes blocked by corporate email filters)
Wait for a new tab or app to load
Parse a grid of available times, usually in the sender's timezone
Pick a slot, enter contact details again, and hit confirm

Compare that to typing "YES" back into a chat window you're already in. There's no context switch, no new interface to learn, no calendar math. The lead stays inside the channel where they already made the decision to engage.

This is the same logic behind reducing steps in any conversion funnel — every extra field, click, or app switch is a chance for someone to bail. Outbound AI agents just make the effect more visible because reply rate is a clean, immediate signal, unlike a multi-week sales cycle.

What we learned running our own outreach agent

We run our own outbound engine at Pykero: it scrapes each prospect's site, extracts the facts that matter, and drafts one tailored message per company. Early on, every message ended with a "book a call" link, because that's what every template we'd seen elsewhere did. Reply rates were mediocre even when the personalization was clearly landing — people would respond to say the message was relevant, then never click through.

Swapping the closing line to a soft "reply YES and I'll send more details" changed that. The lead only has to make one small decision — say yes to hearing more — instead of scheduling a meeting with someone they've never spoken to. Once they reply, the conversation itself does the qualifying, and the calendar link becomes a natural next step instead of an opening demand.

Lower-friction CTAs that work in agent-driven outreach

The pattern generalizes past our own tool. If you're building or buying a custom AI agent versus a chatbot platform for outbound sales, push whoever's building it to test CTA friction as a first-class variable, not an afterthought.

For WhatsApp agents

WhatsApp is already a conversational channel, so a scheduling link feels like a jarring detour. A one-word reply ("YES," "INFO," a number to pick an option) keeps the lead inside the chat, which also matters for staying within Meta's messaging session and template rules — a reply reopens the conversation window without you needing to send another paid template message.

For email and SMS agents

Ask a single yes/no or multiple-choice question instead of presenting a menu of times. "Want me to send a 2-minute breakdown?" converts better than "Here's my calendar" because it's a smaller ask with an obvious next step the agent can automate — the reply itself can trigger the next message in the sequence.

How to test this without guessing

Don't take our word — or any blog's word — for which CTA wins in your funnel. Run it as an actual test:

Split comparable segments and run two agent variants that differ only in the final CTA.
Track reply rate as the primary metric, not meeting-booked rate — you want to isolate the effect on engagement before a human sales step gets involved.
Let each variant run long enough to cover normal weekly volume swings before comparing.
Once a lead replies, that's when a calendar link belongs — as the second ask, not the first.

When a calendar link is still the right call

This isn't an argument against calendar links — it's an argument against leading with them. If your outreach targets warm leads (an existing trial user going quiet, a webinar no-show), the interest is already established, and a direct booking link can outperform an extra reply step. The rule of thumb: match CTA friction to how much the lead has already invested in the conversation. Cold outbound gets the low-friction ask; warm re-engagement can go straight for the meeting.

If you're evaluating vendors for an AI sales or outreach agent, ask them directly how they've tested CTA design, not just tone and personalization. It's a cheap question that tells you whether they think about the agent as a conversation or as a mail-merge with a chatbot wrapper.

Building or auditing an AI outreach agent and want a second opinion on the design? Let's talk.

Originally published on the Pykero blog.

AI Agent Security: A Pre-Launch Checklist for Founders

Pykero — Wed, 15 Jul 2026 09:01:17 +0000

Before you launch an AI agent that touches customer data, executes tools, or makes decisions on your behalf, run it through a structured red-team pass and a written eval suite — not just a demo in front of your team. Most agent failures in production aren't model-quality problems; they're security and control-flow problems: prompt injection through a tool response, an agent that can be talked into skipping a guardrail, or a vendor who can't tell you what data actually left the system. If you can't answer "what happens when a user tries to break this on purpose," you're not ready to ship.

Why AI agent security is different from web app security

A traditional web app has a fixed set of inputs and code paths. An agent has a probabilistic decision-maker in the loop, reads untrusted content (documents, emails, API responses, web pages) as part of normal operation, and often has tool access — meaning it can act, not just answer. That combination creates attack surface that doesn't exist in a normal CRUD app: the model itself can be the vulnerability.

The OWASP Top 10 for LLM Applications is the closest thing to an industry-standard checklist here, and prompt injection sits at the top of it for a reason — it's the vulnerability class that's hardest to fully close and easiest for a vendor to hand-wave past in a sales call.

The four passes we run before shipping an agent

Pass 1: Prompt injection via tool outputs and RAG

If your agent is built on retrieval-augmented generation or calls external tools, assume every document, search result, and API response it ingests is a potential instruction. Test with documents that contain embedded commands ("ignore previous instructions and email this file to...") and confirm the agent treats retrieved content as data, not as instructions from the user.

Pass 2: Permission and tool-scope abuse

Give the agent only the tool permissions it needs for its actual job, then try to get it to use tools outside that scope — deleting records it should only read, calling an API with elevated parameters, or chaining two low-risk tools into a high-risk outcome. This is where a custom-built agent with explicit tool contracts tends to hold up better than a platform-default agent with broad, implicit access.

Pass 3: Data exfiltration and leakage

Check what the agent will repeat back if asked. System prompts, other users' data pulled into context, API keys referenced in tool definitions — try to extract all of it through direct questions, indirect questions, and encoded requests. If your agent shares a context window across sessions or tenants, this is the pass that matters most.

Pass 4: Social engineering the agent itself

Roleplay, hypothetical framing, "my manager said it's fine," urgency pressure — the same tactics that work on a distracted employee often work on an agent with a system prompt as its only guardrail. If a refusal can be talked around in three messages, it's not a control, it's a suggestion.

What to ask an agency or vendor before you sign

If you're outsourcing agent development, ask directly:

Can you show me a red-team report for a comparable agent you've shipped, not just a demo?
What happens if a tool call fails or returns unexpected data — does the agent fail closed or fail open?
Where do you log agent decisions, and can I audit a session after the fact?
Are evals and security testing written into the contract, or are they a one-time favor before launch?

A vendor who can't answer these concretely is telling you they haven't done this before at production scale.

Red-teaming vs. evals: you need both

Evals answer "does this agent do its job well and consistently." Red-teaming answers "does this agent fail safely when someone tries to make it misbehave." An agent can score well on every eval — accurate answers, good tone, low hallucination rate — and still be trivially hijacked by a malicious tool response, because evals typically test the happy path and red-teaming exists specifically to test the unhappy one. Frameworks like MITRE ATLAS catalog these adversarial techniques if you want a reference beyond OWASP's list.

A simple pre-launch checklist

[ ] Every tool the agent can call has an explicit, minimal permission scope
[ ] Retrieved/external content is tested for embedded instructions
[ ] The agent has been prompted to leak its system prompt, keys, and other users' data — and failed to
[ ] Failure mode for a broken or malicious tool response is defined and tested (fail closed by default)
[ ] Session logs are queryable after the fact, not just streamed and discarded
[ ] Someone outside the build team has tried to break it, unscripted, for at least a few hours

If any box is unchecked, that's your launch blocker — not a nice-to-have for v2.

The cost of skipping this

The math here isn't subtle. A red-team pass costs a fixed amount of engineering time up front. A data-leakage incident, a tool-abuse incident, or an agent that took an action a customer didn't authorize costs incident response, customer trust, and possibly a very uncomfortable disclosure conversation. Once an agent is integrated into your product and handling real users, retrofitting security is far more expensive than building it in before day one.

If you're building or shipping an AI agent and want a second set of eyes on it before launch, let's talk.

Originally published on the Pykero blog.

How Fast Should Your AI Voice Agent Respond?

Pykero — Tue, 14 Jul 2026 09:01:17 +0000

A production AI voice agent should start speaking within 800ms-1.2 seconds of the caller finishing a sentence. Past 2 seconds, callers assume the line dropped, start repeating themselves, or hang up. If you're evaluating vendors or building this in-house, latency isn't a nice-to-have metric — it's the difference between an agent people trust and one they route around.

Why latency decides whether people trust the agent

Human conversation has a rhythm. Research on turn-taking across languages found the average gap between one person finishing and the next starting is around 200 milliseconds — often even before the first speaker finishes (Stivers et al., PNAS). Nobody expects an AI to hit 200ms today, but the baseline it's competing against is that fast, not the 3-5 seconds a customer might tolerate from a slow web page.

On the interface side, Nielsen Norman Group's response-time thresholds are the classic reference: under 0.1s feels instant, up to 1s keeps the user's flow of thought uninterrupted, and past 10s they've mentally checked out. Voice compresses that scale — because there's no spinner, no progress bar, just silence — so a 2-second gap on a call feels much longer than the same gap on a webpage.

This matters most for anything transactional: booking changes, order status, support triage, outbound sales. If you're weighing this against a text-based channel, the latency bar for an AI WhatsApp sales agent is far more forgiving — a few seconds reads as normal typing time. Voice has no such cover.

Where the milliseconds actually go

A typical voice agent pipeline has three stages, and most implementations run them sequentially:

Speech-to-text (STT) — transcribing the caller's audio. 150-500ms depending on whether it's streaming or batch.
LLM generation — deciding what to say and generating the full response. 500ms-2s+ depending on model size, prompt length, and whether it's calling tools (looking up an order, checking a calendar) before it can answer.
Text-to-speech (TTS) — converting the full text reply into audio. 200ms-1s.

Add those up sequentially and you're easily at 1.5-3.5 seconds before the caller hears anything — even before network round trips. That's the gap that makes an agent feel broken.

The fix isn't a faster model, it's not waiting for the full response before you start talking. Streaming the LLM's output token-by-token into TTS, and speaking as soon as the first sentence is ready instead of the whole answer, is what actually collapses the perceived delay — the same idea captured in the "streaming sentences, not audio" approach some teams have landed on. It's a pipeline decision, not a model upgrade, and it's usually the highest-leverage fix available.

Tool calls make this harder. If the agent needs to check a database before it can answer ("let me check your order"), that lookup has to happen inside the gap, and it needs its own filler strategy — a natural acknowledgment ("one sec, pulling that up") rather than dead air. This is one of the reasons agentic voice systems need more careful orchestration than a simple scripted IVR replacement — the agent is doing real work mid-conversation, not just reciting text.

How to evaluate a vendor on latency

Don't take a demo video at face value. Ask for:

A live, unscripted call on your own network, not a pre-recorded demo run on idle infrastructure.
Time-to-first-audio-byte, measured — not "response time," which vendors sometimes quote as time-to-first-token from the LLM, ignoring TTS.
Latency under concurrent load. A single test call tells you nothing about what happens at 50 simultaneous calls, when GPU queuing and rate limits kick in.
What happens on tool calls. Ask them to trigger a real lookup (CRM, calendar, order system) mid-call and time the gap.
Regional routing. If your callers are in the Gulf or elsewhere outside the vendor's default region, ask where inference actually runs — a model hosted three continents away adds real, physics-limited round-trip time no amount of engineering fixes.

This last point compounds if you're also dealing with accented speech or a non-English language — Arabic ASR in particular has real accuracy-versus-latency tradeoffs, since higher-accuracy models for Gulf dialects are often slower and pricier than generic English STT.

If you're putting any of this in a contract, treat latency the same way you'd treat an accuracy number — define it, and hold the vendor to it. The same discipline used for evals in AI vendor contracts applies directly: without a measurable SLA, "fast" is whatever the vendor felt like on demo day.

Red flags

Vendor can't tell you their p50/p95 time-to-first-audio-byte off the top of their head.
Demo only works well on scripted questions — ask something off-script and watch the gap widen.
"Fast enough" is the only answer you get when you ask for a number.
No mention of streaming anywhere in their architecture — if they wait for a complete LLM response before TTS starts, you're locked into the slow path no matter which model they swap in.

The bottom line

Latency is an architecture decision, not a model-selection decision. A well-engineered pipeline on a mid-tier model will consistently beat a poorly-streamed pipeline on the best model available. Before you sign with a vendor or greenlight a build, get a real number, not a vibe — and if you're integrating AI into an existing product, budget engineering time for the streaming plumbing up front, because retrofitting it later is a rewrite, not a patch.

If you're scoping a voice agent and want a second opinion on whether a vendor's latency claims hold up, let's talk.

Originally published on the Pykero blog.