DEV Community: Pykero

AI Agent Memory: Build vs Buy

Pykero — Sat, 15 Aug 2026 09:01:19 +0000

Build your own agent memory when your retrieval pattern is simple and stable: a Postgres table with a few typed columns and a recency filter will outperform a platform integration in both cost and debuggability. Buy a managed memory layer only once you have many users, each accumulating their own long-lived memory graph, and no spare engineering time to own decay, summarization, and ranking logic yourself.

"Agent memory" became a crowded category almost overnight: every AI agent vendor now ships a "memory" product, and every founder building an agent asks whether they need one. The honest answer is that most teams conflate three very different problems and end up buying a platform for a problem that a single SQL table would have solved.

What "memory" actually means for an agent

Strip away the marketing and agent memory is one of three things:

Session state: what happened in this conversation. This is just context window management, not memory. Don't buy anything for this.
Structured facts: a user's plan tier, their last five orders, a preference they stated once. This is a database problem with a lookup key.
Emergent recall: the agent surfaces something relevant that nobody explicitly indexed, weeks after it was mentioned, ranked by relevance and recency. This is the only piece that's genuinely hard.

Most products that ask "do we need agent memory" are actually asking about the second bucket, and the second bucket is a facts table with a user_id, a type, a value, and an updated_at column. No platform required.

The build case

If your memory lookups are predictable, roll your own. A typical schema is a Postgres table keyed by user or tenant, a type enum for the kind of fact, a JSON value column, and pgvector on a summary field for the cases where you genuinely need semantic search rather than exact lookup. See the PostgreSQL docs on JSONB and pgvector for the primitives.

The advantage isn't just cost, it's control. When memory misbehaves (an agent surfaces a stale fact, or forgets something it should have retained), you need to be able to SELECT * FROM facts WHERE user_id = ... and see exactly what's stored and why it was or wasn't retrieved. A managed platform turns that into a support ticket. We've found the same principle holds in our own single-call vs agent chains work: the fewer indirection layers between your code and the underlying store, the faster you can diagnose why an agent did something odd.

Build also wins when:

You have one primary retrieval pattern (by user, by conversation, by entity) rather than open-ended "find anything relevant."
Your data volume per user is small (tens to low hundreds of facts), so brute-force filtering beats a specialized ranking engine.
You already run Postgres and don't want another vendor in your AI agent vendor evaluation list.

The buy case

Managed memory platforms earn their keep when the retrieval problem becomes genuinely open-ended: many users, each with a large and growing memory graph, where the agent needs to decide which memories are relevant to the current turn without you writing that ranking logic by hand. That's a real engineering problem involving decay functions, contradiction resolution (the user's preference changed, which fact wins), and summarization of old memories into compressed ones so the graph doesn't grow unbounded.

Buy when:

You're running consumer-scale, multi-tenant agents where memory volume per user will keep growing indefinitely.
You need automatic summarization or forgetting, and building that well is a multi-week project you don't have room for.
Your team's differentiation is the product experience, not the memory infrastructure, and the platform's pricing is genuinely cheaper than the engineering time to replicate it.

The risk on the buy side mirrors what we've written about self-hosting LLMs vs API cost and compliance: you're trading a one-time build cost for an ongoing per-call or per-user fee, plus a dependency whose roadmap you don't control. Read the vendor's actual retrieval and ranking behavior before committing, the same way you'd read evals in an AI vendor contract before signing.

A pattern that generalizes: match the tool to the retrieval shape

We ran into a version of this same build-vs-buy question on our own outreach agent, which scrapes each prospect's site and drafts a tailored email. Early on we considered a multi-step pipeline with a separate memory step to track what it had already learned about a company across runs. It turned out a single-call pattern with the scraped facts passed directly in context beat the multi-step version on both cost and quality, because the "memory" we needed was really just "the last scrape result," not an evolving graph. The lesson carries over directly: don't reach for infrastructure shaped for open-ended recall when your actual retrieval pattern is a lookup.

A simple decision test

Ask three questions before picking a side:

Can you name the query in advance? If you can write the WHERE clause today, build it.
Will memory per user grow without bound? If yes and you have no plan to prune it, buying decay/summarization logic saves real time.
Do you have a person who owns this for the next year? Built systems need an owner. If nobody has bandwidth, buying shifts that ownership to the vendor, at a cost.

Most agent products, especially B2B tools with a few hundred structured facts per account, land on build. Consumer-scale personalization products land on buy. Know which one you're building before you shop for a platform.

If you're scoping an agent and want a second opinion on whether memory is even the right layer to invest in, let's talk.

Originally published on the Pykero blog.

Managed vs Self-Hosted Agent Runtime: The Real Tradeoffs

Pykero — Thu, 13 Aug 2026 09:01:25 +0000

A managed agent runtime is a vendor-run service that executes your AI agent's loop for you, so you write the prompts and tool definitions and they handle orchestration, retries, and scaling. Self-hosting means you run that loop yourself. The right choice depends less on cost and more on who needs to see your data and how much control you need over failure behavior.

Most founders don't think about the runtime at all until something breaks: a tool call hangs, a retry storm triples your LLM bill overnight, or a customer asks where their conversation transcripts are stored. That's usually the first moment "managed vs self-hosted" becomes a real decision instead of a default.

What the runtime actually does

Strip away the marketing and an agent runtime handles four things:

State management between steps, so the agent remembers what it already tried
Tool execution, calling your APIs, databases, or external services
Retry and timeout logic when a step fails or hangs
Observability, logging what the agent did and why

A managed platform (LangGraph Cloud, CrewAI's hosted offering, various vertical agent platforms) gives you all four out of the box, usually with a dashboard. Self-hosting means you build or configure each piece yourself, often on top of a queue (SQS, Redis) and a worker process you control.

What you give up either way

With a managed runtime, you give up:

Visibility into exactly how retries and timeouts are implemented, which matters when an agent silently retries a non-idempotent action (charging a card twice, sending a duplicate WhatsApp message)
Data residency guarantees, since prompts and tool outputs typically transit the vendor's servers, which is a real problem for healthcare or government clients who require in-region processing
Pricing predictability, because most charge per execution or per token pass-through on top of your LLM bill

With self-hosting, you give up:

Time. Building reliable retry logic, dead-letter handling, and step-level observability is a multi-week project, not a config flag
The vendor's SLA. If your worker process falls over at 2 a.m., that's your on-call rotation now
Ready-made debugging tools. Vendors bake in step-by-step replay UIs that take real effort to replicate

Neither side is free. The question is which cost you can absorb right now.

Where this bites founders in practice

We've seen the tradeoff most clearly in agent execution patterns, not just hosting location. When we built our own outreach engine, the first version chained multiple LLM calls together: one to extract facts about a prospect, another to draft the email, a third to refine tone. Running that chain through a managed orchestration layer worked, but every extra hop added latency and another point where a hung tool call could cascade into a stuck workflow the dashboard didn't clearly explain. Collapsing it into a single call that both extracted facts (via a self-hosted Firecrawl instance) and drafted the email in one pass cut both cost and failure surface, because there was simply less runtime state to manage or lose visibility into. The lesson generalizes: the fewer moving parts your agent's loop has, the less it matters whether you're managed or self-hosted, and the more it matters once your workflow grows past two or three steps. That's a variant of the same tradeoff covered in single-call vs agent chains.

Serverless self-hosting has its own trap: if your agent's tool calls can run long (waiting on a slow third-party API, a document parse, a human-in-the-loop approval), a platform like AWS Lambda enforces a hard 15-minute execution ceiling per invocation, per AWS's documented limits. Teams that don't check this upfront discover it when an agent workflow that worked fine in testing starts silently truncating in production under real-world API latency.

A rough decision framework

Go managed if:

You're validating whether the agent use case works at all before committing engineering time
The data passing through isn't regulated or customer-sensitive
You don't yet have someone who owns infrastructure reliability

Go self-hosted if:

The agent touches health records, financial data, or anything with a data residency requirement, similar reasoning to why we recommend self-hosting an LLM over API calls once compliance is in scope
You're running enough volume that per-execution fees are a real line item, not a rounding error
You need custom retry semantics because some of your tool calls aren't safe to retry blindly (payments, outbound messages, database writes)

Either way, decouple your agent's logic (prompts, tool schemas, state definitions) from the runtime's specific SDK. If every tool function calls a vendor-specific orchestration API directly, migrating later means rewriting the agent, not just redeploying it. This is the same due-diligence question we walk through in our agent vendor evaluation checklist: ask the vendor upfront what happens to your logic if you leave.

The maintenance cost nobody quotes upfront

Whichever you pick, budget for ongoing maintenance, not just initial build. Managed runtimes still need someone watching cost per execution and updating prompts as your product changes. Self-hosted runtimes need someone patching the worker infrastructure and handling incidents. We break down what that ongoing cost typically looks like in AI agent maintenance cost, and it's rarely zero on either path.

If you're weighing this decision for a specific product and want a second opinion on where the line should sit for your data and volume, let's talk.

Originally published on the Pykero blog.

Self-Hosting an LLM vs. API: When It Actually Pays Off

Pykero — Tue, 11 Aug 2026 09:01:24 +0000

Self-hosting an LLM pays off in two situations: you're running high enough volume that GPU cost per token beats API cost per token, or you have a data residency requirement that API calls can't satisfy no matter the price. Outside those two cases, calling OpenAI, Anthropic, or Google's API is cheaper, faster to ship, and easier to maintain than running your own inference stack.

We get this question a lot from healthcare and govtech founders specifically, because "can the data leave our network" is often not a cost question at all, it's a legal one. So the framework below splits the decision into cost and compliance, because founders usually only need one of the two answers, not both.

The cost math, worked through

A single NVIDIA A100 80GB costs roughly $2 to $3 per hour on-demand from providers like Lambda or CoreWeave. Run it 24/7 for a month and you're at $1,500 to $2,200, regardless of whether it processes one request or one million. That's the core problem with self-hosting: you're paying for capacity, not usage.

Compare that to API pricing, where you pay per token and nothing when idle. For a workload doing, say, 5 million tokens a day on a mid-tier model, API costs typically land well under $1,500/month, and you have zero ops burden. The crossover point where a dedicated GPU starts winning is usually somewhere north of 20 to 50 million tokens/day of sustained, predictable traffic, and even then you need someone maintaining the serving stack (vLLM, TGI, or similar) and handling GPU failures, driver updates, and scaling.

We went through a version of this math ourselves. Our cold-outreach tool scrapes each prospect's site with a self-hosted Firecrawl instance and a local model to extract facts and draft an email in a single call. At our volume, running that step locally was cheaper and gave us more control over rate limits than routing every scrape through a hosted API, but we didn't self-host the actual sales copywriting, that still goes to a hosted model because the volume doesn't justify dedicated GPU capacity and the quality bar is higher. Mixing the two, self-hosted for cheap deterministic tasks, API for high-stakes generation, is usually the right shape for early-stage products. It's the same reasoning we lay out in LLM cost optimization: match the model tier and hosting model to the task, not the other way around.

Where the "hidden" self-hosting costs come from

Serving infrastructure. vLLM or Text Generation Inference handle batching and KV-cache management, but someone has to operate them, patch them, and handle OOM crashes under load.
Model updates. Open-weight models improve fast. Committing to self-hosting means you own the fine-tuning and evaluation cycle every time a better base model ships, instead of a provider swapping it in behind an API.
Redundancy. One GPU node is a single point of failure. Real uptime needs at least two, which roughly doubles the baseline cost we quoted above.

The compliance case, which is a different question entirely

If your driver is data residency, PHI, or a client contract that prohibits sending data to a third party, the cost math above is mostly irrelevant. You're not comparing dollars, you're comparing "can we legally do this at all." That's the situation a lot of our healthcare and government-adjacent clients are in, and it's the same territory we cover in court-ready architecture for healthcare AI.

A few things worth being precise about here:

Self-hosting solves the "data leaves our network" problem, but it does not by itself make you HIPAA compliant. You still need encryption at rest and in transit, access logging, and a documented retention policy around the model and its logs.
Managed API providers do offer BAA-covered, HIPAA-eligible endpoints (both OpenAI and Anthropic offer these for enterprise customers), so "we need HIPAA compliance" doesn't automatically mean "we need to self-host." Check that option before assuming you need your own GPUs.
Government and defense contracts are a different story. Many require the model and data to sit inside an accredited environment (FedRAMP, IL4/IL5, or fully air-gapped), where a public API is a non-starter regardless of any BAA. That's where self-hosting stops being optional.

Open-weight model quality, honestly

For narrow tasks, classification, extraction, structured triage, intent routing, an open-weight model fine-tuned or well-prompted on your data can match a frontier closed model, and you keep full control over latency and privacy. Meta's Llama family and Alibaba's Qwen models are both reasonable starting points for self-hosted deployments.

For open-ended reasoning, long documents, or anything where the failure mode is "subtly wrong answer that sounds confident," the frontier API models still have an edge. Don't self-host your way into a quality regression to save money on a task where the model's judgment is the product.

A practical decision path

Estimate real token volume, not requests. Multiply average tokens per request by daily request count. If you're under roughly 10 to 20M tokens/day, start with an API.
Check if a HIPAA/BAA-eligible endpoint from a major provider satisfies your compliance requirement. If yes, you likely don't need to self-host.
If neither cost nor compliance forces your hand, don't self-host. The ops overhead is a distraction from building product in the first 12 to 18 months of a company's life.
If you do self-host, budget for redundancy and a standing on-call rotation, not just the GPU line item.

Same logic that applies to picking between managed and bring-your-own-key providers in BYOK vs. managed LLM keys: the cheapest-looking option on paper is rarely the cheapest option once you count the operational load it puts on your team.

If you're trying to work out where your product actually falls on this line, let's talk.

Originally published on the Pykero blog.

How to Design Escalation Paths for AI Agents

Pykero — Sun, 09 Aug 2026 09:02:06 +0000

An AI agent should escalate to a human whenever it hits a pre-defined confidence boundary, touches money or an irreversible action, or sees an input pattern outside what it was built to handle. If you're not designing for that moment before you ship, you're finding out about it from an angry customer instead.

Why "just make it more accurate" isn't the fix

Most teams treat escalation as a failure mode to eliminate rather than a feature to build. The instinct is: if the agent is wrong sometimes, add more training examples, tune the prompt, add another retry. That works up to a point, then hits diminishing returns, because some fraction of real-world inputs are genuinely ambiguous. A refund request that references a policy exception. A support message in a dialect the model handles poorly. A sales lead whose intent doesn't match any of your qualification categories.

No amount of prompt engineering removes ambiguity from the world. What you can control is what the agent does when it encounters it: guess and hope, or stop and ask.

Guessing is cheap until it's wrong. And it compounds if the agent operates in a chain, since one bad guess several steps in can send everything downstream (see AI agents vs. workflows for why chains fail differently than single-call systems). An agent that escalates instead of guessing costs you a small amount of latency on the hard cases and nothing on the easy ones.

What actually triggers a good escalation

Three signals are worth building around, and they're mechanically different from each other:

Confidence threshold. A confidence signal doesn't have to be a probability score the model reports about itself. It can be something you compute externally, like how many concrete facts the agent actually extracted before it tries to act. That's the exact mechanism behind the outreach-agent gate described below: not "is the model sure," but "does it have enough material to work with." Below whatever floor you set, stop and route to a human instead of returning the best guess.
Stakes classification. Tag actions by blast radius before the agent runs, not after. Sending a templated email is low stakes. Issuing a refund on a request that cites a policy exception, or messaging a customer on something with legal implications, is high stakes. High-stakes actions get a lower autonomy threshold regardless of confidence, because being 90% sure isn't good enough when the 10% failure is expensive or irreversible.
Out-of-distribution detection. Log what the agent has actually seen in production. A support message in a dialect the model wasn't tuned on, or a sales lead whose intent doesn't match any qualification category you built for, are both out-of-distribution in the same way: the input looks nothing like your training or eval set, and that's a signal independent of how confident the model claims to be about its own answer.

None of these require a research team. They require you to decide, up front, what "I don't know" looks like for your specific agent, and to build a path for it that isn't a generic error message.

Build the queue before you need it

The most common mistake we see is agencies and in-house teams building the happy path first and bolting escalation on after a bad outcome forces the issue. By then it's reactive: a human is triaging a mess instead of catching it at the decision point.

Build the escalation surface as part of the initial scope:

A place for flagged cases to land. This can be as simple as a Slack channel with the full context: the raw input (the refund request with its policy citation, the dialect message, the off-taxonomy lead), what the agent tried, and why it flagged rather than acted. Don't make a human dig through logs to reconstruct what happened.
A clear resolution path. Someone needs to own responding to escalations within a defined window, or the queue becomes a graveyard and the agent's flags become pointless. If nobody answers, the customer experience is worse than if the agent had just guessed.
A feedback loop back into the agent. Every human resolution is training data. If the same category of case keeps escalating, like the same dialect or the same off-taxonomy lead type, that's a signal to either expand the agent's scope for that category or accept it'll always need a human and design the UX around that permanently.

A pattern from our own agent work

We run an outreach agent internally that scrapes each prospect's site and drafts one tailored email per company. Early versions tried to always produce a draft, even when the scraped page had almost no usable content, a thin "coming soon" site, or a page that was mostly navigation with no actual business description. Forcing a draft out of thin content produced generic, obviously-templated emails that hurt more than they helped.

The fix wasn't a better prompt. It was adding a check: if the extracted facts fell below a minimum threshold of specificity, the agent skips the draft and flags the company for a human to either research manually or drop from the list. That one gate improved the average quality of what actually got sent, because the agent stopped forcing output in cases where it had nothing good to say. The lesson generalizes: an agent that can say "I don't have enough to work with" is more useful than one that always produces something.

Where this fits in scoping a project

If you're evaluating a vendor or planning your own build, escalation design should show up in the initial architecture discussion, not as a change request after launch. Ask any agency pitching you an agent: what happens when it's wrong, and how does it know? If the answer is "we'll monitor it and fix issues as they come up," that's a maintenance cost you're signing up for indefinitely (see our breakdown of AI agent maintenance costs). If the answer includes a specific confidence mechanism and a defined human handoff, that's a team that's thought about failure, not just the demo.

It's also worth checking this alongside your broader risk posture. Escalation design overlaps with the same questions covered in an AI agent security checklist: what can this system do without a human in the loop, and who's accountable when it does something wrong.

The bottom line

Escalation isn't a fallback you add when things break. It's the mechanism that keeps things from breaking in the first place, by giving the agent a legitimate third option beyond "succeed" or "fail silently." Design it at the same time you design the happy path, tie it to concrete triggers (confidence, stakes, distribution shift), and staff the queue like it matters, because the cases that land there are, by definition, the ones your agent couldn't handle alone.

If you're scoping an agentic system and want a second opinion on where the escalation boundaries should sit, let's talk.

Originally published on the Pykero blog.

Do You Need an llms.txt File?

Pykero — Fri, 07 Aug 2026 09:01:30 +0000

llms.txt is a plain markdown file you put at yourdomain.com/llms.txt that summarizes your site for AI models. It's worth adding, it takes an afternoon, and it costs you nothing. But don't confuse it with an actual AI visibility strategy: no major AI product has confirmed it reads the file, and the things that really determine whether your site gets crawled and cited by AI answers are the same things that determine whether Google can crawl it.

Where llms.txt came from

The spec was proposed in late 2024 by Jeremy Howard at Answer.AI, modeled loosely on robots.txt: a single, predictable location where a site tells automated readers what matters. The idea is that a language model has a limited context window and can't (or shouldn't have to) crawl your entire site to answer a question about it, so you hand it a curated index instead: your product pages, your docs, your pricing, in one short file with links and one-line descriptions. The full spec is at llmstxt.org.

It's a reasonable idea. The problem is adoption. As more than one developer has pointed out publicly this year, nobody has confirmed their crawler actually fetches it. OpenAI, Anthropic, and Google have not published documentation committing to parse llms.txt as part of how ChatGPT, Claude, or Gemini answer questions about your product. That doesn't mean it's useless, it means you should size the investment to match the uncertainty.

What it does and doesn't do

What it does: gives you a clean, low-effort way to hand-curate what an AI system sees if it does decide to look. It costs nothing to maintain if your site doesn't change often. It's a reasonable hedge, the same way you'd add a sitemap.xml even though most of your traffic doesn't come from crawlers reading it directly.

What it doesn't do:

It's not a ranking signal for Google or any traditional search engine.
It doesn't override or supplement your actual page content, if the linked pages are JavaScript-rendered client-side with no server-rendered HTML, an AI crawler that does try to follow the links will hit the same wall a search engine crawler does. We've written about this tradeoff before in the context of server-side rendering and SEO.
It doesn't fix a slow, bloated site. If your Core Web Vitals are bad, an AI agent trying to fetch and parse your pages within a reasonable timeout will bail the same way a human does.

The decision framework

If you're a founder or CTO deciding whether to spend engineering time on this, here's the honest breakdown:

Worth doing

You have a docs site, developer product, or API that AI coding assistants (Claude Code, Cursor, Copilot) might reference when a user asks "how do I integrate X."
Your marketing site already has clean, server-rendered pages, so llms.txt is additive, not a patch over a broken foundation.
You can generate it once and regenerate it on deploy with a small script rather than hand-maintaining it forever.

Not worth prioritizing

Your core pages are client-side rendered with no SSR, meaning the links in your llms.txt point to content a crawler can't actually read anyway. Fix that first.
You're treating it as an SEO strategy. It isn't one. Google has not indicated llms.txt affects ranking at all.
You'd need to build custom tooling to keep it in sync with a fast-changing site. At that point the maintenance cost exceeds the unconfirmed upside.

What actually gets you cited by AI answers

If the real goal is "when someone asks an AI assistant about a problem we solve, we want to show up," the leverage is almost entirely in things that predate llms.txt:

Server-rendered, crawlable content. Every major AI crawler (GPTBot, ClaudeBot, PerplexityBot) behaves like a search crawler: it fetches HTML, and if your content only appears after a client-side fetch, it often sees an empty shell.
Clean semantic structure. Real headings, real paragraph text, schema.org markup where relevant. This is the same discipline that makes a page rank well and the same discipline that makes it easy for an LLM to extract a clean answer.
Fast, stable pages. Crawlers time out. A bloated bundle that takes eight seconds to become interactive gets skipped the same way a slow page gets a lower crawl budget from Google.

This lines up with something we've seen directly in our own tooling, not on the SEO side but on the consumption side. We run an outreach engine that scrapes each prospect's site with a self-hosted Firecrawl instance and a local model to draft a tailored email per company. The pattern that actually worked reliably was a single call that extracts the facts we need and drafts the email in one pass, not a multi-step chain that tries to crawl a sitemap, summarize each page, then synthesize. The lesson translates directly to llms.txt: a clean, well-structured page that an LLM can read in one pass beats a curated index pointing at pages the model still has to fight to parse. If you're weighing similar tradeoffs in your own AI features, we've written more on single-call versus agent-chain design and on RAG for founders if the underlying question is really "how do we make our content retrievable by an LLM," which is the same problem from a different angle.

How to actually implement it

If you've decided it's worth the afternoon:

List your 10-20 most important pages: product, pricing, docs, key integration guides.
Write one honest sentence per page, no marketing copy, just what's there.
Serve it as a static file at /llms.txt using the format from the spec (H1 title, blockquote summary, H2 sections with linked bullet points).
Regenerate it as part of your build if your page set changes often, don't let it go stale.
Move on. Don't build a dashboard for it.

llms.txt is a cheap insurance policy, not a strategy. Spend real effort on server-rendered, fast, well-structured pages, that's what both search engines and AI crawlers actually need to cite you, and it's work you should be doing regardless of whether any model ever reads your llms.txt file.

If you're trying to figure out whether your site or product is actually AI-crawlable, and not just checking a box, let's talk.

Originally published on the Pykero blog.

Why Average Latency Is the Wrong Metric for AI Agents

Pykero — Wed, 05 Aug 2026 09:02:45 +0000

Average response time is the wrong number to optimize for AI agents because it hides exactly the requests that break trust: the slow tool call, the retried LLM step, the request that timed out and silently fell back. Track p95 and p99 latency per step instead, and ask any vendor for the same before you sign a contract.

The problem with averaging

Say your agent responds in 800ms on average. That sounds fine. But if 90% of requests finish in 400ms and the remaining 10% take 6 seconds because they hit a retry, a rate limit, or a slow downstream API, the average buries the part of the distribution your users actually feel. Nobody experiences the average. They experience their own request, and for one in ten users, that request is 15x slower than what your dashboard implies.

This is not a new idea in distributed systems generally, it's why the Google SRE book treats percentile latency (p50, p95, p99) as the standard for monitoring, not the mean. AI agents make this worse than typical web services because the tail is fatter: LLM inference time is itself variable, tool calls can hang, and agents often chain multiple calls where one slow link stalls the whole request.

Where the tail comes from in agent systems

A single well-scoped LLM call has one source of latency variance: the model's inference time, plus network. An agent that chains steps, retrieve context, call a tool, call the model again, validate output, call the model a third time, multiplies that variance at every hop. Each step also carries its own failure and retry probability, and retries don't just add latency, they add it unevenly. A request that needs one retry on step 3 might take twice as long as one that sails through, and that unevenness is exactly what averages erase.

We saw this directly building our own outreach tool, which scrapes a prospect's site and drafts a tailored email. Early versions used a three-step chain: extract facts from the scraped page, draft the email, then refine it. The extraction step was the one at the mercy of someone else's server, some prospect sites loaded fast, some were slow or bloated with tracking scripts, and however long that page took to fetch and parse became a floor under everything after it, since drafting couldn't start until extraction finished. The average looked fine because most prospect sites are fast. The tail was rough because the slow sites weren't rare, they were just unevenly distributed, and each one produced a full-length stall that a fast average never showed. Collapsing extraction and drafting into a single well-designed call didn't make individual sites load faster, but it removed the sequential dependency, there was one less handoff for a slow fetch to block. If you're deciding between an agent chain and a single call for your own product, that tradeoff is worth thinking through before you build, see our breakdown of single-call vs agent chains.

What to measure instead

For any AI agent, whether you're building it or evaluating a vendor's, ask for these numbers broken out by step, not just for the request as a whole:

p50 (median): what a typical request feels like.
p95: what one in twenty users experiences. This is usually where the real product complaints start.
p99: your worst-case tail. For a support bot handling thousands of conversations a day, p99 is not an edge case, it's dozens of real conversations every day.
Time to first token vs. total completion time: if the agent streams output, users tolerate a slower total time much better than a slow start. Measure both separately.
Retry rate and where retries happen: a 2% retry rate on one step sounds small until it's the step every request depends on.

Tools like OpenTelemetry exist specifically so you can trace latency through each step of a distributed call, including agent chains, rather than only seeing the total. If your current stack (or a vendor's) can't show you per-step traces, that's itself a useful data point.

How this shows up in vendor conversations

If you're evaluating an outside team to build or operate an AI agent for you, latency claims are one of the easiest places for a pitch to overstate reality. "Sub-second responses" is a claim about the average, almost always, and it's exactly the kind of number that hid the stall in our own outreach tool until we broke it down by step. The right follow-up questions are:

What's the p95 and p99 under realistic concurrent load, not a single warm request in a demo?
Which step in the pipeline is slowest, and how does that change under load?
What happens when a downstream call (a database, a search index, another API, or in our case a prospect's own website) is slow? Does the agent degrade gracefully or hang?

These questions belong in the same conversation as pricing and support terms. We cover the rest of that evaluation, including questions that have nothing to do with speed, in our AI agent vendor evaluation checklist.

Latency and cost are the same conversation

Worth noting: the same chain that produces a bad p99 usually also produces a bad bill. Extra steps mean extra tokens, extra retries mean paying twice for the same work, and extra model calls mean paying for coordination overhead that adds no value to the output. If you're already auditing latency, audit spend at the same time. Our guide on LLM cost optimization walks through the same trimming exercise from the cost side, and in practice the fixes overlap: fewer, better-scoped calls beat more, smaller ones on both dimensions.

The practical takeaway

Don't let "it feels fast in the demo" or "average response time: 800ms" stand in for real measurement. Before you ship an agent, or before you sign off on one someone else built for you, get the p95 and p99 numbers under load, broken down by step. If nobody can produce those numbers, that's your answer about how much testing has actually happened.

If you're building an AI agent and want a second set of eyes on the architecture before latency becomes a production problem, let's talk.

Originally published on the Pykero blog.

BYOK vs Managed LLM Keys: How to Price AI Features

Pykero — Mon, 03 Aug 2026 09:02:02 +0000

Bundle LLM costs into your subscription price by default (managed keys), and offer bring-your-own-key (BYOK) as an opt-in for enterprise accounts that already have their own model contracts or compliance requirements. The wrong default either kills your margin at scale or adds enough setup friction to tank your self-serve conversion.

This decision shows up the moment you add any LLM-powered feature to a SaaS product: a chat assistant, an AI search box, an agent that drafts emails. Someone has to pay for tokens, and someone has to hold the API key. Get it wrong and you either eat unpredictable inference costs as your product goes viral, or you force every trial user through an OpenAI signup before they see any value.

The two models, plainly

Managed keys: your backend holds one (or a few) API keys, calls the LLM on behalf of every tenant, and you fold the token cost into your subscription price. The customer never sees a key or a provider name.

BYOK: the customer generates their own API key from Anthropic, OpenAI, or Azure, pastes it into your settings page, and your app uses their key and their billing relationship. You charge for the software and workflow, not the inference.

Most products default to managed because it's the better onboarding experience. Nobody wants to leave your signup flow to go set up a separate account somewhere else before they can try your product. But managed pricing means your gross margin moves with token prices and usage patterns you don't fully control, which is exactly the failure mode covered in multi-llm-provider-failover: a single vendor price change or outage becomes your problem, not a line item you can pass through.

Why this is a pricing decision, not just an architecture decision

Teams often treat BYOK as a technical toggle: add a settings field, store the key, done. It's actually a pricing and packaging decision that determines your unit economics.

With managed keys, every heavy user erodes your margin. A customer who runs 50,000 completions a month on your $99/month plan is subsidized by the customers who run 500. That's fine at small scale, but it breaks down once usage-heavy customers cluster on your cheapest tier, which is a version of the same trap covered in saas-pricing-models: flat pricing works until usage variance gets wide enough to punish your best customers or your margin.

With BYOK, your margin is protected because the customer pays the LLM provider directly. But you've now pushed setup friction onto every user, and you've taken on a support burden: expired keys, hit rate limits, wrong model access tiers, and confused customers debugging why "your product" returned an error that was actually their provider's account issue.

The cost math worth doing before you decide

Before picking a default, instrument actual token usage per feature, not a guess from one demo session. Log input and output tokens per request in staging, run your real prompts and expected output lengths, and multiply by projected monthly active users at each pricing tier. The gap between "what we assumed" and "what real usage looks like" is usually 3-5x, especially for agentic features that make multiple tool calls per user action.

This is also where call architecture matters as much as pricing model. In our own outreach tool, we originally split lead qualification into separate calls: one to extract company facts from a scraped page, another to draft the email. Collapsing that into a single call that extracts the facts and drafts the email in one pass cut token spend meaningfully and, if anything, improved output quality, because the model had full context in one shot instead of losing it across a handoff. If you're running managed keys, this kind of consolidation is the difference between a feature that's profitable at $49/month and one that isn't. It's worth doing the audit described in llm-cost-optimization before you finalize either pricing model, because a chain-heavy feature makes managed pricing much harder to sustain and makes BYOK friction much harder to justify.

When BYOK is the right default

Run the same 50,000-completions-a-month customer from the margin example above through a BYOK lens: on managed keys they're the one subsidized by your 500-completion customers; on BYOK, their token bill goes straight to their own OpenAI or Anthropic account and stops being your problem. That's the pattern to watch for, not a checklist to apply blind. BYOK tends to be the right call when:

Your customers are enterprises that already have negotiated LLM contracts (volume discounts, data residency terms, or a specific vendor mandated by their own compliance team)
Usage is inherently high-volume and spiky, like a coding assistant or bulk document processor, where per-seat pricing can't absorb the variance the way it broke down in the $99/month tier above
Your buyer's security team requires that no third party (including you) can access their prompts or outputs, which is common in healthcare and legal verticals
You're selling a developer tool where the audience already manages API keys as part of their normal workflow

When managed keys are the right default

Managed keys make the most sense for exactly the profile our outreach-tool anecdote describes: a feature where you control the call pattern closely enough that consolidating two calls into one is your lever for protecting margin, rather than pushing that cost onto the customer. That only works if you're the one holding the key and the bill. Managed tends to be the right call when:

You're selling to non-technical buyers who will never generate an API key on their own
Usage per customer is predictable enough to price into flat or tiered plans, the way a $49-or-$99-a-month tier assumes
Time-to-value matters more than margin protection, which is almost always true pre-product-market-fit
The AI feature is one part of a broader product, not the whole product, so token cost is a small fraction of your overall COGS

The hybrid approach most mature products land on

Start managed for self-serve and lower tiers, where onboarding friction is the bigger risk than margin. Add BYOK as an enterprise-tier option once you have customers asking for it, usually the same customers who need SSO, audit logs, and a signed DPA anyway. This mirrors how most multi-tenant-saas-architecture decisions get made generally: build the simple default first, add the configurable path only when a real customer segment needs it, and keep the two paths behind the same feature interface so switching a tenant from managed to BYOK doesn't require a rewrite.

Whichever default you pick, don't hardcode it. Store the key source (managed vs tenant-provided) as a per-tenant config value from day one, even if every tenant starts on managed. Retrofitting BYOK into a codebase that assumed one global API key is a bigger project than it sounds, and it's the kind of technical debt described in technical-debt-management that's cheap to avoid up front and expensive to unwind later.

If you're scoping an AI feature and aren't sure which model fits your customer base and margin targets, let's talk.

Originally published on the Pykero blog.

Do You Need Multi-LLM Failover, or Just One Good Provider?

Pykero — Sat, 01 Aug 2026 09:01:34 +0000

Most products don't need multi-LLM failover on day one. Build on one provider, wrap the model call behind a thin interface so swapping providers later is a config change, and only add a second provider once an outage, rate limit, or pricing change has actually cost you something measurable.

That's the boring answer, and it's the right one for maybe 90% of the founders who ask us about it. But the question keeps coming up, especially now that tools promising "one API key for every model" (GPT, Claude, Gemini, Grok, DeepSeek, Kimi) are getting attention. The pitch is appealing: never get locked into one vendor, route around outages, arbitrage pricing. The reality is more nuanced, and getting it wrong in either direction costs you.

Why the question feels urgent right now

Every provider has had a bad day. OpenAI's status page and Anthropic's status page both show periodic degraded-performance incidents, usually measured in minutes to a couple of hours. If your product sends a customer-facing request straight to a single model with no fallback, a 45-minute outage becomes 45 minutes of your product being down, even though the rest of your stack is fine.

Rate limits are the more common failure mode. OpenAI's rate limit docs and Anthropic's rate limit docs both scale limits with usage tier, which means the moment you get a spike in traffic (a product launch, a press mention, a viral feature) is exactly when you're most likely to hit a ceiling you've never hit before.

Model deprecation is the quiet one. Providers retire model versions on a schedule, and if your prompts were tuned against a specific model's quirks, the replacement can shift output quality in ways your users notice before you do.

The case against building it now

We see teams reach for a full multi-provider abstraction before they've shipped a single feature that uses it in anger. That's premature infrastructure, and it has a real cost: every layer of indirection between your business logic and the model call is a place a bug can hide, and it's code you have to maintain even if you never actually fail over.

We learned a version of this lesson building our own outreach tooling. We scrape each prospect's site and draft a tailored email with a single LLM call that both extracts the relevant facts and writes the draft, no multi-step chain, no orchestration layer. We tried the more "robust" multi-call pipeline first, on the assumption it would be more reliable. It wasn't: it was slower, more expensive per email, and failed in more places. The single call was cheaper, faster, and easier to debug. The lesson generalizes: don't add architectural complexity to defend against a failure mode you haven't actually experienced yet. The same logic applies to provider redundancy. If you've never had an outage cost you a customer, you're probably optimizing for a problem you don't have. This is the same instinct behind why simple, single-call design usually beats agent chains for well-scoped tasks.

When it does pay off

Multi-provider support earns its complexity in a few specific situations:

You're past a few thousand requests a day and rate limits are a recurring line item in your incident log, not a hypothetical.
Your product is customer-facing and synchronous (a chat widget, a voice agent), where even a few minutes of provider downtime is visible to end users in real time, versus a background job that can just retry in an hour.
You have a specific cost-sensitive workload where routing cheaper requests (classification, extraction) to a smaller model and reserving a frontier model for complex reasoning meaningfully changes your unit economics. This overlaps heavily with general LLM cost optimization work you should be doing anyway.
You're under a contractual or compliance requirement to avoid single-vendor dependency, which shows up more often in healthcare and government procurement than most founders expect.

If none of those apply, you're building insurance against a risk you haven't priced.

How to do it without over-building

If you've decided you actually need it, don't hand-roll a router from scratch. Projects like OpenRouter and LiteLLM already solve the unified-API problem: one interface, multiple backends, consistent error handling. The engineering work you should focus on instead is:

Isolate the model call. One function or module that owns "send this prompt, get this response." Everything else in your codebase calls that, never the provider SDK directly.
Normalize your prompts, not just your API calls. Different models respond differently to the same instructions. A router that swaps providers but sends the exact same prompt will quietly degrade output quality on the fallback model unless you've tested it.
Add evals before you add failover. You can't safely fail over to a second model if you don't have a way to measure whether its output is good enough. This is the same discipline we recommend when negotiating evals into AI vendor contracts: know what "good" looks like before you need to prove it under pressure.
Decide your fallback policy explicitly. Retry the same provider, switch providers, or degrade gracefully (cached response, simpler feature, human handoff)? Each has a different cost and a different failure mode, and "just fall back to GPT" is not a policy.

The actual decision framework

Ask three questions before you write a line of routing code: How many times in the last quarter has a provider outage or rate limit actually cost you a customer or a broken feature? What would it cost, in engineering time, to build and maintain a second integration path? And is the risk you're insuring against getting worse or better as your provider's infrastructure matures? If the answer to the first question is "never, but I'm worried," ship on one provider and revisit this in three months. If it's "twice last month," you already have your business case.

Vendor risk is real, but so is the risk of shipping a more complex system than your usage justifies. If you're weighing this trade-off for a specific product, let's talk about what your actual failure modes look like before you build around hypothetical ones.

Originally published on the Pykero blog.

How to Evaluate an AI Agent Vendor Before You Sign

Pykero — Fri, 31 Jul 2026 09:02:04 +0000

Score AI agent vendors on how they handle failure, not on their demo. A polished demo tells you the agent can handle the five inputs the sales team rehearsed. It tells you nothing about what happens on input six, when a customer types something unexpected, uploads a bad file, or asks the agent to do something it was never scoped for. The vendors worth hiring can answer that question in detail. The ones who can't will hand you a maintenance bill later.

Here's the checklist we'd use if we were on the buying side.

Architecture and design

1. Why did you choose an agent instead of a simpler workflow?
Agentic loops with multiple LLM calls and tool-calling chains are slower, more expensive, and harder to debug than a single well-prompted call. We learned this the direct way building our own outreach tool: our first pass chained four steps (plan, extract, draft, revise), and it cost more per email than the single-call version we shipped instead, for worse copy. A vendor should be able to give you that same kind of before/after reasoning for their own architecture, not default to more agents because "agent" sells better than "script." Read up on the tradeoff in AI agents vs. workflows before the first call, so you can push back if the answer is vague.

2. Can you show me the simplest version of this that would work?
In our own outreach tooling, we scrape each prospect's site with a self-hosted Firecrawl instance and a local model, then draft the email in a single call that extracts the relevant facts and writes the pitch at the same time. We tried a multi-step chain first, plan then extract then draft then revise, and it cost more per email and produced worse copy, because each hop introduced its own chance to drift off the source material. If a vendor's default answer to every problem is "more agents, more steps," that's a signal they're optimizing for billable complexity, not for your outcome. Compare notes with single-call vs. agent chains.

3. What tools and data does the agent have access to, and why?
Every tool the agent can call is a potential blast radius. A support agent that can issue refunds needs tighter guardrails than one that can only draft a reply for human approval. Ask the vendor to walk through the permission model, not just the feature list.

Quality and evaluation

4. How do you measure whether the agent is actually working?
"We test it before launch" is not an answer. You want a repeatable eval set: a fixed collection of real (or realistic) inputs the vendor runs the agent against, with pass/fail criteria you both agree on. Without this, you're trusting vibes, and vibes don't hold up when the model provider ships a silent update. This should be a contract line item, not a verbal promise, per evals in AI vendor contracts.

5. What happens when the agent is confident and wrong?
Every LLM-based system produces confident, fluent, wrong answers sometimes. Ask specifically how the vendor's design catches this, whether that's a confidence threshold that routes to a human, a second model checking the first, or hard-coded guardrails on high-stakes actions. If the answer is "the model is pretty accurate," keep asking.

6. Can I see it fail on purpose?
Ask the vendor to run three inputs you choose, live, on the spot, including at least one that should be rejected or escalated rather than answered. This takes ten minutes and tells you more than a week of reading proposals.

Cost and ownership

7. What's the unit cost per interaction, and how does it scale?
Token costs, tool calls, and any third-party API fees (speech-to-text, search, embeddings) should roll up into a per-conversation or per-task number, broken out by step, not just a blended average. That breakdown is exactly what surfaced the problem in our own outreach tool: once we costed the chain step by step, the extra plan/extract/revise hops were visibly adding cost per email without adding quality, which is what pushed us back to a single call. A vendor who can't hand you that same kind of step-by-step number hasn't actually load-tested their own system.

8. Who owns the code, prompts, and evals when the engagement ends?
This should be explicit in the contract. If the vendor keeps the prompts, the eval set, or the fine-tuned weights, you're locked into paying them for every future change, no matter how the relationship is going.

9. What does month two look like, after launch?
Models get updated, edge cases surface, and prompts drift out of sync with your product. Ask what ongoing maintenance costs, who's on call for it, and what's included versus billed separately. We wrote a full breakdown in what AI agent maintenance actually costs if you want the specifics before you negotiate.

10. Fixed price or time and materials, and why for this project?
Neither is universally safer. Fixed price works when the task is well-defined and the failure modes are known. Time and materials fits better when you're still discovering what "working" means. A vendor who pushes one model regardless of the project shape is optimizing for their own risk, not yours. More on the tradeoff in fixed-price vs. time and materials.

Security and data handling

11. Where does customer data go, and who else can see it?
If the agent calls a third-party model API, ask exactly what's sent, whether it's used for training, and where it's logged. This matters even more for anything touching health records, financial data, or PII. Run through the fuller list in the AI agent security checklist.

12. What's the rollback plan if this goes wrong in production?
A kill switch, a way to route everything to a human, a versioned prompt history you can revert. If the vendor hasn't thought about this, you'll be the one improvising it during an incident.

Reading the proposal itself

Watch for the same red flags that show up in any software project: vague scope, no mention of who tests what, timelines that don't account for iteration. AI agent projects amplify these risks because the output is probabilistic, so ambiguity in the contract becomes ambiguity in production behavior.

Three proposals is usually enough to compare architecture choices and pricing philosophy without turning this into a part-time procurement job. Ask the same twelve questions of each one, and pay closer attention to the vendor who says "it depends" and explains why than the one with the slickest deck.

If you're evaluating vendors for an AI agent build and want a second opinion on a proposal, let's talk.

Originally published on the Pykero blog.

What AI Agent Maintenance Actually Costs After Launch

Pykero — Wed, 29 Jul 2026 09:01:59 +0000

Building an AI agent typically costs less than keeping it working. Budget 15 to 25 percent of your original build cost per year for maintenance, and expect the biggest expense to be things that fail silently rather than things that crash loudly.

Most founders scope an AI agent project like a normal feature: fixed build cost, ship date, done. Then three months post-launch, something quietly breaks, nobody notices for two weeks, and the "finished" project needs another sprint. That's not a scoping failure, it's a category difference. Traditional software fails loudly: a null pointer, a 500 error, a failed test. AI agents fail quietly, because the model doesn't know it's wrong. It just produces a confident, plausible, incorrect answer and moves on.

Why AI agents cost more to maintain than typical software

Silent failures are the real threat

We run our own outreach engine that scrapes each prospect's site with a self-hosted Firecrawl instance plus a local LLM, extracting facts and drafting one tailored email per company in a single call. The failure mode we hit most often wasn't the LLM hallucinating, it was upstream: a prospect's site redesigned its homepage, the scraper still returned a 200, and the agent kept writing emails referencing a service the company had quietly discontinued months earlier. Nothing crashed. Nothing alerted. The output just got worse.

That's the pattern to plan for. Your AI agent's inputs (scraped pages, third-party APIs, CRM records, PDFs) change without telling you, and the agent has no built-in way to know the ground shifted under it.

Model and API churn

The model provider behind your agent will deprecate versions, change rate limits, or adjust behavior on updates you didn't ask for. A prompt tuned carefully against one model's tendencies can behave differently against its replacement, even when the interface looks identical. We saw this on the local LLM behind our own outreach engine: swapping in a newer point release of the same model family produced tighter, less repetitive summaries of scraped page content, but it also started dropping the specific service names our personalization step depends on. That meant re-running our extraction prompts against a batch of past scrapes and comparing outputs line by line before we'd trust the new version in production. This isn't hypothetical, it's a scheduled event: every foundation model has a deprecation timeline, and "just swap the model string" undersells the retesting that responsible teams actually do first.

Data and workflow drift

Business rules change. Pricing tiers change. The support agent that was trained on your refund policy needs an update the day that policy changes, not whenever someone remembers to tell engineering. Static software degrades slowly through neglect; agent logic degrades the moment the business changes and nobody re-syncs it.

What a realistic maintenance budget actually covers

Monitoring and alerting: something needs to flag when output quality drops, not just when the process crashes. This is usually the most underbudgeted line item.
Data source health checks: scheduled validation that scrapers, APIs, and integrations are still returning what you expect, structurally, not just returning something.
Model version management: testing against new model releases before you're forced onto them, and keeping a rollback path.
Prompt and logic updates: a standing process for feeding business changes (pricing, policy, new products) back into the agent's instructions.
Fallback and escalation logic: a defined path for when confidence is low, so the agent hands off to a human instead of guessing.
On-call ownership: someone accountable when the agent misbehaves at 2am, even if the fix waits until morning.

None of this is exotic. It's the same discipline you'd apply to any production system with external dependencies, applied to a system whose failure mode is "confidently wrong" instead of "down."

A rule of thumb for budgeting

For a mid-complexity agent (a WhatsApp sales assistant, an internal support bot, a data-enrichment pipeline), plan on 15 to 25 percent of the initial build cost per year in ongoing maintenance, split roughly between monitoring/infrastructure and prompt/logic upkeep. Our own outreach engine sits comfortably inside that range: the recurring cost is mostly the hours spent re-checking scraper output after a site redesign and re-validating extraction prompts after a model swap, not new infrastructure spend. Agents touching money, health data, or compliance-sensitive decisions should be budgeted higher, because the cost of a silent failure is not "an annoying email," it's a real liability. If you're negotiating with a vendor, this is exactly the kind of commitment worth pinning down in the contract rather than assuming; see our breakdown of what to demand in evals as part of an AI vendor contract.

Build vs. buy changes the shape of the cost, not whether it exists

Choosing a no-code agent platform over a custom build doesn't remove maintenance, it relocates it. The vendor handles infrastructure and model upgrades, but you still own your prompts, your data connections, and your business logic, and now you're also exposed to the vendor's pricing changes and roadmap decisions. We've laid out that trade-off in more detail in custom AI agent vs. chatbot platform. Either way, run the numbers the same way you'd evaluate any recurring infrastructure cost, alongside model spend; our guide on LLM cost optimization covers the token-cost side of that math, and it pairs directly with the maintenance side covered here.

If you're scoping an AI agent and the proposal in front of you doesn't mention monitoring, model updates, or a maintenance retainer, ask why. It's either been priced in quietly (rare) or it hasn't been priced in at all (common, and expensive later).

Weighing whether an agent needs this level of upkeep at all, or whether a simpler workflow would do? Worth reading alongside AI agents vs. workflows before you commit to either the build or the budget. If you want a second opinion on what a realistic maintenance plan should look like for your use case, let's talk.

Originally published on the Pykero blog.

Permission-Aware RAG: What It Really Costs to Get Right

Pykero — Mon, 27 Jul 2026 09:02:03 +0000

Permission-aware RAG means your retrieval step filters by who's asking, not just what they asked — and that filter has to live inside the vector query itself, not as a second pass the LLM does after the fact. Most teams find this out the hard way: their RAG demo works great on a shared knowledge base, then someone asks it to enforce per-user document access and the whole retrieval layer needs to change.

Why "just add a permission check" doesn't work

The instinct is to keep your existing retrieval pipeline and bolt a check onto the end: retrieve the top-k chunks, then ask an LLM (or a rules engine) "should this user see this?" before passing context to the answer step. This fails in two ways:

It's non-deterministic. An LLM asked to police access control will occasionally get it wrong, and "occasionally wrong" is not an acceptable access-control policy, especially with health records, legal documents, or HR files in the mix.
The leak already happened. By the time you're filtering post-retrieval, the restricted content has already been pulled into a context window and sent somewhere. Logging, caching, or a stray debug trace can leak it before your filter ever runs.

The fix is to push permissions into the query, the same way you'd push a WHERE user_id = ? clause into a SQL query instead of fetching everything and filtering in application code. We've seen the same principle play out in our own tooling: in an internal outreach engine we built, a single call that filters and extracts in one pass consistently beat a multi-step chain that fetched broadly and reasoned about relevance afterward — cheaper, faster, and fewer places for something to go wrong. Permission-aware RAG is the security-critical version of that same lesson: filter at the source, don't filter after the fact.

What actually changes in your architecture

If you're running Postgres with pgvector or a managed vector store, permission-aware retrieval typically requires:

Permission metadata on every chunk. Not just a document_id — you need to know which roles, teams, or user IDs can see each chunk at index time. This usually means re-chunking and re-embedding your existing corpus, not a metadata patch.
Row-level security or an equivalent filter in the query path. Postgres has native support for this via row-level security policies, and Supabase builds directly on top of it for multi-tenant RLS. The similarity search and the access filter run as one query, so a user's vector search literally cannot return rows they're not permitted to see.
A permissions model that changes over time. Documents get shared, roles get revoked, teams get restructured. If your embeddings and your permissions live in different systems that sync on a schedule, there's a window where a fired employee's session can still retrieve documents they lost access to an hour ago. Real-time permission checks are non-negotiable once this goes near anything regulated.
Re-indexing strategy, not just retrieval logic. This is the part that blows up estimates. Adding ACLs to a RAG system that already has 50,000 indexed documents means you're not writing a new query — you're re-processing the entire corpus with permission tags attached, then validating that the tags are correct before you trust the system with sensitive content.

Where the cost actually goes

Teams budget for the model and the retrieval logic. On that same 50,000-document corpus, the real cost shows up in:

Data audit. Someone has to determine who's actually supposed to see what, and at 50,000 documents that's not a spreadsheet filled out in an afternoon — it's confirming access department by department, often against tribal knowledge that was never written down, before a single permission tag gets attached.
Re-embedding at scale. Re-chunking and re-embedding 50,000 documents isn't a job you queue up overnight and check on tomorrow — if the permission model changes what counts as one retrievable unit (splitting a shared folder into per-team sections, for instance), the chunk boundaries themselves change, so this is a re-processing pass over the whole corpus, not a metadata update layered on top of it.
Testing for negative cases. Testing that the system retrieves correctly is easy. Testing that it never retrieves something a user shouldn't see, across every role combination, is the part that takes real QA time — this is the same category of work covered in an AI agent security checklist, and it deserves the same rigor you'd apply to authentication code, not "we tried a few prompts and it seemed fine."
Audit logging. In healthcare or government contexts, you often need to prove, after the fact, exactly what a user retrieved and why — which means logging the filtered query, not just the final answer. This overlaps directly with what we've written about court-ready architecture for healthcare AI: the log has to be defensible, not just present.

When it's worth building vs. buying

If your access model is simple — one tenant, one shared knowledge base, everyone sees everything — don't build permission-aware retrieval at all. It's unnecessary complexity for a problem you don't have. Start with the basics in RAG explained for founders and move on.

Build it yourself when:

Your permission model is core to the product (multi-tenant SaaS, healthcare records, internal knowledge bases with confidential HR or legal content).
You need real-time revocation — access changes have to take effect immediately, not on the next sync.
You're in a regulated space where "we filtered it after retrieval" won't survive an audit.

Consider a managed layer (many enterprise search and RAG platforms now ship ACL-aware retrieval out of the box) when your permission model maps cleanly onto standard role/group structures and you don't need to customize the filtering logic itself. Whichever you choose, put the same scrutiny into the vendor's retrieval guarantees that you'd put into any AI vendor contract — see evals in AI vendor contracts for what to actually ask for.

The one-sentence version

Don't ask the LLM to enforce access control after retrieval — enforce it in the query that does the retrieving, and budget for re-indexing your corpus, not just writing new retrieval code.

If you're scoping a RAG system that needs real permission boundaries — healthcare records, multi-tenant SaaS, internal docs with confidential sections — let's talk.

Originally published on the Pykero blog.

What a HIPAA-Compliant AI Voice Agent Actually Costs

Pykero — Sat, 25 Jul 2026 09:01:25 +0000

A HIPAA-compliant AI voice agent for healthcare typically costs $40,000-$150,000 to build, depending on call complexity and EHR integration, plus $2,000-$15,000/month to operate. The build cost isn't dominated by the speech model — it's dominated by the compliance and data-retention layer wrapped around it.

Most cost estimates for "AI voice agents" quietly assume a sales or support use case, where a wrong transcription costs you an annoyed customer. In healthcare, a wrong transcription in a medication name or a dropped consent statement is a liability. That difference reshapes the budget.

Where the money actually goes

1. Speech recognition (10-20% of build cost)

This is the smallest line item, despite being the part founders worry about most. You have three options:

Managed API with a BAA (e.g., enterprise-tier Deepgram, Azure Speech, Google Healthcare API) — fastest to ship, but you're paying per-minute and locked into the vendor's accuracy on medical terminology.
Fine-tuned open-weight model — better accuracy on clinical vocabulary and accents, but adds MLOps overhead.
Self-hosted model — highest control over data residency, needed if your contracts or state law prohibit sending PHI to a third party.

If your patient population speaks Gulf Arabic or another dialect underserved by mainstream ASR, budget separately for this — see our breakdown on Arabic speech recognition costs for how accent and dialect coverage move accuracy and price independently of the base model choice.

2. Compliance infrastructure (30-40% of build cost)

This is where healthcare voice AI diverges hardest from a generic voice bot:

Business Associate Agreements with every vendor in the call path (ASR, LLM, telephony, storage)
Encryption at rest and in transit, with key management you can audit
Role-based access control on transcripts and recordings
Immutable audit logs of who accessed what patient data and when

The U.S. Department of Health and Human Services publishes the actual HIPAA Security Rule requirements — worth reading directly rather than trusting a vendor's compliance checklist, since "HIPAA-compliant" is not a certification anyone issues, it's a set of administrative, physical, and technical safeguards you're responsible for implementing.

3. Data retention and lifecycle policy (15-20%)

Retention isn't a settings toggle you flip once. You need:

Configurable retention windows per data type (call audio vs. transcript vs. structured extraction)
Automated deletion or archival on schedule
A defensible chain of custody if a recording ever needs to be produced for a malpractice claim or audit

If you're also building the surrounding patient record system, this overlaps with what we've written about court-ready architecture for healthcare AI — the same evidentiary standards that apply to clinical documentation apply to voice interaction logs.

4. Integration with EHR/EMR (20-30%)

The voice agent is worthless in isolation. Most of the real engineering effort goes into:

Pulling patient context before the call (so the agent isn't starting cold)
Writing structured summaries back into the EHR after the call
Handling the failure mode where the integration is down mid-call

5. Ongoing operations ($2K-$15K/month)

This scales with call volume and includes ASR/LLM usage, monitoring, and a human-in-the-loop review process for a sample of calls — which most healthcare compliance teams require regardless of how good your model claims to be.

The extraction pattern that actually saves money

A place teams overspend is call processing after the fact — running the transcript through multiple chained LLM calls (summarize, then extract entities, then classify, then draft the EHR note). In our own tooling, we found that a single well-structured call — extract facts and produce the structured output in one pass — consistently beat multi-step chains on both cost and accuracy, because each additional hop introduces a new place for the model to drop or hallucinate detail. The same principle applies directly to post-call processing of a patient conversation: one careful extraction call beats four cheap ones. We go deeper on why in single-call vs. agent chains.

Build vs. buy

Off-the-shelf healthcare voice AI platforms exist and can get you to a pilot fast, but almost none offer a BAA, configurable retention, or audit logging at the tier a small clinic can afford — those features show up once you're paying enterprise pricing. If you're evaluating vendors instead of building, put their claims through the same rigor you'd apply to an internal build; our guide to evals in AI vendor contracts has a checklist for pinning vendors to accuracy and compliance commitments in writing, not just in a sales deck.

For most clinics and health-tech startups, the deciding factor isn't cost — it's whether your patient population, call volume, and compliance obligations justify a custom build now, or whether a pilot on a managed platform buys you time to validate demand before you invest in the infrastructure above.

What to budget for a first version

Pilot (single use case, e.g., appointment reminders + rescheduling): $40K-$60K build, $2K-$4K/month
Mid-complexity (intake, triage questions, EHR write-back): $70K-$110K build, $5K-$8K/month
Full clinical documentation assistant: $120K-$150K+ build, $8K-$15K/month

These ranges assume you're not also building the EHR — if you are, add that scope separately, and check our voice agent latency checklist once you're in build, since a compliant-but-slow agent still fails the patient experience test.

If you're scoping a healthcare voice AI project and want a realistic estimate for your specific call volume and compliance requirements, let's talk.

Originally published on the Pykero blog.