DEV Community: lamingsrb

LLM Evals in 2026: A Practitioner's Field Guide

lamingsrb — Thu, 09 Jul 2026 06:55:56 +0000

LLM Evals in 2026: A Practitioner's Field Guide

Last week I watched a team ship a support agent that passed 94% of their eval suite and then hallucinated a refund policy on day two of production. Their eval was measuring the wrong thing, well. This guide is what I wish someone had handed me the first time I built an LLM eval harness that actually caught regressions before customers did.

What "LLM evals" actually means in 2026

An LLM eval is a repeatable measurement of a model or agent's output quality on a fixed dataset, using deterministic checks, model-graded judges, or human review, run in CI or on a schedule so you can detect regressions before shipping. In 2026 the term covers four distinct things people conflate: offline benchmarks (fixed dataset, run pre-merge), online evals (production traces sampled and scored), guardrails (runtime checks that block bad outputs), and judge calibration (measuring whether your LLM-as-judge agrees with humans). If your team says "we have evals" and cannot tell you which of the four they mean, they do not have evals. They have a vibes-based test set.

The mental model I use with clients: evals are unit tests for probabilistic systems. They will not be green/red. They will be distributions. Your job is to make the distribution shift visible.

The eval framework landscape, honestly compared

I have used all four of these on real projects in the last 18 months. Here is how I actually pick between them.

Framework	Best for	Self-host	LLM-judge built in	Trace-linked	Where it hurts
Braintrust	Teams that want a polished UI and fast iteration on prompts	No (SaaS)	Yes	Yes	Cost at scale, vendor lock on eval definitions
LangSmith	Teams already deep in LangChain/LangGraph	Limited (enterprise)	Yes	Yes, tightly	Framework gravity: hard to leave once your traces live there
Promptfoo	Config-as-code, CI-first, model comparison	Yes	Yes	Weak (bring your own tracing)	UI is basic, no real prod monitoring
Ragas	RAG-specific metrics (faithfulness, context precision)	Yes	Yes	No	Only useful if your problem is retrieval, judge metrics are noisy without calibration

My default stack for a new client engagement: Promptfoo in CI + a lightweight custom dashboard on Postgres for online evals. I add Ragas only when the RAG faithfulness question is central. I recommend Braintrust when a non-engineering PM needs to run experiments without opening a PR. LangSmith is fine if you are already on LangChain and not planning to leave. I do not use it as a starting point in 2026 because the migration cost later is real.

The single biggest mistake I see: teams pick the framework before they have written 20 real eval cases by hand. The framework is 10% of the problem. The dataset is 90%.

The 12 failure modes I check for on every LLM system

This is the table I keep in my head, and now on paper. Each row is a failure I have shipped or watched someone else ship in production. Each has a specific check.

#	Failure mode	What it looks like	How I catch it
1	Prompt drift	Small prompt edit tanks 3 unrelated cases	Snapshot eval on every prompt PR, diff scores
2	Hallucinated citations	Agent invents URL, doc ID, or policy	Regex check against known-good ID list + faithfulness judge
3	Tool-call malformation	JSON args missing required field	Schema validation on every tool call, fail loud
4	Silent tool failure	Tool returns empty, agent proceeds as if it worked	Assert non-empty tool return, log to trace
5	Context poisoning	Bad retrieval chunk flips the answer	Retrieval eval separate from generation eval
6	Judge bias	LLM judge always prefers longer answers	Calibrate judge against 50 human labels quarterly
7	Refusal creep	Model version upgrade adds unwanted refusals	"Should answer" set: track refusal rate over time
8	Loop / infinite planning	Agent replans same step forever	Max-steps guardrail + step-count metric
9	Cost blowup	One query burns 400k tokens	Per-trace token budget alert
10	Latency regression	p95 doubles after model swap	Latency in the eval report, not just quality
11	PII leakage	Model echoes user data into logs or downstream calls	Regex + entity detector on outputs
12	Eval set rot	Test cases become trivial after months of prompt tuning	Rotate 10% of eval set per quarter with real prod failures

If you build a scorecard around these 12, you will catch roughly 80% of what actually breaks LLM systems in production. The remaining 20% is domain-specific, and that is where the interesting engineering happens.

The workflow I use to build an eval suite from scratch

Every engagement follows the same rough sequence. It takes about a week for a scoped agent, three weeks for something ambitious.

Step 1: Collect 50 real inputs, not synthetic ones. If the system is not live yet, sit with the people who will use it and ask them to type the messages they would actually send. Synthetic eval inputs generated by GPT are the reason so many production systems fail on real traffic. The distribution is wrong.

Step 2: Hand-label expected behavior for each input. Not expected output, expected behavior. "Should refuse", "should call search_docs with query about pricing", "should ask a clarifying question". Behavior labels survive prompt changes. Exact-output labels do not.

Step 3: Write deterministic checks first. Anything you can catch with a regex, a JSON schema, or a substring match, catch that way. Deterministic checks are free, fast, and never disagree with themselves. I usually get 30-40% of my checks deterministic.

Step 4: Add LLM-as-judge for the rest, then calibrate. Write your judge prompt. Run it on 50 cases. Have a human (usually me and the domain expert) label the same 50. Compute agreement. If your judge disagrees with humans more than 15% of the time, the judge prompt is broken, not the model. I wrote about this in more depth in my post on Hamel's method vs a lean in-house setup.

Step 5: Wire it into CI and a nightly cron. CI blocks PRs that regress the eval score by more than 2 points on any critical metric. The nightly job runs against a sample of yesterday's production traces so you catch drift the fixed set will not.

Step 6: Publish the numbers where the team sees them. A Slack post every morning with the score, a small table of what regressed, and a link to the diff. If the number lives in a dashboard nobody opens, you do not have evals. You have decoration.

Real numbers from a recent build

For a content generation agent I shipped this year, the eval investment looked like this. I am sharing the actual numbers because "evals are worth it" without numbers is a slogan.

Time to write initial 60-case eval suite: 9 hours
Time to calibrate the LLM judge to 91% human agreement: 4 hours
Model + judge cost per full eval run: $0.42
CI overhead added per PR: ~90 seconds
Regressions caught before merge in first 8 weeks: 11
Estimated cost of those 11 regressions reaching prod (rework + reputation): I do not know, but the two that involved wrong pricing claims would have been bad.

The eval suite paid for itself the first time it caught a prompt tweak that broke tool calling on 4 out of 60 cases. That took 90 seconds in CI. Debugging it in production would have taken hours and would have shipped wrong outputs to real users first.

Where LLM-as-judge quietly lies to you

This is the section I wish someone had written for me two years ago. LLM judges have systematic biases and if you do not measure them, your eval score is a lie you are telling yourself.

Position bias. In pairwise comparison ("is A or B better?"), most judges prefer whichever answer comes first. Fix: run every pair twice, swapped, and only count agreement.

Length bias. Judges prefer longer answers even when shorter is correct. Fix: include "concise" as an explicit criterion in the rubric, or normalize by length.

Self-preference. GPT-4 class judges prefer GPT-4 class outputs. Claude prefers Claude. If you use the same model family to generate and judge, you are grading your own homework. Fix: use a different family for the judge, or triangulate with two judges.

Rubric ambiguity. "Rate helpfulness 1-5" produces noise. "Score 1 if the answer cites the correct policy section by ID, 0 otherwise" produces signal. Push rubrics toward binary or ternary decisions with explicit criteria whenever you can.

I recalibrate judges every time I change the underlying model version. When Anthropic or OpenAI ships a new snapshot, judge behavior shifts. Skipping calibration and trusting the score is how teams end up with green dashboards and angry users.

What I would do if I were starting today

If you are a founder or head of engineering standing up an LLM system in 2026, this is the shortest path I know:

Do not pick a framework yet. Write 50 real eval cases in a spreadsheet, by hand.
Ship Promptfoo in CI on week one. It costs nothing, it works, and the config format is portable if you outgrow it.
Track the 12 failure modes above as a checklist, not a philosophy. Tick them off one by one.
Calibrate your judge quarterly against fresh human labels. Put it on the calendar.
Rotate 10% of the eval set every quarter with real production failures. Otherwise the set rots and your scores lie.
Do not spend more than 20% of your engineering budget on the eval framework itself. The dataset and the judge rubric are the leverage points. The tooling is a commodity.

The teams I see get this right treat evals the way good backend engineers treat observability: a first-class part of the system, built as the system is built, not bolted on after the first outage.

Closing

Evals in 2026 are not a checkbox. They are the difference between an LLM system you can operate with confidence and one you are constantly patching in response to angry users. If you are somewhere in that gap and want a second set of eyes on your eval strategy, or want help standing one up from scratch, I take a small number of these engagements each quarter: lazar-milicevic.com/#contact. More posts on production LLM work live on the blog if you want to keep reading.

Last updated: July 2026. Written by Lazar Milicevic, Senior Technical Engineer, 10+ years in AI automation, cloud architecture and B2B SaaS. Founder of BizFlowAI.

How to Build AI Agents From Scratch: A Blueprint

lamingsrb — Thu, 09 Jul 2026 06:55:53 +0000

How to Build AI Agents From Scratch: A Blueprint

Every framework I have used in production, I have eventually gutted. LangChain, CrewAI, even parts of LangGraph. Not because they are bad, but because when an agent fails at 3 a.m. and you are staring at a stack trace inside three layers of abstractions you did not write, you start thinking: this loop is maybe 200 lines. Why do I need 40 dependencies to run it?

This is the architecture I actually use to ship autonomous multi-agent systems, the ones that publish content, hit APIs, and run unattended for weeks. No framework religion. Just the parts that survive real traffic.

The agent loop is smaller than you think

An agent is a while loop around an LLM call that can invoke tools and decide when to stop. That is the whole thing. Everything else (memory, planning, reflection, multi-agent choreography) is a variation on that core.

Here is the minimal loop I start every new agent from. Python, no framework, uses the Anthropic SDK but the shape is identical for OpenAI:

def run_agent(task: str, tools: dict, max_steps: int = 20):
    messages = [{"role": "user", "content": task}]
    for step in range(max_steps):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            system=SYSTEM_PROMPT,
            tools=[t.schema for t in tools.values()],
            messages=messages,
            max_tokens=4096,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason == "end_turn":
            return extract_text(response)

        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                result = tools[block.name].run(**block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": str(result),
                    "is_error": isinstance(result, Exception),
                })
        messages.append({"role": "user", "content": tool_results})

    raise MaxStepsExceeded(f"Agent did not finish in {max_steps} steps")

That is roughly 30 lines and it will run a real agent. Everything I am about to describe is layered on top of this without changing the shape of the loop.

Two things I have learned to bake in from day one: a hard step cap, and structured tool_result blocks with an is_error flag. Without the step cap, one bad prompt burns $40 in a night. Without the error flag, the model quietly hallucinates that failed calls succeeded.

Tool design is where agents actually live or die

The model is only as smart as the tools you give it. In production, 80% of my debugging time is not prompt tuning, it is tool interface work. Ambiguous tool names, missing error messages, and untyped inputs cause more agent failures than any model weakness.

The rules I follow, without exception:

Every tool is a pure function with a typed schema. No hidden state. If the tool needs context, pass it in.
Tool names are verbs. search_docs, not docs_util. The model prompts itself with the name.
Errors are returned, not raised. Every tool returns {"ok": bool, "data": ..., "error": ...}. The model can then react.
Descriptions include when NOT to use the tool. This alone cut wrong-tool-selection rates by half in one of my content agents.
Idempotency where possible. Retries are cheap when the tool is safe to call twice.

Here is a tool description snippet I have shipped:

{
  "name": "publish_post",
  "description": "Publish a finalized post to the CMS. Only call this AFTER `validate_post` returns ok=true. Do NOT call this to preview or draft. This action is irreversible for 5 minutes.",
  "input_schema": { ... }
}

That last line, "irreversible for 5 minutes", changes model behavior meaningfully. Claude and GPT-4-class models actually respect it. They pause and often call validate_post first, even when I have not explicitly told them to in the system prompt.

Memory: three layers, not one blob

The most common mistake I see in home-rolled agents is treating memory as "just append to the context window". That works for 5 turns. It falls apart at 50, and it becomes unaffordable at 500.

I split memory into three layers, each with its own retention policy:

Layer	Contents	Retention	Storage
Working	Current task messages, tool calls	Current run	In-memory
Episodic	Past task summaries, outcomes	30-90 days	Postgres
Semantic	Facts, entities, learned patterns	Indefinite	pgvector + Postgres

In one of my recent posts I broke down how I cut a 3.26M token conversation down to 118K by moving from a naive append-everything design to this layered model. The key insight: the model does not need history, it needs the right slice of history.

For semantic memory I use pgvector with hybrid search (BM25 + vector, fused with RRF). Pure vector search misses exact-match queries ("what is the SKU for product X"), pure keyword misses semantic queries ("similar customer complaints"). Hybrid wins in every eval I have ever run.

For working memory, I aggressively summarize. Every 10 turns, a small model (Haiku or GPT-4o-mini) compresses the last 10 messages into 200 tokens of "here is what happened and what state we are in". The full log stays in the database for debugging. The context stays lean.

Orchestration: single agent, then supervisor, then graph

Do not start with a multi-agent system. I know it is the fashionable thing. Start with one agent and a lot of tools. Escalate only when you hit a real limit.

Here is the progression I actually use:

Level 1: Single agent, many tools. Works up to about 15-20 tools before the model starts confusing them. Cheap, easy to debug, testable.

Level 2: Supervisor plus workers. A planning agent decides which specialist to call. Specialists are themselves single agents with narrower tool sets. This is how BizFlowAI ContentStudio is built: a research agent, a writing agent, an SEO agent, a publishing agent, orchestrated by a supervisor that owns the workflow state.

Level 3: Graph / DAG. Only when you have deterministic branching, parallel work, and cycles that would be wasteful to let an LLM figure out on its own. This is where LangGraph earns its keep, and honestly, if I am at this level, I sometimes reach for it rather than rolling my own state machine.

The heuristic: if you can draw the workflow on a napkin and it does not need to change per input, the orchestration should be code, not an LLM decision. Every decision you outsource to the model is latency, cost, and a failure mode. Reserve LLM decisions for genuinely ambiguous branches.

Failure handling is 40% of production agents

The demo works. The production version has to survive rate limits, tool timeouts, malformed JSON, hallucinated tool names, infinite loops, context overflow, and the occasional model outage. In my experience, roughly 40% of the code in a production agent is failure handling, not "agent logic".

The layers I always put in:

Retry with backoff on transient errors. Model APIs 529, tool network calls, everything.
Structured output validation. If a tool expects JSON, validate it against a Pydantic schema and hand the model back the validation error on failure. The model will fix it in one turn 90% of the time.
Loop detection. I hash the last three tool calls plus arguments. If the same hash repeats, I inject a system message: "You just repeated the same action. Reconsider your approach." That alone kills most infinite loops.
Budget caps per run. Max steps, max tokens, max wall clock, max dollar. Any breach halts the run and logs it.
Observability from day one. Every step is written to a trace store with inputs, outputs, latency, cost. I use a lightweight Postgres schema for this, not a hosted tool, so I can query traces with SQL. When something breaks at scale, you need to be able to ask "show me every run where the SEO agent called publish_post before validate_post".

Here is the pattern I use for tool retries:

def run_tool_safely(tool, args, max_retries=2):
    for attempt in range(max_retries + 1):
        try:
            result = tool.run(**args)
            return {"ok": True, "data": result}
        except TransientError as e:
            if attempt == max_retries:
                return {"ok": False, "error": f"Transient error after {max_retries} retries: {e}"}
            time.sleep(2 ** attempt)
        except ValidationError as e:
            return {"ok": False, "error": f"Bad arguments: {e}. Please correct and retry."}
        except Exception as e:
            return {"ok": False, "error": f"Tool failed: {e}"}

Notice the model sees the error text. That is deliberate. The model is your last line of recovery, not a helpless victim of a stack trace.

Roll your own vs LangGraph vs CrewAI

I get this question weekly. Here is the honest answer.

Roll your own if:

Your agent has fewer than about 5 distinct roles
You need tight control over cost, latency, and error surface
You want to actually understand what your agent does when it misbehaves
You are shipping to production and will maintain it for years

Use LangGraph if:

You have genuine graph structure: parallel branches, cycles, checkpointing across a long-running workflow
You want first-class human-in-the-loop primitives without building them
Your team already knows the LangChain ecosystem

Use CrewAI if:

You are prototyping a multi-agent concept quickly and the exact wiring does not matter yet
Honestly, I have not shipped anything on CrewAI to production. It is fine for demos. I have always ended up rewriting.

The tradeoff nobody talks about: frameworks lock in a mental model. Once your team thinks in "Crews and Tasks" or "StateGraph nodes", it is very hard to think outside that model when you need to. Rolling your own means the mental model is: functions, state, loops. Universal.

For reference, the 200-line loop I showed at the top has been the base of every production agent I have shipped in the last two years, including the content pipeline that publishes across multiple sites unattended. I have never regretted starting small.

What I would do if starting a new agent today

Write the tools first. Type them, test them without an LLM. If your tools are broken, no model saves you.
Start with the 30-line loop. Get one end-to-end run working with a real task before you add anything.
Add the trace store. Postgres table, columns for run_id, step, role, content, tokens, cost, latency. Query with SQL.
Add budget caps. Steps, tokens, dollars. Non-negotiable.
Add loop detection and structured error returns. These two catch 80% of production weirdness.
Only then, if the task actually needs it, split into supervisor and workers.
Only then, if you have real graph structure, consider a framework.

The teams that ship fast are the ones that resist the urge to adopt heavy abstractions before they have felt the pain the abstraction solves. Every abstraction you adopt without earning it becomes a debugging tax later.

Closing

Production agents are not a framework problem. They are a systems problem: tool design, memory layering, orchestration discipline, and relentless failure handling. Get those right and a 200-line loop will outperform a 10,000-line framework wrapper you do not understand.

If you are building an agent that has to actually run in production and you want a second pair of eyes on the architecture, get in touch at lazar-milicevic.com/#contact. Or read more of what I have shipped and learned on the blog.

WebMCP Runs In Chrome. My 400 Daily Tool Calls Don't.

lamingsrb — Thu, 09 Jul 2026 06:11:50 +0000

WebMCP Runs In Chrome. My 400 Daily Tool Calls Don't.

Google I/O 2026 shipped WebMCP and half the AI Twitter timeline is calling it "the new MCP standard." It isn't. It's a browser-scoped protocol that solves a completely different problem than the MCP servers currently running on your VPS at 3 AM. Here's the boundary Google buried in the docs, and how to decide which side of it your agent belongs on.

What WebMCP actually is (and isn't)

WebMCP is a browser-scoped tool protocol. It exposes tools to an agent from inside a Chrome tab — the tools live in the page, auth is the user's active session, and the runtime is the browser itself. That's the entire surface area.

When Google says "agentic web," they mean an agent that operates inside a tab the user already has open, using the cookies and OAuth tokens already loaded. That's a legitimate and useful pattern:

Booking flows — agent fills a multi-step form on a site the user is signed into
Dashboards — agent pulls a chart, exports it, drops it into a doc
In-app copilots — SaaS product ships tools its own users' agent can call
Form fillers and page-scoped assistants

What WebMCP is not: a replacement for the stdio and HTTP MCP servers running headless on your machine or VPS. Different runtime, different auth model, different lifecycle. Calling it "the new MCP" is like calling a service worker "the new backend." Same protocol family, entirely different deployment target.

The split that actually matters

There's exactly one question you need to answer to pick correctly:

Is a human looking at a screen when the agent runs?

If yes → WebMCP is on the table.
If no → you need a real server-side MCP.

That's it. Everything else is retweet noise.

Dimension	WebMCP	stdio / HTTP MCP
Runtime	Chrome tab	Your process (local, VPS, container)
Auth	User's browser session	Your API keys / OAuth tokens
Trigger	User action in the page	cron, webhook, queue, schedule
Lifecycle	While tab is open	24/7 headless
Credentials scope	Whatever the user is logged into	Whatever you gave the process
Multi-account	Painful (one browser session)	Trivial (one process per tenant)
Runs at 6 AM while you sleep	No	Yes

I run three MCP servers in production. Gmail triage, Telegram messaging, invoicing. They sit on a WSL Ubuntu box, run headless as systemd services, and between them handle over 400 tool calls a day. Zero of those calls involve a browser. There's no user session. There's no tab. The agent wakes on cron or a webhook, pulls email, decides what matters, drafts replies, pushes a Telegram notification, generates an invoice, goes back to sleep.

WebMCP can't do any of that. Not because it's broken — because it's scoped to a runtime where a human is present.

What a real server-side MCP looks like

Here's the shape of one of my production servers, stripped to the bones. This is the Gmail triage server that runs on a 5-minute cron and processes the inbox before I'm awake:

# gmail_triage_server.py — stdio MCP server, runs as systemd service
from mcp.server import Server
from mcp.server.stdio import stdio_server
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build

app = Server("gmail-triage")

# Credentials loaded from disk once at startup.
# No browser, no user session — a service account / refresh token.
creds = Credentials.from_authorized_user_file("/etc/agents/gmail.json")
gmail = build("gmail", "v1", credentials=creds)

@app.tool()
async def list_unread(max_results: int = 50) -> list[dict]:
    resp = gmail.users().messages().list(
        userId="me", q="is:unread -category:promotions", maxResults=max_results
    ).execute()
    return resp.get("messages", [])

@app.tool()
async def get_message(message_id: str) -> dict:
    return gmail.users().messages().get(userId="me", id=message_id, format="full").execute()

@app.tool()
async def draft_reply(thread_id: str, body: str) -> dict:
    # ...creates a draft, does not send
    return {"status": "drafted", "thread_id": thread_id}

if __name__ == "__main__":
    stdio_server(app).run()

The agent that calls this doesn't care what browser you use. It doesn't even know a browser exists. The credentials live in a file with chmod 600, the process runs as a dedicated user, and the tool calls are logged to a local SQLite file so I can audit what happened overnight.

The systemd unit is boring on purpose:

# /etc/systemd/system/gmail-triage.service
[Unit]
Description=Gmail Triage MCP Server
After=network.target

[Service]
Type=simple
User=agents
ExecStart=/usr/bin/python3 /opt/agents/gmail_triage_server.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Now compare that to WebMCP. A WebMCP tool is declared by the page, discovered by the agent through the browser, and executed against the user's session. Rough shape:

<!-- Publisher-side: a page exposing a WebMCP tool -->
<script type="module">
  navigator.mcp.registerTool({
    name: "add_to_cart",
    description: "Add a SKU to the current cart",
    inputSchema: { type: "object", properties: { sku: { type: "string" } } },
    async handler({ sku }) {
      const res = await fetch("/api/cart", {
        method: "POST",
        body: JSON.stringify({ sku }),
        credentials: "include"  // user's session cookie
      });
      return await res.json();
    }
  });
</script>

Different world. That tool only exists while the tab is open, only works for the logged-in user, and only fires when an agent-capable browser decides to invoke it. Try to run that on a cron at 6 AM — you can't. There's no tab, no session, no user.

Where founders and operators keep getting this wrong

The pattern I see repeatedly with SMB owners chasing every I/O announcement: they conflate "AI in the browser" with "AI running my business." They're not the same thing, and the difference costs real money when you build the wrong one.

Concrete examples of workloads that need a server-side MCP, not WebMCP:

Inbox triage at 6 AM — no one is awake, no browser is open
Invoice chase on the 1st of the month — cron-driven, needs your Stripe/QuickBooks keys, not the user's session
Lead follow-up on a 48-hour delay — queue-driven, runs whether you're on a call or on a flight
Cross-tool sync (CRM → email → Slack) — happens on webhooks, not on tab loads
Reports pushed to Telegram/Slack while you're in a meeting
Multi-tenant automations where you serve dozens of clients — one browser session cannot be dozens of users

None of those have a "user looking at a screen" moment. All of them need a process holding your credentials, not a user's session cookie.

The inverse mistake is also real: building a heavy server-side agent to do something that's genuinely browser-scoped, like helping a user complete a checkout on a site they're already signed into. That's where WebMCP is the right answer and a VPS is overkill.

Quick rule of thumb

Agent triggered by a human click in a specific tab → WebMCP
Agent triggered by time, event, queue, or webhook → stdio/HTTP MCP on your infra
Agent needs your API keys → server-side, always
Agent needs the user's logged-in session → WebMCP, always

The other two I/O 2026 updates, quickly

Since everyone's asking:

Built-in AI in Chrome (Gemini Nano exposed to web pages) — useful for tiny client-side tasks. Summarize this form, classify this input, redact PII before it leaves the browser. It's cheap because it runs on the user's device. It's limited for the same reason: small model, no tool use, no persistence, no cross-session memory. Use it for UX polish. Do not build a business process on it. If your "AI feature" breaks the moment the user closes the tab, it isn't a business process.

Skills — reusable capability bundles an agent can load on demand. This one is actually interesting for solopreneurs because it maps cleanly onto the problem of "my prompt library is now 40 markdown files and I can't remember which one does what." Collapsing those into shippable, versioned skill units is a real pattern. I'll write that one up separately — it deserves its own post, not a paragraph.

For context, the Model Context Protocol spec itself still defines stdio and HTTP as the transport surfaces for server-side MCP. WebMCP sits alongside it as a browser transport, not on top of it. Read the actual spec before believing a hot take.

Why bizflowai.io helps with this

The server-side MCP work I described above — Gmail triage, invoicing, Telegram/Slack notifications, lead follow-up, cross-tool sync — is exactly what we wire up for clients at bizflowai.io every week. The build is boring on purpose: real MCP servers on a VPS or your own box, holding your credentials, driven by cron and webhooks, logging every tool call so you can audit what the agent did overnight. Nothing fancy, nothing browser-dependent, nothing that stops working when you close a tab. If your automation needs to run while you're asleep, this is the shape it takes.

Pick based on where the agent runs, not what got retweeted

WebMCP is real, useful, and correctly scoped for browser-resident tools. It is not a replacement for the MCP servers doing the actual work in your business. If your agent needs a human staring at a screen to function, WebMCP is on the table. If it needs to run at 6 AM, on the 1st of the month, or when a webhook fires — you need a server-side MCP holding your credentials, and no keynote is going to change that.

400 tool calls a day. Zero browsers. Pick the runtime that matches the job.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

Anthropic Shipped @Claude For Slack. My Team Runs On

lamingsrb — Thu, 09 Jul 2026 06:11:46 +0000

Anthropic Shipped @claude for Slack. My Team Runs on Telegram.

Anthropic just shipped @Claude inside Slack channels. Tag the bot, it reads the thread, does work async, posts back. Nice product. Except roughly 95% of small businesses don't live in Slack — they run on WhatsApp, Telegram, and Gmail. If you're a solopreneur or a 1-to-10-person team, here's the exact four-part recipe I use to run the same pattern in Telegram for under $12/month.

What Anthropic actually shipped (and who it's for)

Anthropic shipped an enterprise distribution deal wearing a product launch t-shirt. @Claude for Slack lets you tag the bot in a channel or thread, gives it channel memory, connects to your other apps, and returns work asynchronously — but only on Slack Team and Enterprise plans. That's the punchline: it lives where the annual contracts live.

Look at the raw user counts. Slack's own reporting puts it around 35–40 million weekly active users globally. WhatsApp is over 2 billion. Telegram is over 900 million. Gmail sits around 1.8 billion. In the 1-to-10-employee segment outside US tech, Slack penetration is single digits. Small teams in Europe, LATAM, and most of Asia coordinate in WhatsApp groups and run pipeline out of Gmail. They are not about to add Slack seats at $15/user/month just to get an @Claude mention.

That's a rational call for Anthropic — Slack is where the enterprise procurement motion already exists. It's just not a product for the operator segment. And the pattern they productized is trivially replicable on any messenger with a bot API.

Platform	Weekly/monthly active users	Bot API	Cost to run a mention-bot
Slack	~35–40M WAU	Yes, paid plan	$15/user/mo + API
Telegram	~900M MAU	Yes, free	~$5–12/mo API only
WhatsApp Business	~2B MAU	Yes, metered	$0.005–0.08/conversation + API
Gmail	~1.8B MAU	Pub/Sub push	Free tier + API

The four-part recipe (works in any messenger)

Every mention-bot is the same four moving parts: a webhook that fires on mention, a context store that holds recent thread history, a model call that produces the reply, and a send back to the same thread. Total code for a working Telegram version: three files, ~200 lines, running on a $10 VPS or a home server.

Here's the shape, framework-free:

[messenger webhook]  →  [ingest handler]  →  [SQLite context store]
                                              ↓
                                        [Claude API call]
                                              ↓
                                    [messenger send endpoint]

That's it. No Zapier, no n8n, no vendor lock-in. Every piece is a documented public API.

The four parts, one line each

Webhook: Telegram/WhatsApp/Gmail fires a POST when your bot is mentioned or a filter matches.
Context: SQLite table keyed by thread_id, last 20 messages.
Model: Claude API with a system prompt describing the bot's job + thread context as the user turn.
Reply: Same messenger's sendMessage endpoint, back to the same thread.

Part 1 + 2: Webhook and SQLite context store

The first 80% of the work is capturing the trigger and storing context. Do this step alone this week — don't wire Claude yet. Prove you can catch every mention and log the thread. It costs zero dollars.

Register a Telegram bot with @BotFather, add it to a group, set a webhook URL pointing at your server, and enable group message access. The webhook payload arrives as JSON on every message.

# webhook.py — FastAPI ingest, ~40 lines
import sqlite3, os
from fastapi import FastAPI, Request

app = FastAPI()
DB = "bot.db"

def db():
    c = sqlite3.connect(DB)
    c.execute("""CREATE TABLE IF NOT EXISTS msgs(
        chat_id INTEGER, msg_id INTEGER, user TEXT,
        text TEXT, ts INTEGER, mentions_bot INTEGER)""")
    return c

BOT_USERNAME = os.environ["BOT_USERNAME"]  # e.g. "myops_bot"

@app.post("/tg")
async def tg(req: Request):
    u = await req.json()
    m = u.get("message") or {}
    text = m.get("text", "")
    if not text:
        return {"ok": True}
    chat_id = m["chat"]["id"]
    mentions = int(f"@{BOT_USERNAME}" in text)
    con = db()
    con.execute("INSERT INTO msgs VALUES (?,?,?,?,?,?)",
        (chat_id, m["message_id"], m["from"].get("username",""),
         text, m["date"], mentions))
    con.commit()
    if mentions:
        # enqueue for the Claude worker — covered in Part 3
        pass
    return {"ok": True}

Ship that, add the bot to one real group, and let it log for two days. You'll immediately see which threads matter and how much context 20 messages actually covers. On my ops group, 20 messages is roughly the last 45 minutes of conversation — usually enough for the bot to answer without asking clarifying questions.

For WhatsApp, swap the webhook for the Meta Cloud API endpoint and the same table works. For Gmail, use a Gmail push notification via Pub/Sub with a label filter — the "thread" is the email thread ID.

Part 3: The Claude API call, wired to thread context

When a mention fires, pull the last N messages from SQLite for that chat, wrap them as one user turn, and call the API. That's the entire agent. Memory is not magic — it's a SELECT ... ORDER BY ts DESC LIMIT 20.

# worker.py — the Claude call
import os, sqlite3, anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

SYSTEM = """You are the ops assistant for a 4-person consulting team.
Answer briefly. If the group is discussing a client, pull the relevant
client facts from the thread. Never invent numbers. If unsure, ask."""

def build_context(chat_id: int) -> str:
    con = sqlite3.connect("bot.db")
    rows = con.execute(
        "SELECT user, text FROM msgs WHERE chat_id=? ORDER BY ts DESC LIMIT 20",
        (chat_id,)).fetchall()
    rows.reverse()
    return "\n".join(f"{u}: {t}" for u, t in rows)

def answer(chat_id: int) -> str:
    ctx = build_context(chat_id)
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=600,
        system=SYSTEM,
        messages=[{"role": "user", "content":
            f"Recent thread:\n{ctx}\n\nRespond to the latest @mention."}]
    )
    return msg.content[0].text

Real numbers from my server last month across three groups (roughly 40 mentions/day):

Input tokens billed: ~2.1M
Output tokens billed: ~180K
Total Anthropic bill: $11.42
VPS: home server, $0 marginal
Slack seats saved for a 4-person team at $15/user/mo: $720/year

That's the whole economic argument. The Slack version isn't better — it's just packaged.

Part 4: Reply to the same thread + reliability tricks

Sending back to Telegram is one HTTP POST. The trick is quoting the original message so the reply lands in-thread and doesn't spam the channel, and rate-limiting so a chatty group doesn't burn your API budget on one bad afternoon.

import httpx, os

TG_TOKEN = os.environ["TG_TOKEN"]

def reply(chat_id: int, reply_to: int, text: str):
    httpx.post(
        f"https://api.telegram.org/bot{TG_TOKEN}/sendMessage",
        json={
            "chat_id": chat_id,
            "reply_to_message_id": reply_to,
            "text": text,
            "parse_mode": "Markdown",
        }, timeout=15)

Three cheap reliability moves that have saved me real money and real embarrassment:

Debounce: if two mentions fire within 10 seconds in the same chat, collapse to one call. Group chats double-tap constantly.
Daily cap: hard-stop each chat at 200 mentions/day. One runaway loop between two bots almost cost me $80 in a single evening before I added this.
Fail-quiet: if the Claude call errors, log it, don't reply. Silent failure is better than "I'm sorry, I encountered an error" flooding the group.

For WhatsApp Business the send endpoint is a POST to graph.facebook.com/v20.0/{phone_id}/messages with a context.message_id for the reply target. For Gmail, users.messages.send with the original threadId keeps replies in-thread. Same four parts, different vendor.

Why the moat is the wiring, not the model

Every frontier lab is going to ship the polished version of this workflow for the platform where their enterprise buyers already pay. Anthropic picked Slack. OpenAI's team offering is heading the same direction. Neither of them is going to ship a first-class WhatsApp Business or Telegram bot because there's no procurement contract at the other end.

That leaves a specific opening for operators. The API is public. The messenger webhooks are public. The only thing that isn't handed to you is knowledge of your own workflow — which client threads matter, which questions repeat, which context the bot needs to be useful instead of annoying. That's not a technical moat. It's an operator moat, and it takes about a weekend to build the first version.

The one practical takeaway for this week: pick the messenger your team actually uses, register a bot, and set up the webhook + SQLite logger. Don't add Claude yet. Just prove you can capture every mention and the surrounding thread. Once that's stable, adding the API call is an afternoon.

Where bizflowai.io fits in

This is exactly the pattern we assemble for clients at bizflowai.io — mention-bots and inbox agents wired into the messenger the team already lives in (Telegram, WhatsApp Business, Gmail), backed by a small SQLite or Postgres context store, running on a $10 VPS or the client's own server with no per-seat fees. Same four parts as above, hardened with the debouncing, daily caps, and fail-quiet handling that keep the API bill boring. It's not a SaaS subscription and it's not a Slack replacement — it's the wiring most small teams don't have time to write themselves.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

Claude Fable 5 Global Access Restored: Enterprise Playbook

lamingsrb — Mon, 06 Jul 2026 06:11:37 +0000

Claude Fable 5 Global Access Restored: Enterprise Playbook

Your Claude-based agent pipelines went dark on June 12 when Anthropic pulled global access to Fable 5 overnight. As of today, access is back — but re-enabling production agents isn't a toggle flip. If you ship automation for a living, here's what changed, what didn't, and how to bring your workloads back online without breaking SLAs.

What Happened: Export Controls Withdrawn, Access Restored

The U.S. Department of Commerce issued emergency export controls on June 12, 2026, that classified Claude Fable 5 — and its cybersecurity counterpart — under technology export restrictions. Anthropic's immediate response was to suspend all global access outside the U.S. to both models rather than risk compliance violations. Last night, Commerce withdrew those controls, and Anthropic began restoring global availability today.

For engineering teams, the timeline looked like this:

Date	Event	Impact
June 12, 2026	Export control order issued	Anthropic suspends Fable 5 globally
June 12 – July 4	Appeals, negotiations, industry pressure	Workloads either downgraded or offline
July 4, 2026	Commerce withdraws controls	Legal barrier removed
July 5, 2026	Anthropic restores global access	Pipelines can be re-enabled

The withdrawal means enterprises outside the U.S. can resume using Fable 5 through the standard channels: Anthropic's API, Amazon Bedrock, and Google Cloud's Vertex AI Model Garden. There is no separate "international" tier — the same model endpoints serve all regions.

Where Enterprises Can Access Claude Fable 5 Right Now

Fable 5 is available through three primary enterprise channels. The right choice depends on your existing cloud commitments, compliance requirements, and whether you need features like guardrails, provisioning, or private endpoints.

Anthropic API (direct) — The fastest path if you already have an Anthropic console account. You get the latest model versions first, direct access to features like extended thinking and tool use, and Anthropic's native rate limits. Best for teams that want the shortest dependency chain.

Amazon Bedrock — If your infrastructure lives in AWS, Bedrock gives you Fable 5 with IAM-based access control, CloudWatch logging, and VPC integration. You trade some feature lag (Bedrock typically lags the direct API by days to weeks on new capabilities) for enterprise governance. Check the Bedrock model catalog page for current regional availability.

Google Cloud Vertex AI — Similar proposition to Bedrock but for GCP shops. You get Fable 5 alongside Google's Model Garden tooling, IAM, and audit logging. If you're already running workloads on Vertex, enabling Fable 5 is a configuration change, not an architecture change.

A practical note: if your pipelines were running on Bedrock or Vertex before June 12, your model access may already be restored. AWS and GCP typically re-enable models at the platform level. If you were on the direct Anthropic API and your workspace region was set to a non-U.S. locale, you may need to verify that your organization's geo settings haven't been permanently altered.

Assessing Pipeline Damage From the 23-Day Outage

Before flipping models back on, take 30 minutes to assess what actually broke during the outage. Rushing a rollback is how you introduce silent failures — a model that technically responds but behaves differently enough to corrupt downstream data.

Start by checking three things:

# 1. Check which pipelines fell back to a downgrade model
grep -r "claude-fable" ./pipeline-logs/ --include="*.json" | jq 'select(.model != "claude-fable-5") | .pipeline_id, .timestamp, .model'

# 2. Find jobs that failed outright (no fallback configured)
grep -r "model_not_available\|access_denied" ./pipeline-logs/ --include="*.json" | jq '.pipeline_id, .error.message'

# 3. Identify any cached responses that may be stale or wrong
find ./cache/ -name "*fable*" -newer ./config/suspension-timestamp.txt

The teams that weathered the outage best had one thing in common: they'd built model-agnostic abstractions. Their agent logic referenced a model alias (primary-reasoning-model) rather than a hardcoded model string. When Fable 5 went down, they changed one environment variable and rerouted to Claude Opus or Sonnet. The teams that suffered hardest had "model": "claude-fable-5" hardcoded into 40 different API calls across their codebase.

If you're in the second group, this outage is your signal to fix that. It won't be the last time a model becomes temporarily unavailable — whether due to export controls, capacity issues, or deprecation cycles.

Safely Upgrading Your Agent Pipelines Back to Fable 5

Restoring Fable 5 access doesn't mean you should point every agent at it and ship. Model behavior can shift between the version you were running on June 11 and the one coming back online today. Anthropic has not confirmed whether the restored model is the exact same checkpoint or a subsequent build.

Here's a rollout approach that minimizes risk:

Phase 1: Smoke test (1-2 hours)

Run your standard evaluation suite against the restored model. If you don't have a standard eval suite, run 20-50 of your most representative production inputs through the model and compare outputs to your last-known-good baseline.

import anthropic
import json

client = anthropic.Anthropic()

# Load your eval cases — real production inputs with known-good outputs
with open("evals/fable5_baseline.json") as f:
    eval_cases = json.load(f)

results = []
for case in eval_cases:
    response = client.messages.create(
        model="claude-fable-5",  # verify exact model ID in Anthropic's docs
        max_tokens=case["max_tokens"],
        system=case["system_prompt"],
        messages=case["messages"]
    )
    results.append({
        "case_id": case["id"],
        "output": response.content[0].text,
        "usage": response.usage.model_dump()
    })

# Compare against baseline — flag any drift > 15%
for r, baseline in zip(results, eval_cases):
    drift = calculate_semantic_distance(r["output"], baseline["expected"])
    status = "✓" if drift < 0.15 else "⚠ REVIEW"
    print(f"{status} Case {r['case_id']}: drift={drift:.2f}")

Phase 2: Canary deployment (24-48 hours)

Route 5-10% of traffic to Fable 5. Monitor error rates, latency, and — most importantly — output quality. A model can return 200 OK responses that are subtly worse. If you have human reviewers in your loop, have them blind-score Fable 5 outputs against the fallback model you've been running.

Phase 3: Full rollout

If the canary shows no regression, ramp to 100%. Keep your fallback model configured and ready for at least a week. Export controls were withdrawn once; they could be reimposed. Treat model availability as ephemeral.

Multi-Model Architecture: The Real Lesson

The Fable 5 outage exposed a structural weakness in how many teams build agent pipelines: single-model dependency. When one model becomes unavailable, everything stops. The fix isn't to avoid Fable 5 — it's to build routing layers that treat models as interchangeable infrastructure.

A basic multi-model router looks like this:

import os
from typing import Optional

class ModelRouter:
    def __init__(self):
        self.primary = os.getenv("PRIMARY_MODEL", "claude-fable-5")
        self.fallback = os.getenv("FALLBACK_MODEL", "claude-sonnet-4")
        self.tertiary = os.getenv("TERTIARY_MODEL", "claude-opus-4")

    def get_model(self, task_type: str, complexity: int) -> str:
        """Route based on task requirements and model availability."""
        if not self._is_available(self.primary):
            return self.fallback if complexity < 7 else self.tertiary

        # Route expensive reasoning to primary only when warranted
        if task_type in ["code_generation", "complex_analysis"] and complexity >= 6:
            return self.primary
        return self.fallback

    def _is_available(self, model: str) -> bool:
        # Check a health endpoint or cache recent failures
        return self._health_check(model)

The point isn't the specific code — it's the principle. Your agent logic should never assume a specific model exists. Define your pipeline in terms of capabilities ("fast reasoning," "long-context analysis," "code generation") and let a router map those to whatever model is available and cost-effective at runtime.

This architecture also saves money. Not every task needs Fable 5. Simple classification, formatting, and extraction tasks run fine on smaller, cheaper models. Routing intelligently can cut API costs significantly without degrading output quality.

Compliance and Data Governance Going Forward

The export control episode will have lasting effects on how enterprises evaluate AI model dependencies. Legal and procurement teams are now asking questions that engineering teams haven't traditionally prepared for.

If you're operating in regulated industries — finance, healthcare, government contractors — document the following:

Data residency: Where does your inference traffic actually get processed? Anthropic's API may route requests through different regions depending on load. If you're on Bedrock or Vertex, requests stay within your configured cloud region. For data-sensitive workloads, pin your region explicitly.

Audit trails: Can you reconstruct which model processed a given input at a given time? If a regulator asks "was this decision made using an AI model subject to export controls on June 15?" you need a precise answer. Log the model ID, version, and timestamp for every inference call.

Vendor lock-in exposure: How quickly could you switch models if Fable 5 became permanently unavailable in your jurisdiction? If the answer is "we'd need to rewrite our entire agent layer," you have a concentration risk. Insurance companies and compliance auditors are starting to ask this question directly.

Contractual fallbacks: If you have an enterprise agreement with Anthropic, AWS, or Google, review what it says about model availability disruptions. Most standard contracts don't guarantee specific model availability. If you need that guarantee, it's a negotiation point.

For U.S.-based teams, the export control withdrawal is mostly good news — access is restored, move on. For teams in the EU, UK, Canada, Australia, and other jurisdictions, the 23-day outage was a stress test. The regulators noticed. Expect forthcoming guidance from the EU AI Office and UK AI Safety Institute on model dependency risk.

Re-Enabling Cybersecurity Workloads

Fable 5's cybersecurity counterpart — the less restricted variant mentioned in the original export order — was also caught in the suspension. If you were running security-focused agent workloads (threat intelligence parsing, vulnerability analysis, security log triage), those need the same careful re-enablement.

Cybersecurity models carry additional considerations: they may be subject to separate usage policies, require enhanced verification, or have different logging requirements depending on your sector. Before pointing a security agent back at the restored model, verify that your usage patterns still comply with Anthropic's usage policies — those may have been updated during the 23-day window.

If you're building security automation on Fable 5's cybersecurity variant, test against your threat intelligence corpus first. The model's behavior on adversarial inputs, prompt injection attempts, and malformed security data should be verified before production traffic resumes.

How BizFlowAI Approaches This

We've spent the last several days helping clients reroute and restore Claude-based agent pipelines after the Fable 5 suspension. The pattern that works — and that we build into every engagement — is model-agnostic agent architecture with automatic fallback. When Fable 5 went dark on June 12, our clients' pipelines degraded gracefully to alternative models rather than crashing. Now that access is restored, we're running canary evaluations before flipping traffic back.

If your pipelines broke hard during the outage and you want to build something that survives the next disruption — whether it's export controls, capacity issues, or model deprecations — book a discovery call. We'll look at your current architecture and map out where the single-point-of-failure risks are.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.

3 Businesses, 1 Server, 7AM: The Operations Loop Nobody

lamingsrb — Mon, 06 Jul 2026 06:11:34 +0000

3 Businesses, 1 Server, 7AM: The Operations Loop Nobody Teaches

At 7AM every morning, my phone buzzes once. That single Telegram message tells me whether my three live systems survived the night — or if something broke at 3AM and I need to fix it before a client notices. Nobody teaches you this part. Courses show you how to ship a prototype in an afternoon. They never show you what happens on day 47 when an API times out and a client's invoice queue stalls.

I run three automated systems on one home server — a Gmail-Telegram bot ecosystem, an invoicing automation tool, and a lead-gen engine. Real users pay for them daily. Here's exactly how I built the operations loop that keeps all three alive without hiring anyone.

The Morning Check: One Telegram Message Instead of 45 Minutes

The core insight is this: a solopreneur doesn't need dashboards. You need a single, scannable summary that tells you "all green" or "here's the red." If it's green, you go back to sleep. If it's red, you know exactly where to look.

Every morning at 7AM, my operations bot sends a Telegram message with three sections:

📊 MORNING REPORT — 2026-07-05

✅ Gmail Ecosystem
   Processed: 47 emails (3 accounts)
   Flagged: 4 urgent
   Errors: 0

✅ Invoice System
   Generated: 12 invoices
   Queued for approval: 3
   Errors: 0

🔴 Lead-Gen Engine
   Outreach sent: 89
   Replies received: 7
   Errors: 1 (5:30AM — SMTP timeout, retried x3, escalated)

That red block tells me everything I need: which system, what time, what error, and what the retry logic already tried. In this case, the SMTP timeout resolved itself after retry — but I still know it happened.

The alternative most solopreneurs default to is opening 6 browser tabs, checking each tool's dashboard, manually cross-referencing timestamps, and trying to remember what "normal" looks like. That's 45 minutes gone before you've done any actual work.

The message itself is generated by a Python script that runs at 7AM sharp:

import asyncio
from datetime import datetime

async def generate_morning_report():
    """Pulls status from all 3 systems and sends one Telegram message."""
    gmail_stats = await get_gmail_stats(since="midnight")
    invoice_stats = await get_invoice_stats(since="midnight")
    leadgen_stats = await get_leadgen_stats(since="midnight")

    sections = [
        format_section("Gmail Ecosystem", gmail_stats),
        format_section("Invoice System", invoice_stats),
        format_section("Lead-Gen Engine", leadgen_stats),
    ]

    status = "✅ All systems nominal" if all(
        s["errors"] == 0 for s in [gmail_stats, invoice_stats, leadgen_stats]
    ) else "🔴 Issues detected — see below"

    message = f"📊 MORNING REPORT — {datetime.now():%Y-%m-%d}\n\n{status}\n\n"
    message += "\n\n".join(sections)

    await send_telegram(message)

No Grafana. No Datadog. A text message.

The Architecture: Three Systems, One Home Server, Zero Cloud Bills

All three systems run on a single PC-PC home server using WSL Ubuntu on consumer hardware. No cloud instances. No Kubernetes. No AWS bill that makes you nervous. The entire stack costs the electricity to run a desktop PC — roughly $15-20/month depending on your local rate.

The three systems are completely decoupled, each running as an independent process with its own cron schedule:

System	Schedule	Function	Dependencies
Gmail-Telegram bot	Every 15 min	Triage, categorize, forward	Gmail API, Telegram API
Invoice system	Every 2 hours	Pull billing emails, generate invoices	Gmail API, PDF library, SQLite
Lead-gen engine	3x daily (8AM, 1PM, 5:30PM)	Outreach, qualification	SMTP, LinkedIn API, PostgreSQL

The decoupling is the critical part. If the Gmail bot crashes at 2AM, the invoice system and lead-gen engine keep running. They don't share a process, they don't share a database, and they don't share a failure mode. Here's the process structure:

# Each system runs as an independent systemd service
$ systemctl --user list-units | grep active | grep -E "bizflowai.io|gmail|leadgen"
gmail-bot.service        active running
invoice-engine.service   active running
leadgen-engine.service   active running

Each service has its own working directory, its own virtualenv, and its own log files. If one needs to restart, the others don't even notice. This is the poor-person's microservices architecture — and for a one-person business, it's exactly right.

The directory structure on the server:

/home/lazar/
├── systems/
│   ├── gmail-ecosystem/
│   │   ├── venv/
│   │   ├── src/
│   │   ├── config/
│   │   └── logs/
│   ├── invoice-system/
│   │   ├── venv/
│   │   ├── src/
│   │   ├── config/
│   │   └── logs/
│   └── leadgen-engine/
│       ├── venv/
│       ├── src/
│       ├── config/
│       └── logs/
├── shared/
│   ├── telegram-bot.py     # Shared alerting
│   └── error_handler.py    # Shared retry logic
└── deploy.sh               # Single deploy script

The shared/ directory contains the Telegram bot and error handling logic that all three systems import. Everything else is isolated.

The Monitoring Layer: Three Message Types, Not Three Dashboards

Most one-person operations fall apart at the monitoring layer because solopreneurs build like enterprise teams — dashboards, metrics, alerts routed to email. Then they stop checking the dashboard because it's always green, and miss the one time it goes red.

My monitoring layer is three Telegram message types. That's it.

Message 1: Morning summary (described above) — daily at 7AM.

Message 2: Immediate failure alert — within 60 seconds of any unhandled error.

async def alert_failure(system: str, error: Exception, context: dict):
    """Send immediate Telegram alert with stack trace."""
    msg = f"""🔴 FAILURE — {system}
Time: {datetime.now():%H:%M:%S}
Error: {type(error).__name__}: {str(error)[:200]}
Stack: {traceback.format_exc()[-500:]}

Retries attempted: {context.get('retries', 0)}
Action needed: {'Auto-retry queued' if context.get('will_retry') else 'Manual fix required'}
"""
    await send_telegram(msg, priority="high")

Message 3: Weekly trend report — every Sunday at 6PM.

📈 WEEKLY TRENDS — Week of Jun 28

Email volume: 312 (+8% vs last week)
Invoice errors: 2 (stable)
Lead-gen conversion: 4.1% (down from 5.2% — investigate)

Top failure mode: SMTP timeout (3 occurrences)
Avg resolution time: 11 min

That weekly report is where I catch slow degradation before it becomes an emergency. If email volume creeps up 8% per week for three weeks, I know I'm going to hit a rate limit soon and can adjust preemptively.

The point isn't the specific messages — it's that I can check the state of my entire infrastructure from my phone, whether I'm at my desk or walking outside. Three message types. No browser tabs. No "I should really set up Grafana someday."

When Things Break at 3AM: A Real Failure Walkthrough

Here's a real failure from last Tuesday at 3:12AM. The Gmail API started returning 429 rate limit errors because one of the managed accounts received an unusually large batch of forwarded emails. Here's exactly what happened, step by step:

3:12:04 AM — Gmail API returns HTTP 429 on batch processing request.
3:12:05 AM — Error handler catches the 429, logs it, starts exponential backoff retry.

async def call_gmail_with_retry(request_fn, max_retries=3):
    """Gmail API call with exponential backoff for 429 errors."""
    base_delay = 30  # seconds
    for attempt in range(max_retries):
        try:
            return await request_fn()
        except HttpError as e:
            if e.resp.status == 429 and attempt < max_retries - 1:
                delay = base_delay * (2 ** attempt)  # 30, 60, 120
                logger.warning(f"429 rate limit, retry in {delay}s (attempt {attempt + 1})")
                await asyncio.sleep(delay)
                continue
            raise
    # All retries exhausted — escalate
    await alert_failure("Gmail Ecosystem", e, {"retries": max_retries, "will_retry": False})

3:12:35 AM — First retry (30s delay). Still 429.
3:13:35 AM — Second retry (60s delay). Still 429.
3:15:35 AM — Third retry (120s delay). Success on 6 of 8 pending emails.
3:15:40 AM — Two emails remain unprocessed. Telegram alert sent to my phone.

When I woke up at 7AM, the morning report showed the alert from 3:15AM. Six of eight emails were handled automatically. Two needed manual intervention — one had an attachment format the parser didn't recognize, and one was flagged as suspicious by the spam filter but wasn't actually spam.

Here's the fix loop. I opened Claude Code in the project directory, pasted in the error log, and asked it to diagnose. Claude read the error, looked at the retry logic in the codebase, and identified that the rate limit threshold was set too aggressively — processing 10 messages per second when the API allows 20 per minute. Claude wrote the fix, updated the config, and I reviewed the diff before committing. Total time: 8 minutes.

# What the diff looked like
- RATE_LIMIT_REQUESTS = 10  # per second
- RATE_LIMIT_WINDOW = 1     # seconds
+ RATE_LIMIT_REQUESTS = 18  # per minute (safe margin under 20)
+ RATE_LIMIT_WINDOW = 60    # seconds

Then I pushed to the repo, the deployment script pulled the update on the server, the service restarted, and the two stuck emails processed within 90 seconds. Zero downtime for the other two systems because they're independent processes.

This is the loop that keeps a one-person business alive: build, monitor, fail, fix, deploy, repeat. The fix-deploy cycle needs to be fast because you don't have a team. Claude Code is what makes 8-minute fixes possible. I'm not writing code at 7AM with coffee in one hand — I'm reviewing code that Claude wrote based on a real error log. That's the difference between a system you can operate alone and a system that needs a developer on retainer.

Deploying New Features Without Taking Systems Down

The deployment workflow is the part where most solo-built systems die — not because the code is wrong, but because deploying changes feels risky, so you stop making changes, and the system rots. Here's how I keep the cycle fast.

I never push directly to the branch running on the server. The git workflow is simple:

# 1. Feature branch
git checkout -b fix/gmail-rate-limit

# 2. Claude Code writes the fix
claude "Update rate limit config based on this error log: [paste]"

# 3. Review the diff
git diff

# 4. Test locally against sample input
python -m pytest tests/test_gmail_rate_limit.py

# 5. Merge and deploy
git checkout main && git merge fix/gmail-rate-limit
git push origin main
# Deployment script handles the rest

The deployment script on the server does three things: pulls the latest commit, runs a health check, and restarts the service. If the health check fails, it rolls back automatically.

#!/bin/bash
# deploy.sh — runs on server after git push
set -e

SERVICE=$1
BRANCH="main"
HEALTH_URL="http://localhost:8000/health"

echo "Deploying $SERVICE..."

# Save current commit for rollback
PREV_COMMIT=$(git rev-parse HEAD)

# Pull latest
git fetch origin
git checkout $BRANCH
git pull origin $BRANCH

# Restart service
systemctl --user restart $SERVICE

# Health check — 30 second window
for i in {1..6}; do
  sleep 5
  if curl -sf $HEALTH_URL > /dev/null 2>&1; then
    echo "✅ Health check passed. Deployed $(git rev-parse --short HEAD)"
    exit 0
  fi
done

# Health check failed — rollback
echo "🔴 Health check failed. Rolling back to $PREV_COMMIT"
git checkout $PREV_COMMIT
systemctl --user restart $SERVICE
echo "Rolled back. Manual investigation needed."

Total time from feature request to production: about 12-18 minutes for a straightforward change. The auto-rollback has saved me twice — once when a dependency update broke the PDF library in the invoice system, and once when I accidentally tightened a regex too aggressively in the email parser.

The key practices that make this safe:

Never deploy on Friday afternoon (classic rule, still true)
Always test locally against a sample input before merging
If the health check catches it, the rollback handles it — you haven't broken anything for users

How bizflowai.io Fits Into This Operations Loop

Running three live systems solo taught me what breaks at 2AM and what actually matters at 7AM. The operations loop — the monitoring layer, the retry logic, the morning summary, the deploy-with-rollback — is what I folded into the tooling at bizflowai.io. The platform handles the repetitive parts: invoice generation pipelines, email triage with retry built in, lead-gen scheduling with health checks. You bring your Gmail accounts and your business rules; the retry logic, the Telegram alerts, and the deploy workflow are already wired. Less time building plumbing means more time actually fixing the 3AM problems when they matter.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

How to Create an AI Agent: A Production Walkthrough

lamingsrb — Mon, 29 Jun 2026 06:24:35 +0000

How to Create an AI Agent: A Production Walkthrough

The first agent I shipped to production failed at 3am on a Sunday. It looped on a tool call, burned through $40 in tokens before my budget alarm fired, and left a half-written draft in the database with no way to resume. That night taught me more about agent design than any framework tutorial. Since then I have built a pattern I trust enough to leave running unattended for weeks at BizFlowAI, where agents research, write, optimize and publish content without me touching them.

This is that pattern, stripped down to what actually matters.

Start with the job spec, not the framework

Before you pick LangGraph, CrewAI, or roll your own, write the agent's job spec like you would for a junior engineer. One paragraph. What it owns, what it must never do, what "done" looks like, and which signals tell you it failed.

Here is the spec for one of my production agents:

The Topic Researcher owns generating a ranked list of 20 content topics per site per week. It reads from keyword_pool and search_console_perf, writes to topic_queue. It must never publish, never call paid APIs more than 8 times per run, and must finish in under 6 minutes. Done = 20 topics with score >= 0.6 and zero duplicates against the last 90 days. Failure signal = empty queue after a run, or any topic flagged by the dedupe check.

If you cannot write this paragraph, do not build the agent. You will end up with a "do everything" prompt that hallucinates its way through ambiguous tasks. The job spec becomes your evaluation rubric later, so write it carefully.

Rule of thumb I use: if the spec needs more than 5 tools or more than 3 decision branches, it is two agents, not one.

Design the tools before you write the prompt

Most agent failures I have debugged were not prompt failures. They were tool failures. The model called a tool with wrong arguments, the tool returned a 4MB JSON blob, or two tools had overlapping responsibilities and the model picked the wrong one.

Treat tools like a public API you are shipping to a difficult customer. The customer is the LLM.

Three rules I follow:

Each tool does one thing and returns a small, structured result. If search_database returns 200 rows, the model will choke or pick poorly. Return 10 with a has_more flag and a next_cursor.
Tool names and parameter names are the prompt. A tool called fetch_recent_topics(days: int, min_score: float) is self-explanatory. A tool called get_data(query: str) is a coin flip.
Every tool has idempotency keys and a dry-run mode. When the agent retries after a timeout, you do not want duplicate publishes.

Here is the actual signature I use for a publishing tool:

def publish_post(
    site_id: str,
    draft_id: str,
    idempotency_key: str,  # hash of draft_id + content_hash
    scheduled_at: datetime | None = None,
    dry_run: bool = False,
) -> PublishResult:
    """Publishes a draft to the target CMS. 
    Returns PublishResult with url, published_at, and cms_post_id.
    If idempotency_key was used in the last 24h, returns the original result.
    """

The idempotency key has saved me at least four times. EventBridge retries, Lambda cold-start timeouts, network blips: all of them caused duplicate execution attempts in production. Without the key I would have shipped duplicate content.

Write a prompt that survives contact with reality

I no longer write monolithic system prompts. I write a system prompt that is mostly constraints and a runtime context block that gets rebuilt every turn. The split matters because the system prompt is the contract and the context is the working memory.

My template:

SYSTEM PROMPT (stable, ~600 tokens):
- Role and goal (3 sentences max)
- Hard constraints ("never call publish_post without dry_run first on first attempt")
- Tool inventory with one-line guidance per tool
- Output format for the final answer (JSON schema)
- Stop conditions ("when topic_queue has 20 entries, call finalize and stop")

RUNTIME CONTEXT (rebuilt per turn):
- Current task ID and attempt number
- Tool call history compressed: last 3 calls in full, older ones summarized
- Relevant memory entries pulled from pgvector (top 5 by relevance)
- Budget left: tokens, tool calls, seconds

Two specific things I have learned the hard way:

Tell the agent its budget. When I added "you have 8 tool calls remaining and 4 minutes" to the runtime context, my average run cost dropped roughly 30%. Models are surprisingly good at rationing when they know the limit.

Make stop conditions explicit and machine-checkable. "Stop when the task is complete" is not a stop condition. "Stop when topic_queue count returned by count_queue() is >= 20" is.

Memory: short, long, and the part everyone gets wrong

Agents need three kinds of memory and most tutorials only cover one.

Memory type	What it stores	Where I put it	TTL
Scratchpad	Current turn's reasoning, tool results	In-context, compressed each turn	Single run
Episodic	What happened in past runs (decisions, outcomes)	Postgres table, summarized	30-90 days
Semantic	Facts the agent should "know" (brand voice, prior topics)	pgvector + BM25 hybrid (RRF)	Indefinite

The part everyone gets wrong is episodic memory. Without it, your agent makes the same mistake every Tuesday. With it, you can write rules like "before generating a topic, check if a similar topic failed evaluation in the last 60 days, and if so, vary the angle."

For semantic memory I use Postgres with pgvector and a BM25 index, then combine results with Reciprocal Rank Fusion. Pure vector search consistently missed exact-match keywords ("Q3 pricing" returned posts about Q1). RRF is 30 lines of SQL and fixes it.

-- Simplified RRF combining vector and BM25
WITH vec AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rnk
  FROM memory WHERE site_id = $2 LIMIT 50
),
bm AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank(tsv, plainto_tsquery($3)) DESC) AS rnk
  FROM memory WHERE site_id = $2 LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rnk)) AS score
FROM (SELECT * FROM vec UNION ALL SELECT * FROM bm) x
GROUP BY id ORDER BY score DESC LIMIT 10;

The control loop: the part that decides if you sleep at night

A naive agent loop is while not done: llm_call(); execute_tool(). That is how you get a 3am token explosion. Mine looks like this:

def run_agent(task, max_tool_calls=10, max_tokens=80_000, max_seconds=300):
    state = load_or_init_state(task.id)
    budget = Budget(max_tool_calls, max_tokens, max_seconds)

    while not state.done:
        if budget.exhausted():
            return handoff(state, reason="budget")

        context = build_context(task, state, budget.remaining())
        response = llm.call(SYSTEM_PROMPT, context, tools=TOOLS)
        budget.charge_tokens(response.usage)

        if response.stop:
            state.done = True
            break

        for call in response.tool_calls:
            if not policy.allows(call, state):
                state.append_tool_result(call, error="policy_denied")
                continue
            result = execute_tool(call, idempotency=state.run_id)
            state.append_tool_result(call, result)
            budget.charge_call()

        persist_state(state)  # so we can resume on crash

    return state.final_output

Five things in there that matter more than they look:

persist_state every turn. When Lambda times out at 15 minutes, the next invocation resumes from the last persisted turn. No work lost.
policy.allows is a separate function, not a prompt instruction. Hard rules ("never publish before 9am UTC", "never call paid APIs if budget < $0.20") live in code, not in English.
handoff is a real state, not a failure. When the budget runs out, the agent writes its current state and the reason to a queue. A human or another agent picks it up. Agents that cannot ask for help are agents that fail silently.
Idempotency tied to run_id. Replays do not double-execute side effects.
build_context charges into the budget too. Big context windows are deceptively expensive. I cap retrieved memory at 5 entries and summarize old tool calls aggressively.

Evaluation: the part that turns a demo into a product

If you cannot tell me your agent's success rate on a fixed eval set, you do not have a production agent. You have a demo that has not failed yet.

I run three evaluation layers:

Layer 1: Unit-level. Each tool has tests with golden inputs and outputs. Boring, fast, runs on every commit.

Layer 2: Trajectory eval. A frozen set of 30-50 tasks with expected outcomes. Run the full agent, score the trajectory and the final output. For scoring I use a mix: deterministic checks where possible (did the topic queue have 20 entries? was the schema valid?) and an LLM judge for subjective parts (was the topic on brand?). The LLM judge runs with a calibrated rubric and I spot-check 10% of its scores against my own judgment monthly.

Layer 3: Production telemetry. Every run logs: tool calls made, tokens used, wall time, budget exhausted, handoff reason, and a sample of final outputs. I look at the dashboard every Monday. Drift shows up in tool call counts before it shows up in output quality.

A real number from my own systems: when I added trajectory eval to the content agent and started gating deploys on it, my "weird output" rate dropped from roughly 1 in 25 runs to under 1 in 200. Not zero. Never zero. But low enough to leave running unattended.

Deployment: serverless, scheduled, and boring

I deploy almost every agent as a Lambda triggered by EventBridge on a schedule, with state in Postgres (Supabase), secrets in AWS Secrets Manager, and observability through CloudWatch + a thin custom dashboard. Nothing exotic.

A few opinions:

Scale to zero or stay home. If your agent runs 4 times a day, a 24/7 container is a waste. Lambda + EventBridge costs me cents per agent per month.
One agent per Lambda. Co-locating agents to "save cold starts" creates dependency hell. Cold starts are 2 seconds. You will live.
Dead-letter queue for everything. Failed runs go to SQS with the full state. I review the DLQ weekly.
Feature flags for tools. I gate every new tool behind a flag. New tool ships disabled, gets enabled for one site, evaluated, then rolled out. This has caught two tool-design mistakes before they hit all sites.

The full production stack for a typical agent in my BizFlowAI ContentStudio:

EventBridge (cron)
  -> Lambda (agent runner, max 15 min)
     -> Postgres (state, memory, queues)
     -> Claude API or local Ollama (LLM)
     -> Tool Lambdas (publish, fetch, analyze)
  -> CloudWatch (logs, metrics, alarms)
  -> SQS DLQ (failed runs)
  -> Dashboard (Next.js, reads Postgres)

What I would do if I were starting today

If you are building your first production agent in 2026, my opinionated shortlist:

Pick one workflow that already exists as a human task. Document it step by step. That document becomes your job spec and your eval set.
Build the tools first, with tests. Stub the agent loop. You can test 80% of the system without an LLM in the mix.
Use Claude or GPT for the main loop, not a local model. Local LLMs are great for embeddings and classification side-tasks; for the agent's reasoning you want the strongest model you can afford. The cost difference is smaller than the reliability difference.
Hard budgets in code, not in prompts. Always.
Evaluation before scale. Do not deploy to 10 use cases until you have 50 eval cases passing on 1.
Plan the handoff path on day one. What happens when the agent gets stuck? Who or what picks it up? If the answer is "nothing", you will find out at 3am.

The agents I trust to run while I sleep are not the smartest ones. They are the ones with the tightest tool contracts, the most boring control loop, and the eval set I actually run.

If you are working on an agent that needs to leave the demo stage and survive in production, or you want a second pair of eyes on an architecture before you commit to it, I am happy to talk. You can reach me at lazar-milicevic.com/#contact, or browse more posts on the blog where I write about RAG, evaluation, and the unglamorous parts of shipping AI systems.

The AI Implementation Process I Use With Every Client

lamingsrb — Mon, 29 Jun 2026 06:24:31 +0000

The AI Implementation Process I Use With Every Client

Most AI projects do not fail at the model. They fail in the six weeks before anyone writes a prompt, and in the six weeks after the demo lands in a Slack channel and nobody knows who owns it. I have run enough of these now (from one-off automations to multi-agent content systems running unattended) that the process has converged into something stable. This is the version I actually use.

It has five phases: scoping, POC, integration, evaluation, operations. Each phase has an exit criterion. If we cannot meet the exit criterion, we do not move forward. That single rule has saved more projects than any clever architecture choice.

Phase 1: Scoping (1 to 2 weeks, fixed price)

Scoping ends with a written document that names the workflow being automated, the system of record it touches, the success metric in hours or dollars, the data we have access to, and the smallest possible first slice. No model is chosen yet. No code is written. If we cannot produce that document, the engagement stops here and the client keeps the document.

The hardest part of scoping is resisting the urge to solve the interesting problem. Clients almost always describe the AI-shaped fantasy ("an agent that handles all support tickets") when the real opportunity is narrower and uglier ("triage tier-1 tickets that mention billing, route to the right queue, draft a reply for human approval"). The narrower version ships. The fantasy does not.

I run scoping as three sessions:

Workflow walkthrough. Someone who actually does the work shows me their screen for an hour. I record it. I take timestamps. The point is to find the moments where a human is doing pattern matching that an LLM can do, and to find the moments where they are doing judgment that an LLM should not do.
Data audit. Where does the input live? Where does the output need to go? What is the auth story? If the data is locked inside a SaaS product with no API and no export, that is the project, and we deal with it now, not in week six.
ROI sizing. Hours per month times burdened hourly cost, minus realistic infrastructure and maintenance. If the answer is under $20k/year in savings, I usually tell the client to wait. The build and the babysitting are not worth it at that scale yet.

Exit criterion: a one-page scope with a single first slice, a measurable success metric, and a named human owner on the client side. No owner, no project.

Phase 2: Proof of Concept (2 to 4 weeks)

The POC has one job: kill the project cheaply if it cannot work. I treat the POC as adversarial. I am trying to find the reason this will not ship, before we spend integration money on it.

Concretely, a POC for me looks like this:

A thin script or notebook, not a product. No UI unless the UI is the risk.
The real model we plan to use in production (Claude Sonnet, GPT-4 class, or a local Llama/Qwen via Ollama if data residency matters), not a cheaper proxy. Cheap models lie about feasibility.
A hand-curated set of 30 to 50 real examples from the client's actual data, with expected outputs written by a human who knows the domain.
A rough eval harness, even if it is just a spreadsheet with pass/fail and notes.

The POC answers four questions in order:

Question	What "no" means
Does the model produce the right shape of output reliably?	Schema issues, structured-output failures. Fixable.
Does it produce the right content on easy cases?	Capability gap. Sometimes fixable with retrieval or examples.
Does it handle the long tail without catastrophic failures?	The real risk. Often the project killer.
Can we detect when it is wrong?	If no, the project cannot ship to production. Full stop.

That last question is the one most people skip. An AI system you cannot evaluate is an AI system you cannot trust, and an AI system you cannot trust is a demo, not a product. I have walked away from POCs that worked 90% of the time because there was no signal to catch the 10%.

Exit criterion: measurable performance on the eval set that the client agrees is good enough to justify integration cost, plus a documented failure mode list.

Phase 3: Integration (3 to 8 weeks)

This is where most of the actual work lives, and where most of my time goes. The model is usually the easy part by now. The integration is what makes it real.

My default stack for production AI work:

Orchestration: simple, explicit code first. I reach for LangGraph or a hand-rolled state machine only when the workflow genuinely has branches and loops. Most "agents" are a sequential pipeline pretending to be agentic.
Storage: Postgres for everything, with pgvector when retrieval matters. Supabase if the client wants managed. I do not introduce a separate vector DB until pgvector measurably stops scaling, which is later than people think.
Retrieval: hybrid search (BM25 + dense) with reciprocal rank fusion. Pure semantic search loses on exact identifiers, SKUs, error codes, names. Pure keyword loses on paraphrase. RRF is the cheap fix.
Compute: AWS Lambda + EventBridge for scheduled and event-driven work, API Gateway when something needs to be called. Scales to zero, which matters for workloads that run hourly or in bursts.
Frontend (when needed): Next.js with server actions. Boring is good here.

Three integration details I now treat as non-negotiable:

1. Idempotency keys on everything

Any external action (send email, create ticket, post to CRM) gets an idempotency key derived from the input. Retries are inevitable, duplicate side effects are not.

def idempotency_key(workflow_id: str, input_hash: str, step: str) -> str:
    return f"{workflow_id}:{step}:{input_hash}"

2. A human-in-the-loop seam, even if unused

I always build the approval queue before I build the auto-send. Even if the client wants full automation eventually, shipping with human review for the first 2 to 4 weeks catches the failure modes the eval set missed. Turning approval off later is one config change.

3. Cost guardrails per workflow

Token budgets per execution, hard cutoffs, alerts at 50/80/100% of monthly budget. I have seen a single retry loop burn $400 in an hour. Never again.

Exit criterion: the system runs end to end on real production data, with logging, retries, idempotency, and a kill switch. Not perfect outputs yet, but the pipes are sound.

Phase 4: Evaluation (continuous, but formalized for 2 weeks)

Evaluation is not a phase you finish. It is a system you build once and keep running forever. But there is a discrete block of work to set it up, and that is what this phase is.

I build three layers of evaluation:

Offline eval set. The 30 to 50 examples from the POC, grown to 100 to 300, with expected outputs and a scoring rubric. Run on every prompt or model change. This is your regression test.
LLM-as-judge for open-ended outputs. For anything where there is no single correct answer (drafted emails, summaries, classifications with reasoning), I use a separate, stronger model with a calibrated rubric to score outputs. I have written about how to actually calibrate this so the judge does not just rubber-stamp. The short version: you score the judge against human labels on a held-out set, and you do not trust a judge you have not calibrated.
Production telemetry. Every run logs inputs, outputs, model version, prompt version, latency, tokens, cost, and the downstream outcome (was the draft email sent as-is, edited, or rejected?). That last signal is gold. It is the closest thing to ground truth you get in production.

The trap here is treating eval as a one-time gate. Models change. Prompts drift. Data shifts. The eval set has to be re-run on every change and the production telemetry has to feed back into growing the eval set. If a real production failure happens, it goes into the eval set the same day.

Exit criterion: the client can answer "is the system still working correctly?" without calling me.

Phase 5: Operations and Handoff (2 to 4 weeks)

This is the phase that separates a project that survives from one that dies six months in when something breaks and nobody knows where to look.

What I deliver in operations:

Runbook. A markdown doc with the top 10 things that can go wrong, how to detect them, and how to fix them. Real ones, from this system, not generic.
Dashboards. Usually a simple internal page or a Grafana board: success rate, cost per day, queue depth, latency P50/P95, model errors. The client looks at this weekly.
Alerts. Pager-worthy alerts on hard failures (pipeline stopped, cost spike, eval regression). Low-noise. If alerts cry wolf, they get muted, and then the real failure goes unnoticed.
Versioned prompts and configs. In git, with a changelog. Prompt changes are deploys, not Slack messages.
A maintenance retainer or a clean exit. Either I stay on for a defined number of hours per month, or I hand off to an internal team with a transition period. No silent fade-outs. Those end badly for both sides.

What I would do differently if I were starting over

A few opinions, after running this loop enough times:

Spend more on scoping, less on the POC. A bad scope makes a great POC useless. I have never regretted an extra week of scoping. I have regretted skipping it.
Pick the boring model. Use the strongest reliable model in your tier (Claude Sonnet or GPT-4 class) until you have a reason not to. Optimizing for cost too early picks fights you cannot win yet.
Build the eval before the agent. Sounds backwards. It is not. If you cannot define what good looks like, you cannot build toward it.
Treat the first 30 days in production as part of the build. Most of the real bugs surface there. Budget for it. Tell the client.
Say no more often. The projects I have turned down have, on average, been better decisions than the ones I took. Wrong-shaped projects do not get better with effort.

The shape of this process is not unique to my work. What is mine is the calibration: which phases I now know to invest in, which exit criteria I refuse to skip, and which mistakes I have made enough times to write them down. That last category is the actual deliverable when you hire someone like me, more than the code.

If you are scoping an AI implementation and want a second pair of eyes on it before you commit budget, I am happy to look at it. Reach out at lazar-milicevic.com/#contact, or browse the rest of the blog for more on evaluation, RAG, and getting agents into production.

What Is Process Automation? A Practical Guide

lamingsrb — Mon, 29 Jun 2026 06:11:44 +0000

What Is Process Automation? A Practical Guide

You're a founder paying yourself last while still spending three hours every Friday reconciling Stripe payouts against invoices in QuickBooks. You know it should be automated. You've heard "BPA," "RPA," and "workflow automation" used interchangeably by vendors who all want a $500/month seat. This guide cuts through that.

Process automation, in plain terms, is using software to do the repetitive work a human used to do — receiving a form, updating a record, sending a notification, moving a file, generating a report. The hard part isn't the tools. The hard part is choosing the right kind of automation for the work in front of you, and not over-engineering it.

What process automation actually means

Process automation is the use of software to execute a defined sequence of steps end-to-end, with little or no human input, against a measurable business outcome (an invoice paid, a lead routed, a ticket resolved). It is not "AI doing your job." It is a deterministic pipeline that triggers on an event, performs steps, and ends in a known state — sometimes with an AI step inside it.

Three things have to be true for something to be a candidate for automation:

The process is repeatable. You can describe the steps without saying "it depends" four times.
The inputs are structured or can be made structured. A PDF, a webhook, a form submission, a database row.
The outcome is verifiable. You can tell, programmatically, whether it worked.

If a process fails one of those three, automate the parts that pass and leave the rest to a human. That's where most automation projects die — trying to automate decisions that should still be human-reviewed.

BPA vs RPA vs workflow automation: what's the difference

These three terms get sold as if they mean the same thing. They don't. Picking the wrong category will cost you money and time.

Type	What it automates	Best for	Tools (examples)	Typical limit
Workflow automation	Steps inside or between SaaS apps via APIs	"When X happens in app A, do Y in app B"	Zapier, Make, n8n, Pipedream	Breaks when an API doesn't exist
RPA (Robotic Process Automation)	UI-level actions on apps that have no API	Legacy systems, desktop apps, scraping a portal	UiPath, Automation Anywhere, Power Automate Desktop	Brittle when UIs change
BPA (Business Process Automation)	End-to-end processes spanning multiple teams/systems	Onboarding, procurement, AR/AP, compliance flows	Camunda, ServiceNow, Pega	Heavier setup, needs process modeling
AI/agentic automation	Steps that require interpretation (unstructured input, decisions)	Email triage, document extraction, classification	LLM APIs orchestrated by code or platforms	Non-determinism, needs guardrails

A real automation usually mixes categories. Example: a webhook fires from Stripe (workflow), an LLM extracts line items from a vendor PDF (AI), the result is written to QuickBooks via API (workflow), and once a quarter an RPA bot exports a report from a legacy bank portal that still doesn't have an API in 2026.

If your process lives entirely inside modern SaaS, you almost never need RPA. If you're stuck reading from a desktop app or a portal with no integration, RPA is sometimes the only door. If you're orchestrating something across finance, ops, and sales with approvals at each stage, that's BPA territory.

Where to start: pick a process worth automating

The single biggest mistake I see solo founders and ops people make is automating something interesting instead of something expensive. Use this scoring approach on a sticky note before you write a line of code:

Frequency: How often does this run? (per day / week / month)
Minutes per run: How long does the human take?
Error cost: What happens if a step is missed? ($, churn, compliance)
Stability: How often does the process change?

Multiply frequency × minutes × 52 to get yearly hours. If that's under 20 hours/year, leave it alone unless the error cost is high. If it's over 100 hours/year and stable, automate it this month.

A simple Python sketch for prioritizing a backlog:

processes = [
    {"name": "invoice_reconciliation", "per_week": 1, "minutes": 180, "stability": 0.9, "error_cost": 3},
    {"name": "lead_routing",          "per_week": 25, "minutes": 4,   "stability": 0.95, "error_cost": 4},
    {"name": "weekly_report",         "per_week": 1, "minutes": 45,  "stability": 0.7, "error_cost": 1},
    {"name": "customer_offboarding",  "per_week": 2, "minutes": 25,  "stability": 0.8, "error_cost": 5},
]

for p in processes:
    yearly_hours = (p["per_week"] * p["minutes"] * 52) / 60
    score = yearly_hours * p["stability"] * p["error_cost"]
    print(f"{p['name']:<25} hours/yr={yearly_hours:6.1f}  score={score:7.1f}")

Sort by score. Automate top of the list. Don't get fancy yet.

Four examples that are worth building

These are the patterns I see ship for solopreneurs and small teams. They're not exciting, which is exactly why they pay back.

1. Inbox triage and reply drafting

Trigger: a new email lands. Steps: classify (sales lead, support, vendor, noise), extract key fields (company, ask, urgency), draft a reply in your voice, attach it as a Gmail draft. Human still hits send.

trigger: gmail.new_message
steps:
  - classify:
      categories: [lead, support, vendor, billing, noise]
      model: claude-sonnet
  - if: category == "lead"
    extract:
      fields: [company, role, ask, budget_signal]
    draft_reply:
      tone: "direct, no fluff, mention pricing page"
      save_as: gmail.draft
  - if: category == "support"
    create_ticket: linear
    draft_reply:
      template: support_ack

The "human sends" rule is doing a lot of work here. It catches LLM mistakes and keeps your reputation intact while you tune the prompts.

2. Lead-to-CRM with enrichment

Form submission → enrich company from domain → score → route to the right rep → notify in Slack with a one-line summary. This is the highest-ROI automation for any business with inbound leads, because every minute a lead waits costs conversion. The classic MIT/InsideSales study found odds of qualifying a lead drop sharply after the first hour — even if the exact multiplier varies by industry, the direction is settled.

3. AR/AP: invoices in, invoices out

Vendor invoice arrives by email → PDF parsed → line items extracted → matched against PO → written to accounting system → flagged for approval if over a threshold. On the AR side: recurring invoices generated from contract data, sent on schedule, with dunning emails staged for overdue accounts.

This is the kind of work that quietly eats 4–8 hours a week in a 5-person company. It's also boring enough that nobody volunteers to own it, which is why it stays manual for years.

4. Reporting that builds itself

Pull metrics from Stripe, your product DB, and your ad platforms every Monday at 7am. Compute deltas. Generate a markdown summary. Post it in Slack and email it to yourself. No more "I should really look at the dashboard."

# weekly_report.py — runs via cron Monday 07:00
import os, datetime as dt
from clients import stripe, postgres, slack, llm

week = dt.date.today() - dt.timedelta(days=7)

mrr = stripe.mrr_snapshot()
churn = stripe.churn(since=week)
signups = postgres.count("users", since=week)
active = postgres.dau_avg(since=week)

prompt = f"""Write a 5-bullet summary for a solo founder.
Numbers: MRR={mrr}, churn={churn}, signups={signups}, DAU avg={active}.
Compare to prior week. Flag anomalies. No fluff."""

summary = llm.generate(prompt)
slack.post("#metrics", summary)

The architecture that doesn't fall over

After you've built five or six of these, you'll notice the same architecture under all of them. Build with it from day one.

┌──────────┐   ┌──────────────┐   ┌─────────────┐   ┌────────┐
│ Trigger  │──▶│ Orchestrator │──▶│ Steps       │──▶│ Sink   │
│ webhook/ │   │ (queue +     │   │ (API calls, │   │ (DB,   │
│ cron/    │   │  retries +   │   │  LLM, RPA   │   │ Slack, │
│ email    │   │  logs)       │   │  bot)       │   │ CRM)   │
└──────────┘   └──────────────┘   └─────────────┘   └────────┘
                      │
                      ▼
                 ┌─────────┐
                 │ Audit   │  every run, every step, every input/output
                 │ log     │
                 └─────────┘

Non-negotiable pieces:

Idempotency keys. If a webhook fires twice, you should not invoice the customer twice. Use the event ID as a dedup key.
Retries with backoff. APIs fail. Plan for it. 3 retries, exponential backoff, dead-letter queue for the rest.
Audit logs. Every step writes input, output, duration, and status. When something goes wrong at 2am on a Saturday, you need to be able to answer "what happened?" without re-running anything.
A kill switch. One env var, one config flag, one button. When the LLM hallucinates a refund, you want to stop the pipeline in under a minute.
Human-in-the-loop on anything irreversible. Sending money, deleting data, emailing a customer — require approval until you've watched it run clean for weeks.

As Werner Vogels put it bluntly: "everything fails, all the time." Build like you believe him.

Common mistakes that kill automation projects

I've watched these patterns burn through budgets at companies from 2-person shops to 200-person ops teams.

Automating before mapping. If you can't write the process on a whiteboard in 10 minutes, you can't automate it. Write it first. Most "automation problems" are actually undocumented-process problems.

Optimizing rare work. That 90-minute task you do once a quarter is not worth a sprint. The 4-minute task you do 30 times a day is.

One giant flow. A 47-step Zap is a debugging nightmare. Break processes into small pipelines that pass state to each other through a queue or a database. Each pipeline does one thing and can be tested in isolation.

No observability. If your only signal that automation failed is a customer complaining, you've built a liability. Log every run. Alert on failures. Track success rate weekly.

LLM where regex would do. If you're parsing a fixed-format invoice number, don't burn tokens on it. Use the LLM for the messy parts (unstructured email bodies, varying PDF layouts) and deterministic code for everything else. This also keeps your costs predictable.

Believing the demo. Every no-code tool looks great in a 90-second video. Build a real flow with your real data before you commit to a $200/month plan. Most pilots fail not because the tool can't do it, but because the team's processes weren't ready — a pattern Gartner and others have flagged in RPA program research for years.

A 30-day plan to ship your first three automations

You don't need a transformation initiative. You need three weeks and a coffee budget.

Week 1 — Map and pick.

List every repetitive task you or your team did this week.
Score them with the formula above.
Pick the top three. Write each process down step by step. Note the inputs, outputs, and edge cases.

Week 2 — Build the smallest version.

Build automation #1 end-to-end, even if half of it is just "post a message in Slack with the data for a human to act on."
Run it manually-triggered for a few days. Watch where it breaks.
Add retries, logging, and a kill switch.

Week 3 — Harden and expand.

Move automation #1 to its real trigger (webhook, cron, email).
Build automation #2 using the same scaffolding (queue, logs, retries).
Add a weekly review: success rate, failures, time saved.

Week 4 — Measure and decide what's next.

Calculate actual hours saved per week.
Pick automation #4 from your backlog, or harden what you have.

If after 30 days you haven't saved at least 5 hours/week, you picked the wrong processes. Go back to the scoring step.

How BizFlowAI approaches this

We build process automations for solopreneurs and small teams — not platforms, not seats, not a course. The pattern is consistent across every engagement: map the process, score it honestly, build the smallest version that ships, add the boring infrastructure (idempotency, retries, audit logs, kill switches), then layer AI only where deterministic code can't reach. Most of our wins come from inbox triage, lead routing, AR/AP, and reporting pipelines — the unglamorous work that adds up to a day a week back.

We default to standard tools (n8n, Python workers, Postgres, the major LLM APIs) so clients aren't locked into a proprietary runtime they can't maintain. If you want to see what this looks like applied to your actual workflows, that's the conversation we're best at.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.

Make.com vs n8n: I Built the Same AI Agent in Both

lamingsrb — Mon, 29 Jun 2026 06:11:40 +0000

Make.com vs n8n: I Built the Same AI Agent in Both

A client needed a lead-routing agent. Gmail in, AI classifies by industry and urgency, hot leads to a sales rep on Telegram, cold ones to the CRM with a tagged follow-up. Nine steps. I built it three times — Make, n8n, then raw code — because I picked wrong the first two times. Here is the decision rule I wish someone had given me.

The exact workflow I rebuilt three times

Before I argue about tooling, here is the spec, because vague comparisons are useless. Nine nodes total: Gmail trigger → extract sender + body → OpenAI classification (industry, urgency 1-5) → branch on urgency → Telegram message to sales rep (hot) → CRM upsert with tag (cold) → follow-up sequence enqueue → log to Postgres → error catch.

That is the surface. What killed me was the second-order work:

Retry logic when OpenAI returns a 429 or the CRM webhook 502s
A custom HTTP call to an invoicing API with a non-standard signed auth header
Volume: roughly 40,000 executions/month once you count retries and polling

Every comparison post online benchmarks the happy path. Real client work is the unhappy path. That is where Make and n8n diverge hard.

Where Make actually beats n8n (I will say it out loud)

For a workflow with under ten steps, no retry logic, no custom auth, and under 1,000 ops/month, Make is the better choice. The UX is genuinely cleaner and you will ship faster. I built the happy-path version of this agent in Make in eight minutes flat.

Specifically:

The Gmail trigger authenticated in two clicks. n8n's OAuth flow took me about four minutes the first time because you have to register a Google Cloud project.
The OpenAI module has a model dropdown and a structured-output toggle. No JSON wrangling.
The left-to-right scenario canvas reads like a sentence. When I screen-share with non-technical clients, they understand a Make scenario without me explaining.
Built-in modules for Telegram, HubSpot, Pipedrive, Notion — all pre-authed with sensible defaults.

If your automation looks like "trigger → enrich → send" and runs a few hundred times a month, stop reading and use Make. Your time is worth more than the $9–29/month subscription. Most YouTubers who tell you "always use n8n" are selling a self-hosting course.

Where Make falls off a cliff: retries, custom auth, and ops pricing

Make's pricing model is per-operation, and every error handler, every router branch, every iteration consumes ops. The moment you add real production logic — retries, dead-letter queues, idempotency checks — the ops counter accelerates faster than your feature set.

Here is what happened on this specific client:

The client asked for retry logic. If the OpenAI call fails or the CRM is down, the lead can't vanish.
Make's native retry is limited to a few attempts on a small subset of modules. For OpenAI failures, I had to build error handlers that route to a data store.
Each error handler branch costs operations against the monthly quota.
To replay failed runs, I built a separate scenario that polls the data store every five minutes. That scenario alone burned ~8,600 ops/month doing nothing useful.

At around 1,200 real-traffic ops/month plus the polling overhead, the workflow needed the Pro tier. Make's UX is great until you're paying for the privilege of working around its limitations.

Then came the invoicing API. Non-standard auth header, HMAC-signed payload. In Make, the HTTP module on the lower tiers cannot generate the signature inline — you'd need a separate scenario, a webhook, and a data store hop. Ugly. In n8n:

// n8n Function node — 6 lines
const crypto = require('crypto');
const ts = Date.now().toString();
const payload = JSON.stringify($json.body);
const sig = crypto.createHmac('sha256', $env.API_SECRET)
                  .update(ts + payload).digest('hex');
return [{ json: { ...$json, headers: { 'X-Signature': sig, 'X-Timestamp': ts } } }];

Done. That single capability — drop into JavaScript or Python when the visual nodes can't express what you need — is the deciding factor for any non-trivial workflow.

The cost math nobody puts in a comparison post

This is where solopreneurs actually feel the choice. Real numbers from this client, audited from billing dashboards:

Metric	Make.com (Pro)	n8n self-hosted	n8n Cloud (Pro)
Executions/month (with retries + polling)	~40,000 ops	~40,000	~40,000
Monthly cost	$47	$0 marginal*	$50
Annual cost	$564	~$0	$600
Custom code support	Paid tier, limited	Native (JS + Python)	Native
Native retry/error workflows	Limited	Yes	Yes
Self-host option	No	Yes	No

*Marginal cost. My home server (mini PC, 32GB RAM, Ubuntu under WSL on one box, bare-metal Debian on another) runs eight other things. Amortized hardware is roughly $4/month of electricity.

Over a year, that's $564 saved on one client workflow. Run five client workflows and you've paid for a developer-grade server. Run ten and you've funded a year of API credits.

The Make pricing trap is not the headline number. It's that every architectural improvement you make — better error handling, idempotency, observability — costs you ops. n8n charges you once for the engine, then your improvements are free.

Self-hosting n8n in 12 minutes (the actual setup)

If you're going the n8n route, here is the Docker Compose I use on the client server. Postgres for persistence, n8n behind a Caddy reverse proxy for TLS, encrypted credentials at rest.

# docker-compose.yml
services:
  postgres:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: n8n
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: n8n
    volumes:
      - ./pgdata:/var/lib/postgresql/data

  n8n:
    image: n8nio/n8n:latest
    restart: unless-stopped
    environment:
      DB_TYPE: postgresdb
      DB_POSTGRESDB_HOST: postgres
      DB_POSTGRESDB_USER: n8n
      DB_POSTGRESDB_PASSWORD: ${DB_PASSWORD}
      N8N_ENCRYPTION_KEY: ${ENCRYPTION_KEY}
      N8N_HOST: automation.yourdomain.com
      WEBHOOK_URL: https://automation.yourdomain.com/
      N8N_PROTOCOL: https
      EXECUTIONS_DATA_PRUNE: "true"
      EXECUTIONS_DATA_MAX_AGE: 168  # 7 days
    ports:
      - "127.0.0.1:5678:5678"
    depends_on:
      - postgres
    volumes:
      - ./n8n-data:/home/node/.n8n

# .env
DB_PASSWORD=$(openssl rand -hex 24)
ENCRYPTION_KEY=$(openssl rand -hex 32)

docker compose up -d

A few things I learned the hard way running this in production:

Always set EXECUTIONS_DATA_PRUNE. Without it, the Postgres db grew 400MB/week on this client's workload. After a year the n8n UI crawled.
Use the queue execution mode (EXECUTIONS_MODE=queue with Redis) once you're past 10k executions/day. Single-process mode will block on a slow webhook.
Back up n8n-data and the Postgres dump nightly. Credentials are AES-encrypted with N8N_ENCRYPTION_KEY — lose that key, lose every saved credential.
Put it behind Cloudflare Tunnel or Tailscale, not a raw exposed port. n8n's UI authentication is fine but webhooks are public by design.

Twelve minutes from docker compose up to a working webhook in production, if your DNS and TLS are already set up.

The decision rule I give every client

Skip the holy war. Here is the actual rule:

Fewer than 10 steps, no retries, no custom auth, under 1,000 ops/month → Make. Ship in an hour. Worth the $9–29.
Branching, retries, custom APIs, or any real volume → n8n from day one. Self-hosted if you have a server, cloud if you don't.
Workflow needs version control, code review, CI/CD, or multi-environment deploys → skip both. Write it in Python with a task queue. The visual tools become friction.

The trap is building in Make first and migrating later. I have done that rebuild three times. It is always two full days I don't get back, because Make's data shapes don't map cleanly to n8n's, and your error handlers are scenario-specific.

The Make-vs-n8n debate is the wrong question. The right question: does my workflow need logic Make can't express cleanly? If yes, n8n today. If no, Make today. Pick once, build once.

Quick gut check before you start building

Will I ever call an API with a custom signed header? → n8n
Do I need to retry failed runs without burning ops? → n8n
Will I run more than 5,000 executions a month? → n8n
Is the workflow under 10 steps and will stay there? → Make
Will a non-technical client need to read the canvas? → Make

Why bizflowai.io helps with this

For client workflows we run through bizflowai.io, we default to n8n self-hosted on a managed server because the retry/custom-auth/volume math almost always wins for lead-routing, invoicing, and CRM-sync automations. We still build in Make when the workflow is genuinely simple and the client wants to own the canvas themselves. The honest part of the job is telling the client which tool fits before the build, not after — that one conversation saves the two-day rebuild every time.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

How Shopify Built a Model-Agnostic AI Stack

lamingsrb — Thu, 25 Jun 2026 10:46:11 +0000

How Shopify Built a Model-Agnostic AI Stack

You're three months into building an AI feature. It works. Customers love it. Then your model provider emails: "We're deprecating this version in 60 days." Or worse — the API just starts returning 500s on a Tuesday morning and your support inbox explodes.

This is the reality of building on LLMs in 2026. Shopify hit this wall early, and instead of betting on one provider, they built an internal proxy that treats models as interchangeable parts. Here's how it works, why it matters, and how to build the same pattern into your stack — even if you're a team of one.

The problem: every LLM is a moving target

Models get deprecated. Providers change pricing without warning. Rate limits shift. A vendor you trusted last quarter quietly degrades quality on the model you fine-tuned your prompts around. If your production code has client = anthropic.Anthropic() hard-coded at the top of every file, you have a fragility problem disguised as a dependency.

Shopify's engineering team has talked openly about this. In public conference talks and engineering blog posts, their AI infrastructure team has described a pattern where every internal AI call routes through a single proxy layer. When Anthropic ships a new Claude version, when OpenAI deprecates an older endpoint, when a regional outage hits one provider, the proxy reroutes traffic. Application code never changes.

This is not exotic infrastructure. It's the same pattern that made cloud-native apps survive AWS region outages: abstract the dependency, route around failure. The novelty is applying it to LLMs, where the "service" you're calling has unstable APIs, unstable pricing, and a habit of disappearing.

What a model-agnostic proxy actually does

An LLM proxy is a thin service that sits between your application code and any model provider (Anthropic, OpenAI, Google, Mistral, a local Ollama instance, whatever). Your app calls one endpoint with a generic request format. The proxy decides:

Which provider gets the request right now
What to do if that provider returns an error or times out
Whether to retry on the same model or fall back to a different one
How to normalize the response so your app gets a consistent shape

The minimum viable version is maybe 200 lines of Python. The mature version — what Shopify and companies like Anthropic-customer Notion run — adds caching, observability, cost tracking per team, prompt versioning, and policy enforcement.

Here's the core abstraction:

# What your app code looks like with a proxy
response = llm_proxy.complete(
    task="customer_email_classification",
    prompt=user_input,
    max_tokens=500,
)

Notice what's missing: no model name, no provider SDK, no API key handling. The proxy maps task to a routing policy. If customer_email_classification is currently configured to use Claude Sonnet with GPT-4o as fallback, that's a config change, not a code change.

Building the routing layer: a minimal proxy in Python

Here's a stripped-down version of the pattern. This isn't production-ready — it's the skeleton to understand the shape.

import time
from typing import Callable
from dataclasses import dataclass

@dataclass
class ModelConfig:
    provider: str
    model_id: str
    timeout_s: int
    call: Callable

class LLMProxy:
    def __init__(self, routing_table: dict[str, list[ModelConfig]]):
        # routing_table maps task names to ordered fallback chains
        self.routes = routing_table

    def complete(self, task: str, prompt: str, max_tokens: int = 500):
        chain = self.routes.get(task)
        if not chain:
            raise ValueError(f"No route for task '{task}'")

        last_error = None
        for model in chain:
            try:
                start = time.time()
                result = model.call(prompt, max_tokens, model.timeout_s)
                self._log(task, model, time.time() - start, success=True)
                return result
            except Exception as e:
                last_error = e
                self._log(task, model, None, success=False, error=str(e))
                continue

        raise RuntimeError(f"All models failed for {task}: {last_error}")

    def _log(self, task, model, latency, success, error=None):
        # Ship to your logging/metrics pipeline
        print({
            "task": task,
            "provider": model.provider,
            "model": model.model_id,
            "latency_s": latency,
            "success": success,
            "error": error,
        })

And the routing config:

routing = {
    "customer_email_classification": [
        ModelConfig("anthropic", "claude-sonnet-4", 10, call_claude),
        ModelConfig("openai", "gpt-4o", 10, call_openai),
        ModelConfig("local", "llama-3.1-70b", 30, call_ollama),
    ],
    "long_document_summary": [
        ModelConfig("anthropic", "claude-opus-4", 60, call_claude),
        ModelConfig("openai", "gpt-4-turbo", 60, call_openai),
    ],
}

proxy = LLMProxy(routing)

When Claude Sonnet 4 gets deprecated, you change one line in the routing table. When a provider starts returning 529 overloaded errors at 3pm every day, the second model in the chain catches the requests automatically. Your application code doesn't know and doesn't care.

What "failover" really has to handle

Naive failover — "if request fails, try the next model" — is not enough. Real production failover handles at least these cases:

Failure mode	Naive retry	Proper handling
5xx from provider	Retry once on same model	Fast-fail after 1 retry, switch provider
Rate limit (429)	Retry with backoff	Switch provider immediately, don't wait
Timeout	Retry	Switch provider, log the slow one
Bad JSON / schema violation	Crash	Reroute to a model better at structured output
Quality degradation	Invisible	Periodic eval suite catches regressions
Model deprecated	Production breaks	Routing table flagged before sunset date

The quality dimension is the hard one. A provider can return a 200 OK with technically valid output that is subtly worse than what you got last week. Shopify and similar shops handle this with offline eval suites: a fixed set of representative prompts is rerun against every model in the chain on a schedule, and outputs are scored (either by another model or by humans on edge cases). When a model's score drops below threshold, it gets demoted in the routing table before users notice.

You don't need this on day one. You do need a way to swap models in 5 minutes when something breaks.

Normalizing requests and responses across providers

Anthropic's API and OpenAI's API are similar but not identical. System prompts go in different places. Tool-calling schemas differ. Streaming chunk formats differ. Token usage is reported differently. If your app code handles any of this directly, the proxy isn't actually decoupling anything.

The fix is a normalization layer inside the proxy. Define one internal request shape:

{
  "task": "extract_invoice_fields",
  "system": "You extract structured data from invoices.",
  "messages": [
    {"role": "user", "content": "..."}
  ],
  "tools": [
    {
      "name": "save_invoice",
      "schema": { "...": "..." }
    }
  ],
  "max_tokens": 1000,
  "temperature": 0
}

And one internal response shape:

{
  "provider": "anthropic",
  "model": "claude-sonnet-4",
  "text": "...",
  "tool_calls": [],
  "usage": {
    "input_tokens": 1240,
    "output_tokens": 312
  },
  "latency_ms": 1842
}

Inside the proxy, you have adapters per provider that translate to/from these shapes. This is unglamorous code, but it's the work that buys you the freedom to swap providers without touching application logic. The LiteLLM and OpenRouter open-source projects already do most of this if you don't want to write adapters yourself — both are worth evaluating before you build your own.

Cost and observability: the underrated wins

Once everything routes through one place, two things become trivial that were previously impossible:

Per-team, per-feature cost tracking. Every call carries a task label. Aggregate token usage by task and you know exactly which feature is costing what. Most teams discover that 80% of their LLM spend comes from 2-3 features they didn't expect. According to Anthropic's published pricing pages and OpenAI's pricing pages (always check the current ones — they change), input vs. output token costs can differ by 5x, and switching one high-volume feature from a premium model to a smaller one can cut a monthly bill meaningfully.

A single audit log. Every prompt, every response, every model used. When a customer reports a bad output from your AI feature three weeks ago, you can find the exact request and response. When legal asks "do we ever send customer PII to OpenAI," you can answer with data instead of a shrug.

This second one matters more for compliance-sensitive industries (healthcare, finance, anything touching personal data under frameworks like HIPAA or GDPR). A proxy gives you one chokepoint for redaction, logging, and policy enforcement instead of N integration points across your codebase.

Caching: the quietly massive cost lever

Identical prompts get identical-ish responses. A proxy is the natural place to cache. Two layers worth implementing:

Exact-match cache. Hash the full request (system + messages + temperature). If you've seen it in the last N hours, return the cached response. Useful for deterministic prompts at temperature 0.
Semantic cache. Embed the user prompt, look up similar prior prompts, and reuse the response if similarity is above a threshold. Useful for FAQ-style use cases where users ask the same thing five ways.

Anthropic also offers prompt caching at the API level — long system prompts and tool definitions can be cached server-side for a fraction of the input token cost. Your proxy should pass the right cache-control headers to take advantage of this. Check Anthropic's current prompt caching documentation for the exact discount and TTL.

Semantic caching is risky for anything where wrong answers matter (medical, legal, financial). For low-stakes use cases — customer support FAQ, internal docs Q&A — it can cut LLM spend significantly with minimal quality loss.

A practical adoption path for small teams

You are not Shopify. You don't need a dedicated AI platform team. Here's how to get most of the benefit in a week of work:

Pick a wrapper. LiteLLM gives you a drop-in OpenAI-compatible interface that proxies to ~100 providers with retry and fallback built in. OpenRouter does similar with a hosted model. For most solo operators and small teams, one of these gets you 80% of the value with zero infrastructure.
Centralize the calls. Wrap every LLM call in your app behind one internal function. No openai.chat.completions.create(...) scattered across 14 files.
Define your tasks. Name each LLM use case (classify_lead, summarize_ticket, draft_reply). Map each to a primary model and one fallback.
Log everything. Task name, provider, model, latency, token counts, success/failure. A simple Postgres table works. You'll thank yourself in three months.
Set up a deprecation alert. Subscribe to your providers' changelog feeds or RSS. When a model you use shows up on a deprecation list, you have weeks to swap it in the routing table — not hours after production breaks.
Eval suite, eventually. Pick 20-50 representative prompts per task. Run them against each model in your chain on a monthly cadence. Score outputs (model-judged is fine to start). When scores drift, investigate.

Steps 1-5 are achievable in a week for a solo developer. Step 6 is what separates teams that get bitten by silent quality regressions from teams that don't.

How BizFlowAI approaches this

This proxy-and-failover pattern is the default for every client agent we build. When we ship a customer-support automation or a lead-qualification agent, the LLM calls go through a routing layer with primary + fallback providers, structured logging, and a config-driven model map. When Anthropic or OpenAI deprecates a model — and they will — the swap is a config change reviewed in minutes, not an incident.

If you're running AI features in production and the thought of your primary provider going down for an afternoon makes you nervous, that's a fixable problem. Book a call and we'll map out where your stack is fragile and what the smallest version of this proxy pattern looks like for your specific workloads.

The takeaway

Treating LLMs as commodity inputs — not load-bearing dependencies — is the cheapest insurance you can buy against a market where models get deprecated, repriced, and rate-limited on someone else's schedule. The proxy pattern Shopify uses is not exotic. It's the same abstraction that made cloud apps survive their cloud providers' worst days, applied to a new dependency that happens to be more unstable than EC2 ever was.

You don't need a platform team to do this. You need one wrapper, one routing table, and a logging table. Build it once. Swap models forever.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.

Token Sprawl Is Real. Here's How to Cap It.

lamingsrb — Thu, 25 Jun 2026 10:46:03 +0000

Token Sprawl Is Real. Here's How to Cap It.

Your engineering team's Anthropic bill jumped 4x last month. Nobody shipped a new product. Nobody onboarded a major customer. Someone just discovered they can pipe entire log files into Claude Opus to find a typo, and word got around. This is the conversation happening in finance meetings right now, and the fix isn't "ban AI" — it's plumbing.

The tokenmaxxing phase — where every developer was encouraged to throw the biggest model at every task — was brief. We're now in the token rationing phase. CFOs are looking at usage dashboards that didn't exist 12 months ago and asking why a single engineer ran $1,800 of inference in a week to refactor a 200-line file. The companies handling this well aren't rationing access. They're routing it.

Why your AI spend exploded in 90 days

The pattern is consistent across the teams I audit: spend doesn't grow linearly with headcount or output. It grows with habit formation. Once someone gets comfortable pasting a full repo into a frontier model, they stop pasting small chunks into smaller models. The unit cost per task doubles, then doubles again, while the perceived value stays flat.

Three behaviors drive most of the bleed:

Frontier-by-default. Every task — summarizing a Slack thread, writing a regex, naming a variable — gets routed to the most expensive model available. Anthropic's own pricing pages show a roughly 5x cost gap between Haiku-class and Opus-class models per million tokens, and that gap matters when you multiply by 50 engineers making 200 calls a day.
No context discipline. Engineers paste 80k tokens of context to answer a question that needs 2k. Long-context windows trained people to stop curating inputs.
No caching. The same system prompt — your coding standards, your architecture doc, your style guide — gets re-billed on every single request because nobody flipped on prompt caching.

According to a 2025 a16z analysis of enterprise AI spend, the median company underestimated its first-year LLM cost by roughly 2-3x, primarily because individual developer usage scaled faster than procurement modeled. The number isn't the point. The pattern is: bottom-up adoption + top-down pricing = surprise.

The four categories of token waste

Before you cut anything, sort your usage into these four buckets. The fix is different for each.

Category	What it looks like	Typical % of spend	Fix
Model mismatch	Opus answering "format this JSON"	30-50%	Routing
Context bloat	Pasting whole repos for one-file edits	15-25%	Retrieval + scoping
Repeat context	Same system prompt billed 10k times/day	10-20%	Prompt caching
Failed agents	Loops that retry 8 times before giving up	5-15%	Budgets + guardrails

Most teams attack this with a Slack announcement: "Please use Haiku for simple tasks." That fails within a week because individual engineers don't know what counts as simple, and the cost of guessing wrong (a worse answer) feels personal while the savings feel abstract. You need the routing to happen below the engineer, not above them.

Model routing: the highest-leverage fix

A router sits between your application (or your developers' tooling) and the model APIs. It looks at the request, decides which model is appropriate, and forwards it. Done right, it cuts spend 40-70% with no quality drop on the easy 80% of tasks, and you keep frontier models for the hard 20%.

Here's a minimal router that classifies by task complexity:

import anthropic

client = anthropic.Anthropic()

# Cheap classifier decides routing
def classify_complexity(prompt: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=10,
        messages=[{
            "role": "user",
            "content": f"Classify this task as SIMPLE, MEDIUM, or HARD. "
                       f"Reply with one word only.\n\nTask: {prompt[:500]}"
        }],
    )
    return resp.content[0].text.strip().upper()

MODEL_MAP = {
    "SIMPLE": "claude-haiku-4-5",
    "MEDIUM": "claude-sonnet-4-5",
    "HARD":   "claude-opus-4-5",
}

def route(prompt: str, **kwargs):
    tier = classify_complexity(prompt)
    model = MODEL_MAP.get(tier, "claude-sonnet-4-5")
    return client.messages.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        **kwargs,
    )

This is intentionally crude. A production router will use signals like: input token count, presence of code, whether the user is a paid customer, whether the request is part of an agent loop, and historical accuracy on similar tasks. Tools like OpenRouter, Martian, and Portkey solve the boring parts (failover, observability) so you can focus on the routing rules.

A few rules of thumb that have held up across my client projects:

Default to the mid-tier model, not the frontier one. Use the cheap model as the classifier, not the worker.
Route by intent, not by user. "All engineers use Sonnet" is a worse policy than "all single-file edits use Sonnet."
Log the routing decision. You need to A/B the classifier itself when accuracy complaints come in.

Prompt caching is free money

If you're sending the same system prompt with every request — your coding standards, your tool definitions, your few-shot examples — you're paying full price for tokens the provider has already processed for you minutes ago. Anthropic's prompt caching reduces the cost of cached input tokens to roughly 10% of the standard rate, with cache writes priced at about 1.25x. The break-even is usually 2-3 hits.

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_STATIC_SYSTEM_PROMPT,  # 8000 tokens of standards
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_question}],
)

What to cache:

System prompts over ~1k tokens
Tool definitions for agents
Long reference documents queried repeatedly (style guides, schemas, API specs)
Conversation history in long-running sessions

What not to cache: anything that changes per user, retrieval results, or short prompts where the cache write overhead eats the savings.

Right-sizing agents: the 5x problem

Agents are where token bills go nonlinear. A poorly-built agent doesn't cost 2x a single call — it costs 10-50x, because each step adds the full conversation history back into context. A 12-step agent with growing history can re-bill the same opening tokens twelve times.

Three controls that actually work:

1. Hard budget per task. Every agent invocation gets a token ceiling. When it's hit, the agent must return whatever it has or escalate to a human.

class TokenBudget:
    def __init__(self, max_tokens: int):
        self.max = max_tokens
        self.used = 0

    def charge(self, n: int):
        self.used += n
        if self.used > self.max:
            raise BudgetExceeded(
                f"Agent exceeded budget: {self.used}/{self.max}"
            )

# In your agent loop
budget = TokenBudget(max_tokens=50_000)
for step in range(MAX_STEPS):
    resp = client.messages.create(...)
    budget.charge(resp.usage.input_tokens + resp.usage.output_tokens)

2. Context window pruning. After every N steps, summarize the conversation back down. Yes, you lose some fidelity. You also stop paying for tokens about a tool call you made eight steps ago.

3. Tool-call short-circuits. If the agent calls the same tool with the same arguments twice in a row, that's a loop. Kill it. Most agent frameworks now support this natively but very few teams configure it.

A practical FinOps stack for LLMs

The companies handling token sprawl well treat it like any other infra cost: meter it, attribute it, alert on it. Here's the minimum viable stack:

Layer	Purpose	Open-source options	Commercial
Gateway	Single endpoint, key management, routing	LiteLLM, Helicone	Portkey, OpenRouter
Observability	Per-request logs, cost, latency	Langfuse, Phoenix	Datadog LLM, Helicone
Caching	Semantic + exact match	GPTCache, Redis	Provider-native (Anthropic, OpenAI)
Evals	Catch quality regressions when routing	Promptfoo, DeepEval	Braintrust, Patronus

You don't need all of this on day one. The order that has worked for me: gateway first (so you have one chokepoint), observability second (so you know what's actually happening), caching third (the easy wins), evals fourth (so you can route more aggressively without fear).

A 2025 OpenAI cookbook recommendation puts it bluntly: "Optimize for the cheapest model that meets your quality bar, not the smartest model you have access to." That's the whole game.

Policies that don't make engineers hate you

Token rationing fails the moment it feels like surveillance. Here's what works in practice:

Budget by team, not by person. A team with a $5k/month inference budget will self-police. A person with a $200/month budget will hide usage on personal keys.
Surface cost in the IDE. If an engineer sees "this prompt will cost ~$0.40" before sending, behavior shifts immediately. Cursor, Continue, and Claude Code all support this now.
Make the cheap path the default path. If using Haiku requires a config change and Opus is the default, you've lost. Flip it.
Publish a "when to use what" doc. Three lines, not three pages: "Use Haiku for: extraction, formatting, classification. Sonnet for: code, drafts, reasoning. Opus for: anything where a wrong answer costs more than $50."
Show the savings publicly. "Routing saved us $14k in May" goes in the engineering all-hands. People want to be on the team that ships the optimization.

What doesn't work: blanket bans, per-request approval workflows, weekly "AI cost" shame emails. Those just push usage to personal credit cards and Anthropic Console accounts you don't control.

How BizFlowAI approaches this

Most of our client engagements that start as "build us an agent" end up including a token spend audit, because the bill is often the reason they called. We typically find a 40-60% reduction available within two weeks: model routing on the obvious traffic, prompt caching on the long system prompts, and tightening the agent loops that were spinning. The work is unglamorous — instrumenting the gateway, classifying historical traffic, swapping models behind a feature flag — but it's the difference between an AI line item that scales with revenue and one that scales with team morale.

If your monthly AI bill has more than doubled in the last quarter and you can't draw a straight line from that increase to revenue or hours saved, that's the audit. We map every call to a category, route the easy 80% to right-sized models, set hard budgets on agent loops, and leave you with a dashboard that shows cost per task instead of cost per month. Book a discovery call and we'll walk through your actual usage before recommending anything.

What to do this week

If you read this far, here's the order I'd attack it in starting Monday:

Pull the last 30 days of usage from your provider's console. Sort by user and by endpoint. The Pareto distribution will be brutal — usually 3-5 people account for 60%+ of spend.
Enable prompt caching on your largest system prompts. This is the cheapest 20% win available.
Put a gateway in front of everything. Even if you don't route yet, having one chokepoint means you can route tomorrow.
Set per-team budgets, not per-person. Make them generous enough that nobody hits them in week one — you want data, not friction.
Add cost-per-task to your engineering dashboards alongside latency and error rate. Make it visible.

The companies that handled cloud cost sprawl in 2014-2018 are the same ones handling token sprawl in 2026: the ones who built FinOps muscle early. You don't need to be Netflix about it. You do need a gateway, a routing rule, and someone who looks at the dashboard once a week.

The era of "just throw the biggest model at it" is ending not because the models got worse, but because the bills got real. The teams that adapt fastest will be the ones that treat inference like compute — measured, attributed, and right-sized — instead of like magic.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.