DEV Community: lamingsrb

Agentic Workflows That Survive Real Inputs

lamingsrb — Thu, 30 Jul 2026 07:17:39 +0000

Agentic Workflows That Survive Real Inputs

Most agent demos work because the demo input is clean. Real inputs are not. They arrive half-formed, with missing fields, contradictory context, PDFs that are actually images, and users who change their mind mid-thread. The gap between a working demo and a workflow that runs unattended for six months is almost entirely about how you decompose the problem and where you put the guardrails.

I design multi-agent workflows daily at BizFlowAI, and the pattern below is the one I keep coming back to. It is not glamorous. It is the reason my content pipeline runs 24/7 without me babysitting it, and it is the reason my serverless AWS integrations hit SLA instead of paging me at 2am.

Start by writing the workflow as a deterministic pipeline, then decide where an agent earns its keep

Before I touch an LLM SDK, I write the workflow as if agents did not exist. A boring sequence of functions with typed inputs and outputs. This is the single most useful exercise in agentic design, and it is the one most teams skip.

Here is the decomposition I run on every new workflow:

What is the trigger? A webhook, a cron, a queue message, a human uploading a file. Write the exact payload shape.
What is the final artifact? A JSON object, a published post, a Zendesk ticket update, a database row. Write the exact schema.
What are the intermediate artifacts? Between trigger and final, list every object that has to exist. Each one gets a name and a schema.
For each transition between artifacts, ask: does this require judgment, or is it a rule? Rules go in code. Judgment goes to an LLM. That is the whole heuristic.

On my ContentStudio pipeline, the flow looks like this:

trigger (topic gap detected)
  -> ResearchBrief        [LLM: judgment on angle + gap]
  -> OutlineDraft         [LLM: judgment on structure]
  -> DraftPost            [LLM: generation]
  -> SEOAuditReport       [code: deterministic checks]
  -> RevisedPost          [LLM: targeted rewrites only]
  -> PublishPayload       [code: schema validation]
  -> published (WordPress API)

Notice that four steps are code, three are LLM. In an earlier version I had the LLM doing the SEO audit and the publish payload assembly. It hallucinated slugs, invented canonical URLs, and once tried to publish to a site that did not exist. Moving those to deterministic code cut error rate to near zero and made the whole pipeline debuggable.

The rule I follow: an agent should be responsible for a decision, not for the plumbing around the decision.

Pick tools like you're hiring, not shopping

Tool selection is where most agent projects quietly go wrong. Teams give the agent 15 tools because "more tools = more capable" and then wonder why it picks the wrong one 30% of the time.

My working numbers, from actual production agents:

Tools available	Correct tool selection rate
3-5 tools, tightly scoped	96-99%
6-10 tools	88-93%
11-20 tools	70-82%
20+ tools	often below 70%, highly prompt-dependent

These are directional numbers from my own pipelines, not a benchmark. But the pattern is real and it holds across model families. The mitigation is not a smarter model. It is tool routing: a small dispatcher agent that picks the sub-agent, and each sub-agent has 3-5 tools.

When I write a tool spec, I treat it like a job description:

Name: verb-first, unambiguous. search_customer_by_email, not customer_lookup.
Description: one sentence on what it does, one sentence on when NOT to use it. The "when not" line is what stops confident-wrong calls.
Input schema: strict types, required fields marked. No optional-with-a-default fields hiding as strings.
Output schema: the same shape every time, including error shape. Never let an agent parse a free-form error string.

If you cannot write the "when not to use it" sentence, the tool is too broad. Split it.

Guardrails belong at the boundaries, not in the prompt

Prompt-level guardrails ("do not make up customer names", "always validate the email") are advisory at best. In production I treat them as backup, not primary defense. The primary defense is at the boundary of every step.

There are four boundary types where I always put guardrails:

1. Input validation on tool calls. Every tool validates its input with a strict schema (I use Zod in TypeScript, Pydantic in Python) before doing anything else. If the agent hallucinates a field, the tool returns a structured error and the agent gets to retry with feedback. No exceptions, no silent coercion.

2. Output validation on LLM responses. Every LLM call that produces structured output goes through a validator. If the structured output is required, I use the provider's structured output mode (Claude tool use, OpenAI response_format) and then re-validate. Belt and suspenders. I have caught model regressions this way.

3. Cost and loop caps. Every workflow has a hard budget: max tokens, max tool calls, max wallclock. If it exceeds any of them, it fails loudly, writes to the dead-letter queue, and pages me. An agent that silently loops for 40 minutes and spends $80 is a bug you only find on the invoice.

4. Side-effect gates. Any tool that writes to the real world (send email, publish post, create ticket, charge card) has a separate authorization check that is NOT part of the LLM prompt. It reads from a config or a feature flag. This is how you sleep at night when you give an agent write access.

Here is a minimal side-effect gate pattern I use:

async function publishPost(input: PublishInput, ctx: AgentContext) {
  // 1. Schema validation
  const parsed = PublishSchema.parse(input);

  // 2. Authorization check (NOT in prompt)
  if (!ctx.permissions.canPublishTo(parsed.siteId)) {
    return { ok: false, error: "unauthorized_site" };
  }

  // 3. Idempotency: has this artifact already been published?
  const existing = await db.publishedPosts.findOne({
    contentHash: parsed.contentHash
  });
  if (existing) {
    return { ok: true, alreadyPublished: existing.url };
  }

  // 4. Actual side effect, wrapped in retry with backoff
  const result = await wpClient.publish(parsed);
  await db.publishedPosts.insert({ ...parsed, url: result.url });
  return { ok: true, url: result.url };
}

Four things happen before we ever hit the WordPress API. Every one of them has caught a real bug in production.

Observability is the difference between "it works" and "I can prove it works"

You cannot debug an agent workflow by reading logs after the fact if you did not instrument it up front. And you will need to debug it, because real inputs will do things you did not predict.

The minimum I ship with every workflow:

A trace ID per workflow run, propagated through every tool call, LLM call, and DB write. One ID to grep on.
Structured logs, not print statements. JSON with trace_id, step_name, duration_ms, token_in, token_out, cost_estimate, and outcome.
Every LLM call persisted: prompt, response, model, temperature, tokens, latency. Storage is cheap; not being able to reproduce a bad run is expensive.
Every tool call persisted: inputs, outputs, duration, error.
A run summary record per workflow: trigger, final status, total cost, total duration, artifacts produced.

I use a simple Postgres schema for this. Not because it is fancy, but because I can write SQL against it when something breaks. When a run fails, I can pull the entire history in one query and replay it.

The metric I actually watch, across every workflow I run:

Signal	Why it matters
Success rate per workflow version	Regressions after prompt changes
p50 / p95 latency per step	Which step is the bottleneck
Cost per run (tokens + tools)	Unit economics, budget alerts
Tool call error rate by tool	Which tool spec is confusing the agent
Retry rate per step	Silent flakiness
Human-intervention rate	The honest measure of autonomy

The last one is the one nobody wants to look at. If a workflow needs a human 15% of the time, it is not autonomous, it is assisted. Fine, but call it what it is and price the ROI accordingly.

Design for failure modes you have already seen

After building enough of these, the failure modes rhyme. Design against them from day one.

Confident-wrong outputs. The agent produces something plausible but wrong. Mitigation: a validation step that checks the output against ground truth (a DB lookup, a rules check, a second-model critique for high-stakes outputs). For my content pipeline, every draft goes through a deterministic SEO/AEO audit before it can be published. If it fails, it goes back to a targeted revision agent with the specific failures listed.

Silent tool failures. The tool returns a 200 with an empty body, or a partial success that looks fine. Mitigation: tools must return an explicit { ok: boolean, ... } shape and the agent's tool-use loop must treat ok: false as a first-class case, not a string to interpret.

Runaway loops. The agent keeps calling the same tool with slightly different inputs. Mitigation: track recent tool calls in the loop and inject a system message when the same tool+input hash repeats. Also: the wallclock cap from earlier.

Context rot. Long conversations where the agent forgets or contradicts earlier decisions. Mitigation: don't run one long conversation. Break the workflow into steps with fresh context windows and pass forward only the artifact each step needs. This is the single biggest reliability win in multi-step agent design.

Model drift. The vendor updates the model and your prompts subtly break. Mitigation: pin model versions in code, and run a small eval set on every deploy. Even 20 representative cases will catch most regressions.

What I'd do if I were starting a new agentic workflow tomorrow

The order matters. This is what I actually do:

Write the workflow as a plain pipeline, no LLM calls. Get the schemas right.
Identify the 2-4 steps that genuinely need judgment. Everything else stays as code.
Write tool specs like job descriptions. If you have more than 5 per agent, split into sub-agents with a router.
Add strict input/output validation on every tool and every structured LLM call.
Add cost caps, loop caps, and side-effect gates before you connect it to anything that writes.
Instrument every call with a trace ID and persist the full history.
Build an eval set of 20-50 real inputs (not synthetic ones) before you ship.
Deploy behind a feature flag. Run in shadow mode against production traffic for a week.
Watch human-intervention rate. Iterate on the step with the highest rate first.
Only then, expand scope.

Steps 1-6 take about 60% of the total build time and prevent about 90% of the incidents. Skipping them is how you end up with a demo that crashes in week two.

The trending mistake I see repeatedly: teams pick a framework (LangGraph, CrewAI, whatever is fashionable this quarter) before they have written the pipeline as a boring function. The framework is not what makes it work. The decomposition is.

If you are building an agentic workflow that has to run against real production inputs, and you want a second set of eyes on the decomposition or the guardrails, I am open to a few consulting engagements this quarter. Reach out at lazar-milicevic.com/#contact, or read more on the blog for related posts on RAG evaluation, agentic context, and shipping agents that survive contact with users.

AI Agent Stack in 2026: LangGraph vs Custom vs DIY

lamingsrb — Thu, 30 Jul 2026 07:17:36 +0000

AI Agent Stack in 2026: LangGraph vs Custom vs DIY

I ship agents into production most weeks, and the framework question keeps coming back in every kickoff call. The honest answer is that the "right" stack depends on how much branching, how much state, and how much blast radius you can tolerate. I've built the same agent three ways in the last year, and the trade-offs are sharper than the framework marketing lets on. Here is how I actually decide.

The three stacks I keep reaching for

When I start a new agent, I pick from three buckets. Everything else is a variation of these:

A DIY loop, meaning a plain while loop around an LLM call with tool execution and a message array. Roughly 150 lines of Python or TypeScript.
A framework, almost always LangGraph now, occasionally Pydantic AI or the OpenAI Agents SDK for narrow cases.
Custom orchestration, meaning I write my own state machine, usually on top of a message queue or a durable executor like Temporal, Inngest, or AWS Step Functions.

The decision comes down to three questions: how many steps, how much concurrency, and what happens when it crashes at 3am. Everything else (observability, evals, human-in-the-loop) can be bolted onto any of the three.

When a DIY loop is the correct answer

For any agent with fewer than about five tools and a linear think-act-observe loop, a plain loop wins. I use this for ~40% of the agents I ship, including most of the sub-agents inside BizFlowAI ContentStudio. Frameworks add abstraction cost that only pays back when you have real branching.

Here is what the whole thing looks like, minus logging:

def run_agent(task: str, tools: dict, max_steps: int = 12):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": task},
    ]
    for step in range(max_steps):
        resp = client.messages.create(
            model="claude-sonnet-4-5",
            messages=messages,
            tools=list(tools.values()),
            max_tokens=4096,
        )
        messages.append({"role": "assistant", "content": resp.content})

        if resp.stop_reason == "end_turn":
            return resp.content[-1].text

        tool_results = []
        for block in resp.content:
            if block.type == "tool_use":
                try:
                    result = tools[block.name]<a href="**block.input">"fn"</a>
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": str(result),
                    })
                except Exception as e:
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": f"ERROR: {e}",
                        "is_error": True,
                    })
        messages.append({"role": "user", "content": tool_results})

    raise RuntimeError("Agent exceeded max_steps")

That is it. The advantages are real:

You debug in your IDE. Stack traces point at your code, not at three layers of framework internals.
Token usage is transparent. You can see and control every message that gets sent. When I cut LLM spend 40% on one project last quarter, most of it was pruning the message array in exactly this loop.
You own retries and error surfaces. Tool errors go back to the model as is_error: true so it can recover, and if the model loops on a broken tool, you catch it because you wrote the counter.

Where DIY breaks: parallel tool execution across long-running jobs, multi-agent handoffs, or anything with checkpointing that needs to survive a process restart. At that point you are either building a framework or reaching for one.

When LangGraph earns its complexity

LangGraph is the framework I keep coming back to in 2026, mostly because it treats agents as graphs of nodes with typed state, and that model actually matches how complex agents behave. I reach for it when the workflow has real branching (three or more meaningful paths), when I need durable execution across hours or days, or when I have multiple agents talking to each other.

The value shows up in three specific features:

Typed state as a first-class object. You define a TypedDict or Pydantic model for state, and every node returns partial updates. This is worth more than it sounds because it forces you to think about state transitions instead of passing message arrays around.
Checkpointing. With SqliteSaver or PostgresSaver, the graph resumes from the last node if the process dies. For long-running research agents or human-in-the-loop workflows, this is non-negotiable.
Interrupts for human review. You can pause the graph at a node, wait for external input (a Slack approval, a form submission), and resume without re-running earlier nodes.

Where I get burned: LangGraph's abstraction over LLM calls hides token accounting unless you wire in callbacks explicitly. The docs move fast, and some patterns that worked six months ago are now discouraged. If you use it, pin your versions and write integration tests that snapshot the actual API calls being sent.

A rough rule of thumb from what I've shipped:

Agent type	Nodes	Best fit
Single-purpose tool caller	1-3	DIY loop
Research + write + review	4-8	LangGraph
Multi-agent with handoffs	8+	LangGraph or custom
Long-running workflow (hours+)	any	Custom orchestration
Fan-out to 50+ parallel jobs	any	Custom orchestration

When custom orchestration is worth the write

Custom orchestration means treating the agent as a distributed system: a durable executor (Temporal, Inngest, Step Functions, or a homegrown thing on SQS) drives the workflow, and each LLM call or tool execution is a task in that system. I use this when I need to survive machine failures, when a single "agent turn" might take an hour, or when I have to fan out to hundreds of parallel branches.

The serverless AWS + Zendesk integration I built for SLA compliance was custom orchestration. Not because it was an "agent" exactly, but because the same reasoning applies: EventBridge drove the workflow, Lambda functions were the tasks, and DynamoDB held the state. If a Lambda died, EventBridge retried. If a tool failed, the state machine knew where to resume. That pattern maps cleanly onto agentic workflows once you accept that an LLM call is just another idempotent task.

What custom orchestration buys you that a framework does not:

True durability. LangGraph's checkpointer is fine for hours. Temporal or Step Functions is fine for months.
Fan-out at scale. When I need an agent to review 500 documents in parallel and aggregate results, I want a real queue and worker pool, not a graph library trying to manage concurrency.
Blast radius control. Each task runs in isolation. A poisoned tool response cannot corrupt the whole run because state is materialized between steps.

The cost is real. You write more code, you own the infrastructure, and you have to build your own visualization of what the agent did. For anything under an hour of runtime with fewer than 10 parallel branches, this is overkill. Above that, frameworks start to feel like toys.

The stack I actually pick, decision by decision

Here is the flowchart in my head when a new agent project starts:

Is the workflow linear with fewer than five tools? DIY loop. Ship it in a week. Add structured logging, token counting, and a retry wrapper.
Does it have real branching, or need to pause for human input? LangGraph, with PostgresSaver for checkpointing and OpenTelemetry callbacks for observability.
Does a single run last hours, or fan out to hundreds of parallel branches, or need to survive infrastructure failures? Custom orchestration on Temporal or Step Functions. Treat each LLM call as an activity.
Multi-agent with three or more distinct agents talking to each other? LangGraph if runs are short, custom if runs are long. Never DIY, because message routing between agents is where DIY loops turn into unmaintainable spaghetti.

I lean DIY more often than framework marketing suggests I should. Most agents in the wild do not need a graph. They need one good loop, careful prompt engineering, tight tool schemas, and honest evals. The framework becomes useful the moment the workflow diagram stops fitting on one screen.

The things nobody tells you until you ship

A few lessons that cost me real time:

Tool schemas matter more than model choice. I have seen the same task go from 60% to 95% success just by rewriting tool descriptions and parameter names. Before you switch models or add a framework, spend a day on your tool schemas. The model can only be as smart as your tools let it be.

Message array pruning is where the token savings live. In a 20-step agent, the message array grows quadratically in tokens if you keep all tool results. I keep the last 3-5 tool results verbatim and summarize the rest. On one content agent this cut cost by 40% with no measurable quality loss.

Error strings are prompts. When a tool fails, the error message goes back into the model's context and becomes part of the next prompt. "Database connection failed" is a useless prompt. "Database query returned no rows for user_id=42. Try a different user_id or check if the user exists." lets the model recover.

Checkpointing without idempotency is a lie. If your graph resumes from step 7 and step 7 called a payment API, you just charged the customer twice. Every tool that mutates state needs an idempotency key. This is true for any framework.

Evals are not optional. I run every agent I build against a fixed set of 20-50 test cases before every deploy. Not fancy LLM-as-judge stuff, just deterministic checks: did it call the right tools, did the final answer contain the required fields, did it stay under the token budget. This is the single highest-ROI engineering practice in agent work.

What I'd do if I were starting a new agent tomorrow

Start with the DIY loop. Get the tools, the system prompt, and the eval set right first. Instrument token usage and step count from day one. Ship it, watch it fail on real inputs, and fix it.

Only reach for LangGraph when the workflow diagram has real branches or when you need durable pause-and-resume. Only reach for custom orchestration when a single run is longer than an hour or fans out beyond what a single process should manage. Do not skip the DIY stage: it teaches you what your agent actually does, and that knowledge is what makes the framework choice obvious later.

The framework debate is mostly noise. The agents that ship and stay shipped are the ones with tight tool schemas, honest evals, and observability that lets you debug production failures in under 10 minutes. Everything else is scaffolding.

If you are building an agent right now and stuck between these stacks, I'm always happy to look at the workflow diagram and tell you what I'd pick. Reach out at lazar-milicevic.com/#contact, or read more on the blog where I've written about agent evaluation, context engineering, and cutting token spend in production.

I/O 2026: 47s to 12s On My Headless Chrome Loop

lamingsrb — Thu, 30 Jul 2026 06:12:33 +0000

I/O 2026 Scorecard: 1 Of 3 Chrome AI Updates Actually Helps Bots

Google shipped three Chrome AI updates at I/O 2026. If you run headless Chrome workers at 3am — scraping, invoice parsing, email enrichment — two of them are worthless to you and every keynote recap glossed over that. I benchmarked all three purely as agent infrastructure. One cut a DOM inspection loop from 47 seconds to 12. The other two scored zero.

The Filter Every I/O Recap Missed

The reason most reviews of these three updates are misleading isn't that they're wrong — it's that they're answering a different question. Every recap I watched scored these features as developer productivity: how much faster can a human sitting at a laptop ship a feature. That's a fine scorecard if you're that human. It's the wrong scorecard if the thing consuming Chrome is a Python process running under systemd on a machine you haven't logged into for a week.

I have a home server running WSL Ubuntu with three headless Chrome workers pinned 24/7. One triages Gmail, one parses supplier invoices, one enriches inbound leads by scraping public company pages. Every millisecond of every loop shows up on my power bill. So the only scorecard I care about is:

Does the bot get faster (fewer CPU-seconds per task)?
Does it get cheaper (fewer LLM tokens, fewer retries)?
Does it get more reliable (fewer 3am failures)?

Everything else is keynote choreography. That one filter kills two of three updates before we even open the changelog.

Update 1 — Modern Web Guidance: Useful Once, Then Cache It

Google now publishes a canonical, machine-readable description of current web platform best practices — how to structure a page, handle permissions, build accessible components, use modern APIs correctly. As context for a coding LLM, this is genuinely helpful. When your planner is writing new automation code, feeding it this doc means fewer hallucinated APIs, fewer references to document.registerElement from 2015, fewer patterns that broke two Chrome majors ago.

But here is the honest verdict: it is static. I integrated it in about 40 minutes:

# one-time fetch, cached to disk, injected into planner system prompt
import pathlib, hashlib, requests

GUIDANCE_URL = "https://web.dev/.../modern-web-guidance.json"
CACHE = pathlib.Path(".cache/web_guidance.json")

def load_guidance():
    if CACHE.exists():
        return CACHE.read_text()
    data = requests.get(GUIDANCE_URL, timeout=10).text
    CACHE.parent.mkdir(exist_ok=True)
    CACHE.write_text(data)
    return data

SYSTEM_PROMPT = f"""You write Playwright automation code.
Follow these current web platform patterns:
{load_guidance()}
"""

That's it. Refresh the cache monthly with a cron job. It's a one-time integration, not an ongoing unlock. Zero recurring impact on latency or spend. Score for autonomous work: mildly useful, one-shot.

When it actually matters

You have an LLM writing fresh Puppeteer/Playwright code from natural-language tasks
You've been getting hallucinated selectors or deprecated APIs
You do NOT already have a working code-gen pipeline (in which case, skip)

Update 2 — DevTools AI Assistance: Architecturally Wrong For Bots

This got the most stage time. Open Chrome DevTools, an AI panel narrates what's happening on the page, suggests fixes for performance regressions, walks you through a failing network request. The demo was clean. It's completely irrelevant to autonomous agents.

My workers launch Chrome with roughly this:

google-chrome \
  --headless=new \
  --disable-gpu \
  --no-sandbox \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/chrome-worker-1

There is no DevTools panel. There is no human to click "Apply suggestion." There is no tooltip to read. The entire feature assumes a person in the loop who can make a judgment call, and my whole point is that no person is in the loop between midnight and 6am when these jobs run.

This is not a bad feature. It's just built for a different architecture. Score for unattended workflows: flat zero. I won't mention it again.

Update 3 — DevTools MCP Bridge: 47s → 12s On A Real Loop

The one nobody clapped for. Chrome exposed the DevTools protocol as a programmable surface for agents via an MCP (Model Context Protocol) bridge. In plain terms: your agent can drive the same underlying protocol the AI assistant uses, but headless, without a UI, at machine speed. Same primitives, no human wrapper.

This changed real numbers on my stack. My Gmail-triage agent does DOM inspection cycles on rendered email bodies — pulling structured data out of receipts, invoices, shipping notifications where the sender didn't bother with clean HTML. Old pattern using a plain CDP wrapper:

# OLD: ~47s per email, multiple round-trips per field
async def extract_receipt_old(page):
    await page.wait_for_load_state("networkidle")   # ~8s
    dom = await page.content()                       # full serialize
    # LLM extracts fields from raw HTML string
    fields = await llm.extract(dom, schema=RECEIPT) # ~30s, big prompt
    # verification pass with second selector query
    total = await page.locator(".total").inner_text()
    return validate(fields, total)

The bottleneck wasn't the LLM call. It was the full DOM serialize + a giant prompt + a second verification pass because the first pass hallucinated fields half the time. With the MCP DevTools bridge, the agent queries structured accessibility trees and computed styles directly, in one round-trip, and only sends the LLM the relevant subtree:

# NEW: ~12s per email using MCP DevTools tools
async def extract_receipt_new(mcp_client, url):
    await mcp_client.call("navigate", url=url)
    tree = await mcp_client.call(
        "accessibility_snapshot",
        interesting_only=True,
    )
    # small, structured, no HTML noise
    fields = await llm.extract(tree, schema=RECEIPT)
    return fields

Same task, same page, same extraction schema. Numbers from a 200-email batch on my box:

Metric	Old CDP wrapper	MCP DevTools bridge
Wall time per email	47 s	12 s
LLM input tokens/call	~14,200	~3,100
Verification retries	38 / 200	4 / 200
Batch cost (Claude)	$2.84	$0.61
Failed extractions	11 / 200	3 / 200

Roughly 4x speedup, 4.6x cheaper, and reliability went up because the accessibility tree is already structured — no more asking the model to hunt through 400KB of table soup. Multiply that across three workers running around the clock and it's not a demo, it's an infrastructure line item that pays for the server.

Where you'll see the gains

DOM inspection loops (form parsing, receipt/invoice extraction, table scraping)
Network waterfall analysis (finding the one XHR that actually has the data)
Any place you were serializing full HTML and shoving it into a prompt

If your bottleneck is elsewhere — captcha solving, login flows, actual network latency to a slow origin — this update won't move your numbers. But you'll know in an hour of benchmarking, not a sprint.

How To Benchmark Your Own Loop In Under An Hour

Don't take my numbers on faith. Pick one loop that runs often enough to matter and one that you already have baseline data on. Here's the checklist I ran:

Pick the loop. Something that runs at least 50 times a day. Bonus if you already log wall-time per iteration.
Capture baseline. Run it 20 times, log p50 and p95 wall time, log token usage per LLM call, log failure rate.
Rewrite the DOM-inspection portion only. Keep the same schema, same LLM, same prompt structure — only swap how you extract page state.
Run 20 more iterations on the same input set. Cache the target pages if they're live, so you're comparing extraction, not network variance.
Compare four numbers: wall time p50, input tokens per call, retry rate, and dollar cost per 1,000 iterations.

If you don't see at least a 2x improvement on wall time or a 3x cut in input tokens, the MCP bridge is not your bottleneck. Move on. If you do, rewrite the other loops with the same pattern and go audit your monthly compute bill next week.

What This Says About The Next Two Years Of Browser Automation

Here's the pattern I keep watching for. Every browser vendor is currently optimizing for the humans in the room at their keynote — developers with laptops open who want prettier tooltips and smarter autocomplete. The much larger, quieter audience is agents running headless in data centers, home servers, and cheap VPS boxes. That audience doesn't clap. It also doesn't tweet. It just quietly generates load 24/7 and pays cloud bills.

Whichever vendor stops shipping human-in-the-loop features and starts shipping unattended-agent primitives — programmable protocols, structured page snapshots, reliable headless permission models, deterministic timing — is going to own the next five years of automation infrastructure. Chrome accidentally shipped one of those primitives at I/O 2026 and buried it on slide fourteen while spending twenty minutes on the DevTools chatbot.

My working assumption for the rest of the year: audit every browser announcement with one question. Does this feature do anything when nobody is watching? If the answer is no, it's not for me, and it's probably not for you either if you're reading this far.

Where bizflowai.io fits

The kind of headless Chrome workflows in this post — invoice parsing from email, lead enrichment from public pages, receipt data pulled out of shipping notifications — are the exact loops we already run in production for small business clients on bizflowai.io. When a new browser primitive like the MCP DevTools bridge shows up, we re-benchmark the affected client loops that week, roll the faster path into the workers, and the client sees a lower monthly compute line without having to think about any of it. It's the boring, invisible half of automation — the part that decides whether your bot is a $40/mo utility or a $400/mo money pit.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

What Gartner's Magic Quadrant Means for SMBs

lamingsrb — Thu, 30 Jul 2026 06:12:30 +0000

What Gartner's Magic Quadrant Means for SMBs

You're a 6-person team drowning in manual work — invoice routing, lead handoffs, onboarding tickets — and someone forwarded you a Gartner Magic Quadrant PDF suggesting the "right" answer is a six-figure enterprise platform. It isn't. The Magic Quadrant is a useful buying signal for Fortune 500 procurement teams, but the evaluation criteria don't map cleanly to how a 1-10 person business actually adopts automation.

This post breaks down what Gartner actually evaluates in its workflow and process automation research, which vendor archetypes tend to occupy which quadrants, and — more usefully — how to translate the underlying criteria into a decision framework for tools you can actually afford, deploy, and maintain without a platform team.

What the Magic Quadrant actually evaluates

The Gartner Magic Quadrant plots vendors on two axes: Ability to Execute (vertical) and Completeness of Vision (horizontal). Each axis rolls up roughly 7-8 sub-criteria, weighted by Gartner analysts. The result is four quadrants: Leaders, Challengers, Visionaries, and Niche Players.

For workflow-adjacent categories — Gartner has published Magic Quadrants for Enterprise Low-Code Application Platforms, Robotic Process Automation, and Integration Platform as a Service (iPaaS) — the evaluated criteria typically include:

Ability to Execute

Product/service (functional breadth and depth)
Overall viability (financial health, R&D investment)
Sales execution and pricing
Market responsiveness
Marketing execution
Customer experience (support, professional services, communities)
Operations

Completeness of Vision

Market understanding
Marketing and sales strategy
Offering strategy (roadmap)
Business model
Vertical/industry strategy
Innovation
Geographic strategy

Read that list again with an SMB filter on. "Geographic strategy," "vertical industry coverage," and "sales execution" are heavily weighted — and they mostly reward vendors with global field sales teams and 50+ industry accelerators. Those attributes are irrelevant, and often actively harmful, when you're a solo operator who wants to ship an email-triage workflow by Friday.

That's not a knock on Gartner. Their audience is enterprise procurement. It's a knock on using the quadrant as-is for SMB decisions.

The vendors you'll typically see in the Leaders quadrant

Across the RPA, iPaaS, and low-code MQs over recent cycles, the same names recur as Leaders or strong Challengers: UiPath, Microsoft (Power Automate + Power Platform), Automation Anywhere, SAP, ServiceNow, Salesforce (via MuleSoft and Flow), Appian, Pegasystems, Boomi, and Workato. Vision-heavy Visionaries and Niche Players often include Nintex, Tray.io, Celonis (process mining), and newer AI-native entrants.

A rough map of what each archetype is actually good at:

Vendor archetype	Real strength	SMB reality
UiPath / Automation Anywhere	Attended + unattended RPA on legacy desktop apps	Overkill unless you're screen-scraping SAP or Citrix daily
Microsoft Power Automate	Deep M365 tie-in, per-user licensing	Actually SMB-viable if you're all-in on Microsoft 365
ServiceNow / Appian / Pega	Case management, BPM, regulated industries	Six-figure floor. Not for SMB.
Workato / Boomi / MuleSoft	iPaaS with enterprise governance	Priced per connection or workspace; usually mid-market up
Zapier / Make / n8n	Long-tail app connectors, self-serve	Not in the enterprise MQ. This is where most SMBs live.

Notice what's missing: Zapier, Make, n8n, and most AI-native automation tools do not appear in the enterprise MQs. That's not because they don't work — it's because Gartner's methodology filters them out on revenue thresholds, reference customer size, and functional breadth criteria that reward enterprise features (RBAC hierarchies, on-prem deployments, 200+ pre-built connectors for legacy ERPs) SMBs don't need.

Which MQ criteria actually matter for a 1-10 person team

Strip the enterprise-specific weightings out and re-weight for how an SMB actually buys and runs automation. Here's a practical rewrite of the evaluation criteria:

Keep (with re-weighting):

Product depth in the workflows you actually run. Not "does it have 400 connectors" but "does it handle Gmail, Stripe, HubSpot, Slack, and your CRM cleanly."
Customer experience. For SMBs this means docs, community, and forums — not a named CSM.
Overall viability. Yes, you still care if the vendor will exist in 24 months. But an open-source project with 60k GitHub stars can be more viable than a Series B startup burning $8M a quarter.
Innovation / offering strategy. Specifically: is the vendor shipping useful AI features (LLM steps, agent nodes, RAG hooks) without turning them into a separate SKU?

Discard or heavily deprioritize:

Geographic strategy
Vertical accelerators (healthcare, banking, telco)
Sales execution (you don't want a sales cycle)
Marketing execution
Professional services depth

Add (not in the MQ at all but critical for SMBs):

Time-to-first-working-workflow. Under 1 hour or it's the wrong tool.
Self-hostability or data portability. Can you export your workflows as JSON/YAML and leave?
Pricing that scales with runs, not seats. Per-seat pricing kills SMB automation adoption because the ops person building the flow isn't the person receiving the output.
LLM-native primitives. First-class support for calling OpenAI/Anthropic/local models inside a step, not as a bolt-on.

A decision framework you can actually use

Here's the framework I use with clients before recommending any workflow platform. Answer each question honestly:

automation_fit_check:
  team_size: <10                     # if >100, revisit MQ Leaders
  primary_workflow_type:
    - api_to_api                     # → Zapier, Make, n8n, Workato
    - screen_scraping_legacy         # → Power Automate Desktop, UiPath Community
    - document_processing_with_ai    # → n8n + LLM, or custom
    - human_in_the_loop_approvals    # → Slack/email + n8n
    - long_running_agents            # → custom (LangGraph, Temporal)

  data_sensitivity:
    - public_or_low                  # SaaS fine
    - customer_pii                   # prefer self-host or SOC2 SaaS
    - regulated_phi_pci              # self-host, or enterprise MQ vendor

  budget_monthly_usd:
    - "<100"                         # Make, n8n cloud, Zapier starter
    - "100-500"                      # n8n cloud pro, Zapier pro, Workato starter
    - "500-2000"                     # Workato, Tray, custom infra
    - ">2000"                        # you're now shopping the actual MQ

  who_maintains_it:
    - non_technical_founder          # Zapier > Make > n8n
    - technical_operator             # n8n or code-first
    - engineering_team               # code-first (Temporal, Inngest, custom)

If your answers cluster in the top rows, the Gartner MQ Leaders are the wrong shortlist. If they cluster in the bottom rows, use the MQ.

How to translate MQ signals into an SMB shortlist

Even if you're not buying an enterprise platform, MQ research still has salvageable signal. Here's how I extract it:

1. Use the "Cautions" section, not the "Strengths" section. Every MQ vendor profile ends with 3-5 Cautions. These are the honest bits — where the analyst says "pricing is opaque," "the AI features are marketing more than product," or "the roadmap slipped last cycle." That's the real product research.

2. Watch quadrant movement over cycles. A vendor sliding from Leader toward Niche across two consecutive MQs is a signal, regardless of whether you're buying enterprise. It often means R&D has stalled or the market moved past their architecture.

3. Cross-reference with the Peer Insights reviews. Gartner Peer Insights (their review site) has verified customer reviews with company size filters. Filter to "50-200 employees" and read those specifically — the complaints from smaller shops are the ones that will apply to you.

4. Ignore the "hot" categories. Every year Gartner promotes a new umbrella term — hyperautomation, composable enterprise, agentic process automation. These are useful for analyst positioning; they rarely change what workflow you should build on Monday.

Adam Preset, a longtime Gartner analyst, has publicly noted that "the Magic Quadrant is a snapshot, not a recommendation." Take that literally. The dot's position tells you where a vendor is relative to their peers on Gartner's axes — it doesn't tell you whether the vendor fits your workflow.

The practical SMB stack in 2026

Here's what I actually see working in 1-10 person businesses right now, organized by workflow archetype:

Trigger-and-forget integrations (order confirmations, lead routing, notifications)

Zapier if the person building it doesn't write code
Make if they can read a visual DAG and want more control per dollar
n8n (self-hosted or cloud) if they want to own the workflow and add custom nodes

AI-heavy workflows (email classification, document extraction, drafting)

n8n + OpenAI/Anthropic nodes, or
A thin Python service (FastAPI + a queue) if the workflow needs retries, observability, and version control

Example: a lead-qualification pipeline in ~40 lines that any of these platforms can host:

import anthropic, json
from typing import Literal

client = anthropic.Anthropic()

def qualify_lead(lead: dict) -> dict:
    prompt = f"""Classify this inbound lead. Return JSON only.

Lead:
- Company: {lead['company']}
- Size: {lead.get('employees', 'unknown')}
- Message: {lead['message']}

Return: {{"tier": "hot|warm|cold", "reason": "...", "next_action": "..."}}"""

    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(resp.content[0].text)

# Wire this into a webhook from your CRM. Route hot leads to Slack,
# warm to a nurture sequence, cold to a monthly digest.

This is the kind of thing the MQ Leaders will sell you as a "cognitive automation platform" for $60k/year. For a 5-person business, the code above plus a $20/month n8n instance is the correct answer.

Human-in-the-loop approvals (expense approvals, content publishing, refund decisions)

Slack as the UI, n8n or a small backend as the orchestrator
Skip the BPM platforms entirely unless you have audit requirements

Long-running agents (research, multi-step negotiation, ongoing monitoring)

Still early. Most "agent platforms" in 2026 are wrappers around the same LLM APIs. Build thin, own the orchestration.

Common mistakes SMBs make when reading analyst reports

Three patterns I see repeatedly:

Buying the Leader because it's the Leader. A team of 8 signs a Workato or ServiceNow contract, spends three months on implementation, and ends up using 4% of the platform. The Leader position rewards breadth. You need depth in five workflows.

Confusing Visionary with "up-and-coming." Visionaries in the MQ often have interesting product but weak execution. For an SMB betting on a single tool, execution matters more than vision. A boring, well-run Challenger beats an exciting Visionary that ships bugs.

Treating MQ absence as absence of quality. n8n, Make, Inngest, Temporal, Windmill, and most modern AI-native automation tools aren't in the enterprise MQs. That's a methodology artifact, not a quality signal. Some of the most productive automation stacks I've built for clients use zero MQ-listed vendors.

How BizFlowAI approaches this

We sit deliberately outside the enterprise MQ conversation. Our clients are 1-10 person teams who need working automation in weeks, not RFPs and 6-month rollouts. The stack we build on — n8n, direct LLM APIs, small Python services, Slack and email as the interface — is chosen for the criteria the MQ under-weights: fast time-to-first-workflow, per-run pricing, data portability, and the ability for a technical founder to read and modify what we ship.

That doesn't mean the MQ Leaders are wrong. If you're a 400-person company with SOX requirements, buy the Leader. But if you're the ops person at a 6-person business trying to stop routing invoices by hand, the framework earlier in this post — team size, workflow type, data sensitivity, budget, maintainer — will get you to a better shortlist than the quadrant will. That's the shortlist we work from, and it's why our engagements look more like "here's your working system in 3 weeks" than "here's your platform-selection deck."

The takeaway

Read the Gartner Magic Quadrant the way you'd read a Michelin Guide when you're planning a weeknight dinner: interesting context, wrong shortlist. The evaluation criteria — product depth, viability, customer experience, innovation — are the right questions. The weightings and the vendor cutoffs are calibrated for buyers you are not.

Reweight the criteria for your actual constraints. Add the ones Gartner doesn't score: time-to-first-workflow, per-run pricing, portability, LLM-native primitives. Build the shortlist from there. And when in doubt, ship the smallest working version of the automation this week — the platform decision gets easier once you have real workflow requirements instead of vendor slides.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.

Check Your Website Analysis Score in 15 Minutes

lamingsrb — Mon, 27 Jul 2026 06:12:23 +0000

Check Your Website Analysis Score in 15 Minutes

Your website feels slow, leads are dropping off, or Google traffic has stalled—but you do not know whether the problem is performance, SEO, or the page itself. You do not need a week-long audit to find the first useful answers. You need a repeatable way to check the pages that affect revenue and turn the results into a short fix list.

A website analysis score is a starting point, not a verdict

A website analysis score combines different signals—page speed, technical SEO, accessibility, and user experience—but no single number can tell you whether a site will rank or convert. Use scores to identify likely problems, then inspect the underlying diagnostics before changing code, content, or design.

The trap is treating a score of 100 as the objective. A technically perfect landing page can still confuse visitors. A page with an average Lighthouse score can still generate qualified leads if it answers a search query clearly and loads reliably on a phone.

For a practical small-business review, separate the analysis into four layers:

Layer	What you are checking	Why it matters
Performance	Load speed, rendering, blocking scripts, image weight	Slow pages create friction before visitors see your offer
SEO	Indexability, metadata, headings, links, structured data	Search engines need to find, understand, and trust the page
UX	Clarity, navigation, forms, mobile behavior, calls to action	Visitors need to complete a task without guessing
Accessibility	Contrast, labels, keyboard access, semantic markup	Better access also tends to improve interface quality and maintainability

Think of the process as a triage system. Start with the highest-value URLs, identify failures that affect real visitors, and fix those before polishing low-impact warnings.

For most small businesses, review these pages first:

Homepage
Primary service or product page
Highest-traffic blog post or landing page
Contact, booking, checkout, or quote-request page
One page that receives traffic from paid campaigns

Do not begin by auditing every URL on a 500-page site. If the template, navigation, image handling, or analytics scripts are broken, a focused review of five pages will usually reveal the pattern.

Start with real-user performance data before lab scores

Real-user data is the most useful performance signal because it reflects what actual visitors experienced across devices, networks, and locations. Google PageSpeed Insights shows both field data from the Chrome User Experience Report when available and a controlled Lighthouse test, so use both views rather than relying on one synthetic run.

Open Google PageSpeed Insights, enter a page URL, and review both mobile and desktop results. Start with mobile. Many small-business sites look acceptable on a developer laptop but struggle on a mid-range phone over a mobile connection.

Google describes Core Web Vitals as “the subset of Web Vitals that apply to all web pages.” The primary metrics to inspect are:

Largest Contentful Paint (LCP): How quickly the largest visible page element appears. On many marketing sites, this is the hero image, headline block, or video.
Interaction to Next Paint (INP): How quickly the page responds after a visitor interacts with it.
Cumulative Layout Shift (CLS): How much visible content jumps around while the page loads.

Do not copy a recommendation into a task list without checking what it refers to. For example, “serve images in next-gen formats” may point to one oversized hero image that is worth fixing—or to a tiny image that will make no practical difference.

A useful first-pass performance worksheet looks like this:

Check	Where to look	Common cause	First fix to investigate
Poor LCP	PageSpeed Insights diagnostics	Large hero image, slow server response, render-blocking CSS	Compress or resize hero media; preload the critical asset only when justified
Poor INP	Field data and interaction diagnostics	Heavy JavaScript, third-party widgets, long tasks	Remove unused scripts; defer non-critical widgets
High CLS	Lighthouse diagnostics	Images without dimensions, late-loading banners, web-font swaps	Set image dimensions; reserve space for embeds and notices
High total page weight	Network panel or Lighthouse	Uncompressed images, video embeds, large font files	Remove unused assets and load video on interaction
Slow backend response	Time to First Byte observations	Slow hosting, uncached dynamic pages, expensive database calls	Add caching and inspect server-side work

Run at least three tests before declaring a lab score “the score.” Results can vary based on test conditions and third-party requests. If the same issue appears repeatedly, it is likely worth investigating.

You can also run Lighthouse from the command line when you need a repeatable baseline for several pages:

npm install -g lighthouse

lighthouse https://example.com/services \
  --only-categories=performance,accessibility,seo \
  --output=html \
  --output-path=./website-audit.html

This is useful for before-and-after checks after a deployment. It is not a replacement for real-user data. Lighthouse simulates a browser session; it cannot tell you exactly how every customer’s phone and connection behaved.

Check SEO health by following the crawl path

A page cannot earn search visibility if search engines cannot crawl it, index it, or understand its purpose. The fastest SEO analysis checks the technical path first, then verifies that the page gives a clear answer to the search intent it targets.

Start with one important URL and inspect these items in order:

Does the URL return a successful HTTP status?
Is it blocked by robots.txt, a noindex tag, or a login wall?
Does it have a self-referencing canonical URL?
Is it linked from your navigation, sitemap, or another crawlable page?
Does the title explain the page’s purpose?
Does the H1 match what a searcher expects to find?
Is the main content substantial enough to answer that need?
Does the page link to relevant next steps?

You can inspect response headers with curl:

curl -I https://example.com/services

A typical healthy response begins with a successful status code:

HTTP/2 200
content-type: text/html; charset=UTF-8

Then inspect the source or rendered HTML for the canonical and robots directives:

<link rel="canonical" href="https://example.com/services">
<meta name="robots" content="index,follow">

These tags are not automatically proof that the page is indexed. They are instructions and hints. Use Google Search Console to inspect the actual indexing status of important URLs on a verified property.

For a wider review, use a crawler such as Screaming Frog SEO Spider, Sitebulb, or an equivalent tool that can crawl your site and export findings. The tool matters less than the questions you ask it.

Export a list of pages with:

Missing or duplicate title tags
Missing H1s
Redirect chains
Broken internal links
Pages with noindex
Canonical mismatches
Duplicate or near-duplicate content
Large images
Orphaned pages, where your crawler can detect them

AI can help summarize a crawl export, cluster repeated metadata issues, and turn technical findings into developer tickets. It should not be allowed to decide that a page is “thin” or “duplicate” without human review. Two service pages may share a template while serving genuinely different customer needs.

A good title tag is specific, readable, and aligned with the page. This is more useful:

<title>Commercial HVAC Maintenance in Austin | Acme Mechanical</title>

Than this:

<title>Home | Acme Mechanical</title>

The second title wastes one of the clearest signals available to search engines and visitors. It also gives your team no clue what the page is meant to rank for.

Review UX by testing one real customer task

A UX score matters only when it reflects whether a visitor can complete the job they came to do. Test one concrete task on a phone and desktop—for example, request a quote, book a call, buy a product, or find a service area—and document every point where you hesitate.

Do this without looking at the site as its owner. Better yet, ask someone unfamiliar with the business to do it while you watch. Give them a simple prompt:

“You need help with [service]. Find out whether this company serves you and request a quote.”

Watch for behavior, not opinions. Did they scroll past the call to action? Did they try to click something that was not clickable? Did they stop at the form because it asked for information they did not have?

Use this practical UX checklist:

Make the page purpose obvious

Within the first screen, a visitor should be able to identify:

What you offer
Who it is for
Where you operate, when location matters
What action they should take next
Why they should trust you enough to take that action

A vague headline such as “Solutions That Move You Forward” forces the visitor to decode marketing language. A service page should say what it does.

Make the next action low-friction

Every key page needs a visible next step. For a local service business, that might be “Request an estimate.” For a SaaS product, it might be “Start a trial” or “Book a demo.”

Check the form itself:

Are required fields clearly marked?
Does the submit button describe the outcome?
Does the form work with keyboard navigation?
Does it show a useful error message?
Does it confirm what happens after submission?
Does it work on a narrow mobile screen?

Avoid asking for information you do not need to handle the first response. A lead form that requires a detailed project brief, budget, job title, company size, phone number, and mailing address may produce fewer submissions than a shorter form with a reliable follow-up process.

Test mobile navigation and content priority

Mobile UX is not a scaled-down desktop layout. Check whether the menu is usable with one hand, whether the sticky header covers content, and whether pop-ups block the page before a visitor can read it.

Also inspect the content order. If your mobile visitor must scroll through a giant image, a mission statement, and three generic feature cards before seeing your service, the page is not prioritizing the user’s task.

Include accessibility in the review

Accessibility testing catches problems that also damage general usability. Run WAVE or Lighthouse accessibility checks, then manually test essential navigation with a keyboard.

Automated tools are good at finding missing image alternative text, low contrast, unlabeled form controls, and invalid markup. They cannot tell you whether alternative text is meaningful or whether a form flow makes sense.

Use AI to interpret findings, not to manufacture a score

AI tools are useful when they reduce analysis time and turn raw reports into clear work—but they are unreliable when asked to make unsupported claims about rankings, conversions, or “SEO quality.” Give an AI tool source material from trusted audits, then ask it to classify and prioritize the findings.

A practical input package might include:

PageSpeed Insights URL and exported report
Lighthouse JSON output
Search Console indexing details
A crawl export in CSV format
Screenshots of desktop and mobile pages
Your business goal for the page
Known constraints, such as a WordPress theme, Shopify app, or required CRM script

For example, you can ask an AI assistant to convert a Lighthouse report into an engineering-ready task list:

You are reviewing a website performance report.

Use only the evidence below. Do not invent traffic, ranking, conversion,
or revenue impact.

For each issue:
1. State the evidence.
2. Explain the likely visitor impact.
3. Label it P1, P2, or P3.
4. Suggest the smallest safe fix.
5. Identify what must be tested after deployment.

Business goal: generate quote requests from mobile visitors.
Page type: local service landing page.
Constraints: WordPress site, contact form plugin, analytics required.

[Paste PageSpeed or Lighthouse findings here]

The output is only as trustworthy as the evidence you provide. If you paste a vague sentence such as “the website is slow,” you will get a vague response dressed up as analysis.

AI is particularly useful for these jobs:

Task	Useful AI role	Human review required
Crawl export cleanup	Group repeated errors and identify templates behind them	Confirm whether pages are intentionally different
Metadata drafting	Suggest title and description variations based on page content	Check accuracy, tone, and search intent
Performance prioritization	Translate diagnostics into a plain-English fix list	Validate the technical recommendation
UX review	Compare page copy against a defined user task	Test the actual interface on devices
Ticket writing	Turn findings into scoped tasks with acceptance criteria	Estimate effort and deployment risk

Do not publish AI-generated metadata or rewrite service pages in bulk without reviewing each page. That creates generic copy, factual mistakes, and a site that sounds like every other site.

Turn the audit into a 7-day repair plan

The fastest way to improve a website analysis score is to fix recurring, high-impact issues at the template or system level. A seven-day plan keeps the audit from becoming another spreadsheet that nobody acts on.

Start by assigning every finding to one of three groups:

P1 — blocks users or search engines: broken forms, pages blocked from indexing by mistake, server errors, broken checkout, major mobile failures.
P2 — creates measurable friction: oversized hero media, slow scripts, unclear calls to action, missing page titles, poor navigation.
P3 — worthwhile cleanup: minor heading inconsistencies, low-priority image alt text, cosmetic warnings, metadata refinements.

Then create tasks that can be tested. “Improve SEO” is not a task. This is:

task: Reduce homepage hero image payload
priority: P2
owner: developer
evidence: "PageSpeed Insights flags the hero image as the LCP element"
change:
  - Export responsive AVIF and WebP versions
  - Add width and height attributes
  - Serve the appropriate image size by viewport
acceptance_test:
  - Confirm image renders correctly on mobile and desktop
  - Re-run Lighthouse before and after deployment
  - Check that no layout shift was introduced
rollback: Restore the previous image asset and markup

Keep a simple audit log with the date, URL, original issue, change made, and post-deployment result. This helps when a performance fix breaks a visual layout, an app update adds a script back, or someone asks why a particular plugin was removed.

A sensible sequence is:

Fix broken forms, indexability mistakes, and errors.
Remove or delay unnecessary third-party scripts.
Optimize the largest images and media embeds.
Clarify page purpose and calls to action.
Repair metadata, internal links, and heading structure.
Re-test on mobile and desktop after every meaningful deployment.
Review Search Console and real-user performance trends over time.

Do not chase every audit warning. A third-party chat widget may be worth its performance cost if it reliably creates qualified conversations. A tracking script may be necessary for a campaign. The right decision is based on business value, visitor impact, and the smallest acceptable technical cost.

How BizFlowAI approaches this

BizFlowAI builds and runs website analysis workflows that combine performance checks, SEO crawl findings, and practical UX review into an actionable fix list. The goal is not to produce a decorative scorecard; it is to identify what is slowing down a real customer journey and what can be safely fixed in the existing stack.

For clients, that often means automating repeatable checks for priority URLs, flagging regressions after deployments, and translating raw tool output into scoped tasks a small team can actually complete. AI helps organize the evidence, while the final recommendations remain tied to the site’s code, content, and business goal.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.

Chrome Just Killed 8 Of My 12 Puppeteer Scrapers

lamingsrb — Mon, 27 Jul 2026 06:12:20 +0000

Chrome Just Killed 8 Of My 12 Puppeteer Scrapers

Google I/O 2026 shipped three Chrome updates that made two-thirds of my production scraping code obsolete overnight. I run browser automation for bookkeeping SMBs — receipts pulled from vendor portals, PDFs OCR'd, line items pushed to accounting software. Last week I opened my scraper repo, ran the tally against the new Chrome primitives, and cut eight of twelve scripts. Here's exactly what died, what survived, and which SaaS categories on your invoice list are on borrowed time.

The tally: 12 scrapers in, 4 scrapers out

Before the updates, my repo had twelve Puppeteer scripts in production. Each one was a small business — logging into a supplier portal, downloading invoices or receipts, extracting line items, pushing them to the client's books. After migrating to Chrome primitives on cooperating sites, four scripts remain. The other eight are archived.

Scraper category	Count before	Count after	Killed by
Portals with hidden JSON endpoints	5	0	WebMCP
PDF OCR + classification pipelines	3	0	Built-in AI (Gemini Nano)
Multi-step login/download/reconcile	0	0	Skills in Chrome (folded in)
Hostile portals (rotating selectors, captchas)	4	4	Nothing. Still Puppeteer + proxies.
Total	12	4

Zero code from me on the eight I killed. Google shipped the replacement into a browser update. The rest of this post walks through each primitive, what it actually replaces, and how to run the same audit on your own SaaS bill.

WebMCP replaces the "reverse-engineer the hidden JSON" scraper

WebMCP lets a site expose tools directly to a browser agent through a standard protocol — an API contract that lives inside the page. Instead of your agent parsing HTML, clicking buttons, and praying the DOM didn't change, the site declares what actions exist and what data comes back. On cooperating sites, this replaces every scraper whose job was to reverse-engineer the network tab and hit undocumented JSON endpoints.

Five of my twelve scrapers were doing exactly that. Here's what one of them looked like before:

// Old: Puppeteer chasing a hidden endpoint
await page.goto('https://vendor-portal.example.com/invoices');
await page.waitForSelector('#invoice-table');

// Intercept the XHR the table fires internally
const invoicesJson = await page.evaluate(async () => {
  const res = await fetch('/api/v3/invoices?range=90d', {
    credentials: 'include',
    headers: { 'X-CSRF-Token': window.__csrf }
  });
  return res.json();
});

// Handle pagination, 429s, DOM shape changes, session expiry...

That script broke on average every 6-8 weeks whenever the vendor rotated their CSRF pattern or added a rate-limit header. The WebMCP version, on a cooperating portal:

// New: ask the site what it offers
const tools = await chrome.webmcp.listTools(tabId);
// -> [{ name: 'list_invoices', params: { range: 'string' } }, ...]

const invoices = await chrome.webmcp.callTool(tabId, 'list_invoices', {
  range: '90d'
});

No DOM parsing. No CSRF chasing. No breakage when the vendor redesigns their UI, because the contract is the tool declaration, not the HTML. Every SaaS that priced itself on "we handle DOM changes for you" just lost its moat on cooperating sites.

What WebMCP does not fix

Sites that refuse to publish tools (any adversarial portal — banks, government, hostile suppliers).
Auth flows behind SSO providers that block automated user agents.
Anything that requires captcha solving.

Built-in AI collapses the OCR + classification middle layer

Gemini Nano now runs on-device inside Chrome. That means OCR, classification, entity extraction, and simple structuring happen locally with no API bill and no network hop. Three of my scrapers existed only to shuttle downloaded PDFs through a cloud OCR pipeline (roughly $0.015 per page at my volume) and then run a classifier to tag vendor line items. That entire middle layer collapses into a browser call.

The old flow, per receipt:

Download PDF (Puppeteer)
  → Upload to Google Document AI ($0.015/page)
  → GPT-4o-mini for line item classification (~$0.002/receipt)
  → Structured JSON to Postgres
Cost: ~$0.017/receipt × 1,800/month = $30.60/month, plus 2-4s latency per receipt.

The new flow:

// Chrome Built-in AI, on-device
const session = await window.ai.assistant.create({
  systemPrompt: 'Extract line items from receipts. Return JSON: [{vendor, date, amount_usd, category}]'
});

const pdfText = await window.ai.ocr.extract(pdfBlob);
const items = JSON.parse(await session.prompt(pdfText));

Cost: $0. Latency: 400-900ms per receipt on the client's laptop CPU. No data leaves the machine, which happens to be a nice side effect for accountants who don't want client receipts hitting a third-party OCR vendor.

The catch: Gemini Nano is small. It handles structured extraction and clean OCR well. It falls over on multi-column receipts, low-contrast phone photos, and anything requiring reasoning across multiple pages. For those, I still route to a cloud model. But that's maybe 8% of receipts, not 100%.

Skills chain the workflow, so RPA-style scripting goes away

Skills in Chrome let an agent chain multi-step workflows on a site — log in, navigate, download, reconcile — as a single reusable unit. This is the piece that replaces the "click bot" category: RPA tools, form-fillers, onboarding automation platforms. On any site that publishes a Skill (or that cooperates enough that you can define one), you stop writing selector-based scripts.

A skill definition for a bookkeeping workflow looks roughly like this:

name: monthly_vendor_reconcile
inputs:
  - vendor_id: string
  - month: string  # YYYY-MM
steps:
  - login_with_saved_credentials
  - navigate: /vendors/{vendor_id}/statements
  - download_statement: { month: {month} }
  - extract_line_items          # uses Built-in AI
  - reconcile_against: quickbooks.vendor_ledger
outputs:
  - reconciliation_report

That replaces roughly 180 lines of Puppeteer per vendor in my old repo. The agent knows how to handle intermediate failures (retry the download, re-auth if the session expired, flag the report if amounts don't tie). I don't write that logic anymore.

Skills need site cooperation to work reliably. On a hostile portal, a Skill will fail exactly the same way a Puppeteer script fails — selectors change, captchas fire, the account gets flagged. Which brings us to what survived.

What Chrome did not kill: the 4 hostile-portal scrapers

Four of my twelve scrapers hit portals run by companies that actively don't want to be scraped. Rotating selectors on every deploy. Captchas that trigger on the second request from any IP. User-agent fingerprinting. These sites will never publish a WebMCP tool declaration or a Skill, because their business model depends on friction.

Those four scripts still run on Puppeteer with:

Residential proxy rotation ($75/month for the pool I use)
Stealth plugin patches for headless detection
Manual captcha budget (~$4/month at current volume)
A dedicated retry queue because success rate hovers at 82-88%

Total operating cost for the four survivors: about $110/month, plus the maintenance tax of rewriting a selector or two every couple of weeks. That's the market that stays profitable for scraping vendors. It's smaller, uglier, and more industry-specific than the cooperating-portal market — but it's the one that actually pays a premium, because the alternative for the customer is a human doing data entry.

Which SaaS line items to cut this week

Three categories on your monthly invoice list just had their moat drained. If you're paying for any of these against cooperating sites, you're on borrowed time:

RPA bots that click through vendor portals — UiPath-lite tools, Automation Anywhere for SMB, low-code "click recorder" platforms typically running $80-400/month.
Form-fillers and onboarding automators — anything that pretends to be a user typing into a form.
Browser automation platforms that charge per run — Browse AI, Apify starter tiers, Zapier's browser actions add-on, custom scraping-as-a-service vendors.

Run this audit against your last three invoices:

List every SaaS labeled browser automation, web scraping, RPA, form filler, workflow bot.
For each, name the specific site(s) it hits.
Ask one question per site: does the site cooperate, or does it fight back?
Cooperating → migration candidate. Cancel in Q1 2026 once your Chrome primitive replacement is stable.
Hostile → keep paying. That's a real moat.

For the cooperating half, budget a week per replacement. WebMCP and Skills migrations in my repo took 3-6 hours each including tests. That's cheaper than one month of what most of these tools charge.

The distribution story nobody is telling

Every I/O recap is calling this "the agentic web," like it's a philosophical shift. It isn't. It's a distribution shift. Google put agent tooling inside three billion browsers overnight. The venture-funded browser-agent startups raising at billion-dollar valuations to build what Chrome now includes are cooked. So are the SaaS wrappers that priced themselves on solving DOM fragility.

The winners are the thin wrappers that solve the four scrapers Chrome can't replace — the ones aimed at industries where portals fight back. Bookkeeping suppliers in emerging markets. Regional healthcare portals. Legal filing systems. Local government. That's a smaller, less sexy market than "AI agents for everyone," but it's the one where a customer will pay $200/month for a working tool because their only alternative is a $25/hour data entry contractor.

Where bizflowai.io fits

The bookkeeping automations we run for SMB clients at bizflowai.io sit exactly on this line. On cooperating platforms — QuickBooks, Stripe, most modern SaaS accounting exports — we're already migrating client integrations onto WebMCP and on-device extraction, which drops OCR costs to zero and removes the per-run browser automation fees clients used to pay. On hostile portals — the regional supplier sites and legacy vendor systems that fight scraping — we keep the Puppeteer + proxy stack, because that's the only thing that works and it's the layer where an automation partner still earns its keep. The audit process above is what we run for a client before quoting any new engagement.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

Cutting LLM Token Spend 40% Without Losing Accuracy

lamingsrb — Thu, 23 Jul 2026 07:09:39 +0000

Cutting LLM Token Spend 40% Without Losing Accuracy

The first production LLM system I shipped burned through $8,400 in API costs its first month. The demo had cost me $12. Nobody warned me that the gap between "works in a notebook" and "works for 40,000 daily requests" is measured in five-figure invoices. Since then I've spent a lot of time on the boring, unglamorous work of getting that number down without letting quality slip. A recent paper from Writer describing their orchestration "harness" hit a nerve, because the techniques they systematize are the same ones I've been stitching together by hand for the last two years.

Here's what actually works, what the Writer approach adds, and where I'd start if you're staring at an AWS bill you can't defend.

The ROI paradox nobody warns you about

In the pilot phase, you route every request to the strongest model available. Claude Opus, GPT-5, whatever the current top-of-stack model is. It works. The demo goes well. Someone signs a contract. Then you deploy, traffic scales 100x, and your gross margin turns negative before the end of the quarter.

The Writer paper frames this well: enterprise AI economics break not because models are bad, but because teams treat orchestration as an afterthought. Every request gets the same treatment. Every prompt carries the same bloat. Every retrieval dumps the same context window. You are, in effect, paying premium pricing for premium reasoning on requests that a 7B model could handle in 200ms.

In my own systems, I've measured the split. Roughly 60-70% of production requests do not need frontier-model reasoning. They need retrieval, formatting, classification, or extraction. The remaining 30% is where you actually earn the token spend. Once you internalize that split, the optimization strategy writes itself.

Where the money actually goes

Before you optimize, instrument. I cannot stress this enough. Most teams I talk to have no idea where their tokens are going. They know the monthly bill. They don't know the per-endpoint, per-model, per-prompt-template breakdown.

Here's the breakdown I typically see in a mid-scale production RAG or agent system:

Component	Share of token spend	Usual bloat factor
System prompt (repeated every call)	15-25%	3-4x
Retrieved context (top-k stuffed in)	30-45%	2-5x
Conversation history	10-20%	2-3x
User query	2-5%	1x
Model output	15-25%	1.5-2x

The two fat targets are almost always retrieved context and system prompts. If you fix nothing else, fixing those two gets you to a 30% reduction on most systems. Add smart routing on top and you're at 40% or more, which lines up with what Writer reports and with what I've hit on my own projects.

Step 1: Route requests to the cheapest model that can actually do the job

This is the single highest-leverage change. Not every request is worth a Claude Opus call. My routing layer is a small classifier, sometimes a fine-tuned local Llama 3, sometimes just Haiku or GPT-5-mini with a tight system prompt, that decides which downstream model to call.

The taxonomy I use:

Tier 0 (local, free): intent classification, PII detection, safety filtering, simple extraction. Ollama with a quantized 3B or 7B model on the same box as the API. Zero API cost, sub-100ms.
Tier 1 (cheap API): structured extraction, summarization, formatting, single-turn Q&A over already-retrieved chunks. Haiku, GPT-5-mini, Gemini Flash. Roughly 1/20th the cost of the flagship.
Tier 2 (flagship): multi-step reasoning, code generation, tool use with planning, anything the customer actually judges you on. Claude Opus or GPT-5.

The router itself is dumb on purpose. A 40-line prompt, a few-shot rubric, and a hard fallback to Tier 2 if confidence is low. In one autonomous content pipeline I run, this split alone cut monthly spend by 34% and, counterintuitively, improved latency on the majority of requests because the cheaper models are also faster.

The gotcha: you must eval per tier. A model that scores 92% on your golden set at Tier 2 is not the same model at Tier 1. Build a small eval harness (I use a mix of Ragas for RAG-heavy paths and custom LLM-as-judge for open-ended outputs), and track accuracy per route. When Tier 1 accuracy drops below a threshold on a specific intent, bump that intent up.

Step 2: Compress prompts without breaking them

Prompt compression is where the Writer harness idea gets clever. The naive version, "just shorten the system prompt," works up to a point and then breaks in weird ways. The version that actually holds up in production is more surgical.

Three techniques I use, in order of leverage:

1. Separate static from dynamic. Your system prompt has two parts: the instructions that never change (persona, rules, output format) and the parts that change per request (user context, current date, retrieved facts). Cache the static part. On Anthropic's API, prompt caching cuts the repeated portion to 10% of the input token cost. On a 4,000-token system prompt hit 500,000 times a month, that's real money. OpenAI has automatic caching too, but you have to structure the prompt with the stable prefix first for it to trigger.

2. Distill the instructions. Most system prompts I inherit are 3,000+ tokens of accumulated fear. Every past bug fix added a "PLEASE DO NOT" clause. Every stakeholder demand added an "IMPORTANT:" section. Take your current prompt, run it through a compression pass with Claude ("rewrite this system prompt to preserve every behavioral rule but in 1/3 the tokens, then list what you changed"), then eval the compressed version against your golden set. If accuracy holds, ship it. I've cut prompts from 3,800 tokens to 1,100 with zero measurable regression on a customer-support agent.

3. Use LLMLingua-style compression on retrieved context. For long retrieved passages, a small model can compress the passage while preserving the tokens that matter for the query. Microsoft's LLMLingua is the reference implementation. In practice, for a RAG pipeline pulling 8 chunks of 500 tokens each, you can compress the total context by 40-60% with minimal answer-quality loss. The trick is to compress conditional on the query, not blindly.

Step 3: Retrieve less, but retrieve better

Stuffing top-k chunks into context is the laziest thing we do in RAG, and it's the biggest token bleed. If your k is 10 and each chunk is 400 tokens, you're paying for 4,000 tokens of context on every single request, most of which the model ignores.

What I do instead:

Hybrid search with RRF fusion. Semantic search alone misses acronyms, product SKUs, and exact-match terms. BM25 alone misses paraphrases. I run both in parallel over pgvector plus a Postgres full-text index, then merge with reciprocal rank fusion. Retrieval quality goes up, which means I can drop k.
Rerank aggressively. After the initial retrieval of 20-30 candidates, a cross-encoder reranker (Cohere Rerank, or a local BGE reranker) trims to the top 3-5 that are actually relevant. The reranker adds ~80ms but lets me drop from k=10 to k=4 without losing answer quality.
Dynamic k based on query type. A simple factual lookup often needs one chunk. A synthesis question might need five. I have the router predict the required breadth as a side output, and the retrieval layer adjusts. This alone dropped my average context size by 35% on one system.

The result: better answers on fewer tokens. Not a trade-off, a Pareto improvement, once you take the time to build it.

Step 4: Cap the output, and structure it

Output tokens are typically 3-5x more expensive than input tokens. And yet almost no team I audit sets max_tokens aggressively or forces structured output.

Concrete moves:

Force JSON output when downstream code consumes the result. Structured output kills the "here's your answer, and let me also add some context..." tail that models love. On some endpoints I've cut output tokens by 40% just by switching from prose to a JSON schema.
Set max_tokens to what you actually need. If your UI shows 3 bullets, don't let the model generate 15 and then truncate. Set max_tokens to something like 250 and design the prompt around it.
Stream and cut. For chat interfaces, stream the output, and let the client cut off when the answer is complete. Combined with a stop sequence, you can save 20-30% of output tokens on chatty models.

Step 5: Build a feedback loop so this doesn't decay

The uncomfortable truth about token optimization is that it decays. New features add new prompts. Retrieved context grows as your corpus grows. Somebody adds a "just in case" clause to the system prompt. Six months later, you're back where you started.

What I put in place on every production LLM system now:

Per-endpoint token dashboards. Grafana, one panel per endpoint, tracking input tokens, output tokens, cache hit rate, and model tier distribution. Anomalies trigger Slack alerts.
Weekly eval runs. Golden set of 200-500 examples per endpoint. Runs every Sunday. Regression on accuracy or cost triggers a ticket.
Prompt versioning in git. Every prompt is a versioned file. Changes go through review. This alone catches half the accidental bloat before it ships.

Writer's harness formalizes this loop, which is what makes the paper interesting to me. Most teams treat orchestration as glue code. Treating it as a first-class system, with its own eval, versioning, and monitoring, is what turns a 15% cost reduction into a 40% one.

What I'd do this quarter if I ran a production LLM system

If I were parachuted into a team burning money on GPT-5 for everything, here's my 30-day sequence:

Week 1: Instrument. Log input tokens, output tokens, model, and endpoint for every call. Build one Grafana dashboard. Find the top 5 endpoints by spend.
Week 2: Enable prompt caching on the top 5. Distill their system prompts. Ship. This alone is usually a 20-25% cut with no risk.
Week 3: Build a router. Start with a hard-coded rule table (endpoint X always goes to Haiku, endpoint Y always to Opus), then add a small LLM classifier for the ambiguous middle. Eval per route.
Week 4: Attack retrieval. Add reranking, drop k, add hybrid search where you don't have it. Compress context with LLMLingua on the longest passages.

By day 30 you're usually at 35-45% cost reduction with equal or better accuracy. It's not glamorous. There's no announcement post to write. But the difference between an LLM product that scales and one that gets shut down for burning cash is exactly this kind of work.

Closing thought

The Writer paper is a useful signal that orchestration is finally getting the attention it deserves. Foundation models are commodities now. The engineering around them, the routing, the compression, the retrieval, the eval loop, is where the real leverage lives, and where the margin between a healthy AI product and an expensive science project is decided.

If you're running an LLM system in production and the math isn't working, I'm happy to take a look. Get in touch at lazar-milicevic.com/#contact, or read more of what I've written about production LLM systems and agent architecture on the blog.

Building Agentic Workflows with Claude Code

lamingsrb — Thu, 23 Jul 2026 07:09:36 +0000

Building Agentic Workflows with Claude Code

I run Claude Code as the execution layer behind several of my autonomous systems, including the content and SEO machine that powers BizFlowAI ContentStudio. It writes, refactors, publishes, and cleans up after itself while I sleep. What surprised me over the last year is how much of the reliability came from workflow design, not from smarter prompts. Below are the patterns I keep reaching for, and the ones I stopped using after they burned me.

Why Claude Code became my default execution layer

Most agent frameworks give you an orchestrator that pretends to hold state. Claude Code gives you a shell, a filesystem, a real editor loop, and a model that can plan across many tool calls without losing the thread. That combination matters more than people admit. When an agent needs to read a config, run a script, inspect the output, patch a file, and re-run, you want the environment to be the source of truth, not a JSON blob in memory.

For me the practical wins are three: long-horizon tasks stay coherent past 40 or 50 tool calls, subagents can be spawned for isolated work without polluting the parent context, and failures leave real artifacts on disk that I can inspect the next morning. That last point is what makes unattended runs possible at all. If your agent only exists in RAM, you cannot debug it after the fact.

I still use LangGraph and my own orchestrators for hard-typed pipelines. But when the work involves code, files, shell, and judgment, Claude Code is where I go first.

Pattern 1: The plan file as durable memory

The single highest-leverage habit I picked up is forcing the agent to write and update a plan file on disk. Not the internal todo list. An actual PLAN.md in the working directory that persists across sessions, restarts, and human interventions.

Here is the shape I use:

# Task: Publish weekly SEO brief for cluster "RAG in production"

## State
- [x] Pull last 7 days of GSC data for cluster
- [x] Cluster queries by intent (informational / commercial)
- [ ] Draft brief with 3 candidate H1s  <-- current
- [ ] Score against existing posts (cannibalization check)
- [ ] Write final brief to /briefs/YYYY-MM-DD.md

## Notes
- GSC export at /tmp/gsc-2026-07-20.csv
- Skip "AI agents" head term, already covered
- Cannibalization threshold: cosine > 0.82 against embeddings/posts.jsonl

The rule I give the agent is simple: before any non-trivial tool call, read PLAN.md; after any state change, update it. This costs maybe 200 tokens per loop and buys me the ability to kill a run, restart it two hours later, and have it pick up exactly where it left off. It also gives me a diff to review when something goes sideways.

If you take one thing from this post, take that one.

Pattern 2: Tool orchestration by capability, not by API

Early on I made the mistake of exposing tools that mirrored the underlying APIs: create_post, update_post, delete_post, get_post_by_slug, and so on. The agent would happily call three of them in the wrong order and leave a half-published draft.

The fix was to design tools around capabilities the agent actually reasons about, and hide the sequencing inside them. Instead of five WordPress endpoints, I expose one publish_brief(slug, markdown, metadata) that internally handles idempotency, slug conflicts, image uploads, and category mapping. The agent thinks about "publish this thing" and the tool handles "how".

A rough hierarchy I keep in mind:

Layer	Example	Who owns the retry logic
Raw API	`wp.posts.create`	Nobody uses this directly
Capability	`publish_brief(...)`	Tool, with idempotency key
Workflow	`weekly_publish_cycle()`	Agent, via plan file

The agent should live at the workflow layer. It should never see raw APIs unless you are explicitly asking it to write integration code.

A second rule I enforce: every capability tool returns a structured result with status, artifact_path, and next_hint. The next_hint is optional freeform text the tool can use to nudge the agent (for example, "post created but featured image failed, run attach_image before announcing"). It sounds minor. In practice it cut my error-recovery loops roughly in half.

Pattern 3: Error recovery that does not spiral

The failure mode I see most often in agentic workflows is the death spiral: something breaks, the agent tries to fix it, that fix breaks something else, twenty tool calls later you have a mess and a burned context window.

Three guardrails I always add:

1. A retry budget per tool, not per task. Each capability tool tracks its own attempt count in a small state file. On the third failure of the same tool with the same arguments, the tool refuses to run and returns a blocked status. The agent is instructed to treat blocked as a signal to escalate, not to try harder.

2. A "diagnose before you patch" prompt. When any tool returns an error, the agent's system prompt requires it to:

Read the full error output
Write a one-paragraph hypothesis to PLAN.md
Only then propose a fix

This sounds like ceremony. It stops the agent from reflexively adding try/except around everything or deleting files it does not understand.

3. A hard budget on tool calls per phase. I set numeric limits like "no more than 15 tool calls in the drafting phase" and enforce them by having the agent count in the plan file. When it hits the limit, it stops and writes a status report. I would rather review a half-finished run at 8am than find out at noon that it burned through my Claude API budget looping on a broken regex.

For unattended runs, the escalation path is a message to a Slack channel and a JSON snapshot of the plan file plus recent tool outputs to S3. In practice I check it once a day and clear stuck jobs in a few minutes.

Pattern 4: Subagents for isolation, not for cleverness

Claude Code lets you spawn subagents. It is tempting to use them to parallelize everything. Do not.

The rule I converged on: spawn a subagent only when you need context isolation, not when you need speed. The main cases where it earns its keep:

Research spikes. When the agent needs to grep through a large codebase or read fifty files to answer one question, a subagent can do that and return a summary without polluting the parent context.
Risky refactors. A subagent can experiment on a branch, and if it fails, the parent never saw the failed attempts.
Untrusted content. If the agent is processing scraped web content, I run the extraction in a subagent so any prompt injection lives and dies in that isolated context. This one matters more every month.

What I do not do: use subagents to run steps of a linear workflow in parallel. The coordination overhead almost always costs more than it saves, and debugging becomes painful. If two things truly are independent, run them as separate top-level jobs, not as siblings under one parent.

Pattern 5: The unattended run checklist

An unattended run is not a normal run with the human removed. It is a different animal, and I treat it that way. Before I let any Claude Code workflow run overnight, it has to pass this list:

Idempotency. Every capability tool must be safe to call twice with the same input. I usually implement this with a content-hash key. If the same brief hash arrives twice, the second call is a no-op.
A dry-run mode. The whole workflow can run end-to-end with side effects disabled. I use this to test prompt changes without publishing junk.
Structured logging. Every tool call writes a JSONL record with timestamp, tool, args hash, status, and duration. I feed this into a small dashboard, but even grep on the file catches most issues.
A kill switch. A HALT file in the working directory that the agent checks at the top of every loop. If it exists, the agent writes a final status and exits cleanly. This has saved me from myself more than once.
A budget. Hard caps on tool calls, wall-clock time, and API spend. Enforced by the runner, not by the agent's good intentions.
A post-run report. At the end of every run, the agent writes a short report to runs/YYYY-MM-DD-HHMM.md summarizing what it did, what it skipped, and what needs human attention. This is the artifact I actually read in the morning.

The content loop that powers my own site follows this exact template. It runs on a schedule, produces a report, and only pings me when something is genuinely stuck. That is what "autonomous" looks like in practice: not zero human involvement, but human involvement that scales sub-linearly with volume.

A real prompt fragment

I get asked for prompts often. Here is a lightly edited slice of the system prompt for the publishing agent in my content loop. It is not clever. That is the point.

You are the publishing agent for the content pipeline.

Rules:
- Read PLAN.md before every non-read tool call.
- Update PLAN.md after every state change. Never delete history, only append.
- Never call a capability tool with args you have not first written into PLAN.md
  under "Next action".
- On any tool error, do NOT immediately retry. Write a hypothesis paragraph to
  PLAN.md under "Incidents", then propose ONE fix and try it.
- If the same tool fails 3 times with the same args, stop and write "BLOCKED:"
  followed by what a human needs to do.
- Check for a HALT file at the top of every loop. If present, write a final
  status to PLAN.md and exit.
- You have a budget of 40 tool calls for this run. Track the count in PLAN.md.

Style:
- Prefer capability tools over shell.
- Use shell only for read-only inspection or when a capability tool does not exist.
- Never edit files under /published/ directly. Use publish_brief.

The strict tone is deliberate. Claude Code responds well to explicit, enumerated rules, and it holds them across long runs. Loose "please try to" language degrades faster than I would like.

What I would do if you are starting now

If you are building your first serious agentic workflow with Claude Code, here is the shortest path I would take:

Start with one boring workflow you already do by hand. Not something ambitious. Something you understand cold, so you can tell when the agent is wrong.
Design your capability tools before you write any prompts. Half the reliability of the system lives in the tool boundaries.
Use the plan file from day one. Retrofitting durable state later is painful.
Instrument before you scale. JSONL logs and a post-run report will teach you more in a week than any framework choice.
Give the agent less power, not more. Narrow tools with clear contracts beat broad tools with clever prompts. Every single time.
Run it unattended only after it survives a week of supervised runs with zero human intervention. If you are still nudging it, it is not ready.

The teams I see struggling with agent projects are almost always the ones that started with a framework and worked backwards to the problem. The ones shipping started with the problem, wrote sharp tools, and let the model do what it is good at: reasoning across a well-defined surface.

If you are working on something like this and want a second pair of eyes, or you are trying to move an agent from a nice demo to something that runs on its own, I am happy to talk. You can reach me at lazar-milicevic.com/#contact, or dig around the blog for more field notes from systems I have actually put into production.

94 CVs Triaged in 11 Min for $0.31: n8n Beats Your ATS

lamingsrb — Thu, 23 Jul 2026 06:11:53 +0000

94 CVs, 11 Minutes, $0.31: The n8n Triage Your ATS Won't Touch

Ninety-four applications for one junior role over four days. A founder's Saturday morning gone. Two strong candidates missed because they were buried under referral spam and a "Dear Hiring Manager at [Wrong Company]" cover letter. That's the actual pain point in early-stage hiring — not sourcing, not scheduling, triage. And every ATS on the market wants $50/seat/month before it reads a single resume, then hides its ranker behind a black box you can't audit. Here's the five-node n8n workflow I built instead, the exact classifier prompt, and the spots where it breaks.

The five-node workflow that replaces the ranker

The whole thing is a Gmail trigger, an attachment parser, a Claude Haiku classifier, an Airtable log, and a switch node. No queue, no cron, no dashboard. Applications land in a shared Gmail with the label inbound-applications (set by a filter on the jobs@yourdomain address), the workflow fires on the label, and eleven minutes later the founder gets a Telegram ping for the candidates worth a real look.

Here's the node-by-node shape:

Gmail Trigger — watches the inbound-applications label. Fires on every new match. No polling guesswork.
Function (parser) — checks for PDF attachments first, falls back to email body text.
Claude Haiku classifier — structured JSON output, four criteria rubric.
Airtable "Applications" base — audit log for every single application, scored or not.
Switch — three branches by score: >=7 → Telegram, 4–6 → review queue, <4 → auto-archive + rejection draft.

The n8n workflow itself runs in milliseconds. The eleven minutes is Haiku API latency across 94 sequential calls. If you batch or parallelize, you can push that under three minutes, but for a hiring workflow the wall-clock doesn't matter — the founder isn't sitting there watching it run.

The parser is where 80% of tutorials quietly fail

Attachment parsing is the boring part everyone skips, and it's the reason most "AI resume screener" demos fall apart in production. Roughly 90% of applications arrive as PDF attachments. The rest paste the CV into the email body, or send a Google Docs link, or attach a .docx from an old Word install.

The naive path is pdf-parse → text → prompt. That breaks on:

Two-column resumes that interleave columns line-by-line (Name Skills Email Python Phone JavaScript)
LinkedIn PDF exports with private-use Unicode block characters where bullets should be
Scanned PDFs — someone printed their CV, scanned it, attached the scan. Zero extractable text.
Pages exports with weird font embedding that comes out as \u0001\u0002 garbage

Here's the function node I actually use:

const attachments = $input.item.binary || {};
let text = '';
let source = 'body';

for (const key of Object.keys(attachments)) {
  const att = attachments[key];
  if (att.mimeType === 'application/pdf') {
    const buffer = Buffer.from(att.data, 'base64');
    const parsed = await pdfParse(buffer);
    text = parsed.text.replace(/\s+/g, ' ').trim();
    source = 'pdf';
    break;
  }
}

if (!text || text.length < 100) {
  text = $json.textPlain || $json.snippet || '';
  source = text.length < 100 ? 'unparseable' : 'body';
}

return {
  json: {
    candidate_email: $json.from,
    subject: $json.subject,
    cv_text: text,
    parse_source: source,
    parse_length: text.length,
    thread_id: $json.threadId,
  }
};

The parse_source: 'unparseable' flag is the important part. Anything under 100 characters skips the classifier and goes straight to a manual-review branch. Scoring garbage text as 2/10 and auto-rejecting a scanned CV is the worst possible failure mode — you can't get that candidate back.

The classifier prompt (Claude Haiku, structured JSON)

The classifier is Claude Haiku because it's cheap and fits a scoring rubric perfectly. You do not need Sonnet for this. The prompt does one thing: score the CV against four criteria and return JSON. No prose, no candidate summary, no "let me walk through this resume" filler.

You are a resume screener for a {role_title} position.

MUST-HAVES (score heavily):
- {must_haves}

NICE-TO-HAVES (bump score modestly):
- {nice_to_haves}

RED FLAGS (deduct heavily):
- Wrong country when role is on-site in {location}
- Cover letter addressed to a different company
- Copy-paste generic application with no role reference
- Obvious keyword stuffing without context

SENIORITY: The role is {seniority}. A significant mismatch (e.g. staff engineer
applying to junior role) should reduce the score.

Return ONLY valid JSON, no prose:
{
  "score": <integer 0-10>,
  "reason": "<one sentence, max 20 words>",
  "human_check": <true|false>,
  "flags": ["<optional short tags>"]
}

CV TEXT:
{{ $json.cv_text }}

Set the Claude node to response_format: json and max_tokens: 200. Haiku returns in ~2–4 seconds per CV. Average CV parses to ~800 tokens. On the 94-CV batch:

Metric	Value
Total input tokens	~78,000
Total output tokens	~9,400
Wall clock	11 min
API cost	$0.31
Cost per CV	$0.0033

Compare that to $50/seat/month for the entry tier of a mainstream ATS. You'd need to hire for 200+ roles a year before the ATS math even starts to make sense on cost alone, and that's before you factor in the week you spend configuring it.

Routing: why score buckets beat a ranked list

The switch node has three branches, not a ranked list. Ranked lists sound smart in a demo and fall apart in practice, because they force a human to look at position 8 to decide if it's a maybe. Buckets give the founder a clear action per candidate.

Score >= 7 → Telegram ping. Candidate name, one-line reason, Google Drive link to the CV. Arrives in the same Telegram the team already uses. No dashboard, no unread counter, no login flow.
Score 4–6 → Airtable "review later" view. The maybes. Hiring manager batches through them once a week for ~15 minutes.
Score < 4 → Auto-archive + draft rejection. Gmail thread archived, polite rejection saved to Drafts. Never sent automatically. That's a human decision, always.

On the 94-CV run:

7 candidates pinged to Telegram
14 dropped into review-later
73 auto-archived with rejection drafts ready to batch-send

The founder spent ~20 minutes total across the four days. Not four hours on a Saturday. The two strong candidates that would've been buried under referral spam got surfaced within an hour of applying.

Where this breaks — and the human-review gates

Every automation tutorial that skips this section is selling you something. Here's what actually goes wrong and the mitigations I ship with every deployment.

Unparseable PDFs. Older exports, Pages files, scans. Fix: the parse_length < 100 fallback routes them to a manual-review Airtable view with a flag. Never auto-reject an unparseable CV.

Referrals. Your CTO forwards a friend's CV with "hire this person." The classifier has no context and scores it 5. Fix: a separate referrals Gmail label that bypasses the classifier entirely and pings the hiring manager directly. Referrals get a human read, always.

Senior roles. The four-criteria rubric is fine for a junior role where must-haves are concrete (specific stack, years of experience). For a senior or staff hire, the nuance matters and the cost of a false negative is enormous. Fix: for senior roles, invert the routing — score >= 4 goes to human review, only clear spam (<4 with red flags) gets auto-archived.

Rejection emails. Never let an agent send rejections. Draft only. A wrong rejection to a strong candidate is a permanent reputation hit that costs you far more than the 30 seconds it takes to hit "send" on a batch of drafts.

Prompt drift. If your role description changes, re-run the classifier against a labeled set of 20 CVs you've hand-scored. Compare against the model's scores. If it's more than 1.5 points off on average, tune the rubric before you trust the next batch.

Add a labeled test set to Airtable and re-score on every prompt change
Log every classifier response with human_check: true for weekly review
Rotate the sample the hiring manager reads — pull 3 random rejections a week and verify the reason held up

Why bizflowai.io helps with this

The Gmail-to-triage pattern is one of the workflows I ship for small teams every month — hiring inboxes, sales inbounds, support routing, refund requests. Same shape: a labeled inbox, a parser that handles the messy edges, a small classifier prompt with structured output, an audit log, and a router that respects human review gates. The reason it works isn't the model — it's the boring parts most tutorials skip: the unparseable-file fallback, the referral bypass, the draft-not-send rule for anything that touches a candidate or customer. If you want this built for your inbox with your rubric, that's the day job.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

40 Take-Home Tests Graded in 14 Min: The n8n Rubric Node

lamingsrb — Thu, 23 Jul 2026 06:11:50 +0000

40 Take-Home Tests Graded in 14 Min: The n8n Rubric Node

Forty GitHub links land in your inbox on a Monday. Somebody — usually the founder or a senior engineer — is about to lose a Saturday cloning repos, running tests, and writing feedback. At nine minutes per submission that's six hours, and by submission thirty the reviewer is tired and rounding everyone down. That's not hiring. That's fatigue-based rejection, and it's the reason most small teams either drop take-homes entirely or hire the wrong person.

Here's the workflow I built to fix it: five n8n nodes, 14 minutes total runtime for 40 submissions, about $0.60 in API cost. The whole thing hinges on one prompt change nobody talks about.

Why every "AI code reviewer" demo fails

Most automation tutorials skip take-homes because everyone asks the model the wrong question. They paste code into GPT and ask "is this good?" The model answers confidently, plausibly, and unreliably. It praises garbage. It rejects solid work. Ask it the same question twice and you get two different scores. So the industry decided take-homes can't be automated and moved on to booking calendar invites.

The fix is a reframing, not a bigger model. Language models are terrible at subjective judgment ("is this idiomatic Python?") and genuinely good at yes-no checks against fixed criteria ("does this repo contain a README file that describes setup steps?"). Take-home grading looks subjective, but 80% of it is actually the second kind of question. You just have to write the rubric that way.

Bad prompt: "Rate this code from 1–10."
Good prompt: "Does README.md contain a section describing how to install dependencies? Quote the exact lines. If no such section exists, score 0."

The second prompt is auditable. You can re-run it on the same repo six months later and get the same answer. That's the property you need for hiring decisions.

The 5-node n8n workflow

The pipeline is deliberately boring. Every node does one thing and hands off structured data. Nothing clever, nothing hard to debug.

Node 1 — Airtable Trigger. Candidate submits their GitHub URL through your application form (Tally, Typeform, whatever). A row lands in Airtable. Trigger fires.
Node 2 — GitHub API / Clone. Pulls the repo into a temp directory on the n8n host. Also fetches commit metadata via the REST API — you'll need commit timestamps and hashes for the rubric later.
Node 3 — Docker Execute. Spins up a clean, network-isolated container, installs dependencies, runs your pre-defined test suite. Captures pass/fail per test, runtime, stderr.
Node 4 — Rubric Scorer. LLM call (Claude Sonnet or GPT-4o-mini both work) with the structured rubric prompt. Returns JSON.
Node 5 — Airtable Write. Score, per-criterion breakdown, evidence citations, and a flag column written back to the candidate's row.

Here's the Docker execute node body — the actual command that runs each submission:

docker run --rm \
  --network=none \
  --memory=512m \
  --cpus=1 \
  --read-only \
  --tmpfs /tmp:rw,size=128m \
  -v "$REPO_PATH:/app:ro" \
  -w /app \
  node:20-alpine \
  sh -c "npm ci --silent && npm test -- --json > /tmp/results.json; cat /tmp/results.json"

Four things matter here: --network=none after install (candidate code cannot phone home), memory + CPU caps (an infinite loop won't take down your n8n box), read-only mount (the container cannot modify the repo you cloned), and a JSON test output so node 4 gets structured data instead of parsing terminal noise.

The rubric prompt (the whole game)

Everything above is plumbing. This is the part that actually decides whether the workflow works or produces nonsense. Four fixed criteria, 25 points each, 100 total.

You are grading a take-home coding submission against a fixed rubric.
You may ONLY score based on evidence you can quote directly from the
repo contents provided below. If you cannot cite a specific file path,
line number, or commit hash as evidence, the score for that criterion
is 0. Do not infer, do not assume, do not give benefit of the doubt.

CRITERION 1 (0-25): Does the code run end-to-end without errors?
  Evidence source: test_runner_output below.
  25 = all tests executed, exit code 0.
  0-24 = partial or failed execution. Quote the error.

CRITERION 2 (0-25): Does it pass the 3 functional tests in the spec?
  Evidence source: test_runner_output.
  Award 8 points per test passed, +1 if all three pass.
  Quote the test name and pass/fail status for each.

CRITERION 3 (0-25): Is there a README explaining setup and design?
  Evidence source: readme_contents below.
  25 = setup steps + design rationale present. Quote both sections.
  12 = one but not both. 0 = missing or under 100 words.

CRITERION 4 (0-25): Does the commit history show incremental work?
  Evidence source: commit_log below.
  25 = 5+ commits spread over 2+ days, each with a message describing
       a specific change. Quote 3 commit hashes + messages as evidence.
  0 = single commit, or all commits within a 60-minute window.

Return JSON:
{
  "criterion_1": {"score": N, "evidence": "..."},
  "criterion_2": {"score": N, "evidence": "..."},
  "criterion_3": {"score": N, "evidence": "..."},
  "criterion_4": {"score": N, "evidence": "..."},
  "total": N,
  "notes": "one sentence, factual only"
}

The line that kills hallucination: "If you cannot cite a specific file path, line number, or commit hash as evidence, the score for that criterion is 0." Without it, the model invents praise. With it, the model has to point at a line, which means it has to actually look at the repo. In 40 submissions I've had zero fabricated citations since adding that constraint. Before it, roughly 1 in 6 submissions got scored on things that didn't exist in the code.

The ≥90 guardrail: catching AI-written submissions

Any submission scoring above 90 gets automatically flagged for human review. Not because the workflow is broken — because near-perfect scores are the exact suspicious signal you want.

Two scenarios produce a 95+:

The candidate is genuinely strong.
The candidate pasted your spec into Claude or GPT and shipped the output.

Both look identical to the rubric. Neither is a problem the LLM scorer can solve, because a well-prompted model can produce code that clears functional tests, includes a README, and even runs git commit in a loop with plausible messages. What it can't fake convincingly is the shape of real development work over time. Thirty seconds looking at the commit log tells you which bucket the candidate is in.

Signals I check on flagged submissions:

Commit timing. 12 commits across 3 days with variable gaps = human. 8 commits inside 20 minutes at 2 a.m. = LLM dump run through a bash script.
Commit message style. "fix off-by-one in date parser" = human. "Implement feature", "Update code", "Fix bug" repeated = LLM default.
File-level churn. Real devs edit the same file 3–5 times as they iterate. LLM dumps write each file once and never touch it again.
README voice. LLMs love bulleted feature lists and the phrase "This project demonstrates". Humans write shorter, more specific setup notes.

This guardrail catches maybe 3–5 of every 40 submissions. Of those, 1–2 are legit strong candidates and the rest are LLM output. Thirty seconds of human eyeballs per flagged repo is a cost I'll take.

What 40 submissions actually looks like

Real numbers from the last hiring round I ran this on (senior full-stack role, Node/React take-home, 40 submissions):

Metric	Manual grading	n8n rubric workflow
Total wall-clock time	~6 hours	14 minutes
Human attention time	~6 hours	~30 minutes (top 8 only)
Cost	1 engineer-Saturday	$0.61 in API calls
Consistency (same repo scored twice)	±15 points	±2 points
Submissions actually clearing the bar (>75)	8	8 (same 8)
Hidden LLM submissions caught	1 (accidentally)	4 (via ≥90 flag)

The workflow didn't change who got hired. It changed how many hours got burned to find them, and it stopped candidate #35 from getting a worse read than candidate #3.

Things that will trip you up

Four failure modes I've watched people hit trying to build this themselves:

Running candidate code on your main box. Someone will submit a repo with rm -rf / in a postinstall script. Docker with --network=none and --read-only isn't optional. Do not skip it.
Writing tests after you've seen submissions. Define the three functional tests before the job posts. If you write tests after submissions arrive, you will unconsciously write tests that pass the candidates you already like. That's not grading — that's confirmation bias with extra steps.
Tweaking the rubric mid-round. If you change the prompt between candidate 10 and candidate 20, you've broken the consistency guarantee that made the workflow worth building. Version the prompt in git. Freeze it when the job posts.
Treating the score as a hiring decision. The rubric picks who deserves 30 minutes of real human attention. It doesn't pick who to hire. That's still a person's job.

Where this sits in a full recruiter stack

The rubric grader is one node in a larger pipeline. For a small team hiring engineers, the full flow I usually build looks like: intake form captures the applicant, an enrichment step pulls public GitHub and LinkedIn data, a resume screener does a first-pass fit check against the job spec, this rubric grader handles the take-home, and a scheduling agent books calls with candidates who cleared every gate. Each node is boring on its own. Chained together, they collapse a two-week hiring cycle into about three days of actual work.

How this looks when we build it for a client

Most of the hiring pipelines we ship at bizflowai.io start with exactly this take-home grader, because it's the step where founders feel the pain most acutely. We set up the Airtable base, the sandboxed Docker runner, the rubric prompt tuned to the specific role (a React take-home rubric and a data-engineering take-home rubric are not the same document), and the ≥90 human-review flag routing into whatever your team uses — Slack, email, a Trello column. Then we hand it over. It runs on your n8n instance, your API keys, your data. Nothing about the workflow is proprietary — everything in this post is what actually runs in production.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

I/O 2026 On A $600 Laptop: 240 Invoices, 2 Died

lamingsrb — Mon, 20 Jul 2026 06:11:18 +0000

I/O 2026 On A $600 Laptop: 240 Invoices, 2 Died

Google demoed three AI platform updates at I/O 2026 on stage machines with unlimited cloud budget and a keynote-tuned network. I ran the same three against 240 real client invoices on a $600 laptop and a home server with no GPU. Two of them are already dead to a small operator — here are the receipts.

The bench: one cheap laptop, one home server, 240 messy PDFs

The rig is deliberately unglamorous, because that's what a solo bookkeeper or a five-person agency actually owns. One $600 Windows laptop (16 GB RAM, integrated graphics, no discrete GPU) running Chrome and a WSL Ubuntu instance. One home server on the same LAN, also no GPU, used for the Python side of the pipeline. The dataset: 240 real invoices from a small invoicing client — a Monday-morning batch of scanned PDFs, phone photos of receipts, and a handful of Word-exported "invoices" that a normal small business considers acceptable.

Success criteria were fixed across all three runs:

Extract vendor name, line items, subtotal, tax, and total
Return structured JSON that a downstream script can load into the accounting system
Fail loudly (not silently) on unparseable pages
Finish the whole batch on the same hardware the client already owns

# The harness (simplified)
for pdf in batch/*.pdf; do
  ts=$(date +%s%3N)
  result=$(python run_extractor.py --engine "$ENGINE" "$pdf")
  echo "$pdf,$ENGINE,$((`date +%s%3N` - ts)),$result" >> results.csv
done

Same input, same harness, three engines: WebMCP, Skills in Chrome, and Gemini Nano in Chrome. Here's what actually happened.

WebMCP: it worked, and it quietly eats your margin

WebMCP is the update every recap channel led with, because the on-stage demo was genuinely slick — browser talks to remote tools, tools talk back, agent orchestrates the whole flow. On my machine, feeding it invoice pages through the reference client, it worked. That is not the problem. The problem is what it costs to keep working.

Measured on the 240-invoice batch:

2.1 seconds per call, round-trip, from a residential US connection
$0.008 per call at current pricing
Requires an always-on internet path to the remote tool endpoint
No local fallback when the network drops

Scale that honestly. A client processing 2,000 invoices a month at $0.008 a call is $16/month for WebMCP alone — sounds fine until you notice invoice extraction is usually 3–5 calls per document (page split, classify, extract, validate, retry). That's realistically $48–$80/month in per-call fees on top of whatever LLM you're already paying for. Not catastrophic in isolation, but it silently eats the margin the automation was supposed to create, and it grows linearly with the client's business.

The bigger issue is offline. Four of the six small businesses I work with have a hard offline requirement — either the field tech is in a basement with no signal, or the accountant will not let invoice data leave the physical machine. WebMCP breaks that on day one. Cool protocol, wrong tool for this audience.

Skills in Chrome: dead on arrival without an enterprise policy

Skills got the most stage time as "the future of browser-native AI." Here's the part that didn't make the keynote: to activate the tier they demoed, Chrome needs an enterprise policy pushed to the browser through admin controls.

If you run a company with a real IT department and a Chrome admin console, you push the policy and you're done. If you're a two-person shop, a solo bookkeeper, or a small marketing agency, you don't have an MDM. You are the IT department. I spent about an hour trying to install this the way a normal small-business owner would — profile-level toggle, extension, dev flag, anything — and got nowhere useful. The good tier stays invisible.

What I actually tried, in order

Standard install path from Chrome settings — feature not present
Enabling every related chrome://flags toggle — surfaces the API, blocks the useful capability
Registry-level policy on Windows to fake an "enterprise" install — activates it, but this is not a step a normal SMB owner will (or should) take
Same policy pushed via plist on macOS — same story

Skills may become genuinely useful once Google ships a consumer path. Today, at least in the current rollout, it is not shipping to the audience Google demoed it for. Skip until that changes.

Gemini Nano in Chrome: the one Google buried

Update three got the least stage time and is the only one that finished the job on the bench. Gemini Nano runs locally, inside Chrome, on the device. No round-trip, no per-call fee, no data leaving the laptop. I wrapped it in a ~90-line Chrome extension that reads the invoice page, chunks the text, and hands each chunk to the on-device model.

Measured on the same 240 invoices:

1.4 GB RAM steady-state while running
380 ms per line item (average across the batch)
Fully offline — Wi-Fi off, airplane mode on, still ran
$0.00 per invoice in inference cost
238 of 240 invoices extracted cleanly; 2 failed on badly rotated phone photos (same 2 that broke my old cloud pipeline)

// Chrome extension wrapper, trimmed
const session = await window.ai.languageModel.create({
  systemPrompt: "Extract vendor, line_items[], subtotal, tax, total as JSON."
});

async function extractInvoice(pageText) {
  const chunks = chunkByTokens(pageText, 1200);
  const results = [];
  for (const chunk of chunks) {
    const raw = await session.prompt(chunk);
    results.push(JSON.parse(raw));
  }
  return mergeInvoice(results);
}

The number that matters for a small business is the last one. The previous cloud pipeline cost about 1.1 cents per invoice all-in (OCR + LLM + orchestration). Nano dropped that to zero. Not "zero-ish." Zero marginal cost per document. For a client running a few thousand invoices a month, that is the difference between the automation paying for itself in week one and the founder cancelling another SaaS line item in month six.

The three updates, side by side

Metric	WebMCP	Skills in Chrome	Gemini Nano in Chrome
Finished the 240-invoice batch on target hardware	Yes	No (couldn't install)	Yes (238/240)
Latency per unit	2.1 s / call	n/a	380 ms / line item
Marginal cost per invoice	~$0.024–$0.040	n/a	$0.00
Works offline	No	n/a	Yes
Install path for a solo operator	Sign up, add API key	Requires enterprise policy	Enable flag, load extension
Data leaves the machine	Yes	Depends	No
Verdict for a 1–10 person business	Skip for high-volume batch	Skip until consumer rollout	Ship it this weekend

Two failures out of three isn't a hit piece on Google — it's a reminder that keynote applause and small-business fit are almost unrelated variables.

What to actually do with this if you run a small team

If any part of your workflow parses documents — receipts, invoices, contracts, intake forms, order confirmations — the practical move is the same regardless of which local model you pick. Push the model to where the data already lives instead of shipping the data to a cloud model.

The pattern that keeps working for my clients:

Local model first, cloud model as fallback. Nano handles the 90% of documents that are normal. A cloud call handles the 10% that are weird (rotated photos, multi-page contracts, non-English vendors). You pay cents per month instead of dollars per day.
Keep the orchestration boring. A Python script on WSL + a small Chrome extension is enough. You do not need an agent framework to extract five fields from a PDF.
Log every failure with the raw input. The two failures in my batch became the training signal for the fallback path.
Measure per-document cost weekly. If it drifts above $0.01/doc, something is wrong. Usually it's a retry loop.

The rough decision rule I use with clients

Under 200 documents/month → cloud model is fine, don't over-engineer
200–2,000/month → local-first with cloud fallback (the Nano pattern above)
Over 2,000/month → local-first is not optional, it's the only path where the numbers work

The louder an AI platform announcement is at a keynote, the less likely it ships value to a ten-person business in the same quarter. The winners for small operators are the quiet, local-first updates that run on hardware you already own with no contract, no policy, no credit card on file. Two of the three I/O 2026 updates fail that test. One passes. Bet accordingly.

Where bizflowai.io fits in this

The invoice batch above is the same shape of work I've been wiring up for clients through bizflowai.io — document intake pipelines that run local-first on the hardware the client already owns, with a cloud fallback only for the edge cases that actually need it. The point isn't the model, it's the plumbing: a small extractor, a validation layer that flags the 2-out-of-240 that need human review, and a boring hand-off into whatever accounting or CRM system the business already runs. When Nano-class models get better, we swap the extractor and the rest of the pipeline doesn't move.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.

I/O 2026 p99 Test: 1,847 Invoices, One Feature Timed Out

lamingsrb — Mon, 20 Jul 2026 06:11:15 +0000

I/O 2026 p99 Test: 1,847 Invoices, One Feature Timed Out 4.2%

Google's I/O 2026 keynote demoed three AI platform updates and every single one ran under 300ms on stage. I wired all three into a live invoice pipeline, logged p50/p95/p99 for seven straight days, and 77 invoices never returned. Here's the tail latency the keynote didn't show you.

The setup: three features, one production pipeline, real customers

I run an invoice generation tool used by small business owners — the workflow is "click button, wait, get PDF, or get angry." That's the perfect test bed for I/O 2026's three headline updates because it's latency-sensitive, transactional, and the users are non-technical. If it breaks, they don't file a bug report. They generate a duplicate and email support.

The three features under test:

WebMCP — browser-side agent calls, the Model Context Protocol wired into Chrome so a page can invoke agent tools without a server round-trip. Got the most stage time.
Built-in AI (Gemini Nano on-device) — inference running locally in the browser. Boring segment. Barely any applause.
Skills in Chrome — pages ship reusable agent capabilities that other pages can load and invoke.

I routed every incoming invoice request through a round-robin dispatcher so each feature got roughly a third of live traffic. Same input schema, same output requirement (a rendered PDF), same customers hammering it during business hours. Seven days. 1,847 requests. Paying users, not synthetic load.

The dispatcher looked like this — small, deliberately dumb, no retries so I could see raw failure:

FEATURES = ["webmcp", "builtin_ai", "skills"]

async def dispatch(request):
    feature = FEATURES[request.id % 3]
    t0 = time.perf_counter()
    try:
        result = await asyncio.wait_for(
            run_pipeline(feature, request), timeout=8.0
        )
        latency_ms = (time.perf_counter() - t0) * 1000
        log(feature, latency_ms, status="ok")
        return result
    except asyncio.TimeoutError:
        log(feature, 8000, status="timeout")
        raise

8-second hard timeout. That's already generous for a click-to-PDF flow. Anything past 3 seconds and users start clicking again.

WebMCP: beautiful p50, catastrophic tail

WebMCP was the star of the keynote. It's also the feature I would not ship into a customer-facing path today.

The p50 is genuinely good — 280ms, right in line with the stage demo. If I'd stopped measuring at the median I'd have written a glowing recap. But the distribution has a tail like a comet:

Metric	WebMCP
p50	280ms
p95	2,100ms
p99	6,400ms
Timeouts	77 / 1,847 (4.2%)

Under concurrent load — meaning three or four SMB users generating invoices in the same minute — p99 collapsed to 6.4 seconds and 4.2% of requests never returned at all. In a real SMB workflow that means: user clicks generate, sees a spinner, waits, clicks refresh, generates a duplicate invoice with a new number, then calls support to figure out which one to void.

The failure signature was consistent. Concurrent calls into the browser-side MCP transport queued behind each other and eventually the transport dropped the connection. This isn't a Chrome bug — it's the current shape of the WebMCP transport under contention. It'll improve. It's not there yet.

When WebMCP is safe to use

Prototypes and internal tools where one user hits it at a time
Async workflows where a 6-second occasional delay doesn't cascade
Anything where you can queue and retry server-side without user visibility

Do not put it on the click path.

Built-in AI (Gemini Nano): the update nobody clapped for

Built-in AI got the shrug segment of the keynote. On stage it clocked 220ms and everyone moved on to the next slide. In production over seven days, it was the only feature that held up under real traffic.

Metric	Built-in AI (Gemini Nano)
p50	340ms
p95	780ms
p99	1,200ms
Timeouts	0 / 1,847

Zero timeouts. p99 sits at 3.5x p50, which is a normal, healthy distribution. The reason is architectural: on-device inference doesn't care about network jitter, edge congestion, regional outages, or Google's cloud load. Once the model is warm in memory, it's a local function call.

The tradeoff is model size — Nano is not GPT-4. For invoice generation the workload is narrow: parse a natural-language line item, normalize a currency string, classify an expense category. Nano handles all of that. If your workload is "write me a 2,000-word landing page," this is not your feature.

For SMB software specifically — where the AI touch is usually classification, extraction, or short generation — Built-in AI is the update you can ship this quarter without an asterisk.

Skills in Chrome: the cold-start trap

Skills in Chrome looked fine at first glance. p50 of 410ms, p95 of 1.4 seconds. Both acceptable. But the aggregate hid the real problem, which only showed up when I broke the data down by session position:

Session request #1: p50 = 2,510ms  (cold start)
Session request #2: p50 = 390ms
Session request #3+: p50 = 340ms

The first request in any new session paid ~2.1 seconds of cold-start overhead while the skill initialized in the page. For a power user hitting the tool fifty times a day, that cost amortizes to essentially zero. For a small business owner who opens the app twice a week to run payroll invoices, every single session feels broken.

This is the classic mismatch between how developers test features and how SMB users actually behave. Devs open the app, hit it a hundred times in a dev loop, and the cold start disappears into the noise. Real SMB traffic is bursty and infrequent — the majority of sessions are the first request. Cold start dominates.

Rules of thumb for cold-start sensitive features

If p50 request-#1 is more than 4x p50 request-#5+, you have a cold-start problem
Infrequent-user products (payroll, taxes, weekly reports) feel cold start harder than daily-use products
A background pre-warm on page load can hide it, but costs battery/CPU on mobile

The two-rule filter I now run before shipping any new platform feature

Two heuristics from this test that would have kept me from shipping WebMCP into a client invoice flow. Both are cheap to check.

Rule 1: p99 must be less than 5x p50 under real concurrent load. If it's not, the feature has a tail-latency problem that will hit your users during exactly the moments they care most (peak business hours, month-end). WebMCP's ratio was 23x. Built-in AI's was 3.5x.

Rule 2: Cold-start on session request #1 must be under 1.5 seconds. If it isn't, your low-frequency users will experience the app as broken even though your dashboards look fine, because your dashboards are averaging warm requests.

Here's the shadow-test pattern I now use for any new platform feature — route 5-10% of live traffic through it in parallel with your existing path, log percentiles, don't user-facing the results for at least a week:

async def shadow_route(request):
    # Always serve from the known-good path
    primary = asyncio.sessions_spawn(current_pipeline(request))

    # Shadow 10% through the new feature, discard result
    if random.random() < 0.10:
        asyncio.sessions_spawn(measure_only(new_feature_pipeline, request))

    return await primary

The shadow task's result is thrown away. You're only recording latency and timeout rate. No user is exposed to the new feature's failures. After seven days you have a real distribution, not a keynote number.

Why the keynote ranking is inverted from production value

Google spent the most demo minutes on WebMCP, moderate time on Skills, and treated Built-in AI as a footnote. On real SMB workloads over a real week, that order is exactly backwards:

Feature	Keynote time	Production readiness (SMB)
WebMCP	High	Not yet — 4.2% timeout rate
Skills in Chrome	Medium	Conditional — fine for power users, breaks infrequent-use apps
Built-in AI	Low	Ship it

This isn't a knock on Google. Stage demos optimize for wow — a new agent capability with a compelling narrative will always beat "a smaller model runs faster locally." But your customers optimize for the request that doesn't time out at 4pm on a Thursday. The tail is where they live.

Every time a major platform ships a batch of features, the loudest one gets the recaps and the quiet one gets ignored. Nine times out of ten the quiet one is the durable pick, because it's usually the one that removes a network hop or a dependency, not the one that adds a new abstraction.

Why bizflowai.io helps with this

Most of the invoice, lead-follow-up, and CRM-sync automation I ship for SMBs at bizflowai.io runs on the boring pick — a small model, close to the data, with a hard timeout and a fallback path. Every new platform feature gets a seven-day shadow test against real client traffic before it touches a customer's click path, and the results get logged with p99 alongside p50 so we know what the tail actually looks like, not what the demo promised.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.