Daniel Kim

Posted on Aug 18

Kane CLI's Real Customer Isn't the QA Engineer. It's Claude Code and Cursor, Reading NDJSON.

#testing #ai #cli #devtools

Somewhere in the last two years, "write a test" quietly turned into a different job. It used to mean opening a .spec.ts file and hand-rolling a selector chain that would rot the moment a designer renamed a data-testid. Now a growing share of the code showing up in pull requests is written by an agent — Claude Code, Cursor, GitHub Copilot, OpenAI Codex — and none of those agents can tell you, on their own, whether the button they just wired up actually opens the modal in a real browser. They can read the DOM they generated. They can't watch it run.

Kane CLI, from TestMu AI (the company formerly known as LambdaTest), is a bet on exactly that gap. It's a terminal-first browser and mobile testing tool where you describe what should happen in plain English and it drives a real Chrome session to check. That pitch — "natural language testing" — is not new; tools like mabl and various Playwright wrappers have chased it for years. What's specific about Kane CLI, and worth a closer look than the launch copy invites, is who it's actually built for. Read the flag list and the answer isn't "QA engineers." It's coding agents, reading structured output, deciding what to do next without a human in the loop.

What it actually does

Point Kane CLI at an objective — "log in as admin, open the billing page, confirm the plan shows Enterprise" — and it opens a real Chrome browser, works out the steps, and returns a verdict: pass or fail, backed by evidence. No selectors, no page-object classes, no custom DSL to learn. Under the hood it can push through up to 50 steps per flow, retrying and re-interpreting when the page doesn't match what it expected.

It ships as an npm package, @testmuai/kane-cli (registry entry), installable via npm, Homebrew, or a shell script, and needs Node 18+ and a locally installed Chrome. Mobile testing against an iOS Simulator or Android Emulator is also supported, but only on macOS with Apple Silicon plus Xcode or Android Studio — Windows and Linux users get browser testing only.

The repository — LambdaTest/kane-cli on GitHub, Apache-2.0 licensed — sat at 237 stars and 24 forks at the time of writing, with 71 open issues and zero merged pull requests. The npm package's first version was published April 16, 2026; the latest, 0.8.3, landed August 14, 2026 — four days before this piece — and the registry lists 49 published versions in that four-month window. That's a release roughly every two and a half days. This is very young, very actively churning software, still pre-1.0, and none of the launch material foregrounds that.

How it works

Kane CLI deliberately ships three interchangeable front doors onto the same engine, and the split is the most interesting design decision in the product.

Interactive TUI (kane-cli --tui) opens a full terminal UI for a human to author and iterate on a flow against a live browser session — this is the "QA engineer at a keyboard" mode.

Headless CLI (--headless) runs the same flow without a display, meant for shell scripts and CI pipelines where nobody is watching.

Agent mode (--agent --headless) is the one that explains the product's real shape. It emits structured NDJSON instead of human-readable terminal output, and TestMu AI explicitly documents that Claude Code, Cursor, OpenAI Codex, Gemini CLI, GitHub Copilot, and Google Antigravity can read this output natively to decide their own next move. There's even a packaged skill (npx @testmuai/kane-cli-skill) to drop straight into those agents' tool configs. This is not a testing tool that happens to have a CLI. It's a verification API for AI coding agents that happens to also have a UI for humans.

A few other pieces round out the workflow:

Testmd format — test flows are stored as committable, replayable markdown files with caching, so a flow lives in your repo like any other source file instead of a binary recording locked inside a vendor's dashboard. That matters for code review: a teammate can read a diff on a test the same way they'd read a diff on application code, rather than clicking through a recorded session to understand what changed. Caching means a flow that hasn't changed doesn't re-spend LLM interpretation on every run, which is also a direct lever on the credit cost discussed below.
Evidence packs — every run seals a .evidence file containing the test definition, per-step screenshots (including annotated copies of what the agent actually clicked), browser console and network logs attributed to each step, and failure records. This is the accountability layer: when an LLM decided a click landed on the right element, the evidence pack is how you check its work after the fact, instead of taking the pass/fail verdict on faith.
DevTools Assertions — added in a mid-June 2026 update (per the Business Wire announcement), these let a flow assert on network responses, console/JS errors, performance metrics, cookies, and local storage, all still described in natural language rather than code. Before this, a Kane CLI flow could only judge what was visible on screen; DevTools Assertions push it into the same territory Playwright's page.on('response') or page.on('console') hooks cover, but phrased as an objective rather than an event handler.
Assurance system — kane-cli context extracts use cases from requirements documents, design tests turns them into requirement-linked tests, and cover reports what's proven versus what's still owed. This is aimed less at individual developers and more at teams that need to show an auditor or a compliance process that stated requirements actually have test coverage — the kind of traceability matrix that regulated industries (fintech, healthcare, anything ISO- or SOC2-adjacent) currently maintain by hand in spreadsheets next to their test suite, not inside it.
Variable substitution with masking — parameterized objectives can carry secrets (passwords, tokens) without printing them into logs or evidence packs in plaintext, so the same flow can run against staging and production credentials without the credentials themselves ending up committed to a testmd file.
Any flow can be exported to a native Playwright script, which matters more than it sounds — it's the tool's built-in exit ramp for teams that don't want to be permanently dependent on it. Generate the suite with natural language, export once it stabilizes, and hand-maintain the Playwright code from there if the LLM interpretation layer stops earning its keep.

What changed versus Playwright, Selenium, and its own predecessor

The baseline it's replacing is familiar to anyone who has maintained an end-to-end suite: Playwright or Selenium scripts built around CSS/XPath selectors that break the moment a <div> gets a new class name, requiring someone to go back in and patch locators. Kane CLI's core wager is that describing intent in English and letting an LLM re-locate elements each run — with "auto-healing" when the page has changed — removes that maintenance tax entirely.

It also explicitly positions itself against Playwright MCP, Microsoft's MCP server that exposes Playwright actions as tool calls for an LLM to invoke one at a time. TestMu AI's framing is that MCP-style tool calls still require the calling agent to plan each low-level action, whereas Kane CLI takes a whole natural-language objective and handles the step-by-step planning, retrying, and healing internally, returning only a final verdict plus evidence.

There's also a company history worth noting: TestMu AI's earlier product was KaneAI, a cloud-hosted, browser-based "end-to-end testing assistant" pitched at QA teams working through a web dashboard. Kane CLI's own npm description still reads "KaneAI Terminal UI." In other words, this isn't a green-field bet — it's the same underlying AI testing engine getting a second interface, moved from a web dashboard into a terminal specifically because that's where coding agents now live. A browser-based dashboard is a dead end for an AI agent: there's no NDJSON stream, no piped stdout, nothing an agent framework can consume without screen-scraping a GUI meant for a human. Rebuilding the product as a CLI wasn't a stylistic choice, it was a prerequisite for the "coding agent verifies its own work" use case to exist at all.

That's a broader pattern worth watching across developer tooling right now: as more of a codebase's daily traffic comes from agents rather than humans clicking around a GUI, vendors that started as SaaS dashboards are re-platforming the same core engine onto CLI/NDJSON interfaces to stay reachable by that traffic. Kane CLI is a clean, self-contained example of the pattern rather than an isolated oddity — the same forcing function (agents can't click a dashboard) is going to hit every dev tool that currently assumes a human is the one operating it.

Why developers should actually care

Maintainability is the real sell. If auto-healing works as advertised, it removes the single biggest recurring cost of browser test suites — selector rot — at the price of trusting an LLM's judgment about "which element did the user mean."

Cost is where the model changes shape most sharply. Playwright and Selenium are free once written; the machine time to run them is the only marginal cost, and it barely moves whether the suite has 10 tests or 10,000. Kane CLI runs on a credit system instead — the free tier gives 200 credits every 30 days, Starter is $19/month for 2,000 credits, Pro is $99/month for 10,000. Every browser step an LLM has to interpret costs a credit, and a flow that needs several of its 50 possible retries to converge burns more credits than one that resolves cleanly on first try. Run the free tier's math forward: 200 credits a month is a handful of short smoke-test flows, not a full regression suite — anyone piloting this beyond a proof of concept should expect to land on a paid tier fast. That's a genuinely different cost model from anything in the Playwright/Selenium world: your test suite's monthly bill now depends partly on how confidently your app's UI matches what the model expected, which means a UI redesign doesn't just cost engineering time to update tests — under this pricing model it can directly cost more in credits too, since more retries get consumed while the model relearns the new layout.

Lock-in risk is lower than it looks, because of the Playwright export path — you can treat Kane CLI as a way to generate a first draft of a suite and then own the resulting .spec.ts files outright. That's a meaningfully different posture than tools that only let you run inside their platform.

Security gets a real answer (variable masking for secrets in flows and evidence) but also a new question the launch material doesn't raise: an evidence pack is, by design, a bundle of screenshots, console output, and network logs from every step of a test run. If a flow touches a page with real customer data — which staging environments seeded from production sometimes do — that's now sitting in a .evidence file that may get uploaded to TestMu AI's cloud dashboard or committed alongside test code. Worth a policy decision before this goes anywhere near a production-adjacent environment.

Latency isn't heavily documented, but a flow that has to send natural-language objectives to an LLM at each decision point, wait for a real Chrome page load, and potentially retry up to 50 times is inherently slower per-step than a compiled Playwright script executing pre-resolved selectors. TestMu AI's own marketing cites full flows resolving in "about two minutes," which is fine for a smoke test but a different order of magnitude than a Playwright suite that runs hundreds of assertions a second.

Practical use cases

Agent self-verification: wire Kane CLI's agent mode into a Claude Code or Cursor workflow so that after the agent edits a UI component, it runs a Kane CLI flow against a local dev server and reads the NDJSON verdict before opening a PR — closing the loop between "code compiles" and "feature actually works" without a human needing to manually click through the change first. This is the use case the flag names (--agent, the packaged skill, the six explicitly supported agents) make clear the product was built around, even though the marketing copy still talks mostly to a QA audience.
Smoke tests without selector maintenance: teams tired of a Playwright suite's weekly locator-fixing tax can hand Kane CLI the handful of critical user journeys (login, checkout, core CRUD flow) and let auto-healing absorb minor UI churn, reserving hand-written Playwright for the tests where determinism actually matters.
Requirements-to-coverage tracking: the context / design tests / cover pipeline is aimed at teams — likely regulated ones — that need to demonstrate stated requirements map to actual test coverage, not just that tests exist. Feed it a requirements doc, get back generated tests linked to specific requirement IDs, and a report of what's proven versus what's still owed — a workflow that today usually lives in a separate spreadsheet nobody keeps in sync with the actual suite.
CI gating with evidence: headless mode plus the sealed .evidence artifact gives a debuggable trail (screenshots, console, network) attached to every CI failure, without hand-instrumenting Playwright's trace viewer or wiring up a separate video-recording step.
Onboarding a legacy app to E2E testing: teams with zero existing browser test coverage — common for older internal tools — can get a first pass of smoke coverage in plain English far faster than writing selector-based Playwright from scratch, then export to Playwright once the flows stabilize and hand them to the team that owns long-term maintenance.

Limitations the launch page doesn't lead with

Chrome only. No Firefox, no Safari/WebKit. Anyone needing real cross-browser coverage — which is most of what Playwright and Selenium exist for — isn't covered.
Mobile testing is macOS-Apple-Silicon-only, gated behind Xcode or Android Studio. Windows and Linux teams get zero mobile story.
71 open issues against 0 merged PRs, on a repo with 237 stars, is a governance signal worth reading literally: either the maintainers haven't opened the review process to outside contributions yet, or issue triage is behind intake. Either way, "open source" here currently means "source visible," not "community-maintained."
Pre-1.0, fast-churning software. 49 releases in four months is a healthy sign for a team iterating quickly, and a caution sign for anyone expecting API stability in a CI pipeline they don't want to babysit.
Retries are not free. The credit-based pricing means the tool's own resilience mechanism (up to 50 retries) is also its own cost multiplier when a flow struggles — a flaky app doesn't just cost you engineering time under this model, it costs you a metered resource.

Where it sits against the field

Against Playwright and Selenium: Kane CLI trades explicit, deterministic selectors for LLM-interpreted intent, in exchange for lower maintenance and less code — but it inherits the questions any LLM-in-the-loop system carries: given the same English instruction six months from now, against a redesigned UI, will it reliably click the same logical element? Auto-healing is the proposed answer; it's also just an LLM making a judgment call under a different name, and unlike a broken selector — which fails loudly and points at a specific line — a mis-healed step can silently click the wrong-but-plausible element and report a false pass.

Against Playwright MCP and similar tool-call-based agent-browser setups: Kane CLI's bet is that a single natural-language objective with internal planning beats exposing granular tool calls (click, type, navigate) to the calling agent one at a time. With MCP, the calling LLM plans every low-level action and can be interrupted or redirected mid-flow; with Kane CLI, the planning happens inside the tool and the caller only sees the final NDJSON verdict plus the evidence pack. That's a reasonable design opinion — it offloads planning cost from the calling agent's context window — but it also means more of the decision-making happens inside a black box you can't easily inspect step by step while it's running, only reconstruct afterward from the evidence.

Against no-code AI testing platforms like mabl: Kane CLI's differentiator is being CLI-native and git-friendly (testmd files, Playwright export) rather than a hosted recorder-and-dashboard product, which matters if your team already lives in the terminal and version control rather than a SaaS UI. Kane CLI's published $19–$99/month tiers are also unusually transparent for this category — most no-code AI testing platforms quote enterprise pricing on request — though that transparency comes with the trade-off that a self-serve credit model has no built-in cross-browser cloud grid the way hosted platforms typically bundle one in.

An independent read

The most honest way to describe Kane CLI is that it's hedging on an unresolved question in developer tooling: will AI coding agents become dev tools' primary users before humans stop needing a GUI? Rather than pick a side, TestMu AI shipped one engine with two front doors — a TUI for people, NDJSON for agents — and named six specific agent integrations up front. That's a smart positioning move regardless of how the underlying reliability question shakes out.

But the reliability question is still open, and the project's own numbers say so more clearly than its marketing does. A tool whose core promise is "you don't have to think about brittleness anymore" is shipping with auto-healing and 50-step retries baked in — which is itself an admission that the underlying browser interactions are not deterministic. That's not a flaw unique to Kane CLI; it's the same trade-off every LLM-driven automation tool makes. The difference is that here it's wrapped in a testing product, where the entire cultural expectation — dating back to xUnit — is that the same input produces the same result every time. Evidence packs exist specifically to compensate for that expectation gap: you can't fully trust the verdict, so you get an audit trail instead.

For a young repo (four months old, pre-1.0, 71 open issues, zero merged community PRs) that's a reasonable place to be. It's not yet a place to be for anything customer-facing or compliance-critical.

Who should try it, wait, or skip it

Try it now if you're already running Claude Code, Cursor, or Codex against a real codebase and want a cheap way to have the agent verify its own UI changes before you review them — agent mode plus evidence packs is a genuinely useful missing piece there. Also worth a look if your current Playwright suite's biggest cost is selector maintenance on low-stakes smoke tests.

Wait if you need multi-browser coverage, Windows/Linux mobile testing, or a stable API surface you're not prepared to re-pin every couple of weeks. Watch the open-issues-to-merged-PRs ratio over the next couple of quarters as a proxy for whether the project is maturing or just accumulating backlog.

Skip it if you're deep in a mature, low-flake Playwright or Cypress suite with no appetite for credit-metered LLM costs, or if your test data ever touches anything you wouldn't want captured in a screenshot-and-log evidence bundle.

Kane CLI is a small, concrete example of a bigger shift: dev tools are starting to ship two separate interfaces — one for a person, one for a coding agent reading structured output — and deciding which one is actually the primary customer. If verification, not code generation, becomes the bottleneck in agent-driven development, tools built explicitly for agents-as-users like this one are going to multiply fast.

Discussion: if your CI pipeline started gating merges on an LLM's "pass" verdict instead of a deterministic assertion, what would you need to see in the failure evidence before you'd trust it enough to block a deploy on it?

Sources:

DEV Community