DEV Community

Commoditech
Commoditech

Posted on Originally published at commoditech.pl

Grok CLI and terminal agents: 90,000 tokens before the test says 'wrong'

#ai #cli #mcp #llm

Originally published on commoditech.pl.

Give a terminal agent a repository and the instruction "fix this test", and the loop does not end after one model call. Every step appends a search result, a file window and some stdout to the context.

Take a 2,000-token prefix (instruction, AGENTS.md, the task), 1,000 tokens of new observation per step, and no prefix cache. Twelve steps send 90,000 input tokens. One call with the same prefix and one observation sends 3,000. That is a ratio of 30. The last three steps cost 36,000 tokens, 40% of the bill; the first three cost 9,000, or 10%.

You can redo that sum on paper. It is not an invoice from a named rollout. But it is what the invoice looks like when nobody set a step cap and the loop stops when the model says "done" instead of when a test goes green.

The second risk matters more than the tokens. An agent that reads an issue, a README or a tool result also acts on whatever a third party wrote there, and it has a shell underneath. AgentDojo (Debenedetti et al., arXiv:2406.13352) has 97 tasks and 629 security test cases. Models fail some tasks even when nothing is attacking them, and a prompt injection hidden in tool output breaks some security properties and leaves others intact.

So a leaderboard score combines three things: the model, the scaffold, and how clean the test set is. The API bill is the sum of a growing context. What you can control is a stop condition tied to a test, plus a list of tools the agent may call. The name of the binary on your $PATH is none of these.

1. Anatomy: host, loop, MCP

A terminal agent is a loop, not a chat. The host keeps the history, calls the model, runs an action and appends the observation. Grok CLI is one such host. xAI documents it as Grok Build, and the command is grok (docs, product page). The docs describe:

  • an interactive TUI and a headless mode (grok -p "...");
  • a plan before edits, and subagents;
  • a session per git worktree (grok -w);
  • AGENTS.md, skills, hooks, plugins and MCP servers;
  • model selection with -m and grok models;
  • grok inspect (or grok inspect --json), which shows the rules, skills, plugins, hooks and MCP servers discovered in the current directory;
  • authentication by browser login, grok login --device-auth on remote machines, or the XAI_API_KEY environment variable.

The official install is a curl … | bash one-liner; there is also an npm package, @xai-official/grok.

The flags that matter most for the rest of this article are the control flags in the CLI reference:

# cap the loop, pick a sandbox profile, restrict tools, set permission rules
grok -p "make tests/test_parser.py::test_dates pass" \
  --max-turns 8 \
  --sandbox <profile> \
  --tools <allowed-tools> \
  --deny <rule>
Enter fullscreen mode Exit fullscreen mode

--max-turns is the step cap from the arithmetic above, built in. --sandbox, --tools / --disallowed-tools and --allow / --deny are where "which tool may write" gets decided. The flag to avoid on any machine that can reach production is --always-approve, alias --yolo. (Placeholders above are deliberate; see the Sandbox and Permissions pages for the actual profile and rule syntax.)

None of this is a resolve rate. As of October 2026, xAI's public docs do not attach a SWE-bench score to the grok binary. If someone measures one, it belongs to a specific model, step cap and harness.

Model Context Protocol

MCP separates the host from its tools. The specification (revision 2025-11-25) is built on JSON-RPC 2.0. The host (Grok CLI or another client) holds the client side. A server exposes tools (functions the model may call), resources (data to read) and prompts (templates). The lifecycle is: initialize negotiates capabilities, tools/list returns schemas, and tools/call runs one tool. On the client side, the spec also defines sampling (the server asks the host to call a model) and roots (the directories the server may see).

The local transport is stdio: a process on the same machine with no port. The remote transport, introduced in revision 2025-03-26 and kept in 2025-11-25, is Streamable HTTP; the older HTTP+SSE setup is no longer the main remote option. One MCP server can serve several hosts, but that does not settle which tool is allowed to write. In Grok CLI, servers are managed with grok mcp list|add|remove|doctor.

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_issue",
    "arguments": {"key": "PAY-1842"}
  }
}
Enter fullscreen mode Exit fullscreen mode

The message shape comes from the MCP tools spec. get_issue is a made-up read-only tool, not a built-in Grok CLI tool.

step = 0
context = [system, AGENTS_md, task]
while step < LIMIT and not target_test_green():
    action = model(context)
    if action.kind == "mcp":
        result = server.tools_call(action.name, action.arguments)
    elif action.kind == "shell":
        result = sandbox(action.command)  # network, .env and git push live here
    context.append(truncate(action, result))
    step += 1
Enter fullscreen mode Exit fullscreen mode

The interface matters even when the model stays the same. SWE-agent (Yang et al., arXiv:2405.15793, NeurIPS 2024) ran GPT-4 Turbo on SWE-bench Lite:

  • An agent limited to a plain shell resolved 11.00% of tasks; SWE-agent resolved 18.00% (54 of 300).
  • In the file-viewer ablation, a 100-line window scored 18.0% and pasting the whole file scored 12.7%.
  • In the editor ablation, shell-only editing fell to 10.3%, against 18.0% for edits that run a syntax lint.

Running cat on a whole file floods the context with text the model never uses in the patch.

On the full SWE-bench, the same setup resolved 12.47% (286 of 2,294) at an average of $1.59 per instance at the paper's prices. The 87.7% in the paper's abstract is HumanEvalFix, a set of short function repairs, not the 2,294 GitHub issues.

2. Numbers you can recompute, and numbers you shouldn't carry over

SWE-bench (Jimenez et al., arXiv:2310.06770, ICLR 2024) has 2,294 problems from 12 Python repositories. The best model in that paper, Claude 2, resolved 1.96% with no tool loop. Putting 1.96% next to 12.47% mixes up the model, the year and the scaffold. The ablation that holds the model fixed and changes only the interface is 11.00% against 18.00% on Lite: seven points on 300 tasks. That is enough to stop treating raw bash as a neutral API for a model. It is not enough to write "the agent resolves issues" into a proposal.

SWE-bench Verified (OpenAI, 13 Aug 2024): annotators reviewed 1,699 tasks and 500 were kept. 196 of them are "easy" (under 15 minutes) and 45 are "hard" (over an hour). GPT-4o resolved 33.2%. In the annotation, 38.3% of samples had an underspecified problem statement, 61.1% had tests that could fail a valid solution, and 68.3% were filtered out overall.

Verified retired (OpenAI, 23 Feb 2026): the frontier moved from 74.9% to 80.9% in six months. OpenAI audited 27.6% of the set, focusing on tasks that models often fail. In at least 59.4% of the audited problems they found material issues in the test design and/or the problem description; tests that reject a correct patch are the biggest group, but not the only one. Contamination is a separate problem. OpenAI stopped reporting Verified and pointed to SWE-bench Pro.

Pro questioned too (OpenAI, 8 Jul 2026): on the public split of SWE-bench Pro (731 tasks), the frontier pass rate rose from 23.3% to 80.3% in eight months. An automated pass flagged 200 tasks (27.4%) as broken, and a human campaign flagged 249 (34.1%). In the same post, OpenAI estimates that roughly 30% of Pro is broken and walks back its earlier advice to switch to Pro. Read the jump and the broken-task rate together.

Vendor tables. In the GPT-5.2 announcement, GPT-5.2 Thinking scores 80.0% on Verified and 55.6% on Pro public. That 24.4-point gap between two sets for the same model is larger than the whole 2024 shell-versus-ACI ablation. The GPT-5.4 announcement reports 57.7% on Pro public and 75.1% on Terminal-Bench 2.0. That is three percentages and three definitions of "can program". These are vendor figures, not independent measurements.

Developer productivity. METR's randomized trial (Becker et al., arXiv:2507.09089) followed 16 experienced open-source developers on 246 tasks from February to June 2025:

  • With AI tools allowed, tasks took 19% longer (CI +2% to +39%).
  • Before the study, the same developers forecast a 24% speedup. Afterwards they still believed they had been 20% faster.
  • Economists had forecast a 39% speedup and ML researchers 38%.

In a February 2026 update, METR says a follow-up with newer tools gave no reliable signal either way. The +19% describes early 2025, not Q4 2026.

The takeaway is that the interface moves the score even with fixed weights. A team's perception can have the opposite sign from the stopwatch. And a Verified or Pro percentage with no scaffold and no test audit attached does not predict your sprint.

3. Case: 90,000 tokens before the test says "wrong"

The assumptions are explicit so you can swap in your own token counter:

  • Prefix P = 2,000 tokens. Each step appends D = 1,000 tokens. With no prefix cache, the input at step k is P + (k−1)·D.
  • N = 12: 12·2,000 + 1,000·(12·11/2) = 24,000 + 66,000 = 90,000 input tokens. A single call costs 3,000, so the ratio is 30.
  • Steps 1–3 cost 9,000 tokens (10%); steps 10–12 cost 36,000 (40%).
  • With a cap of N = 8 (--max-turns 8): 16,000 + 28,000 = 44,000, which is 51% less. That comes from a shorter loop, not from a measured dashboard p99.
  • A vendor prefix cache discounts the shared beginning of the context. It does not remove a bad step, and it does not turn the stop condition into a test.

A cut that still keeps a check in place looks like this:

  • a step cap, and a stop only when the target test passes;
  • file views of about 100 lines;
  • MCP limited to reading the issue and the CI status;
  • a shell with no network and no secret files;
  • one git worktree per agent;
  • a human doing the merge.

If the test still fails at step 8, the loop stops. You don't get a patch, but you do get a bounded bill and a diff to review.

METR's result explains why teams still say "it feels faster". The gate therefore isn't a survey after the session. It is a test you knew before the agent started, plus a token counter on the session.

4. Decision table

Pattern Complexity Latency Cost When to use it
In-editor suggestion, a human accepts the diff Low No agent-loop tail About one call Local edits whose diff a person can read in full
Terminal agent with a file window, step cap and test-as-stop Medium Sum of steps until green or cap P=2,000, D=1,000: 90k tokens at N=12, 44k at N=8 The issue has a test, the repo is familiar, a human merges
The same agent plus read-only MCP Medium Plus tool round trips Every tool result lands in D; wide JSON breaks the 1k assumption Repeatable reads: issue text, logs, CI status. No writes. Audit with grok inspect, grok mcp doctor
Networked shell, no cap, an MCP server that writes, --yolo Low at first, high in an incident Unbounded tail Retrying the last step resends the largest context (13k at N=12), plus side effects Not in a repo that holds secrets, and not near production
Two agents, one working directory High Wall time may fall, git conflicts rise Two bills, one of which gets thrown away Don't. Use a separate worktree per session (grok -w)

5. Anti-patterns the host tutorial doesn't mention

  1. A leaderboard percentage written into a contract. Verified was retired in February 2026, and by July 2026 roughly a third of public Pro was flagged as broken. A contract should be judged on your tickets and your tests.
  2. A shell-only agent running cat on whole files. In the SWE-agent paper that scored 11.00% and 12.7%, against 18.0% with a 100-line window. Keep observations short, or D stops being 1,000.
  3. Stopping when the model says "done". At N=12, steps 10–12 carry 40% of the input tokens. A loop with no test spends its most expensive third on retries.
  4. A writing MCP tool and a networked shell under one grant, plus an MCP server committed to the repo without review. grok inspect exists so you can see what a directory loads. READMEs and issue bodies are an injection channel. And running a curl | bash installer on a laptop that can reach production is a policy decision, not an onboarding step, even when it is the official install.

FAQ

Does 80% on SWE-bench Verified mean the agent closes 80% of a senior engineer's tickets?
No. The score depends on the model, the scaffold and how clean the benchmark is, and both Verified and Pro have been publicly questioned by the vendor that promoted them. Your sprint is measured by your own tests.

Should an MCP server run over stdio or remotely on the company network?
Stdio runs a process next to the host and opens no port, so it suits a tool that may only read that directory anyway. Streamable HTTP gives the server an address, which means it also needs authentication and a separate list of tools that may write. Reads of issues and CI can go remote; a networked shell and production writes don't inherit that grant.

Can a terminal agent work in a repository under NDA?
Yes, if the API key, the transcript and the shell stay inside your boundary. Exporting a session (grok export) copies the diffs and anything the loop pulled into context, which can include a secret from .env.

Sources

Konrad Wojtasik is the CEO of Commoditech, a Warsaw IT staff-augmentation company that places software, data and AI engineers into client teams. Related: LLM orchestration and inference cost.

Top comments (0)