DEV Community

dubleCC
dubleCC

Posted on • Originally published at heycc.cn

Running AI Coding Agents Locally in 2026: What Actually Works

#localllm #aicodingagents #llamacpp #ollama

Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.

Running AI Coding Agents Locally in 2026: What Actually Works

Last updated: 2026-07-02

Every few months a new open-weight model claims to have "closed the gap" with frontier coding assistants, and every few months developers spend a weekend wiring it into Continue.dev or a terminal agent, only to hit a wall on the fourth tool call in a chain. The gap has genuinely narrowed since 2024 — but it hasn't closed uniformly, and where it hasn't closed maps almost exactly onto where local setups break in practice: long tool-call chains, ambiguous intermediate results, and context windows that quietly shrink the moment you plug into an agent framework instead of a chat box.

This piece works through the actual mechanics: how tool-calling is implemented in llama.cpp and exposed through Ollama, how Continue.dev and OpenCode paper over models that don't support native tool calling, what a real 16-model capability-ladder benchmark says about exactly where local models fall off a cliff versus where they're already at parity, and what real hardware you need to run a model that's good enough not to fight you.

Local coding agent request path: tool-call routing through llama.cpp/Ollama, branching on native tool-calling support, funneling through three independent context-truncation risks

The inference layer: Ollama and llama.cpp are not interchangeable defaults

Both projects wrap the same underlying GGUF/GGML inference engine, but they default to different tradeoffs, and those defaults directly determine whether tool calling works at all.

llama.cpp's llama-server is the lower-level surface. A basic launch is:

./llama-server -m models/7B/ggml-model.gguf -c 2048
Enter fullscreen mode Exit fullscreen mode

GPU offload is controlled with --n-gpu-layers (alias -ngl, default auto), and concurrent request handling is controlled with --parallel (server slots, default -1/auto) plus --cont-batching (continuous/dynamic batching, enabled by default). The server exposes OpenAI-compatible endpoints — /v1/chat/completions, /v1/completions, /v1/embeddings, plus newer /v1/responses and token-counting endpoints — alongside a Prometheus /metrics endpoint and a /slots endpoint for watching what each concurrent slot is doing. Speculative decoding is available via --spec-draft-model if you want to pair a small draft model with a larger target model for latency reduction.

Tool calling in llama.cpp is implemented in chat.h (introduced in PR #9639) as a set of native format handlers per model family: Llama 3.1/3.2/3.3 (including built-in Wolfram Alpha, web search, and code-interpreter tool templates), Functionary v3.1/v3.2, Hermes 2/3, Mistral Nemo, Firefunction v2, and Command R7B each get their own dedicated handler. Qwen 2.5 and Qwen 2.5 Coder are supported too, but per llama.cpp's own docs they're routed through the shared Hermes 2 Pro format handler rather than a Qwen-specific one — the docs list "Hermes 2/3, Qwen 2.5, Qwen 2.5 Coder" together under one template family. Everything outside this list falls back to a generic handler that the project's own docs warn "may consume more tokens and be less efficient than a model's native format." DeepSeek R1 gets a specific callout in the docs as "WIP / seems reluctant to call any tools" — a blunt admission that not every reasoning-tuned open model plays well with structured tool calls, independent of raw capability.

Three operational details matter and are easy to miss:

  1. Tool calling depends on Jinja templating being active. In current llama.cpp builds --jinja is enabled by default (use --no-jinja to fall back to the legacy prompt path). You can confirm what a given model actually supports by hitting http://localhost:8080/props and inspecting chat_template and the chat_template_caps object — specifically its supports_tool_calls / supports_tools booleans, which replaced the older single chat_template_tool_use field.
  2. Parallel tool calls are opt-in per request — you must pass parallel_tool_calls: true explicitly; it isn't inferred from the model or the --parallel server flag (which controls concurrent requests, not tool calls within one turn).
  3. Aggressive KV-cache quantization degrades tool calling specifically. The docs flag that extreme settings like -ctk q4_0 "can substantially degrade the model's tool calling performance" — a tradeoff that's tempting because it's one of the easiest levers for fitting more context into limited VRAM, and exactly the lever most likely to silently break your agent's tool loop.

Ollama sits on top of a similar engine but optimizes for zero-config defaults, and one of those defaults is a trap for agent workloads specifically. As of the current docs, Ollama auto-scales context length to detected GPU VRAM in three tiers: under 24 GiB VRAM → 4,096 tokens, 24–48 GiB → 32,000 tokens, 48 GiB+ → 256,000 tokens. That default is fine for chat. It is not fine for an agent that's stuffing a system prompt, tool schemas, file context, and multi-turn history into every request — Ollama's own guidance says workloads like "web search, agents, and coding tools" should be set to at least 64,000 tokens, which means anyone running below a 48GB card must manually override with OLLAMA_CONTEXT_LENGTH or the num_ctx parameter (precedence: API param > env var > Modelfile PARAMETER > built-in default). Skip this and the failure mode isn't a crash — it's silent truncation, which shows up as an agent that "forgets" earlier tool results or hallucinates file contents it never actually saw. This same failure mode — a context budget quietly too small for an agent loop — recurs below in both Continue.dev and OpenCode, from different root causes.

Agent clients: how they cope with models that don't speak native tool-calling

This is the part most comparisons skip, and it's the actual crux of whether "any open model" can be an agent.

Continue.dev explicitly does not require native tool-calling support. Per its own docs, agent mode can convert tool definitions into XML embedded in the system message, so "any model capable of following instructions can use tools, not just those with native tool support." The client auto-detects which path to use — native function-calling API or XML/system-message emulation — with no manual configuration. That's a meaningfully different design point from assuming every model exposes an OpenAI-style tools parameter that actually works. The tradeoff: XML-in-system-message tool calling is inherently less reliable than a model's native format, because it depends on the model correctly formatting and closing tags in free text rather than being constrained by grammar-based decoding.

Even so, Continue's own documentation is candid about where this breaks down. Its Ollama integration guide notes that some models — DeepSeek R1 by name — may report "Agent mode is not supported" or "does not support tools" even after you manually configure a tool_use capability flag, and lists Llama 3.1 and Mistral as confirmed-working alternatives. The same guide separately warns that Continue "may use a higher default context length than other tools," recommending users drop contextLength to 2048 if they hit out-of-memory errors — a second, independent context-budget constraint layered on top of Ollama's VRAM-tiered defaults above; the two interact, and the practical fix is to explicitly set both rather than trust either default for agent use.

For model recommendations, Continue's own docs for Agent/Plan mode list only three open-weight families: Qwen3 Coder (480B and 30B variants), Devstral (27B), and Kimi K2 (1T). Notably absent: anything under roughly 20B parameters. That's a strong, official signal — from the tool that built its whole client-side tool-calling story around broad model compatibility — that broad compatibility and reliable agent-mode performance are different claims. XML fallback means small models technically "can use tools"; it does not mean they use them well enough for multi-step agent workflows.

OpenCode, the terminal-based alternative (positioned as the most-starred fully open, provider-agnostic coding agent CLI, reporting 180k+ GitHub stars and supporting 75+ providers via Models.dev), takes a more infrastructure-agnostic approach: local backends are wired in as @ai-sdk/openai-compatible providers pointed at whatever server you're already running — Ollama at http://localhost:11434/v1, LM Studio at http://127.0.0.1:1234/v1, or a raw llama.cpp llama-server at http://127.0.0.1:8080/v1. All three are configured identically in opencode.json as a provider block with a baseURL and a model map. If tool calling misbehaves against an Ollama backend specifically, OpenCode's own fix is the same lever as above: raise num_ctx to 16k–32k. Three clients, three different default-context mechanisms, one underlying failure mode — worth internalizing as a checklist rather than three separate debugging sessions.

Where local models actually stand: the capability-ladder data

Aggregate leaderboard percentages hide the thing that matters for agent work, which is where in a task a model starts failing. A useful recent data point here is AgentFloor ("How Far Up the Tool Use Ladder Can Small Open-Weight Models Go?", Karmakar & Chatterjee, arXiv:2605.00334, submitted May 2026), a deterministic 30-task benchmark built specifically to separate that out. It defines six capability tiers with increasing tool-use complexity, run across 16,542 scored trials:

Tier What it tests Step budget
A0 Instruction-following, no tools —
A Single tool call 1–2
B Sequential two-tool chaining ~3
C Conditional branching on intermediate results ~5
D Multi-source synthesis with conflict recovery ~7
E Long-horizon planning under persistent constraints up to 10

It evaluates 16 open-weight models from 0.27B to 32B parameters against GPT-5 as the frontier reference. The model roster spans functiongemma:270m, qwen3:0.6b/1.7b/8b/14b/32b, qwen3.5:2b, granite4:3b, ministral-3:3b/8b/14b, gemma4:e4b, nemotron-3-nano:4b, gpt-oss:20b, mistral-small3.2:24b, and the strongest open-weight model in the study, gemma4:26b. (The "gemma4" and "qwen3.5" naming is the paper's own — later-generation model releases than the more familiar Gemma 2/3 and Qwen 3 lines readers may recognize from 2024–2025.) The per-tier numbers for gemma4:26b versus GPT-5, from the paper's Table 2:

Tier gemma4:26b GPT-5 Gap
A0 (no tools) 100% 80% open model ahead
A (single tool call) 96% 98% near-parity, frontier narrowly ahead
B (two-tool chain) 72% 82% frontier ahead
C (conditional branching) 59% 51% open model ahead
D (multi-source synthesis) 32% 42% frontier ahead
E (long-horizon planning) 0% 10% frontier's real advantage lives here

Read across the row, not down the average: through tier C, gemma4:26b is at or near parity with GPT-5 — only tier A shows a narrow 2-point gap in GPT-5's favor, and gemma4:26b is clearly ahead on tier C; tiers B and D swing back in GPT-5's favor by 10 points each. The clearest, least noisy signal in the table is tier E: the open model scores zero and even GPT-5 only clears 10%. The paper's own summary is blunter than any single tier number — "C, D, and E never clear at any threshold in [60, 90]" and "no zero-shot model in the corpus clears any threshold" on those three tiers, with the authors noting that targeted interventions exist but "each effective intervention we tested helped one model and was null on the others" — i.e., no prompt trick generalizes across the model families tested. That's a meaningfully different picture from "open models are behind," and a more useful one for deciding whether local is viable: if your agent workflow is single- or double-tool-call automation (linting, targeted refactors, single-file edits with a build check), a 26B local model is not meaningfully behind a frontier hosted one on this benchmark. If it's a long multi-file migration requiring the agent to hold state across ten-plus steps and recover from contradictory intermediate results, every model in this study — hosted included — struggles, and the frontier model struggles less.

The cost and latency side reinforces this. At matched aggregate accuracy (~60%), the paper reports gemma4:26b on a Mac Studio (amortized at $0.50/hr) reaching $0.0022 per passed task against GPT-5's $0.0327 — about 15x cheaper, and on H100 spot pricing ($2.50/hr) about 3x cheaper than GPT-5 per passed task. Per-task latency at matched accuracy is 16.0 seconds for gemma4:26b versus 40.8 seconds for GPT-5 — roughly 2.5x faster. At the small end, sub-5B models clear the easy tiers (A0/A) at 80%+ reliability for as little as $0.00007 per passed task (ministral-3:3b on tier A) and $0.0002 (nemotron-3-nano:4b on tier A0). For narrow, high-volume, low-tier agent tasks, local isn't just "good enough" — it's a different cost regime entirely.

Verification note: the AgentFloor numbers above were re-pulled directly from the paper's abstract, Table 2, and cost-analysis section on arXiv (2605.00334, HTML rendering) rather than taken from a secondary summary, since headline-precision numbers like these are exactly where transcription errors compound. The tier B and D figures for gemma4:26b in particular (72.0% and 32.0%) differ from numbers that circulate in some secondary write-ups (73.3% and 40%) — use the values in this table, sourced directly from the paper's Table 2.

SWE-bench: the aggregate number, and why the frontier comparison needs a caveat

SWE-bench Verified is the standard aggregate comparison point for coding-specific capability (distinct from AgentFloor's tool-use-focused ladder). Current primary-source numbers:

Model Params (open weight: total/active) SWE-bench Verified Context License
Claude Opus 4.5 closed 80.9% — closed
Devstral 2 (Mistral) 123B dense 72.2% 256K Modified MIT
Qwen3-Coder-Next 80B / 3B active (MoE) 70.6–71.3%* 262,144 open weights
Devstral Small 2 (Mistral) 24B dense 68.0% 256K Apache 2.0
Devstral (original) smaller predecessor 46.8% — Apache 2.0

*Qwen3-Coder-Next's SWE-bench Verified score depends on agent scaffold: 70.6% with SWE-Agent, 71.1% with Mini-SWE-Agent, 71.3% with OpenHands — a reminder that the harness around the model, not just the model, moves the number by roughly a point either way.

Anthropic's own Claude Opus 4.5 announcement states it is state-of-the-art on SWE-bench Verified at 80.9% — widely reported as the first model to cross 80% — with its comparison chart showing GPT-5.1 (76.3%) and Gemini 3 Pro (76.2%) trailing. Both competitor figures are self-reported: OpenAI's own GPT-5.1 announcement states 76.3% on SWE-bench Verified, and Google's Gemini 3 announcement states 76.2%, so the 80.9/76.3/76.2 trio is three vendor-reported headline numbers rather than one apples-to-apples comparison Anthropic verified under its own test conditions. It's tempting to read this alongside a footnote elsewhere in the same announcement disclosing that Anthropic re-ran competing models under an improved hosting environment and its own harness, moving their scores to "56.7%" (Gemini 3) and "48.6%" (GPT-5.1) — but that footnote is not about SWE-bench Verified. Per the announcement's own methodology section, the Terminus-2-harness re-run it describes applies to Terminal-Bench, a separate agentic benchmark, not the SWE-bench Verified comparison discussed above (third-party Terminal-Bench 2.0 leaderboard figures for GPT-5.1 and Gemini 3 Pro sit in the same range as those footnoted numbers, confirming the benchmark). Conflating the two would wrongly suggest Anthropic caught GPT-5.1 and Gemini 3 Pro inflating their SWE-bench Verified scores by 20+ points — it didn't say that, and making that leap is the same primary-source-conflation risk this piece flags below for SWE-bench Pro versus Verified: a headline number's credibility depends on which benchmark, which harness, and who ran it, and those details don't transfer across footnotes. Whichever framing you use, Opus 4.5 does this at medium reasoning effort while using 76% fewer output tokens than Sonnet 4.5 needed to hit a comparable score, which matters for agent loops because output-token count is directly proportional to latency per turn.

The best current open-weight coding models cluster at 68–72% on SWE-bench Verified as reported by their own developers — roughly 8–13 points behind Opus 4.5's 80.9%. That's a real, consistent gap, and it's narrower than it was in 2024 (when the original Devstral's 46.8% was itself a >6-point jump over prior open state-of-the-art) but it has not closed.

Qwen3-Coder-Next's own technical report is unusually candid about where the remaining gap actually shows up in practice, beyond the aggregate number: the authors state there is "a gap in solving highly complex, large-scale software engineering tasks," that the model may need "more interaction turns to reach correct solutions" than frontier models for equivalent tasks (i.e., it's less efficient per solved task, which compounds latency in an agent loop), and that "frontend and UI-related capability remains an area for improvement." The training pipeline behind it is worth understanding if you're deciding between model families: a staged approach of mid-training on code/agent-centric data, SFT on agentic trajectories, specialization into multiple expert sub-models, then distillation back into one deployable model, using best-fit packing, fill-in-the-middle objectives, multi-turn agentic trajectory learning, and RL with execution feedback. That's a materially more agent-aware training recipe than earlier code-completion-focused open models, and it shows up as the SWE-bench number — but the authors' own limitations section is the more honest signal for what to expect on tasks outside that training distribution.

One caution worth stating plainly: benchmark aggregator sites (e.g., third-party SWE-bench Pro leaderboards) are now tracking newer model generations than the numbers above and use a stricter, differently-scored variant (SWE-bench Pro, not Verified) where even frontier models score in the 50–90% range on a completely different scale. Do not cross-reference a Pro percentage against a Verified percentage — they are not the same benchmark, and vendor-reported numbers on aggregator pages are explicitly flagged by those same pages as "not independently verified." This is the identical caveat that applies to the Opus 4.5 comparison chart above: a headline number's credibility depends entirely on whether it came from the vendor being praised, a competitor's own announcement, or an independent re-run — and those three are not interchangeable even when they sit in the same chart.

Hardware reality: what actually fits, and what that does to latency

The AgentFloor and SWE-bench numbers above assume the model runs cleanly. Whether it does depends on quantization headroom that's easy to get wrong on a single consumer GPU.

A dense 32B-class model at Q4_K_M quantization needs real VRAM headroom: Qwen3-32B's Q4_K_M GGUF, per its Hugging Face model card, is 19.8 GB for weights alone — close enough to a 24GB card's ceiling (RTX 4090-class) that little headroom is left for KV cache, meaning your effective context window shrinks fast under real agent workloads once you add a few thousand tokens of system prompt, tool schemas, and file context on top. Independent, citable per-GPU throughput numbers for this exact model/quant/card combination are not consistently published — third-party benchmarks for similarly-sized 32B-class dense models on a 4090 report figures anywhere from roughly 20 to 60 tokens/sec depending on quantization, context length, and batch settings, so treat any single "X tokens/sec" figure you see for this class of setup as configuration-dependent rather than a fixed spec, and benchmark your own stack before relying on it for latency planning.

The MoE alternative changes this math substantially. Qwen3-Coder-30B-A3B (30B total parameters, only ~3.3B active per token) is listed on Ollama's own model page at 19GB for its default pull — though exact size varies a point or two by quant tag and measurement method across mirrors, so treat "~19GB" as a reasonable planning figure rather than an exact spec — and needs far less compute per token than its total parameter count implies, because only the active experts fire on each forward pass. This is the practical reason MoE architectures (also used by Qwen3-Coder-Next at 80B total/3B active) are becoming the default shape for "runs locally and doesn't crawl" coding models, versus dense architectures like Devstral 2's 123B, which Mistral's own materials indicate realistically wants multi-H100-class hardware and isn't positioned for consumer deployment. Devstral Small 2 (24B dense, Apache 2.0) is the explicit consumer-hardware counterpart in Mistral's lineup — single-GPU, including NVIDIA DGX Spark and consumer GeForce RTX cards, with CPU-only fallback supported.

This gives a concrete decision framework:

Your hardware Ollama context tier (auto) Realistic model choice What you're trading away
<24GB VRAM (e.g. RTX 4060/4070, M-series ≤16GB unified) 4k tokens (override to ≥64k manually) Devstral Small 2 (24B), Qwen3-Coder-30B-A3B at Q4 Long tool-chain reliability (tier D/E territory); must manually raise context or agent truncates silently
24–48GB VRAM (RTX 4090/5090, single A6000, 32–48GB unified) 32k tokens auto Qwen3-Coder-30B-A3B comfortably, dense 32B tight Still short of the ≥64k Ollama itself recommends for agent workloads — override anyway
48GB+ VRAM (multi-GPU, Mac Studio 96GB+ unified) 256k tokens auto Qwen3-Coder-Next (80B/3B active) Approaching but not matching Opus 4.5 on aggregate SWE-bench; still behind on tier-E long-horizon tasks per AgentFloor
Multi-H100 / cloud GPU cluster 256k tokens auto Devstral 2 (123B dense) Not "local" in the laptop sense — this is self-hosted infrastructure, not a workstation setup

The practical takeaway: "local" coding agents split into two very different regimes. Below ~48GB VRAM, you're running MoE models in the 24–30B-total-parameter class, and per the AgentFloor data that's genuinely competitive with frontier hosted models through tier C (conditional branching) — but you must manually fight Ollama's and Continue's default context truncation to get there, or the agent will silently misbehave in ways that look like model incompetence but are actually configuration. Above ~48GB, you can run near-frontier open weights like Qwen3-Coder-Next, which closes most of the SWE-bench gap but, per its own authors, still needs more turns per task and still lags on the long-horizon, many-step workflows where frontier hosted models retain their clearest edge.

Reproducible check: does your local server actually support tool calling?

Before wiring any model into an agent client, verify tool-calling support directly against the running server rather than trusting the model card:

# 1. Launch llama-server (jinja templating is enabled by default in current builds;
#    --jinja is passed explicitly here for clarity and for older builds)
./llama-server -m models/your-model.gguf -c 65536 --jinja --n-gpu-layers 999

# 2. Check what chat template the server detected and whether it reports tool-call support
curl -s http://localhost:8080/props | jq '.chat_template, .chat_template_caps'

# 3. Send a minimal tool-calling request and inspect whether it returns
#    a structured tool_calls array or just narrates the call in prose
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "What is the weather in Boston?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }' | jq '.choices[0].message'
Enter fullscreen mode Exit fullscreen mode

This one you have to run yourself — the output below is the documented contract, not a capture from a local build, and it is the single check worth doing before you commit to a model for tool-calling work.

Expected behavior to check for: if the model has a native tool-calling template (Qwen 2.5/3 Coder, Hermes, Llama 3.x, Functionary, Mistral Nemo, Firefunction v2, Command R7B per llama.cpp's supported list), chat_template_caps.supports_tool_calls will be true and the response message will contain a structured tool_calls array with a function.name and function.arguments JSON string. If the server falls back to the generic handler, the message.content will instead contain the tool call narrated as prose text (sometimes with malformed or unclosed pseudo-XML/JSON), and no tool_calls array will be present at all. That distinction — a structured array versus prose that merely describes an intent to call a tool — is the actual pass/fail line for whether a client like Continue.dev's XML-fallback path or OpenCode's provider integration will work reliably, and it's worth checking per-model rather than assuming anything on llama.cpp's supported-format list behaves identically once wrapped in a specific client's prompt scaffolding.

Is running it locally worth it

None of this maps to a single "yes, local works" or "no, it doesn't." It maps to a checklist: confirm --jinja and check /props before assuming tool calling is live; override Ollama's and Continue's context defaults explicitly rather than trusting auto-detection for anything agentic; pick a model class (dense 24–32B vs. MoE 30B-A3B vs. 80B-A3B) based on your actual VRAM ceiling rather than a leaderboard number; and match your workload's step count to AgentFloor's tier data — tiers A0 through C are close to solved locally, tier E is not close to solved even by frontier hosted models. The SWE-bench gap between the best open-weight coding models and Opus 4.5 is real (roughly 8–13 points) but it's a different question from whether a given agent task needs that last 10% of aggregate capability, and for a large share of routine, short-horizon agent work, current evidence says it doesn't.

Sources

Verification note (2026-07-02): SWE-bench Verified figures for Claude Opus 4.5, Devstral 2, Devstral Small 2, and Qwen3-Coder-Next checked against each vendor's own announcement/model card. The Opus 4.5 vs. GPT-5.1/Gemini 3 Pro comparison was re-checked directly against Anthropic's announcement page, which discloses in a footnote that its own re-scoring of the two competing models under a controlled harness produced substantially different (lower) numbers than the vendor-self-reported 76.3%/76.2% figures — this piece now treats those as vendor-reported rather than Anthropic-verified. The AgentFloor model list, per-tier accuracy table, and cost/latency figures were re-pulled from the paper's own abstract, Table 2, and cost-analysis section on arXiv rather than a secondary source, and two figures (tier B and D for gemma4:26b) were corrected from an earlier draft to match the paper exactly. Qwen3-32B Q4_K_M size (19.8GB) was checked against its Hugging Face GGUF model card. Qwen3-Coder-30B-A3B's ~19GB pull size was checked against Ollama's own model page but is flagged here as an approximate planning figure since third-party mirrors report a range. The RTX 4090 tokens/sec figure for 32B-class dense models in the previous draft could not be traced to a specific citable primary-source benchmark and has been removed in favor of an explicit range with the uncertainty stated. Qwen 2.5/2.5 Coder's tool-calling handler was corrected from "dedicated parser" to "shared Hermes 2 Pro format handler," matching llama.cpp's own documentation grouping.

Verification note (2026-07-16): Re-checked every named tool/model/hardware/integration claim against current primary sources. Four corrections: (1) The Opus 4.5 vs. GPT-5.1/Gemini 3 Pro discussion previously conflated Anthropic's footnote about re-scoring competitors under the Terminus-2 harness with the SWE-bench Verified comparison — that footnote and re-run are actually about Terminal-Bench (confirmed via the announcement's own methodology section, cross-checked against third-party Terminal-Bench 2.0 leaderboard figures for GPT-5.1 and Gemini 3 Pro), not SWE-bench Verified; the section has been rewritten to remove that conflation. (2) The AgentFloor tier-A row for gemma4:26b/GPT-5 was wrong (had 100.0%/97.8%; the paper's Table 2, pulled directly from arXiv:2605.00334's HTML rendering, shows 96%/98%) and spurious decimal precision on GPT-5's B/C/D/E figures was removed since Table 2 reports whole-number percentages only — all other AgentFloor figures (model roster, cost/latency numbers, deployment-recipe costs) were verified correct against the same source. (3) llama.cpp's --jinja flag is enabled by default in current builds per the llama-server README's CLI reference (previously stated as off-by-default). (4) llama.cpp's /props endpoint no longer exposes a chat_template_tool_use field; it now returns a chat_template_caps object with granular booleans (supports_tool_calls, supports_tools, etc., per common/jinja/caps.h) — the three references to the old field name were updated accordingly. Everything else — Ollama's VRAM-tiered context defaults and 64k agent recommendation, Continue.dev's XML tool-calling fallback and Agent/Plan model table (including Kimi K2 1T's listing as an open model), OpenCode's star count (186k+, confirmed live via GitHub API) and local-provider config, the Devstral/Qwen3-Coder-Next SWE-bench and hardware figures, and the Qwen3-32B/Qwen3-Coder-30B-A3B GGUF sizes — checked out against current sources with no changes needed.

Top comments (0)