DEV Community: Thomas Landgraf

The SDK You Pick Matters More Than the Model — A 13-LLM Benchmark on the Same Agentic Task

Thomas Landgraf — Fri, 01 May 2026 09:16:36 +0000

If you have ever built an agent that walks a codebase, calls tools, and writes structured output, you have hit the same wall I kept hitting: the same model produces wildly different results on the same task depending on what harness you wrap it in. Swap Claude for GPT behind a single OPENAI_BASE_URL and you lose half your output quality. Everyone blames the model. The model is rarely the variable.

I ran an experiment to put a number on it. Thirteen LLMs — Claude Opus 4.7, Sonnet 4.6, Haiku 4.5, GPT 5.4, GPT 5.4 Mini, two Gemini 3.1 previews, and six open-weights locals (Qwen 3.6 35B A3B, Gemma 4 at three sizes, GPT-OSS 20B, Nemotron 3 Nano) — on the same real agentic task. Same codebase (excalidraw), same MCP tools, same system prompt. Only the model changes. The output is a specification tree: goal → feature → requirement hierarchies of Markdown files.

What if the SDK is doing more of the work than anyone admits?

Every provider ships an SDK. Most teams assume the SDK is a thin wire-protocol wrapper. It usually isn't. Here's what the Anthropic SDK ships with by default, alongside the MCP tools I expose:

A persistent Todo-List the model reads from and writes to across turns.
A planner for multi-step reasoning that doesn't burn the main conversation budget.
A scratchpad for cross-turn notes that never reach the final output.

Here's what the OpenAI SDK ships with by default when you give it MCP tools: the MCP tools. Nothing else.

I'm the creator of SPECLAN, a VS Code extension for spec-driven development, and the pipeline in this benchmark is one of SPECLAN's agents (full disclosure). But the lesson generalizes to any multi-provider agent harness — and the numbers are genuinely jarring.

The band gap

Requirements produced on the same codebase, same prompt:

Model	SDK	Requirements
Claude Opus 4.7 (1M)	Anthropic	197
Claude Sonnet 4.6	Anthropic	196
Claude Haiku 4.5	Anthropic	203
GPT 5.4	OpenAI	43
GPT 5.4 Mini	OpenAI	60
Gemini 3.1 Pro preview	OpenAI-compat	17
Gemini 3.1 Flash preview	OpenAI-compat	13
Qwen 3.6 35B A3B (local)	OpenAI-compat	174
Gemma 4 31B dense (local)	OpenAI-compat	60
Gemma 4 8B (local)	OpenAI-compat	21
GPT-OSS 20B (local)	OpenAI-compat	17
Nemotron 3 Nano (local)	OpenAI-compat	12

Two things jump out immediately.

The three Claude models cluster at 196–203 regardless of size. Opus is several times the size of Haiku. If model size were driving volume, you would see variance. You don't. That flatness is the scaffolding floor — the shape of what the Anthropic agent loop produces on this benchmark, not the ceiling of what Opus can do.

Every OpenAI-SDK model except one sits at 13–60. An order of magnitude below the Anthropic band. Different vendors (OpenAI, Google, Meta-derived, Chinese open-weights), different sizes (8B to 120B+), same roughly-converged output volume. That convergence is what you would expect if the binding constraint were the harness, not the model.

Why spec authoring exposes scaffolding so brutally

Here's the technical meat. Spec authoring is fundamentally a list-management problem: enumerate the features the code implements, write a requirement, cross it off, move to the next. A human technical writer does this with a notepad.

Without a Todo-List, an LLM has to re-derive from conversation history every turn: did I already write requirements for Shape Drawing Tools? Let me check the last 12 turns… yes I did. Element Organization? Let me check… no, that's next. Every single turn, this bookkeeping consumes context window and decision budget that could have gone into writing the actual requirement.

With a persistent Todo-List, the model does one tiny tool call (todo_list_read), sees next undone: Element Organization, and gets to work. It's doing a fundamentally easier version of the task. That's why you get 197 requirements from Opus and 43 from GPT 5.4 on the same brief — the first model was given a list-management abstraction, the second had to reinvent it in every turn.

If you've ever wondered why your Claude-via-Anthropic-SDK agent seems to "remember things twelve files ago" and your GPT-via-OpenAI-SDK agent feels like it restarts every turn — this is why. Anthropic's SDK implements memory as a tool. OpenAI's SDK expects you to bring your own.

The one exception — and why it matters

Look at the table again: Qwen 3.6 35B A3B produced 174 requirements on the no-scaffolding OpenAI-SDK path. Running locally in LM Studio on a Mac M4 Max, 50k context. Within 12% of the Anthropic cluster. It is the one outlier in an otherwise tight 13–60 band.

Our best guess for why: Qwen's training mix is heavy on agentic tool-call trajectories, and that data seems to have internalized some of the bookkeeping the Anthropic SDK externalizes as tools. The model brought its own list-management to the task.

This matters because it proves the gap is closable without the SDK help — just not by most models. You can think of the benchmark as a 2×2:

	Scaffolding in harness	No scaffolding in harness
Scaffolding-trained model	Opus / Sonnet / Haiku (196–203)	Qwen 3.6 35B A3B (174)
Not trained for agentic bookkeeping	— (we don't have data)	GPT 5.4, Gemini, Gemma, GPT-OSS, Nemotron (13–60)

Three of the four quadrants are populated. The missing one — "scaffolding-trained model on a scaffolding-free harness" — is the obvious follow-up: run Opus on an OpenAI-SDK harness with the Anthropic tools explicitly stripped, so Opus operates on the same MCP-only surface as the others. The delta between that number and Opus's 197 is the SDK's contribution. Whatever's left is the pure-Opus delta. That's the experiment we're shipping next.

The Gemini anomaly — same vendor, two outcomes

One finding from the benchmark lands harder than any single row: Gemma (local open-weights) succeeded at the task; Gemini (frontier cloud preview) failed. Same company. Same pipeline on our end. Same adapter layer (OpenAI-compatibility).

Gemma 4 8B wrote a coherent on-domain tree — every one of 21 requirements landed on a real excalidraw feature. Gemini 3.1 Pro preview wrote Account and Billing Management, Personalized Analytics Dashboard, full acceptance criteria for Subscription tier management (upgrade, downgrade, cancel). Excalidraw has no accounts and no billing.

Our working hypothesis: our OpenAI-compatibility shim round-trips tool-call payloads in a format Gemma tolerates but Gemini treats differently. Gemini falls back to training priors when the adapter produces turns it cannot fluently continue — and "enterprise SaaS reference architecture" is over-represented in those priors. Before anyone dismisses Gemini previews as weak at agentic work, that rerun through Google's native GenerateContent API with planning primitives enabled is on the follow-up list.

The generalizable lesson for anyone building multi-provider agents: every harness silently privileges some providers over others. Our harness privileges Anthropic (full SDK integration), accidentally privileges Gemma-like models (MCP-only works for them), and does not fit Gemini 3.1. Your harness will have the same asymmetry in a different shape. The answer to "which model is best for my harness?" is not "whichever has the most parameters" — it's "whichever your harness actually fits."

What to take from this if you are building agents

Audit what your SDK ships by default. If you picked the Anthropic SDK and haven't looked inside, a non-trivial share of your agent's competence is coming from the built-in Todo-List / planner / scratchpad. Switch providers without replacing that layer and you will measure a model-quality drop that is actually a scaffolding drop.
Invest in the scaffolding layer before investing in a bigger model. In our benchmark, scaffolding was worth roughly an order of magnitude of output volume. A bigger model on a thin harness will not close that gap. A smaller model on a thick harness often will.
Multi-provider support is a harness problem, not a config-flag problem. If you're offering users a choice of providers, you're offering them a choice of how well your harness fits their provider. That's architectural work, not one-line-of-YAML work.
The training mix sometimes bridges the gap for you. Qwen 3.6 35B A3B is the proof. Agentic-tool-call-heavy training data appears to internalize what other models rely on the SDK to externalize. If you're picking a local model for agentic workloads, pick one whose training mix matches that shape.

Try it yourself

All 13 spec trees are browsable side-by-side at speclan.net/compare/ with URL-sharable deep links. Two pairings worth five minutes of your time:

?left=opus&right=qwen3.6-35b-a3b — Anthropic SDK + Opus vs. OpenAI SDK + Qwen. The gap visible in the UI is the SDK story.
?left=gemini3.1-pro&right=gemma4-8b — same vendor, two outcomes. The Gemini adapter-fit anomaly in isolation.

The full canonical post with the complete 13-row table, every caveat (including the excalidraw-in-training-data one), and the follow-up experiments planned lives at speclan.net/blog/2026-04-29-model-comparison.

If you've built a multi-provider agent and seen the SDK-layer drop I'm describing — especially if you've measured it — I'd love to see your numbers in the comments. Particularly curious about LangGraph users who added a Todo-List abstraction and measured a lift across providers.

Four failure modes you'll hit running a local LLM in a multi-step agentic loop

Thomas Landgraf — Sat, 25 Apr 2026 09:56:49 +0000

Most local-LLM benchmarks measure single-turn chat quality. Agentic workflows are a different beast: the model has to read state, call a tool, inspect the tool's result, decide whether it's done, and — if not — call another tool. A model that scores 95% on chat benchmarks can fail catastrophically on this loop in characteristic, reproducible ways.

I spent three weeks trying to get local LLMs to reliably run the agentic workflows in a VS Code extension I maintain. Full disclosure: I'm the creator of SPECLAN, an extension that manages product specs as Markdown files with YAML frontmatter — Git-native, one file per requirement, organized in a tree. A core feature, Infer Specs, walks a codebase and proposes a Goal → Feature → Requirement tree by calling MCP tools (create_feature, update_requirement, read_file, etc.) in a loop until it decides the tree is complete. This is a heavy agentic workflow: multi-turn, tool-heavy, and the model has to know when to stop.

The concept works without the tool. Markdown-plus-YAML-plus-Git as a spec format is older than SPECLAN and is the generalizable pattern this article assumes. The failure modes below will hit any agentic workflow that uses MCP tool calls plus structured output — SPECLAN is just where I observed them on seven different models across two local servers.

Here are the four failure modes, in the order you'll probably hit them.

1. The tool-call loop

Setup: an instruction-tuned model, reasonable size, MCP tool wired, seed a requirement and ask the agent to populate it.

What you'll see in the trace:

18:56:25  update_requirement  → R-0049
18:56:25  update_requirement  → R-0049
18:56:26  update_requirement  → R-0049
18:56:26  update_requirement  → R-0049
...  (12 more times, same arguments)

Same tool, same target, same arguments, repeated until the agent runs out of turns. On disk: garbage. The requirement's description got jammed into the YAML title: field, the body is still the untouched template placeholder, and the "Acceptance Criteria" section ends up in the wrong place.

This is not a bug in your code. Google's own Gemma 4 docs acknowledge it: Gemma can emit multiple tool calls per turn and has no built-in loop termination. The model sees the tool's success response but doesn't recognize "I am done." MoE and MatFormer-style elastic variants hit this hardest.

Mitigation (application layer): track tool-call fingerprints in the agent runner. If you see the same (tool_name, stable_arg_hash) three times in a row, interrupt the loop with a synthetic tool result that says "this tool has already produced the expected effect; proceed to the next step or terminate." This works because the loop is usually driven by the model not trusting the first success.

const callHistory: string[] = [];
for await (const step of agent.stream()) {
  if (step.type === 'tool_call') {
    const fp = `${step.name}:${stableStringify(step.args)}`;
    if (callHistory.slice(-3).every(x => x === fp)) {
      yield { role: 'tool', content: 'Already applied. Continue or finish.' };
      continue;
    }
    callHistory.push(fp);
  }
  yield step;
}

Not beautiful but survives every MoE variant I've thrown at it.

2. The hallucinated success

Second failure is worse, because it passes superficial validation.

Trace:

17:22:01  update_requirement  → R-8881   [tool call happened]
17:22:03  assistant: "I read the current state of R-8881 and updated its
                     description with a full specification: [long convincing
                     summary of changes]"

File on disk: unchanged.

The tool call fired. Your logs show it. The agent's final answer says the task succeeded. But the tool-call arguments were malformed in a way your MCP server silently ignored — or the model narrated its intent as a completion without ever carrying it out.

This is the "hallucinated success" mode. It's worse than the loop because:

Tests that assert "the agent called update_requirement at least once" pass.
Tests that assert the file changed fail — but only if you actually assert that.
Manual review sees a confident, detailed "I did it" message and believes it.

Mitigation (observability layer): every tool-call MCP server should return a diff summary as part of its response, not just {"success": true}. Something like { changed: true, hash_before: '...', hash_after: '...', fields_modified: ['description', 'acceptance_criteria'] }. Then your agent runner can verify that the model's final claim is consistent with the actual diff history. If the model says "I updated the description" but the diff summary shows changed: false, flag the session as inconsistent.

I also keep a diff_since_seed field that the agent can read at any time — so the model can literally look at what it has and hasn't changed, rather than relying on its own memory of the conversation.

3. Edit-as-replace

Different workflow: user runs /add Acceptance Criteria on an existing 5-section spec. Claude and GPT-5 default to echoing the full document with the addition merged in. A weaker local model — gemma-4-26b-a4b in my case — returned only the new section. Three sentences. The editor received three sentences and replaced the entire document.

Silent data loss. No error, no warning.

This isn't exclusive to local models; it happens to cloud models too if your prompt doesn't explicitly state the invariant. But strong models infer the invariant ("they want me to add a section, not replace the doc"). Weak models execute the surface instruction. Prompt-engineer it out:

DOCUMENT COMPLETENESS RULE (NON-NEGOTIABLE)

Your response MUST contain the ENTIRE document, not just the portion
you modified. The editor replaces the current document with your full
response. A partial response will delete everything you didn't emit.

If you cannot reproduce the full document (length, context budget,
uncertainty), return the ORIGINAL document unchanged. A no-op is
always correct; silent truncation is never correct.

The fact that this has to be said in capitals to a 26B model is the whole lesson of weak-model prompting: invariants that strong models treat as obvious must be written down.

4. Structured-output non-compliance

Clarification flow: ask the model to propose JSON matching a schema like { changes: [...], reasoning: "..." }. Downstream code does response.changes.map(...).

A local model with no guidance returned a raw array instead of the wrapped object. .map on undefined, crash.

Here's the subtle part: the schema text was never reaching the prompt. A helper signature had changed; the caller was still passing the schema positionally. TypeScript accepted it. The local model made up its own structure because we never showed it the schema in-band.

The lesson: don't rely on OpenAI's response_format or any SDK-level structured-output guarantee for local models. Most local servers implement the OpenAI-compatible API but not the structured-output constraints behind it. Put the schema text directly into the system prompt:

const systemPrompt = `
You return JSON matching this exact schema:

${JSON.stringify(schema, null, 2)}

Critical: the root MUST be an object with a "changes" array and a
"reasoning" string. NEVER return a bare array at the root.
`;

Belt-and-braces with the SDK's structured-output call. Local models will still go off-script occasionally, but the schema-in-prompt approach catches ~95% of the drift in my testing.

The benchmark

Seeded a requirement, asked the agent to populate it with description + acceptance criteria, measured: did it call the right tool? did it call the same tool more than 3× (loop)? did the file on disk actually change?

Server	Model	Type	Heavy workflow	Failure mode
Ollama	`gemma4:latest` (8B)	Dense	PASS	—
Ollama	`gemma4:31b`	Dense	PASS	slow but clean
Ollama	`gpt-oss:20b`	MoE	PASS tools / FAIL schema	output non-compliance
LM Studio	`google/gemma-4-26b-a4b`	MoE	FAIL	tool-call loop ×16
LM Studio	`openai/gpt-oss-20b`	MoE	FAIL	hallucinates completion
LM Studio	`google/gemma-4-e4b`	Elastic	FAIL	"no final response"
LM Studio	`openai/gpt-oss-120b`	MoE	FAIL	tool called, file unchanged

Three findings fall out:

Dense beats MoE / elastic for agentic tool calling. Every MoE and MatFormer variant failed the heavy workflow. Every dense variant passed. The jdhodges 2026 local-LLM tool-calling benchmark shows the same pattern — Qwen 3.5 4B (3.4 GB) at 97.5%, beating models 5× its size. Dense weights + good tool-call fine-tuning dominate.
Ollama beats LM Studio on the same weights. Same gpt-oss:20b, opposite results. The difference is the tool-call translation layer: Ollama maps the model's native tool-call format to the OpenAI-compatible wire faithfully; LM Studio's current implementation loses fidelity in ways that matter. This one surprised me — I'd assumed weights dominated the harness.
Size doesn't rescue you. gpt-oss-120b failed the same way as its 20B sibling. You can't out-parameter a chat-template / tool-call-format mismatch.

What to carry away

If you're building something agentic on top of local LLMs, the checklist is short:

Start dense. Qwen 3.5/3.6 or Gemma 4 dense, on Ollama, 7B minimum.
Add loop detection at the application layer. Don't trust the model to self-terminate.
Return meaningful tool results, not {"success": true}. Diff summaries let you detect hallucinated success.
Put your schema in the prompt, not just the SDK.
Bump context length to 16K+ on LM Studio and reload the model (the setting doesn't apply to already-loaded models — I wasted half a day on "Model did not produce a final response" before I realized).
Prompt against weak-model literal-mindedness. The DOCUMENT COMPLETENESS RULE pattern prevents whole classes of silent data loss.

Everything here is generalizable — none of it is specific to how SPECLAN uses MCP tools. If you've run into different failure modes on your local-LLM agentic workflows (especially with Qwen 3.6 dense, Llama 3.3, or GLM-4.7), drop them in the comments. I'm particularly interested in anyone who's gotten Qwen3.6-35B-A3B to self-terminate reliably on a 10+-step tool-calling loop — the MoE training for agentic coding is supposed to have fixed this, but I haven't verified it yet.

References

The living journey version of this post (with SPECLAN-specific details): speclan.net/blog/2026-04-25-we-gave-speclan-a-local-brain
jdhodges 2026 local-LLM tool-calling benchmark: jdhodges.com/blog/local-llms-on-tool-calling-2026-pt1-local-lm
SPECLAN on the VS Code Marketplace: marketplace.visualstudio.com/items?itemName=DigitalDividend.speclan-vscode-extension

Your PO Should Own the Spec, Not the Developer — Here's How Status Gates Fix the AI Handoff Problem

Thomas Landgraf — Sat, 04 Apr 2026 08:56:47 +0000

In most AI-assisted workflows, the developer writes the prompt and owns the outcome. The Product Owner writes a Jira ticket, the developer interprets it, feeds it to an AI agent, and 2,000 lines of code appear. Three sprints later, everyone's still arguing about what was actually specified.

The root cause isn't bad developers or bad POs. It's that nobody owns the spec as a living artifact. Jira tickets describe work to do — they die when the sprint ends. Confluence pages describe features that were planned — they go stale the moment someone changes the code. The actual intent lives in chat logs, Slack threads, and someone's memory.

What if your specs lived in Git?

The idea: each product requirement is a Markdown file with YAML frontmatter, stored in your Git repository right next to the code. One file per requirement, organized in a directory tree that mirrors your feature hierarchy. The frontmatter carries metadata — who owns it, when it was last updated, and crucially, its status.

---
id: R-4201
type: requirement
title: "Add to Cart button"
status: approved
owner: sarah
---

When a user clicks "Add to Cart" on a product page...

No external tools. No Confluence sync. No copy-pasting between systems. The spec is a file, reviewed in PRs, versioned in Git, and readable by both humans and AI agents.

I've been building a VS Code extension called SPECLAN that adds a WYSIWYG editor, spec tree view, and AI implementation tooling on top of this approach (full disclosure: I'm the creator). But the core concept — specs as files with status gates — works with any editor and zero tooling.

Here's the part that changes everything: the status field isn't just a label. It's an ownership protocol.

The status lifecycle

draft → review → approved → in-development → under-test → released

Each transition is a handoff between roles — not a Slack message, not a status change in a project management tool, but a field in the file itself, committed to Git, visible to the entire team.

Here's the key insight: the status isn't a label. It's an ownership signal.

draft means the PO is still thinking. Devs can see it but shouldn't implement it yet.
review means the PO wants the team's eyes on it.
approved means it's been reviewed and is ready to implement — the handoff moment.
in-development means the dev team (or AI agent) owns it now.
under-test means responsibility flows back to the PO — did the result match the intent?
released means everyone agrees it's done. The spec stays as a permanent record.

Walking through it: adding a shopping cart

Sarah the PO creates three Requirements in her spec tree: "Add to Cart" button, cart page with quantity editing, and cart persistence across sessions. She writes each one describing what the feature does, not how to build it. She adds Acceptance Criteria — toast notification within 500ms, cart icon updates without reload, duplicate items increment quantity.

Status: draft. The dev team can see the specs in their tree view, but the status fence prevents premature implementation. Sarah is still thinking.

She moves to review. Marco (senior dev) flags a concern — localStorage has a 5MB limit that could bite them with large carts. Sarah updates the spec. Lisa (QA) adds a missing edge case: what happens at maximum stock quantity? Sarah adds it. All of this happens in Git commits, not Jira comments.

Status moves to approved. Now the dev team takes over. The AI agent reads the approved specs directly — not from a copy-pasted prompt, but through structured tools that give it access to the full requirement text, acceptance criteria, and the spec hierarchy. It implements what was specified, not what it guesses.

After implementation, the status moves to under-test. This is the handback moment — responsibility flows back from the dev team to the PO. Sarah tests each acceptance criterion against the running system. The person who defined the requirement is the person who accepts the result.

Status: released. The spec stays in the repo as the permanent record of what the product does. Six months later, when someone asks "why does the cart sync to the server?", the answer is in the spec file — including Marco's review comment about the 5MB limit.

Why this matters more than you think

It's not just about teams

If you're a solo developer, you're already playing all these roles — you just switch between them unconsciously. You're the PO when you decide what to build. The developer when you implement. The QA when you test.

The problem is these mental switches happen mid-sentence. You're halfway through writing a spec when you think "I know how to build this" and jump straight to coding. The spec never gets finished. Two weeks later, you can't remember what you intended.

Status gates give solo developers forced phase separation:

Specificator hat — write specs, think through edge cases. No coding yet.
Implementor hat — code from the approved spec, not from memory.
Verifier hat — check your own acceptance criteria. "Did I build what I intended?"

Specs outlive sprints

This is the real differentiator from Jira. Tickets describe work. Specs describe the product.

Aspect	Jira Ticket	Spec-as-file
Describes	Work to do	Product behavior
Lifespan	One sprint	Product lifetime
Lives in	External tool	Git, next to the code
After completion	Closed, archived	Still authoritative
AI-readable	Copy-paste into prompt	Structured tools read directly

You can use both — Jira for sprint planning, spec files for the actual requirements. But the spec should outlive the sprint.

AI agents need governance, not freedom

The AI agent reads specs through structured tools, not ad-hoc prompts. It only implements approved specs. It updates the status as work progresses. The human governance layer stays intact even when the coding is automated.

This is the part most AI coding workflows get wrong. They give the AI maximum freedom and wonder why month 3 is a mess. The fix isn't more prompting. It's giving the AI a spec to follow and a lifecycle to respect.

Try it yourself

The Markdown + YAML frontmatter approach works without any tooling — it's just files in Git. But if you want the tree view, WYSIWYG editor, and AI implementation assistant on top of it:

SPECLAN on the VS Code Marketplace (free)
speclan.net — docs and methodology guide

What does your spec handoff look like? I'm curious how other teams handle the PO → dev → PO loop — especially with AI agents in the mix.

Stop Vibe Coding: What Happens When You Give Your AI Agent a Real Spec

Thomas Landgraf — Thu, 05 Mar 2026 17:51:01 +0000

Your AI coding agent can write a feature in minutes. But did it write the right feature?

I've been using Claude Code, Cursor, and Copilot for the past year, and the pattern is always the same: you describe what you want in natural language, the agent generates code, and then you spend the next hour fixing the parts it got wrong. Not because the AI is bad — but because your intent was never structured enough for it to get right.

That loop — prompt, wrong output, re-prompt, repeat — is what people call vibe coding. It works for prototypes. It doesn't work for anything you need to maintain.

The missing layer

The gap isn't in the AI's coding ability. It's between your head and the agent's context window. You know what the feature should do, how it fits into the product, what the edge cases are, and which acceptance criteria matter. The agent knows... whatever you typed into the prompt.

Spec-driven development closes that gap by structuring your intent before the agent starts writing code. Not a 40-page requirements document. Just enough structure that the AI knows:

What the feature is and why it exists (business goal)
Where it fits in the product hierarchy (parent feature)
What "done" looks like (acceptance criteria)
What status it's in (can it be implemented yet?)

What this looks like in practice

I've been building a tool called SPECLAN that takes this approach — it's a free VS Code extension that manages specifications as a tree of Markdown files with YAML frontmatter, living in your Git repository.

I recorded a 7-minute walkthrough that shows the full workflow from importing a raw product idea to orchestrating AI agents against structured work packages:

Here's what the video covers:

0:00 — The problem. Why your AI agent keeps getting it wrong, and what's actually missing.

0:25 — Installation. One click from the VS Code Marketplace.

0:40 — Importing an idea. You paste a high-level product description. SPECLAN's AI decomposes it into a hierarchy: goals, features, requirements — each as a separate Markdown file.

1:10 — The specification tree. A navigable tree view in VS Code's sidebar. Goals break down into features, features into sub-features, sub-features into requirements. The hierarchy is your product structure.

1:35 — WYSIWYG editing. A rich text editor inside a VS Code webview, so you can write specs without thinking about Markdown syntax. What you see round-trips cleanly to Markdown + YAML frontmatter.

1:55 — AI chat assistant. Ask questions about your spec, get suggestions, refine requirements — all within the editor panel.

2:15 — Copy AI Prime Context. This is where it gets practical. One click copies a structured prompt containing the spec, its parent feature, the business goal, acceptance criteria, and surrounding context. Paste that into Claude Code or any agent, and it actually knows what to build.

2:40 — Status lifecycle. Specs move through draft -> review -> approved -> in-development -> under-test -> released. Only approved specs can be implemented. This prevents the "building against a moving target" problem.

2:55 — SWARM implementation. Break approved specs into work packages and let multiple AI agents work on them in parallel — with the specification as the shared source of truth.

3:20 — Change Requests. When an approved spec needs modification, you don't edit it directly. You create a Change Request — a separate file that tracks what changed and why. No more spec drift.

3:45 — Git integration. Every spec is a Markdown file in Git. You get diffs, branches, and merge workflows for free. Your specs live next to your code, versioned the same way.

Why Markdown files in Git?

I chose this approach over a database or a cloud service for one reason: portability.

Your specs are plain text files. They work with any editor, any AI agent, any CI pipeline. If you stop using SPECLAN tomorrow, your specifications are still there — readable, diffable, greppable Markdown. No export step, no migration, no vendor lock-in.

The YAML frontmatter carries the structured metadata (ID, status, parent reference, owner), while the Markdown body carries the human-readable content. Git gives you the audit trail. The VS Code extension gives you the GUI.

The ecosystem is growing

SPECLAN isn't the only tool exploring this space. The BMAD Method uses specialized AI agent personas for structured development. OpenSpec adds a spec layer for existing codebases. GitHub's Spec Kit provides CLI templates for spec-driven workflows. Kiro from AWS takes a steering-file approach.

Each tackles the same insight from a different angle: specifications are the missing layer between human intent and AI execution. The methodology matters more than any single tool.

Try it

SPECLAN is free and open source. Install it from the VS Code Marketplace, point it at any project, and see if structured specs change how your AI agent performs.

The docs are at speclan.net. The source is on GitHub.

I'm the creator — full disclosure. I built this because I was tired of re-prompting Claude Code with the same context every session. If you have questions or feedback, I'm in the comments.

What's your experience with spec-driven development? Are you structuring your prompts before sending them to AI agents, or do you find the overhead isn't worth it? Curious to hear what's working for others.

How I Use .claude/rules/ to Give Claude Code Domain Knowledge About My Project's File Structure

Thomas Landgraf — Wed, 04 Mar 2026 22:17:04 +0000

You know that moment when you ask Claude Code to edit a file and it treats your carefully structured project directory like a random pile of Markdown? It adds implementation details to a specification file. It puts a requirement under the wrong feature. It invents an ID format you never asked for.

The problem isn't that the AI is dumb. It's that it has no idea what your files mean.

I've been building a VS Code extension called SPECLAN that manages layered specifications as Markdown files with YAML frontmatter. The speclan/ directory in any project has a well-defined structure — entity types, ID schemes, status lifecycles, nesting rules. And Claude Code kept stepping on all of them until I discovered the paths frontmatter in .claude/rules/.

The problem: Claude doesn't know your conventions

My project has a directory like this:

speclan/
├── goals/           G-###-slug.md
├── features/        F-####-slug/F-####-slug.md
│   ├── requirements/  R-####-slug/R-####-slug.md
│   │   └── change-requests/  CR-####-slug.md
│   └── change-requests/  CR-####-slug.md
└── templates/

Every file is Markdown with YAML frontmatter. IDs are random, not sequential. Features nest recursively. Requirements always belong to exactly one feature. Status goes draft → review → approved → in-development → under-test → released → deprecated. Only approved specs can be implemented. Locked specs need a ChangeRequest to modify.

None of this is obvious from the files alone. Without guidance, Claude will:

Create sequential IDs (F-0001, F-0002) instead of random ones
Put requirements at the wrong nesting level
Mix implementation concerns into specification files
Skip required frontmatter fields
Ignore the status lifecycle entirely

The solution: path-scoped rules

Claude Code loads .claude/rules/*.md files as persistent context. That alone is useful for project-wide conventions. But the feature that makes it powerful for structured directories is the paths frontmatter:

---
paths:
  - "speclan/**/*.md"
---

This tells Claude Code: "Only inject these rules when I'm working with files that match this glob pattern." The rules file is invisible when you're editing TypeScript, writing tests, or doing anything else. But the moment you touch a file under speclan/, it kicks in.

Rules without a paths field load unconditionally — they're the equivalent of putting instructions in CLAUDE.md. Rules with paths only activate when Claude reads files matching the pattern. That distinction is what makes them useful for domain-specific knowledge.

What goes in the rules file

Here's the actual rules file I use (condensed — the real one is ~96 lines):

---
paths:
  - "speclan/**/*.md"
---

# SPECLAN Specification Rules

## Entity Hierarchy

Goal (G-###) → Feature (F-####) → Requirement (R-####)

ChangeRequest (CR-####) modifies locked entities.

## Directory Structure

speclan/
├── goals/           G-###-slug.md
├── features/        F-####-slug/F-####-slug.md (self-named dirs, recursive)
│   ├── requirements/  R-####-slug/R-####-slug.md
│   │   └── change-requests/  CR-####-slug.md
│   └── change-requests/  CR-####-slug.md
└── templates/<entityType>/  UUID-slug.md

## Frontmatter (YAML)

All specs are Markdown with YAML frontmatter. Required fields:
id, type, title, status, owner, created, updated

## ID Rules (NON-NEGOTIABLE)

- Goal: G-### (3 digits)
- Feature: F-#### (4 digits)
- Requirement: R-#### (4 digits)
- ChangeRequest: CR-#### (4 digits)
- IDs are random, not sequential
- Check collisions before creation

## Status Lifecycle

draft → review → approved → in-development → under-test → released → deprecated

Only approved specs can be implemented.
Locked statuses (approved+) require a ChangeRequest for modifications.

## Invariants

1. Requirements belong to exactly one Feature
2. Features may have sub-features AND requirements
3. ChangeRequests reference exactly one parent

IMPORTANT: files under speclan/ are specifications that tell
WHAT from user perspective, not HOW from developer perspective

That last line is the most important one. It's the semantic boundary that prevents Claude from mixing concerns. Without it, you ask for a new requirement and get implementation pseudocode.

Why this works better than CLAUDE.md alone

You could put all of this in your project's CLAUDE.md. I actually did that first. The problem is context pollution — when Claude is editing a React component, it doesn't need to know about SPECLAN's ID scheme. And when it's editing a spec file, it doesn't need your TypeScript lint rules.

Path-scoped rules solve this cleanly:

Focused context — rules only activate when relevant
No noise — Claude's context window isn't cluttered with irrelevant conventions
Composable — you can have multiple rules files for different parts of your project

The rules file acts like a domain expert sitting next to Claude, whispering "that's a specification file, here's how they work" exactly when it matters.

Designing a good rules file

After iterating on this for a few months, here's what I've found works:

Be declarative, not procedural. Don't write step-by-step instructions. Describe the structure, the constraints, the invariants. Claude is good at applying constraints if you state them clearly.

Mark hard boundaries. I use (NON-NEGOTIABLE) for rules that must never be violated — like the ID format. Claude respects this surprisingly well.

Include the "why" for non-obvious rules. "IDs are random, not sequential" needs the implicit why: collision avoidance across branches and contributors. "Files tell WHAT not HOW" needs no explanation but does need emphasis.

Keep it under 100 lines. This is context that gets injected into every relevant interaction. If your rules file is 500 lines, you're eating into the context window Claude needs for actual work. Compress ruthlessly. Tables over prose. ASCII trees over paragraphs. The official docs recommend targeting under 200 lines for any CLAUDE.md file — for path-scoped rules, I'd argue even tighter is better.

Quote your glob patterns. This is a gotcha that'll bite you: YAML treats * and { as reserved indicators. Always quote your patterns — "**/*.ts" not **/*.ts. Unquoted patterns can silently fail.

Use brace expansion for related types. Instead of listing patterns separately, combine them: "src/**/*.{ts,tsx}" matches both TypeScript and TSX files in one pattern. Same works for directories: "{src,lib}/**/*.ts".

Test it by asking Claude to create something. After writing the rules file, ask Claude to "create a new requirement for feature F-1234." If it gets the file path, ID format, frontmatter, and directory nesting right on the first try — your rules file works.

Beyond project directories: glob patterns for other domains

The speclan/**/*.md pattern is one application. The same mechanism works for any file pattern where Claude needs domain context. Here's what I use across my NX monorepo:

Test files (`/.spec.ts`)* — inject your testing conventions: which frameworks, which patterns, how to mock, what not to test. I have rules for Jest vs Mocha conventions since my project uses both (libraries vs VS Code extension).

Webview files (`/webview/`) — inject your browser-context constraints: no Node APIs, specific CSS framework rules, message-passing protocols between webview and extension host.

Infrastructure files (`cdk//.ts`)* — inject your CDK conventions, naming standards, tagging policies, security guardrails. Claude loves to create overly permissive IAM roles unless you tell it not to.

Security-sensitive code (`src/auth//, src/payments//`)** — guardrails for sensitive areas: never log tokens, always parameterize queries, validate all inputs at function boundaries. These rules are especially valuable because the cost of Claude getting them wrong is high.

Database migrations (`prisma/migrations//`)* — safety rules: always include rollback instructions, never delete columns in the same migration that removes the code using them, add columns as nullable first.

The pattern is always the same: you have files where the semantics aren't obvious from the syntax, and you need Claude to understand the domain rules before touching them.

Tips from the trenches

A few more things I've learned from running 12+ rules files across a monorepo:

One concern per file. A testing.md shouldn't also contain API design guidelines. Separation of concerns applies to instructions just as much as code. Descriptive filenames like api-validation.md beat rules1.md.

Subdirectories work. All .md files are discovered recursively, so you can organize rules into frontend/, backend/, infra/ subdirectories. No configuration needed.

Symlinks for shared rules. If you maintain coding standards across multiple projects, symlink a shared rules directory: ln -s ~/company-standards .claude/rules/shared. Circular symlinks are handled gracefully.

User-level rules for personal preferences. Put rules in ~/.claude/rules/ for things that apply to everything you work on — your preferred commit message format, your testing style, your debugging workflow. These load before project rules, so project rules can override them.

Don't duplicate between CLAUDE.md and rules. If a convention is path-specific, put it in .claude/rules/ with a paths field. If it's truly universal (build commands, project architecture), keep it in CLAUDE.md. Conflicting instructions across files get resolved arbitrarily — not what you want for your ID scheme.

Check what's loaded with /memory. When something isn't being respected, run /memory to see which rules files Claude actually has in context. If your file isn't listed, the glob pattern isn't matching.

The compound effect

One rules file doesn't feel like much. But once you have 3-4 of them covering different parts of your project, Claude starts behaving like a developer who actually read the architecture docs. It stops guessing and starts following your conventions. The number of "no, that's not how we do it" corrections drops dramatically.

I think of .claude/rules/ files as executable documentation. They serve double duty: they document your conventions for human readers and they enforce those conventions when AI touches the code. That's a pretty good return on 50-100 lines of Markdown.

I'm the creator of SPECLAN, a VS Code extension for managing layered specifications as Markdown files in Git. The path-scoped rules described here are how I keep Claude Code aligned with SPECLAN's file structure conventions — but the technique works for any project with well-defined directory semantics.

I built a spec management extension with a WYSIWYG Markdown editor in a VS Code webview — lessons learned

Thomas Landgraf — Tue, 03 Mar 2026 20:18:28 +0000

I've been building a VS Code extension for spec management over the past 3 months (full disclosure: I'm the creator, it's called SPECLAN — free side project). The idea is that specifications need the same structure we give source code: hierarchy, types, lifecycle tracking. So the extension organizes specs as a tree of Markdown files with YAML frontmatter — goals break down into features, features into sub-features, sub-features into requirements. Each file has a status lifecycle (draft → review → approved → in-development → released) so you always know what's specced, what's being built, and what needs to change.

The interesting VS Code challenge: making this usable for non-technical people. Product managers and business analysts define what to build, but they won't write raw Markdown with YAML frontmatter. So I needed a WYSIWYG editor inside a webview that round-trips cleanly to Markdown — same file in Git, two editing experiences.

That editor ate about 40% of total development effort. Here's what I learned.

The stack:

Quill 2.x in a VS Code webview (rich text editing)
remark + remark-gfm for Markdown → HTML on load
turndown + turndown-plugin-gfm for HTML → Markdown on save
gray-matter for YAML frontmatter — strips on load, reattaches on save
quill-table-up for GFM tables (Quill has no native table support)

What actually hurt:

Round-trip fidelity. The pipeline is Markdown → HTML → Quill Delta → HTML → Markdown. Every step is lossy. Links, emphasis, nested lists — they all drift across conversions. I spent weeks writing custom turndown rules to keep Markdown output stable. If you're building something similar: start with the save pipeline, not the editor. The round-trip is the constraint that shapes everything.
Frontmatter is invisible but critical. Each spec file has 10+ YAML fields — status, entity ID, parent references, timestamps. The editor only sees the Markdown body, but the file is meaningless without its frontmatter. gray-matter handles parsing, but you need to be careful that editor changes don't conflict with frontmatter values (e.g., someone editing a title in the body that's also in the YAML).
Tables. Quill doesn't do tables. quill-table-up adds them, but serializing table HTML through turndown into GFM pipe tables has edge cases everywhere — empty cells, inline formatting in cells, nested content.
Webview communication. Everything between the editor (iframe) and the extension host is a postMessage call — load, save, dirty state, undo, external file change detection. I ended up building a structured message protocol with typed handlers on both sides. console.log in the webview doesn't show up anywhere useful, so I added a logging bridge that routes webview logs to the extension's output channel.
Custom editor API. Using CustomTextEditorProvider means the document model is VS Code's TextDocument but the visual state is Quill's Delta. Keeping these in sync — especially during concurrent edits or Git operations that change the file underneath — required careful event sequencing.

What worked well:

The file-system-as-data-model approach. Directories ARE the spec hierarchy. speclan/features/F-1234-auth/requirements/R-5678-login/R-5678-login.md — any tool (or AI agent) can understand the structure by reading the file system. No database, no server.
Snapshot testing the conversion pipeline. Take a Markdown file, push it through the full round-trip, diff the output. Catches regressions fast.
Tree views for navigation. VS Code's TreeDataProvider is excellent. The spec tree (goals → features → requirements) renders as a native sidebar with status icons, drag-and-drop reordering, and context menus. Much less effort than the WYSIWYG editor.

Happy to answer questions about webviews, the conversion pipeline, or the spec structure approach.

Marketplace | speclan.net | GitHub