DEV Community: Josh

Your coding agent's summary of its own work is not evidence

Josh — Sun, 09 Aug 2026 16:53:17 +0000

I spent three days last week doing real work on a production codebase with an agentic coding tool. Small site, live customers, money moving through it. The kind of place where a bad commit costs something.

The work went well. Three commits shipped, all verified, all live. But a pattern showed up on day one and repeated often enough that I now build my sessions around it:

The agent's summary of what it did was wrong often enough that I stopped reading summaries.

Not wrong in a dramatic way. Wrong in the quiet way that passes a skim.

Four times it happened

It told me nothing was staged. It had already committed.

I asked for a diff, got one, and got a closing line: "Nothing staged." Two turns later I asked it to stage and commit. It replied that the commit already existed — same hash, same message — and correctly refused to commit unrelated files under a message that didn't describe them.

Good refusal. But the earlier report had been wrong, and I'd built a step on top of it.

It silently dropped half a request.

I asked for two checks in one message: a repo-wide grep, and a CSS dump. It returned the CSS in detail, said nothing about the grep, and signed off with "Nothing was edited or staged" — which reads like a complete report. The missing half was the half that would have told me whether the thing I was about to edit was generated by a build script or hand-written markup.

That's the check that decides whether your edit survives the next build.

It added a table up by eye and got the wrong number.

It reported "21 links changed across 8 files" and said this matched an earlier count. It didn't. When I asked it to grep and count properly, the real number was 20 — and its own per-file table, printed two messages earlier, had summed to 20 all along.

Worth noting: I'd independently eyeballed the same table and got 18. Both of us added up a list without counting it. Neither of us caught it until a grep -o settled it.

It asserted byte-identity where it meant functional equivalence.

A refactor moved a chunk of HTML into a shared function. The agent said the output was "byte-identical, not just equivalent." It wasn't — the new version built the same tag through a different string-escaping route, so the emitted attribute delimiters differed. The browser doesn't care. The claim was still false, stated with more confidence than the situation supported.

Why this isn't a complaint about a tool

Every one of these is a model reporting on its own recent work, which is the one thing it's structurally worst at.

A model that just generated something carries the reasoning that produced it. When it then reviews that work, it isn't approaching the artifact fresh — it's approaching its own intention, which is always cleaner than the output. That's why "check your work" produces so little, and why a second, independent pass catches things the first one can't.

Same reason code review works better when someone else does it.

The practical upshot is that the agent's report and the agent's output are two different artifacts with different reliability. The output was almost always fine. The report was the thing that drifted.

What I do now

These are session rules, not prompts. I paste them at the top of every session.

Print it, don't summarise it. "Print the complete unified diff, every hunk, in full. Do not summarise what you changed." The difference in outcome is large. A summary is the model's model of the change. A diff is the change.

Staging is not a verification step. The agent had staged past a verification checkpoint four times in three days. Now the rule is explicit: print, then I read, then stage, then commit, then push — and push only on a separate instruction.

Locate fresh from disk every time. Line numbers in your notes go stale the moment anything is edited. Any prompt that says "the function at line 1640" is a prompt that will eventually edit the wrong thing. Say "find it by searching for this literal string" instead.

Ask for the check that would falsify you. Not "confirm the tests pass" but "print the output of the test run." Not "is this still there?" but "grep for it and print every match with file and line." One of these can be answered from memory. The other can't.

Grep fragments, not sentences. After removing a phrase, searching for the whole sentence tells you the sentence is gone. It doesn't tell you a fragment survived somewhere else. Search the shortest distinctive substring.

Count with a tool, not with your eyes. Both the agent and I got the link count wrong from the same table. grep -c doesn't have that failure mode.

Verify on the artifact, not the diff. For a web change, that means loading the live page in a fresh private window. A clean diff and a green build still leave room for a broken deploy, a stale cache, or a file that never regenerated.

The part that generalises

If you're building agentic systems rather than just using them, this maps onto something specific: the distinction between what a model reports and what actually happened is an architectural concern, not a prompting one.

You don't fix it by asking nicer. You fix it by designing the loop so that verification comes from somewhere the model can't narrate — a tool result, a test exit code, a diff, a fresh instance without the generating context. If your agent's only evidence that a step succeeded is the agent saying so, you have a system that degrades silently.

That principle shows up all over production agent design: independent review passes rather than self-review instructions, programmatic prerequisites rather than prompt-based ordering, structured tool errors rather than a model's account of what went wrong. It's also, not coincidentally, a large chunk of what the Claude Certified Architect – Foundations exam tests — the difference between deterministic guarantees and probabilistic compliance is a recurring theme, and it's recurring because it keeps mattering.

What I'd tell someone starting

Agentic coding is genuinely good. Three commits across those days, on a live codebase, each properly scoped, each verified, none reverted. That's a real gain and I'm not going back.

But the failure mode isn't the one people warn about. It isn't hallucinated APIs or confidently wrong code — those are loud and you catch them immediately.

It's the quiet report that reads complete and isn't. It costs you nothing at the time and one debugging session later.

Make it print.

Claude Certified Architect Exam: How to Know If You're Ready Before Test Day

Josh — Fri, 12 Jun 2026 15:35:30 +0000

The Claude Certified Architect Exam is becoming one of the most sought-after certifications for professionals building AI-powered applications with Claude. As organizations increasingly deploy AI into production environments, they need architects who understand far more than prompting—they need professionals who can design reliable, scalable, and effective AI systems.

But one question stops almost every candidate:

"Am I actually ready for the exam?"

Why the Claude Certified Architect Exam Is Challenging

The exam tests practical architectural knowledge across multiple domains. Candidates are expected to understand:

Agentic Architecture & Orchestration
Tool Design & MCP Integration
Claude Code Configuration & Workflows
Prompt Engineering & Structured Output
Context Management & Reliability

Many questions require applying concepts to real-world scenarios rather than simply recalling facts.

The Problem with Most Study Plans

Most candidates begin by reading documentation, watching videos, and experimenting with projects.

Weeks later, they still don't know:

Which topics they're strongest in
Which areas need the most improvement
Whether they're actually ready for the exam

Without a baseline assessment, preparation becomes guesswork.

Start with a Free Readiness Diagnostic

Before spending dozens of hours studying, take a free diagnostic assessment at:

https://www.claudecertifiedarchitects.com/

The diagnostic evaluates your knowledge across all five exam domains and helps identify the areas where additional study will have the biggest impact.

Coverage Across All Five Domains
Agentic Architecture & Orchestration

Understand your readiness for questions involving workflow design, agent coordination, task decomposition, and orchestration patterns.

Tool Design & MCP Integration

Evaluate your knowledge of tool calling, integration strategies, MCP concepts, and architectural decision-making.

Claude Code Configuration & Workflows

Assess your familiarity with project setup, workflows, configuration patterns, and development best practices.

Prompt Engineering & Structured Output

Measure your understanding of prompt design, output reliability, validation strategies, and structured generation.

Context Management & Reliability

Identify strengths and weaknesses related to context handling, reliability engineering, memory management, and failure recovery.

Why a Diagnostic Matters

A focused study plan consistently outperforms a broad study plan.

When you know your weakest domains, you can prioritize your preparation and avoid wasting time reviewing concepts you've already mastered.

Take the Free Claude Certified Architect Diagnostic

If you're preparing for the Claude Certified Architect Exam, start by understanding where you stand today.

Take the free diagnostic now:

https://www.claudecertifiedarchitects.com/

Your future exam score may depend less on how much you study and more on whether you're studying the right things.

# Cracking the Claude Certified Architect Exam: Production-Grade AI is a Different Beast

Josh — Thu, 11 Jun 2026 15:44:11 +0000

If you’ve been building LLM applications for a while, you know that the gap between a cool local demo and a robust, enterprise-grade production system is massive. Local scripts don’t have to handle multi-agent race conditions, cascading tool failures, or prompt dilution across thousands of tokens.

Anthropic explicitly targeted this gap when they launched the Claude Certified Architect – Foundations (CCA) exam.

Unlike entry-level cloud certifications that test your ability to memorize product names or parameter limits, the Claude Certified Architect exam is a purely scenario-based, architecture-level challenge. It doesn't care if you've memorized the documentation; it cares if you know how to build systems that don’t break when real-world data hits them.

If you are thinking about sitting for the proctored exam, here is exactly what you need to know about its architecture, the core anti-patterns Anthropic will use to trick you, and how to prepare.

What the Exam Actually Tests (The 5 Domains)

The exam structures its 60 multiple-choice questions across 5 core technical domains. The weighting leans heavily into orchestration and tool execution:

Agentic Architecture & Orchestration (~27%): Designing deterministic agentic loops, subagent spawning, and state management.
Tool Design & MCP Integration (~18%): Writing robust Model Context Protocol (MCP) tool schemas and descriptions that prevent routing confusion.
Claude Code Configuration (~20%): Mastering CLAUDE.md hierarchy, plan mode vs. direct execution, and CI/CD automation.
Prompt Engineering & Structured Output (~20%): Enforcing JSON schemas through tool selection rather than raw text prompting.
Context Management & Reliability (~15%): Preventing context window degradation, managing progressive summarization, and building fallback/human-in-the-loop loops.

Instead of generic questions, the test pulls from 4 out of 6 hidden production scenarios (ranging from a multi-agent automated research engine to a code generation pipeline running in CI/CD). Every single question drops you into one of these environments and forces you to make a high-stakes architectural decision.

3 Core Anti-Patterns to Memorize Before the Test

Anthropic designs its distractor choices brilliantly. They will offer you answers that sound like totally reasonable "prompt engineering" fixes, but are actually architectural brittle points. Look out for these three traps:

1. Parsing Natural Language for Loop Termination

When building autonomous agent loops, a classic mistake is instructing Claude to write "STOP" or "FINISHED" in the text when it is done, and having your Python/TypeScript app regex-parse the output.

The Architect Way: You must track the underlying stop_reason field via the Messages API. The loop should cleanly continue while stop_reason == "tool_use" and terminate only when it shifts to "end_turn".

2. Overloading Single Agents with Tools

Giving a single customer support agent 18 different tools degrades its reasoning quality and leads to misrouting (e.g., calling get_customer_data instead of lookup_order because the text descriptions overlap).

The Architect Way: Adhere to the strict rule of keeping 4–5 tools per agent. If you have more requirements, implement a Hub-and-Spoke architecture where a main coordinator agent decomposes the query and dispatches it to isolated, specialized subagents.

3. Prompt-Based Compliance vs. Programmatic Hooks

The exam will present scenarios where sensitive corporate policies must be enforced (e.g., never process a refund greater than $500 without manager approval). A distractor answer will suggest adding this rule to the system prompt.

The Architect Way: Prompts have a non-zero failure rate. Critical business logic must be programmatically enforced using PreToolCall or PostToolUse hooks in the Claude Agent SDK to physically intercept and block unauthorized actions outside the model’s discretion.

How to Practice and Diagnose Your Knowledge

Because this test evaluates your engineering instincts rather than rote memorization, traditional flashcards won't cut it. You need to simulate the cognitive friction of diagnosing real architectural problems.

To help the developer community bridge this gap, I built a completely free diagnostic tool that mimics the style, depth, and scenario-driven nature of the actual Anthropic exam:

🛠️ Test your readiness instantly: claudeCertifiedArchitects.com

A Recommended Study Strategy

If you want to clear the 720/1000 passing score threshold on your first try, structure your prep like this:

Read the Community Guide: Clone the community-driven study resources on GitHub to understand the primitives of the Model Context Protocol (MCP) and the Agent SDK lifecycle hooks.
Take the Diagnostic: Go to claudeCertifiedArchitects.com to uncover your hidden blind spots. Do you actually know how context passing changes when spawning subagents via the Task tool? The diagnostic will tell you.
Build a Tool Failure Loop: Don't just read about error handling. Write a small script where an MCP tool deliberately returns an error, and practice structuring the response metadata (isRetryable, errorCategory) so a coordinator agent can intercept it gracefully without crashing your loop.

Building systems with Claude is incredibly rewarding, but enterprise applications demand deterministic guardrails. Earning your certification proves you know exactly where to draw the line between software engineering logic and model intelligence. Good luck!

The 5 things the Claude Certified Architect exam actually tests (and the gotchas)

Josh — Mon, 08 Jun 2026 07:26:32 +0000

Anthropic's Claude Certified Architect – Foundations exam is scenario-based: 60 questions, 120 minutes, 720/1,000 to pass. It's not an "AI literacy" badge — it tests production judgment. Here's the domain breakdown and the specific traps in each.

Agentic Architecture & Orchestration — 27%
The biggest domain. The agentic loop is simple in principle: stop_reason: "tool_use" means execute the tools and loop back; stop_reason: "end_turn" means stop. The trap is anything else — checking for "done" in the assistant's text, or capping at N iterations as your primary control. A hard cap is a safety net, not the signal.
Claude Code Configuration & Workflows — 20%
Lots of CLAUDE.md and CI/CD. A favorite gotcha: a Claude Code job in CI hangs forever because it was launched without -p. Without the print flag it starts an interactive session and waits on stdin that never comes. -p runs headless: process, print to stdout, exit.
Prompt Engineering & Structured Output — 20%
Few-shot patterns, JSON schemas, output validation. The exam assumes you can prompt; it tests whether you can make output reliable.
Tool Design & MCP Integration — 18%
Tool descriptions, MCP servers, error handling. Key principle: when a tool fails, return a structured error the model can read and recover from — don't throw and kill the loop, and don't return an empty string it'll silently misread.
Context Management & Reliability — 15%
Long sessions, escalation, provenance, and a subtle one: the same Claude instance that generated code will rationalize its own decisions when asked to review them — an independent instance catches more.

How to prep: figure out your weak domain first. I put together a free, no-signup diagnostic that scores you across all five in ~10 minutes: https://www.claudecertifiedarchitects.com/

Independent study resource — not affiliated with or endorsed by Anthropic. Verify current exam details against the official guide before you sit it.

I made a CCA Foundations practice exam platform — 3 free questions inside

Josh — Mon, 11 May 2026 14:15:17 +0000

Been prepping for the Claude Certified Architect exam and couldn't find any realistic practice questions anywhere, so I spent the last few weeks building my own.

There's a lesson plan and you can use the 10-question speed tier as many times as you want (and it will shuffle the questions). There is a paid tier for those who want to practice the full 2-hr exam with all the questions.

Here are 3 free ones to give you a feel for the difficulty level:

Q1 — Agentic Architecture
A coordinator agent spawns three subagents in parallel. One fails mid-task. What is the correct behavior?
A) Terminate all subagents and restart from scratch
B) Log the failure silently and return partial results
C) Surface the error to the coordinator to handle recovery
D) Retry the failed subagent indefinitely
✅ Answer: C — subagents don't self-recover. The coordinator owns all error handling.

Q2 — Claude Code
You want Claude Code to run your test suite automatically after every file edit. Where do you configure this?
A) A shell alias in your terminal
B) PostToolUse hooks in your Claude Code config
C) A custom slash command in CLAUDE.md
D) .claude/settings.json under postEditHook
✅ Answer: B — PostToolUse hooks are designed exactly for this.

Q3 — Context Management
Your agent processes a 200-page document and produces inconsistent outputs. Expanding the context window doesn't help. Why?
A) The model needs more few-shot examples
B) The system prompt is conflicting with the document
C) Attention dilution — quality degrades as context grows, regardless of window size
D) The document format isn't supported
✅ Answer: C — more context ≠ better attention. Split into focused per-section passes instead.

I built 300 questions total across all 5 domains with timed sessions and a full 60-question mock exam. The sprint above is free with no signup.
Search "Claude Certified Architects" to find it, or drop a comment and I'll DM you the link.
Happy to answer questions about any of the exam domains — good luck to everyone prepping 🙌