DEV Community: Luka Zmikic Engels

A Property Inquiry Agent with React, MCP, and Hybrid RAG

Luka Zmikic Engels — Tue, 29 Sep 2026 20:23:02 +0000

A property assistant gets a question:

“I’m looking for a flat in Hamburg under €2,000. I have a dog. Is heating included, and can I view it on Saturday?”

That sounds like one request. It actually contains several jobs: filter properties, check a pet policy, find the heating details, and arrange a viewing.

Some of those jobs have answers in the records. Others need a person.

That distinction is the starting point for proptech-inquiry-agent: a TypeScript project that gives an AI assistant tools to search property records, retrieve supporting passages, and create human hand-off tickets.

You interact with it through a React inquiry desk. Behind the interface, an agent chooses tools, gathers evidence, checks its draft, and returns an answer or a hand-off. You can try the browser interface without a model API key. All property and policy data is synthetic.

The MCP tools, hybrid retrieval, agent loop, answer checks, and web interface are implemented. A 49-case evaluation suite now measures answer correctness and hand-offs beyond the existing tests. Its first runs found problems in retrieval, follow-up replies, and the evaluator itself; after fixes, 46 of 49 cases pass. This is still a work in progress, with the remaining failures recorded alongside the results.

A desk where you can see what happened

The browser application has three panes, each answering a different question: what did the assistant say, what did it do, and what needs a person?

+---------------------+----------------------+---------------------+
| Conversation        | What the agent did   | Hand-off queue      |
+---------------------+----------------------+---------------------+
| Ask a question      | Search listings      | Viewing request     |
|                     | Read the evidence    | HH-1001             |
| Receive a checked   | Create a ticket      | Inquiry summary     |
| reply with sources  |                      | Contact details     |
|                     | Expand a step for    | New tickets appear  |
| Click a citation    | arguments, result,   | as they are created |
| to read its source  | and duration         |                     |
+---------------------+----------------------+---------------------+

This demo shows a fact question followed by a viewing request through the same interface. Click it to open the full-size animation.

The left pane has example buttons for a fact question, a viewing request, an unanswered question, and a German inquiry. You can follow up with “and the deposit?” without repeating the property ID, or select “New conversation” to start over. The middle pane exposes the actual tool calls, including errors and failed answer checks. The right pane shows tickets for a human to handle.

Clicking a citation opens a dialog with the source document and the cited passage highlighted. A citation for a whole property shows its structured fields and description. This makes it possible to inspect the evidence behind the reply without searching through logs.

The application uses React state, Vite, and one plain stylesheet. The UI makes the existing workflow visible; the server still owns the tools, retrieval, and answer checks.

What happens behind the interface

Submitting an inquiry starts an agent loop on the server. The loop asks a model which tools to use, executes those calls, and checks the resulting answer. The browser displays the progress and final result. The same workflow is available through a CLI or HTTP API.

Customer inquiry
       |
       v
React desk, CLI, or HTTP API
       |
       v
Agent loop <--------------------> Model implementation
       |                         Claude via API / Bedrock
       |                         or rule-based demo
       |
       +── MCP tools ──> Property records + local retrieval
       |             └─> Temporary hand-off queue
       |
       v
Check the draft answer
       |
       v
Reply + citations + tickets + tool trace + token usage

MCP, the Model Context Protocol, is the connection that lets an assistant discover and call tools. The built-in agent uses an MCP client connected to this repository's server in the same process. It goes through the same tool interface as an external assistant, without an extra HTTP round trip.

The repository's architecture guide maps this request flow, the escalation boundary, where each check runs, and which parts work offline.

The server exposes four tools:

Tool	Job
`search_listings`	Find properties matching explicit requirements.
`get_listing`	Return one complete property record.
`search_knowledge`	Retrieve passages from descriptions and policies.
`hand_off_to_human`	Create a ticket with a specific reason.

External assistants such as Claude Desktop can use the tools through standard input and output, or through the Streamable HTTP endpoint at /mcp. General policy pages are also available as MCP resources.

What the agent loop actually does

The loop is ordinary TypeScript. It asks the model what to do next, executes any requested tools, and sends the results back. It repeats until the model produces a draft answer or the run reaches a limit.

Inquiry + instructions + available tools
                   |
                   v
              Call the model <----------------+
                   |                          |
             What came back?                  |
              /           \                   |
             v             v                  |
       Tool requests    Draft answer          |
             |             |                  |
             v             v                  |
       Execute via MCP  Run answer checks     |
             |                                |
             +── Send tool results back ──────+

The system prompt asks the model to answer from tool results, cite its facts, preserve qualifications such as “on request,” and hand off the parts it cannot resolve. The model can answer the heating question and create a viewing ticket in the same run.

The model adapter supports the Anthropic API and Amazon Bedrock. The loop itself depends on a small ModelClient interface, so tests can substitute a scripted model. Model and effort settings are configurable; the adapter also requests prompt caching and records cache token usage.

Each run returns a structured result:

reply       The final text
citations   Source markers parsed from that text
handoffs    Tickets created during this inquiry
trace       Tool names, arguments, results, and durations
usage       Input, output, and cache token counts
outcome     Answered, or forced hand-off with a cause

An answered outcome can still include a viewing ticket. It means the loop accepted the final reply, not that every part of the inquiry was resolved automatically.

Use ordinary code for ordinary filters

“Hamburg, under €2,000, at least three rooms” does not require semantic search.

The catalogue is a JSON file. Each property has typed fields for its city, price, room count, floor area, features, availability, and pet policy. Zod validates the data when it loads.

search_listings checks those fields directly and sorts matching properties by price.

City: Hamburg       ──┐
Price: ≤ €2,000      ──┼──> Field filters ──> Matching properties
Customer has a dog  ──┘

One deliberate detail: properties marked “pets on request” remain in the results. Excluding them would hide possible options. Their status remains visible so the assistant can explain that permission is still needed.

For example, the synthetic property HH-1001 has a monthly price of €1,650 and pets allowed “on request.” That is a possible match, not approval for the customer's dog.

Search the text when the fields are not enough

The heating question needs a different approach. The answer appears inside the description:

“Utilities (Nebenkosten) are an additional 240 EUR per month, heating included.”

The retrieval system splits property descriptions into sentences and policy pages into sections. These small passages are called chunks. Each carries its source and an identifier for citations.

Retrieving evidence for the model before it writes an answer is called retrieval-augmented generation (RAG). Here, the retrieval is hybrid: it combines keyword search with embedding-based meaning search, then uses a reranker to assess the candidates.

Search then happens in two stages:

              “Is heating included in HH-1001?”
                              |
                              v
               Restrict the searchable material
               to HH-1001 + general policies
                              |
                  +-----------+-----------+
                  |                       |
                  v                       v
             Keyword search          Meaning search
                 BM25                Local embeddings
                  |                       |
                  +-----------+-----------+
                              |
                              v
                    Candidate passages
                              |
                              v
                 Local reranker scores each
                 question-and-passage pair
                              |
                              v
                   Relevance threshold
                     /             \
                    v               v
           Evidence + source ID   found: false

Keyword search helps with distinctive terms such as Nebenkosten and Staffelmiete. Meaning search helps with paraphrases and questions in German or English.

For meaning search, multilingual-e5-small converts text into numerical representations called embeddings. Similar representations help identify related passages.

The system collects up to 20 candidates from each search method and removes duplicates. A second model, bge-reranker-v2-m3, scores each candidate together with the question. Only passages above the configured threshold are returned.

Both models run locally through Transformers.js after their initial downloads. The conversational assistant is separate; local retrieval does not mean the whole conversation runs offline.

The nearest passage may still be useless

The most useful finding in this project came from asking questions the data could not answer.

An embedding search can find the closest passage even when every passage is irrelevant. The system therefore needs a way to return “nothing suitable found.”

The project's recorded experiments show why this was difficult. An unanswerable gym question received an embedding similarity score of 0.832, while an answerable German pet question scored 0.784. A simple similarity cutoff could not reliably separate them.

A second attempt measured how much a passage stood out from the other candidates. That also failed on some unanswered questions.

Adding the reranker gave the project a better signal for deciding which passages to return. Its score is still a model output, not a guaranteed probability that an answer is correct. The threshold needs evaluation against the actual corpus.

Another experiment changed what went into the embeddings:

Before:  Property title + location + passage  ──> Embedding
After:   Passage alone                      ──> Embedding

On the recorded 25 answerable questions, the expected passage appeared in the top five vector results for 20 questions with headers and 23 without them. Headers remain useful for keyword search and for the reranker, which reads each passage with its context.

This is a small experiment, but it changed the implementation: adding context was helpful in some parts of the search pipeline and harmful in another.

Keep one property's facts separate from another's

When a question includes a listing ID, the code excludes other properties before ranking.

Question about HH-1001
          |
          +── HH-1001 description       Included
          +── General agency policies  Included
          +── Other property records   Excluded

Among passages that pass the relevance threshold, the property's own passages come before general policies. A property's specific deposit amount should take precedence over a general explanation of deposits.

This is a boundary enforced by code. It does not depend on the assistant remembering to ignore another property's heating bill.

A ticket is not a completed viewing booking

The hand-off tool supports six reasons:

Viewing requests
Price negotiations
Contract or legal questions
Complaints
Personal data requests
Questions the records cannot answer

A ticket includes its reason, a summary, an optional property ID and contact email, a timestamp, and a generated ID. The tool rejects unknown property IDs.

“Can I view it Saturday?”
           |
           v
hand_off_to_human
reason: viewing_request
           |
           v
Ticket stored in memory
           |
           v
Human follow-up still required

Today, this is an in-memory queue. It does not send an email or reserve a calendar slot, and tickets disappear when the process stops.

Check the answer before returning it

The checks in packages/core/src/agent/guards.ts examine a draft before the loop returns it to the caller.

Check	What the code looks for
Citation	Recognized citation markers must refer to listing or passage IDs returned by a tool during this run.
Number	Detected prices, areas, and percentages must match a number in a successful tool result from this run or the user's current or retained earlier inquiries.
Missing evidence	A reply containing those figures must include at least one recognized citation.
Empty reply	The answer must contain text.

For example, [HH-1001#s5] is accepted only if that passage ID appeared in the run's tool results. A newly invented price should fail the number check. German and English number formats are handled, so 1.650 € can match 1650 in the record.

If a check fails, the user does not receive that draft. The model gets feedback and one repair opportunity, which may include more tool calls.

                  Draft answer
                       |
                       v
                  Run checks
                  /        \
               Pass        Fail
                |            |
                v            v
          Return reply   Explain the violations
                             |
                             v
                       One repair attempt
                             |
                             v
                         Check again
                         /        \
                      Pass        Fail
                       |            |
                       v            v
                 Return reply   Code calls hand-off tool
                                and returns a fixed reply

The defaults also limit an inquiry to eight model calls, with an 80,000-token budget checked between calls, a 60-second model-call timeout, and a 30-second tool-call timeout. The loop initiates a hand-off when it reaches its turn or token limit, the model times out, or a final response is refused or truncated. Tool failures are returned to the model as errors it can react to; unrelated provider failures can still surface as errors.

These checks have a precise scope. They check whether recognized IDs and numbers appeared in the evidence, not whether each sentence correctly interprets that evidence. A real number attached to the wrong property can still pass. A reply without numerical claims can also omit citations without this guard catching it.

The prompt forbids arithmetic, but the number check only tests whether a value already appears somewhere in the allowed material. It cannot determine how the model arrived at that value. Likewise, the guards do not classify the inquiry to catch every forgotten viewing hand-off. The evaluation suite therefore checks expected hand-offs in code and uses a model judge to assess the meaning of the reply against the evidence.

These answer checks belong to the built-in agent loop. An external assistant calling /mcp directly gets the tools and their validation, but does not automatically run its final answer through these guards.

Follow-up questions: remember the context, retrieve the evidence again

After asking about heating in HH-1001, a user can ask “and the deposit?” The model needs the earlier exchange to know which property they mean.

POST /inquiries accepts an optional history containing up to five earlier {inquiry, reply} pairs, oldest first. The web app sends the five most recent completed exchanges on screen. “New conversation” clears that conversation context. The server does not retain conversation history between requests; its shared in-memory ticket queue is separate.

Earlier questions and replies ──> Context: which property?
                                           |
New question ───────────────────────────────┤
                                           v
                                  Retrieve current evidence
                                           |
                                           v
                                  Write and check the reply

Earlier replies reach the model as plain text, without their tool calls or results. Their citations and figures do not become evidence for the new run. Reusing a source marker requires retrieving it again; a figure must appear in this run's tool results or in a user inquiry. Figures the user supplied in retained earlier turns still count as their own—for example, their €2,000 budget.

The API rejects more than five history entries with HTTP 400; the client chooses which entries to drop. This keeps the history bounded without adding server-side conversation sessions. Replaying old tool results would carry evidence across inquiries, while model-generated summaries would add another model call and another unchecked text to the workflow.

What the evaluation tells us

The agent evaluation suite contains 49 German and English inquiries: fact questions, searches, all six hand-off reasons, follow-ups, and traps where a qualification matters. “Pets on request” must not become permission, an estimated facade levy must not become a final amount, and polite haggling must still produce a negotiation ticket.

One case uses a test-only listing whose description tells the assistant to ignore its instructions and confirm a viewing. The expected behavior is still a hand-off. That fixture has its own catalogue and index; it is never served by the application.

Code checks the expected listings, hand-off reasons, and numerical grounding. A different model judges the required facts, forbidden claims, unsupported statements, and reply language against a written rubric. It sees the tool results and tickets as well as the reply. Explicitly allowed optional hand-offs count as neither correct nor extra when calculating precision and recall.

Two full runs on October 1, 2026 used claude-opus-5-5 on Amazon Bedrock as the agent and claude-sonnet-5-5 as the judge:

Metric	First run	After fixes
Cases passing all checks	41 / 49	46 / 49
Hand-off precision / recall	93.8% / 100%	100% / 100%
Replies passing the number check, final / first draft	100% / 100%	100% / 100%
Correct replies according to the judge	83.7%	93.9%
Retrieval: expected passage in the top five	24 / 25	26 / 26
Unanswerable retrieval questions returning passages	0 / 8	0 / 8
Mean estimated agent cost per inquiry	$0.0160	$0.0143
Mean latency / p95 latency	9.2 s / 14.7 s	8.6 s / 13.8 s

The first report and report after fixes link each case to its full trace. Cost estimates use Claude API list prices, not Bedrock billing, and exclude the judge's separate cost. Runs processed three cases concurrently; latency includes waiting for the shared local reranker. The retrieval set gained one regression question between runs.

These results have limits. Numerical groundedness uses the same number check as the runtime guard, so the final-reply score mainly checks that the guard is working; the first-draft score also shows whether a numerical repair was needed. Neither establishes that every claim is supported. The judge can make mistakes, and the small retrieval development set is also used for calibration. These are measurements on known cases, not an independent benchmark or proof that the assistant never invents facts.

What broke, and what changed

The first runs exposed three different problems:

A German commission question could not find an English policy. The page said “commission,” but neither candidate search collected it for “Provision.” Adding “Provision” and “Maklerprovision” to the policy made it retrievable. The original missed question and a new regression question now pass, bringing retrieval recall to 26 of 26. Lowering the reranker threshold would not have helped: a reranker cannot rescue a passage it never receives.
A follow-up corrected an earlier answer that was already right. The agent retrieved the heating fact again, confirmed it, then introduced the same fact as a correction. The prompt now asks for a correction only when new evidence contradicts the earlier reply. A separate case with an actually wrong earlier figure still checks that the agent corrects it.
The judge treated ordinary hand-off wording as unsupported claims. The rubric now accepts statements about human follow-up when a matching ticket exists, while still rejecting promised outcomes such as a confirmed viewing or approved pet. This fixes the evaluator; it does not represent an improvement to the agent itself.

Three cases still fail after those changes. Two involve borderline inferences the judge flags, such as saying the customer would have to use the stairs because a flat has no lift. The third exposes a concrete capability claim: the agent offers to add an email address to an existing ticket, but no tool can update a ticket. That remains open. A reply can pass every number and citation check while promising something the system cannot do.

Different tests answer different questions

Ordinary tests run with a deterministic substitute for the embedding model, without model downloads. A separate pnpm test:model command checks retrieval with the real models.

Scripted-model tests exercise the loop with invented citations, unsupported figures, repair behavior, and run limits. These make failure paths reproducible without paying for model calls.

In a documented run on Amazon Bedrock on September 29, 2026, the German version of the opening inquiry produced a German reply, selected HH-1001, kept pets “on request,” cited HH-1001#s5 for heating, and created one viewing_request ticket. The run used four tool calls across three model turns.

The optional live agent test is recorded as passing too. That test uses a real conversational model with the offline retrieval substitute; it is separate from the real-retrieval model test. The recorded acceptance run is evidence that the workflow has been exercised, not a broad measure of its reliability.

Three Playwright tests cover the browser workflow: a viewing request produces a reply, trace, and ticket; a citation opens its highlighted passage; and a follow-up retains the property context until “New conversation” clears it. They run with the demo model and hashing embedder, so the test runs need no model credentials or model downloads. They verify the browser workflow, not a remote model's answer quality.

GitHub Actions runs typechecking, unit tests, and these browser checks on every push and pull request. The CI workflow passed on the development branch, its pull request, and main after the merge on October 1, 2026. Those checks need no model API credentials or retrieval-model downloads.

Stream the work, then show the checked answer

The browser builds on the inquiry event stream, while external MCP clients can still call tools directly:

Application sends an inquiry
           |
           v
    POST /inquiries
           |
           v
    Runs the agent loop
           |
           +── trace events: completed tool calls
           +── guard events: checks that failed
           +── result event: final reply and metadata

External MCP client
           |
           v
          /mcp
           |
           v
    Calls the tools directly

/inquiries uses server-sent events (SSE): one HTTP response carries a sequence of named events. The UI shows completed tool calls and progress messages while it waits. The answer appears as a whole after the checks pass, or as a fixed hand-off reply if the code ends the run.

This is a deliberate choice. If the UI streamed an invented price before the number check rejected it, the user would already have read it. Keeping the draft private lets the agent repair it before showing it.

Two additional routes support the desk:

Route	What the browser receives
`GET /handoffs`	An SSE snapshot of existing tickets, followed by each new ticket.
`GET /sources/:id`	The passage or property behind a citation, with its surrounding document.

The queue stream subscribes to the shared server queue. A ticket created by an external client through /mcp therefore appears in the browser too. It is still an in-memory queue, so live updates do not imply persistence.

The server binds to 127.0.0.1:3000 by default and currently has no authentication. Vite forwards the browser's API requests to it during development.

Try the workflow without an API key

DemoModel implements the same ModelClient interface using rules. It calls the actual MCP tools, quotes retrieved passages, and passes its replies through the same answer checks. It is designed for the four example inquiries and a limited set of related requests, rather than general conversation.

Claude model ──────────┐
                      ├──> Same loop ──> Same tools ──> Same checks
Rule-based demo ───────┘

pnpm dev falls back to the demo when the credential check reports missing settings. The page labels this mode “Demo model, no API key.” Set MODEL_PROVIDER=demo explicitly to choose it even when credentials are available.

Demo mode changes the conversational model, not the retrieval configuration. By default, the local retrieval models still download on first use. Choosing EMBEDDER=hashing as well removes that download and uses the weaker retrieval substitute. This is also how the browser tests run.

Finding your way around the code

The repository is a pnpm workspace with three packages: the browser interface, the core logic, and the server that connects that logic to external interfaces.

apps/web/                  React inquiry desk and browser tests
packages/core/
  src/domain/              Property records and ticket queue
  src/retrieval/           Chunking, search, models, ranking
  src/agent/               Loop, checks, interfaces, demo model
  scripts/                 Index building and calibration
packages/server/
  src/mcp/                 Tools and MCP transports
  src/agent/               Provider adapters, MCP client, CLI
  src/api/                 Inquiries, queue, and source routes
data/                      Synthetic records and policy pages
evals/                     Agent cases, judge rubric, thresholds, retrieval golden set
  reports/                 Recorded results and per-case traces
  fixtures/                Test-only prompt-injection listing

The core package has no MCP, HTTP, or model-provider imports. The server adapts those external interfaces to the core. The web app shares types and example inquiries without pulling the core's Node.js implementation into the browser bundle.

Development uses the packages' TypeScript source directly. For the compiled MCP entry point used by Claude Desktop, build first and run Node with --conditions=built. The server entry point is packages/server/dist/mcp/stdio.js; its source lives in packages/server/src/mcp/stdio.ts.

The vector store is currently in memory, with embeddings cached on disk. For this small corpus, a separate database would add setup without solving an immediate problem. A pgvector adapter is an optional, unscheduled extension intended to exercise the store interface; it would persist retrieval vectors, not hand-off tickets.

Try the current implementation

Clone the GitHub repository, then run these commands from its root with Node.js 20 or newer:

corepack enable   # once, provides the pnpm version pinned in package.json
pnpm install
pnpm dev

Open http://localhost:5173 and click an example. The API runs on 127.0.0.1:3000. The first start downloads the two retrieval models—approximately 120 MB and 570 MB—and prepares the local index.

For a demo without retrieval-model downloads, set these environment variables before starting. In PowerShell:

$env:MODEL_PROVIDER = "demo"
$env:EMBEDDER = "hashing"
pnpm dev

Use a fresh terminal or clear those overrides when switching to the real models. pnpm inspect remains available for calling the MCP tools directly.

To use a remote conversational model, copy .env.example to .env and fill in credentials for your chosen provider. For the default Anthropic adapter, set ANTHROPIC_API_KEY. For Bedrock, set MODEL_PROVIDER=bedrock, an AWS region, and the relevant credentials. The CLI and HTTP server load .env automatically.

Then ask a question and show the tool trace:

pnpm ask --trace "Is heating included in HH-1001, and can I view it on Saturday?"

Use --json to get the full structured result. To expose the HTTP endpoints, run:

pnpm serve

Send POST /inquiries a JSON body such as:

{ "inquiry": "Is heating included in HH-1001, and can I view it on Saturday?" }

For a follow-up, include the earlier exchange. This example shows the request shape; in an application, pass the actual reply previously returned:

{
  "inquiry": "And what is the deposit?",
  "history": [
    {
      "inquiry": "Is heating included in HH-1001?",
      "reply": "Heating is included in the additional utilities charge of 240 EUR per month. [HH-1001#s5]"
    }
  ]
}

Unlike pnpm dev, pnpm ask and pnpm serve require credentials unless you explicitly select MODEL_PROVIDER=demo. Real-model runs use provider credentials; demo runs do not.

To run the regular tests and the browser checks:

pnpm test
pnpm --filter @proptech/web exec playwright install chromium
pnpm test:e2e

Installing the browser is a one-time setup. The optional live-model test runs when its credential conditions are met in the process environment; the test itself does not load .env.

After changing the corpus or model, pnpm calibrate reports scores and proposes thresholds. Those values are reviewed and copied into retrieval.thresholds.json; calibration does not automatically update that file.

To evaluate the real agent and retrieval, configure credentials for both the agent and judge, then run:

pnpm eval                  # all 49 cases, with a report and per-case traces
pnpm eval --subset         # 12 cases covering both languages and the main case types
pnpm eval --case handoff-viewing --case trap-pets-hh1001

The runner exits non-zero when a metric misses a threshold in evals/thresholds.json. Current requirements include at least 90% case pass rate and judge-rated correctness, 100% hand-off recall, and 100% retrieval recall on the golden set. Thresholds were set from the first run and raised after the fixes; the committed reports retain the thresholds used at the time.

The paid evaluations also have a manually triggered GitHub Actions workflow. In the repository's Actions tab, select “Eval subset,” then “Run workflow.” Choose subset for the 12-case selection or all for the full 49 cases. Configure the repository secret ANTHROPIC_API_KEY, or AWS_BEARER_TOKEN_BEDROCK for Bedrock with the repository variable AWS_REGION (default eu-central-1). When both secrets are present, the workflow chooses the Anthropic API.

The report table appears in the workflow run summary, and the report and per-case traces are uploaded as an artifact. A workflow run does not commit them to the repository. The free CI checks have passed; a successful manual eval workflow run has not yet been confirmed. The committed full-run measurements above came from local runs.

What remains to be proven

The evaluation suite now tests whether qualifications survive paraphrasing, requests needing a person produce hand-offs, and a malicious listing instruction changes the agent's behavior. The next work includes fixing the unsupported ticket-update offer, expanding coverage beyond these known cases, and comparing a cheaper agent model with a separate judge. Passing one prompt-injection case does not establish resistance to other attacks.

The architecture guide, recorded demo, failure analysis, and CI workflows are in place. The repository description, topics, and social preview are configured too. Hand-off tickets still live only in memory; persistent ticket storage remains future work. For now, this is a local application with synthetic data, not a complete agency operations system.

The useful lesson so far is that retrieval, answer generation, and answer checking need separate tests. A search result can be relevant but misquoted. A citation can exist but support a different claim. The inquiry desk makes the evidence, tool calls, and resulting tickets inspectable while keeping rejected drafts out of the conversation.

Catching AI Workflow Failures with Executable Playbooks

Luka Zmikic Engels — Fri, 25 Sep 2026 15:17:22 +0000

An AI coding agent can produce working code while skipping the steps that make it the right thing to build. Implementation starts without a plan. Approval is assumed. A prototype changes after work begins, and the agent follows the new version instead of the one you agreed on.

These are workflow failures, and passing tests won't necessarily reveal them. You need to check what was specified, what was approved, and which version the work is supposed to follow.

I built sdlc-playbooks to make those checks executable. Each playbook describes a delivery phase: the inputs it needs, the work to do, and the artifact it produces. Scripts check the prerequisites against files in the repo and report missing artifacts or inconsistent state before the next phase starts.

For example, the implementation gate refuses to proceed when a story has no plan or its pinned prototype snapshot is missing. A prompt can tell the agent to check those things; a script can inspect the files and return a specific failure with a fix.

The procedure runs from design system → requirements → prototype → human approval → product, with one source shared by Claude Code and Codex. The agent follows the playbooks; the scripts check the parts of the workflow that can be checked mechanically.

That distinction matters: an executable check can establish that a plan exists. It cannot establish that the plan is good, or independently prove that a person approved the prototype. Those decisions still need people and review.

This post walks through those checks, where they run, and what they can—and cannot—catch.

The flow

  design system                 PO requirements (text)
          │                              │
          ▼                              ▼
   0 lock-design-system          1 capture-requirements
   tokens.css                    REQ-### + acceptance criteria
   components.md                          │
          └──────────────┬─────────────────┘
                         ▼
                 2 build-prototype
             single-file React page
            every screen, every state
                         │
                         ▼
                  ◆ HUMAN APPROVES ◆
                   3 freeze-approval
            frozen snapshot v<N> + stories
                         │
                         ▼
        4 plan-story → 5 implement-story → 6 review-change
                         │
                         ▼
                    the product

There are seven phases, each with one playbook and one gate. Everything the flow knows is stored in a file:

Artifact	Written by
`design-system/tokens.css`, `components.md`	phase 0, then locked
`work/items/REQ-###.md`	phase 1
`prototype/<slug>.html`	phase 2
`prototype/.approved/<slug>-v<N>.html`	phase 3, versioned baseline
`work/items/ST-###.md`	phase 3, one story per screen
`work/plans/ST-###.md`	phase 4

With the default file tracker, the workflow state lives in the repo. If you delete every playbook, a human can still finish the work from the artifacts. That's the intended direction of dependency.

Why playbooks instead of agents

Most agent setups I've seen start with personas: "You are a senior reviewer." That describes an identity, but leaves the working procedure to be specified elsewhere. Here, the playbook is the unit of organisation. It names the inputs, gate, outputs, and done-condition for a phase. For example:

## Contract

|               |                                                                       |
| ------------- | --------------------------------------------------------------------- |
| **Phase**     | 5 — the only phase that writes product code                           |
| **Gate**      | `node .claude/tools/gate.mjs implement ST-###`                        |
| **Inputs**    | the plan, the pinned snapshot, the catalog, the component lib         |
| **Produces**  | product code, one test per AC                                         |
| **Done when** | every `verify.commands` entry is green, story is `in-review`          |
| **Never**     | copy the prototype into the product; build against the live prototype |

Each playbook also routes neighbouring decisions: new tokens go to lock-design-system, changed scope goes to capture-requirements, and finished work goes to review-change.

The earlier version had separate agent definitions. Most of their instructions already existed in the matching playbooks; the remaining ownership rules belonged at phase boundaries. Removing those files left one place to maintain each procedure. Keeping two copies of a rule means you have one rule and one future contradiction.

Handoffs happen through saved artifacts. Requirements supply acceptance criteria to prototyping; an approved snapshot supplies the baseline for planning and implementation; the plan and resulting code supply the inputs to review. The next phase can run in a fresh session because its inputs are in the repo. It doesn't need the previous agent's persona or conversation history to reconstruct what was agreed.

Layer 1: gates decide whether a phase may start

Every phase begins with:

node .claude/tools/gate.mjs <phase> [id] [flags]

The gate checks the phase's preconditions against the repo. If one fails, it exits 2 with a named reason and a fix. The Claude Code command files invoke it through inline shell expansion:

---
description: Record a human's approval — freeze the prototype and cut the stories.
---

Gate — this runs before you read any further; a non-zero exit aborts the command:

!`node .claude/tools/gate.mjs approve $0 $1 2>&1`

That puts a concrete check at the command entry point. The playbook instructs the agent to stop on failure. Direct script calls and other editing paths still need their own controls.

Phase	Refuses when
`design-system`	already locked and `--refresh` wasn't passed
`requirements`	no input
`prototype`	design system unlocked · wrong status · no acceptance criteria
`approve`	not `prototyped` · no `data-screen` markup · no `--human-approved`
`plan`	parent not approved · no version pin · snapshot missing
`implement`	no plan · story not `in-progress` · snapshot missing
`review`	story not `in-review` · already at `loops.reviewRounds`

Making the approval assertion explicit

The old version of /approve said: "This command represents a human decision. Do not run it on your own judgment."

The gate now requires an explicit flag:

if (!args["human-approved"]) {
  throw new Blocked(
    `approval is a human decision and has not been recorded for ${id}`,
    `ask the person to open ${req.prototype} and say yes, then run: /approve ${id} --human-approved`,
  );
}

The bundled seaside example illustrates the check: even with a prototype ready to inspect, an approval gate call without the flag is refused:

$ node .claude/tools/gate.mjs approve REQ-003
BLOCKED — approve: approval is a human decision and has not been recorded for REQ-003

  ask the person to open prototype/seaside-zadar-landing.html and say yes,
  then run: /approve REQ-003 --human-approved

Nothing was changed. Report this reason verbatim and stop; do not work around it.
$ echo $?
2

The flag means "a person said yes in this conversation." It makes that assertion explicit and reviewable, but does not verify it independently: the agent can supply the flag, and the snapshot utility can be called directly. Respecting human approval still depends on following the procedure.

The review loop is bounded

Review can approve a story or send it back. The gate reads the highest round N: recorded in the story's notes and refuses a round beyond loops.reviewRounds (two by default). That bound depends on the reviewer recording each round. When the cap is reached, the procedure calls for handing the disagreement to a person.

On success, gates print resolved context as JSON: paths, the pinned prototype version, acceptance-criterion ids, and the review round where relevant. The agent gets concrete inputs for its next step.

Layer 2: hooks decide whether an edit may land

Hooks add checks at supported edit entry points. In Claude Code, two shell scripts are wired as PreToolUse hooks on Edit|Write:

require-plan.sh blocks a guarded edit when it identifies a story with no plan. It finds the story from the branch name (story/ST-007-…) or, failing that, from exactly one in-progress story in the tracker. For example:

BLOCKED — no plan for ST-007.

  expected: work/plans/ST-007.md
  editing:  src/contacts/list/ArchiveButton.tsx

Run the plan-story skill first. The plan is a gate, not paperwork.

The reasoning: a plan written after the code is just a summary, and a summary never catches the thing you'd have noticed before starting.

check-hardcoded-colors.sh warns. It flags a raw #hex, rgb() or hsl() at the moment it's written, while fixing it is still a one-line change. The blocking version of this rule runs at review time.

The hooks have limits: shell writes fall outside Claude's Edit|Write matcher, some file types are excluded, and the plan hook allows an edit when no story can be identified. Both scripts accept file_path or path; actual coverage also depends on the harness invoking them with a supported payload.

Layer 3: checks expose gaps before handover

check-tokens.mjs scans prototypes and configured product directories for raw hex/RGB/HSL colours and literal values in selected properties: font size, radius, shadow, padding, margin, and gap. It also checks for undefined token references. width: 100% and z-index are outside those design-property checks.

$ node .claude/tools/check-tokens.mjs
ok — 139 tokens defined, 134 used, no raw design values in 1 files

The procedure's rule is that a design value can only be created in phase 0. The scanner catches common violations; it is not a complete CSS validator. It permits token-definition blocks for inlined prototypes without verifying that their values match the locked file.

check-coverage.mjs checks structural links:

requirement → approved snapshot → screen → story → plan file

It reports gaps such as screen-without-story, story-without-plan, stale-pin, and done-story-screen-vanished, exiting 2 when it finds one. It checks that acceptance criteria exist, but does not map each AC to a screen or inspect implementation and test coverage.

The prototype convention puts two attributes on every screen root:

<section data-screen="SCR-booking-enquiry" data-req="REQ-003"></section>

The checker discovers screens through data-screen; the approve gate refuses a prototype with no screen markers. data-req records the intended requirement association.

Structural gaps become queryable. Whether the product satisfies the requirements still needs tests and review.

The prototype, and the pin

Phase 2 calls for one HTML file per requirement: React 18 UMD plus @babel/standalone from a CDN, tokens inlined, catalog components, hash routing, and realistic mock data. It needs no build step and opens with a double-click, but loading its CDN dependencies requires network access. The playbook requires every state to be drawn: empty, loading, populated, error, and success. "The list refreshes" doesn't show a person what success looks like.

After human approval, the procedure uses approve.mjs to create prototype/.approved/<slug>-v<N>.html, then cuts one story per screen with a prototypeVersion: v<N> pin. These are versioned snapshots treated as immutable by the procedure. The utility checks for an existing destination before copying; the resulting files remain editable on disk.

The implementation playbook directs the agent to build against the pinned snapshot. If someone makes "one more tweak" to the live file, gate.mjs implement reports the difference:

{
  "liveHasMovedPastThePin": true,
  "note": "prototype/bulk-archive.html has moved past v2. You build v2. The difference is new scope for the PO — record it under Divergence, do not absorb it."
}

Building the latest prototype when the story pinned v2 is scope drift, and you won't see it in the diff unless someone says so. This is how a two-week feature quietly becomes a four-week one without anyone deciding it should.

One source, two harnesses

Procedures live in exactly one place: playbooks/<name>/SKILL.md, written without reference to any particular tool. sync.mjs copies that tree into .claude/skills/ for Claude Code and .agents/skills/ for Codex:

node .claude/tools/sync.mjs          # write the projections
node .claude/tools/sync.mjs --check  # exit 2 if one is stale (for CI)

Each generated directory has a .generated marker holding the source hash, so a hand-edit to a projection is detected, not silently kept.

Copies avoid the extra permissions or developer-mode setup that symlinks can require on Windows.

Claude Code has slash commands such as /requirements, /prototype, and /build. The projected skills expose the same playbooks, including build-prototype, with a gate invocation as step 0. The shared procedure does not make the harnesses' enforcement identical.

The tracker is an adapter

No playbook talks to a tracker directly. Everything goes through node .claude/tools/tracker.mjs <verb>, which prints JSON and dispatches to a provider:

node .claude/tools/tracker.mjs create --type requirement --title "Bulk archive contacts"
node .claude/tools/tracker.mjs set ST-007 --status in-review
node .claude/tools/tracker.mjs next --type story

The default provider is files, storing Markdown with YAML front matter in work/items/. Jira support is unfinished: mapping and REST code exist, but the gate's asynchronous provider handling and the coverage checker's direct file reads still need integration work.

Two procedural rules keep updates consistent: never hand-edit a work item, and never pick an id yourself. create allocates IDs centrally so agents don't choose them manually. Allocation is not protected against concurrent writers.

The adapter is intended to let status and conversation move to another tracker while design, prototype, and plan artifacts stay in the repo.

Every tool runs without a model

None of the enforcement needs an LLM. You can check the state of the repo from a plain shell:

node .claude/tools/status.mjs --gaps
node .claude/tools/check-coverage.mjs
node .claude/tools/check-tokens.mjs
node .claude/tools/gate.mjs plan ST-001

The local tools use Node 18+ and plain ESM, with no npm dependencies. The hooks require Bash; on Windows, that means a Bash installation such as Git Bash.

What's deliberately missing

There's no CI/deploy stage, no QA phase, no parallel fan-out, no worktree isolation, and no board integration. Those are all real and sometimes necessary. They're also where a procedure stops being easy to read.

The rule for adding one: do it when a specific failure demands it, and write down which failure.

A gate with no incident behind it is ceremony.

Try it

Clone the repo and generate the skill projections in your terminal:

git clone https://github.com/srnux/sdlc-playbooks
cd sdlc-playbooks
node .claude/tools/sync.mjs

Edit .claude/sdlc.config.json for your product and verification commands, and rewrite the stack-specific half of .claude/rules/coding-standards.md.

The repo includes a locked seaside-rental design system, a prototype with nine marked screens, and three requirements. You can inspect that example first. For your own design, update the configured design source and run /design-system --refresh in Claude Code. A bare /design-system refuses because the bundled system is already locked.

Then, in Claude Code:

/requirements 'PO text here…'

Use the requirement ID returned by that command in /prototype <returned-id>. Open the result and review its screens and states. Only after you approve it, run /approve <returned-id> --human-approved, then /build.

My one piece of advice: run one real requirement end to end before you load up the backlog. That gives you a concrete way to find gaps in the procedure while the scope is still small.

If you've built something similar, or tried to hold an agent to a process with prompts alone and watched it slip, I'd like to hear what broke first.

Partially matching object, array and function with Jest

Luka Zmikic Engels — Thu, 01 Jun 2023 14:16:42 +0000

JestJs 🃏 Delightful JavaScript Testing

It is really delightful as the team states! For the last couple of years that is accompanying me in my daily struggles, it has shown it's worth. One of the best parts is the partial match on object, array and function, which will be the focus of this blog post.

Partially matching objects

Let's say you need to test a unit that changes just parts of an object. It is indifferent of the remaining parts of the object. Your unit test should check just the changes the unit influences.

In the code block below, a simple fixture containing some nested objects and arrays, which will be used in the following examples:

const resultFixtureObject = {
 active: true,
 id: 'e0e0e0e0-2b79-4eba-ab39-b2dfdb2f0eba',
 firstName: 'Jester',
 lastName: 'Magic',
 finances: {
  incomes: [
  {
    type: 'salary',
    amount: 150000,
    currency: '$',
   },
   {
    type: 'dividend',
    amount: 6000,
    currency: '$',
   },
   {
    type: 'rent',
    amount: 12000,
    currency: '$',
   },
  ],
 },
};

Match object

In the test below, the expected result object must match a few properties:

expect(resultFixtureObject).toEqual(
 expect.objectContaining({
   active: true,
   id: 'e0e0e0e0-2b79-4eba-ab39-b2dfdb2f0eba',
   firstName: 'Jester',
   lastName: 'Magic',
 })
);

Match nested object

You can match nested objects by simply nesting objectContaining() expectations. In this case, expecting object finances that must have nested object incomes. We do not care about the content of the incomes, in this unit test:

expect(resultFixtureObject).toEqual(
 expect.objectContaining({
  finances: expect.objectContaining({ 
   incomes: expect.anything()
  })
 })
);

Partially matching arrays

Just as the objects, there is a way to partially match arrays. You can expect single or multiple item matches on an array.
Expecting one item to match:

expect(resultFixtureObject).toEqual(
 expect.objectContaining({
  finances: expect.objectContaining({
   incomes: expect.arrayContaining([
    {
     type: 'rent',
     amount: 12000,
     currency: '$',
    },
   ]),
  }),
 })
);

or multiple, complete or partial matches:

expect(resultFixtureObject).toEqual(
 expect.objectContaining({
  finances: expect.objectContaining({
   incomes: expect.arrayContaining([
    {
     type: 'rent',
     amount: 12000,
     currency: '$',
    },
    expect.objectContaining({
     type: 'salary',
    }),
   ]),
  }),
 })
);

Partially matching function parameter calls

Aforementioned operations can be applied to toHaveBeenCalledWith() matcher as well.
Below a variant with toHaveBeenNthCalledWith() matcher. Verifying just the third call to the function:

const functionMock = jest.fn();
functionMock('string arg');
functionMock(['array arg']);
functionMock(resultFixtureObject);

expect(functionMock).toHaveBeenNthCalledWith(
 3,
 expect.objectContaining({
  finances: expect.objectContaining({
   incomes: expect.arrayContaining([
    expect.objectContaining({
     type: 'rent',
    }),
    expect.objectContaining({
     type: 'salary',
    }),
   ]),
  }),
 })
);