DEV Community: Joe Buckle

Hardening an AI coding agent: the failures, and the code that fixed them

Joe Buckle — Fri, 31 Jul 2026 11:33:04 +0000

At Univoco we build retrieval-augmented assistants over a customer's own documentation. One of them is a coding agent that writes code for a proprietary document layout engine.

The useful thing about that engine is that there is effectively nothing about it on the public internet. The model cannot have learned it. Everything the agent knows comes from a private documentation index and the tools we give it to search that index.

That makes it an unusually clean laboratory. When memorisation is off the table, you get to watch the problem-solving process itself, and watch it fail.

It failed a lot, and the failures were rarely the ones I expected. What follows is the list, in the order we hit them, with the code that fixed each one.

#	Failure	Fix
1	Search tools are literal	filename matching, construct footprints
2	It searches the same thing forever	five layers of loop detection
3	One phrasing is a single point of failure	query inflation
4	Goal words vs. index words	a deterministic phrasebook
5	The retrieval tool invented APIs	self-auditing lookups
6	Deciding an API is invented	the checker stack
7	Findings get ignored	finish gates
8	The wording of a correction is load-bearing	the envelope and the silence trailer
9	A deletion that grew the file	take the generation out

1. Search tools are literal

The standard shape is fetch-then-answer: embed the question, pull the top-k chunks, hand them to the model. It works for a great many tasks, and it has one structural problem. The folder gets assembled before anyone has understood the problem, so you are deciding what might be useful before you know what is needed.

Giving the agent search tools of its own fixes the ordering and immediately produces a different class of bug, because search tools are literal.

Content search cannot see filenames

The agent searched every file in the project for json, found nothing, and offered to create a sample data file. sampleData.json was sitting in the project. The word it needed was on the front of the folder, not inside it.

for entry in project_paths.iter_project_files(ctx.project_dir, glob=glob):
    # A model hunting "the json"/"the xml" searches by TYPE, and content search can
    # never find a file whose content doesn't mention its own format (live failure:
    # sampleData.json was invisible to pattern "json" and the agent offered to create
    # a file that existed). Names are part of the searchable surface.
    if rx.search(entry["path"]):
        name_hits.append(f"{entry['path']} (filename match)")

A grep hit is a line, but the thing you edit is a construct

Asked to remove a feature, the agent searched, found the one matching line, deleted it, and declared success. The five lines above it declared and configured the same object.

So find_in_files now returns grep-style context plus an automatic variable footprint. A hit on layout.blockStart(box) also lists var box = new Block() and every property line of that variable. Only declared variables expand, so globals like layout never blow the output up.

The rule it encodes: the construct is all of its lines, not the one that matched.

Neither fix is clever. Both exist because something shipped.

Two layers of tools

There are two layers of tools, and the split explains several of the fixes that follow.

The base tools ship with the platform and are deliberately generic. list_files, read_file, find_in_files, write_file, edit_file, apply_edit, delete_lines: every customer gets them, and not one of them knows anything about anybody's domain.

The client tools are authored per customer and live in that customer's own directory. Here that means lookup_docs, resolve_topic, find_construct, check_symbols and dry_run, none of which mean anything without this engine's documentation and this engine's object model.

The mechanic that matters is that a client tool shadows a base tool of the same name. Two do. The client find_in_files adds something the generic one could not: when a literal search finds nothing, concept words like "block" or "the box" expand to the engine's real symbols, so the agent never has to translate a concept into code before it can search for it. And rag_lookup, the knowledge lookup itself, is shadowed too, which is what makes the fix in section 5 possible.

That split is the architecture. The generic layer is table stakes and everyone has it. The layer that knows what a "block" is called in this customer's world is the part that cannot be bought.

The two knowledge tools

These two need separating, because the gap between them drives a lot of what follows.

Tool	What it does	What it costs
`lookup_docs`	full-text search over the indexed documentation, returning the chunks that literally contain the term. No embeddings, no model, no generation.	milliseconds, and it cannot invent anything
`rag_lookup`	semantic retrieval, then a model composes an answer out of whatever came back	ten to thirty seconds, and the composing step can invent

The second one is what most people mean by RAG: retrieve some relevant text, hand it to a model, let the model write the answer. It is the right tool for "how do I build a table", and the wrong tool for "does tableColumnRule exist", which is the distinction the agent is told to route on:

TWO knowledge tools, and the choice matters. lookup_docs is ~1000x cheaper. Naming a symbol you already know is a FACT CHECK, not a research question.

That framing looks like a cost optimisation. It turns out to be a correctness one, for reasons section 5 gets to.

2. It searches the same thing forever

My favourite failure of the whole project:

The agent ran the same knowledge lookup twelve times in one turn, identical arguments each time, and the results themselves contained notes pointing out it had already asked. It made no difference.

The variant version was worse, and no exact-match check can catch it, because the arguments differ every time. One table turn ran 12 steps and 170k input tokens, and three of its five knowledge lookups re-confirmed the same three API names. Those names were already in its context, listed in the prompt fragment it had been handed at the start of the turn.

It was not searching. It was checking something it already knew, at ten to thirty seconds a time. The operator's note in the ledger puts it better than I can:

once it knows how to do stuff it shouldn't need to keep trying to find it at the rag end

A human has an internal sense of going in circles. The model does not, and telling it so in the tool output does nothing, because that text is just more context. Worse, on a provider without prompt caching the fixed ~11k of system prompt and tool schemas is re-sent every step, so cost is roughly fixed times steps plus transcript. Fewer steps is the only real lever, and a redundant lookup adds steps.

The fix is five layers, and the layering is the point: each layer only catches what the one before it structurally cannot see.

Layer 1: identical calls

A counter in the tool loop, tripping at _MAX_IDENTICAL_CALLS = 4. Past that the call is not executed, but it still has to be answered, because every tool call in the transcript needs a result.

# Stuck-loop detector: the same call over and over is fixation, not progress.
# Answer it with a stub (every tool_call must be answered) and head for the
# forced final answer instead of burning the rest of the budget.
key = (name, call.arguments or "")
call_counts[key] = call_counts.get(key, 0) + 1
if call_counts[key] > _MAX_IDENTICAL_CALLS:
    stuck = True
    messages.append({"role": "tool", "tool_call_id": call.id,
                     "content": f"[not run - you have made this exact call "
                                f"{call_counts[key]} times this turn]"})

Layer 2: rewordings within a turn

Reworded questions are distinct argument strings, so the identical-call counter never fires on them. Catching those needs a budget rather than a counter:

Constant	Value	Meaning
`_MAX_LOOKUPS_PER_TURN`	`12`	live lookups across all wordings
`_MAX_LIVE_LOOKUPS`	`2`	per identical query, then the turn cache answers

Layer 3: rewordings across turns and sessions

This one I like, because the obvious solution is wrong. You could embed every query and compare vectors. We do not, because that is a per-probe cost on the hot path for a problem that plain string handling solves.

Matching is deliberately embedding-free: normalise (lowercase, drop stopwords, light stemming with a double-consonant collapse so splitting to split), token-sort, then exact key hit or a difflib close match. Deterministic, stdlib, zero per-probe cost, and it catches exactly the reworded-not-rethought queries above. Genuinely different questions stay distinct.

The similarity threshold is 0.8, and the calibration comment is the reason I trust it:

the live pair whose only difference was the domain word scores 0.833
two genuinely different questions, create table vs create box, score about 0.73

The window between them is the sweet spot, and it was measured rather than guessed.

Layer 4: concurrent duplicates

Read-only calls get dispatched together, so three rewordings can all be in flight before any of them returns, and a dedupe that runs on completion never sees them. That needs a leader/follower collapse on the normalised query: the first runs, the rest wait and receive its answer.

Layer 5: do not ask for what you were already given

None of the above touches the failure this section opened with. A budget caps how much re-asking costs, but the agent re-confirming names it was handed at turn start should not be asking at all.

Writes to domain code are normally refused unless a knowledge lookup has run in that turn, which is what forces grounding. That gate was also what forced the redundant lookups: the agent had the names, but the rule did not know that.

So the prompt fragments now declare what they supply. Each construct fragment carries a provides list, generated rather than hand-written and scoped to one family, so the table fragment declares the table companions and not the rule ones. The task brief unions those into provided_symbols, and the gate exempts a write whose new X() constructors are all covered. A write reaching for a construct that was never delivered still grounds, and check_symbols remains the backstop before anything lands.

The general shape: a rule that forces good behaviour will also force pointless behaviour unless it can see what has already been satisfied.

The counter-lesson

This was the most useful thing in the section. The budget once cut a model off mid-way through correctly verifying eleven symbols it was about to use. It was doing exactly the right thing and we throttled it.

So budget is now charged per tool rather than per call:

_MAX_LOOKUPS_PER_TURN caps LIVE lookups across all wordings; cached serves and cheap tools are free (projects.budget_tools, default rag_lookup): a millisecond lookup_docs fact-check per symbol is diligence we demand, not cost to ration.

If your rate limit punishes the behaviour you are trying to encourage, the limit is wrong, not the behaviour.

One last piece of absurdity, included because it is a genuine class of bug and not just a funny story. An earlier nudge injected into the tool results said "look up the interface's REAL members". The model ran that sentence, verbatim, as a semantic search query. Twelve times.

Nudges now name an action, and only ever suggest exact-name lookups, never a phrase, because anything phrase-shaped in an agent's context is a candidate query.

3. Query inflation

One user phrasing is a single point of failure for recall. If the question happens to be worded unlike the corpus, retrieval returns nothing useful and the agent concludes the knowledge base is empty rather than that it asked badly.

So before the lookups run, a forced tool call reformulates the question into several deliberately different queries. Not paraphrases: different angles.

_QUERIES_TOOL = {
    "name": "search_queries",
    "description": (
        "Reformulate the question into optimised knowledge-base search queries using the right "
        "domain keywords / API terms. For a how-to / implementation / 'how do I ...' question, "
        "return 2-3 queries covering different angles (e.g. the core API/object, and a usage "
        "example). Return a single query ONLY for a trivial factual lookup (e.g. 'what is X')."
    ),
    ...
}

The user message that drives it is where the actual behaviour lives. Almost every clause is there to stop a specific degenerate output:

Generate {max_queries} search queries covering DIFFERENT angles so retrieval is thorough:
1. A focused how-to query for the specific task.
2. A BROAD keyword query - just the core nouns / concepts / terms (e.g.
   'user login session token'), to surface related material the specific query might miss.
3. If more are allowed, add a query aimed at a concrete example.
Do NOT return only a verbatim copy of the question. Only a trivial factual lookup gets a
single query.

Examples (apply the SAME pattern to the question's own domain):
- 'set up user authentication' -> ['how to set up user authentication',
  'authentication login session token configuration']
- 'export a report to CSV' -> ['how to export a report to CSV',
  'CSV export columns rows formatting']

The system line is one sentence and does most of the work:

You generate DISTINCT knowledge-base search queries using precise keywords / API terms. Never echo the question verbatim.

Without it you get the question back with the word order shuffled, which is the reworded-not-rethought problem from section 2, except now we are paying to generate it.

Three details that are not obvious

Each query carries a human label. The tool schema requires an about field alongside the query: "a short, natural human phrase for what this looks up, e.g. 'creating tables with borders'". That is not for retrieval, it is for the activity line the user watches. Generating the explanation in the same call as the query is free; deriving it afterwards is another model call or a bad heuristic.

The queries are not fanned out by code. They are injected back into the prompt as an instruction, so the agent still owns sequencing and can stop early.

def _guided_prompt(question, query_objs, adaptive=False):
    """Wrap a question with guidance to search each expanded query, then answer (or dig deeper)."""
    qlist = "; ".join(o["query"] for o in query_objs)
    tail = _ADAPTIVE_TAIL if adaptive else " Then answer fully."
    return f"{question}\n\n[Search the knowledge base for: {qlist}.{tail}]"

Expansion without a brake is just a licence to over-search, which is section 2 all over again. So _ADAPTIVE_TAIL ships with it:

Then judge the results: if they already answer the question, STOP and answer now, do not run extra, redundant, or tangential searches. Only if a genuine gap remains (or the results reference something specific you still need) search again for exactly that. Aim for the fewest searches that fully cover the question.

There is a ceiling of 6 regardless of configuration, and max_queries: -1 means the model decides within that ceiling.

One clarification, since the two get conflated: this is multi-query rewriting, not HyDE. No hypothetical answer document is generated. In a domain the model has never seen, a hypothetical document is a hallucination with extra steps, and embedding it just retrieves whatever is nearest to the invention.

4. The agent's words are not the index's words

This is the failure that cost the most and looked the least like a bug.

The agent phrases things by goal: "display the data in a table", "put the XML in a table". The corpus is indexed by the engine's own vocabulary: openStream, xmlDoc, selectNodes.

To a human expert those are obviously the same topic. To a retriever they have almost nothing in common, so the lookups returned nothing, the agent concluded the capability did not exist, and it reached for browser JavaScript instead.

Missing real content and inventing names is the expensive combination, because the invention then looks like a hallucination and you go and debug the wrong component.

Three layers, in increasing order of force.

A deterministic phrasebook

resolve_topic maps a goal to the real symbols, with no model involved at all:

The agent phrases requests by goal; the docs are indexed by object-model vocabulary. That mismatch made it MISS real content and INVENT names, spiralling. This reads the terminology datagraph (_topics.json) and hands back the real symbols, a note on the pattern, and the exact next lookup, so the agent searches truth instead of guessing.

Zero LLM, deterministic. Call it BEFORE rag_lookup when you don't already know the API names.

The map is mined rather than hand-written:

human phrasings come from query-pattern tags on the worked examples
symbols come from a regex over the example code, validated against the authoritative API dictionary
a curated overlay adds goal aliases for the high-value topics

Matching is stopword-filtered token overlap ranked by hit count. No embeddings anywhere in it.

A confidence ladder

Because a phrasebook that is confidently wrong is worse than none. The first version applied full force to any match at all:

# Confidence ladder (live miss: 'footer with page numbers' shared ONE word with the
# layers topic and got the full-strength 'do NOT invent' route, walling the agent off
# from the real constructs). A weak match is a HINT, not a law: strict only when the
# route explains at least half the goal's significant words, or 2+ of them.
coverage = top / max(1, len(_words(goal)))
if top < 2 and coverage < 0.5:
    return ("WEAK MATCH - the goal only loosely fits a mapped topic; treat this as a "
            "hint, NOT the answer:\n\n" ...)

A guardrail with a false positive does not fail neutrally. It actively steers the agent away from the answer, and it does it with authority.

A de-nouning rewrite

Users phrase goals in their own nouns ("display parts from an XML file grouped by category") and documentation indexes mechanisms ("load an XML file, iterate elements into table rows, heading per group"). One cheap call normalises the retrieval query before the lookup stack sees it, layered outermost so that dedupe and the persistent cache both key on the normalised form and hit more often.

The constraint on that rewriter is the interesting part:

#: Operations only, never API names: the rewriter half-knows jargon, and an invented
#: symbol in the query would poison retrieval with exactly the hallucination the
#: lookup exists to prevent. API discovery stays retrieval's job.

It ships disabled, incidentally. The doctrine version in the prompt does most of the work and the mechanical version is held in reserve for when doctrine leaks.

Where the phrasebook is applied matters more than the phrasebook. The terminology hint is prepended to every lookup result, whether or not the model asked for it, and whether or not retrieval returned anything, because a gap is exactly when the real names matter most.

5. The tool was the liar

The agent kept writing invented API calls into its work. It would be caught, it would look the correct ones up, and then it would do it again. Seven rounds of this in one session.

I assumed the model was inventing them. That is the famous failure mode and I never questioned it.

Then I read what the lookup tool had actually returned, rather than the model's account of what it returned.

The retrieval pipeline was generating the invented APIs. The style that answers conceptual questions composes an answer from retrieved examples, and that generation step had invented layout.tableCellEnd and Table.cellSpacing, neither of which exists, while asserting they came "directly from the example in the provided context".

That is not really a bug, it is what generation does. Ask an obliging model for a worked example and it will produce a worked example, and where the retrieved material runs thin it fills the gap with the most plausible name available. The word "lookup" was what fooled me: I had been treating an endpoint that ends in a generate step as though it returned facts, when the only thing separating it from the agent was that its inventions arrived with a citation attached.

The model was faithfully copying bad information from the one source it had every reason to trust. Worse, the loop was closed: the check flagged the file, the agent re-ran the same lookup to find the right answer, and got the same poison back.

The debugging rule that came out of it, because I lost days to not knowing it: read the actual tool results in the log, not the model's narration of them. A large proportion of what looked like model failures turned out to be a tool feeding it poison.

The fix is a client-side shadow that wraps the built-in lookup and audits its output with the same symbol checker the write path uses:

The examples style GENERATES its answers, and generation can invent APIs. A live run returned examples using APIs that do not exist, the agent wrote them, check_symbols flagged the file, the agent re-ran the same lookup and got the same poison: a perfect loop. This shadow breaks it at the source: run the built-in lookup, then run the SAME symbol audit the write path uses over the returned example, and hand the agent the answer and the corrections in one result, so it learns the example is partly wrong BEFORE writing it, together with the interface's real members to build from instead.

The second-order bug, which is the one worth stealing

The lookup cache had pickled the poison. Of 137 cached entries, 126 carried warning text. Every fuzzy cache hit was re-serving a known-bad answer, forever.

Flagged answers are now uncacheable, and the cache module has a rule list at the top that reads like scar tissue, because it is:

never cache failures or empties, because those are the answers most legitimately retried, and a wrong fuzzy hit on a cached "returned nothing" is poison
serve each entry at most once per turn, because a fuzzy match can be wrong and the corpus can change

One more thing, and I think it generalises

Once the guardrails were in, we probed the same grounded prompt across three model tiers, all with identical retrieval and identical checks:

Model tier	Grounded probe
frontier	5/5
mid	2/5
cheap	0/5

The verdict in the notes: the guardrails make a weak model SAFE (bounced), not GROUNDED.

A gate stack stops bad output reaching the user. It does not turn a model that cannot follow grounding instructions into one that can. Budget for the model tier; do not expect the scaffolding to buy it back.

6. Deciding that an API is invented

This is the check everything else depends on, and the first version was wrong in an instructive way. It asked an LLM to list the symbols in the agent's code, then full-text searched each one.

Every step leaned on the thing that was broken: an LLM summarising another LLM's hallucination, and a search whose only possible verdict was "not found", which tells an agent nothing about what to write instead, so it kept the invented call.

That second clause is the part I underestimated. "Not found" is not actionable. An agent that receives it has no better move available than the one it already made, so it makes it again.

The reframe: the documentation is complete and enumerable, so the question is decidable. Resolve what an object is, then ask whether the member exists on it.

layout.tableCellEnd                       ->  layout is Layout    ->  no such member  ->  INVENTED
var cell = new TableCell(); cell.start()  ->  cell is TableCell   ->  check `start`

Where the index comes from

"Complete and enumerable" is carrying a lot of weight in that paragraph, so here is where the index actually comes from. It is not hand-written, and that is the only reason the verdict is trustworthy. A hand-maintained list of what exists is just one more thing that can drift from what exists.

The member index falls out of the documentation's own structure. Every documented member is its own chunk, titled Interface-member, so parsing titles yields {interface: {members}} for the entire surface. Two patterns in the prose supply the rest of what the resolver needs:

Pattern in the docs	What it yields
`Interface-member` chunk titles	every member of every interface
`accessed using the global property "layout"`	`layout` resolves to `Layout`
`Parent interface: Block`	inherited members

The middle row is what makes receiver typing possible at all. The set of documented globals, the thing that turns layout.x into a decidable question rather than a guess, is extracted from a sentence the documentation happens to write about itself.

A second datagraph describes constructs rather than members: how each one is declared, and which calls open and close it. It is generated from the API definitions rather than curated, and the reason is the best argument in the codebase for never hand-writing ground truth:

Hand-curating it is error-prone: the first cut invented a List/listStart construct that does not exist.

We hallucinated an API while building the tool whose entire job is catching hallucinated APIs. The generator now derives the pairs mechanically: on a global-reachable interface, a <base>Start/<base>End pair whose Start takes a defined interface as its first argument. Pairs whose first argument is a primitive, like keepStart(string), are not constructs anyone instantiates, so they are skipped and reported rather than guessed at. Seven constructs survive that test.

An entry looks like this:

{
  "id": "table",
  "interface": "Table",
  "kind": "paired",
  "aliases": ["table", "grid"],
  "declare": "new Table",
  "start": "layout.tableStart",
  "end": "layout.tableEnd",
  "no_close": "Rows and cells end IMPLICITLY. There is NO tableCellEnd and NO tableRowEnd; those methods DO NOT EXIST."
}

That last field is the interesting one, because it is a negative fact. Some constructs close implicitly, so the end-calls a developer would reasonably expect are absent, and tableCellEnd is precisely what the model kept reaching for. The fact is authored once and read by both the checker and the terminology builder, so the two can never drift into disagreeing about it.

It also upgrades the finding. Instead of "no such member", which sends the agent hunting for a replacement that does not exist, the checker can say the call is unnecessary and the row closes on its own, so deleting it loses nothing. A checker that only knows what is absent can report; one that knows why it is absent can tell the agent what to do instead.

With the index in place, three further properties are what make the check survivable in production.

Precision over coverage

A member is only judged when the receiver's type is certain: a documented global, a new X(), or an interface named directly. A parameter, a function's return value, a nested property, all skipped.

a wrong accusation is worse than a miss: it sends the agent off to "fix" working code, and it teaches the user to ignore the check.

Eight check families, each of which exists because something landed

A member check alone is not enough, and every gap here was found in production rather than in design.

What landed	Why it slipped the previous checks	Check added
`engine.loadXML(...)`	undeclared receiver, so no type to resolve	object check
`loadXMLData()`	a bare call defined nowhere	function check
`const`, arrow functions	`node --check` passes what the target engine cannot run	syntax check (ES3)
`layout.blockEnd({ borderColor: 'red' })`	an invented key inside an object literal, not a dotted member	option check
`new ActiveXObject("Microsoft.XMLDOM")`	member check cannot type an unknown constructor	constructor check
`Object.keys`	valid JS, absent from an ES3-era engine	engine check
a referenced data file that does not exist	not a symbol at all	file check

The option check is the one I would not have thought of. An invented member on a dotted call is obvious. An invented key smuggled through an options object is invisible to a member check, and at runtime it is silently ignored, so the behaviour you promised the user simply never happens and nothing anywhere reports an error.

A checker's false positive is worse than a miss

We learned this twice, expensively.

Once by accident. A markdown fence in a knowledge-base snippet tripped a template-literal check and a real API was branded invented. The model then wrote that into its session memory: it learned that a real API does not exist, and carried that forward.

Once systematically. The target engine implements a genuine W3C DOM, and our browser-API denylist had been written on the assumption that DOM methods were browser-only. Of the 34 names on it, 23 were real APIs in this engine. The check had been flagging the agent's correct code, and it had helped cause the spiral in section 5.

Every new check now ships with a known-good-input case for both shapes it audits, whole files and fenced snippets.

The runtime layer

Static analysis has a hard limit: it must skip any receiver whose type is uncertain, which is precisely where dynamic code lives. So the last check generates a fake engine from the documentation index and actually runs the build against it.

The docs are COMPLETE and enumerable. So a fake engine can be GENERATED from them: every documented global and constructor exists; every documented member is a callable, chainable stub; touching an UNDOCUMENTED member on a typed object records an INVENTED finding. loadJS/importJS resolve real project files, so wiring is exercised for real.

It catches syntax errors and crashes, loads of files that do not exist, and invented members reached through dynamic flow: parameters and return values, exactly the receivers the static pass must skip. It runs once at the end of a turn, never per write, because one process launch per draft is cheap and one per write is not.

The limits are stated in the same docstring, which is the part I would want if I were adopting this:

What it can NOT prove: rendering. Data APIs return permissive stubs (no real XML), so loops may run zero iterations and zero-iteration paths go unexercised. This is a dry run, "the code executes against the documented surface", not a render test.

7. A finding that does not block is decoration

I built the checker, wired it in, and watched it work. It caught the invented APIs and reported them accurately, and nothing whatsoever happened as a result:

Told "these APIs do not exist, fix them now", the model wrote the user a prose essay ("you may need to rely on...") and finished the turn with the invented symbols still on disk. So findings also gate the END of the turn: a finish_gate the tool loop consults before accepting a final answer, which pushes the model back to work while anything is outstanding (same lesson as lookup_before_write: enforce it, don't ask for it).

The gate contract fits in one line:

gate(answer, asking=False) -> Optional[str]

Returning a string means the draft was not sent and that string goes back into the model's context. Returning None ships it.

Everything else is about the budget, and the budget is where the design decisions are.

Per-gate bounds compound

Gate	Own push budget
verify	2
bail	2
claim	1
checklist	1

Each is individually reasonable, and together they can bounce one turn six times, which is an unpredictable worst case for latency and cost. So there is a shared turn budget of 3 across every gate, and past it the composed gate goes silent and the turn ends with whatever honesty the pushes bought.

Which produced the next bug

Twice, every push was consumed by intermediate holds and a false final claim shipped unchallenged. Hence a reserve:

reserve_final keeps the last (user-visible) draft policed after the shared budget is spent mid-turn. Past max_pushes, up to reserve_final further pushes may be spent, but ONLY on NEW material: a draft whose text materially changed since the last hold, or new deeds. A verbatim doubled-down draft has already ignored this correction once, repeating it buys nothing, so it ends the turn as before.

Refuse before the write, not after

This was the single biggest behavioural improvement:

Auditing after the fact meant asking the model that just invented an API to un-invent it. It would rather explain, annotate, or stand by it. Refused content simply never becomes a file.

Police claims, not just code

The gate that catches the most is the simplest. Asked to split a file, the model created three modules and reported four:

Changes made:
  1. Created xmlLoader.js ...
  2. Created styledBox.js ...
  3. Created tableCreator.js ...
  4. Updated template.js to import and use the new files.

Step 4 never happened. It had planned it and narrated the plan in the past tense. The check is mechanical: past-tense verbs only, scoped to a single line, matched against ctx.files_changed, which every landed write funnels through.

The tense restriction is not incidental, and the comment explaining why earns its length:

# PAST TENSE ONLY, and deliberately so.
#   - "I will update template.js" is a plan, not a deed - the future tense must not trip this.
#   - The present tense describes what CODE does, not what the agent did: "template.js already
#     imports xmlLoader.js" and "createTable() creates a new Table" are statements of fact
#     about the project, and reading them as claims would gate honest, read-only answers.
# Only the past tense asserts "I did this", which is the thing the file system can contradict.

The scope restriction matters just as much: matching is per line, because a wider window lets one "Created" three bullets up capture every filename mentioned afterwards.

And the gate declares its own non-goal, which I think is why it works:

This checks only what the model VOLUNTEERS. It does not police silence... A false claim is wrong unconditionally; that is why this gate can be mechanical.

Two smaller rules that took real failures to learn:

Strike caps exist because the model can be wrong about how to fix something and must not be trapped re-trying forever.
Findings are split into introduced and inherited, after a user complained: "I only asked it to do 1 simple thing and it did it, but it also started talking about jsonData." Only a finding that a write in this turn introduced may hold the turn.

8. How the system talks to itself

Every gate above works by writing a message into the model's own context. Getting the wording of those messages right turned out to be as load-bearing as the logic that triggers them, and it is the part I have seen discussed least.

Start with the two constants that open and close every one of them.

The envelope

#: Opens every gate message. Injected corrections must not read as the user speaking, or the
#: model treats its own gated draft as user feedback.
ENVELOPE = "[automated pre-send check - NOT a message from the user] "

That is not a stylistic choice. Without the marker the model reads its own blocked draft as something the user saw and objected to, and then reasons about it as if a conversation had happened.

The silence trailer

#: Closes every gate message. Without it the model apologises to the user for a draft they never
#: saw ("You are correct. I apologise..."), explains itself, and asks permission, turning a silent
#: retry into a wall of confusing chat. The gate is plumbing and must stay invisible.
SILENCE = ("Do not apologise. Do not mention this check, your draft, or any mistake - the user "
           "never saw them and will not understand a reference to them. Do not ask permission: "
           "just do the work. Your reply must read as if you had done it right the first time. "
           # Live: a bail push made the model REWRITE a file smaller, deleting footer code
           # that had already passed the symbol check, and surrender in a comment instead.
           "Never respond to this check by deleting or stripping code that already passed "
           "its checks - fix FORWARD (add, complete, correct), never retreat.")

Every sentence in that string is a bug, and the last one is the strangest: a push meant to stop the agent giving up made it delete working, already-verified code and write an apology in a comment.

Correction pressure has a direction, and if you do not specify it, the model will sometimes satisfy you by removing the problem rather than solving it.

Both constants live in one module because they did not, once, and a hand-copied version quietly drifted out of sync.

Three of the messages, chosen for what each one teaches

Escalation on repetition - the second push stops being polite

SECOND refusal: you have made ZERO writes this turn, the file system is the ground truth
and it shows no change, yet your draft still reports work as done. It is false.

Do ONE of these, nothing else: (1) make the change now with a write tool and then
describe it, or (2) answer exactly what is true: that you changed nothing, and what you
found in the file instead.

The first time the model claims work it did not do, it gets a straightforward correction. This is what it gets if it doubles down.

Constraining what the model then tells the user - the budget message

[lookup budget exhausted - {live} knowledge lookups have run this turn] No further
lookups will run. Searching again, in any wording, returns this same message. STOP
searching and WRITE... In that final answer, if information is missing, say the SEARCH
BUDGET ran out, do NOT claim the knowledge base lacks it; the budget ending says nothing
about what the documentation contains.

The obvious message is "stop searching". The last clause took a live failure to learn: without it, our own rate limit came out of the agent's mouth as a factual claim about the customer's documentation. The agent had no way to know the difference, so we had to tell it.

A checker that argues - pre-empting the three moves the model actually makes

This is not "unverified", the documentation is COMPLETE, and these were checked against
every member of the interface. You invented them just now, most likely because a similar
name exists (tableStart exists, so tableCellEnd feels like it must). It does not. The
file will fail on the real engine.

Rewrite those calls using the real members listed above. Do NOT re-run the same
rag_lookup that produced this code; it will return the same wrong example. If you need a
pattern, rag_lookup a DIFFERENT, more specific goal. Do NOT annotate the call and move
on; do NOT keep it because it looks right. If the interface genuinely has no member for
what you intended, leave that part out and say so in your final answer, never leave a
call to a method that does not exist.

Note that it names the failed retrieval explicitly. Without that line the agent re-runs the lookup that produced the bad code in the first place, which is the closed loop from section 5.

The trust boundary between agents

When a plan is decomposed, each item runs as a bounded worker with its own context, and the worker's report comes back to the composer as a user message. Workers are told to end with what they built and what it exposes.

The composer does not believe the "exposes" line:

def exposures(path: str, content: str, answer: str) -> str:
    """One line describing what a landed file exposes - MECHANICAL over the file itself
    (the file is a fact; a model-authored "exposes" line is a claim). Falls back to the
    worker's own short answer for non-JS / no matches."""

That parenthesis is the whole article in one sentence. Everywhere the system has a choice between what a model says happened and what the file system says happened, it takes the file system, and where it cannot, it says so and degrades deliberately.

9. Some mistakes should not be catchable

The fix for this last one runs against every instinct: it is to take intelligence away, not add more.

Asked to remove a table, the agent removed one line, rewrote everything around it, and made the file bigger:

Attempt	Result
original	1527 bytes
"remove the table"	1653 bytes, reported as success
retry	1659 bytes, reported as success

The diagnosis had two layers, and both are worth knowing:

The removal instruction fell into a generative chunk-rewrite path, so "remove" became "rewrite this region, minus the thing".
The forced pre-write lookup for "how to remove a table" returned the documentation for creating one, because that is what the corpus has. That poisoned the intent: the agent came away believing you remove a table by calling tableEnd().

The fix was to take the generation out of that step entirely. The model reads the file and emits L<n>: DELETE markers, nothing else, and ordinary code drops those lines:

Delete-only pass: the model reads the file (or a where=-scoped region) and marks which lines belong to the thing being removed; they are dropped deterministically. It can neither rewrite nor add, so a removal can only shrink the file. This is where EVERY pure deletion goes instead of the generative chunk rewrite (which deleted one line and grew the file live).

The same file went from 1527 bytes to 678, with the right construct gone and the one next to it untouched.

What it all amounts to

Mechanism	Catches
filename matching, construct footprints	literal search missing the obvious
identical-call breaker, turn budget, fuzzy cache, in-flight collapse	fixation and rewording loops
query inflation with an adaptive brake	one phrasing being a single point of failure
`resolve_topic`, confidence ladder, de-nouning rewrite	goal words that miss an API-indexed corpus
self-auditing lookup shadow, uncacheable flagged answers	the retrieval tool inventing APIs
`check_symbols` (8 families) + the `dry_run` shim	invented APIs, statically and at runtime
composed finish gates, pre-write refusal, claim audit	findings being ignored, work claimed but not done
envelope, silence trailer, escalating pushes	the correction itself being misread or over-obeyed
delete-scan	a destructive edit that could grow the file

Across all of this, the model never got better. Every version ran on the same tier that failed the first time.

What changed was the space of things it could get away with: what it could search without a limit, what it could claim without evidence, what it could write without being refused, and what it could remove by generating rather than by marking.

That is unglamorous work, and it is almost all of the work.

Creating a web API with Lua using Nginx OpenResty

Joe Buckle — Sun, 02 Feb 2020 16:58:55 +0000

I'm a bit of an old fashioned, pragmatic developer and tend to stick with frameworks and patterns that I know are reliable - a 'stick in the mud' if you will.

This dive is deviating from my usual path.

I had never considered Lua to be a viable language to create a backend for a web application. Of course I'm not totally oblivious to the language itself - I do know it's used mostly for embedding in other software and game engines - but that's it really.

I came across the OpenResty project whilst on a learning journey of Nginx web server.

Writing code directly on the web server rather than passing the request to some interpreter is a very interesting concept for me.
Lua runs directly inside the Nginx worker which means a very small barrier between the webserver and the application code.
Additionally, Lua is known for being immensely fast compared to interpreted languages particularly when using the LuaJIT compiler.

Nginx is capable of handling a very high number of concurrent connections at a very low memory footprint.

However, there doesn't appear to be that much excitement about it.
It's been around for some time (at least since 2016) but it does appear to be lacking in developer adoption. Saying that, some big names are using it, including Cloudflare and Tumbler.

Despite that - the performance gain from this sort of backend has peaked my interest.

The easiest way (perhaps the only way) to get started with this is by installing OpenResty.
This provides the Nginx web server with the Lua module.

If you're familiar with Nginx this example will make sense to you. The location directive is being passed a Lua script as a file.
The output from that script is expected to be in JSON in this configuration:

location ~ ^/api(.*)$ {
  default_type 'text/json';
  add_header 'Content-Type' 'application/json';
  content_by_lua_file /etc/openresty/sites/api.lua; # < Points to Lua file
}

You can also pass arbitrary code into content_by_lua and content_by_lua_block.

Now, Lua is a reasonable simple language to pick up if you're used to Ruby or Python. I hadn't used it before but I was still able to quickly write a script that allows creation of API endpoints and parses the body and parameters.

There is an object you can access in Lua called ngx. This provides you with the data passed into Nginx from the request such as the body or the path etc...

Example:

ngx.var.request_method -- POST, GET.. whatever
ngx.req.get_body_data() -- The data passed in from the request
ngx.var.uri -- The request path

The script I wrote allows me to set the Method and Endpoints that are allowed.

--[[ api.lua --]]

-- Helper functions
function strSplit(delim,str)
    local t = {}

    for substr in string.gmatch(str, "[^".. delim.. "]*") do
        if substr ~= nil and string.len(substr) > 0 then
            table.insert(t,substr)
        end
    end

    return t
end

-- Read body being passed
-- Required for ngx.req.get_body_data()
ngx.req.read_body();
-- Parser for sending JSON back to the client
local cjson = require("cjson")
-- Strip the api/ bit from the request path
local reqPath = ngx.var.uri:gsub("api/", "");
-- Get the request method (POST, GET etc..)
local reqMethod = ngx.var.request_method
-- Parse the body data as JSON
local body = ngx.req.get_body_data() ==
        -- This is like a ternary statement for Lua
        -- It is saying if doesn't exist at least
        -- define as empty object
        nil and {} or cjson.decode(ngx.req.get_body_data());

Api = {}
Api.__index = Api
-- Declare API not yet responded
Api.responded = false;
-- Function for checking input from client
function Api.endpoint(method, path, callback)
    -- If API not already responded
    if Api.responded == false then
        -- KeyData = params passed in path
        local keyData = {}
        -- If this endpoint has params
        if string.find(path, "<(.-)>")
        then
            -- Split origin and passed path sections
            local splitPath = strSplit("/", path)
            local splitReqPath = strSplit("/", reqPath)
            -- Iterate over splitPath
            for i, k in pairs(splitPath) do
                -- If chunk contains <something>
                if string.find(k, "<(.-)>")
                then
                    -- Add to keyData
                    keyData[string.match(k, "%<(%a+)%>")] = splitReqPath[i]
                    -- Replace matches with default for validation
                    reqPath = string.gsub(reqPath, splitReqPath[i], k)
                end
            end
        end

        -- return false if path doesn't match anything
        if reqPath ~= path
        then
            return false;
        end
        -- return error if method not allowed
        if reqMethod ~= method
        then
            return ngx.say(
                cjson.encode({
                    error=500,
                    message="Method " .. reqMethod .. " not allowed"
                })
            )
        end

        -- Make sure we don't run this again
        Api.responded = true;

        -- return body if all OK
        body.keyData = keyData
        return callback(body);
    end

    return false;
end

Then I can call Api.endpoint() following to create my endpoints:

--[[ api.lua --]]
Api.endpoint('POST', '/test',
    function(body)
        return ngx.say(
            cjson.encode(
                {
                    method=method,
                    path=path,
                    body=body
                }
            )
        );
    end
)

Api.endpoint('GET', '/test/<id>/<name>',
    function(body)
        return ngx.say(
            cjson.encode(
                {
                    method=method,
                    path=path,
                    body=body,
                }
            )
        );
    end
)

So this has got me started. Obviously the logic inside these endpoints would be much more complex and usually have to connect to databases - all of which OpenResty supports.

One thing I did come across was the Lapis Framework created by Leafo who I know mostly from creating the PHP compilers for Less and SCSS.

Github for code the code here - https://github.com/bambattajb/openresty-api-example,

Embedded UI Components with Preact

Joe Buckle — Tue, 24 Dec 2019 10:34:14 +0000

As a front-end developer it's quite common to be tasked with creating a new UI component for an application.

You're given access to some backend API and you have to crack on and build a UI that sits somewhere inside the website as a widget, page or whatever.

The component is complex enough that you need to manage view states, so you'd quite like a system for that.
You also like using the cleaner ES6 syntax and you're already familiar with React and JSX and all that jazz.

Now, you don't have access to the applications overarching build tools - you just need to find a way of passing some reasonably sized and efficiently written code over to the application maintainers who will factor it into their monolith 😊.

I tend to see these UI components as their own small compartmentalised applications but I am fully conscious of the increased size when shipping seemingly unnecessary frameworks in packaged code.

So React is out of the question because it's > 30kb.

I eventually discovered Preact which claimed to come in at 3kb and supports all of the Virtual DOM and State Management features of React.

I totally signed up and was not disappointed. Just set up a Webpack build environment and away you go.

How is Preact so small?

React has its own Synthetic Event System for handling events and Preact uses the browsers native addEventListener.
This could lead to a user experience penalty.

React is trying very hard (30kb + hard) to ensure a consistent feel in whatever browser it's being run in; whereas Preact has chosen to use unpredictable browser-native event listeners for a massively reduced packaged size.

More on the 'subtle' differences here - Main Differences.

What's missing?

Well, not much!

Proptypes (they aren't used all the time anyway)
Children

Conclusion

Preact is a great solution to my problem but it does look as though it might come with some UI performance trade-offs leading to an inconsistent UX.

That said, if you chose to build and embedded UI component without a framework you most likely wouldn't create your own synthetic event system for seamlessly handling differences in native events.

Preact is totally winning for me in this context 👍

I love WordPress, but I don't like its ecosystem

Joe Buckle — Sat, 21 Dec 2019 15:24:21 +0000

Come gather round friends and I'll tell you a tale
Of when I claimed to be a WordPress guru
But was then made to use off-the-shelf themes and plugins
And that claim I sought to undo

I've been using WordPress to solve content management problems for clients in the digital publishing industry for around 10 years and know its core features pretty well having spent countless hours traversing through the codex and reading a plethora of problem-solving developer articles.

I am an advocate for the software and push it as a low-cost, easy to use, extensible and familiar content editing & publishing platform for digital publishing teams.

From a developer perspective the core provides a very good and efficient Query Engine, Database Abstraction Layer, Rest API Framework, User / Roles management and loads more.

It also provides a decent admin interface that is intuitive and adaptable enough that I can tailor it to specific business requirements.

What's not to like?

Well, last year I assumed the role of "WordPress Guru".

Up until then I had been using WordPress as a toolkit for building a unique set of features for my clients. I would be writing a lot of specific business logic around this toolkit just as you would a framework.

At the time of accepting this role I didn't know which sites I'd be working on and just assumed that, armed with this knowledge, I should be able to handle it right?

But I was wrong!

You see, there is, in my opinion, a "Dark Side" to WordPress - its Theme and Plugin culture.

It's not like I was unfamiliar with this and whenever I'd read up or hear of this culture I thought it was individuals or small businesses on small budgets looking to get a website up as cheap as possible and that it shouldn't apply to serious businesses or organisations.

I was wrong again!

I used the term "Dark Side" because I would be dealing with content managers that have a business requirement for something that is more flexible than the feature-restrictive theme or plugin. At some point this system was sold to them either by someone in-house or a developer that is now long gone.

The first problem is you have a client that doesn't understand "why" this feature is difficult to implement (they're right - it shouldn't be, if this system was built properly).
The second is the developer trying to shoehorn features into this system knowing it's not really fit for purpose thus making their ongoing maintenance work more difficult.

Sometimes you have to do things like "modify core code" of plugins only to learn of a "major security update" some weeks later that you now have to factor in.

Sites end up becoming a maintenance and security headache.

Lesson learned

I will never claim to be a "WordPress Guru" again as putting me in an environment where I have to build or extend sites that use WooThemes or Divi (for example) then I am lost!

Someone who is a "WordPress Guru" would understand the entire ecosystem of available WordPress products - not just its core system.

My advice for publishers is if you are looking at purchasing a Theme or Plugin to handle a feature you're needing make sure it does EVERYTHING you're ever going to need.

Otherwise consult a developer - there are some good quality plugins that allow themselves to be extended by using custom hooks and filters, perhaps enough so to fill the void in requirements.