DEV Community: sunnydachs

I changed a tool's schema four ways and watched three agent frameworks break differently — 36 runs measured

sunnydachs — Fri, 25 Sep 2026 18:23:01 +0000

If you run agents with tools, this day always comes:

"I need to change this tool's arguments — will the agent survive?"

Tool schema changes are a daily event in agent development. An argument gets renamed. A type changes. An argument is removed. A new required argument appears.

The ideal is to fix the prompt and all call sites at the same time. In practice, "the schema changed first, and the prompts and callers haven't caught up" is a state that always happens.

In my previous article, I ran a comparison across Strands, LangGraph, and CrewAI with the same task. An argument rename (text → content) passed 9/9 in all three frameworks.

That raised a fair question:

"Was that rename the first schema change you tried — or the one that survived after a harsher one broke everything?"

In other words: did rename only work because it happens to be the easiest change?

To answer it, I split schema changes into four harshness levels and re-ran the measurement across 36 runs.

Repo (all code, all traces, all analysis scripts):

https://github.com/sunnydachs/agent-framework-showdown

The experiment design

Same task, same model across all runs, same recording proxy: 3 frameworks × 4 levels × 3 runs = 36 runs.

[The four schema-change levels]
rename: argument renamed        text -> content
type:   type changed            str -> int (a document id)
remove: argument deleted        the text arg is gone entirely
add:    new required argument   "note", never mentioned in the prompt

At each level, the only thing that changes is the word_count tool's signature.

The task prompt stays fixed at every level: "write an 80–120 word digest and verify the word count with word_count" — not one character changes.

What I measured:

wrong-arg calls: calls whose arguments don't match the schema
schema errors: how often the tool layer rejected a call
recovery: did the model self-correct within the run, give up, or never notice the error at all
silent failure: the process exits 0 but the word count was never actually verified

The result: each level dies a different way

3 runs per cell:

Level	Strands	LangGraph	CrewAI
rename	✅ 3/3 verified	✅ 3/3 verified	✅ 3/3 verified
type	⚠️ 0/3 verified (silent)	❌ 3/3 crash	❌ 3/3 dead
remove	⚠️ 0/3 verified (silent)	❌ 3/3 crash	⚠️ 0/3 verified (silent)
add	✅ 3/3 verified	❌ 3/3 crash	✅ 3/3 verified

The direct answer to the question: rename was not "the first change I tried" — it was the one that survived.

And it was the only level that survived. One step up the ladder, and all three frameworks die in three different ways.

Strands: degrades silently

At the type level (the argument goes from str to int), Strands behaved like this:

model:  dutifully sends a numeric document id
tool:   returns "document not found"
model:  outputs the digest anyway and finishes
result: exit 0, zero verifications, no error surfaced

No error ever surfaces. Only the token consumption swells to 2.4× (rename mean 2,505 → type mean 6,004).

At the remove level, the argument-less tool returned a quiet placeholder (8 words), and the model accepted it as "verification passed" in 3/3 runs.

The process exits 0 — and nothing was verified.

That is a silent failure.

I wrote in the previous article that agent failures happen quietly. Schema change is one of the triggers that can cause exactly that.

LangGraph: fails loudly

LangGraph is the opposite.

Because the tool is called directly inside Python code, a signature change becomes an immediate TypeError.

verify node: word_count(state["draft"]) runs
tools_harsh: TypeError (the signature changed)
graph:      the draft is finished, but the result is never emitted — dead

9/9 runs across type, remove, and add crashed inside the verify node. Each run gets as far as the draft-generation LLM call, then dies at the verification point.

Silent failure effectively does not exist in LangGraph. In exchange, no run ever produces output.

The failure is loud — and it is a full outage. (Because the error is explicit and the process halts, you notice immediately — but as a service, it is completely down.)

CrewAI: does both

CrewAI sits in between.

At the type level, the model stuffed the digest text into the int field: 9 wrong-arg calls across 3 runs. The tool layer returned a JSON parse error, and the next request carrying that error was rejected by the provider with a 400.

model:          stuffs the text into an int field
tool:           JSON parse error
next request:   provider 400 -> dead

CrewAI's failure shape is not the schema mismatch itself — it dies on the way back from it.

At the remove level, CrewAI believed the quiet placeholder just like Strands and exited 0. (The word-count check the agent was supposed to perform is effectively skipped, yet the run still reports completion.)

"add" was the only harsh change that model-driven frameworks survived

One interesting reversal.

The add level — a new required argument the prompt never mentions — looks like the most unreasonable change of the four.

But Strands and CrewAI fabricated a plausible value in 6/6 runs and resumed verification:

model's fabrication: "Draft digest of AI agents news"
-> a natural description, inferred from the schema itself
-> verification resumes, 3/3 pass

The model can read the schema and reason about it.

Changes the model can infer from the schema itself get absorbed.

Changes that contradict the prompt (type) or delete the verification path itself (remove) do not.

There is a practical lesson here: when you add a new required argument, if its name and description are designed so a model can fill them in by inference, the agent is surprisingly likely to survive.

The failure shapes, summarized

Across 36 runs, each framework's way of breaking is now quantified:

Framework	Failure shape	How you detect it
Strands	silent degradation (exit 0, no verification)	traces only
LangGraph	full outage (immediate TypeError death)	process monitoring, instantly
CrewAI	dies during recovery (provider 400)	both process monitoring and traces

Which framework is "safe"? The data's answer: none of them, alone.

The one that fails quietly needs auditing. The one that fails loudly trades that for availability. To get a schema change through safely, you have to layer defenses.

Honest limitations

3 runs per cell show direction, not statistical claims
The tool is a local function — real HTTP API schema changes also involve network, latency, and status codes
The quiet placeholder at the remove level is a design choice of the experiment; a real deployment would normally return an error
Single model measured; schema-inference ability varies by model

Reproduce it

All code, all traces, and all analysis scripts are public. The README has the commands for every experiment so far (36 + the earlier 108 runs).

https://github.com/sunnydachs/agent-framework-showdown

This is a personal OSS project — no warranty, use at your own risk. Issues and improvement ideas are welcome.

Previously in this series: what happens when enterprise requirements hit the same three frameworks (45 runs) and do agents survive a crash (34 runs).

AI answers are easy to get but expensive to read — a CLI that filters them down

sunnydachs — Fri, 25 Sep 2026 01:02:52 +0000

You ask an AI assistant a question and an answer comes back. Reading it is the expensive part. I find the long round-trips tiring.

Ask it how to remove duplicates from a CSV in Python and you get 800 characters covering BOMs, encodings, and pandas design philosophy. What you actually need is twelve lines of code and one way to check it works.

Telling the assistant "be concise" ahead of time doesn't always help. Asking "tldr pls" costs another round-trip.

So I built a CLI that filters the answer after it's produced. Feed it a Markdown answer and it emits three sections: conclusion, minimal code, and verification. It takes stdin, so you just pipe it.

https://github.com/sunnydachs/concise-md

$ concise answer.md
## Conclusion
- The standard approach is exponential backoff with jitter:
## Minimal code
<emits the code block, verbatim>
## How to verify
- Test it with a mock that returns 429 three times and then 200.

The code section carries the actual code block from the answer, unchanged:

RETRYABLE = {429, 502, 503, 504}

def get_with_retry(url, **kwargs):
    for attempt in range(4):
        try:
            r = requests.get(url, timeout=30, **kwargs)
        except requests.RequestException:
            time.sleep(2 ** attempt)
            continue
        if r.status_code not in RETRYABLE:
            return r
    raise RuntimeError("gave up")

Zero dependencies, Python 3.11+, no LLM calls. Runs in 0.02 seconds.

Why not summarize with an LLM

I built that version first and tried it.

On a long Japanese input, the summary itself came out clean, but the code was broken.

In the deduplication example, the model generated a line that re-ran the same duplicate-count calculation on an already-deduplicated DataFrame, as part of "reorganizing" the code. Run it and you get removed 0 rows.

For a learning artifact, that is fatal.

Summarizing prose is what LLMs are good at. Code is a different story, so I drew the line there: don't touch code. With rule-based parsing, code blocks come out byte-for-byte identical. If the goal is keeping an AI answer as study material, that guarantee is the whole point.

The rules behind it

The implementation is plain Markdown parsing.

It splits the input into heading-delimited sections, detaches code fences from prose, joins wrapped paragraphs, and breaks prose into sentences.

The conclusion is the first sentence containing a marker such as tl;dr, "in short", or "the standard approach", falling back to the opening sentence of the first section.

Code is capped at three blocks of 25 lines each. A block that opens with "DON'T" is treated as an anti-pattern and placed after the preferred example, not before.

For verification, a "Verification" / "How to verify" heading wins first; otherwise it picks up sentences containing words like "verify" or "check" from the prose.

Finally, if the prose is under 10 lines and there's at most one code block, the input is short enough already and passes through unchanged. Restructuring a short answer just costs the reader.

All of these decisions are covered by tests. Twelve of them run in CI on every commit (Python 3.11-3.13).

Honest limitations

The Markdown parser is a collection of regexes, not a full CommonMark implementation.

Table rows are treated as prose, so a table-heavy answer can produce an odd conclusion.

Also, the tool only shortens how the document looks. It can't judge whether the content is correct. A wrong answer comes out as a shorter, wrong answer. If you are using this for study, run the trimmed code yourself.

Measured

On my own sample set, an 88-line English answer came out as 46 lines, and a 57-line Japanese answer as 26 lines. Both in 0.02 seconds.

Feed it an answer that is already to the point and nothing happens — it comes back unchanged. That's the spec.

Closing

When I use AI for learning, I realized I was losing time on reading, not thinking. This CLI is my one-lane fix for the "read it again" tax.

It's a personal OSS project with no warranty, but if something breaks, an issue is welcome.

https://github.com/sunnydachs/concise-md

Do agents survive a crash, and why does an LLM retry execute the same side effect twice? 34 runs measured

sunnydachs — Wed, 23 Sep 2026 15:28:32 +0000

When you hand an agent work with external effects, two worries always come up.

First: when the process dies, what happens to the work in progress? An agent paused at an approval gate crashes — can it resume, or does everything restart from scratch?

Second: when an LLM retries, does it execute the same side effect twice? Retrying a send or a data mutation carries a real double-execution risk.

My previous article measured HITL approval, audit trails, and structured output across three frameworks. It recorded a failure where Strands double-fired a publish call with the identical draft. This article measures what comes next: 34 more runs, same task, same model, same recorder proxy.

Repo (all code, traces, and analysis scripts open):

https://github.com/sunnydachs/agent-framework-showdown

(Read the previous article here. here.)

The experiment design

Same task, same model, same recorder proxy: 3 frameworks x 3 cells x multiple runs.

[agent] → [approval gate] → [publish (destructive)]
      ↑               ↑
  crash it here    pauses waiting

The three cells:

Cell A — crash recovery: SIGKILL a process paused at the approval gate; can a NEW process resume from where it stopped?
Cell B — idempotency: call publish under 3 key strategies (position key / content hash / no key) and count duplicate executions on retry
Cell C — audit under retry: score whether an auditor reading the traces alone can detect the duplicate

The three idempotency key strategies:

position key:   {workflow}:{step}:{tool}
content hash:   sha256(the raw arguments)
no key:         nothing

Cell A: crash recovery — a durable checkpointer is the difference between resume and redo

SIGKILL the agent while it waits for approval, then resume in a new process:

Framework	Persistence	Resume time	State survived	LLM calls to resume
LangGraph (durable checkpointer)	checkpointer on disk	0.01-0.02s	3/3	zero
LangGraph (no checkpointer)	none	-	0/3	-
Strands	none built-in	4.9s avg	full re-run	5.3 avg
CrewAI	none for agents	4.2s avg	full re-run	2.0 avg

LangGraph's durable checkpointer persists the graph state to disk even while the process is dead. The new process restores to that position in 0.01s — with zero LLM calls. The difference between resume and redo is only whether the state lives outside the process.

Without a checkpointer, an identical-looking "resume" is a full re-run: Strands runs its average 5.3 LLM calls again and pays the full token cost a second time.

What if the crash happens before the first checkpoint?

LangGraph has a known issue here (#8764): if the process dies before the first checkpoint is persisted, recovery may find no checkpoint and no record that the run was ever accepted. On the version I tested, resuming the empty thread succeeded without raising — so the behavior is version-dependent. Don't rely on the error either way; keep an external acceptance ledger.

Cell B: idempotency — the content-hash key silently fails exactly when the model rewords

Average duplicate executions when an LLM retry re-calls publish:

Key strategy	Same-args retry	Reworded retry
Position key	1 dup, all deduped	3 dups, 33% deduped + rest rejected as caller bug
Content hash	1 dup, all deduped	1 dup, 0% deduped — it slipped through
No key	1.33 dups, 0% deduped	1 dup, 0% deduped

This is the core result. The content-hash key silently fails the moment the model rewords the arguments on retry.

The reason is simple. An LLM retry does not replay the saved HTTP request. It reasons again from a context that now includes the timeout error, and emits a new tool call. The arguments get reworded, the order changes, fields appear. With sha256(args) as the key, the retry produces a different hash, sails past the dedup check, and executes the side effect a second time.

The position key ({workflow}:{step}:{tool}) identifies the intent — where the call sits in the workflow — not the bytes. The same position with the same operation yields the same key no matter how the arguments change.

One more measurement: every duplicated publish carried a different tool_call ID on the wire. Nothing at the protocol layer detects "this is the second time for this operation." Detection lives at the recording level only.

Note: the position-key strategy rejects "same position, different arguments" calls as a caller bug (2.67 of the runs here). That is by design — it flags intent drift instead of letting the key be reused.

Cell C: audit under retry — the trace-only auditor

Can an auditor reading the traces alone recover these four facts:

Decision rationale is readable: 100% in all cells
The duplicate is detectable: 100% in all cells
Dedup is provable (which attempt was deduped, and why): 100% in all cells
The retry directive is visible: 100% in all cells

But this is only because the recording is at the wire level. The proxy keeps every attempt — first call, dedup, caller-bug rejection — as its own record, so the auditor can reconstruct everything.

Framework-level trace surfaces show none of the double-firing. What matters in audit design is where the evidence lives.

The most important finding: the failure shape changes with the key strategy

Across the 34 runs, each key strategy fails differently:

Content-hash key: fails silently. The hash just changes — no error, dedup passed, and only the side effect runs twice
Position key: fails loudly. "Same position, different arguments" is rejected as a caller bug, so intent drift surfaces as an error
No key: prevents nothing. Every retry executes twice

For enterprise use, the silent failure is the scariest class. The intuition "a hash key makes retries safe" breaks the moment the caller becomes non-deterministic — that is the conclusion from these measurements.

Honest limitations

3 runs per cell is directional, not a statistical claim
The approver is scripted; no real UI or notification flow
The destructive action is simulated (though whether the gate held is read directly from the recorded traffic)
Single model. A different model may reword arguments differently

Reproduce it

Everything is open. The README has the commands for all experiments (34 runs here + 72 from the earlier articles):

https://github.com/sunnydachs/agent-framework-showdown

This is a personal OSS project — no warranty. Use at your own risk, and issues are welcome.

What happens when enterprise requirements hit Strands, LangGraph, and CrewAI - 45 runs measured

sunnydachs — Mon, 21 Sep 2026 15:46:51 +0000

What happens when enterprise requirements - human approval gates, audit
trails, structured output - hit three agent frameworks? The first article
measured how Strands, LangGraph, and CrewAI differ on a plain task. This one
measures what happens when the task grows up: 45 more runs, same recorder
proxy, same model, same tools.

The headline finding: the frameworks fail differently. Strands - the
model-driven one - finished with an empty output three times while exiting

LangGraph and CrewAI never did.

https://github.com/sunnydachs/agent-framework-showdown

(Read the previous article here. here.)

Why the enterprise angle

Two walls keep coming up in developer communities when agents touch regulated
work:

Audit: "why did the agent make this decision" must be answerable, or the workflow can't enter regulated territory
Approval: a human must be able to stop an agent before a destructive action (data mutation, sending, publishing)

The EU AI Act makes automatic logging and retention a legal obligation for
high-risk AI systems. So past the "working demo", these are the two
validations that matter. I ran them.

Experiment 1: human approval gates (18 runs)

The task: write a news digest, ask a human to approve, and only publish if
approved. Publishing is a simulated destructive action that must never fire
before approval.

The three frameworks implement the gate differently:

LangGraph: interrupt() suspends the whole graph; Command(resume=...) continues it after the decision
CrewAI: Task(human_input=True) - the crew pauses for console feedback after the task completes
Strands: prompt-only ("ask the reviewer before publishing") - the model-driven loop is the design

Results:

LangGraph suspended 6/6 runs and resumed in 0.0s - the checkpointer
restores state without re-executing. Reject routes away from the publish node
via an edge condition, so the gate is enforced by the graph's structure, not
by the model's good behavior.

Strands respected the order too: ask -> check -> publish 3/3, and never
published on reject. But one run called publish_article twice with the
identical draft. The prompt was followed; the model just double-fired the
destructive action. In a model-driven design, a second execution after
approval is a real risk.

CrewAI is the simplest shape: 1 call for approve, 2 for reject (the
feedback re-runs the task). One operational gotcha measured along the way:
returning the same rejection on every prompt spins an infinite loop - 131 LLM
calls with the prompt growing from 240 to 6,561 tokens. CrewAI re-runs and
re-prompts on every non-empty feedback, so the repetition policy is the
caller's responsibility.

Experiment 2: audit-trail reconstruction (36 runs analyzed)

The regulated question is "why did the agent decide this". Because every run
goes through the same recorder proxy, the traces have one shape - so I scored
whether an auditor can recover the seven audit-relevant facts (decision
rationale, tool call order, tool arguments, model identity, and more) from
each framework's traces:

Framework	Rationale	Tool order	Args	Silent failures
Strands	100%	100%	100%	2
LangGraph	100%	0%	0%	0
CrewAI	100%	50%	50%	0

Strands is model-driven, so everything the model saw and reasoned about stays
in the trace - the strongest audit story of the three. The flip side is
exactly those 2 silent failures.

LangGraph's 0% is not a defect: its tool calls live in code, not on the wire.
Read the code and you know the order; read only the trace and you don't. That
is the real audit-design trade-off: where the evidence lives changes with
the framework's control-flow model.

Experiment 3: structured output (9 runs)

Output the digest as strict JSON with exactly 4 keys (summary,
word_count, topics, publish_ready). All three frameworks hit 100%
compliance, and word_count matched the actual summary length in every run -
putting the count inside the schema makes the model's self-verification
effective. Strands ran a validate loop averaging 2 calls (4 revisions in one
run).

The most important finding: they fail differently

Across all 45 runs, only Strands finished with an empty output three
times. The model built the complete result, handed it to the validation tool,
and then emitted nothing as the final answer. The run exits 0 - it looks
successful. You only catch it by reading the trace.

LangGraph and CrewAI: zero. In a pipeline design, the output node IS the
deliverable, so an empty answer is structurally hard to produce.

For enterprise use, this is the scariest class of failure: not an error that
stops the run, but a success-shaped empty result that breaks everything
downstream. (The empty outputs were recoverable - the model passes its full
result to the tool as an argument, so the trace holds it.)

Honest limitations

One model, 3 runs per cell - directional, not a definitive ranking. The human
is scripted; no real UI or notification flow. The destructive action is
simulated - though whether the gate held is read directly from the recorded
traffic, which is the part that's solid.

Reproduce it

Everything is open. The repo README has the commands for all five experiments
(72 recorded runs total, one proxy in front of every framework - that's the
whole foundation):

https://github.com/sunnydachs/agent-framework-showdown

This is a personal OSS project - no warranty. Use at your own risk, and
issues are welcome.

Cover image: generated with a local flux-schnell pipeline.

I built the same agent in Strands, LangGraph, and CrewAI — and recorded every LLM call to see how they actually differ

sunnydachs — Mon, 21 Sep 2026 05:47:42 +0000

Which framework should I actually use? Every comparison article has an
opinion. Almost none of them has data.

So I built the same agent three times — once in Strands, once in
LangGraph, once in CrewAI — ran 27 executions, and routed every
single LLM call through a local recorder proxy so the logs are directly
comparable. The frameworks write different log formats, different trace
shapes, different everything. A one-file proxy in front of all of them fixed
that.

The headline finding: LangGraph's explicit verify/revise loop cut output
variance by 77% (word-count spread 13 -> 3) — at 2.5x the tokens and 2.5x
the latency. Explicit control buys determinism, and you pay for it in tokens
and latency.

https://github.com/sunnydachs/agent-framework-showdown

The task

A tech-news digest agent, identical across all three frameworks:

collect 5 headlines via a fetch_headlines tool
write a ~100-word digest
verify the word count via a word_count tool, revising if out of band

The tools are deterministic and local — no network, no LLM inside them —
because the thing being measured is the framework's behavior, not the tool's.
Same model behind the proxy for all three runs.

How the three frameworks differ (in code)

The philosophy split is real, and it shows up in the implementation:

Strands (model-driven): you hand the model two tools and a system prompt. The LLM decides which tool to call, in what order, and when it's done. My implementation is 78 lines.
LangGraph (graph-driven): you define a typed state, every node, every edge, and the loop condition. The model only makes decisions inside nodes. 115 lines.
CrewAI (role-based): agents are team members with role/goal/backstory; the Crew orchestrates the handoff. 110 lines.

The dependency weight differs too: Strands 262MB (81 packages), LangGraph
71MB (45), CrewAI 699MB (142). CrewAI's weight is the flip side of
"fastest to prototype."

The observability design (the part that made this honest)

One recorder proxy sits in front of every framework
(proxy/rec_proxy.py): an HTTP server that forwards to any
OpenAI-compatible endpoint and writes every request/response pair as JSONL —
the full messages the framework sent (system prompt, tool schemas,
conversation history), the raw response (SSE-streamed or JSON), token usage,
latency, and status. The API key is stripped before writing.

An X-Run-Label header splits traces per run. Strands and LangChain stream
SSE, so a small reassembler (parse_sse.py) normalizes streamed responses
into the same shape as non-streamed ones (content / reasoning / tool_calls /
usage).

Same trace shape across frameworks is what lets you diff messages, tokens,
and latency exactly. Without it, "which framework is slower" is vibes.

The numbers (27 runs, 3 frameworks x 3 scenarios x 3 runs)

Three scenarios:

base — 80-120 word band (the original task)
tight — 95-105 words, which forces the verify/revise loop to fire
drift — the word_count tool's argument renamed (text -> content), testing how each framework copes with schema change

Word-count spread across 3 runs (max - min, lower = more stable):

Scenario	Strands	LangGraph	CrewAI
base	15	13	0
tight	11	3	2
drift	12	16	0

	Strands	LangGraph	CrewAI
LLM calls	3-5 (adaptive)	1 (base) / 2 (tight)	4 fixed
Total tokens (base)	2,370	2,007	2,532
Total latency	4.2s	4.7s	3.8s

What the traces actually show

LangGraph in base mode made a single LLM call and wrote the draft in one
shot. Word-count spread: 13 words. Switch to tight, and the explicit
verify/revise loop fires: spread drops to 3 (−77%), tokens go 2,007 -> 5,262,
latency 4.7s -> 11.8s. The graph structure guarantees the loop runs — that
is the product you're buying, and the price is 2.5x on everything.

CrewAI produced byte-identical output across all 3 base and drift runs
(96/96/96 words). temperature=0 plus the role prompt dominates. The flip
side: the call structure is always 4 calls, fixed. The framework that "just
ships it" also just repeats it.

Strands self-corrected inside its own loop. In one tight run the model
wrote a 77-word draft, called check_word_count, reasoned "77 is under 95,
I need to expand it," revised itself to 103, and verified again — five LLM
calls, variable depth per run ([5, 3, 4]). The model deciding when to stop
is part of the design, and the call count being adaptive is what you get for
trusting it.

Schema drift: models adapted perfectly (this time)

The rename was absorbed 100%: all three frameworks' models called the tool
with the new argument name, zero wrong-arg calls, no error recovery
triggered. Worth being precise about what this does and doesn't show: a
one-argument rename is the gentlest possible schema change. Type changes,
removed arguments, or changed return shapes would likely break the
model-driven side (which trusts its prompt over the wire), while LangGraph
would be immune — its tool calls live in code, not in a prompt.

Honest limitations

3 runs per cell is a trend check, not a statistical claim. Standard guidance treats ~30 runs as the floor for median estimation.
One task, one tool. This is exactly where model-driven frameworks shine. LangGraph's graph pays off in complex branching / approval / parallel workflows that I did not test.
One model. Reasoning volume, tool-call tendencies, and output variance are model-dependent and will change with a different model.
Dependency footprints move fast; numbers are from the initial install snapshot.

Reproduce it

python3 runs/run_matrix.py       # 27 runs in ~270s
python3 runs/analyze_matrix.py   # -> artifacts/matrix_report.json

The repo README has the full setup (three venvs, the recorder proxy, the
per-framework run commands). One proxy in front of everyone is the whole
trick — when the frameworks write different log formats, a same-shape
recorder is the simplest honest answer.

https://github.com/sunnydachs/agent-framework-showdown

Your GDScript arrays are slower than you think? — I built a CLI that detects the pitfalls and benchmarks them

sunnydachs — Sun, 20 Sep 2026 02:33:36 +0000

"Why is my loop stuttering? I'm not even doing anything heavy here."

Every Godot tutorial tells you to just use arrays. Nobody tells you that an
untyped array holds Variants, that a sequential-key Dictionary is ~2x slower
than a plain Array, or that the official docs contradict themselves about
packed arrays. This is about gd-bench, a CLI I built that scans your
.gd files for array-type performance pitfalls and generates runnable Godot
micro-benchmarks for each one — with zero dependencies and no LLM anywhere.

https://github.com/sunnydachs/gd-bench

What it does

Point it at a Godot project. It scans every .gd file and reports where the
array-type choice is costing you performance, then writes a runnable
benchmark script per finding so you can measure before you rewrite.

pip install gd-bench

gd-bench .              # scan the current project
gd-bench . --gen-bench  # also generate Godot micro-benchmarks per finding

Example output (a small demo project, 7 files):

game/main.gd:4  🐢 UNTYPED ARRAY
    var inventory = []            # untyped array — Variant soup
    ->
game/main.gd:5  🔧 TYPED ARRAY
    var scores: Array[int] = []   # typed array — fine but Packed is faster
    ->
game/main.gd:6  ✅ PACKED ARRAY
    var points: PackedVector2Array = PackedVector2Array()  # already optimal
    ->

summary: {"typed_array": 5, "dict_as_array": 2, "untyped_array": 1, "packed_array": 3}

It reports 4 kinds of findings:

UNTYPED ARRAY — var x = [1, 2, 3] holds Variants; slowest to iterate
TYPED ARRAY — Array[int] gives compile-time checks but is still Variant-backed; PackedInt32Array is faster for the same element type
PACKED ARRAY — contiguous memory, the fast path (reported as ✅)
DICT AS ARRAY — var d = {} filled with sequential keys is ~2x slower and uses about half the memory again versus a plain Array

A static type checker will never tell you which of these is actually slower
in your loop — and the official docs can't agree either. Only a
measurement can.

Why deliberately generate benchmarks instead of just linting

This was the core design decision. A linter could just say "use
PackedInt32Array here" and be done. But I deliberately built it as
scanner + benchmark generator instead. Three reasons:

The docs are contradictory. The GDScript reference (still, as of issue #10300 being discussed) says packed arrays are slower than generic arrays; the class reference says they are faster. A tool that parrots either line would be wrong half the time. A benchmark settles it per call site.
Performance advice without a measurement is folklore. The same operation (append + iterate) can flip the winner depending on element type and size. Generating the benchmark next to the finding makes "measure before you rewrite" the default workflow, not a discipline.
Generated benchmarks are honest about what they test. Each script measures exactly two data structures doing exactly one operation — no framework, no warm-up noise, no statistical smoothing. If the number looks wrong, the script is 40 lines you can read in a minute.

Deterministic work deserves deterministic tools.

The rule: no rewrite without a benchmark run

The detail I obsessed over: the tool must never hand you a rewrite suggestion
it hasn't equipped you to verify. So the rule is:

Allowed: findings with a generated benchmark script next to them
Forbidden: "packed_array" findings get no benchmark — they're already the fast path, and benchmarking the optimal case just adds noise

That single line (書き換え前に必ず計測 / "always measure before rewriting")
is what keeps the tool from becoming another "just use X" voice. The
generated script is plain GDScript (extends SceneTree, runs headless) so
you can run it in CI or on a machine without the editor:

godot --headless --script gd-bench-out/bench_gd_untyped_array_4.gd
# gd-bench: untyped_array
#   untyped_array: 8231 usec
#   typed_array: 5120 usec
#   ratio: 1.61x

Testing against real-world patterns

Not a toy example — the detection patterns come from the same shapes real
Godot projects and issues use:

23 unit tests covering all 4 finding kinds, comment lines, non-sequential Dictionary keys (not flagged), and end-to-end CLI runs
The demo project above caught 8 findings across 5 files — including two DICT AS ARRAY cases that were doing 1000 sequential-key inserts per frame
The scanner correctly skipped its own generated benchmarks' comment lines and only flagged real declarations

I also scanned 5 popular open-source Godot projects with it
(GodotProjectZero, both GDQuest demos, godot-open-rts, Advanced-Movement-System).
That scan found real findings — godot-open-rts has 30 untyped arrays in its
source/ — and it caught the scanner lying: every DICT AS ARRAY hit on
String keys or Node keys was a false positive (e.g. units_synced[unit] = 1
where unit is a Node — a legitimate keyed Dictionary). v0.1.1 tightened the
rule: only int literals and range() loop variables are flagged, everything
else is left alone. One real finding was sacrificed for that; documented.

The real win wasn't just "it works" — it's that the tool's advice and the
docs' advice can be compared in one command, which is exactly what the
godot-docs issue is asking for.

Honest limitations

Line-based detection. GDScript has no stable public AST, so this is regex-based pattern matching. It catches the common declaration shapes (var lines, {} + indexed assignment); it does not catch arrays built through multi-line expressions or passed as literals into function calls. The for i in N: int-loop form is deliberately not treated as sequential (it's indistinguishable from element enumeration) — a documented false negative.
Benchmarks measure one operation. append + iterate-sum. Real frames do more (splice, sort, resize) — the generated script is a starting point, read it and extend it for your loop's actual shape.
No Godot required to scan, required to bench. The scanner runs anywhere Python runs; running the generated benchmark needs a Godot 4 binary (headless is fine).

Wrap-up

Frame time scales with your worst inner loop. Array-type choice is exactly
the class of problem where "scanner + generated measurement" earns its keep:
the advice is cheap to give, but only a benchmark tells you if it was true
for your code.

https://github.com/sunnydachs/gd-bench

This is a personal OSS project with no warranty. If you hit bugs or have
suggestions, GitHub issues are the best way to reach me.

Your OpenAPI spec is a contract. Nobody is checking it. I built a CLI that detects spec↔code drift with zero dependencies.

sunnydachs — Sat, 19 Sep 2026 07:13:34 +0000

"The generated client keeps calling DELETE /items/{id}."
"The server returns 405 Method Not Allowed."

An OpenAPI spec is not just documentation. SDKs, frontend types, API docs, mock servers — everything is generated from the spec. When the implementation drifts from it, everything generated starts lying quietly.

This is about oas-drift, a CLI I built that detects that drift — with zero dependencies (Python 3.11+ standard library only, no LLM).

https://github.com/sunnydachs/oas-drift

What it does

You give it an OpenAPI JSON spec and a Python codebase. It finds three classes of drift:

SPEC ONLY — defined in the spec, but no matching route in code
CODE ONLY — route exists in code, but is not in the spec
METHOD MISMATCH — path matches, but the HTTP method differs

# scan the current directory against a spec (read-only)
oas-drift --spec openapi.json

# scan a specific root, machine-readable output
oas-drift --spec openapi.json ./src --json

Here is a real run against a deliberately-drifted demo app:

oas-drift — scanned src
  spec: 5 endpoint(s) | code: 4 route(s)

/health  ➕ CODE ONLY
    route implemented (GET) but not defined in spec — src/app.py:21
/items/{id}  ⚠️ METHOD MISMATCH
    /items/{id}: in spec but not implemented: DELETE; implemented but not in spec: POST — src/app.py:17
/users/{id}  ⚠️ METHOD MISMATCH
    /users/{id}: in spec but not implemented: DELETE — src/app.py:13
/admin/stats  ⬜ SPEC ONLY
    defined in spec (GET) but no matching route in codebase
/items  ⬜ SPEC ONLY
    defined in spec (GET) but no matching route in codebase

summary: {"code_only": 1, "method_mismatch": 2, "spec_only": 2} | ok: 2

"The spec is the contract" only works if someone checks both sides. Code review sees the diff against the last commit — not against a spec written three months ago.

Why it's a detector, not a gate

This was the core design decision. Most CI-facing drift tools fail the build when they find anything. oas-drift's exit code is 0 either way. Three reasons:

Adoption survives. A tool that fails your build on day one gets removed on day two. A detector stays.
Drift has priorities. A /health endpoint missing from the spec is usually fine. A METHOD MISMATCH on a payment route is not. Which drift fails the build is policy — the tool shouldn't decide that for you.
Fully deterministic. It parses sources with Python's ast module — never imports, never executes, never writes. Same input → same report, always.

If you do want to fail on specific statuses, wire it into CI with --json and jq — the report is machine-readable by design.

Same principles as the sibling tools I shipped this month — doc-drift (README↔code) and plan-drift (tracking plan↔code). Deterministic work deserves deterministic tools.

The rule: paths match literally

The detail I obsessed over: path parameters and router prefixes. The rule:

/users/{id} in the spec matches /users/{id} in code — and only that.
Router prefixes are NOT resolved. A router with prefix="/items" serving /{id} does not match a spec's /items/{id}.

I validated this against a real, widely-used codebase — the backend of FastAPI's official full-stack-fastapi-template (25 files, 14 paths, 23 routes detected). Scanning it with a spec written in prefixed paths produces exactly the false-positive pair this rule predicts:

/items/{id}  ⬜ SPEC ONLY
    defined in spec (DELETE, GET, PUT) but no matching route in codebase
/{id}  ➕ CODE ONLY
    route implemented (DELETE, GET, PUT) but not defined in spec — backend/app/api/routes/items.py:48, ...

Two names for the same route. oas-drift doesn't guess — it reports what literally exists. If your project uses router prefixes, normalize the spec side first (the surest source is the /openapi.json your app actually serves).

And the important detection works: flipping the implemented POST /login/access-token to PUT in a test spec gets reported precisely:

/login/access-token  ⚠️ METHOD MISMATCH
    /login/access-token: in spec but not implemented: PUT; implemented but not in spec: POST — backend/app/api/routes/login.py:23

Testing against real-world data

15 pure-function tests (no network, no fixtures on disk), all passing. Runtime: 0.04s.
e2e on a real install path: wheel build → install → actual drift detection, verified.
A real codebase: FastAPI's official full-stack-fastapi-template backend — the deliberately planted METHOD MISMATCH was caught, and the router-prefix false-positive pair was something I discovered and documented.

Honest limitations

JSON only (OpenAPI 3.x JSON; YAML is a future item).
Literal path matching — normalizing router prefixes is on you.
f-string paths are invisible — @app.get(f"/users/{id}") isn't extracted; string literals only.
Routes and methods only — request/response schema comparison is a future item.

These are documented in the README. The current version prioritizes minimal, honest detection over coverage.

Wrap-up

Specs drift the moment the code moves and nobody updates them. A contract is only a contract if something checks it — oas-drift is that checking part, built read-only, fully deterministic, and dependency-free.

https://github.com/sunnydachs/oas-drift

This is a personal OSS project with no warranty. If you hit bugs or have suggestions, GitHub issues are the best way to reach me.

The tracking plan said "implemented". The dashboard says otherwise. I built a CLI that detects plan↔code drift using only AST.

sunnydachs — Fri, 18 Sep 2026 02:09:41 +0000

"Did shipping auth tracking make it into the release?"
"Yep, it's in."

Two weeks later, someone opens the dashboard and says "…this doesn't look right."

This is about plan-drift, a CLI I built that detects exactly that gap — the silent drift between your analytics tracking plan (what you intended to measure) and the actual implementation — without using an LLM at all.

https://github.com/sunnydachs/plan-drift

What it does

You give it a repository and a tracking plan (a JSON file). It finds four kinds of drift:

UNEXPECTED EVENT — implemented in code, but not in the tracking plan
UNIMPLEMENTED EVENT — in the plan, but no track() call found
PROPERTY MISMATCH — event name matches, but its properties diverge (undeclared props, missing required)
DYNAMIC — the event name or properties can't be resolved statically (manual check needed)

# scan the current repo against your plan (read-only)
plan-drift --plan tracking-plan.json

# another path, JSON output
plan-drift --plan tracking-plan.json ./src --json

Output looks like this:

plan-drift — scanned /home/dev/myproject
  tracking plan events: 4 | track() calls found: 6 (dynamic: 1)
app/events.py:12  [!] PROPERTY MISMATCH
    event 'Signed Up': undeclared property 'campaign' found in code but not in plan
app/analytics.py:34  [+] UNEXPECTED EVENT
    'Add To Cart' implemented (1 call) but not in the tracking plan
app/analytics.py:56  [ ] UNIMPLEMENTED EVENT
    'Checkout Started' declared in plan but no track() call found

"The event shipped but the properties are off" is invisible to IDEs and static analysis. Tracking code is rarely covered by tests. So nobody notices until the numbers look wrong.

Why deliberately not use an LLM

This was the core design decision. LLMs could parse ambiguous tracking code with more "understanding." But I built it on nothing but Python's standard ast (Abstract Syntax Tree). Three reasons:

Deterministic. Same input → same report, always. You can't put a tool in CI whose verdict changes run to run.
Read-only and safe. It never generates or edits code. It parses the syntax tree and compares.
Zero dependencies. Python 3.11+ is enough. pip install git+... and you're done.

Tracking calls like analytics.track(...) are statically analyzable — an LLM here would add cost, latency, and nondeterminism with no accuracy gain.

This is one answer to a question I keep coming back to when automating: how much should AI handle? Deterministic work deserves deterministic tools.

The design core: check both directions

The thing I cared most about was checking plan↔implementation in both directions:

An implemented event NOT in the plan is a dangerous signal — could be an unexpected addition, could be a misconfiguration
A planned event with no implementation is also a problem — the classic "the plan got written, nobody shipped it"

So the tool cross-references and enumerates every mismatch. Test files (tests.py, test_*.py) are excluded, because fixture events showing up as "unplanned implementations" would be pure noise.

Where it fits

Marketing team writes the plan → devs instantly see how far implementation has come
Renaming an event or dropping a property without updating the plan → caught immediately
Wire into CI to warn on critical drift

The "plan exists but implementation never got updated" problem I've seen at workplaces — this gets you most of the way there.

Honest limitations

Dynamic events are reported, not resolved — track(event_name, props) shows up as DYNAMIC for manual review.
Python only for now (.py). JS needs a different parser — planned.
Only key presence is checked, not values — if the plan says plan: string and the code sends an int, that's a future improvement.

These are documented on the roadmap. The current version prioritizes correctness and low false positives over coverage.

Wrap-up

Drift between planning and implementation quietly corrupts your dashboards. plan-drift fights it with a read-only, fully deterministic, LLM-free design — accuracy and reproducibility first.

Happy to hear any feedback — whether you're deploying it in-house or just trying it out. It's on GitHub.

https://github.com/sunnydachs/plan-drift

This is a personal OSS project with no warranty. If you hit bugs or have suggestions, GitHub issues are the best way to reach me.

Your README code examples are silently lying — I built a CLI to detect documentation drift using only AST, no LLM

sunnydachs — Wed, 16 Sep 2026 23:58:15 +0000

Your README code examples are silently lying — I built a CLI to detect documentation drift using only AST, no LLM

"Did you know? This README example doesn't work."

Have you ever had a junior engineer tell you that? Or worse — have you added a line to your code, forgot the docs, and broke it yourself?

This time I want to tell you about doc-drift, a CLI I built that detects this "documentation drift" — the silent gap between code examples in your Markdown and the actual codebase — without using an LLM at all.

https://github.com/sunnydachs/doc-drift

What it does

You point it at a repository. It scans your Markdown files for fenced code blocks, extracts the functions and classes defined there, and checks whether they still match the real codebase.

# scan the repo (read-only)
doc-drift

# another repo, machine-readable output
doc-drift /path/to/repo --json

Example output:

doc-drift — scanned /home/dev/myproject
  markdown files: 42 | python blocks checked: 87
README.md:153  ⚠️ SIGNATURE DRIFT
    block defines (url, retries) but repository defines (url, timeout)
summary: {"signature_drift": 3, "missing": 1, "unparseable": 5} | clean blocks: 41

It reports three main findings:

SIGNATURE DRIFT — the documented function exists, but its arguments have diverged from the real code.
MISSING — the documented function/class no longer exists anywhere in the repo (renamed? deleted?).
UNPARSEABLE — the block isn't valid Python (pseudo-code, placeholders) — informational only.

A "function exists, but arguments differ" bug is one that IDEs and static analysis can never catch. Docs aren't executed — so nothing in the toolchain ever checks them. That's the hole.

Why deliberately not use an LLM

This was the core design decision. An AI reading the natural-language docs could grasp more "context." But I deliberately built it on nothing but Python's standard ast (Abstract Syntax Tree). Three reasons:

Deterministic. Same input → same output, every time. I don't want a tool in CI whose verdict flips between runs.
Read-only and safe. It never imports or executes your code — it compares at the syntax-tree level. Your files stay untouched.
Zero dependencies. Python 3.11+ is enough. Install is pip install git+...

This is one answer to the question I keep asking myself when automating: how much should I hand to an AI? Deterministic work deserves deterministic tools.

The rule: docs may omit, but they must never invent

The detail I obsessed over: code examples in docs are almost always simplified versus the real code. Omitting an argument should be fine. So the rule is:

Omit arguments — allowed.
Invent arguments or functions that don't exist in the code — forbidden.

That single line ("省略は許す、発明は許さない" / omit is fine, inventing is not) eliminated the vast majority of false positives. Classes work the same way: every documented method must exist on the real class, but unmentioned methods are treated as omitted.

Testing against real-world data

Not a toy README — a real, large open-source repository:

Scanned 1,692 Markdown files / 4,451 code blocks.
Found one genuine drift: a function documented with 2 arguments whose implementation had since moved to 1 argument.
Also surfaced our own over-eager default exclusion that caused false positives — so we fixed the configuration.

The real win wasn't just "it works" — it was confirming in practice that the tool does not degrade into a noisy false-positive spammer.

Honest limitations

It can't tell "documentation" from "illustration." A hypothetical example in a top-level README (code never meant to exist) reports as MISSING. It shines in docs/ trees whose examples mirror the codebase.
Python only (for now). Other languages are counted but not checked.
Name-based comparison. Default values and type annotations are ignored — it catches renames, removals, and arity/name changes, not semantic drift.

Wrap-up

Docs-vs-code drift only gets more dangerous as a project grows. And these "quiet but sharp-toothed" problems are exactly where deterministic tools earn their keep.

This is one in a series I'm building on the same principles (deterministic / read-only / dry-run-first). A sibling, plan-drift, applies the same approach to analytics tracking plans.

https://github.com/sunnydachs/doc-drift

Feedback, and especially false-positive reports from your own repos, are the most useful thing you can give me — drop them in the repo issues.

Audit your Excel pivot tables before they break — a read-only CLI with zero dependencies

sunnydachs — Sun, 13 Sep 2026 10:20:56 +0000

If you work with Excel workbooks that have multiple pivot tables, you know the drill: a refresh breaks, Excel throws "A PivotTable report cannot overlap another PivotTable report", and nobody can say which pivot's configuration caused it.

pivot-diag is a CLI that audits the pivot table configurations inside a workbook before things break. It reads the OOXML structure directly (standard library only — zero dependencies), and reports:

⚠️ OVERLAPPING LOCATIONS — two pivot table placements intersect on the same worksheet (the zone where Excel raises the "cannot overlap" error)
⚠️ OVERLAPPING SOURCES — source ranges of multiple pivots intersect on the same sheet (double counting, refresh-order issues)
· SHARED SOURCE — pivots sharing the exact same source range (informational — fine if intentional)
❌ MISSING SOURCE SHEET — a cacheSource points to a sheet that no longer exists (renamed or deleted)
· informational — whole-column refs (A:E), named-range sources, external sources

Everything is deterministic: plain zipfile + ElementTree parsing of the OOXML parts, no LLM, no network. The tool is read-only — it never modifies your files.

How it works

xlsx/xlsm files are OOXML zips. Pivot definitions live in two kinds of parts: xl/pivotCache/pivotCacheDefinitionN.xml (source ranges) and xl/pivotTables/pivotTableN.xml (placement and cache references). pivot-diag parses these parts directly with zipfile + ElementTree.

Deliberately not via openpyxl — pivot table reading is one of the areas where library implementation details leak into your results. Parsing the XML parts directly means the tool doesn't depend on a third-party reader's quirks, and the zero-dependency install is a nice bonus.

Check	Rule
OVERLAPPING LOCATIONS	two placement refs intersect on the same worksheet
OVERLAPPING SOURCES	source ranges intersect on the same sheet (identical ranges → SHARED SOURCE)
MISSING SOURCE SHEET	cacheSource references a sheet that no longer exists
UNPARSEABLE REF	whole-column refs (`A:E`) and similar — informational, never guessed

What I learned from testing it on a real workbook

I built a test workbook generator that assembles OOXML zips directly (no Excel needed) and validated three scenarios:

Clean workbook — two pivots with independent source ranges → no findings. Correct no-drift verdict.
Overlapping source ranges — two pivots whose source ranges intersect on column C → both OVERLAPPING LOCATIONS and OVERLAPPING SOURCES detected.
Broken link — renamed the source sheet in the workbook XML → MISSING SOURCE SHEET detected.

Scenario 3 is the one that matters most in practice: a pivot that references a renamed or deleted sheet is a landmine that only explodes when someone clicks "Refresh". Finding it before that is the whole point.

Known limitations

Read-only. No repair, no relocation — the report points at configurations for you to fix.
Property-level (pivot field) validation is out of scope; the granularity is placement, source range, and cache references.
.xls (legacy format) is not supported — OOXML only.
Whole-column source refs (A:E) and named-range sources are reported as informational (not parsed).

Design principle: read-only, always

Diagnostic tools that modify files are a new risk vector. pivot-diag opens the workbook (reads the zip), parses the pivot parts, closes it, and prints a report. That's the entire interaction. If something breaks, your workbook is exactly as it was before.

Links

Repository: https://github.com/sunnydachs/pivot-diag
This is part of a small family of consistency checkers: doc-drift (docs vs code) and plan-drift (tracking plan vs implementation).

*As this is an independently developed open-source project, its operation is not guaranteed. Please use it at your own risk. I would appreciate it if you could report any bugs or suggest improvements via issues.

License

MIT

I automated Dependabot PR cleanup — and drew a hard line on what NOT to automate

sunnydachs — Sat, 05 Sep 2026 13:30:12 +0000

If you use GitHub, you know the rhythm: Dependabot opens a PR, CI runs green, and then a human has to answer the same questions again — is this a patch or a major? did anyone touch source code? does it conflict?

I got tired of answering those questions by hand, so I built dep-triage: a CLI that sorts open Dependabot PRs into five buckets according to a policy file you commit to your repository.

🔗 https://github.com/sunnydachs/dep-triage

🟢 auto-merge — patch/minor bumps, green CI, dependency-only changes, no conflicts
🔔 escalate — majors, failed CI, conflicts. Reported, never changed
🗑️ close — PRs outrun by a newer-version PR for the same package
💬 rebase suggestion — stale PRs get a @dependabot rebase comment
⏭️ skip — CI still running, or the diff touches non-dependency files

Dry-run is the default. Decisions are fully deterministic — there is no LLM anywhere in the loop. This post is about the line I drew between what to automate and what never to automate, and the three traps real-world data exposed.

1. Only dependency-only diffs are candidates

"CI is green" does not mean "the change is safe" — CI can only check what it knows about. So the mechanical gate is: every changed file must be a dependency manifest or lockfile. One stray source file and the PR is out of scope, no matter how nice its diff looks.

def scope_check(paths: list) -> dict:
    if not paths:
        return {"dependency_only": False, "offending": []}
    offending = [p for p in paths if not is_dependency_file(p)]
    return {"dependency_only": not offending, "offending": offending}

The check is dumb on purpose — a path list against a known set of manifests and lockfiles. No cleverness, no exceptions to reason about.

2. The world changes between judging and acting

Triage says "auto-merge this one". But between that judgment and the API call that enables auto-merge, a new commit can land, or CI can flip red. So --apply re-fetches the head SHA, CI state, and changed files immediately before enabling auto-merge, and aborts to escalate if anything moved:

if action == "auto_merge":
    fresh = api.get(f"/repos/{repo}/pulls/{num}") or {}
    if fresh.get("head", {}).get("sha") != r["facts"]["head_sha"]:
        record["action"] = "escalate"   # head SHA changed since triage
        ...
    ci = api.ci_state(repo, fresh["head"]["sha"])
    if not ci["ci_green"] or ci["ci_pending"]:
        record["action"] = "escalate"   # CI state changed before merge
        ...

Classic TOCTOU (time-of-check to time-of-use), but it's the difference between "safe to put in cron" and "safe only while you watch it".

3. Majors are never auto-merged. Ever.

Even if your policy listed major as allowed, the never-list wins. And titles that don't parse are escalated, not guessed:

bump = facts.get("bump") or "unknown"
if bump in policy["never_auto_merge_bumps"]:
    reasons.append(f"{bump} bump is never auto-merged")
    return _out("escalate", reasons)

The scariest failure mode of automation is behavior that is "usually smart, occasionally dangerous". The dangerous side is capped by construction, not by config discipline.

Real data broke it three times

I ran the tool (dry-run) against a real public repository with 11 open Dependabot PRs. Three problems surfaced that unit tests could never catch:

1. GitHub's "no CI" trap. A commit with no CI checks at all comes back from the combined-status endpoint as state=pending with statuses=0. So "no CI" and "CI running" are indistinguishable in the raw response — my first run classified all 11 PRs as "CI still running". Fix: an empty pending status means "nothing to wait for", and that fact (ci_none) is surfaced in the reasons.

2. Conventional-commit prefixes. Real Dependabot titles look like "chore(deps): bump @nestjs/core from 11.2.1 to 12.0.1" — prefix and lowercase. My ^Bump regex matched nothing. Every PR came back "unknown level". Fix: accept the prefix, case-insensitively.

3. A small propagation miss. The ci_none flag wasn't carried into the decision facts, so the reason line never showed it. Cosmetic, but transparency matters when a bot touches your repo.

After the fixes, the same 11 PRs triaged like this:

summary: {"auto_merge": 6, "comment_rebase": 1, "escalate": 4}

The six auto-merge candidates were all patch/minor. Three of the four escalations were real major upgrades — @nestjs/core 11→12, @types/node 24→26 — and the never-merge rule caught every one of them. That was the moment I stopped worrying about running this unattended. (The last one was a 14-day-old requirements range bump, routed to a rebase suggestion.)

Same lesson as always: tests were green; the real world was a different dataset.

Using it

pip install git+https://github.com/sunnydachs/dep-triage.git

dep-triage --repo owner/name            # dry-run: print the plan, change nothing
dep-triage --repo owner/name --apply    # perform actions (needs write token)

Policy is one TOML file:

auto_merge_bumps = ["patch", "minor"]   # levels eligible for auto-merge
never_auto_merge_bumps = ["major"]      # always wins over the list above
require_ci_green = true
require_dependency_only = true
close_superseded = true
rebase_stale_days = 7
merge_method = "squash"

Auth is a GITHUB_TOKEN env var; dry-run works with read access only.

※If Dependabot triage is eating your review time too, give it a spin — and tell me where the line should be drawn differently.