DEV Community: Sho Naka

I Used Claude Code Heavily. Then Codex Changed How I Hand Off Work.

Sho Naka — Wed, 12 Aug 2026 09:47:31 +0000

I used to hand work to Codex the moment Git, a CLI, or code entered the conversation. I thought that would make execution faster.

What grew was not execution speed. It was the number of handoffs: frame the requirement in ChatGPT, inspect the repository in Codex, return to ChatGPT to change an assumption, then send it back to Codex. A round trip that brings back a new fact from the real environment is valuable. I was also moving questions that could still be decided without running anything.

TL;DR: I stopped routing work by whether it looked “technical.” While the next decision can be made safely from the current context, I let AI do the foundation work too: research, comparison, and structure. When the answer moves into the real repository, Git history, CLI, tests, or browser, I hand Codex an end-to-end execution unit using five fields.

The problem was my routing rule, not Codex

I have used Anthropic's Claude Code as more than code completion. It can read a codebase, edit files, and run commands. I gave it research, design, implementation, tests, and Git work as one long unit.

Claude Code Hooks were an important part of that workflow. A Hook is a user-defined command that runs at a particular lifecycle event in Claude Code. It can format after an edit, stop before a dangerous command, or start a verification step. In practical terms, Hooks let me turn repeated instructions into repeatable stopping points.

I still keep the operating idea I learned there: once I give an AI a well-defined unit of work, I want it to finish the investigation, decisions, execution, and verification.

When I started using the Codex app seriously, I tried to apply the same idea across ChatGPT and Codex. But I used a rough rule for where to cut the work:

If it becomes technical, move it to Codex.

That sounds natural, but “technical” is not an execution requirement. Defining a reader, comparing existing approaches, choosing constraints, and writing a done condition can all include technical language. None of those decisions necessarily needs the real repository yet.

I was confusing the topic of the work with the evidence the work required.

What a handoff actually carries

By handoff, I mean moving work from ChatGPT to Codex, or from one AI task to another. It is not just copying a prompt. It is transferring enough information for the next worker to continue the same decision.

That information is context. In AI systems, context is the information visible to the model when it decides its next response or action. Operationally, I treat context as a handoff packet: the goal, current state, decisions already made, fixed constraints, next action, and evidence that will prove success.

In my workflow, I rebuilt that packet for each move. If the decisions are still unstable, the next AI may research the same question again or act on a different assumption.

In an earlier Zenn article, I wrote about delivering important context reliably after a handoff without making the injected packet too large. That problem still matters. But there is a cheaper question to ask first:

Does this handoff need to happen yet?

Route by where the next answer lives

My current routing rule is:

If the next decision can be made safely from known goals and constraints, continue in the current surface. If getting the answer requires touching the real environment, hand it to Codex.

“Continue in the current surface” does not mean I write everything myself. I am moving foundation work to AI to test how much of my own load it can remove.

Research and compare existing information
Frame the reader's or user's problem
Propose scope and non-scope
Build an outline and first draft
Propose done conditions and verification methods

The human keeps a different class of decisions: whose problem matters, what must not change, where the publication, deletion, spending, or other irreversible boundaries are, and what observed result should count as success.

This is not a compromise between dumping everything on AI and reviewing everything as a human. It is an operating design that gives the human and AI different kinds of ownership.

When the real environment holds the next answer, I move the work to Codex.

Next question	Where the answer lives
Whose problem should this change solve?	Current requirements and human judgment
Does a similar implementation already exist?	The real repository and Git history
Does this command work in the current environment?	CLI output
Did the change break existing behavior?	Tests, lint, and build
Can a reader use the actual page?	The real browser surface

The boundary is not between thinking and building. It is between deciding from available information and collecting evidence from the real artifact.

Use a five-field handoff

Once the boundary is clear, I do not need a long specification. This is the smallest packet I now try to fix before Codex starts:

goal: Who has what problem, and what changes for them?
constraints: Which decisions are fixed? What must not change? Where are the publish, delete, and spend gates?
execution: What is the current state? What remains to inspect or change in the real environment?
done_when: What observable state means the work is complete?
verify_with: Which command, screen, URL, test, or diff proves it?

This preserves implementation freedom while fixing the semantic boundaries.

For example, suppose the goal is “make a Zenn article reviewable.” Before I ask Codex to open a repository, AI can research candidate readers, check overlap with existing articles, propose the claim, build an outline, and define what the cover image needs to communicate.

The human decides whether the claim matches lived experience, what is safe to publish, and what “reviewable” means. Codex can then find the canonical files, place the article and image, run Zenn Preview, and check the table of contents, code blocks, links, image, and frontmatter on the real rendered page.

The five fields preserve technical choice while giving Codex enough meaning to act autonomously.

Ask for evidence, not routine decisions

After the handoff, I do not want Codex to ask me about every small reversible choice. I want it to own this loop:

Inspect -> Decide -> Change -> Test -> Repair -> Verify the real artifact

A return trip from Codex to the planning surface is valuable when execution produced a new fact:

The API we expected does not exist
A test disproved a requirement assumption
The real UI is technically correct but hard to read
An existing implementation makes new code unnecessary

Those handoffs are worth their cost because they carry evidence that was unavailable at the desk.

Reversible choices such as “should this variable be named A or B?” or “which of these equivalent libraries should we use?” usually are not worth a return trip. I am testing whether transferring ownership of the execution loop reduces the human load too.

What I cannot claim yet

I cannot say this boundary reduced my handoff count or completion time by a particular percentage. I have not measured that yet.

What I can claim today is improved explainability. When I stabilize decisions until the real environment is needed, it becomes easier to state what I am handing to Codex, why it is moving now, and what evidence will finish it.

To test the method, I plan to measure:

Handoffs per unit of work
Premises rebuilt after a handoff
Clarification requests returned to the human
Time from execution start to a verified artifact

I may change the boundary again in a few weeks. That would not make this a failure. It would mean the operating idea became a measurable hypothesis.

Try it on your next task

On your next AI-assisted task, stop before writing “technical work.” Fill the five fields, then ask one question:

Can the next decision be made safely without the real repository, CLI, test result, or browser?

If yes, let AI continue the foundation work in the current surface.

If no, move the work to the environment that has the answer and hand over execution and verification together.

The important idea I kept from Claude Code was to let AI own a long unit of work. What I changed for Codex was where that autonomous unit begins.

My current hypothesis is that handing off only the code leaves too much coordination with the human. So I am testing a broader split: move the foundation work to AI, keep the semantic boundaries human-owned, and move execution when the real environment has the answer.

What signal currently tells you to hand work to another agent?

This article was adapted from my Japanese Zenn article with AI assistance for English restructuring, copy editing, and cover art. The workflow described here is my own; factual claims were checked against the source article and official product documentation.

My AI Answered in 5.8 Seconds and Said Nothing Useful. I Almost Blamed the Model.

Sho Naka — Sat, 08 Aug 2026 09:42:56 +0000

I put an AI into a Google Meet call. It transcribed Japanese, generated a reply, and spoke it
out loud. Total new spend: $0.

Then I asked it the one question I actually needed answered, and it said:
"I think there's still room for discussion. How about we set up a session to align our understanding?"

That is exactly what a person says when they don't know.

TL;DR: I had a latency problem and an "is this model smart enough" problem. Neither was real.
Same model, same question, 5.80s → 5.68s — 2,545 characters of context turned a deflection into
a claim you could argue with.

The stack, and what it replaced

I wanted an AI participant in a real meeting. Not a note-taker — something that answers when
someone demands specifics.

The obvious stack bills you three times: a hosted meeting-bot API, a speech-to-text vendor, and
a text-to-speech vendor. I replaced all three.

Layer	Obvious choice	What I used	Why
Meeting bot	Recall.ai, $0.50/hour	Attendee (OSS, self-hosted)	no per-hour billing
Speech-to-text	Deepgram / AssemblyAI	Google Meet's own captions	the meeting already generates them
Text-to-speech	Google Cloud TTS	raw audio POST (below)	no GCP project at all
Reasoning + voice	LLM + TTS, two hops	Gemini Live (speech-to-speech)	one model, one hop

Attendee is 699 stars, last pushed 2026-08-07.
Google Meet exposes no bot API, so it drives a full Chrome instance — which is why setup hurt
before anything else did.

The setup tax, compressed

Two problems were routine. The image pins FROM --platform=linux/amd64 ubuntu:22.04, and my
machine is Apple Silicon, so colima with Rosetta:

colima start --vm-type=vz --vz-rosetta --cpu 6 --memory 12 --disk 60
docker run --rm --platform=linux/amd64 alpine:3.20 uname -m   # x86_64, 5.6s cold

Then the build died at step 35 of 42 with the --chmod option requires BuildKit — colima's
docker CLI ships without the buildx plugin. brew install docker-buildx, point
~/.docker/config.json at /opt/homebrew/lib/docker/cli-plugins via cliPluginsExtraDirs,
rebuild. docker images then reported three images at 5.91 GB each.

The third problem is the one worth your time, because it produced no error at all:

/usr/bin/python: can't open file '/attendee/init_env.py': [Errno 2] No such file or directory

The file existed on the host. dev.docker-compose.yaml mounted .:/attendee. Both true — and
the mount was still empty, because colima only mounts $HOME, and my checkout was under
/private/tmp/.... No warning, no failed-mount message, just an empty directory that looked
exactly like a correctly-mounted one.

Moving the checkout under $HOME fixed it. If a bind mount looks empty inside a colima
container, check the VM's mount list before you check anything else.

Making it speak without a TTS vendor

Attendee's documented endpoint for speaking is /speech, and the source told me it was a dead
end: "Check for Google TTS credentials. This is currently the only supported text-to-speech
provider." That meant a GCP project, a service account, and a billing profile — for something
I'd already solved elsewhere.

But the URL list had another entry, and it doesn't care where the bytes came from:

subprocess.run(["ffmpeg", "-y", "-i", "out.wav",
                "-codec:a", "libmp3lame", "-b:a", "128k", "reply.mp3"], check=True)

requests.post(f"{API}/bots/{bot_id}/output_audio",
              headers={"Authorization": f"Token {key}"},
              json={"type": "audio/mp3",
                    "data": base64.b64encode(open("reply.mp3","rb").read()).decode()})
# 200

MediaBlob.VALID_AUDIO_CONTENT_TYPES is [('audio/mp3', 'MP3 Audio')]. Generate audio
anywhere, convert, POST. The entire Google Cloud dependency left the critical path.

Why my "conversation" was a bot talking to itself

I ran a live loop: poll the transcript, generate a reply, speak it, repeat. Then I read the log.

→ [AI participant] ...how much time has passed, you don't know?
  ← replied in 3.82s
→ [AI participant] I don't know.
  ← replied in 7.07s

The bot's own speech landed in Meet's caption stream, and the loop read it back as new input.

That alone is an obvious bug. The part I didn't expect was the collateral damage. I had a
cooldown so the bot wouldn't talk over people — and the self-loop kept that cooldown permanently
hot, so it started dropping the human's turns:

(skipped: cooldown 3.0s) [Human] I measured that one at about 7 seconds.
(skipped: cooldown 0.0s) [Human] so you'd need it to say "got it, thinking" immediately

The human on the call said "this isn't connecting." They were right, and not for the reason
either of us assumed. The fix is one line — filter transcript rows by speaker_name.

The lesson is not the line. It's that a self-loop does not present as an error. It presents as
poor conversational quality, and you will debug the wrong layer for as long as you trust your
impression of the conversation over the log.

The number that changed my mind

With the loop fixed, latency across seven consecutive turns looked like this:

6.16  3.82  7.07  6.44  5.31  4.84  8.77   (seconds)

Median 6.16s. A single request outside the loop was 1.77s to first audio chunk (110 chunks,
5.32s of speech). The ~4s gap is connection setup, repeated every turn.

So I had a latency problem, and I started planning fixes for it. Then I looked at what the bot
was actually saying, and realized latency was the wrong problem.

I ran a controlled comparison. Same model (gemini-2.5-flash), same question, same prompt
scaffold. One variable: 2,545 characters of context appended to the system instruction —
a short digest of ongoing work, competitor pricing, and our own pricing.

The question was the kind anyone gets asked in a strategy meeting:
"What is our value again — what makes us different from these people?"

	Latency	Shape of the answer
No context	5.80s	A deflection. "There's still room for discussion — how about we set up a session to align our understanding?"
With context	5.68s	A position with specifics: named our pricing model, cited what the competitor comparison had shown, stated where we differ operationally.

Latency was identical. Within noise. The model was the same. The only thing that changed was
what it knew.

The first answer is what you say to buy time. The second can be argued with — someone can check
the pricing claim, disagree with the positioning, push back on the comparison. That is the
difference between an assistant and a stalling tactic.

Note the shape of the failure: ask a deep question, get a shallow answer. Not a wrong answer —
a smooth one, with nothing in it. That's the tell. It reads as "the model isn't good enough at
reasoning" when it's actually "the model has nothing to reason over."

The rule I extracted

Measure what your agent knows before you upgrade what it thinks with.

The reflex when an agent underperforms is to reach for a bigger model. That reflex has a cost —
more money, more latency, sometimes a migration — and in my case it would have bought nothing,
because the agent was reasoning from an empty context and doing that competently.

A cheap test: take one question your agent handled badly. Paste the context a human colleague
would have had in their head when answering it. Re-run with the same model. If the answer
transforms, your problem was never the model.

Which also means the build work isn't model selection. It's the layer that decides what your
agent knows at the moment it's asked — and that layer has its own problems, starting with what
you're allowed to put in it.

The free tier has a price, and it isn't money

From Gemini's pricing page, on the Live API audio model row:

Used to improve our products: Yes (free tier) / No (paid tier)

For meetings containing customer names, that's a real decision. So the context layer needs a
filter in front of it: a deterministic masker — no model judgment, just a lookup table.

Here's the part that surprised me. My hand-built table of 82 names, generated from canonical
company records, scored 1 replacement across 86 transcripts — and it was a false positive.

Speech transcripts don't contain canonical company names. They contain what people say, and
speech recognition mangles proper nouns. One person's surname never appeared in its correct form
in a 36,240-character transcript.

Switching to an existing alias registry — built for a different purpose, listing the terms people
actually use out loud — changed everything:

Table	Replacements	Files with ≥1 hit (of 86)
Hand-built, 82 canonical names	1 (false positive)	—
Registry `strong` terms	3,324	41 (47%)
`strong` + a dedicated `mask` field	5,071	58 (67%)

Measured across 1,728,094 characters. If you're redacting speech, build your table from spoken
forms, not from your database.

A note on running this in Japanese

I expected the language to be the hard part. It wasn't — not where I expected it.

What worked: Gemini Live took language_code: "ja-JP" and produced speech a native speaker
in the call judged natural on the first attempt. Google Meet's own captions
(google_meet_language: "ja-JP") transcribed conversational Japanese cleanly enough to feed
straight into the model. Neither needed tuning.

What broke: proper nouns, and only proper nouns — the same failure the masking numbers above
already show. If your pipeline keys on names (redaction, entity linking, CRM lookup), that's
where the work is, and it's a per-name problem you close by growing a table, not a model problem
you fix by upgrading.

One structural difference if you've only done this in English: Japanese has no spaces between
words, so substring matching has no natural boundaries. My masker replaced a fragment inside an
unrelated word on its first run, because two characters of a company name happened to sit inside
a common noun. In English a \b word boundary catches that for free. Here you need explicit
negative lookarounds per term, or a real tokenizer.

So: viable, and cheaper than I expected. Budget your effort for the name layer, not the speech
layer.

Try it yourself

You need about an hour and no paid accounts.

brew install colima docker docker-compose docker-buildx, start colima with --vm-type=vz --vz-rosetta, and verify docker run --platform=linux/amd64 alpine uname -m prints x86_64 before building anything.
Clone Attendee under $HOME, then follow its dev flow: docker compose -f dev.docker-compose.yaml build, generate .env via init_env.py, up, migrate, create an account at localhost:8000, and confirm it with the link printed to the server log. The API key comes from the UI afterward.
Join a meeting with POST /api/v1/bots, then push any mp3 through /output_audio and confirm you hear it. Do this before writing any agent logic; it isolates the plumbing from your code.
Now the real test: take one question your agent answered badly. Add the context a colleague would have had. Re-run with the same model.

If step 4 changes the answer, you know where your next week goes.

What did your version of step 4 tell you? I'd like to know whether the gap shows up outside my
setup.

This article was drafted with AI assistance. All measurements in it were produced on my own
machine, and responsibility for their accuracy rests with me.

My First Day With OpenTelemetry: Spans That Arrive but Measure Nothing

Sho Naka — Fri, 07 Aug 2026 15:00:36 +0000

How this piece came together. A Zenn writing contest put OpenTelemetry back on my radar, and this is the record of learning it by doing — picking it up, hitting confusion, and working through the side-questions with an AI while writing it up. Every command, number, and output shown below was run and confirmed on my own machine.

I touched OpenTelemetry for the first time today. I knew the name; I'd never installed it. This is the record of adding instrumentation to a piece of my own code and seeing what actually showed up.

The short version: the first version of my instrumentation didn't work. All 15 spans arrived cleanly, with the right names and the right attributes, and they lined up in the Jaeger UI exactly as expected. I was about to call it done — until I looked at the actual numbers. The parent span read 1,006 ms. The 15 child spans summed to 0.085 ms.

The missing ~1,000 ms wasn't recorded anywhere. No error, either.

The cause was how I'd written it. I ran the work, then built spans while reading the results that came back. The child spans were timing "how fast the loop that reads results runs," not the checks themselves.

I rewrote the same logic to wrap the actual execution instead, and the same 15 children summed to 1,183 ms. Span count, names, attributes — none of that changed.

What follows is how I got there, and then — after I thought I had a way to tell a "healthy" trace from a broken one — how I broke that judgment method too, and fixed it. Some of this will be obvious to anyone who already knows this stuff; I'm writing it down as a record of what a first-timer actually gets wrong.

Environment: A Docker-less Local Setup

I don't have Docker, podman, or colima installed. Most OpenTelemetry getting-started guides assume docker compose, but both the Collector and Jaeger ship as plain binaries, so this works without any of that.

$ curl -sSL -o jaeger.tar.gz \
    https://github.com/jaegertracing/jaeger/releases/download/v2.20.0/jaeger-2.20.0-darwin-arm64.tar.gz
$ tar xzf jaeger.tar.gz && ./jaeger-2.20.0-darwin-arm64/jaeger

Jaeger v2 has a built-in OTLP receiver, so it listens on port 4317 and serves the UI on 16686.

$ curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:16686/
200

The sending side is Python, with two dependencies.

$ uv add opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc

This is a setup for local experimentation, not a recommended production configuration. In production the default path is OTLP → Collector → backend. Putting a Collector in front gets you batching, retries, sampling, and the ability to swap destinations without the app ever knowing where the data actually goes. None of that is needed for what this article covers, so I skipped it — but please don't read "Jaeger alone was enough for me today" as "Jaeger alone is enough."

What I Measured

A pre-publish check for blog articles. Before an article goes out, it runs checks — topic classification, leaked sensitive information, tag rules, posting-interval limits, and so on — and stops publication if anything trips. It's Python, and calling run_gates(platform, article_path) returns a list of check results.

The Bad Example: Building Spans After the Result Comes Back

Here's what I wrote first.

with tracer.start_as_current_span("publish gate") as root:
    result = run_gates("qiita", article)          # everything actually runs here
    for c in result.checks:                        # this just reads the finished results
        with tracer.start_as_current_span(f"check: {c.id}") as s:
            s.set_attribute("outcome", c.outcome)

It looks reasonable at a glance. There's a span per check, and each one carries a name and an attribute. And Jaeger did show 15 child spans, right on cue.

But run_gates is completely finished by line 2. Everything from line 3 down is just reading results that already exist. What the span wraps is "read one result out of a list," not the check that produced it.

bad example: spans built after the fact
  parent: 1006.0 ms / sum of 15 children: 0.085 ms
    0.046 ms  check: topic_class_qiita
    0.006 ms  check: sensitive_identifiers_qiita
    0.004 ms  check: unsourced_experience_qiita

Every child comes in under 0.05 ms, and the parent is a full second. Roughly 1,006 ms isn't recorded in any span at all.

What makes this dangerous is that nothing throws an error. The SDK exports fine, Jaeger renders fine, and the trace looks complete on the screen.

The Good Example: Wrapping the Actual Execution

The fix is to start the span inside the function that actually runs, and keep it open until that function returns. For code I own, the natural way to do that is a decorator.

def traced(name=None):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*a, **k):
            with tracer.start_as_current_span(name or fn.__name__) as s:
                try:
                    return fn(*a, **k)
                except Exception as e:
                    s.record_exception(e)
                    s.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
                    raise
        return wrapper
    return deco

@traced("check: authority verification")
def check_authority(): ...

Passing exceptions through record_exception and set_status, instead of swallowing them, is part of what makes this a workable minimum. Measuring three checks with this shape, the children summed to 100.0% of the parent.

This particular task came with the constraint "don't touch the production code," so instead I swapped in a wrapped version of the function that runs a single check, at runtime, just for the measurement.

orig = gate_runner._run_check

def wrapped(entry, *a, **k):
    with tracer.start_as_current_span(f"check: {entry.get('id')}") as s:
        r = orig(entry, *a, **k)                   # execution happens inside the span
        s.set_attribute("outcome", str(r.outcome))
        return r

gate_runner._run_check = wrapped
run_gates("qiita", article)
gate_runner._run_check = orig

This kind of runtime swap is a temporary measurement device, not something to leave in place. Its blast radius is hard to reason about, it's easy to forget to restore, and it breaks if something else swaps the same function mid-flight under concurrency. For ongoing measurement, add a decorator instead — or, if the target is a well-known library, lean on automatic instrumentation via opentelemetry-instrument. That said, automatic instrumentation only covers libraries it already has support for; your own functions aren't covered, so those still need manual instrumentation either way.

Re-measuring the same article, the same checks, the same span names, gives this:

good example: wrapping the execution
  parent: 1221.2 ms / sum of 15 children: 1183.058 ms
    644.6 ms  check: publish_authority_qiita
    538.4 ms  check: qiita_pre_publish_check_cli
      0.015 ms  check: topic_class_qiita

Parent at 1,221 ms against a child sum of 1,183 ms. The 38 ms gap is explainable — result assembly and similar overhead — and stays in a reasonable range.

And for the first time, the contents were actually visible. Of that roughly one second, 644 ms was an authority check for publishing, and 538 ms was a call into an existing check script. Together those two account for 97% of the total — a number nobody had before instrumentation existed.

Judgment Method: My First Attempt Broke Under Parallelism

Both traces have the same span count, the same names, the same attributes. Side by side in the Jaeger UI, unless you already know what you're looking for, you won't spot the difference. I wanted a number to judge by.

My first idea was: sum the child span durations, divide by the parent. On the two examples above, that separates cleanly.

bad example: spans built after the fact   0.0%
good example: wrapping the execution      96.9%

I thought that settled it — until I ran it on parallel execution, and it broke.

async def check(name, sec):
    with tracer.start_as_current_span(f"check: {name}"):
        await asyncio.sleep(sec)

with tracer.start_as_current_span("run three checks in parallel"):
    await asyncio.gather(check("A", 0.3), check("B", 0.3), check("C", 0.3))

parent 301 ms / sum of 3 children 903 ms
naive sum ÷ parent = 299.8%

With three things running at once, a plain sum comes out to roughly three times wall-clock time. Building the metric from a single sequential example was the mistake.

The fix is to collapse overlapping intervals before comparing. Sort the intervals and merge them — no new dependency needed.

def union_us(spans, parent):
    """Sum the child spans' occupied intervals, with overlaps removed (microseconds).

    Clips to the parent's own time window — otherwise a child that spills
    outside the parent can push the result past 100%.
    """
    iv = []
    for s in spans:
        a, b = s["startTime"], s["startTime"] + s["duration"]
        pa, pb = parent["startTime"], parent["startTime"] + parent["duration"]
        a, b = max(a, pa), min(b, pb)
        if b > a:
            iv.append((a, b))
    iv.sort()
    total = 0
    cur_s = cur_e = None
    for s, e in iv:
        if cur_e is None or s > cur_e:
            if cur_e is not None:
                total += cur_e - cur_s
            cur_s, cur_e = s, e
        else:
            cur_e = max(cur_e, e)
    if cur_e is not None:
        total += cur_e - cur_s
    return total

That took a detour through units. I had another model review this article's draft, and it flagged that startTime and duration are nanoseconds by spec, so naming the function union_us (microseconds) and converting to milliseconds was off by a factor of 1,000. It sounded plausible enough that I nearly "fixed" it — but I checked first.

The parent span's duration for three asyncio.sleep(0.3) calls run in parallel was 301311. As microseconds, that's 301.3 ms. As nanoseconds, it would be 0.3 ms. The code actually waits 300 ms, so microseconds is correct — the review comment was the one that was wrong.

When you're new to something and someone who sounds like they know it tells you "that's wrong," the instinct is to just go fix it. But running one case with a known, expected value and checking it against the output settles it in seconds — faster than tracking down the spec. And getting the order of magnitude wrong doesn't leave an obvious trace either: the ratio still looks correct on its face, so nothing points back to the mistake later.

Running all three cases through it:

publish gate (decorator version)     naive sum  100.0% / union 100.0%
good example: wrapping the execution naive sum   96.9% / union  96.9%
running three checks in parallel     naive sum  299.8% / union 100.0%

The union stays within 100% even under parallel execution. Fetching the trace looked like this. The same service name also returns past runs, so remember to take only the most recent one — I skipped that at first and got a "good example" that read 0.0%, because a pre-fix run was still mixed into the results.

data = json.load(urllib.request.urlopen(
    f"http://localhost:16686/api/traces?service={service}&limit=20"))["data"]

def root_of(tr):
    return [s for s in tr["spans"] if not s.get("references")][0]

latest = max(data, key=lambda tr: root_of(tr)["startTime"])
root = root_of(latest)
kids = [s for s in latest["spans"] if s.get("references")]
print(f"union / parent = {union_us(kids, root) / root['duration']:.1%}")

How Far Should I Trust This Metric?

A low number doesn't, by itself, mean "the instrumentation is broken." It also comes out low in these cases:

The parent is carrying I/O wait. If waiting on an external API's response counts toward the parent's time and the child spans sit outside that wait, an uncovered interval remains.
Some work was never turned into a child span. Data transformation, cache checks — anything in between that wasn't instrumented shows up only in the parent's time, never in a child's.
The context broke across an async boundary. Cross a thread or an asyncio boundary without care and the parent-child link doesn't survive, so that work never gets tallied as a child at all.

So this metric isn't "proof the instrumentation is broken." It's closer to a prompt to go find out why, when a number looks off — not an established practice, just something I put together because I ran into this exact problem today. If it comes out near 0% and there's no I/O wait to explain it, the first thing to suspect is how the spans were built. That's exactly what happened to me here.

When You Have to Connect Things After the Fact

There are legitimate designs where a result arrives somewhere else, at some other time — batch jobs, async queues. OpenTelemetry has two mechanisms for this, but they do different jobs.

Span Link: connects a span in another trace in as a reference. Useful for a causal-but-not-parent-child relationship — "the work that got enqueued" and "the work that consumed it," for example.
Event: marks a specific point in time inside a span. Records something that happened along the way.

The thing to watch for: neither one stands in for measuring duration. An Event is a point, not an interval, so it can't be used to split a duration after the fact. If you want to measure time, you have to put a span where the work actually executes. A Link is for reconnecting causality after the fact — reconnecting the causal chain doesn't reconstruct the timing.

What This Article Doesn't Cover

I only measured a single local process, so this leaves out everything production actually needs:

Sampling. Sending every span overwhelms the backend at scale; head-based vs. tail-based sampling is a real decision to make.
Context propagation. Cross a thread or an asyncio boundary carelessly and contextvars breaks, severing the parent-child relationship.
Running a Collector. Batching, retries, splitting destinations — the production default, and I skipped it entirely here.
Backend persistence and auth. I used a local, unauthenticated Jaeger instance as-is.

Summary

Building spans after processing finishes makes the duration vanish without an error. The spans still arrive, and the names and attributes are still correct.
Wrapping the actual execution surfaces the real numbers. The same target went from 0.085 ms to 1,183 ms without any change in span count, names, or attributes.
For ongoing measurement, a decorator or automatic instrumentation seems to be the right shape. A runtime swap is a temporary investigation tool, not something to keep.
The judgment metric I ended up using is union of child spans ÷ parent. A naive sum hit 299.8% under parallel execution and was unusable.
A low number means "go find the reason," not "it's broken." I/O waits and uninstrumented sections pull it down too.
Link and Event, for connecting things after the fact, exist for causality and points in time — neither one measures duration.

Adding instrumentation makes it easy to relax, just because you "added it." But a span arriving and a span measuring the thing you actually wanted to measure turned out to be two different things. Even the metric I used to judge that difference was wrong on the first pass — built from a single example that didn't generalize.

The Limits of This Article

I wrote this without fully understanding the tool — reading the official docs and other people's write-ups, running things locally to check as I went. I expect there are places here that depart from normal practice.

What I'm least confident about:

The "union of child spans ÷ parent" metric I used for judgment is something I built because I ran into this exact problem today — it isn't a general convention. There may be a more straightforward way to check this.
I only looked at three ways to add instrumentation: a decorator, a runtime swap, and automatic instrumentation. Real-world practice surely has other patterns.
I only measured a single process — sequential, and asyncio parallelism. Threads, multiple processes, and distributed environments are a different story entirely.

If something here is off, I'd like to know. Every number and every piece of code shown was run locally and the output checked against what's printed, so if you reproduce it and get something different, that difference is the interesting part.

References

~/.ssh/config's Permission Check: When OpenSSH Enforces It, and What Changes Between ssh -F and Include

Sho Naka — Thu, 06 Aug 2026 13:28:55 +0000

A draft I was reviewing claimed that OpenSSH's client doesn't enforce permissions on ~/.ssh/config. The evidence was a chmod sweep — 600, 644, 664, 666 — with every run exiting 0. The commands were real. They ran. The output was genuine. The conclusion was backwards: permissions are enforced.

Re-running doesn't catch this one. Run the same commands again and you get exit=0 every time, forever. What was broken wasn't the output — it was the path that produced it. Point -F at a config file directly and OpenSSH's permission check never runs at all. Four runs down a path that skips the check got read as four pieces of evidence that the check doesn't exist.

Even when a command and its output are genuine and reproducible, that still does not show that the test exercised the path it claims to — and that gap survives a re-run. What follows is four checks that caught both failures above, roughly cheapest first: a contradiction check, an existence check, a positive control, and an ablation test. All four run inside ssh and a terminal. This isn't a complete taxonomy of measurement errors — it's the cheap heuristics that happened to work on these two cases; what it doesn't cover comes at the end.

A note on how this was checked. Every command and output below was re-run in a disposable scratch directory immediately before publishing and matched verbatim, with one substitution: the scratch directory's absolute path was replaced with the placeholder /tmp/ssh-demo for readability. The numbers, output, and line order are otherwise the run's own. None of the real ~/.ssh on this machine was touched.
$ command -v ssh
/usr/bin/ssh

$ /usr/bin/ssh -V
OpenSSH_10.2p1, LibreSSL 3.3.6

$ codesign -dv /usr/bin/ssh 2>&1 | grep Identifier
Identifier=com.apple.ssh
Every command below runs this same /usr/bin/ssh. How the default config path — ~/.ssh/config itself — was safely tested, and the limits that testing ran into, are covered in a later section.

Check 1 — `ssh -G myhost` Returns `port 2200`: Two Explanations for the Same Result

The cheapest check needs no terminal at all. Read a single passage for one observed result that gets explained two incompatible ways.

Take this config:

Host myhost
    HostName 127.0.0.1
    User specific-user
    Port 2200

Match host myhost user specific-user
    Port 2300

$ /usr/bin/ssh -F config_match_test -G specific-user@myhost | grep '^port '
port 2200

Match's Port 2300 doesn't win here. A draft I reviewed offered two different reasons for this, in the same section. First: Port 2200 locks in before Match is evaluated, so first-value-wins discards 2300. Second: Match user is checked against the OS login account, and since neither that account nor the Host block's User specific-user has taken effect yet, Match never fires in the first place.

Those can't both be true. First-value-wins is a rule about what happens once Match's condition is already satisfied. If the second explanation is right, the condition was never satisfied, and first-value-wins never got a turn to do anything. If the first explanation is right, Match must have fired — which is exactly what the second explanation denies.

Knowing they can't both be true doesn't say which one is. The single port 2200 above is consistent with either story: Match firing and losing to first-value-wins looks identical, from the outside, to Match never firing at all. Telling them apart needs a config built so the two predictions come apart.

A narrower config decides which half is wrong:

Host myhost
    HostName 127.0.0.1
    User specific-user

Match user specific-user
    Port 2300

$ /usr/bin/ssh -F config_match_user -G myhost | grep -E '^(user|port) '
user specific-user
port 2300

Match user specific-user succeeds, and the shell running this command is logged in under a different OS account entirely, not specific-user. The only way Match fires here is by reading specific-user back out of the Host block's User line, not the OS account. The second explanation doesn't survive this — for this config.

Calling it flatly wrong would overreach, though. Match user reads a resolved remote-user value, and when nothing has resolved one yet, that value falls back to the OS login name by default. A config with no explicit User line shows the fallback:

Host myhost
    HostName 127.0.0.1

Match user deploy
    Port 2400

$ /usr/bin/ssh -F config_match_localdefault -G deploy@myhost | grep -E '^(user|port) '
user deploy
port 2400

$ /usr/bin/ssh -F config_match_localdefault -G myhost | grep '^port '
port 22

Match fires only when the command line supplies deploy@. Called as plain myhost, the unresolved user value falls back to whoever's logged in locally — not deploy — so Match doesn't fire. The second explanation describes a real fallback path; it's just misapplied to the original config, where the Host block already resolves User to specific-user before Match is evaluated, leaving no unresolved value for a local login name to fall back into.

There's a keyword for exactly the mechanism the second explanation was reaching for: Match localuser matches the OS login account directly, unconditionally. Match user doesn't — it matches a resolved remote-user value that only falls back to the OS login name once nothing else, like a Host block's User line, has already set it. readconf.c's match_cfg_line() recomputes that value fresh every time it starts evaluating a Match line: ruser = options->user == NULL ? pw->pw_name : options->user; — whatever a prior Host block already put in options->user wins; the OS login name is only what's left when nothing did. The second explanation wasn't describing a mechanism that doesn't exist; it was describing localuser's behavior and attaching it to user.

That leaves the first explanation standing for the original config — Check 1 can only show that the draft contradicted itself, not settle which half, if either, was actually right in general. Check 4 comes back to close that out.

Nothing about this is specific to ssh_config. The same shape shows up wherever a write-up explains one failure two ways: a CI failure blamed on both "the code regressed" and "the test environment never came up" in the same paragraph, a support ticket blamed on both a caching bug and a race condition. In each case the two stories can't both be the reason, and the write-up that offers both hasn't actually settled which one is.

This check catches a draft that states two competing explanations for the same result in the same place; it does nothing for a draft that's wrong consistently. It costs zero commands, which is exactly why it's the first pass on anything that lands.

Check 2 — `chmod 600/644/664/666` All Exit 0 on `ssh -F`: Looking for `Bad owner or permissions`

A different draft's permissions chapter made a clean claim: SSH's client doesn't enforce permissions on ~/.ssh/config — 600, 644, 664, 666 all pass.

$ for m in 600 644 664 666; do chmod $m config && stat -f '%Lp' config && \
    /usr/bin/ssh -F config -G myhost >/dev/null 2>&1; echo "exit=$?"; done
600
exit=0
644
exit=0
664
exit=0
666
exit=0

(stat runs right after chmod here, rather than chaining chmod && ssh — that way a chmod that silently failed can't produce a false pass.)

The commands are real, they ran, the exit codes are genuine. "Not enforced" is still backwards. What catches this one: before accepting a negative claim — "not enforced," "not required," "not checked" — confirm the code that would do the enforcing exists in the binary at all.

$ strings /usr/bin/ssh | grep "Bad owner"
Bad owner or permissions on %s

It's there. Finding nothing wouldn't have proven the check is absent — it could live in a shared library, or surface as a different message — but finding this string means "not enforced" needs more than the sweep above to stand. This costs one strings call, cheap enough to run before accepting any negative conclusion: absent code kills the claim right here; present code is the cue for Check 3.

Check 3 — `ssh -F` and `Include` Give Different Results: Adding the `chmod 664` Case Into the Measurement

strings only shows that the check exists somewhere in the binary; it can't say which path reaches it. That needs the source, and the caveat from the note at the top applies here too: this is a static reading of openssh-portable upstream, not a trace of what the local binary actually executes.

From ssh.c, process_config_files():

if (config != NULL) {
    if (strcasecmp(config, "none") != 0 &&
        !read_config_file(config, pw, host, host_name, cmd,
        &options,
        SSHCONF_USERCONF | (final_pass ? SSHCONF_FINAL : 0),
        want_final_pass))
        fatal("Can't open user config file %.100s: "
            "%.100s", config, strerror(errno));
} else {
    r = snprintf(buf, sizeof buf, "%s/%s", pw->pw_dir,
        _PATH_SSH_USER_CONFFILE);
    if (r > 0 && (size_t)r < sizeof(buf))
        (void)read_config_file(buf, pw, host, host_name, cmd,
            &options, SSHCONF_CHECKPERM | SSHCONF_USERCONF |
            (final_pass ? SSHCONF_FINAL : 0), want_final_pass);

    /* Read systemwide configuration file after user config. */
    (void)read_config_file(_PATH_HOST_CONFIG_FILE, pw,
        host, host_name, cmd, &options,
        final_pass ? SSHCONF_FINAL : 0, want_final_pass);
}

A file handed to ssh via -F only ever carries SSHCONF_USERCONF. SSHCONF_CHECKPERM — the flag that turns permission checking on — is set only on one call in the else branch, the one that builds and reads the default per-user ~/.ssh/config path; the systemwide config read right below it doesn't carry the flag either. The four-pattern sweep in Check 2 ran entirely on the -F branch, which never sets SSHCONF_CHECKPERM at all; it never had a chance to fail, independent of what the actual permissions were.

One path that does set the flag is Include. From readconf.c, Include handling inside process_config_line_depth():

r = read_config_file_depth(gl.gl_pathv[i],
    pw, host, original_host, remote_command,
    options, flags | SSHCONF_CHECKPERM |
    (oactive ? 0 : SSHCONF_NEVERMATCH),
    activep, want_final_pass, depth + 1);

Every file pulled in through Include gets SSHCONF_CHECKPERM unconditionally. What the flag actually does, further down the same function, read_config_file_depth():

if (flags & SSHCONF_CHECKPERM) {
    struct stat sb;

    if (fstat(fileno(f), &sb) == -1)
        fatal("fstat %s: %s", filename, strerror(errno));
    if (((sb.st_uid != 0 && sb.st_uid != getuid()) ||
        (sb.st_mode & 022) != 0))
        fatal("Bad owner or permissions on %s", filename);
}

Re-run the same permission sweep, but through a path that actually sets the flag: an outer file, held at 600, that Includes an inner file whose permissions vary. Both ssh calls below use -T (it suppresses an unrelated pty warning and doesn't affect the port or permission-check results these checks read) and the same absolute paths for -F and Include. stat runs immediately before each call, so the permission and path actually observed are the ones under test, not assumed:

# outer.conf, permissions 600, fixed
Include /tmp/ssh-demo/inner.conf

# inner.conf, permissions varied 600 -> 644 -> 664 -> 666
Host myhost
    HostName 127.0.0.1
    User specific-user

$ cat perm.sh
set -euo pipefail
D=/tmp/ssh-demo
for m in 600 644 664 666; do
  chmod "$m" "$D/inner.conf"
  stat -f '%d %i %u %Lp %N' "$D/inner.conf"
  /usr/bin/ssh -T -F "$D/inner.conf" -G myhost >/dev/null && s=0 || s=$?
  echo "  direct -F      exit=$s"
  stat -f '%d %i %u %Lp %N' "$D/inner.conf"
  /usr/bin/ssh -T -F "$D/outer.conf" -G myhost >/dev/null && s=0 || s=$?
  echo "  via Include    exit=$s"
done

$ bash perm.sh
16777230 296585647 503 600 /tmp/ssh-demo/inner.conf
  direct -F      exit=0
16777230 296585647 503 600 /tmp/ssh-demo/inner.conf
  via Include    exit=0
16777230 296585647 503 644 /tmp/ssh-demo/inner.conf
  direct -F      exit=0
16777230 296585647 503 644 /tmp/ssh-demo/inner.conf
  via Include    exit=0
16777230 296585647 503 664 /tmp/ssh-demo/inner.conf
  direct -F      exit=0
16777230 296585647 503 664 /tmp/ssh-demo/inner.conf
Bad owner or permissions on /tmp/ssh-demo/inner.conf
  via Include    exit=255
16777230 296585647 503 666 /tmp/ssh-demo/inner.conf
  direct -F      exit=0
16777230 296585647 503 666 /tmp/ssh-demo/inner.conf
Bad owner or permissions on /tmp/ssh-demo/inner.conf
  via Include    exit=255

stat's columns, in order: device, inode, owner uid, permissions, path. Across the 8 stat calls immediately preceding each ssh run, device and inode stayed constant and the owner uid matched this account's. That's as far as the observation goes — whether the file could have changed between stat and ssh's own open(), or whether both paths' open file descriptors actually resolved to that same device/inode, wasn't checked.

The only edit made to this transcript for publication was substituting the working directory's absolute path with /tmp/ssh-demo, as noted at the top. The numbers, output, and order are the run's own.

Same file, same permissions, only the path differs. -F direct passes all four permission levels; Include fails at 664 and 666 with the exact string Check 2 found sitting in the binary. Because 664 and 666 disagree between the two paths on files that are otherwise identical, path is what's gating the check — and once a path does trigger it, group- or other-writable is what fails it.

I stopped there. The default path itself, ~/.ssh/config, was never touched — the real ~/.ssh on this machine stayed out of scope for this kind of audit. Redirecting $HOME to a scratch directory looks like a shortcut around that, but it doesn't work: the else branch in ssh.c builds the default path from pw->pw_dir, the account's actual home directory from getpwuid(), and never reads $HOME. That same branch always carries SSHCONF_CHECKPERM regardless of which directory it ends up reading, so the source gives no reason to expect the default path at 664/666 to behave any differently from inner.conf above — but that's a prediction from the source, not a run I have to show for it.

The same pattern shows up outside SSH. In CI, plant a known break in the exact path a test claims to cover and check whether the intended assertion fails for the intended reason; adding one assert false to force red only proves the test runner executes, not that the target path was ever reached. For a health check, take the monitored service down in an isolated environment and check whether the contract's failure signal — non-2xx, connection refused, timeout, whichever applies — actually fires; if it stays green, the thing being watched is a different process or a different environment. Mock-heavy tests invert this: with the real implementation swapped for a mock, breaking that implementation should leave the test green, so the thing to break instead is the mock's return value or the contract behind it, or route through an integration test that exercises the real implementation. Same goal every time: plant a known failure and check it gets caught. If it doesn't, the measurement may never have reached the path it claims to test. A sweep with no case that has to fail can't tell a silent mechanism from an absent one — adding 664/666 through a path that actually sets the flag is what turns "everything passed" into a result that means something.

Redirecting `$HOME` Doesn't Move the Default Path — Measuring the Boundary Through `Include`

The previous section closed by leaving the default path as inference. Here's how far that can actually be pushed.

The obvious next move looked like redirecting $HOME to point the default path at an isolated environment:

$ mkdir -p /tmp/ssh-demo/isolated-home/.ssh
$ cat /tmp/ssh-demo/isolated-home/.ssh/config
Host myhost
    HostName 127.0.0.1
    Port 2200

$ chmod 700 /tmp/ssh-demo/isolated-home/.ssh
$ chmod 600 /tmp/ssh-demo/isolated-home/.ssh/config
$ HOME=/tmp/ssh-demo/isolated-home /usr/bin/ssh -G myhost | grep -E '^(hostname|port) '
hostname myhost
port 22

Neither HostName 127.0.0.1 nor Port 2200 took effect. The file itself isn't broken — pointing -F straight at it reads it fine:

$ HOME=/tmp/ssh-demo/isolated-home /usr/bin/ssh -F /tmp/ssh-demo/isolated-home/.ssh/config -G myhost | grep -E '^(hostname|port) '
hostname 127.0.0.1
port 2200

Drop -F and keep only the $HOME redirect, and it's back to hostname myhost (unsubstituted) and port 22 (the compiled-in default). -vvv shows which file actually got opened:

$ HOME=/tmp/ssh-demo/isolated-home /usr/bin/ssh -vvv -G myhost 2>&1 | grep "Reading configuration data"
debug1: Reading configuration data $REAL_HOME/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config

$REAL_HOME stands in for this machine's actual home directory, substituted for publication; the rest of the line is the run's own. The default path is built from pw_dir, not from $HOME, so redirecting the environment variable never moves it. The source explains why: the default path is assembled from pw->pw_dir, and that pw comes from getpwuid(getuid()), not from getenv("HOME").

/* ssh.c, line 729 */
pw = getpwuid(getuid());

/* ssh.c, lines 589-590, the `config == NULL` else branch */
r = snprintf(buf, sizeof buf, "%s/%s", pw->pw_dir,
    _PATH_SSH_USER_CONFFILE);

Since the default path's construction never reads $HOME, giving it a pw_dir different from the real one takes a different UID — which takes root, which this working environment doesn't have. Touching the real ~/.ssh on this machine was never on the table either. So the literal thing — set the default path to 664 and watch it fail — wasn't measured here.

What the source does confirm directly: reading readconf.c, the default-path branch (the else in ssh.c) and the Include branch both end up in the same read_config_file(), hitting the same check (if (flags & SSHCONF_CHECKPERM) { ... }, testing sb.st_mode & 022). The only difference between the two callers is whether SSHCONF_CHECKPERM gets set before getting there; once it's set, there's exactly one check to pass. The Include measurement above exercises that shared check directly. It isn't a measurement of the default path itself, but it is a measurement of the exact code the default path relies on — a proxy, and one worth keeping distinct from a direct measurement.

Three paths, side by side:

Path	`SSHCONF_CHECKPERM`	How it was tested	`600`	`644`	`664`	`666`
`-F` direct	not set	measured directly, isolated environment	exit 0	exit 0	exit 0	exit 0
via `Include`	set	measured directly, isolated environment	exit 0	exit 0	`Bad owner...` (255)	`Bad owner...` (255)
default path (no `-F`)	set (confirmed from source — same branch as `Include`)	can't be measured directly in this environment; `Include`'s result stands in as a proxy	—	—	—	—

That 664 and 666 both fail is already established from the measurement above. What isn't yet established is which bit does it. Starting from 600 and adding one bit at a time — read bits (040, 004) kept separate from write bits (020, 002) — settles that:

$ cat /tmp/ssh-demo/boundary.sh
D=/tmp/ssh-demo
for m in 600 604 620 640 660 662 664 666; do
  chmod "$m" "$D/boundary_inner.conf"
  /usr/bin/ssh -T -F "$D/boundary_outer.conf" -G myhost >/dev/null 2>&1
  echo "mode=$m exit=$?"
done

$ bash /tmp/ssh-demo/boundary.sh
mode=600 exit=0
mode=604 exit=0
mode=620 exit=255
mode=640 exit=0
mode=660 exit=255
mode=662 exit=255
mode=664 exit=255
mode=666 exit=255

boundary_outer.conf stays fixed at 600 and contains a single line, Include /tmp/ssh-demo/boundary_inner.conf. Only boundary_inner.conf's own permissions moved:

Mode	owner	group	other	group write	other write	Result
`600`	rw-	---	---	no	no	exit 0
`604`	rw-	---	r--	no	no	exit 0
`620`	rw-	-w-	---	yes	no	255
`640`	rw-	r--	---	no	no	exit 0
`660`	rw-	rw-	---	yes	no	255
`662`	rw-	rw-	-w-	yes	yes	255
`664`	rw-	rw-	r--	yes	no	255
`666`	rw-	rw-	rw-	yes	yes	255

Whether a read bit is set (604, 640) makes no difference to the result. One write bit on group or other, and it fails (620 onward) — matching readconf.c's condition, (sb.st_mode & 022) != 0, exactly. 022 in octal is group-write (020) OR'd with other-write (002); the read bits (044) never enter into it.

The earlier Include measurement — outer file pinned at 600, only the inner file's permissions varied — already satisfied this condition. A 600 parent doesn't shield an included file sitting at 664 or 666. The parent's own permissions play no role in the check applied to whatever it Includes.

`Bad owner or permissions`: One Message for Two Different Causes

Bad owner or permissions comes out of a single fatal() call in readconf.c:

if (((sb.st_uid != 0 && sb.st_uid != getuid()) ||
    (sb.st_mode & 022) != 0))
    fatal("Bad owner or permissions on %s", filename);

The two conditions joined by || produce the same one-line message regardless of which one fired. A file owned by neither root nor the current user (st_uid mismatch) and a file with a group- or other-write bit set (st_mode & 022) are indistinguishable from the message alone. Telling them apart takes a separate stat:

$ stat -f '%u %Lp' /tmp/ssh-demo/boundary_inner.conf
503 664

503 matches this account's own id -u, so ownership checks out — the failure here traces to permissions (664's group-write bit), not to a mismatched owner. The owner-mismatch branch — a file that ends up owned by someone else after being copied over shared storage, say — wasn't exercised here, since setting up a second user account was out of reach; that side stays at what the upstream source's condition says, not something measured directly.

The path in the message deserves attention too. filename names whichever specific file was open at the time. In the Include measurement above, the parent outer.conf passes the check at 600, so the error only ever names the included inner.conf. Once a config is split across files, check which file's path the error actually names before touching anything — fixing the parent's permissions does nothing if the real cause sits in the included file.

One path that actually lands on this boundary, confirmed here: with the shell's umask set to 002, a new file created by redirection comes out at 664.

$ umask
022
$ bash -c "umask 002; printf 'Host x\n    Port 22\n' > /tmp/ssh-demo/umask_test_config"
$ stat -f '%Lp' /tmp/ssh-demo/umask_test_config
664

This environment's default umask is 022, which tops out at 644 on its own — 002 is a value someone sets deliberately for group sharing, not something a default-umask environment ever produces by itself. cp behaves differently:

$ bash -c "umask 002; cp /tmp/ssh-demo/src_600.conf /tmp/ssh-demo/umask_test_cp600.conf"
$ stat -f '%Lp' /tmp/ssh-demo/umask_test_cp600.conf
600

cp on this machine duplicates the source file's own permissions and doesn't go through umask at all. What trips the boundary isn't copying a config into place — it's generating one fresh: a shell redirect, an editor save, a template rendering. Running the umask-created file through Include fails for the same reason as the boundary sweep above:

$ /usr/bin/ssh -T -F /tmp/ssh-demo/umask_outer.conf -G x >/dev/null 2>&1; echo exit=$?
exit=255

Check 4 — `Match host` vs `originalhost`: Removing `Port 2200` to See Which One Fired

The last check: delete the value you suspect matters, and check whether the result flips the way the theory predicts — rather than taking one matching run as confirmation.

Back to the contradiction Check 1 left open. What settles it is in the man page:

The criteria for the host keyword are matched against the target hostname, after any substitution by the Hostname or CanonicalizeHostname options. The originalhost keyword matches against the hostname as it was specified on the command-line.

— ssh_config(5), OpenBSD manual pages

host is compared after HostName substitution. The config sets HostName 127.0.0.1, so Match host myhost compares myhost against 127.0.0.1 and never matches. originalhost compares against what was literally typed on the command line, so Match originalhost myhost does match — but matching alone still doesn't hand Port to 2300 while Host's Port 2200 is set first:

Host myhost
    HostName 127.0.0.1
    User specific-user
    Port 2200          # present in the first run, removed in the second

Match originalhost myhost
    Port 2300

$ /usr/bin/ssh -F config_match_originalhost_only -G myhost | grep -E '^(user|port) '
user specific-user
port 2200

$ /usr/bin/ssh -F config_match_fixed -G myhost | grep -E '^(user|port) '
user specific-user
port 2300

Deleting Port 2200 flips the result exactly as predicted. That's still one data point, though, on a config where Match was already confirmed to fire.

The actual ablation test applies the same move to the config that started this section — Match host myhost user specific-user, the one where Match was never established to fire at all. Remove Host's Port 2200, so first-value-wins has nothing left to defend, and leave Match's Port 2300 in place. If Match had ever fired here, this is exactly the config where 2300 should now win by default:

Host myhost
    HostName 127.0.0.1
    User specific-user

Match host myhost user specific-user
    Port 2300

$ /usr/bin/ssh -F config_match_noport -G myhost | grep '^port '
port 22

$ /usr/bin/ssh -F config_match_noport -G specific-user@myhost | grep '^port '
port 22

Both calls land on 22, SSH's compiled-in default — not 2200 (nothing left for first-value-wins to defend), and not 2300 either. Match host myhost never fired, under either way of calling the host, exactly as the man page's substitution rule predicts. Neither of Check 1's explanations was actually the reason: there was no Port conflict to resolve, because Match never matched in the first place.

One grep detail worth flagging:

$ /usr/bin/ssh -F config_match_noport -G myhost | grep port
port 22
gatewayports no

grep port without an anchor also catches gatewayports no — every command above uses grep '^port ' for exactly this reason.

To pin down when Match host does and doesn't fire, and separate that cleanly from the Port conflict, run all four combinations of condition keyword × Port 2200 present/absent, holding everything else fixed — same user specific-user clause, same -G myhost call in all four cells:

$ cat cell.sh
set -euo pipefail
D=/tmp/ssh-demo
for cond in "host myhost user specific-user" "originalhost myhost user specific-user"; do
  for port in 2200 none; do
    { printf 'Host myhost\n    HostName 127.0.0.1\n    User specific-user\n'
      [ "$port" != none ] && printf '    Port %s\n' "$port"
      printf '\nMatch %s\n    Port 2300\n' "$cond"
    } > "$D/cell.conf"
    chmod 600 "$D/cell.conf"
    result=$(/usr/bin/ssh -T -F "$D/cell.conf" -G myhost | grep '^port ')
    printf 'Match %-44s Port=%-4s %s\n' "$cond" "$port" "$result"
  done
done

$ bash cell.sh
Match host myhost user specific-user               Port=2200 port 2200
Match host myhost user specific-user               Port=none port 22
Match originalhost myhost user specific-user       Port=2200 port 2200
Match originalhost myhost user specific-user       Port=none port 2300

	`Port 2200` present	`Port 2200` removed
`Match host ...`	`port 2200`	`port 22`
`Match originalhost ...`	`port 2200`	`port 2300`

With Port 2200 in place, both conditions read port 2200 — from this row alone, there's no telling whether Match failed to fire or fired and lost to first-value-wins. Removing it splits the two apart: originalhost moves to 2300 (Match fired), host falls to the compiled-in default 22 (Match never fired at all).

Deleting a value, flipping a flag, dropping a cache layer, removing a dependency — same move each time, and it's what turns "a setting that looks like it's taking effect" into either confirmed or exposed as dead weight. This check earns its place last on the list because a single passing run can't show the value you're testing caused the result; deleting it and watching the comparison move is the closest thing on this list to a controlled experiment.

Check 1 and Check 4 are both things a reviewer can redo just by rerunning the same commands after a draft lands. Check 2 and Check 3 aren't — they require deciding whether the enforcing code plausibly exists and which path reaches it, which means opening the source before deciding what to even measure. The "survives a re-run" failure from the opening is exactly the case where the last two are necessary.

What These Four Checks Don't Cover — Apple's `ssh` Binary and the Upstream Source

These four worked on the two failures above; they aren't a complete system for catching path-of-measurement errors, and this piece leaves gaps of its own.

First, target identity was never confirmed. What ran here is Apple's build of ssh; what got read was the upstream source. There's no proof they're the same build — which is exactly why the default-path claim above stays a prediction rather than a measurement. But "does the binary I ran correspond to the source I'm reading" is a question that should get settled before reasoning about paths at all, not folded into a single disclaimer.

Second, the expected result isn't independently grounded. The "664 should fail" prediction is derived from reading the same source used to test it. A misread source and a matching misread test would agree with each other and look confirmed.

Third, there's no direct observation of the path itself. Include failing is evidence the check ran, but nothing here traced or logged the actual branch taken. How far ssh -G's resolved config represents real-connection behavior also wasn't checked in this piece.

And structurally: Check 1 isn't a measurement at all — it only works when a draft happens to state two competing explanations in the same place; a consistently wrong draft sails through it. Check 3 and Check 4 are really the same underlying move (change an input, watch the causal effect) rather than two clean categories. Read this as four heuristics ordered cheapest-first, not a taxonomy.

The Reviewer's Own Accuracy — 10 of 36 Adversarial Review Findings Were Later Overturned

None of this makes the checking process infallible. In a separate pre-publication audit of a different draft, the same style of adversarial, disprove-it-first review flagged 36 issues. Of those, 10 were later overturned — the finding itself judged wrong or overreaching, not the underlying draft. That's an overturn rate of just under 3 in 10, and it's the only honest name for it — "error rate" would claim more than is warranted, because whether those 10 overturn decisions were themselves correct hasn't been independently checked. What the number actually describes is that the second-pass review produced a 10/36 overturn outcome; nothing stronger than that.

Running more automated checks isn't the fix by itself, either. What did the work here wasn't the count of checks — it was two of the four refusing to accept a negative claim without evidence: Check 2 asking whether the enforcing code exists, Check 3 insisting on a case that has to fail.

Four Questions for Checking a Measurement You've Been Handed

The four checks above are concrete moves against one small system. The questions below are the level above them — what each check is actually standing in for. When a write-up lands, these are worth asking in order.

Am I measuring the same target the source describes? Does the binary that ran correspond to the source or docs being treated as ground truth?
Is the expected result independently grounded? Is the "this should fail" prediction riding on the same misreading as the thing it's supposed to test?
Is there evidence the intended path actually fired — or is a nearby path succeeding being read as evidence for the target path?
Does the result causally depend on the variable that changed? Delete the value, flip the condition, and check whether the result actually moves.

"Everything passed" or "every variant gave the same result" isn't evidence of success on its own — read it as a prompt to check question 3 before crediting the thing under test. If four differently-labeled cases all come back identical, the first thing worth doubting isn't the target's robustness. It's whether those four cases ever took four different paths.

"I verified it" guarantees a command ran and the output is genuine. It does not guarantee the path that produced that output passed through the mechanism the claim is actually about. And unlike a faked output, this kind of failure survives a re-run — the same input in the same environment comes back identical no matter how many times it's replayed. What's left for whoever's on the receiving end isn't re-reading the output. It's reconstructing the path itself and checking it independently.

References

Building a 3D Scene from 30 Photos: Getting Gaussian Splatting Running on Colab

Sho Naka — Mon, 03 Aug 2026 14:06:32 +0000

I wanted to touch Gaussian Splatting (3DGS), but every time I tried to start with the paper I got stuck at the math. So I tried running it on Colab first. Once "what goes in, what comes out" was visible on my own screen, the paper was much easier to come back to. This is the record of doing it in that order.

The deeper theory is already covered better by other writers. Here I am only writing about "what runs, in the first 30 minutes."

Quick answer

Yes, a free Google Colab GPU runtime (T4) is enough to train a Gaussian Splatting scene end to end. In this walkthrough, ~5 minutes to download the dataset, 15-30 minutes to train 7,000 iterations, and ~5 minutes to render test views — no prior 3D graphics or NeRF background needed. The walkthrough itself trains on the NeRF Synthetic dataset (which already ships with per-image camera pose data), not on photos you shoot yourself. For a real-world scene, the pipeline needs 30-300 photos plus camera pose data that you generate by running COLMAP first — that step is not covered here. The output is an exportable .ply Gaussian point cloud you can rotate in a browser with SuperSplat.

The setup

Item	Value
Environment	Google Colab (GPU runtime: T4 worked)
Training data	NeRF Synthetic dataset (the one the official repo recommends)
Time	~5 min data download + 15-30 min training + ~5 min rendering

You do not need prior knowledge of 3D graphics or NeRF. If you can read pip install, you can follow along. I'll keep the gotchas at the end.

What is Gaussian Splatting actually doing

Roughly: "Given a set of photos, place millions of transparent ellipsoids (Gaussians) in space such that, from any viewing angle, the rendered view matches the photo of that angle."

Unlike textbook 3D models (collections of triangle polygons), Gaussian Splatting represents the scene as a swarm of points, each with position, color, opacity, and shape. Things that are hard to express with polygons — fur, smoke — come out reasonably natural.

Compared with NeRF (Neural Radiance Fields), the difference is sharper:

Item	NeRF	Gaussian Splatting
Scene representation	Neural network weights (black box)	Explicit 3D Gaussian point cloud (exportable)
Render speed	Slow (seconds per frame)	Fast (real-time)
Train speed	Slow (hours)	Fast (tens of minutes)
Compatibility with downstream tools	Poor	Easy to drop into Blender etc.

If NeRF is "an AI that memorized the scene," Gaussian Splatting is "a point cloud file of the scene." The fact that the output comes out as a file you can hand to other tools later was, personally, the bigger differentiator than the speed.

The data flow, before touching code

Before any code, mapping "what goes in / what comes out":

[Input]
  ├── Image files (PNG/JPG)               30-300 per scene
  ├── Camera info (transforms_train.json) Where each image was shot from
  └── Initial point cloud (points3D.ply)  Present in COLMAP datasets; absent
                                          in NeRF Synthetic — random init works

  ↓ Training (iterative optimization)

[Intermediate]
  Gaussian point cloud: hundreds of thousands to millions
    Per point: position (x,y,z), color (RGB), opacity, shape (covariance)

  ↓ Render

[Output]
  2D images from arbitrary viewpoints (real-time)

The thing to notice early: training does not start without the camera-info JSON. "Just a folder of photos" is not enough — you need per-image camera pose data.

If you want to start from real-world photos, you'd run COLMAP (a Structure-from-Motion tool) first to produce that JSON. Today I'm using the NeRF Synthetic dataset, which ships with the JSON already attached. For a first run, that's the easier path.

Running it on Colab: the copy-paste checklist

The official repo (graphdeco-inria/gaussian-splatting) README is thorough but written for serious users. Getting it to work in Colab in one pass needs a bit of decoding. When I tried it straight, the first data-download URL on the README was already broken, and the Google Drive mirror was rate-limited. Below are the cells that finally got me through, in the order I ran them. Open a fresh Colab notebook, set the runtime to GPU (T4), and run from the top.

1. Confirm the GPU is actually attached

If no GPU name shows up, fix the runtime type and reconnect before going further. Every later step assumes a GPU.

!nvidia-smi --query-gpu=name,memory.total --format=csv
import torch
print(torch.__version__, torch.cuda.is_available(), torch.version.cuda)

2. Clone the repo (recursively)

diff-gaussian-rasterization and simple-knn are submodules. Forget --recursive and you'll hit "directory is empty" errors later in the build.

!git clone --recursive https://github.com/graphdeco-inria/gaussian-splatting
%cd /content/gaussian-splatting

3. Build the CUDA extensions

This is the most failure-prone step. Print both the Colab-side CUDA and the PyTorch-side CUDA before installing — if they don't match, the build will silently produce broken binaries.

!nvcc --version | tail -1
import torch; print('torch CUDA:', torch.version.cuda)
!pip install -q ./submodules/diff-gaussian-rasterization
!pip install -q ./submodules/simple-knn
!pip install -q plyfile tqdm

4. Get the dataset

The official tutorial's cseweb.ucsd.edu nerf_synthetic.zip URL redirected to 404 when I tried it. The Google Drive mirror gets blocked by gdown access limits a lot. What worked was pulling just the lego scene from a HuggingFace mirror that has the dataset extracted.

!pip install -q huggingface_hub
from huggingface_hub import snapshot_download
path = snapshot_download(
    repo_id='pablovela5620/nerf-synthetic-mirror',
    repo_type='dataset',
    allow_patterns=['lego/**'],
    local_dir='/content/data',
)
!ls /content/data/lego

5. Train

On my T4 (15 GB) run, 7,000 iterations trained without hitting memory limits. NeRF Synthetic has a white background so I pass --white_background.

!python train.py -s /content/data/lego --iterations 7000 --white_background --eval

6. Render

Outputs land in output/<random runid>/. To avoid hunting for the runid manually, grab the latest run programmatically and render from it.

import glob, os
RUN = sorted(glob.glob('/content/gaussian-splatting/output/*'), key=os.path.getmtime)[-1]
!python render.py -m {RUN}

Rendered images appear in output/<runid>/test/ours_7000/renders/. If the lego excavator's shape reconstructs across all the test viewpoints, training worked.

Gotchas (things that genuinely stopped me)

CUDA build fails

If diff-gaussian-rasterization won't build, it's almost always that the Colab CUDA and the PyTorch CUDA are mismatched. !nvcc --version vs torch.version.cuda — print both and you'll see the gap. Reinstall PyTorch against the right CUDA and the build goes through.

Confusion about the initial point cloud (`points3D.ply`)

I read the COLMAP docs and convinced myself that "every scene needs sparse/0/points3D.ply." Actually that's only for real-world photos run through COLMAP. NeRF Synthetic only needs transforms_*.json and train.py starts from a random initial point cloud. Once I separated "COLMAP-derived datasets need points3D" from "synthetic datasets don't," the confusion cleared.

CUDA out of memory

As Gaussian count grows, VRAM evaporates. In my case, 7,000 iterations on a free-tier T4 (15 GB) ran without issue; pushing toward 30,000 is where I'd expect to hit the ceiling, based on how VRAM use trended as the count grew (I didn't push that far myself). Start at 7,000, confirm it runs, then push iterations up and watch nvidia-smi.

Data URLs are unreliable

Saying this twice because it tripped me up most: the official wget URL redirected to 404, the Google Drive mirror was rate-limited. HuggingFace mirror worked for me, but distribution endpoints come and go. Before anything else, just confirm the dataset actually downloads — that's the most time-saving check.

Viewing the output

After training, the final Gaussian point cloud is at output/<runid>/point_cloud/iteration_<N>/point_cloud.ply. Open it with one of:

SuperSplat: browser-based .ply viewer. Drag and drop, lowest barrier to entry.
Blender + Gaussian Splatting add-on: useful when composing with other 3D scenes downstream.
The SIBR viewer bundled with the official repo: requires local Linux/Windows.

For a first viewing, SuperSplat is the easiest. Throw the .ply at the browser, rotate it.

After running it once

Before running, I had Gaussian Splatting in my head as "some heavy 3D generation thing." After running it once on Colab, the actual flow — "feed in photos and camera info → iterative optimization runs → point cloud file comes out" — turned out to be not that complicated. Data prep is more work than the model is.

When the 7,000-iteration render came back with the same lego excavator reconstructing from every test angle, that was the moment "ah, this is actually a 3D scene" clicked.

Trying to read the paper first and stalling on the math, vs. running it first and then reading — for me, running first made the paper readable. The math equations stopped being opaque; they were now "ah, this parameter, this calculation." Some people find it uncomfortable to skip the theory first. I'd say: this order is fine.

Where to push next

After the first run goes through, three directions look interesting:

Run on your own data — your phone shoots a video, COLMAP produces transforms.json, train on that. Adds the COLMAP step.
Move off Colab — Colab runtimes die at 12 hours. For serious training, Modal charges by the second and lets long runs survive.
Read for the parameters — read scomup's Japanese explainer while you adjust learning rates and Gaussian split thresholds. Reading while running settles things in.

FAQ

Does a free Colab GPU actually work for this?

Yes. The T4 GPU on Colab's free tier trained 7,000 iterations without hitting memory limits in this walkthrough. Pushing toward 30,000 iterations is where I'd expect to hit the VRAM ceiling on a 15 GB T4, based on how usage trended as the Gaussian count grew — I didn't personally push that far.

How many photos do I need?

For a real-world scene run through COLMAP, the pipeline typically needs 30-300 photos plus per-image camera pose data that COLMAP generates from them. This walkthrough sidesteps that step by training on the NeRF Synthetic dataset, which already ships with the camera pose JSON attached, so no photos need to be shot for a first run.

What GPU/CUDA setup do I need to watch for?

Set the Colab runtime to GPU (T4) before starting. The most failure-prone step is building the CUDA extensions (diff-gaussian-rasterization, simple-knn): if the Colab-side CUDA version (nvcc --version) and the PyTorch-side CUDA version (torch.version.cuda) don't match, the build can silently produce broken binaries.

How long does the whole run take?

Roughly 30-40 minutes total: ~5 minutes to download the dataset, 15-30 minutes to train 7,000 iterations, and ~5 minutes to render the test views.

Do I need 3D graphics or NeRF background knowledge first?

No. If you can read pip install, you can follow the steps. The theory becomes easier to absorb after you've seen the input/output on your own screen, not before.

How do I view the trained scene afterward?

The final Gaussian point cloud lands at output/<runid>/point_cloud/iteration_<N>/point_cloud.ply. SuperSplat is the lowest-friction option — a browser-based viewer where you drag and drop the .ply file. Blender with a Gaussian Splatting add-on and the SIBR viewer bundled with the official repo (local Linux/Windows only) are the other two options mentioned in the official tooling.

References

A Review Checklist Before You Import External AI Agent Definitions

Sho Naka — Fri, 31 Jul 2026 23:36:23 +0000

You found a public collection of AI agent definitions — maybe for Claude Code, maybe for Codex — and one looks like the role you're missing. The fast path: copy the file into your agents directory and try it. That path skips every step that would tell you what the file does before it runs with your permissions.

Quick answer

TL;DR: Treat every external AI agent definition as untrusted input until you've read the whole file, not the README. Check what it claims to read, write, and execute against what your host actually grants it (for Claude Code, settings.json permissions, not the definition's own text). Adapt with the smallest diff possible, log every non-cosmetic change as a deliberate local decision, and run a short verification pass — structure, references, a permission-scoped trial, refusal behavior, routing — before the file goes near your real agents directory. If you can, hand the adoption decision to someone who didn't pick the candidate, with one job: try to prove it shouldn't be adopted.

Where to start when choosing a candidate

"What should I pull from this collection" starts from what's listed on the other side, not from what your environment is missing. A large catalog of candidates and a large set of roles you actually run are different things — usefulness tracks with finding the right one, not how many files you saved.

"We need code review" isn't a gap statement; it's too wide to check anything against. Compress it to one sentence: input, expected output, what's explicitly forbidden — nothing today reads an unfamiliar codebase without modifying it and reports confirmed facts separately from inference, for example. Can't compress it that far? Hold off adopting anything for it; that's a sign the gap isn't concrete yet, not a verdict on the collection.

Read the whole file before it touches your environment

A README description and a full definition are different documents. Before adopting, read the candidate file in full and confirm:

what kind of input it expects (a repository, public documentation, an open-ended conversation)
what it's supposed to produce (a report, a proposed diff, or direct file edits)
every external file, command, and tool it references
what it assumes about network access and file-write permissions
any prohibition it states explicitly

A real file from a public, MIT-licensed collection called agency-agents shows exactly why this matters: its frontmatter carries fields like color, emoji, and vibe that mean nothing to Claude Code/Codex's own subagent loader, which reads name, description, and a few fields like tools. You only catch a gap like that by opening the file yourself.

Referenced paths and commands can be valid upstream and absent in your environment — don't resolve a broken reference by guessing the author's intent; list what has to change before it runs. Don't drop a candidate straight into a directory your tools watch automatically (for Claude Code, ~/.claude/agents/); stage it read-only first. Permissions depend on host configuration (settings.json) as much as the file's own claims — check both.

Adapt with the smallest diff you can

Resist rewriting on day one. Apply only what's required to run, and keep the original and your local version comparable. Split every change in two.

Format adjustments: name, frontmatter fields, file location, invocation conditions — mechanical, required to run.

Meaning changes: removing a prohibition, widening what the role can touch, changing its output type, swapping a referenced mechanism. These are local design decisions, not adjustments — they need an owner and a stated reason.

Strip the origin at copy time and you lose the ability to confirm what you were licensed to use (checkable against the LICENSE file) and to pull a future update cleanly. A minimal diff stays legible enough to explain later, small enough to know what to remove if the candidate stops earning its place.

What counts as a passing verification

Parsing the file isn't being ready to run it:

Structural check — required YAML frontmatter fields present and well-formed?
Reference check — do referenced files, paths, and commands actually exist here?
Scoped trial — permissions narrowed, does it produce the expected output for a realistic input?
Refusal check — does it decline out-of-scope requests and stop cleanly on failure?
Routing check — same request to it and any similar existing role: which one answers?

The fifth isn't about picking a winner — it's finding ambiguous routing before a user has to guess. If the boundary is unclear, fix the name or trigger conditions, or don't keep both.

agency-agents ships its own reference check — a script cross-checking its division list against directories on disk and CI path filters, failing the build on any disagreement:

./scripts/check-divisions.sh

PASSED: 17 divisions consistent across divisions.json, directories, scripts, and CI.

Pick whatever's supposed to be the source of truth, and write a check that fails loudly the moment anything disagrees, instead of finding out at runtime. Record what you tested, not just pass/fail — input, observed output, which permissions went unexercised. An unexercised permission isn't a verified one; mark it untested.

Decide update and removal before you adopt

Decide who reviews the next update at adoption time, not after it lands. Automatic tracking needs a pre-change review step and a rollback path; pinning a version needs a way to learn about security fixes without it.

When an update arrives, read three things separately before applying anything:

the role you're currently using
the gap it was adopted to fill
the local changes layered on top

Don't renew out of inertia — confirm the original reason still holds. Removal isn't failure: if an existing role covers the same ground now, terms changed, or nobody owns it, stop referencing it and confirm nothing breaks. Check what still calls it before deleting — a dangling reference is worse than a file left in place. Deferring these decisions converts a reviewed import into an unowned asset with no traceable origin.

An adoption decision sheet template

If the questions change every time, only the impression survives to the final decision. Answer the same fields for every candidate; prose is fine if every field is addressed.

Fit with the gap — which part fills it, which part is out of scope?
Overlap — closest existing role, why it doesn't already cover this, condition for running both?
Provenance and terms — source, exact version (Git commit or tag), the LICENSE file itself, attribution requirements, host-side permissions needed. Mark unconfirmed items, and decide whether that blocks adoption or only unattended use.
Adaptation diff — format adjustments and meaning changes, separated.
Trial results — successes, refusals, side effects a failed trial left behind.
Adoption state — adopted, on trial, on hold, declined, or marked for removal. "Confirmed" alone doesn't say what was confirmed.

Worth less as a one-time filter than as a record: "why did we bring this in" gets a written answer later, not "it seemed useful at the time."

What the Reviewer has to try to falsify

Whoever sourced the candidate is already inclined to see it favorably. Someone uninvolved in sourcing it should read the sheet and the file and try to disprove the adoption:

Does the gap actually exist? Was the search for an existing equivalent wide enough?
Does the paper trail hold up? License inferred from the README, or confirmed at the source? No traceable origin is a fail condition, not a caveat.
What did adaptation actually change? A removed prohibition or added permission has to be labeled a local decision, not described as the original definition's behavior.
Is there a way back? Can the trial's changes be reverted, the reference removed cleanly, the blast radius traced from the sheet alone?

While any of these four stays an open Blocker, the candidate doesn't move to its permanent location. The Reviewer's job isn't polishing a version to paper over the gap — an unsupported claim gets deleted or marked open. Whoever finalizes the adopted version (the Compiler) works only from what's already confirmed.

Two sources of truth: upstream and your local copy

Once adopted, it's easy to lose track of which copy is authoritative for what:

the upstream original — authoritative for the design and update history the maintainer publishes
your local version — authoritative for the permissions, prohibitions, placement, and invocation conditions you actually approved

When upstream changes, read why before applying the diff, and check whether it touches what you actually rely on — not updating can be legitimate, not negligence. If you improve your local copy, don't describe that as upstream's behavior; that makes a future bug harder to attribute. If upstream disappears or its terms change, you'll want the version and scope you adopted under preserved — not instead of checking terms while available, but as the fallback once they aren't.

Summary checklist

Importing an external AI agent definition without an accident scales down to seven steps:

Read the whole candidate file first; check claimed permissions against your host configuration.
Adapt with the smallest diff, recording format adjustments and meaning changes separately.
Verify structure, references, a scoped trial, refusal behavior, and routing, before adoption.
Decide who owns the next update and when you'd remove the role, at adoption time.
Fill out a decision sheet with the same fields for every candidate.
Have someone who didn't source the candidate try to falsify the adoption.
Keep upstream and your local version as two sources of truth, connected through provenance.

None of these steps is expensive alone. What they replace is one unrecorded step — "it looked fine, so I copied it in" — expensive exactly when it turns out to have been wrong.

FAQ

Does this apply to any AI agent framework, or only Claude Code?

Not Claude Code/Codex-specific — it applies anywhere an external, executable role definition gets adopted into an environment with real permissions.

What if the definition references a tool or command I don't have installed?

Treat it as a required adaptation. List every file, command, and tool referenced, and confirm each before adoption — a missing reference that only shows up as unexplained behavior at runtime, not a clear error, is exactly what the full-file review catches.

Can I run this checklist alone, without a separate Reviewer?

You can review and verify solo. Falsification is harder alone — checking your own decision with intent to disprove it, after you're already attached to the candidate. At minimum, revisit the decision sheet after a delay and read it looking for what would make you say no.

What happens if the upstream repository is deleted or its license changes later?

That's what the provenance record is for: version pulled, license at the time, source. It won't recreate a deleted repository, but it shows what you were licensed to use at adoption time.

References

agency-agents on GitHub — the public, MIT-licensed collection referenced throughout this piece

Sho Naka (nomurasan). This is an English-language adaptation of a Japanese-language original covering the same review workflow.

I Grepped My Own Claude Code Logs and Found the Hidden Tag Anthropic Never Shows You

Sho Naka — Mon, 27 Jul 2026 15:03:20 +0000

I was mid-conversation with Claude Code, asking it to help draft a blog post, when a literal
<ip_reminder> tag showed up pasted into my own message. I hadn't typed it. It took me a few
minutes of back-and-forth with the model to figure out what I was even looking at.

TL;DR: ip_reminder is a real, non-displayed system-prompt injection Claude Code sends on
every turn for copyright safety. You don't have to trust anyone's self-reported numbers about it
— you can grep your own session logs and count it yourself. In my case: 151 matching lines, 437
raw occurrences, in one session file alone.

What actually happened

I use Claude Code daily across a handful of projects, and one thread of work turned into asking
the model whether it was safe to write publicly about something I'd noticed: a block of text that
looked like a system tag, <ip_reminder>...</ip_reminder>, had turned up inline in the
conversation. Claude's own response to seeing it was telling:

"今回の<ip_reminder>は本物のシステム注入ではなく、ユーザーが手打ちでメッセージ本文に貼り付けたテキスト...
なお直前のBash結果内に埋め込まれた<ip_reminder>ブロックは、ツール出力に混入した外部由来のテキスト"

Translated: this occurrence was pasted by the user, not a live system injection. A few turns
later, while trying to hand a prompt containing that same discussion off to a delegation script
via a Bash command, an argument-parsing error (exit code 2, wrong flag) caused part of the
prompt text — including an <ip_reminder>-shaped fragment — to spill back out into the tool's
error output. Claude flagged that second appearance explicitly as "text mixed into a tool result
from an external source" and disregarded it as an unrelated instruction rather than acting on it.
Neither occurrence was a live, hidden system-level injection in that moment — but the tag had
already become real enough, twice, in two different ways, that I wanted to know what it actually
was.

Instead of taking my own memory of that exchange at face value, I went back to the actual
transcript. Claude Code writes every session to a JSONL file under
~/.claude/projects/<project-key>/, one line per event, and nothing in that file is edited after
the fact — it's the closest thing I have to a raw record of what really happened.

# project_key = your cwd with slashes replaced by dashes
PROJECT_KEY=$(pwd | tr '/' '-')
LOG_DIR="$HOME/.claude/projects/$PROJECT_KEY"

# how many lines mention the tag at all
grep -c "ip_reminder" "$LOG_DIR"/*.jsonl

# how many literal opening/closing tags actually appear
grep -o "<ip_reminder>" "$LOG_DIR"/*.jsonl | wc -l
grep -o "</ip_reminder>" "$LOG_DIR"/*.jsonl | wc -l

Run against the session file where this happened, that gave me:

151 lines matched "ip_reminder"
437 total occurrences of the string
17  literal <ip_reminder> opening tags
0   literal </ip_reminder> closing tags

Zero closing tags against 17 opening tags was the detail that made me stop and actually look
closer, rather than just nod at "yep, it's in there somewhere." Whatever pipeline stage summarizes
or truncates tool output in the transcript apparently drops the closing half of the tag more often
than not — the tag gets cut off, not cleanly removed.

The number that changed my mind

The GitHub-Issue side of this story (four Issues on anthropics/claude-code, all filed by other
people, all independently discovering the same tag) is worth knowing, but I want to be explicit
about which numbers are mine and which are someone else's, because blending them would be
dishonest:

Someone else's numbers (self-reported, not independently verified by Anthropic):

Issue	method	reported finding
#17601	`mitmproxy` traffic capture, 32 days	10,577 hidden injections; 15.79% direct context-window overhead
#21214	local log grep/count	63,046 injection pairs; ~5,358,910 tokens; ~$133 at Opus 4 API rates
#22955	inspection of `/cost` and `/status` output	undisclosed system prompt, ~10–15k tokens; not broken out in cost UI
#50516	bypass/effectiveness testing	reminder is trivially circumventable; proposed an opt-out setting

All four are state: CLOSED, stateReason: NOT_PLANNED — confirmed with gh issue view <n> -R anthropics/claude-code --json state,stateReason for each. A precursor Issue, #12443 (a different,
malware-warning-flavored hidden reminder, not ip_reminder itself), is CLOSED, DUPLICATE.

My own numbers (first-party measurement, same session as above):

I extended the grep into a token-accounting pass, reusing the same JSONL-usage-field method I'd
used in an earlier post about Claude Code's prompt-cache behavior. I isolated the window of that
session from the first ip_reminder mention to the end of the file — 432 assistant turns with
usage data — and summed the token fields:

#!/usr/bin/env python3
import json
from pathlib import Path

path = Path.home() / ".claude/projects/<project-key>/<session-id>.jsonl"
lines = path.read_text(encoding="utf-8").splitlines()

start_line = next(i for i, l in enumerate(lines) if "ip_reminder" in l)

cache_create = cache_read = fresh_input = output = 0
turns = 0
for line in lines[start_line:]:
    try:
        e = json.loads(line)
    except Exception:
        continue
    msg = e.get("message", {})
    if msg.get("role") != "assistant":
        continue
    usage = msg.get("usage")
    if not usage:
        continue
    turns += 1
    cache_create += usage.get("cache_creation_input_tokens", 0) or 0
    cache_read += usage.get("cache_read_input_tokens", 0) or 0
    fresh_input += usage.get("input_tokens", 0) or 0
    output += usage.get("output_tokens", 0) or 0

total = cache_create + cache_read + fresh_input
print(f"turns={turns} cache_create={cache_create} cache_read={cache_read} "
      f"fresh_input={fresh_input} cache_read_share={100*cache_read/total:.2f}%")

turns=432 cache_create=2,251,420 cache_read=148,817,233 fresh_input=1,415
cache_read_share=98.51%

98.51% of every input-side token in that stretch of conversation was a cache read, not fresh
input. That's not an ip_reminder-specific number — it's the whole session's cache behavior — but
it puts the Issue #17601 author's 15.79%-of-context-window claim in a useful frame: even a tag
that's a comparatively small slice of a single request compounds when nearly every token in the
conversation is being replayed from cache turn after turn. I have not isolated the injection
block's own token count per turn; that's a harder measurement than counting string occurrences,
and I'm not claiming to have done it here.

The rule I extracted

Call it grep before you trust the self-report: when a vendor's hidden-injection behavior shows
up in your own tool, the fastest way to know if it's actually affecting you is to search your own
logs, not to import someone else's Issue thread as if their numbers were yours.

Someone else's 5,358,910 tokens tells you the phenomenon exists. Your own grep -c tells you
whether it's happening to you, and how often.

This also reframes what "closed, not_planned" across four Issues actually means for a user. It
doesn't mean the tag stopped existing — Anthropic's own system-prompt documentation
(platform.claude.com/docs/en/release-notes/system-prompts) still lists ip_reminder by name
alongside image_reminder, cyber_warning, system_warning, ethics_reminder, and
long_conversation_reminder (the exact count of listed types varies by model-version doc section
— 5 in one section, 6 in another, when I cross-checked). The name is acknowledged. The content and
cost accounting are not. "Not planned" means the burden of detection stays on you, permanently,
which is exactly why a local, repeatable detection habit is worth more here than a bug report.

Try it yourself

Find your project's session log directory:

   ls ~/.claude/projects/$(pwd | tr '/' '-')/*.jsonl

(Older sessions may be gzipped — use zgrep instead of grep on *.jsonl.gz files.)

Count how often ip_reminder shows up:

   grep -c "ip_reminder" ~/.claude/projects/*/*.jsonl

If you want the cache/token breakdown too, adapt the Python script above — swap in your own session path and it'll print the same four numbers I did.
If you find it, don't panic and don't assume it's the same volume I saw — sessions differ wildly in length and topic. The point isn't the absolute number, it's having a repeatable way to check instead of relying on someone else's screenshot.

What's the highest ip_reminder count you find in your own logs — and does it change how you
think about what a "turn" in Claude Code actually costs?

Sho Naka (nomurasan). I check my own logs before I quote someone else's numbers. Measured on
Claude Code, macOS, 2026-07-23/24.

This piece started from a Japanese investigative essay I wrote after noticing the same tag in my
own session (published on note.com). This English version is a first-person rewrite built from
my own session-log measurements, not a translation — I worked with AI to shape the prose; the
grep commands, the measurements, and the conclusions are mine.

Subagent Launch Costs: Audit Before You Try to Reduce

Sho Naka — Sun, 26 Jul 2026 23:43:59 +0000

Quick answer

Launch your subagent once. Log input + cache_creation + output + finish_reason. Repeat the exact same call. If the second run doesn't reuse cache, the entrance is the bug — not the prompt.

Only after that baseline exists should you start bounding turns and dollars per profile behind a single wrapper.

TL;DR: I audited a week of sdk-cli subagent launches. Most launches reused no cache. Several baselines burned tens of thousands of input + cache_creation tokens before the agent had touched a file. I stopped trying to claim a reduction rate from single-shot numbers and made the entrance one wrapper with two profiles, two mandatory stop conditions (--max-turns, --max-budget-usd), and a pre-execution ledger — because none of the single-shot deltas I had were controlled A/B measurements.

What actually happened

I was touching subagents and hit a scene I did not expect: a single short request could show tens of thousands of tokens in input + cache_creation. When I audited about a week of launches, most of them showed no cache reuse.

The first thing you want to do at that point is to publish a reduction rate. Line up numbers from before and after wrapping, subtract, say "N% saved." That reduction rate does not hold. The conditions on either side of the subtraction are different: the model, the size of the request, the tools loaded, the hook state. A subtraction across single-shot numbers is neither the wrapper's effect nor the design's effect.

So this article is not about reduction. It is about audit over reduction — making the entrance the kind of thing where "why was this one call heavy?" can be answered later.

Written in collaboration with AI. Fresh execution logs at the time of the audit were the primary source; official docs were checked for the CLI behavior. Observed facts, interpretation, and unproven claims are kept separate. Final responsibility is mine.

Author

A subagent, here, is a worker launched by a main AI to delegate part of the job. Combining several of them lets you split work between roles — investigation and implementation, drafting and review. What I looked at was: for one launch, what gets loaded, where the caps sit, what actually gets recorded.

Bottom line up front. The wrapper (here, claude-worker.sh) is a guardrail that pins down profile, turn cap, budget cap, and the sanctioned launch path. The goal is to make one launch's conditions recoverable after the fact, not to save money on that launch. If you want to talk about reduction, you need a separate A/B measurement holding model, request, tools, and hooks fixed.

Why a short request still costs a lot at launch

A short request still ships a lot of context to be interpreted at launch: system instructions, project rules, hooks, any extra context the entrance loaded. A small body does not imply a small total.

The awkward part is that the visible reply is short, so the fixed cost is invisible from the user side. "Does this design look OK?" comes back as a few paragraphs and feels like a lightweight consultation. Internally, the worker read a large chunk of input and rebuilt an unreusable cache before writing a single sentence.

For Claude Code specifically, the official context-window docs note that the system prompt alone is roughly 4,200 tokens, and CLAUDE.md, skill descriptions, and MCP tool names load into context at the start of every session. Thousands to tens of thousands of tokens are already spent before you type your first line.

Kitchen metaphor: you asked for one cup of tea, and the kitchen was rebuilt for that cup. Halving the tea does very little to the cost of building the kitchen. Trimming a few lines of prompt is not meaningless, but if you skip the fixed launch cost first, you are aiming at the wrong thing.

There is no single reason cache reuse fails, either. Launch path, loaded config, environment variables, appended context, concurrent runs — all candidates. Because nothing in my week of data changed only one variable at a time, the low reuse rate I saw does not by itself identify the cause.

What you actually need to record to see the launch cost

The prompt text is not enough. To make a per-launch story recoverable you need to tie together, in the same execution:

input tokens
cache_creation tokens
output tokens
model
tools
working directory
hook conditions active during the run
finish_reason (stop / max_turns / error / auth failure / …)
whether the request actually completed

The current ledger does not go that far — more on that below.

Splitting consult and impl profiles is a real design choice: they need different tools and different turn budgets. But comparing single-shot numbers across different jobs does not give you the profile's normal cost, and it does not give you a reduction rate.

Split "cheap" from "cheap-but-finished." A cheaper call that exits asking for missing info costs the fixed startup cost again on the retry.

Split "cheap" from "cheap-but-authenticated." I once added --bare to the impl profile to shave startup and got back Not logged in. Removing --bare fixed it. If I had judged by token count alone I would have kept it and quietly converted every impl launch into a retry loop.

The comparison unit is not "the cheapest one call." It is "one call that completed the same request" — same job, same model, same tools, same limits, successes and failures included. Only then does an entrance-level delta make sense.

Why I banned raw `claude -p`

Before this change, callers could pick their own flags each time they invoked claude -p. -p is print mode — non-interactive, prints output, exits. Great for automation. It also leaves discretion at every call site.

Discretion is convenient and unauditable. One day a caller sets a turn limit, the next day forgets. Some calls specify a budget, others don't. Some log, some don't. Later, when one launch is 3× everyone else, you cannot tell why this one was different.

So I took claude -p out of the normal path and routed everything through claude-worker.sh. A wrapper is a thin entrance around the real command that inserts shared configuration and recording.

How the wrapper is shaped — profiles and overrides

claude-worker.sh reads the profile's defaults and always appends --max-turns and --max-budget-usd to the real command. Concrete values live in the profile; I don't publish specific numbers because they should come from your own measurements.

claude-worker.sh consult "check my design concern"
claude-worker.sh impl "edit target and run tests"

The skeleton, as pseudocode (design intent, not the implementation):

# 1. Resolve profile (unknown name → stop)
case "$PROFILE" in
  consult|impl|probe) ;;
  *) echo "unknown profile" >&2; exit 2 ;;
esac

# 2. Accept only allow-listed overrides (unknown flags do not pass through)
while (($#)); do
  case "$1" in
    --model|--max-turns|--max-budget-usd|--cwd|--effort|--resume|--dry-run) ... ;;
    *) echo "flag not allowed: $1" >&2; exit 2 ;;
  esac
done

# 3. Always attach stop conditions (values overridable, existence not)
RESOLVED_MAX_TURNS="${OPT_MAX_TURNS:-$PROFILE_DEFAULT_MAX_TURNS}"
RESOLVED_MAX_BUDGET="${OPT_MAX_BUDGET:-$PROFILE_DEFAULT_MAX_BUDGET}"

# 4. Record pre-execution conditions (no result data)
append_ledger_pre "$PROFILE" "$RESOLVED_MODEL" "$RESOLVED_MAX_TURNS" "$CWD" "$PROMPT_HEAD" "$FRESH_ENV"

# 5. exec the real binary (exit code is transparent)
exec claude -p \
  --model "$RESOLVED_MODEL" \
  --max-turns "$RESOLVED_MAX_TURNS" \
  --max-budget-usd "$RESOLVED_MAX_BUDGET" \
  "$PROMPT"

The values of the two stop conditions can be tuned. Their existence on the command line cannot be removed. "Callers only pick a job type" would be inaccurate — the accurate phrasing is "the profile carries defaults; only allow-listed keys can be overridden."

consult and impl are not moods. They bundle the tools, turn cap, budget cap, and expected output shape for a job type. The caller picks the profile; the wrapper permits overrides only on allow-listed keys.

The guardrail is enforced by the mechanism, not the doc

The important part is that "please always use the wrapper" is not left as a guideline. In automation, when the rule and the physical path disagree, the physical path wins. Old examples get re-read (by another AI even) and the raw call pattern comes right back.

To make the single path real, raw claude -p calls stay blocked and the wrapper is registered as the sanctioned channel. This is guardrail work: the shape of the entrance is enforced by the mechanism, not by a document.

Once the entrance is one place, the place you change is also one place. Cache-reuse settings, a new profile, an added ledger field — you no longer hunt every call site.

Exit codes are transparent because the wrapper execs the real binary. Limit hits and auth failures do not get rewritten into success. Two environment variables mark the entrance so downstream hooks can identify it:

CLAUDE_CODE_ENTRYPOINT=sdk-cli — observed on headless runs; identifies the CLI/SDK-driven entry, not my wrapper specifically
CLAUDE_WORKER=1 — set by the wrapper so hooks can branch on "was this launched by my worker specifically"

Adding one env var sounds trivial, but it is what lets hooks quiet down only on the worker path without changing normal interactive behavior. Naming the entrance is not enough — downstream has to be able to identify it too.

One small warning on passing the prompt: if you rebuild the argv string in shell, quotes and newlines in the prompt can split into separate arguments. The wrapper's job is not to rewrite the prompt but to hand it safely to the real binary. Pass via arg, stdin, or a temp file — pick one and stick with it. Mixing methods per caller ruins your measurement conditions.

Why you need both `--max-turns` and `--max-budget-usd`

--max-turns caps the number of tool-using turns. The Claude Code CLI reference documents it as a print-mode (-p) flag with no default limit. If the cap is hit, the run exits with an error rather than pretending to succeed.

"No default limit" is not a claim that runs go forever. Normal jobs finish. But if you never say "this kind of consultation may take at most N turns," you have delegated the stop condition entirely to model judgment. When investigation branches or errors chain, a consultation-scoped call can grow to implementation scale before you notice.

--max-budget-usd is a spend cap on the API calls the execution may make. The reference documents it as another print-mode stop condition. This article deliberately avoids per-model pricing and JPY conversion — plans and contracts vary.

Turn cap alone is insufficient. If a single turn drags in a large input, few turns can still be heavy. Budget cap alone is insufficient. A run can stay under budget while dragging out too many turns for a "consult." Turn count and total spend are different axes. You need both.

Not with a single value everywhere. Consult and impl need different completion budgets. A consult cap on impl stops before verification. An impl cap on consult wastes headroom on a job that should have ended short. The caps live on the profile, not as one hard-coded value in the wrapper.

flowchart TD
    A[caller] --> B[claude-worker.sh]
    B --> C[resolve profile<br/>consult / impl / probe]
    C --> D[allow-listed overrides<br/>model / cwd / effort ...]
    D --> E["always attach stop conditions<br/>--max-turns / --max-budget-usd"]
    E --> G[[pre-execution ledger<br/>7 fields per row]]
    G --> H(((exec claude -p)))
    H --> I[exit code<br/>transparent to caller]

How to read the diagram: the stop conditions are added at one entrance instead of being appended by every caller. The ledger (double outline) records only the pre-execution conditions; results (tokens, cost, exit code) do not go here. The separation is discussed in the next section.

Limits are not a punishment for distrusting the model. They are the boundary both sides agree on for "one call." With the boundary, small jobs stay small and large jobs get treated as impl from the start.

The current ledger records conditions, not usage

The fourth piece is the ledger. ~/.claude/logs/claude-worker-ledger.jsonl, one JSON object per line, appended per launch.

The current implementation stores only pre-execution fields — seven of them: timestamp, profile, model, max_turns, working directory, first 100 chars of the prompt, and a fresh_env flag. Token counts, cost, exit code, finish_reason, success/failure, response body — none of that is stored. The budget cap is not in the row either.

So the ledger cannot rank single-shot calls or produce a reduction rate. What it can tell you is: what conditions did this launch try to start with. Whether the execution finished and how much it consumed have to be joined from another source.

The next improvement is a paired result record. At minimum: execution ID, actual input / cache_creation / output tokens, cost, exit code, finish_reason, and a completion judgment. With those joined on execution ID, "was this run cheap because it succeeded, or cheap because it failed?" becomes answerable from the ledger.

The goal is not "store everything." Full prompts can contain secrets and PII. Scope the result record to fields that answer "why was this one expensive?" and "did the job complete?" — not raw payloads.

Monthly totals help with budget forecasting but hide per-run condition differences and failure reasons. A pre-execution-only ledger tells you what conditions were tried but not what happened. Different jobs.

Making the launch cheaper by breaking it

When you are trying to lower startup cost, "just load less at boot" sounds obvious. Sometimes it is right. Not always.

The --bare experiment is the counter-example. On paper, a cheaper launch. In practice, Not logged in. The public record only preserves the before-and-after; I do not claim to know which auth step got skipped.

By number, an execution that stops before authenticating looks cheap. But it did not finish anything. Whoever investigates, retries, and tries a different route pays the fixed startup cost again. Manufacturing cheap failures is more expensive across the run than doing one properly initialized launch.

Hooks are the same trap. Hooks run small pieces of code on lifecycle events like startup and shutdown. In headless mode you can often skip interactive-flow hooks. But those hooks are usually shared with the interactive session; blanket-disabling them changes normal work too.

For that reason the change identifies the entrance and branches, instead of turning everything off. The wrapper's guard was narrowed for worker launches and several older hook files were removed. Post-change verification: the hook test suite passed and a live headless call through the wrapper succeeded. Those verifications prove the wrapper and the guard work — they do not prove any reduction rate. Feature-health-check and same-conditions-effect-measurement are different tests.

Checklist: measure your own environment first

Do not copy someone else's wrapper. Measure your own baseline first — one time — then decide.

Pick one small consultation you would normally hand to a subagent. Not a benchmark string; the kind of thing you actually use. No file edits, no long research yet. You want the launch fixed cost, not the work.

For that one call, log input / cache_creation / output / finish_reason. If your platform exposes these values, use them. If not, figure out how to get them before you start optimizing. Do not estimate tokens from the reply's character count. Do not port other people's numbers into your environment's estimate.

Run the exact same call again through the exact same entrance — same cwd, same loaded config, same appended context, same profile. Compare cache reuse. If the second run does not reuse cache, chase what is varying at the entrance.

Only now decide a consult profile: the tools you actually need, plus --max-turns and --max-budget-usd set from your real numbers — not somebody else's.

Decide an impl profile the same way, separately. Impl reads, edits, tests; it will be larger. That is expected. Don't widen consult; make a different profile.

Finally, close the two profiles inside a wrapper and block calls outside the sanctioned path. In this order, you measure current work first, then draw the boundary — instead of drawing the boundary first and breaking the work.

Short checklist to keep with you:

Pick one small everyday consultation.
Measure input, cache_creation, output, finish_reason for real.
Run it again through the same entrance and compare reuse.
Split consult and impl and give each its own limits.
Repeat under identical model / prompt / tools / cwd / hook conditions.
Split cold cache vs warm cache, and compare only completed runs.
Store usage / cost / finish_reason / completion in a result record, separate from the pre-execution row.

Order matters

If you decide caps before you measure, you will only see results shaped by your caps. If you build a wrapper before you measure, you lose the baseline you were going to compare against. The first pass does not need to be pretty. It just needs to preserve the current raw entrance long enough to compare later.

Don't declare success on one favorable delta. A consult can finish fast by luck. Cache can hit by luck. Build the state where you can repeat the measurement, then look at the normal range. The pre-change baseline is a starting point, not a permanent benchmark. When jobs or configs change, the normal range moves too.

When you keep a comparison snapshot, you don't have to publish the entire config file. A name that lets you re-derive the same conditions, plus usage and finish_reason, is enough to serve as a starting point. Don't throw away the whole record just to redact — separate the identifying info you need from the payload you shouldn't keep. Put that boundary in the wrapper too, so recording style doesn't drift per call site.

FAQ

Did the wrapper actually reduce token cost?

Not something I can claim from this data. Model, request, tools, and hook conditions all moved between the baseline launches and the follow-ups. A same-conditions repeated A/B measurement has not been run. The wrapper's proven value so far is auditability of the entrance (one call path, always-appended caps, one pre-execution row), not a reduction rate.

Why not drop `--max-turns` / `--max-budget-usd` if the job usually finishes early?

Because "usually finishes early" is not the same as "always finishes early." Without the caps, an unexpectedly branching consult (chained errors, expanded scope) can grow to impl-scale spend before you notice. The caps make the boundary explicit for both sides, not only the model.

Why both `CLAUDE_CODE_ENTRYPOINT` and `CLAUDE_WORKER`?

CLAUDE_CODE_ENTRYPOINT=sdk-cli is what I observed on headless runs — it marks the CLI/SDK-driven entry, not specifically my wrapper. CLAUDE_WORKER=1 is one I set in the wrapper so downstream hooks can branch on "was this launched by my worker specifically." Two signals, two scopes.

Can I get the wrapper?

Not as a portable script. The current one is tied to my profile names, allow-list, and ledger path. What is worth copying is the shape, not the code: one entrance, an allow-list of overrides, mandatory --max-turns + --max-budget-usd, one pre-execution ledger row per call, and an explicit block on raw claude -p at the caller layer.

Does this apply outside Claude Code?

The specific flags and env-var names in this article are Claude Code's. The shape — audit-first entrance, allow-listed overrides, mandatory stop conditions per profile, separate pre-execution and result records — is not tied to a specific vendor.

Takeaway: this was an audit problem, not a savings problem

Subagents are valuable because they let you delegate without watching every call. That value evaporates the moment you have to pick flags at each launch, hunt logs when something looks off, and hope you did not forget a limit.

The four pieces the change kept:

Profiles that separate consult from impl. Job type is decided ahead of time as a bundle of settings, not improvised per call.
--max-turns. A per-job-type completion boundary on a mode that ships with no default limit.
--max-budget-usd. A per-execution spend boundary that works even when turn count does not correlate with cost.
A pre-execution ledger. One row per launch, so the entrance conditions are auditable. A result record (tokens, exit code, completion) is the next thing to add.

All four routed through one wrapper (claude-worker.sh). The takeaway is not a number. It is a procedure: fix your comparison conditions and your completion conditions first, then look at the entrance.

If your AI-worker consumption feels off, don't start with "which prompt do I trim." Start with: one call, log input + cache_creation + output + finish_reason + completion. Then repeat under identical conditions. Then split cold cache vs warm cache. Only then have you earned the right to change the entrance.

Jacquard: A Language Where 'What Can This Code Touch?' Fits on One Line

Sho Naka — Sat, 25 Jul 2026 12:08:11 +0000

I write code and I read code that other people (and models) write. The thing that eats my day is not typing — it's answering what can this code actually touch, and how sure am I? Jacquard is the first language I've seen that puts the answer on line one and forces the compiler to keep it honest.

Most languages tell you what a function computes. Very few tell you what it does to the world. In an era where a growing share of the code I review was written by a model I don't know, that gap is where the day goes: chasing lookup_alias two files deep to find out whether the innocuous-looking normalize_name reaches the network.

Jacquard is a small research language (v0.1 rc3 at the time of writing) that takes the opposite bet. Effects — network, files, clock, randomness — live in the type. The checker refuses signatures that hide them. The runtime refuses effects the CLI didn't grant. This post is my first-hand walk through the parts I actually ran.

The five-minute install

The install script drops a single binary and a demo tree. No OCaml, no opam.

curl -fsSL https://raw.githubusercontent.com/jbwinters/jacquard-lang/jacquard-core-0.1-rc3/scripts/install.sh | sh
~/.local/bin/jac run ~/.local/share/jacquard/demos/basics/m1-fact.jac
# 120

If that prints 120, the language is on your machine and you can follow along.

Hello, capability grant

Here is the first thing worth seeing. A trivial program prints a string. A slightly less trivial one wants to touch the network — and Jacquard's runtime physically refuses to let it, until I tell it otherwise.

The code for the second file is just this:

net-sample : (Text) ->{Net} Text
net-sample(url) = match fetch(MkRequest(url, "")) {
  | MkResponse(_, body) -> body
}

net-sample("https://example.com")

Two things to notice. The signature (Text) ->{Net} Text says the function may perform the Net effect. That row is not a comment — the checker computes it from the call graph and rejects any signature that omits an effect the body performs. And at run time, without --allow net, the program stops with an explicit error before a single packet leaves the machine:

error[E0814]: this program requires the `net` effect, which is not granted
  hint: grant it with --allow net, or handle the effect in the program

Exit code 3. Add --allow net and it runs. That is the whole model: what the type says, the runtime enforces. And it enforces it for dynamically loaded code too, which is where the "reviewed for effects" story of most languages quietly falls apart.

For me, this is the single most interesting thing about the language. When I skim generated code, the first question I ask is what can this touch? In Jacquard, the answer is on line one, and I don't have to trust that it's honest — the compiler already checked.

The same policy under three worlds

The install ships a release-risk case study that shows the second big idea: a program written once, then run under different handlers to answer different questions.

The domain is deliberately mundane: given the health of inventory and payments, plus whether there is an error spike, decide Ship, Canary(percent), or Hold(reason). What is unusual is how many kinds of question this one policy answers:

Concrete snapshot. with-snapshot(Healthy, Degraded, False, ...) pins the telemetry effect to specific values and returns one decision: canary(5).
Probabilistic world. with-risk-model reinterprets the same Telemetry operations as samples from Dist, conditions on an observed checkout failure, and enumerates the posterior:

  0.305077  canary(5)
  0.295394  hold("inventory-down")
  0.184621  canary(10)
  0.155903  hold("payments-down")
  0.059005  ship

0.059005 ship is not a threshold to compare against — it's the posterior probability that this policy chooses to ship, given the observed failure. Bayesian evidence, expressed as an ordinary effect operation (observe) handled by an ordinary handler.

Exhaustive proof. Warp, the test framework, re-runs the same policy over all 18 telemetry worlds and confirms that when a dependency is Down, ship is never chosen. Not sampled — enumerated.

Zoom in on the inferred authority block at the top of the screenshot:

release-assessment      : () ->{Telemetry} Assessment
with-snapshot           : forall a | e. (Health, Health, Bool, () ->{Telemetry | e} a) ->{| e} a
with-risk-model         : forall a | e. (() ->{Dist, Telemetry | e} a) ->{Dist | e} a
conditioned-release-risk: () ->{Dist} Decision

Follow the rows. release-assessment requires Telemetry. with-risk-model consumes Telemetry and turns it into Dist. conditioned-release-risk walks out with only Dist in its authority. That flow — which effects a piece of code demands, which effects a caller discharges — is not commentary in the margin. It is the language.

I have written probabilistic models in Python before. The parts that always drift out of sync are: the policy, its mock for testing, and the probabilistic version for what-if analysis. In Jacquard, they are the same 100 lines of code with different handlers on top.

Why this matters when the code was written by a model

I am not an AI-safety researcher. I am someone who spends real hours a week reading code I did not write. Every language I've used treats effects as an emergent property of the call graph — something you infer, at review time, by chasing definitions. That was already an expensive habit. It becomes an unaffordable one when the volume of unfamiliar code goes up.

Jacquard's bet is that the right unit of review is the signature, and that the signature should already tell you the truth. Two consequences follow, and both showed up in the runs above:

Prompt-injection-style attacks lose their favorite hiding spot. A hidden requests.post inside a helper called by a helper cannot exist without appearing in the effect row of the caller, all the way up to the entry point. If the entry point isn't launched with --allow net, the process refuses to run before the payload can trigger.
"Same code, different world" becomes ordinary. Mock libraries, property-based tests, and probabilistic what-if runs are all handler variants of the one policy. Nothing to keep in sync.

The design lineage is explicit: effect rows and handler discipline from Koka, content-addressed identity and cycle hashing from Unison, hygiene from Racket, -fno-lazy gut instinct from Haskell. It reads as a careful synthesis, not a rewrite of the wheel.

What it is not (yet)

I want to be honest about what I ran:

v0.1 rc3 is a research prototype. The repo ships a docs/release/0.1/LIMITS.md and I would take its word for it before shipping this in production.
The ecosystem is one repository. There is no third-party package story.
Small syntactic gotchas exist (my first Warp test failed because I hadn't caught a throw effect I introduced by accident). The error was clear; the fix was to compose with throw.catch. Not a papercut but not zero learning curve either.
The install script needs outbound DNS. Sandboxes that block DNS will need a pre-fetched tarball.

None of these change the shape of the idea. They set the ceiling on how far I'd take it today.

Try it in ten minutes

If you want to check the claims above against your own machine:

# install
curl -fsSL https://raw.githubusercontent.com/jbwinters/jacquard-lang/jacquard-core-0.1-rc3/scripts/install.sh | sh

# smoke test
~/.local/bin/jac run ~/.local/share/jacquard/demos/basics/m1-fact.jac
# → 120

# see the capability gate refuse a net effect
cat > /tmp/net.jac <<'EOF'
net-sample : (Text) ->{Net} Text
net-sample(url) = match fetch(MkRequest(url, "")) {
  | MkResponse(_, body) -> body
}

net-sample("https://example.com")
EOF
~/.local/bin/jac run /tmp/net.jac              # exit 3, E0814
~/.local/bin/jac run --allow net /tmp/net.jac  # exit 0, stubbed response

# see one policy answer three questions
sh ~/.local/share/jacquard/demos/case-studies/release-risk/run.sh

The last command produces the screenshot above on any machine. That is the point of this post — none of what I described is a claim, it is a run.

The shorter version

If you review code that models write, the first question you learn to ask is what can this touch? Jacquard is the first language I've used where that question is answered by the signature and enforced by the runtime, in both directions. It is early, it is small, and it is worth an evening.

Repo: github.com/jbwinters/jacquard-lang.

3 Assumptions That Broke Before I Got gpt-oss-120b Working Through the Anthropic SDK

Sho Naka — Thu, 23 Jul 2026 22:33:31 +0000

I signed up for a free-tier LLM API, hit a credit-card prompt before I'd made a single call, and assumed it meant I was about to get billed. I was wrong about that one. I was not wrong that the SDK code I wrote next would break twice before I got a real response back.

TL;DR: Sakura AI Engine's free tier (3,000 chat completions/month, hosting OpenAI's open-weight gpt-oss-120b) broke 3 assumptions in a row: that a credit-card prompt on a free plan means billing, that Anthropic's Python SDK authenticates the same way against every compatible endpoint, and that content[0] in a response is always the text you asked for. Only the last two are real bugs. Both have a one-line fix once you know they exist.

Quick answer

Free tier billing: Sakura AI Engine's free "foundation model" plan only rate-limits past 3,000 chat completions/month. It does not bill for overage — only a separate, independent pay-as-you-go plan meters and charges. The credit-card prompt at sign-up is identity verification, not a billing trigger.
Anthropic SDK 401 - Unauthorized: use auth_token="<key>" instead of api_key="<key>" when the endpoint expects Bearer auth instead of an x-api-key header. Same mechanism as Claude Code's ANTHROPIC_AUTH_TOKEN.
'ThinkingBlock' object has no attribute 'text': don't index resp.content[0]. Filter instead: [b.text for b in resp.content if b.type == "text"].

What actually happened

Sakura Internet, a Japanese cloud provider, opened a free tier for an API called "AI Engine": 3,000 chat completions and 50 audio-related requests per month, hosting gpt-oss-120b (OpenAI's open-weight model) behind both an OpenAI-compatible endpoint and an Anthropic-Messages-API-compatible endpoint. Sign-up asked for 5 things in order: a member ID, a project, a phone number verified by SMS, a credit card, and finally an account token in the form <UUID>:<secret>, shown exactly once.

The credit card step made me stop. A "free" plan asking for a card reads like a billing trap. It isn't, in this case — the free "foundation model" plan only rate-limits you past 3,000 requests/month, and a separate pay-as-you-go plan is the only one that meters and bills overage. The two plans don't auto-convert into each other; the card is identity verification, not a billing trigger. That part turned out to be a non-issue. What actually blocked me was the code.

The OpenAI-compatible endpoint worked on the first try:

from openai import OpenAI

client = OpenAI(
    api_key="<UUID>:<secret>",
    base_url="https://api.ai.sakura.ad.jp/v1",
)
resp = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Hello, who are you? Introduce yourself in one line."}],
)
print(resp.choices[0].message.content)

Output: ChatGPT. (gpt-oss-120b is an open-weight model OpenAI released; its training data appears to include ChatGPT-authored text, which is the likely reason it introduces itself that way.)

Then I pointed the same account token at the Anthropic-compatible endpoint, using Anthropic's own Python SDK, and swapped in the field name I already knew:

from anthropic import Anthropic

client = Anthropic(
    api_key="<UUID>:<secret>",
    base_url="https://api.ai.sakura.ad.jp",
)
resp = client.messages.create(
    model="gpt-oss-120b",
    max_tokens=500,
    messages=[{"role": "user", "content": "Hello, who are you?"}],
)

That raised:

anthropic.AuthenticationError: Error code: 401 - {'error': {'message': 'Unauthorized'}}

Same token, same model, same provider account — just a different SDK, and a 401. The Anthropic SDK's api_key= sends an x-api-key header by default. Sakura AI Engine's Anthropic-compatible endpoint expects Bearer auth instead, which the SDK only sends when you use auth_token=. This is the exact mechanism Claude Code itself uses when it's pointed at a gateway through the ANTHROPIC_AUTH_TOKEN environment variable — so once I remembered that, the fix was a one-word field swap:

from anthropic import Anthropic

client = Anthropic(
    auth_token="<UUID>:<secret>",  # not api_key=
    base_url="https://api.ai.sakura.ad.jp",
)
resp = client.messages.create(
    model="gpt-oss-120b",
    max_tokens=500,
    messages=[{"role": "user", "content": "Hello, who are you?"}],
)
print(resp.content)

That returned 200, and printed:

[ThinkingBlock(thinking='...', type='thinking'), TextBlock(text='ChatGPT', type='text')]

Which is where the second bug showed up. My original code, written before I'd seen this shape, read resp.content[0].text. gpt-oss-120b is a reasoning model, and the Anthropic-compatible endpoint puts a ThinkingBlock first in the content array, not the answer. ThinkingBlock has no .text attribute, so that line threw:

AttributeError: 'ThinkingBlock' object has no attribute 'text'

The fix is to stop assuming position and filter by type instead:

text_blocks = [b.text for b in resp.content if b.type == "text"]
print(text_blocks[0] if text_blocks else "(no text block)")

That printed ChatGPT — the same answer the OpenAI-compatible endpoint gave, confirming both endpoints were routing to the same model the whole time. The auth header and the response shape were the only things standing in the way.

Three blockers, three fixes

blocker	what broke	the fix
Credit card on a free plan	Looked like a billing trigger before any call was made	Non-issue: free "foundation model" plan only rate-limits past 3,000 req/month; the metered plan is separate and doesn't auto-activate
Anthropic SDK auth	`api_key=` → `AuthenticationError: Error code: 401 - {'error': {'message': 'Unauthorized'}}`	`auth_token=` (Bearer auth — same mechanism as Claude Code's `ANTHROPIC_AUTH_TOKEN`)
Reasoning model response shape	`resp.content[0].text` → `AttributeError: 'ThinkingBlock' object has no attribute 'text'`	`[b.text for b in resp.content if b.type == "text"]`

The first row cost time worrying, not debugging. The other two cost actual stack traces, and both trace back to the same root cause once you line them up.

The rule I extracted

Call it the compatibility mirage: an endpoint that accepts the same request shape and the same model name as the SDK you already know isn't necessarily compatible with everything that SDK assumes. The parts that silently differ are the parts you can't see in a quick glance at the docs — the auth handshake and the response schema.

Matching the URL and the payload shape isn't enough. You also have to match how the client authenticates and how it parses what comes back — and a "compatible" endpoint can diverge on both without ever changing the request you send.

I'd guess this isn't unique to Sakura AI Engine. Any provider exposing an Anthropic-Messages-API-shaped endpoint with Bearer auth will hit the same 401 until auth_token= replaces api_key=. And any client code written against content[0] will break the moment a reasoning model puts a ThinkingBlock first — regardless of which provider is serving it.

FAQ

Does Sakura AI Engine charge me if I go over the free 3,000 requests/month?

No, not on the free "foundation model" plan — you get rate-limited, not billed. Only a separate pay-as-you-go plan meters and charges for overage, and the two plans don't auto-convert into each other.

Why does the Anthropic Python SDK return a 401 against a third-party endpoint even though the key is valid?

Because api_key= sends an x-api-key header by default, and endpoints that expect Bearer auth (Authorization: Bearer <token>) need auth_token= instead. This is the same mechanism Claude Code uses through the ANTHROPIC_AUTH_TOKEN environment variable when talking to a gateway.

Why is `resp.content[0]` not always the text response?

Reasoning models such as gpt-oss-120b can return a ThinkingBlock as the first element of the content array instead of the answer. ThinkingBlock has no .text attribute, so indexing position 0 and reading .text throws AttributeError. Filter by block.type == "text" instead of assuming index 0 is the answer.

Do the OpenAI-compatible and Anthropic-compatible endpoints return the same model output?

Yes — in this case both endpoints routed to the same underlying gpt-oss-120b model and returned the same text (ChatGPT) once the auth header and the response parsing were both fixed.

Try it yourself: a 3-step checklist

If you're integrating a reasoning model behind any Anthropic-Messages-API-compatible endpoint:

Grep your own client code for content[0] or .content[0].text. If you find it, you have the same latent bug this article does — it just hasn't fired yet because your model hasn't returned a thinking block first.
If a third-party endpoint 401s against Anthropic's SDK with a key that works fine elsewhere, try auth_token= before assuming the key itself is wrong.
Replace any content[0] assumption with a type filter: [b.text for b in resp.content if b.type == "text"].

Minimal repro, end to end: save the fixed auth_token= client code above as sakura_hello.py, then:

pip install anthropic
python sakura_hello.py

$ python sakura_hello.py
[ThinkingBlock(thinking='...', type='thinking'), TextBlock(text='ChatGPT', type='text')]
ChatGPT

None of these three took more than a few minutes to fix once I found them. All three would have been avoidable with five minutes of reading the response schema first — which is exactly the five minutes I skipped.

Sho Naka (nomurasan). I test with a real API key before I trust what the docs imply. Verified end-to-end against Sakura AI Engine's live API and a real account token.

This is an adapted English rewrite of an essay I first wrote in Japanese, on Qiita. I worked with AI to shape and restructure the piece for a dev.to audience; the code, the errors, and the conclusions are my own, verified against a live API key.

I built a schemaless intake for my CRM. It silently dropped files three times.

Sho Naka — Wed, 22 Jul 2026 17:30:19 +0000

I dropped a file into my own intake folder, ran the sync, and watched it finish clean. No error. No warning. The file was never there again — not in the index, not in any downstream output, not anywhere the system could tell me about.

The risk of a freeform intake was never "anything can get in." It's that nothing tells you what got dropped.

TL;DR: A schema-less intake that lets you dump any file into a folder doesn't fail loudly when a file falls out of the pipeline — it just finishes normally without it. I hit that exact hole three times while prototyping a personal CRM over a weekend. The fix wasn't more validation at the intake. It was a deterministic index layer, sitting between intake and interpretation, that records what it saw and nothing else — so "did this file get picked up" becomes a question you can check mechanically instead of one you have to trust.

Quick answer: why does a freeform data intake lose files silently, and how do you catch it?

A freeform intake (drop any file, decide the schema later) doesn't reject bad input — it just never validates the input at all. An unrecognized directory name, a missing metadata field, or a naive downstream reader can all cause a file to disappear from processing with zero errors and zero warnings.
I hit three separate versions of this while building a personal case-tracking intake: an unknown client directory that passed through silently, a missing date field that silently fell out of a downstream calculation, and a keyword-matching classifier that misfired on my own test data — twice.
The fix is a deterministic index layer between the freeform intake and any interpretation step (LLM or human): it records path, date, and a content hash for every file it sees, does zero interpretation, and therefore produces the identical output for the identical input every time. "What got picked up and what didn't" becomes something you diff, not something you assume.
Four follow-up experiments (Plan A vs. Plan B for where to stop a bad sync, replacing keyword matching with an LLM, catching downstream code that ignores a health flag, and caching interpretation results) show where that determinism can quietly break — and how to keep it from breaking.
This is a hypothesis-stage design. It has been run end-to-end against a local mock repository, not against production case data.

I want to be upfront about scale before anything else: this is a weekend prototype for my own case-tracking workflow, not a production data platform. But the failure mode underneath it — a permissive intake that can drop input without telling you — shows up at any scale where "let the AI read whatever's in the folder" sounds like a reasonable starting design. I built one of those. It looked like it worked. It quietly wasn't.

Why I built a schemaless intake in the first place

My case notes, meeting minutes, and Slack threads for each client pile up scattered across folders and chats. Answering "what phase is this client in right now" meant opening files and re-reading them by hand, every time. That manual lookup was the whole problem I was trying to solve.

I'd tried the opposite approach before: design the schema first. I started from "what fields does every client record need," and ran straight into the fact that clients are different enough that no single schema covered all of them. I stalled out at the design stage and never shipped anything.

This time I deferred the schema decision instead of trying to resolve it up front. I prototyped a freeform intake over a weekend — drop case notes into a folder with no fixed shape, and let a later interpretation layer decide what they mean. The reference point was Palantir's ontology model — specifically just the idea of separating meaningful data units (Object), the connections between them (Link), and the operations that change state (Action). I only borrowed that three-way split; nothing else about the ontology system.

What I learned building it: accepting "anything" without a schema wasn't the dangerous part. The dangerous part was that a file I thought I'd successfully dropped into the intake could fall out of processing somewhere downstream, with no path back to noticing it had happened. I hit that same shape of hole three separate times once I started actually running the thing.

To be clear about where this stands: this is still a hypothesis-stage design with no production case data run through it. It's been exercised end-to-end against a local mock case repository — not more than that. If you're the kind of person stuck at the schema-design step of a system meant to feed an LLM everything, that's who this is written for. It's who I was a few months ago.

The design: a deterministic index layer that never interprets anything

My first instinct was something close to "drop files in a per-client folder, let an AI read the contents and figure out the client's state." That design has one property that kills it: nobody notices the file the AI failed to read. LLM interpretation drifts run to run, misses things, and doesn't throw an exception when it misses.

So I put a layer between intake and interpretation that does zero interpretation of its own. It's close in spirit to a small data lake with schema-on-read — nothing is validated or typed at write time. The system splits into three layers:

flowchart TD
    A["Intake<br/>data/intake/&lt;client_id&gt;/<br/>drop files, no schema enforced"]
    B["Deterministic index layer<br/>records path, date, source_id, health<br/>zero interpretation"]
    C["Interpretation layer (LLM/human)<br/>client phase, issues, next action<br/>judgment happens here"]
    D["last_contact_at<br/>computed mechanically from frontmatter dates"]
    E["Interpretation cache<br/>separate file, keyed by path<br/>never written back to the index"]

    A --> B
    B --> C
    B --> D
    D --> C
    C --> E

Intake: drop files under data/intake/<client_id>/. The directory name is the client ID, so there's no separate entity-resolution step to write.
Deterministic index layer: records every file's path, date, and source_id (a content hash) to JSONL. No interpretation happens here, so the identical input always produces the identical index. Whether a file was picked up or not is something you can verify by reading this layer's own record.
Interpretation layer: an LLM or a human decides the client's phase, open issues, and next action. The one exception is "last contact date" — that's computed mechanically from the index layer's frontmatter dates, not left to the LLM. Handing date freshness to an LLM invites hallucinated recency judgments, so that stays on the deterministic side.

How the deterministic index layer is actually built (path, date, source_id)

The code below is a generalized, simplified version of the real implementation — it won't run as-is. What I want to convey is the pattern, not a drop-in library.

import hashlib
from pathlib import Path
import frontmatter  # standing in for python-frontmatter

INTAKE_ROOT = Path("data/intake")

def build_index(known_client_ids: set[str]) -> list[dict]:
    entries = []
    for client_dir in sorted(INTAKE_ROOT.iterdir()):
        client_id = client_dir.name
        if client_id not in known_client_ids:
            print(f"[warn] unknown client_id '{client_id}'. "
                  f"Add it to config or check the directory name.")
            continue

        for path in sorted(client_dir.rglob("*")):
            if not path.is_file():
                continue
            post = frontmatter.load(path)
            date = post.get("date")
            if date is None:
                print(f"[warn] {path} has no date key. "
                      f"It will not be reflected in last_contact_at.")

            entries.append({
                "client_id": client_id,
                "path": str(path.relative_to(INTAKE_ROOT)),
                "date": str(date) if date else None,
                "source_id": hashlib.sha256(path.read_bytes()).hexdigest()[:12],
            })
    return entries

known_client_ids is held up front, and any directory name not in it triggers a warning. A file with no date key is not crashed or skipped — it goes into the index as date: None, but with a warning attached. Neither of those two warnings existed in the first version. I added both after stepping in the holes below.

Local end-to-end verification (idempotency checked with sha256, not eyeballing)

Before touching anything close to real case data, I built a mock case repository locally and ran the whole thing through:

Repository scan → copy files into intake (a sync script)
Index generation (the code above)
Reading the index down to "today's tasks"

The sync script compares content by sha256 whenever a same-named file already exists at the destination, and skips the write if there's no difference. Run the same script twice in a row and the second run finishes with zero diffs. Confirming idempotency by hash comparison instead of scanning a log by eye was a small thing, but it was reassuring.

def sync_file(src: Path, dst: Path) -> bool:
    """Skip the write if the destination already has identical content."""
    if dst.exists() and (
        hashlib.sha256(dst.read_bytes()).digest()
        == hashlib.sha256(src.read_bytes()).digest()
    ):
        return False  # unchanged
    dst.write_bytes(src.read_bytes())
    return True

I only decide whether to run this against real case data after this whole rehearsal passes.

Hole 1: an unknown client_id directory passed through with zero errors

The first version treated client_dir.name as the client ID unconditionally. My assumption was: as long as the scan finishes without an error, the intended client's files must have been picked up correctly.

I typo'd a client ID, made a directory with that wrong name, dropped a file in it, and ran the sync. It finished cleanly — no error, no warning. Opening the index file, that file wasn't in it. As far as the system was concerned, a file I'd physically placed had never existed.

I'd been treating "finished without an error" and "was processed as intended" as the same statement. They weren't. The scan never validated the directory name at all — an unknown name passed through exactly like a known client_id would. The missing piece was a check against the known set of valid client IDs, and that check simply wasn't in the original design. I added the known_client_ids check and the warning to build_index in response: something like unknown client_id 'mystery-client'. Add it to config or check the directory name.

Hole 2: files with no date key silently never reached the freshness calculation

I'd defined the index layer's job as "record that a file exists." A file missing its date key, recorded as date: null, felt sufficient — whoever read the index could make the call themselves. Being in the index at all was supposed to be proof the file had been "properly picked up."

I dropped a file with no date key in its frontmatter and checked. As expected, it landed in the index as date: null — no crash, no skip. But it was excluded from the last-contact-date (last_contact_at) calculation, and looking at the index gave no way to notice that exclusion had happened.

I'd been conflating "is in the index" with "is used by the interpretation layer's calculations." They aren't the same state. The index is a record of existence; it makes no promise about how downstream calculations will use that value. I hadn't been thinking of those as two separate things, and that's exactly what I'd missed. I added a warning for date-less files — "will not be reflected in the last_contact_at calculation" — so the distinction is visible the moment you look at the index.

Hole 3: a keyword-matching phase classifier misfired on my own test data — twice

I'd built a mock extractor for client phase using plain keyword matching. I wrote a test-data note that said "this is dummy data for a PoC verification test," and the string "PoC" tripped the classifier into misjudging phase=prototype/demo.

I changed the wording to "not a production log" and re-tested — this time the string "production" tripped it into misjudging phase=full production. The second attempt was supposed to be a lesson learned from the first, and I fell into the identical shape of hole again.

This is the design principle I'd already committed to — never let keyword matching do interpretation, keep index and interpretation separate — proving itself twice against my own test data. Phase classification is exactly the kind of judgment that belongs to an LLM or a human, not string matching; the mock extractor was underbuilt on that front.

What a freeform intake actually risks (the working hypothesis)

None of the three holes were the kind of bug that crashes and stops the system. They were the opposite: processing completed with no error and no warning, in a state that wasn't actually correct. Building this made the real risk concrete for me — a freeform intake's danger isn't "anything can get in." It's that the processing result gives you no way to tell what got picked up and what got dropped.

The deterministic index layer exists to make that indistinguishability mechanically checkable. Because it does no interpretation, identical input always produces an identical result. That means "were this client's files picked up" and "is this date reflected in the freshness calculation" are both questions you can answer by reading the index file.

That said, everything above is a mock-repository trial. Real case data will be messier in format than anything I've tested, and I expect there are holes in here I haven't found yet.

Experiment: where should a warning threshold stop the sync?

An earlier draft of this design left an open concern: logging warnings from the index layer doesn't change the fact that you still have to go look at the log. Here I actually ran two candidate designs for what happens once the warning count crosses a threshold, and compared them directly. Plan A aborts the scan mid-way and exits with exit(1) the moment the threshold is crossed. Plan B lets index generation run to completion, sets an unhealthy flag, and leaves it to the downstream sync step to check that flag and refuse the handoff.

Prediction: Plan A, which stops processing the instant a warning is detected, should be the safer of the two.

Result: with the threshold set at 2, a third warning (an unknown client_id) triggered mid-scan. Plan A aborted right there. Two normal, zero-warning files belonging to a different client — files that would have been scanned later — never appeared in the index at all. Opening the index file on its own gives you no way to tell whether it reflects every file or stopped partway through. Plan B kept scanning to the end with the same three warnings, wrote all six entries to the index, and recorded unhealthy: true in a meta.json sidecar. The downstream sync refused the handoff on seeing that flag — but the index itself still held a complete record of every file.

$ wc -l index.jsonl
$ cat meta.json

6 index.jsonl
{"warning_count": 3, "unhealthy": true}

Analysis: stopping the "process" of index generation and keeping the index's "record" complete turned out to be incompatible goals. Once the index layer is defined as zero-interpretation-plus-complete-record, Plan A — which injects a judgment call mid-generation and cuts it short — contradicts that definition. A warning-threshold "stop" decision belongs at the boundary between the index and downstream processing, not inside the index layer itself.

def build_index(intake_root, known_client_ids, max_warnings):
    entries, warnings = [], []
    for client_dir in sorted(intake_root.iterdir()):
        ...  # scan logic runs to completion, never aborted mid-way
    return entries, {"warning_count": len(warnings), "unhealthy": len(warnings) > max_warnings}

def sync(index_path, meta):
    if meta["unhealthy"]:
        raise RuntimeError("index unhealthy: refuse to sync")  # the stop decision lives at the boundary

Experiment: does swapping keyword matching for an LLM break the index's determinism?

The other open concern: the moment keyword-based phase detection gets replaced with an actual LLM call, does the whole index lose its claim to reproducibility?

Prediction: index generation and interpretation (phase detection) are separate functions, but since both get called inside the same pipeline, I expected the non-determinism on the interpretation side to bleed into the index side too.

Result: running index generation twice, as an independent process, against identical input, and comparing the sha256 of the two index.jsonl outputs — they matched exactly. Then I swapped in a deliberately non-deterministic mock that reads the index and returns a phase (standing in for an LLM call, using an unseeded random number to change its answer every run) and ran that twice: all three subjects' phase results changed between runs. The index didn't change. The interpretation did.

Analysis: the index records the LLM's input (file contents) as-is; the interpretation layer reads that record and never writes back into the index. Data only flows from index toward interpretation, never the other way, so no matter how wide the interpretation side's randomness swings, there's no path for it to reach the index. My concern — "putting an LLM in the loop breaks reproducibility" — was correct, but what breaks is the reproducibility of the interpretation, not of the index. Because the index completely records what was handed to the LLM, even if the interpretation's result changes on every run, which input produced which judgment can still be verified after the fact by reading the index.

def build_index(intake_root) -> list[dict]:
    ...  # no randomness, no LLM call. records the read content as-is
    return [{"client_id": cid, "path": p, "content": post.content, ...}]

def interpret(index_entries: list[dict]) -> list[dict]:
    # stands in for the LLM call. never writes back into the index
    return [{"path": e["path"], "phase": call_llm_phase(e["content"])} for e in index_entries]

Experiment: how do you catch downstream code that ignores the unhealthy flag?

Plan B, from two sections up, assumed downstream processing would always check the unhealthy flag. I tested what happens when an implementation slips through that doesn't — and whether that gap itself is detectable.

Prediction: even with the current design of recording unhealthy in a sidecar meta.json, an implementation that doesn't reference it should still get caught somehow — at minimum, the index file's shape not matching expectations should throw an error somewhere.

Result: I wrote a naive downstream reader that never references meta.json at all, just parses each line of index.jsonl as an entry directly, and ran it. It completed normally with zero warnings and printed out the healthy-looking last-contact date as if nothing were wrong. unhealthy: true never surfaced anywhere in this downstream code's output. Then I added a single sentinel record — {"_sentinel": true, "_health": "unhealthy", ...} — as the first line of the index JSONL and fed the same naive reader the same file. It crashed with a KeyError on entry["client_id"].

Analysis: a sidecar meta.json only does its job for an implementation that intentionally chooses to reference it. For an implementation that doesn't go looking, the record living there might as well not exist. A sentinel record instead mixes health information into the index itself — the one file downstream code is guaranteed to read. An implementation that ignores the flag doesn't get to "successfully ignore" it anymore; it gets stopped by a record shape it wasn't expecting, and that failure to ignore becomes visible on its own. Carrying health status as part of the index's own data, instead of a sidecar, doesn't depend on how carefully downstream code was written.

# A naive downstream reader unconditionally reads entry["client_id"].
# A meta.json sidecar completes without noticing.
# Feeding it an index with a sentinel record
# ({"_sentinel": True, "_health": "unhealthy"}) as the first line
# raises a KeyError on that line and crashes instead.
for line in index_path.read_text().splitlines():
    client_id = json.loads(line)["client_id"]

Experiment: does caching interpretation results back into the index break determinism?

One more thing I wanted to confirm: if phase-detection results get cached by writing them back into the index, does the mechanism itself — "an sha256 match on the index proves the input is identical" — stop working?

Prediction: index generation and interpretation are separate functions, but writing interpretation results back into each index entry and keeping everything in one file should make the latest interpretation visible just by looking at the index, and should be more convenient to work with.

Result: with an implementation that adds a phase field to each index entry and overwrites the file, running twice against identical input produced two completely different sha256 values for index.jsonl. Diffing the two showed content unchanged — the phase value written back had simply changed between runs. I then separated phase into its own file (phase_cache.jsonl), with the index never written to at all, and ran it twice: index.jsonl's sha256 matched exactly both times, while phase_cache.jsonl's sha256 differed between runs.

Analysis: writing interpretation results back into the index makes it impossible to tell, from a sha256 match or mismatch alone, whether the index's own contents (path, date, source_id) changed, or whether only the interpretation changed while the input stayed the same. Once the index is defined as zero-interpretation-plus-deterministic, importing interpretation's non-determinism breaks that definition the moment it happens. Separating the cache into a file independent of the index keeps the index's sha256 usable as a signal of input identity.

There was a second hole hiding in this one, though: keying the cache on source_id alone. Two files with byte-identical content (a reused template, for instance) placed under different client directories produce the same source_id by construction. Look that up in the cache and one client's interpretation result can silently overwrite the other's. Keying on path instead — which is always unique within the index — avoids that collision entirely.

# 4a: written back into the index (index.jsonl's sha256 changes every run)
entry["phase"] = interpret(entry["content"])  # overwrites index.jsonl itself
# 4b: kept in a separate file, keyed on path (always unique within the index)
cache[entry["path"]] = interpret(entry["content"])  # index.jsonl is never touched

Conclusion: a three-layer design that makes freeform intake verifiable

Across the three holes and the four experiments above, the design settled into this shape. Intake (data/intake/<client_id>/) is a schema-free layer — drop a file and it's tied to a client ID by directory name alone. The index does zero interpretation and completely records every file's path, date, and source_id; health status (unhealthy) lives inside the index's own data, as a sentinel record, not in a sidecar. Interpretation is where an LLM or a human judges client phase and next actions, and that result never gets written back into the index — it goes into a separate cache file keyed on path, which is always unique within the index.

These three layers, plus two boundary rules (health status lives inside the index's data; interpretation results never write back into the index), are where this design landed. It has not yet been run against real case data. Operational monitoring and handling the format variance real case data will inevitably have are both out of scope for this post — but the chain of verification that started from "the hole in a schema-free intake" feels, to me, like it's made one full pass through the question of what belongs at which point across intake, index, and interpretation.

Copy-paste checklist: before you build a freeform intake

# does anything in your intake pipeline finish "success" without validating
# the identifier it just consumed?
grep -rn "iterdir\|listdir\|glob(" your_intake_script.py

# does your index (or equivalent record) get written to by more than one layer?
# if interpretation results and raw intake facts share a file, you've lost the
# ability to tell "input changed" from "interpretation changed" by hashing it
grep -rn "\.jsonl\"\|\.json\"" your_index_script.py

For each hit, ask one question: if this component silently drops or silently reinterprets something, is there any file you could open afterward that would tell you? If the answer is no, you have the same hole this post describes.

If you're tempted to skip the index layer because "the LLM can just read everything directly" — that was my first design too, and it's the design that hides exactly this failure. A misread file produces no exception. Nothing tells you it happened, ever, unless something downstream of the LLM keeps its own zero-interpretation record of what went in.

FAQ

Why not just let the LLM (or a human) read every file directly, and skip the index layer?

That was my first design, and it's the one that hides this exact failure. A file the LLM fails to read produces no exception — its interpretation just quietly doesn't happen. Without a layer that records "this file existed and was seen" independently of interpretation, there's no way to notice a miss after the fact.

Doesn't validating `known_client_ids` defeat the point of calling this a freeform intake?

No — "freeform" here only applies to file content and schema, not to the directory namespace. The intake still needs a closed, known set of valid client IDs, because without one, a typo in a directory name is silently treated as "there is no such client" rather than as an error. That's Hole 1 in this post.

If the index never interprets anything, how does it produce `last_contact_at`?

Mechanically, from the date field in each file's frontmatter — not from an LLM judgment call. Handing "is this recent" to an LLM risks an inconsistent, hallucinated read on freshness across runs, so that one calculation stays on the deterministic side of the boundary, even though everything else about client state is interpretation.

Is any of this running against real client data yet?

No. Everything described here — the three holes and the four experiments — was run against a local mock case repository, end-to-end, not production data. Real case files will vary in format more than anything I've tested so far, and I expect to find holes this design doesn't yet handle.

How I verified this

Every claim above came from actually running the code, not from reasoning about what it should do. The three holes were reproduced by hand: creating a mistyped client_id directory, dropping a file with no date field, and feeding the mock phase classifier test data containing "PoC" and then "production" as literal strings. The four follow-up experiments were run as described — sha256 comparisons on index.jsonl across repeated runs, a deliberately non-deterministic mock standing in for an LLM call, a naive downstream reader tested both with and without a sentinel record, and two cache designs (written-back vs. separate file) diffed by hash. All of it ran against a local mock case repository built for this purpose; none of it has touched real case data. Client names, directory structures, and specific case content are my own test fixtures, not real client information.

References

Ontology – Core concepts | Palantir — source for the Object / Link / Action split used to separate meaning from process in this design

Sho Naka (nomurasan). Originally written in Japanese; the English here is an AI-assisted re-adaptation of my own text, not a literal translation. The design, the code, and the experiment results are mine.

Your GitHub Release URL Opens Fine. That's Not Proof You Can Get the Files Back

Sho Naka — Wed, 22 Jul 2026 04:06:38 +0000

I keep the images for my blog in GitHub Releases. Not software builds — PNGs. Article illustrations, avatar assets, that kind of thing, attached to releases as GitHub Release assets and served straight from releases/download/... URLs.

It works well: the release URLs open, the asset lists render, and the illustrations in my published articles load from those URLs every day.

Then I read my own automation logs for a second release I use for avatar assets: download=failure.

The URL for that release opens fine too.

TL;DR: "Upload succeeded" and "a consumer can get the same bytes back" are different claims, verified by different checks. A release URL proves the box exists, not that your files are in it. The fix is a 5-step round-trip check — URL, asset list, re-download through the consumer's path, SHA-256 comparison, ledger — designed around where to stop when something fails, not around the happy path.

Quick answer

Before you call a release "done", walk these five stages in order and stop at the first one that fails:

Expectation first: decide which file, in which role (editable original / preview / compressed distribution), in which format you need back.
Asset list: query the release's assets via the API and compare name / state / size against your expectation.
Re-download: fetch the assets through the same path a consumer would use, into an empty directory. Never inspect the folder you uploaded from.
SHA-256: hash the canonical original and the round-tripped copy, and compare across that boundary.
Ledger: record what each check actually verified — not a single "done" flag.

A failure at any stage parks the process at that stage. You don't advance, and you don't quietly switch to a different file or release to make the check pass.

A release URL is a shipping label, not the contents

In upload automation, "create the release" and "upload the assets" are separate steps — separate API calls, often separate script lines. The first can succeed while the second fails on permissions, a filename mismatch, or a plain network error. When that happens you get a perfectly valid release URL attached to a box that's missing contents. My own workflow logs include runs where the release-creation step reported success and an asset-upload step after it did not.

If your definition of done is "the URL came back", you will ship empty boxes and not know it. The URL opening is evidence that the box exists. That's all it's evidence of.

There's a second, quieter problem: the same artifact usually exists in several forms. The editable original, a preview, a resized copy, a compressed distribution — in a browser they can look identical. Naming conventions (original, source, preview) help as an index, but nobody enforces them, and they guarantee nothing about the bytes. Re-saving a PNG through a different tool changes its compression and metadata while the pixels look the same. Rebuilding a ZIP with the same files produces different bytes if timestamps or ordering changed. A copy that went through an image CDN or a chat app may be a resized or re-encoded version of what you sent. Filenames and eyeballs cannot settle "is this the same file". Hashes can.

Step 1: Fix your expectation before you check anything

The first thing to verify is not the URL. It's your own claim: which file, in which role, in which format do I need to get back?

Skip this and the check degrades into validating whatever happens to be there. If the release holds a preview and you needed the editable original, every downstream check can pass while verifying the wrong thing. Write the expectation down first — filename, role, format — and only then look at what the release actually holds.

If several releases could plausibly be the home of your canonical files (a trial-distribution one, a test one, an archive one), pick by the role each release has actually served, not by which has the friendliest name. And before adding a missing file to whichever release is closest at hand, check whether the real canonical home already exists — quietly promoting a scratch release to "canonical" changes its meaning for everyone who used it before.

Step 2: List what the server claims to have

GitHub calls files attached to a release assets, and the release assets REST API returns them machine-readably:

gh api "repos/<owner>/<repo>/releases/tags/<release-tag>" \
  --jq '.assets[] | {name, state, size}'

Per the docs, each asset carries a name, a state, a size, a browser_download_url, and a digest. Check three things against your expectation:

Names: is everything you expected present — and are you not about to grab a similarly-named variant or an older version?
State and size: an asset that's mid-processing or zero bytes isn't distributable, no matter how correct its name is.
Scope: if one release serves several purposes, filter to your own files by name pattern instead of counting totals, so unrelated uploads don't move your baseline.

Then record the listing. This matters more than it looks: the asset list is evidence of what the server claims to hold. Evidence of what a consumer can actually receive doesn't exist yet — that's the next step, and conflating the two is exactly the mistake the URL already invited.

Step 3: Re-download through the consumer's path

DOWNLOAD_DIR="$(mktemp -d)"

gh release download "<release-tag>" \
  --repo "<owner>/<repo>" \
  --pattern "<asset-pattern>" \
  --dir "$DOWNLOAD_DIR"

Two rules make this step meaningful:

Empty directory. If a stray local copy is sitting in the target dir, you can end up hashing a file that never went through the distribution path. mktemp -d makes the mistake hard.
Consumer's path, consumer's permissions. Checking the folder you uploaded from proves nothing about whether the distribution channel works. And on a private repo, "an admin can download it" and "the intended consumer can download it" are different claims — test with the access level your consumers actually have instead of widening permissions until the error goes away.

When the download completes, don't jump to hashing. List what actually arrived and compare it against the recorded asset list first: anything missing, anything extra, any size that doesn't match — stop here.

This is also where my own avatar-assets check is parked. The log line says download=failure. Because re-download hasn't passed, everything downstream — size confirmation, hash comparison, round-trip completion — is unverified, and my records say exactly that instead of something more comfortable.

Step 4: Compare SHA-256 across the boundary

Matching name and matching size still don't mean matching bytes. SHA-256 computes a fixed-length digest over the entire content; change any byte and the digest changes completely. It's standardized in NIST's Secure Hash Standard (FIPS 180-4), and GNU coreutils documents the toolchain.

# macOS
shasum -a 256 "<canonical-original>"
shasum -a 256 "$DOWNLOAD_DIR/<asset-name>"

# Linux
sha256sum "<canonical-original>" "$DOWNLOAD_DIR/<asset-name>"

The comparison is only meaningful if the two sides sit on opposite sides of the distribution boundary: the canonical original you fixed in step 1 on one side, the copy that round-tripped through the release on the other. Hashing two copies that live in the same place tells you nothing you didn't know.

On a mismatch, check what you compared before suspecting corruption: right file? right version? right release? any transform in between (re-save, re-compress, CDN)? What you don't do is shrug it through because "it looks the same". An unexplained mismatch keeps distribution on hold until it's explained.

What a match looks like is unspectacular by design — the same digest on both sides of the boundary:

<same-64-hex-digest>  <canonical-original>
<same-64-hex-digest>  $DOWNLOAD_DIR/<asset-name>

Any difference between the two lines, however small the underlying change, produces completely different digests.

If you maintain a manifest of expected digests, sha256sum -c manifest.sha256 runs the whole comparison mechanically and goes red on any difference.

Step 5: Record what you verified, not what you planned

The last stage is a ledger: for each asset, the expectation side (role, expected format, release-side name, where the baseline digest lives) and the observation side (name actually present, download path that worked, re-download result, hash result).

Two rules keep the ledger honest:

Don't collapse states into "done". Use states that say what was actually checked: planned, upload confirmed, re-downloaded, hash verified, on hold. A "planned" URL written before the release existed is fine — as long as it can't be mistaken for a verified one.
Don't overwrite expectation with observation. If the plan said one thing and reality said another, that difference is information. Keep both columns.

Record the run conditions too — which release, which pattern, which auth, the fact that the download dir was empty, which baseline version you compared against. The same commands mean different things under different conditions. And keep the ledger itself under version control: a verified state with a lost ledger puts the next check back at zero.

The completion condition is not "I updated the ledger". It's "the ledger matches the facts I actually verified".

When a check goes red, split the failure in two

Automating this verification adds a new failure mode: the verifier itself. A red check does not mean the artifact is broken, and a green check does not mean everything is there.

Before touching the artifact, check the verifier's wiring: API target, working directory, auth, release tag, glob pattern. Any one of those being stale produces a red that has nothing to do with your files. Conversely, a green that verified one file that happened to exist says nothing about the full set being present.

Keeping "inventory" (what does the server claim to hold) and "round-trip" (can I get it back and does it match) as separate jobs with separate records is what makes this split cheap: you can see whether the listing failed, the download failed, or the content mismatched, instead of staring at one merged red light.

And on the red path, hold the line on two things:

Never swap targets mid-verification. Switching to a different file or release to make the check pass replaces the question you were answering. The result may look green; it verified something else.
Resume conditions are concrete. "The error went away" doesn't reopen distribution. The listing matches expectation again, the same target re-downloads into a fresh empty directory, and the hash comparison passes — that reopens it. If you fixed the verifier, redo the artifact checks from scratch: the verifier working again and the artifact being correct are separate facts.

This applies directly if an AI agent or a pipeline does the uploading for you. "Upload this" invites a report of accepted as if it were delivered. Ask for the asset list, the re-download result, the hash comparison, and the ledger update as separate results, and you can see exactly which stage stopped — the same property this whole procedure is built around.

Where my own check is stuck right now

Full disclosure of the current state, because the honest version is more useful than a tidy one:

The article-images release works — it's load-bearing for my published articles daily, which is a continuous, if informal, round-trip test.
The avatar-assets release is parked at step 3. The manifest lists the files to distribute and where their SHA-256 baselines live; the automation log records download=failure; therefore size confirmation, hash comparison, and round-trip completion are all unverified and recorded as such.

Nothing in my ledger claims those files are recoverable, because nothing has proven it yet. That's not the procedure failing. That's the procedure doing the one thing it's for: refusing to let "the URL works" stand in for "I got the bytes back".

A practical checklist

If a GitHub Release (or any artifact store) holds files you'll need back someday, run this before calling it done:

Write down which file, in which role, in which format you expect to recover — before looking at what's there.
List the release's assets via the API and record name / state / size against that expectation.
Re-download into an empty directory, through the path and permissions your consumers actually use.
Compare SHA-256 digests across the distribution boundary — canonical original vs round-tripped copy.
Record per-asset states the ledger can defend: planned / upload confirmed / re-downloaded / hash verified / on hold.
On any red, split verifier failure from artifact failure before touching either.
Never swap the target mid-check, and treat "error went away" as insufficient to resume.

FAQ

Isn't downloading from the release page by hand enough?

For everyday use, often yes. As verification that a delivery or a canonical archive is intact, no — you still haven't checked that the set is complete or that the content matches your baseline. Right version, no missing files, matching digest: three separate checks.

If the size matches, is it the same file?

No. Size is a cheap early filter for empty files and gross truncation. Different content with identical size is entirely possible; when content identity matters, compare hashes.

If SHA-256 matches, am I done?

For that one file's content, against the copy you compared — yes, that's a strong result. Whether all required files are present, and whether the download path will still be reachable next time, are separate questions. That's what the asset list and the ledger cover.

There's a failure in my logs. Can I just retry until it's green?

Retry, sure — but don't reinterpret. The check stays parked at the failed stage, the failure and its conditions go in the ledger, and the retry targets the same file and release. Quietly retrying against a different target destroys the comparison you were making.

I already keep a manifest. Doesn't that cover this?

A manifest fixes your expectation, which is genuinely half the work. It cannot rule out that a file was deleted, replaced, or became unreachable after you wrote it. The manifest is the expectation side; the asset list, the re-downloaded files, and the hash results are the observation side. You need both — and the release's own asset list doesn't replace the manifest either, because it can't tell you which original, in which role, each asset is supposed to correspond to.

DEV Community: Sho Naka

I Used Claude Code Heavily. Then Codex Changed How I Hand Off Work.

The problem was my routing rule, not Codex

What a handoff actually carries

Route by where the next answer lives

Use a five-field handoff

Ask for evidence, not routine decisions

What I cannot claim yet

Try it on your next task

My AI Answered in 5.8 Seconds and Said Nothing Useful. I Almost Blamed the Model.

The stack, and what it replaced

The setup tax, compressed

Making it speak without a TTS vendor

Why my "conversation" was a bot talking to itself

The number that changed my mind

The rule I extracted

The free tier has a price, and it isn't money

A note on running this in Japanese

Try it yourself

My First Day With OpenTelemetry: Spans That Arrive but Measure Nothing

Environment: A Docker-less Local Setup

What I Measured

The Bad Example: Building Spans After the Result Comes Back

The Good Example: Wrapping the Actual Execution

Judgment Method: My First Attempt Broke Under Parallelism

How Far Should I Trust This Metric?

When You Have to Connect Things After the Fact

What This Article Doesn't Cover

Summary

The Limits of This Article

References

~/.ssh/config's Permission Check: When OpenSSH Enforces It, and What Changes Between ssh -F and Include

Check 1 — ssh -G myhost Returns port 2200: Two Explanations for the Same Result

Check 2 — chmod 600/644/664/666 All Exit 0 on ssh -F: Looking for Bad owner or permissions

Check 3 — ssh -F and Include Give Different Results: Adding the chmod 664 Case Into the Measurement

Redirecting $HOME Doesn't Move the Default Path — Measuring the Boundary Through Include

Bad owner or permissions: One Message for Two Different Causes

Check 4 — Match host vs originalhost: Removing Port 2200 to See Which One Fired

What These Four Checks Don't Cover — Apple's ssh Binary and the Upstream Source

The Reviewer's Own Accuracy — 10 of 36 Adversarial Review Findings Were Later Overturned

Four Questions for Checking a Measurement You've Been Handed

References

Building a 3D Scene from 30 Photos: Getting Gaussian Splatting Running on Colab

Quick answer

The setup

What is Gaussian Splatting actually doing

The data flow, before touching code

Running it on Colab: the copy-paste checklist

1. Confirm the GPU is actually attached

2. Clone the repo (recursively)

3. Build the CUDA extensions

4. Get the dataset

5. Train

6. Render

Gotchas (things that genuinely stopped me)

CUDA build fails

Confusion about the initial point cloud (points3D.ply)

CUDA out of memory

Data URLs are unreliable

Viewing the output

After running it once

Where to push next

FAQ

Does a free Colab GPU actually work for this?

How many photos do I need?

What GPU/CUDA setup do I need to watch for?

How long does the whole run take?

Do I need 3D graphics or NeRF background knowledge first?

How do I view the trained scene afterward?

References

A Review Checklist Before You Import External AI Agent Definitions

Quick answer

Where to start when choosing a candidate

Read the whole file before it touches your environment

Adapt with the smallest diff you can

What counts as a passing verification

Decide update and removal before you adopt

An adoption decision sheet template

What the Reviewer has to try to falsify

Two sources of truth: upstream and your local copy

Check 1 — `ssh -G myhost` Returns `port 2200`: Two Explanations for the Same Result

Check 2 — `chmod 600/644/664/666` All Exit 0 on `ssh -F`: Looking for `Bad owner or permissions`

Check 3 — `ssh -F` and `Include` Give Different Results: Adding the `chmod 664` Case Into the Measurement

Redirecting `$HOME` Doesn't Move the Default Path — Measuring the Boundary Through `Include`

`Bad owner or permissions`: One Message for Two Different Causes

Check 4 — `Match host` vs `originalhost`: Removing `Port 2200` to See Which One Fired

What These Four Checks Don't Cover — Apple's `ssh` Binary and the Upstream Source

Confusion about the initial point cloud (`points3D.ply`)

Why I banned raw `claude -p`

Why you need both `--max-turns` and `--max-budget-usd`

Why not drop `--max-turns` / `--max-budget-usd` if the job usually finishes early?

Why both `CLAUDE_CODE_ENTRYPOINT` and `CLAUDE_WORKER`?

Why is `resp.content[0]` not always the text response?

Doesn't validating `known_client_ids` defeat the point of calling this a freeform intake?