DEV Community: lleqsnoom

Every Agent Session Is a Test Run

lleqsnoom — Wed, 16 Sep 2026 17:56:14 +0000

The short version

Every morning at 05:00, a scheduled job opens the transcripts of everything my AI coding agent did in the last 24 hours. It leaves me a list of proposed edits to the skills it used. Not a summary of what I did — a list, each item citing the exact line of the transcript and the exact line of the file that caused the problem.

I built it on top of xskills, my collection of agent skills. It is called x-skills-daily-reflection. It turned my skill set from "a folder of instructions I wrote once" into "a tool that gets measurably less wrong every week."

The problem: skills rot silently

In an earlier post of mine I argued for building your own agent skillset. That argument has a hidden bill, and I named it there: maintenance is on you, forever.

Here is what maintenance looks like. A skill is a markdown file of instructions. When it is wrong, nothing crashes: no stack trace, no failed test, no red build. What happens instead is worse: the agent reads a stale instruction, stumbles, improvises, and succeeds anyway. The work gets done. The only trace is a slightly longer session, a repeated tool call, a correction you typed mid-flow and forgot about two minutes later.

That trace lives in a place nobody looks: the session transcript. Every conversation your agent has is a test run of the skills it used, and every transcript is a test report that gets thrown away.

People solved this for code decades ago. CI runs your tests on every push. What is missing is the equivalent for instructions: a job that treats your own agent sessions as the test suite. Your skill files are the code under test.

The loop in one picture

The diagram is a five-step loop; you can see it rendered in the original post.

Three pieces make that loop run, and none of them are clever:

automation/daily-reflection/
├── collect-sessions.mjs   # gather last 24h of sessions, scan for friction
├── precheck.sh            # fail closed: skip the run when there is nothing honest to do
└── runbook.md             # the agent's instructions for the headless run

The collector walks every project directory my agent CLI (Crush) has touched recently. It exports the sessions modified in the last 24 hours and runs the x-autoreflection scanner over each one. The scanner extracts mechanical signals from each session: failed commands, repeated tool calls, moments I corrected the agent, skills loaded but never used, questions asked in prose instead of a structured prompt. Each signal keeps a severity, a suspect skill, and quoted evidence.

The precheck runs before the agent is invoked and fails closed. It skips the day when a binary is missing, when skills/ has uncommitted edits, or when no session in the window touched a skill. That git check is the whole philosophy in four lines: every proposal cites a skill by file:line, and uncommitted edits move those lines. A reflection that runs against a moving target produces plausible garbage. Plausible garbage is the one output you cannot afford from a system whose entire job is to be trustworthy.

The runbook is what the headless agent follows. It is plain markdown, but written for an agent that cannot ask questions:

At most 4 sessions, at most 10 proposals. Caps, not targets. Without a human in the loop, an agent optimizing for "good retro" will find something to say about everything. A quiet day produces a three-line digest. The runbook says: do not invent findings to fill it.
Verify every signal against the real file. A signal is a lead, not a finding. The agent opens the actual SKILL.md and decides. Keep: the file is wrong. Re-grade: real friction, but not that skill's fault. Drop: the transcript misled the scanner. A grep exiting 1 because it found nothing is the answer, not a failure. Dropping is recorded, so tomorrow's run does not re-litigate it.
Blame the instruction, not the agent. "The SKILL.md did not say" is a gap. "I forgot" is not.
Every proposal has Signal, Target, Change, and Check. The Check must be a command that runs: a test, a lint rule, a script that exits 0 once the fix is in. A proposal you cannot check is a wish.

And then the rule that defines the whole system, printed twice in the runbook:

Never edit anything under skills/. Propose, then stop. The review decides.

Why the loop does not close itself

The obvious design is a fully closed loop: cron fires, agent reflects, agent edits the skill, agent commits. It would take an afternoon to build, and it would be the wrong system.

Not because agents edit files badly — they are fine at it. Because a self-editing instruction set has no audit trail and no stopping condition. If today's run misreads a signal and "fixes" a skill into uselessness, tomorrow's run reflects on sessions shaped by that bad edit. The drift compounds. The precheck already knows this, which is why it refuses to run against a dirty tree. Skip the review, and you are not editing a file; you are editing the audit trail.

So the output is a DIGEST.md with proposals, and the output of my morning coffee is five minutes of checkboxes: accept, defer, drop. Accepted proposals route by size. A one-line edit goes in directly, a cluster goes through x-fix, and a design decision goes to x-plan with the reflection attached. The automation read 40 sessions. It handed me three verified, checkable changes instead of a vague feeling that something was off.

This is the same shape as a deploy pipeline. The pipeline builds, tests, and stages everything automatically — but something with judgement presses the button.

What it found, honestly

The loop is honest about its limits. Mechanical scanning only sees friction that leaves a trace. A wrong-but-successful instruction produces no failed command to flag, which is why the runbook lets the agent log manual findings it noticed by reading. And it costs one small agent run per day — on a quiet day, the precheck skips even that.

Most days are the three-line digest. But the pattern of what surfaces is what you cannot see from inside:

The same gap twice is a defect. A missing flag in one session is a hypothesis. The same command failing in three sessions is a bug with a patch.
Loaded but unused skills are a smell. If the agent keeps loading a skill and abandoning it, the description promises the wrong thing or the body answers the wrong question.
Documentation drift is catchable. A documented command that no longer runs leaves one trace: the agent tries it, fails, and improvises. Hours later, that trace is in a transcript. Without the loop, you find it months later, when you run the command yourself.

Build the minimal version

You do not need Orca, a skill framework, or any of my code. You need a place where your agent's conversations are stored, a scheduler, and the agent's headless mode.

# crontab: 5 5 * * *  ~/reflection/run.sh

agent sessions export --since 24h --out /tmp/reflection/sessions/   # collect
agent run --headless --runbook ~/reflection/runbook.md \
  --context /tmp/reflection/sessions/                               # reflect

And a first runbook can be five sentences:

Read every transcript in the folder.
List every moment the user corrected you, retried a command, or worked around a missing capability.
For each, check whether an instruction file could have prevented it.
For each that could, write one proposal: which file, what change, what command proves it.
Output the list to DIGEST.md. Edit nothing.

The first week it will mostly produce noise, and the noise is informative — it shows you which signals to filter. Mine learned to stop flagging grep's exit codes after I made "drop the signal, record why" part of the procedure.

The part people get wrong

Two failure modes, both of which I built first.

The loop that must find something. You schedule a daily reflection. For three days it says "nothing to report." The temptation is to loosen the criteria until something appears. Do not. A reflection that never comes back empty is not reflecting. It is generating content.

Closing the loop to save five minutes. The human review feels like the weakest link, so it is the part everyone automates away first. It is the load-bearing part. Five minutes of checkboxes a day is the price of an instruction set that improves in one direction.

The short version, again

Your agent's conversations are already a test suite for your workflows — the only one that runs on real work instead of synthetic benchmarks. A cron job, a collector script, and a runbook are enough to stop throwing that test suite away.

The system does not improve itself. It improves your half-hour a week into a targeted, evidenced, checkable five minutes. Every accepted proposal makes the next week a little smoother, and the next digest a little shorter. Most mornings it is three lines. That is the system telling you it worked.

Originally published at mielony.com.

Sampling Parameters as Substances: An Honest Analogy

lleqsnoom — Mon, 14 Sep 2026 06:11:00 +0000

Sampling Parameters as Substances: An Honest Analogy

A strange question that works: why does cannabis seem to make people more creative?

It doesn't make anyone smarter — it changes filtering. The mind lets through associations it would normally reject. Distant ideas reach awareness more easily. That feels like creativity from the inside.

Then someone asked a better question: so it's like turning up a model's temperature?

Yes. And that opens a door. Every sampling parameter in a language model sits on the same trade-off — how boldly the model picks its next word — and each one has a rough equivalent in how a mind works.

The one trade-off behind everything

A language model doesn't "know" what to say. At every step it produces a probability for every token it could write next, then picks one.

Pick the most likely word every time and the text is safe, correct, boring. It loops.
Pick something unlikely often enough and the text is surprising — sometimes brilliant, sometimes nonsense.

Every parameter below just moves the needle between those two poles.

The LLM pharmacopoeia

Max tokens is the odd one out — it's a budget, not a mood. It never touches which token gets picked, only how many are allowed.

Why this matters now

One caveat, because it's the most interesting part: the dial is disappearing. The newest reasoning models expose no temperature at all. The industry stopped turning the knob up and started spending the compute on thinking instead.

The full article develops the analogy parameter by parameter, with a careful caveat up front: these are analogies, not neuroscience. No substance has a top_p value, and a model is not a mind with loose filters. The comparison earns its place because a person and a model both face a version of the same decision — choosing among thousands of possible next steps, and deciding how much to trust the obvious one.

→ Read the full post on mielony.com.

Path-Tracing Doom on macOS: Metal Ray Tracing on Apple Silicon

lleqsnoom — Sun, 13 Sep 2026 14:30:45 +0000

There is no Vulkan ray tracing on macOS. Apple ships Metal, so a Vulkan RT app runs only through MoltenVK — and MoltenVK's ray tracing is an experimental, unmerged pull request, not an installable driver. On that missing layer, the 1993 game's path tracer runs: every pixel traced per frame, direct and bounced light, reflections, soft shadows — at ~44 FPS at 800p on an M4 Max.

The short version

I ported path-traced Doom 2 to Apple Silicon on a patched MoltenVK with Metal RT. It translates SPIR-V to Metal Shading Language and runs the same RTGL1 path tracer and Doom port as the Intel Arc article. The port ships as a self-contained zip (game + path tracer + patched MoltenVK + SDL); you supply Doom2.wad.

Download + every patch: https://mielony.com/blog/path-tracing-doom-on-macos/

Why macOS is different

On Intel Arc the bug class was "assumptions about the device." macOS goes further — there is no Vulkan loader at all. The game links MoltenVK (Vulkan-on-Metal), which stops at Vulkan 1.2 graphics/compute. Ray tracing (VK_KHR_acceleration_structure, VK_KHR_ray_tracing_pipeline) exists only on an unmerged MoltenVK PR (#2771), which maps the Vulkan RT entry points onto Metal behind MVK_CONFIG_ENABLE_EXPERIMENTAL_RAY_TRACING=1.

What broke, layer by layer

Driver. MoltenVK fails vkCreateDevice if any requested feature is unsupported, and the path tracer asks for its full set up front — the patch clears every feature bit the device does not report. SPIRV-Cross's MSL backend lacked Metal types for ray-tracing built-ins like gl_LaunchIDEXT, so every ray-generation shader failed to compile until a +21-line patch added them.
Renderer. An sRGB swapchain double-encoded the already display-ready image; a missing barrier loaded stale tiles — 8×8 and 16×16 artifacts on a tile-based Apple GPU.
Game. Metal surface from SDL_Metal_CreateView, true fullscreen, HighDPI matching the panel backing store (3456×2234), and a green FPS: n HUD counter.

Performance

Doom 2 demo1 on the M4 Max: 43.8 FPS at 800p, 51.7 FPS at 450p internal. rt_renderscale sets internal render height, and RT cost scales with it — not the window. For comparison, the Intel Arc A770 reaches 90 FPS at 450p internal on the same renderer.

Should you bother?

Only if the Mac is the point. It runs 44–52 FPS at low internal resolutions, has no music, stands on an open unmerged PR, and needs a commercial Doom 2 IWAD plus a ModDB lighting addon. If nothing ties you to macOS, Arc/Linux or an RTX machine is smoother — same renderer either way.

Full write-up with benchmarks, build scripts and every patch: https://mielony.com/blog/path-tracing-doom-on-macos/