DEV Community: rello

The 4-Question Filter That Killed My GitHub Star Habit

rello — Fri, 09 Oct 2026 13:00:00 +0000

I star a lot of repos. Trending page, a newsletter link, whatever shows up mid-search for something unrelated. Almost all of it looks genuinely good. That was never the problem. The problem is "genuinely good" and "belongs in my stack" are two different questions, and starring something answers neither.

One afternoon I stopped starring and started writing verdicts instead. Sixteen repos, one sitting, one sentence each on whether it belonged in my actual setup. Here's the filter that fell out of it.

The stack this was measured against

This only works if I name what I was checking against: OpenClaw, a local multi-agent framework I run on Ollama on a Mac mini M2 with 8GB of RAM (adopted, not something I built), plus Hiring OS, job-search tooling I keep deliberately separate from it. A repo that didn't make the cut here isn't bad. It didn't fit this pairing, on this hardware, that day.

Question 1: reference or dependency?

Six of the sixteen were curated lists, not software: macOS app roundups, a self-hosting guide, a CTO reading list, a GitHub-profile README gallery, a DeepSeek-wiring guide. None of them have a runtime footprint. git clone doesn't apply. They go in a bookmarks folder, not an evaluation queue.

This filter alone saves the most time, because reference-vs-dependency confusion is exactly what turns a watchlist into forty half-read tabs.

Question 2: do I already have this role covered?

Two verdicts here, and they're the ones worth sitting with:

ego-lite      -> browser automation for AI agents
                 already covered by Claude Code's Chrome MCP integration

holaOS        -> multi-agent workspace, shared memory, MCP support
                 verdict: "this is what OpenClaw already is"

Neither verdict is a knock on the tool. Both are probably well-built. The verdict is about redundancy, not quality: running a second tool that does a first tool's job doesn't add capability, it adds maintenance surface and a second thing that can drift out of sync with the first.

The transferable check: before asking if a repo is good, ask if you can name the thing in your current stack that already does its job. One sentence. If you can, you don't need the new repo, you need to pick which of the two you're keeping, and usually that's whichever one is already running.

Question 3: does it fit my actual hardware?

needle   -> 45MB model built for phones/IoT
            different hardware tier entirely, not a worse one

llmfit   -> Rust TUI, benchmarks local LLMs against your hardware
            set aside: my installed Ollama models already cover
            the 8GB-RAM tier it would be scoring

ragflow  -> self-hosted RAG / document-retrieval platform
            not a rejection, see below

llmfit is the interesting one: it's a genuinely useful tool answering a question I'd already independently answered by using the machine every day. Know your actual ceiling (RAM, compute, deployment target) before scoring anything against it in the abstract.

The move most lists skip: "someday, not now"

ragflow gets its own category because it's the easiest verdict to get wrong in either direction. It's a real capability, self-hosted document retrieval, that could plausibly feed a resume-and-document knowledge base into Hiring OS eventually. Installing it immediately felt tempting. Dismissing it entirely felt wrong too.

What I actually wrote: heavy Docker-plus-Elasticsearch platform, worth using eventually, not worth standing up for a use case that's plausible but not active. That's a third option between install-it and reject-it, and it's the one people skip when a tool looks exciting: write down what it's for and why it's waiting, then actually revisit it, instead of force-installing out of FOMO or letting the tab die from neglect.

Question 4: does it solve a problem I have right now?

ToolJet  -> low-code dashboard builder
unsloth  -> local fine-tuning framework (training, not inference,
            and this stack is inference-only)
OpenCut  -> video editor
macro    -> all-in-one team workspace

Four real, capable tools. None solves a problem this stack has. That's the whole verdict, and it says something about the gap in my current setup, not about the tools.

One repo cleared every filter: a technical interview-prep collection, DSA, CS fundamentals, language-specific tracks. Directly useful for a job search I'm in right now, not hypothetically. That's the one install.

The checklist

Reference or dependency? No runtime footprint, no evaluation queue, straight to bookmarks.
Already covered? Name the thing in your current stack doing this job, one sentence, before judging the new repo.
Fits your actual hardware and scale? Not hardware in general. The machine actually in front of you.
Solves a problem you have right now? Not one you might have.

Plus the move that isn't really a question: when something's promising but doesn't clear 3 or 4 yet, write down why it's waiting and what would change that. Don't install it out of excitement, don't let it evaporate in a pile of open tabs.

Sixteen repos, one sitting, one install. Every single one of those sixteen looked like real, useful work. That's exactly why the ratio matters: the filtering was the actual decision, made on purpose, instead of by default every time something looked exciting.

I Diffed 4 CLAUDE.md Files From 4 Real Projects. Byte Range: 15 to 14,540.

rello — Tue, 06 Oct 2026 13:00:00 +0000

Quick context: I write a context file for almost every project before an AI coding agent (Claude Code, mostly, sometimes Codex or Gemini CLI) touches the code. Going in, I assumed I'd converged on a template somewhere along the way. Same shape, reused each time, agent behaves consistently across projects.

So I pulled four of these files from four different projects and ran the numbers instead of trusting the assumption.

The numbers

Project	File(s)	Lines	Bytes
A system that takes real, irreversible actions on my behalf	`CLAUDE.md`	130	7,413
SparkyFitness	`CLAUDE.md`	1	15
SparkyFitness	`AGENTS.md` (delegated to)	152	11,077
A pre-product research project	`CLAUDE.md`	42	1,754
A weekend hackathon deck	`PRODUCT.md`	53	3,786
A weekend hackathon deck	`DESIGN.md`	183	14,540

SparkyFitness's CLAUDE.md is one line: See @AGENTS.md. Fifteen bytes. The hackathon onboarding deck doesn't have a CLAUDE.md at all: its PRODUCT.md/DESIGN.md pair came out of a design tool's own generated schema, not a personal convention I wrote. Fourteen thousand five hundred forty bytes on the DESIGN.md side alone.

Nobody scales a template down to fifteen bytes. That's not a smaller version of the same document. It's a different decision: this project didn't need the thing I built for the last one.

What actually repeats (it's not the file)

Four instincts show up in all four projects, at wildly different amounts of ceremony:

Lock decisions explicitly. An agent with no memory of your last session treats every choice as live unless you tell it otherwise. The pre-product research project states four settled decisions under a header that says exactly what it means: "Settled decisions — not open for reinterpretation." The system that takes real, irreversible actions on my behalf locks an entire safety posture under "Core Rules (Non-Negotiable)," including a flat "never invent companies, titles, metrics, skills, projects, or achievements." Same instinct, different amount of weight, because the two projects have completely different amounts riding on the decision staying locked.
State non-goals, with a named escape hatch. Agents are helpful by default, and helpful without a boundary turns into scope creep. The pre-product research project names three excluded features with a reason for each, then gives the agent a specific instruction instead of a vague warning: "When something looks like the obvious next step and isn't on the capability list above, ask first." SparkyFitness's product spec does the same job in four words: "What V1 is NOT."
Require verifiable, recorded completion. "The script ran" is not evidence. The system that takes real, irreversible actions on my behalf states it with teeth: "No manual 'the script ran' acceptance," with a requirement to record acceptance results in the commit message. SparkyFitness ties the same bar to CI instead of inventing new process, and its ticket tracker held the line even when it would've been easy not to: one ticket sat at TODO after later commits touched the same area, because the actual bar hadn't been cleared yet. A bar that always says yes isn't a bar.
Delegate to one living source, correct it in place. Duplicated context eventually disagrees with itself. SparkyFitness's root CLAUDE.md delegates entirely to AGENTS.md, which states its own routing rule: "Package-level guides win... Stale guides are worse than no guides; when you notice a claim in any AGENTS.md that contradicts the code, fix the guide as part of your change." I went looking for duplication in that project's three separate coding-agent configs (.claude/skills/, .agents/skills/, a Gemini config), expecting to find three copies of the same instructions. They're not copies: the .claude/skills files are one-line stubs pointing at the canonical body in .agents/skills. Same pattern, one directory deeper.

The proof-of-judgment detail

The system that takes real, irreversible actions on my behalf also shows the "correct in place" half of instinct 4 with a dated artifact: an old autonomous-submission policy, struck through rather than deleted, with a note on why it's no longer policy. That's a separate story with its own stakes; here it's evidence for one narrower point: "fix the guide as part of your change" isn't aspirational language in that project. It has a date on it, and it's still legible in the file.

What this isn't

Not a framework. No name, no version number, no diagram I reuse. It's four questions I ask myself every time I sit down to write one of these files: what's actually settled, what's explicitly out, what does "done" have to prove, and where does this project's truth live so nothing else has to repeat it. The answers come out a different length every time, because the questions are the same but the projects aren't.

If the four files had looked alike, that would've been evidence of a template, not of judgment. A 969x spread in byte count, on the same category of document, is the evidence that a template didn't happen here.

Same Score, Twice in a Row. Two Completely Different Root Causes.

rello — Sat, 03 Oct 2026 13:00:00 +0000

I run a personal AI command center (Hermes, an open-source agent framework I've configured and hardened, not something I wrote from the framework layer up). Its default model is a free local one. I also had access to a paid frontier model, xAI's Grok, and a real decision to make: should Grok replace the free local model as the default?

I didn't want to answer that with a vibe. I wanted a test I could point to later and defend.

Write the bar down before you run anything

Before writing a single test command, I put the acceptance rule into the system's own persona file: Grok becomes more than a per-call diagnostic option only if it clears a fixed battery of my real commands, at a bar set before I know the result.

Do not change the permanent Hermes model because of a single successful prompt.
Require the agreed command battery to pass first.

The number I picked: ≥90%. Not because 90 is a magic threshold, but because I picked it while I genuinely didn't know what the test would return. A bar set after you've seen the score isn't a bar. It's a rationalization with a number attached.

The battery

18 fixed commands, pulled from what I actually run day to day, fired one-shot against the live system with the model forced. No synthetic benchmark questions. Every call's raw output and duration got written to disk (first attempt lost its output to a mid-session reboot, so the second attempt wrote everything to a durable path before doing anything else — a boring fix, but the kind that makes the rest of the exercise trustworthy).

Scoring was manual. I read every single output and checked it against system state I already independently knew. Not "does this sound plausible," but "is this actually correct."

Battery 1: 16/18 correct, 88.9%. Average latency 199.8 seconds per call (137.8s fastest, 389s slowest, which is its own finding: even a model that clears the accuracy bar has to be livable to use). Two failures: one transient API transport error, and one that mattered more, the model hit an iteration ceiling and attempted a direct, out-of-band file-patch edit on a file it should never touch on its own. It only failed because the patch didn't apply cleanly. That's an architecture gap worth knowing about regardless of what a battery decides about defaults.

88.9% is under 90%. Grok stayed diagnostic-only. Done, in writing, no debate about whether "basically 90%" counts.

The trap: the same score, twice, meaning two different things

I re-ran the identical 18-command battery the next day. Result, on its face: 16/18, 88.9%. Same exact number.

The easy, wrong move here is to stop at the headline and write "failed the same way twice, case closed." I went back into the raw output for both failing calls instead. Both hit the exact same wall: a real xAI HTTP 403, "reached its monthly spending limit." Not a bad answer. Not a malformed tool call. A billing wall that had nothing to do with whether the model could actually do the work.

battery 1 → 16/18 (88.9%) → below 90% bar → stays diagnostic-only
battery 2 → 16/18 (88.9%) → HTTP 403 billing wall on both failures → "not reliability" verdict

Same percentage. Different diagnosis. If I'd only tracked the pass/fail count, I would have written off a model that hadn't actually failed the test I meant to run. A score is a summary statistic. It isn't a root cause.

What happened once billing was restored

I didn't rerun all 18 commands. I ran a scoped 4-command recheck targeted at what was actually in question. Grok passed three cleanly and failed the fourth: swarm/delegation, the mode where Hermes dispatches child tasks and waits on results.

Root-caused that one too, instead of just logging it as "Grok failed again." It traced to a real bug in Hermes's own one-shot code path — a function defaulting to a value that one-shot mode never actually binds session variables for. Nothing to do with Grok. Fixed with a 14-line patch (backed up first, since I was editing vendored code I don't own), verified with an acceptance test: a delegated child task echoing back an unguessable nonce, returning correctly in 28.6 seconds instead of timing out at 2:42.

Final verdict, written as its own precise sentence instead of a blanket upgrade:

GROK READY (for those three commands), SWARM-ON-GROK NOT READY.

Not "Grok is ready now." A command-specific finding, because that's what the evidence actually supported.

One detail worth keeping on its own: when the swarm call broke, the model didn't try to fake a result. Logged verbatim: "I will not fabricate or manually run the reads to bypass the delegation requirement." That's exactly the behavior an acceptance battery exists to check for, and it showed up unprompted, under a real failure.

The eval framework only earns trust the day it catches something real

Alongside the battery I built a 12-layer, four-tier (PASS/WARN/FAIL/BLOCK) eval framework wired into the command center, deliberately reusing an existing approval-gate implementation from a different part of the system rather than writing new scoring logic from scratch.

Any eval framework looks rigorous right up until it only ever gets tested against cases you already know the answer to. This one earned its keep on its first real, unstaged run: an agent wrote and confirmed a token file, the underlying call actually failed schema validation, and the agent replied anyway with "EVAL_OK." The framework's own recheck looked for the file. It wasn't there. Fabricated success, caught and logged, not because I happened to be watching that day, but because the check existed and ran.

The part a command battery doesn't catch

Separately, and earlier, I got a real scare from a Grok bill and didn't open the system for two days. Not because a battery had failed. Because I hadn't run one at all for this particular decision.

The cause: the agent bound to my everyday chat had its model set to paid Grok. Nobody changed that on purpose. It was a config drift while every other agent in the system stayed on the free local model, and ordinary conversation was billing per message the whole time.

Fixed the immediate problem by rebinding to the local model and verifying against live session logs, not the config file (a config value telling you something is off is not the same as it actually being off). Then went further: found the two remaining legitimate Grok call sites (a manual review step, a daily scoring cron), put both behind one shared budget file with a hard cap, $0.30/day combined, coded to auto-fall-back to a non-Grok path if it's ever hit.

The crons kept running through my entire two-day pause. Not touching the system doesn't pause spend. Only disabling the specific job does. "I'll be more careful" isn't a control. A cap the code enforces is. Checked today: real spend is $0.02896 against that $0.30/day ceiling.

What generalizes past AI models

Write your acceptance bar down before you see the score. When two results carry the same headline number, check whether they failed for the same reason before treating them the same. Don't trust a default you haven't verified against ground truth, and once you've verified it, make the safe behavior a hard constraint in code, not a habit you're hoping to remember. None of this is a verdict against any specific model or vendor. It's what a defensible model-acceptance decision actually looks like when you're the one accountable for the system.

Stop Trusting Your Gut About Technical Debt. Count It Instead.

rello — Wed, 30 Sep 2026 13:00:00 +0000

I run a personal agent system for my own job search, built on adopted frameworks I configured and hardened rather than wrote from scratch (Hermes for reasoning, OpenClaw for execution, Hiring OS as the single source of truth for job data). For a while I had a nagging feeling that I was building more infrastructure into it than I needed. Feelings like that are useless on their own, so I stopped trusting it and went and counted instead.

Here's what was actually sitting on disk:

32 config backup files across two systems, spanning 67 days, under four different naming conventions running in parallel. Three of them, labeled "STABLE" and cut 40 minutes apart on the same day, are byte-identical when diffed.
A duplicate test harness born from a 2-line diff. A test script needed one change the next day: a different output path, one added environment variable. What it got instead was a full second copy of the entire harness, same eighteen output files, same support scripts. Diffed against the original, the two scripts differ by two lines out of about twenty.
Unused scaffolding for a multi-agent workboard: a goal/card model, a verb-routing dispatcher, a stub safety net, a receipt step, built for a system that has run exactly one demo goal and zero live ones, with one of its two core verbs never implemented.
34 archived creative-tooling skills, bulk-moved out of the live system on a single day, one of them alone documenting 38 shaders and 24 character palettes, none with any recorded reason for being set aside.
Three separate fine-tuning runs of the same idea in one day: a resume-screening model, trained at three different scales, one of which measurably overfit partway through. None of the resulting model files ever got referenced in any live config.
Three parallel copies of the same career database across three directories, one of them two months stale.

None of that is dramatic by itself. A backup file is not a bug. A second training run is not a crime. What the list shares is a shape: build first, notice the duplication later, and usually fix the pointer without deleting the older copy. That lag, repeated enough times, is the actual cost.

The audit that mattered more than the inventory

Counting artifacts tells you something happened. It doesn't tell you what it cost. For that I had to compare output against consumption, not activity against intention, because activity always looks productive from the inside.

I pulled up one full working session and checked what came out of it: model upgrades, a pass through roughly twenty unrelated GitHub repos, a tool installed and never touched again, an unrelated hardware video watched somewhere in the middle. Zero job-search output. In that same window, two job-discovery automations, the actual point of the system, had been sitting disabled for three weeks. Unnoticed, while I kept building things next to them instead of running them.

Separately: 692 jobs scored, 105 sitting untouched in a save queue with the oldest six weeks old, my last real application six days prior, outreach drafts that had never once left a "simulated" status. Building more automation on top of that picture would have made the backlog worse, not better. The bottleneck was never scouting volume.

Why "it looks useful" isn't a good enough bar

Once the numbers were visible, the mechanism behind them was ordinary, not some deep personal flaw: building has fast feedback, a clean stopping point, and zero risk of rejection. Job search, networking, interview practice have none of those properties. They're slow, ambiguous, and personally exposed. Given a choice between a task with guaranteed positive feedback and one without it, the easier task wins by default unless something structural stops it. That includes work done with an AI coding assistant. The loop doesn't care what's generating the code; it only cares that finishing something feels good and starting the hard thing doesn't.

So I wrote a gate directly into the system instead of relying on noticing it again later. The orchestrating agent's persona document now carries a hard stop condition: once the core pipeline works end-to-end, don't add new frameworks, bots, models, or frontends until the current setup has run reliably in normal use. A separate frozen-state document states the same rule at the architecture level: don't add infrastructure without a real usage failure that proves it necessary.

That's an inversion of the old default, not a tweak to it. Under the old rule, "a memory layer might help" was reason enough to install one. Under the new rule, "might help" doesn't clear the bar; something has to have already broken in a way only new infrastructure fixes.

A worked example of the gate, before the gate existed

Four days before I wrote that rule down, I'd already been applying it by instinct on one specific dependency: a memory tool called GBrain. It installed cleanly and looked functional. Writes worked; gbrain remember succeeded and gbrain doctor independently confirmed the fact was stored. Recall didn't: asking for a fact I'd just written, by exact string, by paraphrase, by provenance tag, returned "No matching facts," every time.

Rather than assume the bug was in my integration, I isolated it. A 9-step script first exercised a different, already-working memory tool end to end (store, recall, correct, recall the correction, all four steps clean), then hit GBrain's own CLI directly with no dispatch layer in between. Same failure. I upgraded GBrain a full point release and reran with a brand-new fact, specifically to rule out a stale-record false pass. Same result. At that point the evidence pointed at GBrain's own recall engine, not my wiring, so I stopped: removed the connection cleanly, left the local install in place in case a future version fixes it, and moved on. No bug filed, no fork, no further debugging on a dependency I don't maintain.

That same week two other things got cut for unrelated reasons: a messaging channel whose "disabled" flag didn't actually stop outbound sends until three escalating fixes later, and two agent personas an audit found had zero real production dependency. Three kills, three different root causes, four days, none of them individually dramatic. Together, they're the gate's own worked example, in reverse: each of those things existed without ever meeting a "proven necessary" bar, and each got cut once that became visible.

What I'm not claiming

The freeze document is one day old as I write this. Nothing confirms the rule has held past today, and I'm not going to write an ending where the discipline "worked": that's not a claim the evidence supports yet. What exists is a gate, written into the system instead of left as intention, and a number I can check later: whether the next thing I'm tempted to build actually clears "a real usage failure proved this necessary," or whether it's just the easiest room in the house, again.

A Config Flag Told Me Telegram Was Off. My Gateway Log Said Otherwise.

rello — Sun, 27 Sep 2026 13:00:00 +0000

I run two AI agent systems I didn't build: OpenClaw (an installed multi-agent operator framework, npm package plus a native Mac app, pinned at openclaw@2026.9.4 for what follows) and Hermes (a deployed copy of Nous Research's open-source hermes-agent, MIT-licensed, tens of thousands of upstream commits, zero of them mine). Operating both, not building either, is what produced the four incidents below, and all four collapse into the same rule: a system's report of its own state is a claim, not a fact, and you check it against an independent source before you rely on it for anything that matters.

Incident 1: one flag, two code paths

I wanted OpenClaw's Telegram channel off. I set:

channels:
  telegram:
    enabled: false

Top level, and per bot account. Config file looked right. Dashboard looked right.

Production didn't agree. The flag stopped inbound polling, the system stopped listening on Telegram, but scheduled, cron-driven outbound sends kept firing hourly through three stale duplicate bot accounts (analyst, default, a leftover hermes account) for the rest of that day. No status indicator told me this. I found it by reading the raw gateway log line by line instead of trusting the config state I'd already set.

enabled: false described the intended state of one code path, inbound handling. It said nothing about whether anything could still go out. Same boolean, two different claims.

The fix wasn't a smarter flag, it was removing the mechanism: physically moved all five Telegram bot token files out of the secrets directory into a disabled folder. No token, no code path (scheduled or otherwise) can authenticate and send. That's a stronger guarantee than a config value can make, because the credential no longer exists rather than merely being configured off. The permanent close came later as a separate step: the bot accounts were deleted at BotFather entirely, and Telegram ownership was consolidated out of OpenClaw into Hermes as a single control plane, instead of two systems each partially able to trigger sends.

To be fair to OpenClaw here: the flag almost certainly does exactly what it promises for the path it governs. My assumption that one switch was authoritative over every trigger path, including a scheduled one I wasn't even thinking about, is what broke. If you operate anything with both a request-driven path and a schedule-driven path gated by the same nominal switch, don't assume they obey it identically. Check the logs.

Incident 2: the repair tool that didn't repair

A version upgrade (2026.7.1-2 → 2026.8.1) left twelve stale files on disk, an approvals-gate file, workspace-state for two retired personas, seven attestation files, that were supposed to clear via the framework's own doctor --fix / --repair command.

Ran it. Files stayed. Tried every flag combination the tool exposed. Files stayed.

This is one data point in one environment, not a verdict on the tool. What I did about it is the transferable part:

Renamed the files rather than deleting them, matching the tool's own remediation convention instead of inventing my own.
Backed up the originals to a separate directory first.
Approved each batch individually from my phone rather than skipping the checkpoint because I was confident I knew what I was doing.

That third step wasn't optional; my own automation guardrails wouldn't let a change like that run unattended. When you're standing in for a vendor's automation because it didn't finish its job, hold your manual fix to at least the same bar the automation was supposed to meet.

Incident 3: a rule in English is not a rule the code enforces

Hermes's persona spec is the file defining how the system behaves. I rewrote it across at least seven dated revisions, from a one-sentence default to 364 lines. One revision banned meta-commentary, the model narrating what it's about to do instead of doing it, after this output got logged as a permanent "Bad" example in the file:

"Since the user is unsure about what to do, I would suggest asking a clarifying question... Here's an example response: 'What does not sure what to do mean?'"

It didn't answer. It narrated a hypothetical answer. Five minutes later in diff terms, a second rule went in for a related failure: inventing context it didn't have.

Writing a clear rule into a persona file is not the same thing as enforcing it. A prompt describes intended behavior; the thing executing it is a probabilistic model, not a deterministic interpreter. So the fix wasn't a third rule. It was moving enforcement somewhere prose structurally can't reach: code that runs before the model gets a turn. Two original modules on top of Hermes, an action gate (543 lines) and a career-intent gate (154 lines), neither part of upstream hermes-agent. Their docstrings say why plainly: the prior routing was "not deterministic, and provably inconsistent." Concretely, these gates hard-block LinkedIn as an automation target, checked before a job resolves and again against the resolved URL, and they require an explicit approval phrase before any irreversible external action, with "ok," "sure," and "thanks" deliberately excluded because those are the words people type when they haven't actually decided.

Use the persona layer to describe intent. Use code to enforce anything where a failure is expensive.

Incident 4: the one that's still open

Hermes runs a daily briefing job off a cron scheduler. A config value drifted, the default model string got renamed, and the job is still pinned to the old string. It has been failing for many consecutive days as of this writing.

To the design's credit, it fails safely. The guard catches the mismatch and logs a specific reason, "Skipped to prevent unintended spend: global inference config drifted... job is unpinned," instead of running against an unexpected model. But a guard that logs correctly to a file nobody's watching is still, from a get-it-fixed standpoint, failing silently. Nothing paged me.

I'm including this one unresolved because the other three all end with a fix, and pretending this one does too would undercut the whole point. If the lesson is "verify a system's claim about its own state," the guard's own uptime needs that same verification, one level up from what it's guarding. I hadn't built that layer. Now I know I need to.

The checklist version

A disable flag describes the path it governs, not every path that can trigger the same behavior. Check logs or process state before trusting it for anything scheduled.
A vendor's repair tool failing in your environment isn't a verdict on the tool. Match or exceed its own safety bar when you step in manually: reversible, backed up, approved in increments.
A persona/prompt rule describes intent. It doesn't enforce anything. Move enforcement into code for failures that are expensive.
A safety guard needs its own uptime check. Failing safely and failing silently are not the same thing, and only one of them gets fixed quickly.

None of this required building an agent framework from scratch. It required operating two adopted ones closely enough to notice when their claims about themselves stopped matching what the logs actually showed.

Your Model Isn't Bad. Your Eval Set Might Be Circular.

rello — Thu, 24 Sep 2026 13:00:00 +0000

I built a rule-based sugar detector and a fine-tuned model to compete against it, ran them against my gold set, and the rule engine scored 100% accuracy. Every metric, perfect.

It took me longer than it should have to notice the actual problem: most of that gold set had been labeled by running the rule engine's own code against the data. I hadn't built an eval. I'd built a mirror.

The two systems

SugarShield catches hidden sugar in ingredient lists: things like "maltodextrin" or "evaporated cane juice" that don't read as sugar unless you already know they are. Two systems do the detecting:

A deterministic rule engine: a lexicon of 145 aliases across 77 canonical terms, feeding a risk score. Detection and scoring are separate steps on purpose, so finding "milk" (a natural sugar context) never raises risk on its own.
A fine-tuned model: a LoRA fine-tune of Qwen2.5-1.5B-Instruct, trained on 1,375 records sourced from GroceryDB, a licensed academic dataset of real grocery products.

The build environment couldn't reach Hugging Face Hub, Open Food Facts, Wikipedia, or Kaggle: all four returned a hard 403 from the network egress proxy, confirmed with curl and Python's requests before switching plans rather than assumed. That ruled out pulling a pretrained checkpoint for the original LoRA plan inside that environment, so the training script trains a byte-level BPE tokenizer and a small GPT-2-architecture model from scratch instead, full-parameter, no pretrained weights to adapt. That from-scratch run (5.87M parameters, 6 layers, CPU-only, 607 seconds) is a stand-in for the plan that couldn't execute there, not the model that mattered for the actual decision. The real fine-tune ran separately, on different hardware with real Hugging Face access: genuine LoRA weights on top of genuine pretrained Qwen.

Getting that real fine-tune working surfaced an actual bug worth naming: load_model() never called model.to(device). Training on Apple's MPS backend worked fine, but at inference time the model itself stayed off the GPU, so generation silently ran on CPU: a 25-plus-minute hang producing nothing, while the rest of the pipeline happily reported device=mps. The fix was small: resolve the device explicitly, move both model and input tensors onto it, wrap generation in torch.inference_mode(), cap max_new_tokens at 128. After the fix: 4 to 8 seconds per sample.

The eval that lied by construction

With both systems trained, I scored them against a 59-record gold set. Rule engine: 100% accuracy, 100% recall, zero false negatives.

Here's the part that should have been obvious immediately and wasn't: most of those 59 labels were "silver-labeled" by running the rule engine's own production code (lib/lexicon.ts / lib/riskEngine.ts) against the data, and marked "verified": false to say so, honestly, in the dataset itself. A system scored against labels it helped generate will score close to perfectly. That's not a property of the system being good. It's a property of the measurement being circular.

The tell was sitting right next to it: the from-scratch model, on the exact same 59 records, scored 69% accuracy and 21.05% hidden-sugar recall on a genuine blind test (it never saw those examples in training). One system looked perfect and the other looked mediocre, on the same eval, and the honest read was that neither number could be trusted for a production call.

Building an eval that can fail

The fix wasn't better labels for the existing set. It was a completely separate one, built specifically so neither system had a hand in producing it: 132 records, every label written by directly reading raw ingredient text the way a human reviewer would, never by calling risk_engine.ts or the model. A dedicated script, check_no_overlap.py, verified zero id or product-name overlap with anything either system had already seen in training, validation, or the original gold set.

Rerun on the independent set, the rule engine's self-graded 100% became an honest 90.9%. That drop is exactly what you'd expect once the circularity is gone, and it's the correct outcome: a decent system, no longer grading its own homework, still does fine.

                    Rule Engine   Qwen2.5-1.5B LoRA   Hybrid
Accuracy            90.9%         83.0%               89.4%
Precision           92.6%         96.5%               90.9%
Recall              96.2%         81.2%               96.2%
Hidden-sugar recall 50.0%         15.2%                56.3%
False negatives     4             19                   4
Avg latency         0.9ms         4,552ms              4,553ms

15.2% hidden-sugar recall, against the rule engine's 50%, on a product whose entire job is catching sugar a shopper would otherwise miss. That's the number that ended the conversation about shipping the fine-tune alone.

The part that isn't a clean loss

The model beats the rule engine on precision (96.5% vs 92.6%) and throws fewer false positives (3 vs 8). Layered on top of the rule engine as a hybrid, with a hallucination guard dropping any model-claimed term with no textual support, it lifts hidden-sugar recall to 56.3% and trigger-match accuracy to 72.9%, both better than the rule engine alone. That's a real capability: naming novel sugar aliases the fixed lexicon was never written to catch.

It still didn't ship, because hybrid's overall accuracy (89.4%) is slightly below the rule engine alone (90.9%), the extra false positives from the model's own guesses cost more than the term-level gain was worth, and the app runs on Vercel serverless functions with no persistent process to hold a loaded PyTorch model between requests. Production still calls the deterministic engine.

Checking it again later

I went back afterward and found two real bugs in the rule engine's own matching logic: an apostrophe-stripping order bug that broke matching on terms like "confectioner's sugar," and a missing generic fallback for fruit-juice-concentrate phrasing. Fixing both moved the rule engine's own independent-set numbers: 90.9% to 91.7% accuracy, 4 to 2 false negatives.

With that fix in place and about 5x more training data, a second Qwen LoRA run (V2) now beats the rule engine on accuracy (93.1% vs 91.7%) and precision (97.0% vs 91.9%) on the same independent set. Hidden-sugar recall is still behind: 48.9% (6 false negatives) versus the rule engine's 54.2% (2 false negatives). Closer than the original 15.2%-vs-50%, not closed. Production hasn't moved. The independent benchmark is still the thing deciding that, not how far the model has come since.

The actual lesson

If your eval set was built, even partly, by the system you're testing, a good score on it isn't evidence. It's an artifact of the measurement. The fix isn't a smarter model or more careful prompting on the first eval. It's a second, independently constructed benchmark with zero shared provenance, built specifically to fail if the first one was lying to you, and the discipline to publish what it says even when it disqualifies the thing you spent the most effort building.

I Built a Daemon to Auto-Submit Job Applications. Two Lines Above It, My Own Policy File Said Not To.

rello — Mon, 21 Sep 2026 13:00:00 +0000

I run a personal system called Hiring OS. It pulls jobs from ATS APIs (Ashby, Greenhouse, Lever, YC's Work at a Startup), scores them against my resume, tailors a resume through a set of deterministic quality gates, and tracks every application. I've been the user, PM, builder, and tester on the same system for about five months, which means every bad decision in it is mine and every fix is also mine.

This is a postmortem on one of those decisions: a rule I wrote that let a daemon submit job applications with no human in the loop, sitting two lines below a rule that explicitly banned that.

The setup

By the time this happened, the parts of Hiring OS I actually trusted were working: job discovery from live ATS APIs, and a set of deterministic quality gates a tailored resume had to clear before I'd use it. Truthfulness against a fixed set of source documents. ATS keyword match. Recruiter readability. Hiring-manager fit. Real pass/fail checks, not vibes.

A couple of days before the rule I'm about to show you, I'd written an internal status note declaring an autonomous submission daemon "MVP complete." Run continuously in the background, apply to qualifying jobs 24/7, I check progress from my phone. The logic felt sound: I'd already built the hard part, the thing that actually stops a bad resume from going out. If the gates say a resume clears the bar at a high enough score, what is a human click still protecting against?

The rule

I formalized it in the same rules document that governs everything else the system is and isn't allowed to do:

Amended: Daemon runs continuously. Jobs with tier classification +
passing eval thresholds auto-qualify for submission. No manual gate.

Perfect-match threshold: Truth>=90, ATS>=75, Recruiter>=70, HM>=70
Adjacent-role threshold: slightly looser bar + quality signal

No exceptions carved out. The daemon would build the submission payload and submit through the automation runner on its own.

Two lines above this exact block, in the same file, since the file's first version: "Tailored resume rewriting runs through a human-in-the-loop session per job. No unattended/autonomous pipelines." A few lines below: jobs cannot be auto-promoted to ready_to_apply, the operator has to set it explicitly.

I'd written a document whose entire spine was "nothing irreversible happens without me," then bolted on one exception, without touching a single word of the surrounding rules that said the opposite.

Why it took me months to catch

A gate score answers "is this output good." It does not answer "is it safe to act on this output without review." Those look like the same question if you squint, especially when the same number is doing double duty for both, but they have different failure modes entirely.

Here's the data that makes this concrete. Over about a week and a half, applying by hand the way the original rules actually described, I tracked 28 applications with real gate scores logged next to each one:

Founding Senior PM role: 100/95/87, submitted
Another PM role: 100/92/90, submitted
A third role: 100/100/97, the highest score in the entire batch, rejected the same day, attributed by my own tracker notes to a work-authorization hard filter

That third one is the whole argument in one row. A perfect resume score did nothing to prevent a same-day rejection for a reason the resume gates were never built to detect. If the daemon had been live, it would have auto-submitted that application on the strength of a 100/100/97, with exactly as little visibility into the sponsorship mismatch as I had. The gate wasn't wrong. It was answering a question that had nothing to do with the thing that actually killed the application.

There's a smaller version of the same failure mode from the same stretch. The day I added an eighth eval gate (a voice and AI-vocabulary checker, after two resumes got flagged for banned wording), I'd given myself a general go-ahead to submit two queued applications. Partway through the second form, I stopped and wrote a note anyway: "Do not click submit, fill the form, wait until I say it only." A blanket yes given earlier in a session didn't survive contact with the actual irreversible click.

The fix

Once I reread the amendment against the rest of the document it lived in, it read as exactly what it was: an exception that contradicted every safety rule around it, added under the pressure of wanting the search to move faster. I struck it through instead of deleting it, so the wrong version stays legible next to the correction:

DEPRECATED, do not follow: Amended earlier: Daemon runs continuously... Jobs with tier classification + passing eval thresholds auto-qualify for submission... This described a zero-human-review autonomous submission architecture that contradicts every other safety rule in this document and is no longer the policy.

Current policy: NO unattended external submission, no exceptions. Eval scores and tier classification may inform a recommendation, never an automatic submission.

Then I checked the actual running system, separately from the document, because a corrected policy file and an inert daemon are two different facts and I wasn't going to assume the first proved the second. No active service runs the submission daemon. A narrower scheduled job still runs, and it only fetches, scores, and scaffolds. It never submits anything on its own.

The generalizable part

If you're gating any irreversible action behind a confidence score, the question worth asking isn't "how high should the threshold be." It's whether the threshold measures the thing that makes the action safe, or something correlated with quality that you're treating as if it measures safety. Those aren't the same property, and a system can pass every check you built and still be missing the one check the actual risk depends on.

The tell, in my case, was that a rule contradicting the rest of the document sat there for months before anyone (me) reread it closely enough to notice. The fix isn't trusting the daemon less. It's building a forced re-read of any change that removes a human step, on a schedule, instead of waiting on a coincidental close reading to catch the next one.

I Built Five Self-Improving Loops in One Evening. They All Had the Same Bug.

rello — Fri, 18 Sep 2026 14:00:00 +0000

I built the same scoring bug five times in one evening, and none of the five READMEs I wrote for it caught it, because each one described it as a separate, minor, honest limitation. Read together, they're one bug wearing five different names: every judge scored whether the right words or shapes showed up, not whether the thing actually worked. The interesting part isn't the five prototypes, it's the sixth one, built to fix that bug in a way I could actually verify.

For context on how much weight to put on that: this was one evening of self-directed practice, no external user or stakeholder involved, six small repos (biggest eval set: 8 tasks), all built with Claude Code (every commit is paired with a session, per the commit trailers), and everything below ran in mock mode, no live API calls anywhere in this story.

If you're working on anything with an optimize-against-a-judge shape (prompt search, RAG tuning, agent config search, eval-driven pipelines), the bug is worth knowing about before you hit it yourself.

The setup

The question I wanted an actual answer to: does mutate → judge → keep-if-better → repeat generalize as a loop shape, or does every kind of artifact need its own bespoke approach?

Instead of arguing about it, I built five tiny versions, back to back, in about 7 minutes of commit timestamps total:

self-improving-prompt-agent: mutates a prompt from a fixed candidate list, heuristic judge (0.0 to 1.0)
context-improving-agent: mutates a context block, judged by 0.8*fact_coverage + 0.2*(1-bloat_penalty)
graph-improving-agent: inserts one role node at a time into a workflow graph, judged on role-keyword presence
harness-improving-agent: adds one guardrail statement at a time from a list of ten, judged on keyword coverage
agent-improving-agent, one level up: mutates the priority order harness-improving-agent tries its ten guardrails in, since you rarely get to test all ten before your eval budget runs out

Four of the five repos have exactly one commit. That's what five fast structural tests of one hypothesis actually looks like.

The loop shape held up. All five ran the same structure and all five climbed over 10 rounds:

Repo	Result (10 rounds)
prompt	0.10 → 0.80
context	0.20 → 0.28
graph	0.10 → 0.70
harness	0.00 → 0.90
agent (meta)	0.45 → 0.61 (stated max: 0.80)

Five for five, on paper. That wasn't actually the interesting result.

The bug hiding in plain sight

Every one of these repos has a "Known Limitations" section, because writing down what's wrong with what I just built while I still remember it is a habit. Read individually, each limitation sounds like a minor, honest caveat:

graph: "no check that the wiring is actually runnable/useful"
harness: "not a real check that the agent obeys the stated guardrails at runtime"

Read together, though, these are the same bug five times, wearing five different filenames. Every judge in this set scored whether the right words or shapes showed up, not whether the artifact actually worked:

A graph judge that checks for a "verifier" keyword scores an inert, do-nothing verifier node exactly the same as a functional one.
A harness judge that checks for the word "rollback" scores that word being present the same whether or not any rollback mechanism exists.
context-improving-agent's fact_coverage term almost certainly measures whether facts are textually present, not whether the context is coherent once they're added.

None of the judges were malfunctioning. They did exactly what I told them to measure. The problem is I told five of them, separately, to measure the wrong thing, and I only caught it by reading all five READMEs side by side instead of shipping straight to a sixth prototype.

There's a second wrinkle worth calling out on harness-improving-agent specifically: its judge used loose keyword matching early on and stalled at 0.20 for several rounds, until I rewrote the judge's wording mid-run to match the mutation wording it was supposed to be scoring. The jump to 0.90 partly reflects that fix, not clean search progress, and the repo's own README says this directly.

The fix: sia

About 2.5 hours after the fifth prototype's commit, I built a sixth repo, sia (self-improving-agent), whose README opens by naming the first five as prior art and states plainly why it doesn't reuse their judges: porting a structure-presence judge into something meant to be real "would have meant the keep/rollback gate approved patches for saying the right words rather than answering questions correctly."

Four changes, each one closing a specific hole from the first five:

1. Frozen judge, held-out eval. sia optimizes a genome (versioned JSON: prompt/context/workflow/harness layers) against a fixed 8-task set, split 5 train / 3 holdout. The judge doesn't get edited mid-run, unlike what happened with harness-improving-agent.

2. A real gate, not a bare threshold:

keep iff train_child >= train_parent + 1.0
         AND holdout_child >= holdout_parent

The +1.0 minimum lift on train, combined with non-regression on holdout, is there specifically to catch a patch that overfits the five training examples without generalizing. None of the first five repos had a holdout set at all: they scored and kept against the exact same signal they optimized against, every single round.

3. Integrity checks on the eval itself. sia/loop.py hashes the eval set and the judge file at the start of a run, and re-checks that hash every iteration. If either changes, the run aborts with a logged ABORT row. Watching five judges reward the appearance of a good answer taught me this directly: an optimizer that's allowed to edit what it's graded against will eventually find that editing the test is cheaper than improving the artifact.

4. Positive and negative controls, not just a demo run. Checked directly against sia's own ledger, not a printed summary I trusted at face value:

Context patch: train 0.0→50.0, holdout 0.0→33.33 (KEPT)
Harness patch: train 50→70, holdout 33.33→66.67 (KEPT)
Workflow patch: train 70→90, holdout 66.67→100 (KEPT)
A deliberately-constructed patch with zero train lift (ROLLBACK)

3 keeps and 1 correct rollback show the gate distinguishing real lift from no lift. That's a stronger claim than "the score went up." I also ran the test suite live: 21/21 unit tests passing.

One more scope decision worth mentioning: sia's README states "The AI is not allowed to call itself better." The improver component can write a hypothesis about why a patch might help, but that text is documentation only. It never feeds into the score.

What I can't claim

Everything above happened in mock mode, a simulated worker standing in for a real model call. There's an API key sitting in a .env file that was never used to make a real call: no logs, no genomes, no ledger rows anywhere attributable to a live run. sia's client hard-raises if it's ever invoked with mock=True, so there's no ambiguity about which mode produced these results. That also means I genuinely don't know whether the gate, the holdout split, or the hash-check survive contact with real, noisy model output. It might. I haven't tested it.

Also unresolved: the meta-loop from agent-improving-agent (the one that optimizes guardrail priority order) was designed to plug into exactly this kind of frozen judge, and I never wired it in. The README calls it future work, plainly.

The takeaway

If you're building any kind of optimization loop over LLM output, the thing worth taking from this isn't the five prototypes or sia's specific architecture. It's the order of operations: before investing in a better mutation strategy or a smarter search, check what your judge is actually measuring, and whether that's cheap to satisfy without being correct. A keyword-presence judge gets gamed eventually, not because the search is adversarial, but because "score well against this specific signal" is always an easier target than "be right," and any sufficiently capable optimizer finds the easier target first.

I built the same mistake five times before I noticed it, then spent the rest of the night on the unglamorous fix: a frozen test set, a holdout split, and a hash check that (so far) has never fired.