DEV Community: rello

I Built a Daemon to Auto-Submit Job Applications. Two Lines Above It, My Own Policy File Said Not To.

rello — Mon, 21 Sep 2026 13:00:00 +0000

I run a personal system called Hiring OS. It pulls jobs from ATS APIs (Ashby, Greenhouse, Lever, YC's Work at a Startup), scores them against my resume, tailors a resume through a set of deterministic quality gates, and tracks every application. I've been the user, PM, builder, and tester on the same system for about five months, which means every bad decision in it is mine and every fix is also mine.

This is a postmortem on one of those decisions: a rule I wrote that let a daemon submit job applications with no human in the loop, sitting two lines below a rule that explicitly banned that.

The setup

By the time this happened, the parts of Hiring OS I actually trusted were working: job discovery from live ATS APIs, and a set of deterministic quality gates a tailored resume had to clear before I'd use it. Truthfulness against a fixed set of source documents. ATS keyword match. Recruiter readability. Hiring-manager fit. Real pass/fail checks, not vibes.

A couple of days before the rule I'm about to show you, I'd written an internal status note declaring an autonomous submission daemon "MVP complete." Run continuously in the background, apply to qualifying jobs 24/7, I check progress from my phone. The logic felt sound: I'd already built the hard part, the thing that actually stops a bad resume from going out. If the gates say a resume clears the bar at a high enough score, what is a human click still protecting against?

The rule

I formalized it in the same rules document that governs everything else the system is and isn't allowed to do:

Amended: Daemon runs continuously. Jobs with tier classification +
passing eval thresholds auto-qualify for submission. No manual gate.

Perfect-match threshold: Truth>=90, ATS>=75, Recruiter>=70, HM>=70
Adjacent-role threshold: slightly looser bar + quality signal

No exceptions carved out. The daemon would build the submission payload and submit through the automation runner on its own.

Two lines above this exact block, in the same file, since the file's first version: "Tailored resume rewriting runs through a human-in-the-loop session per job. No unattended/autonomous pipelines." A few lines below: jobs cannot be auto-promoted to ready_to_apply, the operator has to set it explicitly.

I'd written a document whose entire spine was "nothing irreversible happens without me," then bolted on one exception, without touching a single word of the surrounding rules that said the opposite.

Why it took me months to catch

A gate score answers "is this output good." It does not answer "is it safe to act on this output without review." Those look like the same question if you squint, especially when the same number is doing double duty for both, but they have different failure modes entirely.

Here's the data that makes this concrete. Over about a week and a half, applying by hand the way the original rules actually described, I tracked 28 applications with real gate scores logged next to each one:

Founding Senior PM role: 100/95/87, submitted
Another PM role: 100/92/90, submitted
A third role: 100/100/97, the highest score in the entire batch, rejected the same day, attributed by my own tracker notes to a work-authorization hard filter

That third one is the whole argument in one row. A perfect resume score did nothing to prevent a same-day rejection for a reason the resume gates were never built to detect. If the daemon had been live, it would have auto-submitted that application on the strength of a 100/100/97, with exactly as little visibility into the sponsorship mismatch as I had. The gate wasn't wrong. It was answering a question that had nothing to do with the thing that actually killed the application.

There's a smaller version of the same failure mode from the same stretch. The day I added an eighth eval gate (a voice and AI-vocabulary checker, after two resumes got flagged for banned wording), I'd given myself a general go-ahead to submit two queued applications. Partway through the second form, I stopped and wrote a note anyway: "Do not click submit, fill the form, wait until I say it only." A blanket yes given earlier in a session didn't survive contact with the actual irreversible click.

The fix

Once I reread the amendment against the rest of the document it lived in, it read as exactly what it was: an exception that contradicted every safety rule around it, added under the pressure of wanting the search to move faster. I struck it through instead of deleting it, so the wrong version stays legible next to the correction:

DEPRECATED, do not follow: Amended earlier: Daemon runs continuously... Jobs with tier classification + passing eval thresholds auto-qualify for submission... This described a zero-human-review autonomous submission architecture that contradicts every other safety rule in this document and is no longer the policy.

Current policy: NO unattended external submission, no exceptions. Eval scores and tier classification may inform a recommendation, never an automatic submission.

Then I checked the actual running system, separately from the document, because a corrected policy file and an inert daemon are two different facts and I wasn't going to assume the first proved the second. No active service runs the submission daemon. A narrower scheduled job still runs, and it only fetches, scores, and scaffolds. It never submits anything on its own.

The generalizable part

If you're gating any irreversible action behind a confidence score, the question worth asking isn't "how high should the threshold be." It's whether the threshold measures the thing that makes the action safe, or something correlated with quality that you're treating as if it measures safety. Those aren't the same property, and a system can pass every check you built and still be missing the one check the actual risk depends on.

The tell, in my case, was that a rule contradicting the rest of the document sat there for months before anyone (me) reread it closely enough to notice. The fix isn't trusting the daemon less. It's building a forced re-read of any change that removes a human step, on a schedule, instead of waiting on a coincidental close reading to catch the next one.

I Built Five Self-Improving Loops in One Evening. They All Had the Same Bug.

rello — Fri, 18 Sep 2026 14:00:00 +0000

I built the same scoring bug five times in one evening, and none of the five READMEs I wrote for it caught it, because each one described it as a separate, minor, honest limitation. Read together, they're one bug wearing five different names: every judge scored whether the right words or shapes showed up, not whether the thing actually worked. The interesting part isn't the five prototypes, it's the sixth one, built to fix that bug in a way I could actually verify.

For context on how much weight to put on that: this was one evening of self-directed practice, no external user or stakeholder involved, six small repos (biggest eval set: 8 tasks), all built with Claude Code (every commit is paired with a session, per the commit trailers), and everything below ran in mock mode, no live API calls anywhere in this story.

If you're working on anything with an optimize-against-a-judge shape (prompt search, RAG tuning, agent config search, eval-driven pipelines), the bug is worth knowing about before you hit it yourself.

The setup

The question I wanted an actual answer to: does mutate → judge → keep-if-better → repeat generalize as a loop shape, or does every kind of artifact need its own bespoke approach?

Instead of arguing about it, I built five tiny versions, back to back, in about 7 minutes of commit timestamps total:

self-improving-prompt-agent: mutates a prompt from a fixed candidate list, heuristic judge (0.0 to 1.0)
context-improving-agent: mutates a context block, judged by 0.8*fact_coverage + 0.2*(1-bloat_penalty)
graph-improving-agent: inserts one role node at a time into a workflow graph, judged on role-keyword presence
harness-improving-agent: adds one guardrail statement at a time from a list of ten, judged on keyword coverage
agent-improving-agent, one level up: mutates the priority order harness-improving-agent tries its ten guardrails in, since you rarely get to test all ten before your eval budget runs out

Four of the five repos have exactly one commit. That's what five fast structural tests of one hypothesis actually looks like.

The loop shape held up. All five ran the same structure and all five climbed over 10 rounds:

Repo	Result (10 rounds)
prompt	0.10 → 0.80
context	0.20 → 0.28
graph	0.10 → 0.70
harness	0.00 → 0.90
agent (meta)	0.45 → 0.61 (stated max: 0.80)

Five for five, on paper. That wasn't actually the interesting result.

The bug hiding in plain sight

Every one of these repos has a "Known Limitations" section, because writing down what's wrong with what I just built while I still remember it is a habit. Read individually, each limitation sounds like a minor, honest caveat:

graph: "no check that the wiring is actually runnable/useful"
harness: "not a real check that the agent obeys the stated guardrails at runtime"

Read together, though, these are the same bug five times, wearing five different filenames. Every judge in this set scored whether the right words or shapes showed up, not whether the artifact actually worked:

A graph judge that checks for a "verifier" keyword scores an inert, do-nothing verifier node exactly the same as a functional one.
A harness judge that checks for the word "rollback" scores that word being present the same whether or not any rollback mechanism exists.
context-improving-agent's fact_coverage term almost certainly measures whether facts are textually present, not whether the context is coherent once they're added.

None of the judges were malfunctioning. They did exactly what I told them to measure. The problem is I told five of them, separately, to measure the wrong thing, and I only caught it by reading all five READMEs side by side instead of shipping straight to a sixth prototype.

There's a second wrinkle worth calling out on harness-improving-agent specifically: its judge used loose keyword matching early on and stalled at 0.20 for several rounds, until I rewrote the judge's wording mid-run to match the mutation wording it was supposed to be scoring. The jump to 0.90 partly reflects that fix, not clean search progress, and the repo's own README says this directly.

The fix: sia

About 2.5 hours after the fifth prototype's commit, I built a sixth repo, sia (self-improving-agent), whose README opens by naming the first five as prior art and states plainly why it doesn't reuse their judges: porting a structure-presence judge into something meant to be real "would have meant the keep/rollback gate approved patches for saying the right words rather than answering questions correctly."

Four changes, each one closing a specific hole from the first five:

1. Frozen judge, held-out eval. sia optimizes a genome (versioned JSON: prompt/context/workflow/harness layers) against a fixed 8-task set, split 5 train / 3 holdout. The judge doesn't get edited mid-run, unlike what happened with harness-improving-agent.

2. A real gate, not a bare threshold:

keep iff train_child >= train_parent + 1.0
         AND holdout_child >= holdout_parent

The +1.0 minimum lift on train, combined with non-regression on holdout, is there specifically to catch a patch that overfits the five training examples without generalizing. None of the first five repos had a holdout set at all: they scored and kept against the exact same signal they optimized against, every single round.

3. Integrity checks on the eval itself. sia/loop.py hashes the eval set and the judge file at the start of a run, and re-checks that hash every iteration. If either changes, the run aborts with a logged ABORT row. Watching five judges reward the appearance of a good answer taught me this directly: an optimizer that's allowed to edit what it's graded against will eventually find that editing the test is cheaper than improving the artifact.

4. Positive and negative controls, not just a demo run. Checked directly against sia's own ledger, not a printed summary I trusted at face value:

Context patch: train 0.0→50.0, holdout 0.0→33.33 (KEPT)
Harness patch: train 50→70, holdout 33.33→66.67 (KEPT)
Workflow patch: train 70→90, holdout 66.67→100 (KEPT)
A deliberately-constructed patch with zero train lift (ROLLBACK)

3 keeps and 1 correct rollback show the gate distinguishing real lift from no lift. That's a stronger claim than "the score went up." I also ran the test suite live: 21/21 unit tests passing.

One more scope decision worth mentioning: sia's README states "The AI is not allowed to call itself better." The improver component can write a hypothesis about why a patch might help, but that text is documentation only. It never feeds into the score.

What I can't claim

Everything above happened in mock mode, a simulated worker standing in for a real model call. There's an API key sitting in a .env file that was never used to make a real call: no logs, no genomes, no ledger rows anywhere attributable to a live run. sia's client hard-raises if it's ever invoked with mock=True, so there's no ambiguity about which mode produced these results. That also means I genuinely don't know whether the gate, the holdout split, or the hash-check survive contact with real, noisy model output. It might. I haven't tested it.

Also unresolved: the meta-loop from agent-improving-agent (the one that optimizes guardrail priority order) was designed to plug into exactly this kind of frozen judge, and I never wired it in. The README calls it future work, plainly.

The takeaway

If you're building any kind of optimization loop over LLM output, the thing worth taking from this isn't the five prototypes or sia's specific architecture. It's the order of operations: before investing in a better mutation strategy or a smarter search, check what your judge is actually measuring, and whether that's cheap to satisfy without being correct. A keyword-presence judge gets gamed eventually, not because the search is adversarial, but because "score well against this specific signal" is always an easier target than "be right," and any sufficiently capable optimizer finds the easier target first.

I built the same mistake five times before I noticed it, then spent the rest of the night on the unglamorous fix: a frozen test set, a holdout split, and a hash check that (so far) has never fired.