I run a personal system called Hiring OS. It pulls jobs from ATS APIs (Ashby, Greenhouse, Lever, YC's Work at a Startup), scores them against my resume, tailors a resume through a set of deterministic quality gates, and tracks every application. I've been the user, PM, builder, and tester on the same system for about five months, which means every bad decision in it is mine and every fix is also mine.
This is a postmortem on one of those decisions: a rule I wrote that let a daemon submit job applications with no human in the loop, sitting two lines below a rule that explicitly banned that.
The setup
By the time this happened, the parts of Hiring OS I actually trusted were working: job discovery from live ATS APIs, and a set of deterministic quality gates a tailored resume had to clear before I'd use it. Truthfulness against a fixed set of source documents. ATS keyword match. Recruiter readability. Hiring-manager fit. Real pass/fail checks, not vibes.
A couple of days before the rule I'm about to show you, I'd written an internal status note declaring an autonomous submission daemon "MVP complete." Run continuously in the background, apply to qualifying jobs 24/7, I check progress from my phone. The logic felt sound: I'd already built the hard part, the thing that actually stops a bad resume from going out. If the gates say a resume clears the bar at a high enough score, what is a human click still protecting against?
The rule
I formalized it in the same rules document that governs everything else the system is and isn't allowed to do:
Amended: Daemon runs continuously. Jobs with tier classification +
passing eval thresholds auto-qualify for submission. No manual gate.
Perfect-match threshold: Truth>=90, ATS>=75, Recruiter>=70, HM>=70
Adjacent-role threshold: slightly looser bar + quality signal
No exceptions carved out. The daemon would build the submission payload and submit through the automation runner on its own.
Two lines above this exact block, in the same file, since the file's first version: "Tailored resume rewriting runs through a human-in-the-loop session per job. No unattended/autonomous pipelines." A few lines below: jobs cannot be auto-promoted to ready_to_apply, the operator has to set it explicitly.
I'd written a document whose entire spine was "nothing irreversible happens without me," then bolted on one exception, without touching a single word of the surrounding rules that said the opposite.
Why it took me months to catch
A gate score answers "is this output good." It does not answer "is it safe to act on this output without review." Those look like the same question if you squint, especially when the same number is doing double duty for both, but they have different failure modes entirely.
Here's the data that makes this concrete. Over about a week and a half, applying by hand the way the original rules actually described, I tracked 28 applications with real gate scores logged next to each one:
- Founding Senior PM role: 100/95/87, submitted
- Another PM role: 100/92/90, submitted
- A third role: 100/100/97, the highest score in the entire batch, rejected the same day, attributed by my own tracker notes to a work-authorization hard filter
That third one is the whole argument in one row. A perfect resume score did nothing to prevent a same-day rejection for a reason the resume gates were never built to detect. If the daemon had been live, it would have auto-submitted that application on the strength of a 100/100/97, with exactly as little visibility into the sponsorship mismatch as I had. The gate wasn't wrong. It was answering a question that had nothing to do with the thing that actually killed the application.
There's a smaller version of the same failure mode from the same stretch. The day I added an eighth eval gate (a voice and AI-vocabulary checker, after two resumes got flagged for banned wording), I'd given myself a general go-ahead to submit two queued applications. Partway through the second form, I stopped and wrote a note anyway: "Do not click submit, fill the form, wait until I say it only." A blanket yes given earlier in a session didn't survive contact with the actual irreversible click.
The fix
Once I reread the amendment against the rest of the document it lived in, it read as exactly what it was: an exception that contradicted every safety rule around it, added under the pressure of wanting the search to move faster. I struck it through instead of deleting it, so the wrong version stays legible next to the correction:
DEPRECATED, do not follow: Amended earlier: Daemon runs continuously... Jobs with tier classification + passing eval thresholds auto-qualify for submission... This described a zero-human-review autonomous submission architecture that contradicts every other safety rule in this document and is no longer the policy.
Current policy: NO unattended external submission, no exceptions. Eval scores and tier classification may inform a recommendation, never an automatic submission.
Then I checked the actual running system, separately from the document, because a corrected policy file and an inert daemon are two different facts and I wasn't going to assume the first proved the second. No active service runs the submission daemon. A narrower scheduled job still runs, and it only fetches, scores, and scaffolds. It never submits anything on its own.
The generalizable part
If you're gating any irreversible action behind a confidence score, the question worth asking isn't "how high should the threshold be." It's whether the threshold measures the thing that makes the action safe, or something correlated with quality that you're treating as if it measures safety. Those aren't the same property, and a system can pass every check you built and still be missing the one check the actual risk depends on.
The tell, in my case, was that a rule contradicting the rest of the document sat there for months before anyone (me) reread it closely enough to notice. The fix isn't trusting the daemon less. It's building a forced re-read of any change that removes a human step, on a schedule, instead of waiting on a coincidental close reading to catch the next one.
Top comments (0)