Every team has a version of this document: a postmortem, a retro note, a lesson log. Something breaks, someone writes down exactly what should change, and the fix reads as obvious. And then, days later, the same failure happens again, in the same shape, for the same reason -- because the recommendation was never anything more than text.
I hit this in a very literal way while running an autonomous agent that manages its own task queue. Every day it logs a reflection pass over its own output: what worked, what didn't, what to change tomorrow. For four consecutive days, that reflection log identified the exact same problem -- a scheduler that kept over-allocating a high-friction task type while a low-friction, high-throughput one starved -- and proposed the exact same fix, with formula and threshold spelled out. And for four consecutive days, the scheduler did the opposite of what the log recommended, because nothing in the code actually read the log.
That gap -- between "we wrote down the fix" and "the fix runs" -- is worth taking seriously as a design problem, not a process problem. Here's the pattern I used to close it, with enough detail that you can drop the shape into any weighted scheduler.
The setup: weights that already exist
Most non-trivial schedulers already have some notion of weighted selection -- a set of task types (or routes, or workers, or jobs) each with a base weight, and a pick function that samples from that distribution:
def weighted_pick(candidates, weights):
total = sum(weights[c] for c in candidates)
r = random.uniform(0, total)
upto = 0
for c in candidates:
upto += weights[c]
if upto >= r:
return c
return candidates[-1]
If your scheduler is even a little dynamic, you probably also have an override layer on top of the base weights -- a place where you can bump one task type up or down without redeploying. Mine stores it as a small JSON blob in the database: a dict of {task_type: multiplier}, read once per scheduling decision and multiplied into the base weight, with a safe fallback (clamp to a small positive floor) if a multiplier would zero out or invert a weight entirely.
def weight_overrides(db):
raw = db.get_state("weight_overrides")
if not raw:
return {}
try:
overrides = json.loads(raw)
except (TypeError, ValueError):
return {}
return {k: max(v, 0.05) for k, v in overrides.items() if isinstance(v, (int, float))}
This piece usually already exists for a mundane reason -- someone wanted to manually throttle a noisy job type without a deploy. The mistake is stopping there and assuming manual overrides are the only overrides you need.
The missing piece: a signal the scheduler can read for itself
The recommendation that kept getting ignored had a very specific, checkable shape: when the backlog of tasks waiting on a human (or any blocking external step) crosses a threshold, throttle the task types that produce more of that backlog and boost the ones that don't. That's not a vague sentiment -- it's a query and a table.
The query counts, over some recent window, how many completed tasks are still carrying an unresolved follow-up:
def backlog_pressure(db, window=100):
pending = 0
for row in db.recent_tasks(limit=window):
try:
result = json.loads(row["result_json"])
except (TypeError, ValueError, KeyError):
continue
if result.get("action_required"):
pending += 1
return pending
And the table is just the reflection log's own formula, made literal:
BACKLOG_FRICTION_MULTIPLIERS = {
"high_friction_task": 0.4,
"another_high_friction_task": 0.5,
"low_friction_task": 1.6,
}
BACKLOG_THRESHOLD = 7
def apply_backlog_pressure(overrides, pressure):
if pressure <= BACKLOG_THRESHOLD:
return overrides
merged = dict(overrides)
for task_type, multiplier in BACKLOG_FRICTION_MULTIPLIERS.items():
merged[task_type] = merged.get(task_type, 1.0) * multiplier
return merged
Wired into the existing override function, this is maybe fifteen lines of new code on top of infrastructure that was already there. That's the point worth underlining: the fix wasn't a new subsystem. It was making an existing subsystem read one more signal.
Why this shape, specifically
Three properties made this safe to ship without a long review cycle, and they generalize past this one bug:
It's additive, not replacing. The new function composes with the existing override dict rather than branching around it. If backlog_pressure() returns zero -- which it does for every existing test fixture, since none of them simulate a week of unresolved tasks -- behavior is provably identical to before the change. You're not asking anyone to trust new logic in the common case; the common case doesn't touch the new logic at all.
It reuses a read path that's already trusted. I didn't add a new database table or a new query method. recent_tasks() already existed, already had test coverage, and already had a documented row shape. The new code just reads it differently. Every new failure mode you might worry about (malformed JSON, missing keys, a task with no result yet) was already something the surrounding code defended against, so the same defensive pattern -- try/except around json.loads, .get() with a default -- covered the new call site for free.
The threshold is a number, not a vibe. "Throttle when things feel backed up" is not enforceable. "Throttle when more than 7 of the last 100 tasks still have action_required set" is a single if statement. Converting a recommendation into a gate means converting its trigger condition into something a function can evaluate -- if you can't write that condition down, the recommendation isn't finished yet, no matter how sound the reasoning is.
The general lesson
A recommendation that only lives in a log is a wish. It becomes a gate the moment three things are true: the trigger condition is a query against data you already collect, the response is a change to a code path that already runs on every cycle, and the default behavior (trigger not met) is provably unchanged from before. If any of those three is missing -- if the trigger needs data you don't have yet, or the response would need a new code path nobody's reviewed, or you can't show the change is a no-op in the common case -- that's exactly where recommendations go to die, re-written in next week's postmortem with the same formula and the same missing enforcement.
The fix isn't writing better recommendations. Mine were already correct four days running. The fix is treating "convert this into a gate" as its own task, with its own priority, separate from "identify the problem" -- because a system will keep re-discovering the same problem forever if nothing downstream of the discovery ever runs.
Top comments (1)
The four consecutive days of correct-but-ignored reflection logs is painfully familiar. I run into the same shape with agent checkpoints that log exactly what went wrong between runs but never become assertions on the next run.
Your three properties for safe gates are the part I want to steal. Especially the "provably unchanged in the common case" criterion -- that's what makes it shippable without a design review. Most of the time when I've tried to close this gap, the proposed fix touches the default path, and then it sits in review until the next postmortem re-discovers the same bug.
The threshold itself drifts, though. 7-of-100 works today, but six months in the task mix shifts and that number either fires constantly (so someone hardcodes it away) or never fires. Logging every time the gate activates, even as a single counter, keeps it visible enough that someone notices when it goes stale.