DEV Community

jackymenCZ (jackymenCZ)
jackymenCZ (jackymenCZ)

Posted on

13,786 Requests in One Day: How We Taught Our AI Agent to Stop Going in Circles

description: "My agent once sent 13,786 requests to OpenAI in one day and achieved nothing. This is how it got there, and how we taught it to stop: the story for normal people first, then the code, tests and receipts for developers."

13,786 Requests in One Day: How We Taught Our AI Agent to Stop Going in Circles

A long talk over coffee about agent loops. The first half is for anyone. The second half is for people who like stack traces with their coffee.


Before we start: how to read this

This article has two halves.

Part One is for normal people. No code. You don't need to know what an API is. If you've ever had a robot vacuum get stuck under the sofa, or watched a toddler ask "why?" forty times in a row, you already have the background you need. I'll explain what an "agent loop" is, why it costs real money, and what fixes it, using our own bot's history as the example.

Part Two is for developers. Code, git history, OpenAI billing data, a before/after test harness you can run, unit test counts and live production numbers. If you only want that, skip to Part Two.

Every number here comes from one of three places, and I'll say which:

  • The OpenAI bill. Pulled read-only from the OpenAI Admin API (/v1/organization/costs and /v1/organization/usage/completions), daily buckets, UTC, from March 2026 to 17 October 2026.
  • The git history of the repository the bot lives in.
  • The bot's own public read-only endpoints, captured on 17 October 2026 at 05:37 UTC.

If a number is an estimate, I'll call it an estimate. If I don't know why something happened, I'll say that too. That turns out to matter in a story about a machine that kept telling itself it was being productive.


Part One: For Everyone

1. Pull up a chair

Okay. Sit down. You know that project I keep talking about, the one I call Sentinel? The bot that watches a code repository, reads the files and decides whether anything should be improved?

On 13 April 2026 it sent 13,786 requests to OpenAI.

In a single day.

That works out to about one request every 6.3 seconds, around the clock, for 24 hours. If a human employee did that, you'd ask whether they were okay. You'd want to know what they were writing, who to, and why they'd sent you 13,786 emails about a code base that, as far as I can tell from the history, barely changed that day.

The bill for that day was $7.57. That doesn't sound like much. Now look at the month:

  • April 2026: 41,253 requests, $37.06
  • May 2026: 22,995 requests, $50.28
  • June 2026: 48,353 requests, $10.90

And here's how it looks today, after everything I'm about to tell you:

  • 1–17 October 2026: 27 requests, $1.39

Same bot. Same job. Same repository. Same person paying.

From 1 March to 17 July, Sentinel made 123,932 requests. From 18 July to 17 October it made 451. So 99.6% of all requests this bot has ever made happened before mid-July.

Some of those early requests did useful work, and I'm not claiming every one of them was wasted. But a large share of them was the bot doing the same thing again and again: re-reading the same files, re-asking the same question and getting the same answer. In this world that's called looping, or "cycling". This article is about how that happens and how we stopped it.

Grab your coffee. This takes a while, and it's more interesting than it sounds.

2. First: what even is an "agent"?

There are a lot of buzzwords around here, so let's clear them up.

A normal program is like a vending machine. You press B4 and you get crisps. It never decides anything. It does exactly the steps someone wrote down.

A chatbot like ChatGPT is like a very well-read pen pal. You write a letter, it writes one back. It doesn't do anything between letters. It doesn't wake up at 3 a.m. to check on you, which is honestly a good quality in a pen pal.

An agent is a pen pal with hands and an alarm clock. It can:

  1. Look at something (read files, check a website, read your email),
  2. Think about it (often by asking a language model like GPT, which is the "brain"),
  3. Act (write a file, send a message, open a pull request, buy something),
  4. And then do it all again. On its own. On a schedule, or whenever something pokes it.

That last step is what makes an agent an agent. It keeps going without you pressing a button every time.

It's also the step that gets you into trouble.

Sentinel is an agent of that kind. Every couple of minutes it wakes up, looks at a code repository, thinks about each file ("is this file healthy? should it be improved? is anything risky here?") and either does something or decides to leave it alone.

Each time the "thinking" part asks OpenAI's model, that's one request, and requests cost money. Not a lot each time. A fraction of a cent, sometimes a few cents. But fractions of a cent times 13,786 is a real number, and if you multiply that by "every day, forever," you'll understand why the person paying started reading invoices with a frown.

3. So what is a "loop"?

A loop is when an agent keeps doing the same thing over and over without anything new happening, and usually without noticing that it's repeating itself.

You've seen loops in real life:

  • The dog chasing its tail. It's very busy. It's very committed. It's getting nowhere.
  • The kid in the back seat: "Are we there yet?" Every 90 seconds. The answer hasn't changed. The kid asks anyway.
  • Groundhog Day. Bill Murray wakes up on the same day again and again. The difference is that Bill Murray eventually learns. A dumb agent doesn't. It wakes up on the same day, has the same thoughts, makes the same decisions and sends you the same invoice.
  • The robot vacuum under the sofa. Bump, turn, bump, turn, bump. Battery runs out. Floor not cleaned.
  • The colleague who replies-all to "please stop replying-all". That one's a special kind of loop. We'll come back to it, because our bot did exactly that.

The important thing is that a looping agent looks busy. Its logs are full. Its graphs are going up. It's "working". If you only glance at it, you'll think it's productive. You only notice when you ask the boring question: what actually changed because of all this work?

The honest answer is often: nothing.

4. Why should a normal person care?

"Fine," you say, "the robot is silly. Why is that a problem?"

Four reasons.

1. It costs money, and the meter is always running. An agent calling a paid AI model is like a taxi with the meter on. A loop is a taxi driving around the block with you in the back. Every lap costs. Our bot's most expensive stretch was 24–27 May 2026: between 1,310 and 1,382 requests a day to a more expensive model, $7.28–$10.15 a day, $36.59 in four days. Small numbers for a company, but this was one person's hobby project with no revenue. And at a company with a hundred agents, add zeros.

2. It can hurt you, not only your wallet. An agent that acts, writes files or sends messages, can loop on those actions too. It can write the same "fix" twice, undo it and write it again. It can trigger itself: its action causes an event, the event wakes the agent, the agent acts again. That's the reply-all colleague.

3. It drowns out the real signal. If your agent says "I checked everything!" 2,000 times a day, you'll stop reading its reports. Then, on the day something really breaks, you miss it.

4. It gets you blocked. Services like OpenAI and GitLab have rate limits ("you're asking too fast, slow down") and quotas ("you've used up your money"). A looping agent hits those walls. A really dumb agent then retries the request it was just refused, which hits the wall again, which makes it retry again... You get the idea. It's a loop inside a loop.

5. The five kinds of loops (a field guide)

After more months than I'd like to admit of watching Sentinel misbehave, I'd sort agent loops into five families. Once you know them, you'll spot them everywhere.

Loop #1: The Alarm Clock Loop

"It's 3:02. Let me check everything. It's 3:04. Let me check everything. It's 3:06..."

The agent has a timer. Every few minutes it wakes up and re-examines everything from scratch, even if nothing has changed since last time.

That's like a security guard who walks every corridor of the building every two minutes, all night, and writes a full report each time, even though the doors are all locked and nobody has come in. The guard isn't bad at the job. The guard just has no way to know whether a lap is needed, so every lap happens.

This was Sentinel's main problem. In April 2026 it was set to wake up every minute. Later every 2 minutes, which still means 720 wake-ups a day, each one looking at a batch of 5 files.

Loop #2: The Echo Loop (the reply-all)

The agent's own action triggers the agent.

Sentinel watches a repository. When something is pushed to the repository, GitLab sends a notification (a "webhook") saying "something changed, come look." But Sentinel itself can write to that repository. So: Sentinel writes a file → GitLab says "something changed!" → Sentinel wakes up and looks → Sentinel writes something → GitLab says "something changed!" → ...

That's a microphone pointed at its own speaker. One tap on the mic and you get that horrible squeal that only stops when someone pulls the plug.

In early April the bot had a rule "ignore commits that I made myself" to avoid exactly this. The git history shows we kept adjusting what counts as "myself". At one point the rule was literally "skip commits whose message starts with a shield emoji". In May the conclusion, in the words of the commit message, was to remove "webhook-driven chaos" and the "execution storms" it caused.

Loop #3: The Busy Signal Loop

You call someone. It's busy. You redial immediately. Busy. Redial. Busy.

When an agent's request fails (the service is down, too slow, out of money), a naive agent just tries again right away, often in a tight loop. If the failure is temporary, that might work. If the failure is "your account is out of money until tomorrow," then every retry until midnight is pointless.

Loop #4: The Goldfish Loop

"Hmm, let me think about this file... It looks fine. I'll leave it alone."

Two minutes later:

"Hmm, let me think about this file... It looks fine. I'll leave it alone."

The agent has no memory of its own conclusions, or it has a memory and doesn't trust it. So it asks the expensive brain the same question about the same unchanged thing and gets the same answer.

This one has a nice signature in the billing data. In June 2026, on five separate days (16, 17, 19, 22 and 23 June), Sentinel sent exactly 1,726,740 input tokens to OpenAI. Not roughly the same. Exactly the same, to the token. On every other day from 14 to 23 June it was within 0.7% of that number, and from 15 to 23 June the request count stayed between 2,169 and 2,173 every day. That's the goldfish: the same prompts sent again and again, day after day, because nothing told the agent "you already answered this."

Loop #5: The Overthinker Loop

The subtlest one. The agent is allowed to think more deeply sometimes, say to reconsider a file it decided to skip, "just in case." That's a good instinct. But if nothing stops it, "just in case" turns into "every time."

Think of it as the friend who reopens a decision you made together last week, every time you meet, without any new information. "But what if we did paint the kitchen yellow?" We had this conversation. Nothing has changed. Same paint, same kitchen.

6. Our story, in four acts

Here is the whole OpenAI history of the project, month by month. "Requests" means calls to an OpenAI model and "spend" means what OpenAI billed.

Month (2026) Requests Spend Busiest day
March 258 $0.02 200 requests (22 Mar)
April 41,253 $37.06 13,786 requests (13 Apr)
May 22,995 $50.28 4,045 requests (10 May)
June 48,353 $10.90 2,892 requests (12 Jun)
July 11,078 $5.38 4,668 requests (16 Jul)
August 47 $2.13 14 requests (26 Aug)
September 372 $12.36 267 requests (25 Sep)
October (1–17) 27 $1.39 4 requests
Total 124,383 $119.51

A few things to notice even if you never read a line of code.

Act One: April. "Let's make it autonomous!" On 8 April the bot was changed to run "every 1 minute" over a batch of 5 files. The next day it made 7,411 requests. A few days later, the day a version came out that "restored the webhook endpoint" (the doorbell from Loop #2), it made 13,786. I can't prove from the bill alone exactly which of those requests came from the alarm clock and which came from the doorbell. The bill doesn't say why. But the timing lines up uncomfortably well.

Act Two: May. The expensive month. May had fewer requests than April but cost more. Why? On 23 May two changes landed on the same day. One added a more expensive "strategic" model (GPT-4o) next to the cheap one. The other removed a safety check that used to stop the main loop from doing its full scan unless the repository had really changed. The commit says, in its own words, that it removes the check "to enforce strict 2-minute polling." For the next four days the bot sent 1,310–1,382 requests a day to the expensive model, at $7.28–$10.15 a day.

Here's the funny part, if you're in the mood for dark humour. Between 13 and 22 May, right after we'd removed the doorbell-triggered chaos, the bill shows zero requests. Ten quiet days. Then a well-meaning change removed the "only run when something changed" check, because, as the commit put it, it had been "disabling the autonomous 2-minute telemetry polling". In fairness, the bot was partly broken: with the doorbell gone, that check only let it run at boot. But the fix didn't bring back "run when something changed". It brought back "run every 2 minutes, unconditionally", and the meter started spinning again. That's the classic agent trap in one picture: quiet looks like broken, so someone "fixes" it back into a loop. (The same commit promised that a per-file cooldown would keep tokens safe. The bill for the next four days disagrees.)

Act Three: June. The goldfish month. Lots of requests (48,353) but cheaper, because by then it used the small model. And this is where the identical-to-the-token days come from: same prompts, same answers, every day.

Act Four: from mid-July onward. The bot learns to sit still. On 18 July we added what we call the Novelty Gate: "if nothing about this file has changed since you last decided to leave it alone, don't ask the brain again, reuse your old answer." From then on, requests collapsed: 451 in three months, compared with 123,932 in the four and a half months before.

(Two notes for honesty. September had a $12.36 bump. Part of it, 28–29 September, came from a different OpenAI project, not the Sentinel bot: 8 requests that were each very large. 25 September, with 267 requests and $5.09, was the Sentinel project, and I can't attribute it to a specific cause from the data I have. Part Two lists what is and isn't explained.)

7. How we fixed it, without code

"Great, it was a mess. What did you actually do?"

Not one magic trick. Several small, boring guards, each of which closes one kind of loop. The best way to explain them is to give each one a human job title.

The Doorman (Wake Gate) → fixes the Alarm Clock Loop

The alarm clock still rings every 2 minutes. We never removed it. But now, before the agent does any real work, a doorman asks one cheap question:

"Has anything actually happened since last time?"

To answer, it does one tiny lookup: "what is the latest version of the code?" If it's the same as last time, and nothing else is scheduled (no follow-up review due, no new evidence), the doorman says "go back to sleep" and the agent doesn't scan files, doesn't think and doesn't save anything.

Today 93.2% of all wake-ups end exactly like that. Out of 16,314 alarm-clock rings recorded by the current production instance, 15,200 were "nothing happened, go back to sleep".

There's one rule the doorman follows that I'm proud of. If the doorman can't tell whether anything changed (say GitLab didn't answer), it lets the agent in anyway, to be safe. "I don't know" is never treated as "nothing changed." That rule has woken the agent 23 times in production. That costs a little efficiency, and it's the safer direction to be wrong in.

The Notebook (Novelty Gate) → fixes the Goldfish Loop

Even when the agent does wake up, it now keeps a notebook. For each file, it writes down a "fingerprint": the file's contents plus a few other things that matter (how risky it is, what role it plays, which other files it depends on, which rules were in force). Next to the fingerprint it writes the decision it made.

Next time, before asking the brain about that file, the agent checks the notebook: "Same fingerprint as last time? And last time I decided to leave it alone? Then I'll leave it alone again, and I don't need to ask anyone."

Two rules make this safe rather than lazy:

  1. Only "leave it alone" answers are reused. If the last decision was "change this file" or "this was rejected," that's never copied blindly.
  2. Anything that matters changes the fingerprint. If the file changes, or a file it depends on, or the rules, the fingerprint is different and the agent thinks fresh.

In production, the notebook has been used 163,459 times instead of asking for a fresh evaluation. There were 49 fresh evaluations.

The Bouncer (Webhook Guard) → fixes the Echo Loop

The doorbell (the GitLab webhook) still exists. But now there's a bouncer:

  • You need the right password (a secret token) to ring it at all.
  • If the agent is already busy, ringing again does nothing.
  • And it starts at most one cycle every 30 seconds, no matter how many times you ring.

In our test, 20 rings in 10 seconds start exactly 1 cycle instead of 20.

The Single Key (busy flag + runtime lock) → no two copies working at once

Only one copy of the agent may work at a time, like a shared office with one key. If the key is out, whoever wants to start has to wait. This seems obvious, but without it a slow cycle plus a fast timer means two or three cycles running on top of each other, all doing the same work.

The Wallet (daily budget) → a hard ceiling

The agent keeps its own running count of what it thinks it has spent today. When that passes $2, it stops asking the brain until tomorrow and just writes down "deferred, budget exhausted." It's a pocket-money rule: you can't spend what isn't in your pocket.

One honest caveat. This wallet counts the agent's own estimate of each call's price, not the real bill. On 25 September the real bill for the Sentinel project was $5.09, more than the $2 ceiling. The wallet is a guard rail, not a guarantee. A separate read-only "accountant" module now reads the real OpenAI bill and flags days that go over budget. It flagged that day.

The "Shop's Closed" Sign (LLM error protocol) → fixes the Busy Signal Loop

When a request to the brain fails, the agent now looks at why it failed and follows a fixed drill:

  • "Too slow" / "network hiccup": try again a couple of times with a short pause.
  • "Server error": try twice, then stop asking for 10 minutes.
  • "You're asking too fast": try twice with longer pauses, then wait 2 minutes.
  • "You're out of money" (quota): don't retry at all. Close the shop until midnight UTC.
  • "Wrong password": don't retry. Stop for an hour, because retrying won't fix a wrong password.

In our test scenario, the quota runs out at 9 a.m. A hypothetical naive agent that tries each file three times, every 2 minutes, would make 6,750 pointless calls before midnight. Sentinel makes 1, sees "out of money," puts up the "closed until midnight" sign, and quietly holds the other files.

The "New Questions Only" Rule (autonomy budget + triggers) → fixes the Overthinker Loop

Sentinel is allowed, rarely, to reconsider something it decided to skip. But only if:

  1. There's a concrete reason: an error in the production logs, a rollback, a failed prediction, a change in its dependencies, and so on. "I'm just curious" doesn't count.
  2. The reason is new. If it already reconsidered this file because of this same reason, the same reason can't reopen it.
  3. It still has tokens left today (a small daily allowance, default 10, hard cap 100), and the same file can't be reopened again within 24 hours.

In production, this "reconsider" path has asked the brain 28 times in total. When I captured the data, it was refusing to reopen anything, with reason reason_already_answered: "you already discussed this, and nothing new has happened since."

That's the friend who reopens the yellow-kitchen debate, now politely asked: "Has anything changed since last time?" "...No." "Then we're not doing this."

8. Before and after: the same day, two agents

Here's the simplest way to see the difference. Picture one ordinary, boring day:

  • A repository with 5 files.
  • The alarm clock rings every 2 minutes, so 720 times in 24 hours.
  • At noon, someone changes one file. That's the only event of the day.

The "before" agent (roughly how Sentinel worked in June) wakes up 720 times, reads all 5 files each time and asks the brain about each one: 3,600 requests. For one change.

The "after" agent (Sentinel today):

  • Rings: 720.
  • Times the doorman let it in: 2. Once at the start ("I've never seen this place before") and once at noon ("the code changed").
  • The other 718 rings: "nothing happened, go back to sleep."
  • File reads: 10 (5 files × 2 wake-ups).
  • Fresh evaluations (the most it could ask the brain): 6. 5 the first time, plus 1 for the file that changed. The other 4 files at noon were "same fingerprint, same answer, reuse."

3,600 vs 6. Not a percentage improvement. A different way of behaving.

This isn't a made-up chart. In Part Two this scenario runs through Sentinel's real gate code, with a fake clock and a fake brain that only counts calls.

To be fair to the "before" agent: in real life, in June, Sentinel made about 2,170 requests a day rather than 3,600, because some files were skipped by older, simpler rules and some cycles were slower. The model is a simplification. But the pattern, ask about everything, every time, is what the billing data shows.

9. A checklist for non-developers

If someone in your company is building or buying an "AI agent", here are the questions I wish I'd asked myself in April. You don't need to understand the answers in technical detail. Just watch whether the person answering looks relaxed or starts sweating.

  1. "When nothing has changed, what does the agent do?" The right answer is some version of "nothing, cheaply." The wrong answer is "it re-checks everything."
  2. "Can the agent's own actions wake it up again?" If yes, ask what stops the echo.
  3. "What happens if we run out of credit with the AI provider at 9 a.m.?" The right answer is "it stops until the quota resets." The wrong answer is "it keeps retrying."
  4. "Is there a daily spending limit inside the agent, and is it checked against the real bill?" Two different things, and both matter.
  5. "How many times did it call the AI model yesterday, and how many of those calls changed anything?" If nobody can answer, that's your answer.
  6. "When the agent is quiet, how do we know it's healthy and not broken?" Because the most dangerous moment is when someone sees a quiet agent, thinks it's broken, and "fixes" it back into a loop. We did exactly that on 23 May.

That's it for the non-technical half. If you stop reading here, the one-sentence version is:

An agent should only think when something new has happened, and it should know the difference between "nothing changed" and "I don't know."

Developers, fresh coffee. Here comes the long part.


Part Two: For Developers

Still here? Good. Pour the strong stuff. From here on I'll assume you know what a webhook is, why setInterval with an async callback is a footgun, and why "just add a retry" is how half of all outages start.

The plan:

  1. The data: where the numbers come from and what they can't tell us.
  2. Forensics: lining the OpenAI bill up against the git log, incident by incident.
  3. The containment stack: the actual code of each guard, in the order a tick hits it.
  4. A before/after harness: the real gate modules, a virtual clock, a fake LLM, and the output.
  5. Tests: what's pinned, and the counts.
  6. Production today: the live numbers, with the caveats that belong next to them.
  7. What isn't solved: the gaps, because there are some.

A note on context: Sentinel is a Node.js service (Express, deployed on Render) that watches the libs/shared directory of a GitLab repository, runs every file through a deterministic pipeline (translation into a compact intermediate representation, risk/role classification, lifecycle, governance rules) and consults an LLM only as an advisor. The deterministic gates are the authority. That design choice matters a lot for loops, as you'll see: an advisor that can't decide anything on its own is much easier to keep quiet.

The repository is private, so every code excerpt below is quoted inline rather than linked.


10. The data and its limits

10.1 How it was pulled

Two endpoints of the OpenAI Admin API, read-only, with an admin key that lived only in the environment of one shell command:

GET /v1/organization/costs              bucket_width=1d  group_by=project_id   (and group_by=line_item)
GET /v1/organization/usage/completions  bucket_width=1d  group_by=model,project_id
Enter fullscreen mode Exit fullscreen mode

From start_time = 2026-01-01T00:00:00Z to "now" (17 October 2026), following next_page until exhausted. That gave 289 daily cost buckets and 290 daily usage buckets. The first non-zero bucket is in March; January and February are empty.

10.2 The full table

Month Requests Spend (USD) Input tokens Output tokens Cached input Active days Peak day
Mar 258 0.02 38,067 25,677 0 4 200 (22 Mar)
Apr 41,253 37.06 65,599,316 24,519,893 45,054,080 18 13,786 (13 Apr)
May 22,995 50.28 21,139,626 6,020,577 13,585,024 14 4,045 (10 May)
Jun 48,353 10.90 36,458,146 10,922,248 18,348,032 22 2,892 (12 Jun)
Jul 11,078 5.38 11,138,633 6,978,132 6,388,992 5 4,668 (16 Jul)
Aug 47 2.13 208,987 64,392 121,472 6 14 (26 Aug)
Sep 372 12.36 1,571,200 120,372 699,647 11 267 (25 Sep)
Oct 1–17 27 1.39 154,741 6,670 75,967 10 4
Total 124,383 119.51 136,308,716 48,657,961 84,273,214

Split into phases at the two architectural boundaries (Novelty Gate on 18 July, Wake Gate on 2 September):

Phase Spend Requests Active days Requests / active day USD / active day
1 Mar – 17 Jul (no Novelty Gate) $103.63 123,932 62 1,998.9 1.67
18 Jul – 1 Sep (Novelty Gate, no Wake Gate) $2.73 61 8 7.6 0.34
2 – 30 Sep (Wake Gate; includes a second project) $11.77 363 10 36.3 1.18
1 – 17 Oct (full stack) $1.39 27 10 2.7 0.14

10.3 What this data can't tell you

I want these next to the numbers, not in a footnote:

  • The bill doesn't know why. OpenAI's usage API counts requests and tokens per model and project per day. It can't tell you which code path made a request. Everything I say below about causes is a correlation between a daily bucket and a commit date, unless I quote a log or a test. Deploy times on Render aren't in this dataset, and a commit on master isn't proof that it was running in production that afternoon.
  • Daily buckets are UTC days. A loop that starts at 22:00 UTC shows up split across two days.
  • The models changed. Historically this organisation billed gpt-4o-mini-2024-07-18, gpt-4o-2024-08-06, gpt-4.1-mini-2025-04-14, gpt-4-0613, gpt-5.5-2026-04-23 and, recently, gpt-6-astra. "Requests went down" and "dollars went down" are different claims, because the price per request moved under our feet. That's exactly why May (fewer requests, more dollars) happened.
  • There's more than one project. Two project IDs appear. One is the Sentinel production service. The other carried a handful of large experimental/benchmark calls in August and September. I attribute only what's in the Sentinel project to Sentinel.
  • Not every request in the loop era was waste. Some of those calls produced decisions we kept. I don't have a ground-truth label of "useful vs. redundant" per request for March–July, so I won't give you a "wasted dollars" number. What I can show is the signature of redundancy: identical daily token totals, request counts that equal ticks × batch, and the drop when reuse was introduced.

11. Forensics: the bill vs. the git log

Here's the overlay, day by day where it matters. Spend is daily cost across projects; requests are from the usage endpoint.

Date Requests Spend Model mix What landed in git around then
7 Apr 825 $5.84 gpt-4o (early prototype on the big model)
8 Apr 3,426 $0.58 mini (+9 4o) dee34ae6c18 "full autonomous Sentinel – 1min interval, batch 5 files, … rotation, gpt-4o-mini"
9 Apr 7,411 $6.86 mini e1dc8da047c 5-minute loop
10–12 Apr 1,555 / 415 / 288 mini 5-minute cycle (e0b7ea36d3e)
13 Apr 13,786 $7.57 mini bb58a851636 v5.0.5 "restore webhook endpoint and stabilize architect cycle"; d4772d0b72a v5.0.6
15–16 Apr 4,112 / 5,805 $1.93 / $1.04 mini bb8c70201dc hash-mismatch-only + fix for a "self-confirmation loop"
18 Apr 447 $0.13 mini 3cc77c956e2 "Anti-Loop: file is marked with its hash even on SKIP"; 87cc85f62c4 MAX_SAME_ACTION
20–21 Apr 491 / … mini d6a373e6a6d; 555ff14dc4c recursive setTimeout instead of setInterval, MD5 change detection
10–12 May 4,045 / 3,234 / 310 mini ce56e294f89 local isBusy lock; 80fc764e688 1-hour cooldown for failing files
13 May 81e3e044f75 "remove webhook-driven chaos"
13–22 May 0 $0 (silence)
23 May 467 $3.42 gpt-4o 9a33317dba7 "remove dirtyRepo execution gate to enforce strict 2-minute polling"; 59335c63826 adds STRATEGIC_MODEL: 'gpt-4o'
24–27 May 1,380 / 1,382 / 1,380 / 1,310 $9.99 / $10.15 / $9.17 / $7.28 gpt-4o (no change; this is the steady state)
10–12 Jun 2,777–2,892 ~$0.27 mini f3c80fa4654 "initiate autonomous mode"; 853539c999b retryOpenAI
15–23 Jun 2,169–2,173 $0.21–0.22 mini (no executor change; the goldfish)
24 Jun – 1 Jul 2,235–2,425 ~$1.07–1.21 mini 41af0646f61 executor observability
2 Jul 975 $0.53 mini 35750f953da scanner jitter, "anti-loop randomness"
3–14 Jul 0 $0
15–16 Jul 3,005 / 4,668 $1.42 / $2.20 mini large merge day: executor stack, Intent Gate, analytics, benchmarks
17 Jul 0
18 Jul 4c197562590 Novelty Gate
29 Jul 4e63c5ade34 daily budget reset fix (DAILY_LIMIT = 2.00)
2 Sep f1ed6a48ca1 Wake Gate
1–11 Oct 5dfeb61dc79 / 59e21771827 / e7f66e36840 / d5b03f00048 / 09c6334a3c6 autonomy budget, spacing, triggers, "reason already answered"; 36f94f8c896 LLM error protocol
12 Oct adb1f059fc4 webhook token + one cycle per 30 s

Now let's walk the incidents properly.

11.1 April: the alarm clock with a doorbell on top

Here's the scheduler from dee34ae6c18 (8 April). I'm quoting it exactly, because it's so honest:

// --- AUTONOMOUS LOOP - every 1 minute ---
setInterval(() => {
    console.log('\u23f0 Autonomous cycle...');
    runAutoRefactor('autonomous');
}, 60 * 1000);
Enter fullscreen mode Exit fullscreen mode

And inside runAutoRefactor, the batch selection:

const start = batchOffset % jsFiles.length;
const batch = jsFiles.slice(start, start + BATCH_SIZE);
batchOffset = (batchOffset + BATCH_SIZE) % jsFiles.length;
// ...
const aiRes = await axios.post('https://api.openai.com/v1/chat/completions', { /* ... */ });
Enter fullscreen mode Exit fullscreen mode

Spot the three loop ingredients:

  1. setInterval with an async body. setInterval doesn't wait for the previous callback's promise. If a cycle takes 70 seconds (GitLab fetches + an LLM round trip + a commit), the next one starts while the first is still running. Overlap isn't just waste. It's the ingredient that makes "two cycles commit the same thing" possible.
  2. Rotation instead of change detection. batchOffset makes sure the agent eventually looks at every file, which sounds diligent. Without a "has this changed?" check it also guarantees that it keeps looking at every file forever.
  3. The model call is unconditional. Every cycle → one or more completions.

Add the webhook. In v4.2 (aff7fd943f6, 3 April):

app.post('/webhook', async (req, res) => {
    res.status(200).send('Processing');
    const { commits } = req.body;
    if (!commits) return;

    console.log("🛠️ Commit detected. Triggering Audit & Pattern Analysis...");
    runAutoRefactor(commits);
});
Enter fullscreen mode Exit fullscreen mode

Every push triggers a full cycle, and the cycle itself can push. The fix of the week was a filter "skip Sentinel's own commits", first by an [autoRefactor=true] marker, then (35025d2a947) "only skip commits starting with 🛡️". If your loop protection depends on the first character of a commit message, you'll eventually write a commit that starts with something else. Ask me how I know.

The 13 April spike (13,786 requests, ~23 million input tokens) lands on the day the webhook endpoint was "restored". Here's the uncomfortable part: the arithmetic doesn't close. In the 8 April version the batch was concatenated into one prompt, so a 1-minute timer is at most 1,440 calls a day if cycles never overlap. 7,411 (9 April) and 13,786 (13 April) are 5–10× that. So something multiplied the timer: overlapping setInterval cycles, webhook-triggered cycles, a second completion call site in the same file (/propose-library), restarts, or code changes during those days (there were several). Daily buckets can't tell those apart, and I don't have the April Render logs. What the bill does show is that the volume was far beyond anything one timer explains, which is what you'd expect from loops feeding loops.

The first real anti-loop idea shows up on 18 and 21 April:

3cc77c956e2  STABILITA: Zaveden Anti-Loop systém – soubor je v paměti označen hashem
             i při SKIP akci, což zastavuje nekonečné AI konzultace.
             (A file is marked with its hash even on SKIP, which stops endless AI consultations.)

555ff14dc4c  Oprava zacyklení: … mainLoop s Timeoutem místo Intervalů …
             Bot nyní volá OpenAI pouze při detekci skutečné změny v kódu.
             (Loop fix: mainLoop with a timeout instead of intervals.
              The bot now calls OpenAI only when it detects a real change in the code.)
Enter fullscreen mode Exit fullscreen mode

Notice the first one. It's the Novelty Gate idea in embryo: remember the SKIP, not only the EVOLVE. A cache that only remembers the times you acted forces you to re-think every time you didn't act. And not acting is what a healthy watcher does most of the time.

Why didn't it stick? Over the following weeks the core was rewritten several times (v6.8, v6.10, v7.x, v8.x, v9.0...). I haven't traced exactly which rewrite dropped it, but the June executor quoted below has no change check in front of the model call. That's lesson zero, before any code: a loop guard that isn't pinned by a test isn't a guard. It's a phase.

11.2 May: the quiet that looked like a bug

On 12 May a local busy flag appeared (ce56e294f89, "Implementován lokální isBusy lock zabraňující kolizím při souběžném spuštění z intervalu a webhooku", a local isBusy lock that prevents collisions between interval and webhook runs). On 13 May 81e3e044f75 removed webhook-driven execution entirely:

- replaced webhook-driven execution with interval-based deterministic loop
- simplified distributed locking using INSTANCE_ID ownership model
Enter fullscreen mode Exit fullscreen mode

Then the bill went flat: zero requests from 13 to 22 May.

On 23 May:

         await MemoryManager.save(memory);

-        if (!dirtyRepo) return;
-
+        // 📡 ACTIVE MONITORING: Spouští se bezpodmínečně každé 2 minuty (odstraněn dirtyRepo gate)
         console.log(`📡 [${INSTANCE_ID}] RUN MODE: ${CONFIG.SYSTEM.RUN_MODE} | TARGET: ${CONFIG.GITLAB.BRANCH}`);

         const files = await RepositoryScanner.scan();
Enter fullscreen mode Exit fullscreen mode

(The new comment says: "Runs unconditionally every 2 minutes (dirtyRepo gate removed).") The commit message is a structured "agent instruction" and it's worth quoting, because it's the most instructive artefact in the whole history:

{
  "reason": "The 'dirtyRepo' constraint restricted the mainLoop execution exclusively to boot sequences and incoming webhook signals, disabling the autonomous 2-minute telemetry polling.",
  "runtime_impact": {
    "polling_behavior": "Enforces unconditional active repository scans and external observability analysis every 120000ms.",
    "token_safety": "File-level cooldown matrices (FILE_COOLDOWN_MS) remain active to block redundant downstream processing."
  }
}
Enter fullscreen mode Exit fullscreen mode

Two things are true at once here:

  • The diagnosis was right. With the webhook gone, dirtyRepo was only ever true at boot, so the agent really was asleep. The ten zero days weren't "efficiency". They were a coma.
  • The fix was wrong. It replaced "run only when dirty" with "run always" instead of "find out whether it's dirty, cheaply". And the claimed safety net, a time-based cooldown, isn't change detection. A 30-minute cooldown on an unchanged file means the unchanged file gets re-thought 48 times a day instead of 720. Better, but still a loop, and now one running on GPT-4o thanks to 59335c63826 the same day:
-        MODEL: 'gpt-4o-mini',
+        TACTICAL_MODEL: 'gpt-4o-mini',  // Rychlá, super levná hlídka pro periskop
+        STRATEGIC_MODEL: 'gpt-4o',      // Těžký kalibr pro krizové situace a kódování
Enter fullscreen mode Exit fullscreen mode

(Czech, for the curious: "fast, super-cheap patrol for the periscope" and "heavy calibre for crises and coding".) The next four days: $36.59, all gpt-4o. The most expensive stretch of the project, and it came from fixing a bug.

Lesson: cooldowns throttle loops, they don't end them. A cooldown answers "how often may I do this?" The question that ends loops is "is there anything new to do?"

11.3 June: the goldfish, measured to the token

By June the main loop looked like this (d602439fc3c, 14 June), and the shape survived until the Novelty Gate:

SYSTEM: {
    INTERVAL: 2 * 60 * 1000,
    BATCH_SIZE: 5,
},
// ...
for (const file of files.slice(0, CONFIG.SYSTEM.BATCH_SIZE)) { /* ... Executor.processFile ... */ }
// ...
setInterval(mainLoop, CONFIG.SYSTEM.INTERVAL);
Enter fullscreen mode Exit fullscreen mode

And Executor.processFile began, in its entirety before the model call, with loading a context file. Then:

response = await retryOpenAI(() =>
    axios.post(
        'https://api.openai.com/v1/chat/completions',
        // ... STRICT JSON: {action:"EVOLVE"|"SKIP", newCode:string, newFilePath:string}
Enter fullscreen mode Exit fullscreen mode

No gate in front of it. Every processed file meant a completion.

Now the arithmetic. A 2-minute interval is 720 ticks per day. The daily request counts from 15 to 23 June were:

06-15 2169   06-16 2172   06-17 2172   06-18 2169   06-19 2172
06-20 2173   06-21 2169   06-22 2172   06-23 2172
Enter fullscreen mode Exit fullscreen mode

If the 2-minute timer were the only caller, that's 2,172 / 720 = 3.02 requests per tick. I wouldn't lean on that ratio, though: in an earlier investigation, the resolved observation windows showed that in July production ran about one pass every 5 seconds, not one every 2 minutes, and the only other caller of mainLoop() at the time was the then-unauthenticated POST /webhook. So "ticks" in June may well have been far more frequent than 720 a day, with most passes ending before the model. The input-token totals are the stronger evidence:

06-15 1,724,355   06-16 1,726,740   06-17 1,726,740   06-18 1,724,355
06-19 1,726,740   06-20 1,728,011   06-21 1,724,355   06-22 1,726,740
06-23 1,726,740
Enter fullscreen mode Exit fullscreen mode

Five days at exactly 1,726,740 tokens, three days at exactly 1,724,355. That's what deterministic redundancy looks like on an invoice: the same files, the same prompt template, the same context, the same number of ticks, summed over a day. If even one prompt had carried new information, those totals would drift. They didn't.

I don't have per-request logs for June, so I can't say how many passes produced how many calls. What matters is that the number is constant, and constant is the signature.

That week cost only about $0.21 a day, because by then it was on gpt-4o-mini. That's the insidious part. Cheap loops don't get fixed, because nobody gets paged for $0.21. Run the same loop on May's model and it's $7–10 a day.

11.4 July: the last spike and the Novelty Gate

15–16 July (3,005 and 4,668 requests) coincide with a large merge day: the executor stack, analytics, a "TranslatorGateway" for all external AI, and benchmark frameworks. I don't know how much of that traffic was production cycling and how much was testing against the real API. I'm leaving it unexplained rather than inventing a story.

On 18 July, 4c197562590:

feat(novelty): reuse a prior SKIP when nothing changed (novelty gate)

Confidence answers 'how reliable is my judgement?' and must not drift up
from repetition. A new local novelty gate answers 'has anything changed
since the last decision?': it fingerprints content hash + risk + role +
confidence + lifecycle status/rollback pressure, and when that is unchanged
and the previous decision was a SKIP it reuses it (reused:true) instead of
re-reasoning — no LLM, no duplicate memories. Only SKIP is cached (it is
deterministic given identical inputs); EVOLVE/REJECT/rollback/deferred are
not.
Enter fullscreen mode Exit fullscreen mode

From 18 July to 1 September the Sentinel bill shows 61 requests in total.

That's the hinge of this whole story. Everything after it (Wake Gate, budgets, triggers, error circuits) is about making the quiet cheap, safe and honest, and making sure the rare exceptions can't turn into new loops.

11.5 September–October: making quiet cheap, then making exceptions finite

The Novelty Gate stopped redundant thinking, but the agent still did redundant work: every 2 minutes it listed the repo, fetched 5 files, ran them through the deterministic pipeline, fingerprinted them, and saved memory. No LLM, but GitLab API calls, CPU, and a memory write each tick.

On 2 September, f1ed6a48ca1:

feat(executor): event-driven wake gate, no-op when no actionable event

One commit lookup (latest SHA under libs/shared) + evidence version + existing
observation/opportunity dues decide wake_reason before the main loop fetches,
parses or saves anything. reason=none returns immediately.
Enter fullscreen mode Exit fullscreen mode

Then October was about the legitimate reasons to think again, each of which is a potential loop:

  • Autonomy budget (5dfeb61dc79, 59e21771827, e7f66e36840): reconsidering a skipped file costs a token; tokens are limited per day; one token per target; a 24-hour per-file cooldown.
  • Triggers (d5b03f00048): no concrete trigger → no_trigger → no wake.
  • "Reason already answered" (09c6334a3c6): the same trigger can't reopen the same file twice.
  • LLM error protocol (36f94f8c896): classify failures; retry only where retrying can help; open a circuit otherwise.
  • Webhook guard (adb1f059fc4): token-authenticated, and at most one cycle per 30 seconds. The commit's own test case: 20 requests used to mean 20 cycles.

Let's look at the code.


12. The containment stack, in the order a tick hits it

Here is the life of one tick today, top to bottom. Every layer can end the tick early, and the cheapest layers come first.

flowchart TD
    T["timer, every 2 min"] --> B
    W["POST /webhook"] --> G{"1. WebhookGuard<br/>token? busy? under 30 s since last start?"}
    G -- "no" --> R1["401 / 202, nothing started"]
    G -- "yes" --> B{"2. isBusy?"}
    B -- "busy" --> X1["return"]
    B -- "free" --> L{"3. RuntimeLock<br/>240 s lease, owner = INSTANCE_ID"}
    L -- "held by another" --> X2["return"]
    L -- "acquired" --> WG{"4. WakeGate<br/>1 SHA lookup + evidence + dues + autonomy"}
    WG -- "reason = none" --> X3["return: no fetch, no parse, no save"]
    WG -- "wake" --> S["scan batch of 5 files<br/>deterministic pipeline"]
    S --> N{"5. NoveltyGate<br/>same fingerprint + prior SKIP?"}
    N -- "yes" --> RU["reuse SKIP, no LLM"]
    N -- "no" --> P{"6. local policy<br/>confident local SKIP?"}
    P -- "yes" --> LS["local SKIP, no LLM"]
    P -- "no" --> A{"7. AutonomyBudget<br/>needed and allowed?"}
    A -- "refused" --> AS["no LLM"]
    A -- "ok" --> M{"8. BudgetManager.canSpend()<br/>estimated daily spend under $2?"}
    M -- "no" --> D["defer"]
    M -- "yes" --> E{"9. LlmErrorProtocol<br/>circuit open?"}
    E -- "open" --> H["hold file"]
    E -- "closed" --> LLM["LLM advisor"]
    LLM --> GATES["deterministic gates decide"]

The order matters. A loop guard is only as good as the cheapest work it prevents. The Novelty Gate saves an LLM call but still costs a GitLab fetch and a full pipeline run. The Wake Gate saves all of that for the price of one HTTP call. If I had to keep only one layer, the Wake Gate does most of the work, but only because the Novelty Gate is behind it for the ticks that do wake.

12.1 Webhook Guard: authenticate, then admit at most one

app.post('/webhook',(req,res)=>{

    if(!WebhookGuard.authorized(req.get('X-Gitlab-Token'))){
        return res.status(401).json({ ok:false, error:'unauthorized' });
    }

    const admission = webhookLimiter.admit({ busy:isBusy });

    if(!admission.start){
        return res.status(202).json({ ok:true, started:false, reason:admission.reason });
    }

    res.status(200).send('OK');

    mainLoop();

});
Enter fullscreen mode Exit fullscreen mode

And the limiter in libs/core/webhook-guard.js:

const MIN_INTERVAL_MS = 30 * 1000;

// inside createLimiter()
admit({ busy = false, now = Date.now() } = {}) {
    if (busy) return { start: false, reason: 'busy' };
    if (lastStartTs !== null && now - lastStartTs < minIntervalMs) {
        return {
            start: false,
            reason: 'rate_limited',
            retryAfterMs: minIntervalMs - (now - lastStartTs)
        };
    }
    lastStartTs = now;
    return { start: true, reason: 'started' };
}
Enter fullscreen mode Exit fullscreen mode

Three design notes:

  • A busy refusal doesn't consume the window. If it did, a long cycle plus a steady trickle of webhooks could keep the window permanently "just used" and starve legitimate events. There's a test for exactly that (§14).
  • 202, not 429. A refused-but-valid poke isn't an error. Answering "accepted, not started" keeps GitLab from logging a failed delivery (repeated failures can get a webhook auto-disabled), and gives nobody a reason to build retry logic around it, which would be a loop of its own.
  • The token check is constant-time. authorized() hashes both the presented and the expected token to fixed-length digests and compares them with crypto.timingSafeEqual, so a wrong-length token can't throw or leak its length through timing. With no WEBHOOK_SECRET set, the endpoint stays open and reports that (/health shows webhookAuth). In production it's enabled. That's security rather than loops, but an open webhook is also a free "make my agent spend money" button for anyone who finds the URL.

Notice what the webhook does not do any more: it doesn't carry any decision. It just pokes mainLoop(), which goes through the same Wake Gate as a timer tick. If nothing changed, a webhook-triggered tick is a no-op too. The echo loop isn't stopped by recognising "my own commit" (fragile). It's stopped structurally: the agent writes to a sandbox branch (CONFIG.GITLAB.BRANCH: 'sentinel-sandbox'), while the Wake Gate watches the default branch (DEFAULT_BRANCH, master). Its own writes don't move the SHA it wakes on.

12.2 isBusy and RuntimeLock: one cycle at a time

async function mainLoop(){

    if(isBusy) return;

    isBusy=true;

    let memory=null;
    let lockAcquired=false;

    try{
        await memoryRestore;
        memory=await MemoryManager.load();

        lockAcquired=
            RuntimeLock.acquire(
                memory,
                INSTANCE_ID
            );

        if(!lockAcquired){
            return;
        }
        // ...
Enter fullscreen mode Exit fullscreen mode

isBusy handles overlap within one process, which is what setInterval + async used to produce. RuntimeLock handles overlap between processes (a deploy that briefly runs two instances, a restart while the old one is draining):

acquire: (memory, owner) => {
    const now = Date.now();
    memory.runtime_lock = memory.runtime_lock || { owner: null, expires: 0 };

    if (
        memory.runtime_lock.owner &&
        memory.runtime_lock.owner !== owner &&
        now < memory.runtime_lock.expires
    ) {
        return false;
    }

    memory.runtime_lock = {
        owner,
        expires: now + 240000
    };

    return true;
},
Enter fullscreen mode Exit fullscreen mode

The 240-second expiry is the dead-man's switch: a crashed owner can't hold the lock forever. It's a lease, not a mutex. I'm not claiming it's a correct distributed lock in the presence of clock skew and partitions (it isn't; it lives in a JSON memory file). It's enough for "Render briefly runs two containers during a deploy".

The timer is still setInterval(mainLoop, CONFIG.SYSTEM.INTERVAL), by the way. With isBusy as the first line, overlapping callbacks return immediately, so the classic setInterval overlap bug is closed by the callee rather than by the scheduler.

12.3 Wake Gate: "has anything actually happened?"

The whole decision is one pure function in libs/core/wake-gate.js:

function evaluate(memory, { headSha = null, evidenceVersion = null, autonomy = null, now = Date.now() } = {}) {
    const s = state(memory);

    if (!s.lastHeadSha && !s.lastEvidenceVersion) {
        return { wake: true, reason: REASONS.FIRST_RUN, detail: null };
    }

    if (!headSha) {
        return { wake: true, reason: REASONS.SHA_UNAVAILABLE, detail: null };
    }

    if (headSha !== s.lastHeadSha) {
        return { wake: true, reason: REASONS.REPO_CHANGED, detail: { from: s.lastHeadSha, to: headSha } };
    }

    if (evidenceVersion && evidenceVersion !== s.lastEvidenceVersion) {
        return { wake: true, reason: REASONS.EVIDENCE_CHANGED, detail: null };
    }

    const observation = observationDue(memory, now);
    if (observation) {
        return { wake: true, reason: REASONS.OBSERVATION_DUE, detail: { path: observation } };
    }

    const opportunity = opportunityDue(memory, now);
    if (opportunity) {
        return { wake: true, reason: REASONS.OPPORTUNITY_DUE, detail: { path: opportunity } };
    }

    if (autonomy?.eligible === true && autonomy.path) {
        return { wake: true, reason: REASONS.AUTONOMY_DUE, detail: { path: autonomy.path, trigger: autonomy.trigger ?? null, /* ... */ } };
    }

    return { wake: false, reason: REASONS.NONE, detail: autonomy?.reason ? { autonomy: autonomy.reason } : null };
}
Enter fullscreen mode Exit fullscreen mode

And in mainLoop(), before any scan:

const headSha = await GitLabClient.latestCommitSha(
    RepositoryScanner.TARGET_DIR,
    CONFIG,
    CONFIG.GITLAB.DEFAULT_BRANCH
);
// evidenceVersion = governance + dependency manifest version
// autonomy = AutonomyBudget.admit(...)  (pure: may say "eligible" or why not)

wake = WakeGate.evaluate(memory, { headSha, evidenceVersion, autonomy });
WakeGate.remember(memory, wake, { headSha, evidenceVersion });
console.log(WakeGate.oneLine(wake));

if (!wake.wake) {
    await MemoryManager.retryMirror();
    return;
}
Enter fullscreen mode Exit fullscreen mode

latestCommitSha is a single GET /repository/commits?path=libs/shared&per_page=1 and returns null on any failure.

The things I'd point at in review:

1. !headSha wakes. This is the most important line in the file. The tempting version is if (headSha === s.lastHeadSha) return none, which treats null === null... or, after a failed lookup, compares null with a real SHA and calls that a change. Either bug is bad: the first makes the agent go blind whenever GitLab hiccups, the second makes it wake on every hiccup. Unknown gets its own reason, sha_unavailable, and it wakes, because a missed real change is worse than a wasted scan. In production it has fired 23 times out of 16,314 ticks.

2. Order is precedence. An external change (repo_changed) outranks scheduled reasons, which outrank autonomy. There's a test for that. It matters because the wake reason travels into the cycle: WakeGate.prioritize(files, wake) moves the file a due-review is about to the front of the batch, so "woke up for file X" can't end with "and then didn't look at X because it sorted last" (another test).

3. none doesn't write memory. A no-op tick that saves memory isn't a no-op. It's a write per tick, and on this system memory is mirrored to GitLab, so it'd be an API call per tick. Yet you want no-ops counted. The answer is a tiny in-process carry:

// No-op ticks must not save memory (that is the whole point), yet they still
// have to be counted. They accumulate here, in-process, and are folded into
// memory on the next tick that does wake and save.
const carry = { ticks: 0, noops: 0, lastTickTs: null };

function remember(memory, verdict, { headSha = null, evidenceVersion = null, now = Date.now() } = {}) {
    if (!verdict?.wake) {
        carry.ticks += 1;
        carry.noops += 1;
        carry.lastTickTs = now;
        return null;
    }
    const s = state(memory);
    s.ticks = num(s.ticks) + carry.ticks + 1;
    s.noops = num(s.noops) + carry.noops;
    // ...
Enter fullscreen mode Exit fullscreen mode

The trade-off is visible in production as unsavedNoops: 245: ticks counted in RAM but not yet persisted. A restart loses them. I'll take inaccurate statistics over a write per tick.

4. The timer stays. You might ask why not drop polling and go fully event-driven. Because some events aren't pushes: a scheduled review falling due, an observation window closing, a production error appearing in logs. The timer is now a clock for the gate, not a trigger for work. It ticks 720 times a day and almost all of those ticks cost one HTTP call.

12.4 Novelty Gate: "have I already answered exactly this?"

The fingerprint (libs/core/novelty-gate.js):

function fingerprint(input = {}) {
    const {
        content,
        risk = null,
        role = null,
        confidence = null,
        status = null,
        rollbackStreak = 0,
        penalty = 0,
        evidenceVersion = null,
        dependencies = null
    } = input;

    const contentHash = crypto
        .createHash("sha256")
        .update(String(content ?? ""))
        .digest("hex")
        .slice(0, 16);

    const conf = Number.isFinite(Number(confidence))
        ? Math.round(Number(confidence) * 100) / 100
        : "na";

    const parts = [
        contentHash,
        risk ?? "na",
        role ?? "na",
        conf,
        status ?? "na",
        Number(rollbackStreak) || 0,
        Number(penalty) || 0,
        evidenceVersion ?? "na"
    ];
    if (dependencies) parts.push(String(dependencies));
    return parts.join("|");
}
Enter fullscreen mode Exit fullscreen mode

The decision:

function reusable(decision) {
    return !!decision &&
        decision.action === "SKIP" &&
        typeof decision.reason === "string";
}

// evaluate(memory, filePath, fp, { dependenciesKnown })
if (!prev) return { novel: true, reason: "first-seen", previous: null };
if (dependenciesKnown === false) return { novel: true, reason: "dependencies-unknown", previous: prev };
if (prev.fingerprint !== fp) return { novel: true, reason: "changed", previous: prev };
if (!reusable(prev.decision)) return { novel: true, reason: "prev-not-reusable", previous: prev };
return { novel: false, reason: "unchanged", previous: prev };
Enter fullscreen mode Exit fullscreen mode

Things worth stealing:

Confidence is rounded to two decimals before fingerprinting. Floating-point jitter in a computed confidence (0.8700001 vs 0.87) would otherwise flip the fingerprint every cycle and turn the cache into a very elaborate no-op. A test pins "tiny confidence jitter within rounding does NOT flip it".

Bytes aren't the whole world. The file's own content is only the first component. If the governance rules change (evidenceVersion) or a local module this file requires changes (dependencies), the same bytes mean something different, and the agent must look again. The executor's comment puts it nicely: "'Nothing changed' has to mean nothing relevant changed, not merely that the bytes are identical." And if dependency state can't be determined, dependencies-unknown forces a fresh evaluation. Same principle as sha_unavailable: unknown is never unchanged.

Only SKIP is reusable. This is the subtle one. A SKIP ("leave it alone, reason R") is deterministic given identical inputs. An EVOLVE isn't safe to replay: it produced a change, so the next state is different by construction. A REJECT or rollback is a negative event the system should keep learning from. And a deferred outcome (budget exhausted, provider down) is explicitly "I didn't decide", so caching it would turn a temporary refusal into a permanent one, which is a sneaky way to build a loop in the other direction: the "never re-evaluate" loop.

Scheduled reviews punch through. If the Opportunity Engine says a frozen module is due for its periodic review, the cached SKIP must not swallow that, or nextReview would never move and the module would be due forever. That would be a Wake Gate loop: wake for opportunity_due, reuse SKIP, nothing changes, still due, wake again. Here's the guard in the executor:

// A scheduled opportunity review is an event in its own right: a cached
// SKIP must not swallow it, or a frozen module's `nextReview` never moves.
const scheduledReviewDue = /* ...lifecycle says skip && OpportunityEngine.dueForCheck(...) */;

if (noveltyGateEnabled && !scheduledReviewDue) {
    const novelty = NoveltyGate.evaluate(memory, filePath, noveltyFingerprint, {
        dependenciesKnown: localDeps.known
    });
    if (!novelty.novel && !reconsider("novelty-reuse")) {
        NoveltyGate.reuse(memory, filePath, saved);
        // ...
        return { action: novelty.previous.decision.action, reason: novelty.previous.decision.reason, reused: true, envelope };
    }
}
Enter fullscreen mode Exit fullscreen mode

I found that one the hard way. Every guard you add is a new state machine, and every new state machine has its own way of getting stuck.

12.5 Local policy before the model

After novelty, the executor still tries to avoid the LLM. DecisionPolicy.shouldConsultLLM(envelope, referenceGate) looks at the deterministic risk/confidence envelope and only consults the model on low confidence or real novelty. A local SKIP is recorded and fed back into the Novelty Gate:

if (!llmGate.consult) {
    console.log(`💸 LLM SKIPPED: ${filePath} | ${llmGate.reason}`);
    // ...
    NoveltyGate.remember(memory, filePath, noveltyFingerprint, { action: "SKIP", reason: llmGate.reason });
    // ...
}
Enter fullscreen mode Exit fullscreen mode

That last line is the April idea (3cc77c956e2, "marked with its hash even on SKIP") finally in its permanent home, with tests.

12.6 Autonomy: letting it think again, finitely

A watcher that can never reconsider anything is just a very expensive if. Sentinel can reconsider a skipped file, which is exactly the overthinker loop waiting to happen. AutonomyBudget.admit() is a pure function that walks a list of refusals before it ever says yes:

const ADMISSION = {
    // ...
    EXHAUSTED: "autonomy_budget_exhausted",   // no tokens left today
    WINDOW_CLOSED: "autonomy_window_closed",  // spacing: next eligible time not reached
    NO_TARGET: "no_eligible_target",          // every file is inside its 24 h cooldown
    NO_TRIGGER: "no_trigger",                 // nothing concrete happened
    ANSWERED: "reason_already_answered",      // this exact trigger was already answered
    PROVIDER_HELD: "llm_provider_held",       // LLM circuit is open (§12.8)
};

const DEFAULT_DAILY_LLM = 10;
const HARD_CAP = 100;
const FILE_COOLDOWN_MS = 24 * 60 * 60 * 1000;
Enter fullscreen mode Exit fullscreen mode

In order: wrong mode → no; LLM circuit open → no; tokens exhausted → no; spacing window not reached → no; no file outside its 24-hour cooldown → no; no file with a deterministic trigger → no; the trigger's fingerprint equals the one already answered for that file → no. Only then is there an eligible: true with a path and a trigger, which the Wake Gate turns into autonomy_due.

Triggers (libs/core/autonomy-trigger.js) are evidence, not authority: a runtime error for that file in production logs from the last 24 hours, an unresolved rollback, an open Opportunity Engine proposal, dependency drift, a wrong stability prediction. A correct prediction isn't a trigger. A stale (>24 h) log line isn't a trigger. A locked module is never a target, however strong the trigger.

reason_already_answered deserves a paragraph. Before it existed (until 11 October), a persistent trigger, say a peer module that stays unstable for days, would legitimately reopen the same file every 24 hours, get the same answer and spend a token every day. That's the overthinker loop with a daily period. It's slow enough that you won't see it in a request graph, but it's still a loop. Fingerprinting the trigger itself and refusing to answer an identical question twice turns "every day forever" into "once per new fact".

At the capture time, production autonomy was in exactly that state: mode controlled, limit 100, 0 used today, admission eligible: false, reason reason_already_answered.

12.7 Budget: a ceiling, with a known blind spot

// libs/knowledge/budget-manager.js
const DAILY_LIMIT = 2.00;
// canSpend(): resets on a new UTC date, then returns dailySpend < DAILY_LIMIT

// libs/core/executor.js, just before the call
if (!BudgetManager.canSpend()) {
    console.log(`💰 BUDGET EXHAUSTED: deferring LLM for ${filePath}`);
    memory.decision_stream.unshift({
        path: filePath,
        decision: { action: "SKIP", reason: "daily-budget-exhausted" },
        ts: Date.now()
    });
    // ... recorded as "deferred", NOT remembered by the Novelty Gate
}
// ... after a successful call
const costPerCall = Number(config?.AI?.COST_PER_CALL ?? 0.01);
BudgetManager.addSpend(costPerCall);
Enter fullscreen mode Exit fullscreen mode

The blind spot is right there in costPerCall. The guard counts a configured estimate per call (AI_COST_PER_CALL, default $0.01), not the invoice. When a request is much bigger than average (long context, cache writes, a pricier model), the estimate undercounts and the guard bites late.

That's why there's a second, read-only organ, libs/core/openai-cost-organ.js. It polls the same two Admin endpoints this article uses (whitelisted, GET only) and compares real spend against the budget and against the estimate:

const realPerCall = yUsd / yReq;
const ratio = realPerCall / perCall;
// ...
if (ratio >= ESTIMATE_DRIFT_RATIO) verdicts.push('estimate-too-low');
Enter fullscreen mode Exit fullscreen mode

Its advisories from production are a good example of why I insist on "estimate" vs "billed":

When (UTC) Verdict Evidence
21 Sep 00:07 unexpected-model gpt-4o-mini-2024-07-18 billing while another model was configured
25 Sep 04:56 over-daily-budget real spend that UTC day: $5.09
29 Sep 02:44 estimate-too-low real per-call cost ≈ 15.45× the configured $0.01
29 Sep 14:44 over-daily-budget $2.35

Read that third line again: the agent's own wallet thought each call cost a cent, while the real average was about fifteen cents. A $2 estimated ceiling at a 15× undercount is a ~$30 real ceiling. The organ found it, which is the point of having it. But the organ is advisory: it doesn't block calls. (Its summary string says "LLM calls are now deferred". That's only true if the internal estimate also hits $2, and I consider that wording a bug to fix.) The production counters at capture: 30 calls denied with daily-budget-exhausted over the instance's lifetime.

Lesson: a budget guard needs a feedback loop from the real bill, or it guards a number you made up.

12.8 LLM error protocol: retry only what retrying can fix

const PROTOCOL = Object.freeze({
    [CLASSES.TIMEOUT]:      { retries: 2, backoffMs: [1000, 2000],  circuitMs: null },
    [CLASSES.NETWORK]:      { retries: 2, backoffMs: [2000, 4000],  circuitMs: 5 * MIN },
    [CLASSES.SERVER]:       { retries: 2, backoffMs: [2000, 4000],  circuitMs: 10 * MIN },
    [CLASSES.RATE_LIMIT]:   { retries: 2, backoffMs: [5000, 15000], circuitMs: 2 * MIN },
    [CLASSES.QUOTA]:        { retries: 0, backoffMs: [],            circuitMs: "day" },
    [CLASSES.AUTH]:         { retries: 0, backoffMs: [],            circuitMs: 60 * MIN },
    [CLASSES.REQUEST]:      { retries: 0, backoffMs: [],            circuitMs: 60 * MIN },
    [CLASSES.BAD_RESPONSE]: { retries: 1, backoffMs: [1000],        circuitMs: null },
    [CLASSES.UNKNOWN]:      { retries: 0, backoffMs: [],            circuitMs: 5 * MIN }
});
Enter fullscreen mode Exit fullscreen mode

(Advice strings trimmed. Each class also carries a human-readable advice, e.g. quota: "billing quota exhausted: no further calls today; human must top up or raise the limit".)

Classification is the important part:

else if (status === 429) cls = apiType === "insufficient_quota" || apiCode === "insufficient_quota"
    ? CLASSES.QUOTA : CLASSES.RATE_LIMIT;
else if (status === 401 || status === 403) cls = CLASSES.AUTH;
else if ([400, 404, 413, 422].includes(status)) cls = CLASSES.REQUEST;
else if (status >= 500 && status <= 599) cls = CLASSES.SERVER;
Enter fullscreen mode Exit fullscreen mode

Both "too many requests per minute" and "you're out of money" come back as HTTP 429. Retrying the first is correct. Retrying the second is a loop until midnight. If your retry logic keys on status codes alone, it can't tell them apart. Look at error.type / error.code.

Once a class's retries are exhausted, record() opens a circuit in memory.llm_health, open(memory, now) is checked before every call, and the autonomy admission refuses with llm_provider_held while it's open. So a provider outage doesn't just stop calls. It stops the wanting to make calls too. A Retry-After header wins over the table (capped at 60 s), and a success closes the circuit.

One more detail that prevents a quiet, slow loop: when a call fails and the file is deferred, the autonomy token spent on it is refunded, along with its cooldown and "answered" marker. Otherwise an outage would burn the day's tokens and mark triggers as answered when nothing was answered.


13. The before/after harness

Talk is cheap; let's run it. This script uses Sentinel's real WakeGate, NoveltyGate, WebhookGuard and LlmErrorProtocol modules, unmodified, with a virtual clock and an "LLM" that's just a counter. No network.

Three scenarios:

  1. The ordinary day. 5 files, a 2-minute tick (720 ticks), one commit at 12:00 that changes one file. Compare: (a) June-style "every tick, every file, one call each"; (b) July-style Novelty Gate only; (c) today: Wake Gate + Novelty Gate.
  2. The webhook storm. 20 webhook POSTs in 10 seconds; a cycle takes 3 seconds.
  3. The quota wall. The billing quota runs out at 09:00 and every call after that returns 429 insufficient_quota.
// loop-demo.js: before/after with real Sentinel gate modules, virtual clock, no network.
const R = process.env.SENTINEL_REPO || '../../repos/AutoDoc-Sentinel';
const WakeGate = require(`${R}/libs/core/wake-gate`);
const NoveltyGate = require(`${R}/libs/core/novelty-gate`);
const WebhookGuard = require(`${R}/libs/core/webhook-guard`);
const LlmErrorProtocol = require(`${R}/libs/core/llm-error-protocol`);

const MIN = 60 * 1000, DAY = 24 * 60 * MIN, TICK = 2 * MIN;
const T0 = Date.UTC(2026, 5, 16, 0, 0, 0);
const FILES = ['a.js', 'b.js', 'c.js', 'd.js', 'e.js'];

function repoAt(t) {                       // one commit at 12:00 changes c.js
    const changed = t >= T0 + 12 * 60 * MIN;
    const files = FILES.map(p => ({ path: p, content: `// ${p}\nmodule.exports = 1;${p === 'c.js' && changed ? '\n// v2' : ''}` }));
    return { sha: changed ? 'sha-2' : 'sha-1', files };
}

function scenarioDay() {
    const naive = { ticks: 0, reads: 0, llm: 0 };
    const novelty = { ticks: 0, reads: 0, llm: 0, reused: 0 };
    const sentinel = { ticks: 0, wakes: 0, noops: 0, reads: 0, llm: 0, reused: 0, byReason: {} };
    const memN = {}, memS = {};
    WakeGate.resetCarry();
    for (let t = T0; t < T0 + DAY; t += TICK) {
        const repo = repoAt(t);
        // BEFORE (June 2026 style): every tick, every file, one model call each.
        naive.ticks++; for (const f of repo.files) { naive.reads++; naive.llm++; }
        // July 2026: Novelty Gate only. Still reads every file every tick.
        novelty.ticks++;
        for (const f of repo.files) {
            novelty.reads++;
            const fp = NoveltyGate.fingerprint({ content: f.content });
            const v = NoveltyGate.evaluate(memN, f.path, fp);
            if (v.novel) { novelty.llm++; NoveltyGate.remember(memN, f.path, fp, { action: 'SKIP', reason: 'stable' }, t); }
            else { novelty.reused++; NoveltyGate.reuse(memN, f.path, 0, t); }
        }
        // AFTER (today): Wake Gate first (one SHA lookup), Novelty Gate per file.
        sentinel.ticks++;
        const wake = WakeGate.evaluate(memS, { headSha: repo.sha, evidenceVersion: 'ev-1', now: t });
        WakeGate.remember(memS, wake, { headSha: repo.sha, evidenceVersion: 'ev-1', now: t });
        sentinel.byReason[wake.reason] = (sentinel.byReason[wake.reason] || 0) + 1;
        if (!wake.wake) { sentinel.noops++; continue; }
        sentinel.wakes++;
        for (const f of repo.files) {
            sentinel.reads++;
            const fp = NoveltyGate.fingerprint({ content: f.content });
            const v = NoveltyGate.evaluate(memS, f.path, fp);
            if (v.novel) { sentinel.llm++; NoveltyGate.remember(memS, f.path, fp, { action: 'SKIP', reason: 'stable' }, t); }
            else { sentinel.reused++; NoveltyGate.reuse(memS, f.path, 0, t); }
        }
    }
    return { naive, novelty, sentinel, wakeStats: WakeGate.stats(memS) };
}

function scenarioWebhookStorm() {
    // 20 POST /webhook in 10 s (one every 500 ms); one cycle takes 3 s.
    const CYCLE_MS = 3000, N = 20, GAP = 500;
    let naiveCycles = N;
    const limiter = WebhookGuard.createLimiter();
    let busyUntil = 0; const verdicts = {}; let cycles = 0;
    for (let i = 0; i < N; i++) {
        const now = T0 + i * GAP;
        const a = limiter.admit({ busy: now < busyUntil, now });
        verdicts[a.reason] = (verdicts[a.reason] || 0) + 1;
        if (a.start) { cycles++; busyUntil = now + CYCLE_MS; }
    }
    return { requests: N, naiveCycles, guardedCycles: cycles, verdicts };
}

async function scenarioQuota() {
    // Billing quota runs out at 09:00; the rest of the day is 450 ticks x 5 files.
    const quotaErr = () => Object.assign(new Error('quota'), { response: { status: 429, data: { error: { type: 'insufficient_quota', code: 'insufficient_quota' } }, headers: {} } });
    const start = T0 + 9 * 60 * MIN;
    // BEFORE: generic retry (3 attempts) on every file, every tick.
    let naiveCalls = 0;
    for (let t = start; t < T0 + DAY; t += TICK) for (const _ of FILES) naiveCalls += 3;
    // AFTER: classify once, open the circuit until the next UTC day.
    LlmErrorProtocol.setSleep(async () => {});
    const mem = {}; let calls = 0, heldFiles = 0;
    for (let t = start; t < T0 + DAY; t += TICK) {
        for (const _ of FILES) {
            if (LlmErrorProtocol.open(mem, t)) { heldFiles++; continue; }
            try { await LlmErrorProtocol.run(async () => { calls++; throw quotaErr(); }); }
            catch (e) { LlmErrorProtocol.record(mem, e.protocol, t); }
        }
    }
    const c = LlmErrorProtocol.view(mem, start + MIN).circuit;
    return { naiveCalls, protocolCalls: calls, heldFiles, circuitClass: c?.class, circuitUntil: new Date(c.until).toISOString() };
}

(async () => {
    const out = { day: scenarioDay(), webhookStorm: scenarioWebhookStorm(), quota: await scenarioQuota() };
    console.log(JSON.stringify(out, null, 2));
})();
Enter fullscreen mode Exit fullscreen mode

Output (Node v24.19.0, Sentinel master at 1bc30bab7be), trimmed only of the timestamps inside wakeStats:

{
  "day": {
    "naive":    { "ticks": 720, "reads": 3600, "llm": 3600 },
    "novelty":  { "ticks": 720, "reads": 3600, "llm": 6, "reused": 3594 },
    "sentinel": { "ticks": 720, "wakes": 2, "noops": 718, "reads": 10, "llm": 6, "reused": 4,
                  "byReason": { "first_run": 1, "none": 718, "repo_changed": 1 } },
    "wakeStats": { "ticks": 720, "wakes": 2, "noops": 718, "noopRate": 0.9972,
                   "byReason": { "none": 718, "first_run": 1, "repo_changed": 1 },
                   "unsavedNoops": 359, "lastWakeReason": "repo_changed", "lastHeadSha": "sha-2" }
  },
  "webhookStorm": {
    "requests": 20, "naiveCycles": 20, "guardedCycles": 1,
    "verdicts": { "started": 1, "busy": 5, "rate_limited": 14 }
  },
  "quota": {
    "naiveCalls": 6750, "protocolCalls": 1, "heldFiles": 2249,
    "circuitClass": "insufficient-quota", "circuitUntil": "2026-06-17T00:00:00.000Z"
  }
}
Enter fullscreen mode Exit fullscreen mode

Summarised:

Scenario Before Novelty only After (today)
Ordinary day: model calls (upper bound) 3,600 6 6
Ordinary day: file reads + pipeline runs 3,600 3,600 10
Ordinary day: ticks that did any work 720 720 2
Webhook storm: cycles started 20 – 1
Quota wall: provider calls after 09:00 6,750 – 1

How to read it honestly:

  • "llm" in the Novelty and Sentinel columns is an upper bound. It counts fresh evaluations. In the real executor a fresh evaluation still goes through local policy, so many of them end as a local SKIP without a model call.
  • The naive baseline is a model of June, not a replay. Real June was ~2,170 requests/day, not 3,600. The model assumes every file reaches the LLM. The shape (requests proportional to ticks, not to changes) is what the June bill shows.
  • The two columns show different savings. The Novelty Gate alone already gets model calls down to 6. What the Wake Gate adds is the 3,590 avoided reads/pipeline runs and 718 avoided memory saves: not tokens, but GitLab API calls, CPU and noise. Both layers matter, for different resources.
  • unsavedNoops: 359. In this run the counters for the 359 no-op ticks since the noon wake are still in the in-process carry. Same trade-off as production's 245.
  • The quota scenario's "before" is hypothetical: three attempts per file per tick. Sentinel's June retryOpenAI actually made two attempts (retries = 2). With two, it's 4,500 instead of 6,750. Same order of magnitude, same conclusion.
  • The webhook verdicts are worth a look: busy: 5 during the 3-second cycle, then rate_limited: 14 for the rest of the 30-second window. The busy refusals didn't consume the window. That's the property pinned by a unit test.

14. Tests: what's actually pinned

All of the above is covered by the repository's normal suite, node --test tests/*.test.js tests/unit/*.test.js, which on master 1bc30bab7be with Node v24.19.0 reports:

# tests 722
# suites 11
# pass 722
# fail 0
# cancelled 0
# skipped 0
# duration_ms 9800.352678
Enter fullscreen mode Exit fullscreen mode

The anti-loop files, run individually:

File Tests Pass
tests/unit/wake-gate.test.js 14 14
tests/unit/novelty-gate.test.js 12 12
tests/unit/novelty-executor.test.js 2 2
tests/unit/autonomy-budget.test.js 10 10
tests/unit/autonomy-trigger.test.js 12 12
tests/unit/autonomy-wake.test.js 15 15
tests/unit/llm-error-protocol.test.js 8 8
tests/unit/webhook-guard.test.js 6 6
tests/unit/budget-manager.test.js 2 2
tests/webhook.test.js (integration) 4 4
Total 85 85

Test names read like a specification of everything I've ranted about. A selection, verbatim:

sha_unavailable: unknown SHA never masquerades as unchanged
none: identical SHA + evidence, nothing due → no wake, memory untouched
precedence: repo change outranks evidence and dues
prioritize: the due file ends up inside the batch even when it scores last
opportunity_due: a locked module with a stale nextReview does not wake (scanner never returns it)
fingerprint: tiny confidence jitter within rounding does NOT flip it
evaluate: a non-SKIP prior decision is never reused
limiter: 20 rapid admissions start exactly one cycle
limiter: a busy cycle starts nothing and does not consume the window
executor: 429 insufficient_quota → 1 attempt, held until the next UTC day; 401 → 1 attempt, 1 h hold
executor: HTTP 500 → 3 attempts, deferred, token refunded, 10 min circuit; the next file is held without a call
a cooled-down, unlocked file with no signal is NOT a target: no_trigger, no wake, nothing spent
a wrong stability prediction (observation mode) is a trigger; a correct one is not
locked module with a strong trigger is still never a target
existing non-autonomy wake still wins over a triggered autonomy_due
Enter fullscreen mode Exit fullscreen mode

Two of them in full, because they show the style: tiny, deterministic, injected clock, assert the exact shape.

test('none: identical SHA + evidence, nothing due → no wake, memory untouched', () => {
    const memory = settled();
    const before = JSON.stringify(memory);

    const v = WakeGate.evaluate(memory, { headSha: SHA_A, evidenceVersion: 'ev1', now: NOW });
    assert.deepEqual(v, { wake: false, reason: 'none', detail: null });

    const rec = WakeGate.remember(memory, v, { headSha: SHA_A, evidenceVersion: 'ev1', now: NOW });
    assert.equal(rec, null);
    assert.equal(JSON.stringify(memory), before, 'a none tick must not mutate memory');

    const s = WakeGate.stats(memory);
    assert.equal(s.ticks, 1);
    assert.equal(s.noops, 1);
    assert.equal(s.unsavedNoops, 1);
    assert.equal(s.byReason.none, 1);
    assert.equal(s.noopRate, 1);
});
Enter fullscreen mode Exit fullscreen mode
test('limiter: a busy cycle starts nothing and does not consume the window', () => {
    const limiter = WebhookGuard.createLimiter();
    assert.deepEqual(limiter.admit({ busy: true, now: 0 }), { start: false, reason: 'busy' });
    assert.equal(limiter.lastStartTs(), null);
    assert.equal(limiter.admit({ now: 1 }).start, true);
});
Enter fullscreen mode Exit fullscreen mode

The first one asserts something people rarely test: that doing nothing really does nothing. JSON.stringify(memory) === before. If a future refactor adds an innocent memory.lastTick = now to the no-op path, this test fails, and it should, because that "innocent" line is a write per tick.

Remember the April lesson: the hash-on-SKIP idea existed in April and evaporated in a rewrite. These tests are the reason the September/October guards won't evaporate the same way. A test suite is how an agent's lessons survive its developer's enthusiasm.

What the tests don't cover: real GitLab and OpenAI latency, multi-instance races on Render, or the memory mirror under failure. Those are observed in production, not proven in CI.


15. Production today

Snapshot of the public read-only endpoints of the production instance (e09ba037), captured 17 October 2026, 05:37 UTC. The counters are lifetime counters of that memory state, not a fixed time window.

Wake Gate (/analytics, wake):

Metric Value
ticks 16,314
wakes 1,114
no-ops (none) 15,200
no-op rate 93.17%
wakes: opportunity_due 1,063
wakes: sha_unavailable 23
wakes: autonomy_due 23
wakes: repo_changed 4
wakes: first_run 1
unsaved no-ops (in RAM) 245

Novelty Gate (/analytics, novelty):

Metric Value
tracked files 5
fresh evaluations 49
reuses 163,459
estimatedTokensSaved 285,419,457 (estimate, see below)

Autonomy (/autonomy): mode controlled, limit 100/day, 0 used today, admission reason_already_answered. Lifetime: 28 reconsiderations, 27 API calls, estimated $1.08. Verdicts: 14 SKIP, 13 EVOLVE, 1 no verdict. Wake source: 23 autonomy_due, 5 from the regular pipeline. Triggers: 7 trauma-pressure, 7 peer-instability, 14 with no recorded trigger (older rows and pipeline-path reconsiderations; triggers became mandatory for autonomy_due on 5 October).

Observations (/observations): 1,678,211 labelled skips. 46 resolved stability predictions: 45 correct, 1 wrong.

OpenAI cost organ (/ai-costs, read-only, whitelisted endpoints): last 7 days $0.11, last 30 days $12.55, month to date $1.39, projected month $2.54, budgets $2/day and $30/month, 30 lifetime denials daily-budget-exhausted.

Caveats that belong right next to those numbers:

  • estimatedTokensSaved is counterfactual. It's ceil(contentLength / 4) + 400 tokens per avoided evaluation, summed. It answers "how big would the prompts have been", not "how much did OpenAI not bill". Treat 285 million as an order of magnitude, not a refund. The billed reality is the table in §10: ~$1.67 per active day before July, ~$0.14 per active day in October.
  • Reuses (163,459) far exceed waking ticks (1,114). Each waking tick processes a batch of files, and the reuse counter is cumulative per tracked file. As far as I can tell it also includes the period between 18 July and 2 September, when the Novelty Gate existed without the Wake Gate and every tick ran the batch. I haven't reconstructed the exact split.
  • Most wakes are opportunity_due. 1,063 of the 1,114 wakes are scheduled reviews of frozen modules, not code changes. They don't mean 1,063 model calls: a due review re-enters the pipeline under all gates and usually ends as a deterministic SKIP. They do mean the next efficiency target is obvious: review cadence for modules that keep coming back identical.
  • "45 of 46" is a small sample. It says the stability predictions behind the skips have been right so far, not that they will be.
  • The repository is quiet now. 4 repo_changed wakes is a small sample of real change. A busier repository would wake more, and should.

16. What isn't solved (yet)

I promised a list.

  1. The internal budget runs on an estimate that was 15× too low. The cost organ detected it (estimate-too-low, ratio 15.45, 29 September). The fix, calibrating AI_COST_PER_CALL from the organ's measured realPerCall, or letting real spend gate calls, isn't in place as of this writing. Until it is, the real ceiling is "whatever $2 of one-cent estimates turns out to cost".
  2. 25 September is unexplained. 267 requests and $5.09 in the Sentinel project on a day when the merged changes were tests and docs. The organ flagged it. I don't have per-request attribution for that day, and I'm not going to invent one.
  3. The advisory text overstates. "LLM calls are now deferred" in the over-daily-budget advisory is only true if the estimate-based guard also tripped. Advisory text should describe what happened, not what we hope happened.
  4. No-op counters live in RAM between wakes. A restart loses up to "ticks since the last wake". Statistics only, but they're the statistics this article quotes.
  5. sha_unavailable wakes are deliberately wasteful. 23 so far. If GitLab had a bad hour, that would be 30 full scans in that hour. Bounded and safe, but not free.
  6. The lock is a lease in a JSON file. Fine for "two containers for a minute during deploy". Not a distributed consensus protocol, and I don't claim it is.
  7. None of this proves Sentinel "can't loop". It proves that the loop families I've seen are each closed by a tested, measured guard. The next loop will probably be a new family. The defence against that isn't a cleverer gate, it's the habit: measure the boring numbers (requests per active day, no-op rate, fresh evaluations, real $ per call) and get suspicious the moment one goes flat or spiky.

17. The developer's cheat sheet

If you're building an agent and want to skip my March–July:

  1. Gate on "has anything changed?" before you do any work. One cheap lookup (SHA, ETag, version, row count), then return. The Wake Gate pattern.
  2. Treat "unknown" as "changed", with its own reason code. sha_unavailable, dependencies-unknown. Never let a failed lookup masquerade as "same as before".
  3. Cache your SKIPs, not just your actions, keyed on everything that would change the answer (content, rules version, dependencies, rounded scores). And only cache decisions that are deterministic given those inputs.
  4. Don't cache "I didn't decide". Budget-deferred, provider-held and errored outcomes must stay re-evaluable.
  5. A no-op must do nothing. No memory write, no mirror, no log flood. Write a test that diffs state before and after.
  6. Cooldowns throttle, they don't stop. Use them as a second line behind change detection, never instead of it.
  7. One cycle at a time: in-process busy flag first, cross-process lease second. With setInterval + async, the guard must be in the callee.
  8. Webhooks poke, they don't decide. Authenticate, refuse while busy, admit at most one per window, answer 202 so the sender doesn't retry, then go through the same gate as the timer.
  9. Classify provider errors before retrying. 429 rate_limit ≠ 429 insufficient_quota. Retry the first with backoff. Stop for the day on the second. Never retry 401/400.
  10. Make "think again" cost a token, require a trigger, and refuse to answer the same question twice. Fingerprint the trigger, not just the file.
  11. Close the loop with the real bill. An estimated budget plus a read-only reconciliation against the provider's billing API, with a drift alarm.
  12. When the agent goes quiet, prove it's healthy before "fixing" it. Our most expensive four days came from making a quiet agent busy again.

18. Last sip

So, my friend, that's the story. A bot that once sent a request every six seconds, all day, about a code base that didn't change. It cost $36.59 in four days on one occasion because someone (me) thought quiet meant broken. And it now makes a few requests a day, most days zero, because it learned to ask one question before anything else:

Has anything actually happened?

If yes, it looks. If it's seen exactly this before, it says so and moves on. If it doesn't know, it looks anyway. If it wants to think again, it has to say why, and the why has to be new. If the provider says "no money", it believes it until midnight.

None of that is artificial intelligence. It's the plain, boring, deterministic stuff around the intelligence, and that's the part that pays the bill.

The meter is still there, by the way. It's just not spinning.

All OpenAI numbers come from the OpenAI Admin API (costs + completions usage, daily UTC buckets, 1 March – 17 October 2026). Commit hashes refer to the private AutoDoc-Sentinel repository. Production numbers come from the service's public read-only endpoints at 05:37 UTC on 17 October 2026. Test counts come from npm test on master 1bc30bab7be.

Real production data. GPT-4o mini.

Because if I ran this loop on GPT-6 Astra, the agent wouldn't be the only thing going in circles. My wallet would be spinning in orbit. 🛰️💸

Top comments (0)