DEV Community: build996

What's a number you published that turned out to describe your tools, not your subject?

build996 — Mon, 09 Nov 2026 02:09:48 +0000

I once published a host's "21% failure rate" after seven of a speed test's 33 nodes failed. Then the other three hosts returned the same 26/33 result: I had attributed the measuring service's failing probes to the host.

The arithmetic wasn't the problem. The explanation attached to it was.

That is the kind of mistake I'm interested in here: a number you could reproduce, but whose apparent meaning changed when you checked another target, inspected the response, or separated the measurement into phases.

Have you published or shared a number that turned out to describe your tools rather than the thing you were testing? A dashboard, internal report, code review, or benchmark result counts; it doesn't have to have reached a public article.

I'd like to hear:

What did you initially think the number meant?
What check made you change that interpretation?
Did you correct the number itself, its label, or the conclusion people drew from it?

I'm especially curious about the last part. "These probes failed" can remain a valid observation even when "this host is unreliable" isn't supported by it. Keeping the observation while withdrawing the explanation seems useful, but it can be awkward once people have already acted on the original claim.

What did your correction look like, and what would have helped you catch the attribution error earlier?

My benchmark reported 4,393,158 tokens per second

build996 — Thu, 05 Nov 2026 08:12:16 +0000

My API benchmark once reported 4,393,158 tokens per second. The useful result was finding out what that number actually measured.

The harness timed a streaming response and calculated throughput like this:

generation_time = total_request_time - time_to_first_token
tokens_per_second = output_tokens / generation_time

That calculation assumes the client observes tokens arriving over a meaningful interval after the first token. Some endpoints in the run instead held the response and delivered it in a burst. The first observed content arrived close to the end of the request. Subtracting those two timestamps left an interval close to zero.

Dividing the entire output by that tiny interval produced the spectacular number. It described the burst arriving at the client, not the model generating millions of tokens per second.

A streaming request is not proof of observable generation

Asking for streaming does not guarantee that the timing seen by the client exposes the underlying generation process. From that trace alone, I could not locate the buffering inside the serving path. I could only see that the arrivals did not support the throughput calculation I was making.

This is the distinction I needed in the results:

Post-first-content throughput: output divided by the observed interval after first content, when the stream provides a useful interval.
End-to-end throughput: output divided by the whole request duration, including waiting before content arrives.

The second number is useful for the caller, but it is a different metric. Quietly putting both in one ranking would hide the difference.

The guard I added

The benchmark now falls back to end-to-end timing when a streaming response has fewer than three data chunks or the calculated generation interval is below 0.2 seconds. It also withholds the first-token metric for responses classified as buffered. Empty output becomes an error rather than a row claiming zero tokens per second.

Here is the relevant logic, simplified from the harness:

generation_time = total_time - first_content_time
buffered = chunk_count < 3 or generation_time < 0.2

if buffered:
    generation_time = total_time

throughput = output_tokens / generation_time

Those thresholds are conservative measurement guards, not a universal detector of server buffering. A short, fast response can also produce a small interval. The harness has a separate streaming capability probe that uses chunk count; its comments explicitly warn against treating a short duration alone as evidence that streaming is fake.

The fallback gives the run a usable wall-clock measurement. It does not recover the hidden generation speed, and I should not label it as though it did.

Reproduce the failure without calling a model

For a synthetic illustration, suppose 1,000 output tokens arrive after a ten-second wait, with only 0.001 seconds between the first and last observed content:

post-first-content calculation: 1,000 / 0.001 = 1,000,000 tokens/s
end-to-end calculation:         1,000 / 10    =       100 tokens/s

These are illustrative inputs, not the raw timings behind my 4,393,158 result. They show why the denominator matters: the same output can produce radically different rates depending on the interval being measured.

Before trusting an unusually fast row, I now want the total duration, first-content timing, output token count, chunk count, and the metric actually used beside it. If the trace only exposes a final burst, the honest result is end-to-end throughput with the generation rate left unknown.

Changing my agent's default model did not change its existing sessions

build996 — Mon, 02 Nov 2026 02:13:16 +0000

A model setting on disk can be correct without describing the model an existing conversation will use.

I hit that distinction while repairing a local OpenClaw installation in September. This is a historical debugging note about that installation, initially running version 2026.4.1, rather than a guide to the current release's storage format.

The visible symptom was an empty assistant reply. One configured NVIDIA endpoint returned 410 with a message saying the model had reached its end of life on July 27. My catalogue check at the time found all eight configured models gone, and the diagnostic state showed fallbackConfigured: false.

That explained why the old model could not answer. It did not explain why changing a default was insufficient.

The default was only one copy

My setup had model settings in several places:

~/.openclaw/openclaw.json, under agents.defaults.model.primary.
Agent-specific entries in agents.list[], whose model value was a string in that version.
Persisted state under ~/.openclaw/agents/<id>/sessions/sessions.json and the agent model store, agents/models.json.

The persisted session copies overrode the settings above them in this installation. Editing only the main configuration left existing sessions using their saved values.

The useful distinction was between choosing a default and changing an already-created conversation. I had been treating those as the same operation. The application had stored enough state to make them different operations.

Search for the value you are trying to remove

Reading the new default back proves that one setting changed. It does not establish that every consumer uses it.

For this kind of failure, I would start by backing up the configuration directory, then search the relevant configuration and session stores for the old model identifier. Inspect which agent and session each match belongs to before changing anything. A historical transcript mentioning a model is different from a field selecting it.

Then compare an existing session with a newly created one and inspect the model actually selected for a request. That comparison is a diagnostic I recommend; I do not have a recorded old-versus-new session experiment from this incident to present as a result.

If the two differ, the next question becomes concrete: which saved selection does the existing session retain, and what supported operation updates it? Repeating an edit to the default cannot answer that question.

Do not turn a storage detail into a permanent recipe

During a later upgrade of the same installation to 2026.8.1, the configuration schema and session storage changed again. The old session JSON stores became migration blockers, and new SQLite stores were created after the legacy files were moved aside. That procedure lost the old chat history; the entire configuration directory had been backed up first.

That is why I would not prescribe deleting a session file as the general fix for a stale model selection. The file layout belongs to a particular version, and recovery can have a history cost.

The lasting debugging check is smaller: when a default changes but an existing conversation behaves the same way, inspect the conversation's saved selection as well as the default. A valid configuration value is evidence about that value. The dispatched request is evidence about what the agent actually used.

I asked a deleted file to speak. It lied about line 9.

build996 — Wed, 28 Oct 2026 16:21:54 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

What I Built

Deleting a file from a Git repository does not quite kill it. The last surviving version is still in the parent of the deletion commit. I wanted to see what would happen if an open-weight model tried to speak as that file — with the actual source and commit attached, so I could challenge its memory.

ghostline is a small Python CLI. Point it at a repository and a deleted path. It finds the most recent deletion commit, retrieves the file from the preceding revision, and gives numbered evidence blocks to an open-weight model. The model writes a short first-person “ghost” account and selects citations. The CLI refuses to print an answer if any cited evidence ID does not exist.

The eerie part came when the model cited real lines and still told a false story about them.

Demo

Watch the 15-second demo · Read the complete captured run

The demo uses a deliberately fictional, two-commit repository generated by demo/make_history.py. Its legacy_cache.py file contains a fallback read. The second commit deletes the file with the message “Remove legacy cache after switching to direct reads.” I ran ghostline against that history using NVIDIA's hosted Nemotron 3.5 Lightning open-weight model.

The model's voice said the assignment on line 9 “would have stored a potentially failed value.” That sounds plausible. It even cited the source block containing line 9. But the preceding block says:

try:
    value = fetch(key)
except TimeoutError:
    return cache.get(key)
cache[key] = value

On the timeout path, the function returns before the assignment. The model's interpretation was wrong. The video freezes on that moment. I left the mistake in the captured run because the distinction is the point of the tool: a valid citation is a place to check, not a guarantee that the sentence is true.

Code

build996 / ghostline

ghostline

A tiny code séance for Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine. Give it a deleted file in a Git repository. It retrieves the file's last surviving version and deletion commit, then asks an open-weight model to write a short first-person testimony. Every response must cite real evidence IDs; an invented citation makes the run fail.

This is a new project created during the October 26–31, 2026 challenge window. MIT licensed. Python 3.10+, Git, and no Python package dependencies.

Reproduce the demo

The demo history is fictional and generated locally. It is not presented as a production incident.

Watch the captured 15-second demo. The video is rendered from an actual model run saved in demo/captured-run.txt. It shows a mistake the model made about the cache write and the source lines that disprove it.

python demo/make_history.py demo-history
python ghostline.py demo-history --file legacy_cache.py --evidence-only

…

View on GitHub

The MIT-licensed repository was created during this Week 4 challenge window. It has the CLI, a script that builds the fictional Git history, the saved model output, the video renderer, and three tests. The CLI itself uses only Python's standard library and Git.

python demo/make_history.py demo-history
python ghostline.py demo-history --file legacy_cache.py --evidence-only
# Set NVIDIA_API_KEY, then:
python ghostline.py demo-history --file legacy_cache.py
python -m unittest discover -s tests -v

How I Built It

Git does the archaeology. git log --diff-filter=D finds the deletion; git show PARENT:path retrieves the last surviving file. Source lines are numbered and grouped into evidence IDs such as S1 and S2. The deletion commit message and file status become C1 and C2. There is no vector database or fabricated backstory.

Nemotron 3.5 Lightning does the part Git cannot: it turns those fragments into a compact narrative and chooses which evidence it thinks supports the account. The response must be JSON with a voice, 2–5 citations, and an uncertainty sentence. The program rejects missing IDs, duplicates, and malformed responses. It prints the exact evidence behind accepted IDs beside the voice.

I tested the code against a real two-commit Git repository built inside a temporary directory, plus an invented-citation response and a credential-redaction case. All three tests pass. The live demo made one model call; the video was rendered from that captured CLI output, rather than a hand-written mock answer.

The checks validate references, not semantic truth. The line 9 error demonstrates the gap. In particular, this is an aid to reading history, not an automated code reviewer or a source of certainty about why a maintainer deleted something.

Why Does Open Innovation Matter?

The file's “voice” is the part I want to keep replaceable. The default is Nemotron 3.5 Lightning 30B A3B, an open-weight model. The CLI accepts another model and a compatible chat-completions endpoint, so a user with suitable hardware can run inference on infrastructure they control. That matters when the source is private, and it lets people inspect or improve the model-side behavior instead of locking this experiment to one provider.

My recorded run used NVIDIA's hosted API. It was not offline or private. ghostline sends source excerpts to the configured endpoint; its basic secret redaction is best effort. For a sensitive repository, use --evidence-only or a self-hosted endpoint and review the material first.

The Spirit of It

The ghost did not know how it died. It saw a commit message and some old lines, then filled a gap with a confident, incorrect memory. Keeping the source beside the story made that mistake visible. For me, that is a useful kind of haunting: the past can speak, but you still have to read the record.

Three hosting errors that pointed me to the wrong fix

build996 — Fri, 23 Oct 2026 05:58:07 +0000

I deployed the same small PHP and MySQL application to four free hosts in September. Three failures looked familiar enough that I nearly fixed the wrong thing: an FTP login error, an FTP permission error, and a database timeout. The words in each error described what the client saw, not the layer that failed.

The useful question was not “What does this error code usually mean?” It was “Which step had already succeeded before this error appeared?”

1. `530 Login authentication failed` with the correct password

On ProFreeHost, I copied the FTP credentials from the control panel into a client. Login returned:

530 Login authentication failed

I had no reason to think the password was wrong, but that message gave me no other lead. Re-entering it did not help. Changing the connection mode did: the server rejected plaintext FTP and accepted the same username and password with explicit FTPS (FTP over TLS, starting on the usual FTP port).

The discriminating test was to hold the credentials constant and change only TLS. If the same credentials work over FTPS, resetting the password would have addressed the wrong variable. This does not mean every 530 is a TLS problem; it means 530 alone does not establish that a password is wrong.

2. `425 Operation not permitted` after login worked

On AwardSpace and FreeHostingEU, the FTPS login succeeded. I could change directories. Then every file transfer failed:

425 Unable to build data connection: Operation not permitted

“Operation not permitted” sounds like a directory permission problem. But directory changes and authentication were already working. The step that changed was opening the data connection for the upload. FTP uses a control connection for commands and a separate connection for file data; a successful login says little about the latter.

These servers required the data connection to resume the control connection's TLS session. Python's ordinary ftplib.FTP_TLS opened a fresh TLS session for data, and the server refused it. In my deploy script, a small FTP_TLS subclass passes session=self.sock.session when wrapping the data socket; transfers then proceed. The fix belongs at the data-channel TLS handshake, not in a panel checkbox for file permissions. A client that supports TLS session reuse can solve the same problem without custom Python.

I would not generalize this behavior to every FTPS server. ProFreeHost did not require this change; the AwardSpace pair did. Also, FTPS has two separate settings to check: use TLS for the control connection, and request protection for the data connection (prot_p() in Python).

3. A MySQL timeout caused by the address itself

ProFreeHost's panel supplied a MySQL hostname, sql301.ezyro.com. A local database client timed out trying to connect. That initially looked like a blocked port or a firewall rule. I checked DNS before changing either:

sql301.ezyro.com -> 192.168.0.191

That is a private IPv4 address, not a route to a database from my laptop. I checked it again on October 23 and got the same address. The hostname works for code running inside the host's network; it is not an invitation to connect from the public internet. For this account, the workable paths were PHP running on the host or phpMyAdmin in its panel, rather than a local MySQL client.

A timeout can come from many places. This one did not require guessing: the DNS answer showed that the destination was unreachable from where I was running the client.

The check I now do first

For an FTP or database failure, I write down the last step that worked:

Last successful step	Next thing to inspect
FTP credentials entered, login refused	Compare plaintext FTP with explicit FTPS using the same credentials
FTPS login and directory change worked, upload failed	Inspect the data connection, including TLS protection and session reuse
Database hostname resolves, connection times out	Inspect the resolved IP before changing passwords or firewall rules

These were observations from specific accounts, not universal meanings for 530, 425, or a timeout. The point is to change one variable at a time and follow the protocol stage that actually failed.

For the full ProFreeHost deployment record—including the original 530, the private MySQL address, and what happened after the files finally uploaded—I kept the measured application and host notes together. The three diagnostics above stand on their own; that record is for anyone who wants to inspect the larger test.

quietline: an open-weight model decides which nightly log warnings should wake me

build996 — Fri, 23 Oct 2026 04:13:35 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

I have scheduled jobs that occasionally print alarming lines and then recover. Reading the whole log the next morning is work; treating every WARN or ERROR as an alert is even more work.

quietline is a tiny Python CLI that reads only newly appended log lines and tries to wake me only for failures that still need attention. A cheap local filter finds signal words. An open-weight model reads those lines with nearby context and marks each one wake or sleep. The CLI checks that every decision refers to an actual line number, then prints the original evidence beside any alert.

The sample log has three signals: a storage API returns 429 and succeeds on retry, a site build cannot create its output directory and exits with code 1, and a cache miss is followed by HTTP 200. The model should sleep through the two recoveries and flag the failed build. On my captured run, it did exactly that. This is a synthetic test log, not a claim that quietline is already monitoring a production system.

Demo

Watch the 15-second CLI demo · See the release and recording details

The video renders the output of three actual CLI invocations against a temporary copy of the sample log. The first call used NVIDIA's hosted Nemotron 3.5 Lightning model; the next two did not call a model. Here is the first result (the model's wording varies between runs):

Read 9 new lines; 3 signal lines; 1 need attention.
WAKE line 5: job exited with code 1; output directory not created
  Evidence: 2026-10-23T01:01:05Z job=site-build ERROR output directory was not created
  Next: investigate directory permissions and retry

The second run said Quiet: no new complete lines. Model calls: 0. After I appended a routine INFO line, the third said Quiet: 1 new lines, no error signals. Model calls: 0. That is the lazy payoff I can verify: unchanged or harmless input needs neither my attention nor an inference request. I have not measured time saved over weeks of real use.

Code

build996 / quietline

A quiet daily digest for automation logs using an open-weight model

quietline

Sleep through recovered warnings; wake up for the failures that still need you.

I check automation logs after scheduled jobs. A retryable 429 and a failed build both contain alarming words, but only the failed build needs me. quietline reads new lines from one log file, sends only signal lines and nearby context to an open-weight NVIDIA Nemotron 3.5 Lightning model, and prints unresolved failures with line-numbered evidence. When the log has no new signals, it makes zero model calls.

The Python CLI is MIT-licensed and uses the standard library. The model weights are available from NVIDIA on Hugging Face; the default endpoint is NVIDIA's hosted NIM API, so this default setup sends redacted log excerpts to NVIDIA. You can point --endpoint and --model at a compatible service for an on-premises setup. Model availability and provider rate limits can change.

Try it

Python 3.10+ and an…

View on GitHub

The repository was created during this challenge window. It contains the MIT-licensed CLI, the synthetic log, a script that captures and renders the demo, and three focused tests for cursor handling, secret redaction, and rejecting invented line numbers. The CLI itself has no package dependencies.

export NVIDIA_API_KEY=...
python quietline.py demo/sample.log --state .quietline-state.json
python quietline.py demo/sample.log --state .quietline-state.json

How I Built It

The program stores a byte offset and a hash of the log prefix. It waits for a complete line; if the file is rewritten or rotated, it reads from the beginning. Only new lines matching words such as error, warning, timeout, or 429 go to the model, along with neighboring lines. That context is essential: a 429 followed by a successful retry is a different incident from a 429 followed by a failed job.

The model is NVIDIA Nemotron 3.5 Lightning 30B A3B, an open-weight model. I called it through NVIDIA NIM for this demo. The model's role is semantic triage: a regex can find an error word, but it cannot reliably decide whether later lines show recovery. Code handles the cursor, evidence, and output checks. If the API fails or the model returns invalid JSON, the cursor is not advanced, so the same lines can be retried.

I first tried an older model ID from a previous project; the endpoint returned 404. The listed Gemma 4 endpoint timed out during a probe. Nemotron 3.5 Lightning returned a useful result once I disabled its reasoning output for this short JSON task. This is a practical reason to keep the model ID and endpoint configurable rather than hiding them in the code.

Why Does Open Innovation Matter?

The project needs a model that can read the meaning of a short log sequence, not just match a token. Using open weights means the triage logic is not tied to this hosted API: someone with the hardware can serve the model themselves and keep log excerpts on their own infrastructure. My demo uses NVIDIA's hosted endpoint, so I do not claim that the demo was offline or private.

The CLI redacts common token and password patterns before sending excerpts, but that filter is best effort. I would review any sensitive logs before using a hosted endpoint. I also would not replace exit-code monitoring with this tool: the signal filter can miss unusual failures, and the model can misclassify one. quietline is a calmer digest, not a guarantee that a job is healthy.

A DEV comment streak can lose an old qualifying week

build996 — Mon, 19 Oct 2026 02:24:29 +0000

I expected a streak counter to remember which weeks had qualified. Reading Forem's Community Wellness code showed me a different design: it rebuilds the active streak from the comment and reaction rows that exist when the calculation runs.

That means an old week can stop qualifying after the activity in that week is over.

This is a source-code finding, not a claim that I have watched DEV revoke a badge. I checked the public Forem implementation on October 19. DEV's deployed version and operational settings are not exposed by those files.

The qualifying week is a live query

In Comments::CommunityWellnessQuery, comments join a set of comment IDs with surviving reaction rows. IDs with a thumbsdown or vomit reaction in the query's lookback are excluded.

Comment age is divided by 604,800 seconds and truncated to form rolling seven-day buckets. Both comments and reactions are bounded by a 231-day lookback. There is also an eligibility check for more than one comment in the seven-to-fourteen-day interval.

There is no saved record here saying that a particular historical week passed forever.

The award service walks the result again

Badges::AwardCommunityWellness skips bucket zero, requires at least two qualifying comments in a bucket, and advances only through consecutive buckets starting at one. It creates an award when the resulting length matches 1, 2, 4, 8, 16, 24, or 32 weeks.

Consider a hypothetical old bucket with exactly two qualifying comments. If one loses its only qualifying reaction, that bucket can fall below the threshold on a later calculation. The consecutive sequence can then stop before the next milestone.

That affects the calculation of future awards. The service shown here does not remove previously awarded badges when the sequence becomes shorter.

A reaction can disappear without the commenter doing anything

The profile-cleanup path in Users::DeleteActivity deletes the departing user's reactions in batches. Those include reactions they gave to other people's comments.

Following that deletion into the query gives the failure path: a comment can lose the reaction that qualified it, changing an older bucket. Multiple surviving reactions or extra qualifying comments can prevent that particular deletion from changing the result.

I did not delete a test account to reproduce this on DEV. The mechanism follows from these public source paths; its occurrence on a particular account would need separate evidence.

What I can actually verify

My profile currently shows the 1-, 2-, and 4-week Community Wellness badges. My local reconstruction has disagreed with the profile before, so I use the profile as the record of awards received.

I also cannot infer exactly which reactions the live badge job counts from a displayed reaction total. A new comment showing one reaction is not sufficient evidence that an outside reader liked it or that it will qualify.

My practical adjustment is modest: participate across the week, respond when someone adds something specific, and avoid treating exactly two comments as a guaranteed milestone. Additional useful conversations leave some room for changes in old data. They still do not make an award certain.

If you were building this feature, would you preserve each week's qualification once calculated, or deliberately let later deletions change the active streak?

tierwatch: Gemma 4 reads the free-tier pages I depend on, and has to quote every number

build996 — Tue, 13 Oct 2026 02:50:24 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

tierwatch is a small CLI that watches the free-tier and pricing pages your projects depend on and tells you when a limit changes. It reports changed facts, not changed HTML.

I keep a list of free-tier limits for a side project, and I have had to correct that list after the fact more than once. Two of the entries that went stale:

Cerebras. In February the rate-limits page showed a "Free" tier at 30 requests/min and 60K tokens/min on six models. The October version shows a "Free Trial" at 5 requests/min and 30K uncached tokens/min on two models, plus a $5 credit that needs a verified payment method.
Brave Search API. In January the page listed a "Free AI" plan at $0 for up to 2,000 queries a month. Today it lists "$5 in free credits every month" against $5 per 1,000 requests.

Neither change is hidden. Both sit on a docs page that nobody re-reads until their app starts throwing 429s.

Who would pay for it: freelancers and small agencies who leave client projects running on free or near-free plans: a contact form on a free email API, site search on a free search API, a summariser on a free inference tier. They have a dozen of these dependencies spread across clients, and nobody re-reads the pricing pages. My guess is a few dollars a month per watchlist, with alerts delivered to email or Slack. It's a guess: I haven't put it in front of a paying user yet, so there's no "first user" section in this post.

Demo

Video supplement — added October 20, 2026

Watch the video demo (MP4, about 72 seconds) · Transcript and recording details

This recording was made after the submission deadline. It renders output captured from an actual python tierwatch.py --compare --state demo/state run using unchanged October 13 code (927b7a5) and the snapshots already committed in demo/state/. It demonstrates the saved-snapshot comparison, not fresh extraction or live pricing verification. API keys were deliberately unset, so the optional model summary is unavailable; the deterministic fact diff still runs.

The original submission included textual demo output but omitted a deployed-project or video-demo link. This is a transparently dated supplement, not a claim that the video existed before the deadline. No project functionality has changed. The README distinguishes pre-deadline work from this documentation addition; acceptance of the supplement is up to the organizers.

These are two real replays. For each page, tierwatch took a snapshot from an old Wayback Machine capture, then a second snapshot from a newer capture or the live page, and diffed the two. The output below is copied from those runs. I only lined up the table columns. Every snapshot it was built from is in the repo under demo/state/.

My watchlist note for Brave said "my client's site search calls the Brave Search API on the free plan, about 1,500 queries a month":

## Brave Search API
https://brave.com/search/api/

Compared 2026-01-13 (Wayback) with 2026-10-13 (3 vs 4 verified facts;
extracted by google-ai-studio:gemma-4-31b-it).

The free plan no longer provides 2,000 free queries per month. Instead, new
users now receive $5 in monthly credits. The developer should check if these
credits cover their 1,500 monthly queries.

| fact                   | before          | now                   |
|------------------------|-----------------|-----------------------|
| answers.*              | not on the page | 2 limits              |
| free.monthly_credits   | —               | $5                    |
| free_ai.api.*          | 3 limits        | gone from the page    |
| search.qps             | —               | 50 queries per second |

For Cerebras, the note said "my side project's summariser runs on the Cerebras free tier with gpt-oss-120b and I have no card on file". Here is the summary and a selection of rows (the full table has 19 rows and is in the repo):

Compared 2026-02-12 (Wayback) with 2026-10-06 (Wayback) (49 vs 14 verified facts)

Rate limits for gpt_oss_120b have decreased to 5 requests per minute and 30K
tokens per minute, and daily/hourly limits have been removed. The new free
offering requires adding a verified payment method to receive $5 in credits.
Since you have no card on file, you should check if you can still access the model.

| free.gpt_oss_120b.rpm   | 30              | 5                  |
| free.gpt_oss_120b.tpm   | 60K             | 30K                |
| free.gpt_oss_120b.rph   | 900             | —                  |
| free.gpt_oss_120b.rpd   | 14.4K           | —                  |
| free.llama_3_3_70b.*    | 6 limits        | gone from the page |
| free.qwen_3_8_27b.*     | not on the page | 4 limits           |
| free_trial.*            | not on the page | 2 limits           |
| requires_payment_method | —               | True               |

That summary contains a mistake, and I've left it in on purpose. "Daily/hourly limits have been removed" is only half true. The request-per-hour and per-day columns went away, but tokens-per-hour and tokens-per-day are still 1M. The table is the part the code checked; the prose is the model's. More on that below.

Code

build996 / tierwatch

Notice when a free tier or pricing page you depend on changes. Gemma 4 extracts the limits, code verifies every quote, facts get diffed.

tierwatch

Tells you when a free tier or pricing page you depend on changes — in facts, not in HTML diffs.

Free-tier limits change quietly, on a docs page nobody re-reads. Cerebras's rate-limits page went from a "Free" tier at 30 requests/min to a "Free Trial" at 5 requests/min with a $5 credit that needs a verified payment method; Brave Search replaced "2,000 free queries a month" with "$5 of monthly credits". If a side project or a client's site runs on one of those plans, you usually find out when it starts returning 429s.

tierwatch keeps a snapshot of each page you list, has an open-weight model (Gemma 4) pull the limits out as structured facts, throws away every fact whose quoted evidence is not actually on the page, and diffs the facts against the previous snapshot. A second model call turns the diff into a few sentences about…

View on GitHub

It's one Python file using only the standard library, under the MIT license. Running it looks like this:

export GEMINI_API_KEY=...
python tierwatch.py                            # snapshot + diff everything in watchlist.json
python tierwatch.py --wayback 20260113061705   # replay an old capture
python tierwatch.py --compare                  # diff the two newest saved snapshots

How I Built It

The pipeline has five steps:

Fetch and flatten. The page, or a Wayback capture of it, is converted to text. Each table row stays on one line as cell | cell | cell, and <sup> footnote markers become ^1.
Extract with Gemma 4. The model returns {free_offering, requires_payment_method, facts: [{key, value, quote}]}. On later runs, the previous snapshot's keys go back into the prompt so the same fact keeps the same key.
Verify in code. A fact's quote has to appear in the page text, and its value has to appear inside the quote. For a table row, the value has to match one whole cell. Anything that fails is stored as dropped_unverified and never reported.
Diff by key. When a whole model disappears from a plan, the diff collapses its limits into one row.
Summarise the diff against the user's one-line note about what they depend on.

Step 3 started out weaker, and it let a real error through. On October's Cerebras page, the row for qwen-3.8-27b has a footnote marker right after the model name. With the HTML flattened to plain lines, Gemma read that footnote 1 as the requests-per-minute value and shifted every other column over by one. The quote check still passed, because every one of those numbers really is on the page. It just wasn't under the header Gemma assigned it to. That's when I added the table-row flattening and the whole-cell rule. Both are cheap, and that error doesn't get through any more. I don't claim column mistakes are impossible now. What the check guarantees is narrower: it can make a number disappear from the report, but a number that isn't on the page can't get into it.

The check is also strict enough to throw away correct facts. Brave prints $5 and per 1,000 requests as separate elements. Gemma quoted only the second half, the value wasn't inside the quote, and the price was dropped. For this tool that's the right way to fail. Missing a change costs you one more day before you notice. A made-up limit could push a client into a plan they don't need.

Model hosting. I didn't run inference on my own machine for this build. Every call goes to a hosted copy of the same open weights. PROVIDERS lists Gemma 4 on Google AI Studio (26B A4B and 31B), then OpenRouter's free Gemma 4 31B, then NVIDIA NIM's Gemma 4 31B. Each host gets three attempts before the next one is tried. All requests stream, because Gemma thinks for a while before it answers, and long silent requests kept getting cut by my network.

The fallback wasn't a nice-to-have. In the runs behind this post I logged these failures:

Google returned HTTP 500, HTTP 429, and streams that ended with only thinking tokens and no answer.
OpenRouter's free Gemma returned 429 "temporarily rate-limited upstream" on every attempt.
NVIDIA answered one extraction after 91 seconds, after two dropped connections.

The calls that did succeed took between 34 and 284 seconds. That's fine for a daily cron job and far too slow for a web form.

Why Does Open Innovation Matter?

This tool exists because a provider can change its terms whenever it likes. A watcher like that shouldn't depend on one provider's terms itself.

Same weights, different hosts. When Google's endpoint was returning 500s, the next host in the list ran the same model. The extraction prompt, the key naming, and the behaviour the verification step was tuned against all stayed the same. With a closed model, a fallback means a different model with different habits, and I'd have to re-tune the quote check against it.
A host retiring a model doesn't retire the weights. While setting this up, NVIDIA's google/gemma-3-27b-it returned 410 Gone with an end-of-life date of 2026-05-12. That's one host dropping one model. The weights are still published, and I could serve them myself or pick another host. When a closed model ID is retired, the model is simply gone.
The economics. The work is one extraction per page per day, and nobody waits on it. Right now it costs nothing on free hosted Gemma. If those free tiers disappear (which is exactly what this tool watches for), the same weights can run on a small rented GPU or a cheap batch endpoint. Either way the price stays under my control.
Fine-tuning is possible later. Pricing pages follow a handful of layouts. With open weights, I could fine-tune a small Gemma on verified extractions from this tool's own state/ folder. A closed API wouldn't let me do that.

Where the closed models I've used would still beat this: speed, and fewer empty or failed responses. For a cron job that runs once a day, I'll take the open option.

Prize Categories

Best Use of Gemma. Gemma 4 (26B A4B and 31B) does every extraction and summary, served by Google AI Studio with OpenRouter and NVIDIA NIM hosting the same weights as fallbacks.

openclaw doctor --fix reported the problems and fixed none of them

build996 — Mon, 12 Oct 2026 02:19:49 +0000

I run OpenClaw as a systemd user service inside WSL, mostly as a local playground for agent setups. It had been sitting on 2026.4.1 for a while, and I upgraded it to 2026.8.1 in one jump. That is four months and four major versions.

The install finished. The gateway did not start. It exited with code 78 (EX_CONFIG), and the log told me to run openclaw doctor --fix.

I ran it. It printed a list of problems. I started the gateway again. Exit 78.

This post is the order of steps that actually got it booting, and the one habit I would take into any upgrade like this: treat "run doctor --fix" as a suggestion, not a fix.

What doctor did and did not do

doctor was useful for finding things. But on this jump the problems it reported were still there after --fix, and two of its messages were actively misleading:

It warned that agents.list had to become agents.entries. It did not rewrite it. The new shape is an object keyed by agent id instead of an array of objects with an id field, so this is a two-minute edit, but you have to do it yourself.
It got stuck in a loop on a legacy session store:

deferred legacy-main session migration; run openclaw doctor --fix

Running doctor --fix again produced the same line. And again. The message points at the command you just ran.

The real blocker was in journalctl

The gateway's own output only gave me the exit code and the doctor hint. The service log had the actual reasons:

journalctl --user -u openclaw-gateway

That is where the actual blockers showed up. Once I stopped re-running the hint and started reading the journal, the rest was mechanical.

The order that worked

Node first. 2026.8.1 requires Node >=22.22.3. The box had 22.22.1, two patch versions short, and npm refused at preinstall. I installed Node 24 with nvm and made it the default.
Let the install scripts run. npm skipped the package's postinstall by default, so the bundled plugins never landed. Reinstalling with --allow-scripts for openclaw and its native dependencies fixed that.
openclaw gateway install again. The systemd unit still pointed at the old Node binary path, so the service was launching the new code with the old runtime.
Delete the config keys the new schema rejects, one boot at a time. This was the slow part. Each failed boot named some of the rejected keys, not all of them. For me the full set was meta.lastTouchedAt, commands.ownerDisplay, gateway.tailscale.resetOnExit, gateway.nodes.denyCommands and plugins.installs. I found them over several boots.
agents.list → agents.entries, by hand, as above.
Add agents.ownership: "explicit". It is now required when more than one agent is defined.
Move the legacy stores aside. agents/*/sessions/sessions.json and exec-approvals.json were what kept the migration loop alive. Renaming them let the new SQLite stores be created. The cost: old chat history is gone, which is why I tarred the whole ~/.openclaw directory before touching anything.
Disable third-party plugins written against the old SDK. One chat-integration plugin could no longer be loaded from its TypeScript source and blocked boot on its own.

After step 8 it came up. The new UI is a full redesign, with sessions as a first-class pane, which was the thing I wanted from the upgrade in the first place. But nothing about getting there was automatic.

One more trap while iterating

Steps 4 to 8 mean restarting the service a lot. OpenClaw has a crash-loop breaker: three unclean boots within five minutes, and it suppresses provider auto-start on the next boot. At that point the gateway is up, but agent turns fail for a reason that has nothing to do with whatever you just fixed. It looks like your fix broke something else.

What worked: stop the service, wait about 45 seconds, start it once, and leave it alone while you read the journal.

What I would do next time

Back up the whole config directory before upgrading, not just the config file. Some of the fixes are "rename this store aside", and that is only safe with a copy.
Expect schema rejections to arrive in batches. Plan for several boots, not one.
When an error message tells you to run a repair command, run it once. If the message comes back unchanged, the repair command is not going to help, and the answer is in the service log.

If you have upgraded a self-hosted agent stack across several majors at once: did its migration tool do the work, or did you end up editing config by hand too?

Groq returned the same 403 for my real key, a fake key, and no key

build996 — Thu, 08 Oct 2026 02:22:51 +0000

I was running a small Node benchmark script against a few hosted LLM APIs from a Windows machine. One provider failed with a bare fetch failed, which at least looked like a network problem. Groq did something more convincing. It answered, with a status code and a JSON body:

HTTP 403
{"error":{"message":"Forbidden"}}

A 403 from an API that has your key reads like "your key is not allowed to do this". Revoked, suspended, wrong project, account flagged. What stopped me from treating it that way was one boring test.

Three requests, one answer

If the 403 is about the key, then changing the key should change the answer. So I sent the same request three ways:

with my real key
with a key I made up (gsk_ plus random characters)
with no Authorization header at all

All three came back identical: 403, {"error":{"message":"Forbidden"}}.

That settles more than it looks like. A server that has actually looked at your credentials cannot give the same verdict to a valid key, a fabricated one and a missing one. Whatever said "Forbidden" made its decision before anyone checked the key. The key was never the problem, so rotating it, regenerating it or emailing support about it would all have been wasted.

What a real auth failure looks like

For contrast, here is the same endpoint today, from a network path that works, with a fake key and then with no key at all:

curl -s -w "\n%{http_code}\n" https://api.groq.com/openai/v1/models \
  -H "Authorization: Bearer gsk_fake123"

curl -s -w "\n%{http_code}\n" https://api.groq.com/openai/v1/models

Both return:

{"error":{"message":"Invalid API Key","type":"invalid_request_error","code":"invalid_api_key"}}
401

Two differences from the 403, and either one is enough to tell them apart:

The status. A missing or wrong key gets 401. A 403 for a request that carried no credentials at all doesn't mean "this key lacks permission", because there was no key to judge.
The body shape. The real auth error has type and code fields, the same shape as the API's other errors. The 403 body had only message. When an error doesn't look like the rest of the API's errors, it probably didn't come from the API's application code.

I can't reproduce the 403 any more because the network path it came from is gone. Where exactly it was produced, I can't say for certain: something between my machine and the key check, whether a network hop or an edge policy that looks at where a request comes from. What I can say is that it was upstream of authentication, and the three-key test is what proved it.

Why curl worked and Node didn't

The second half of this was a red herring of my own making. The machine had a proxy configured through HTTPS_PROXY, and curl through that proxy reached everything fine. So I assumed my script was taking the same path. It wasn't.

Node's built-in fetch (undici) does not read HTTP_PROXY / HTTPS_PROXY by default. The script inherited the variables and ignored them, so its requests went out directly, which is where the 403 and the fetch failed both came from. One provider answering normally over that same direct path made it look provider-specific instead of path-specific, which sent me further in the wrong direction.

On Node 24 there's an opt-in for this:

NODE_USE_ENV_PROXY=1 HTTPS_PROXY=http://127.0.0.1:PORT node script.mjs

With that set, the same script got normal answers from every provider, including Groq with the same key that had been "forbidden" before.

The checklist I use now

When an API rejects a key that I believe is good:

Send the request with no Authorization header. If you get the same status and body as with your key, the key isn't being evaluated. Stop debugging the key.
Compare the error body's shape with the API's documented errors. Missing fields usually means a different layer wrote it.
Make sure the tool that fails takes the same path as the tool that works. curl honouring HTTPS_PROXY says nothing about Node's fetch, Python's httpx with trust_env=False, or anything else with its own idea of proxies.

Step 1 takes ten seconds, and it's the one I skipped. I'm curious whether other people have a "this error isn't from where it claims to be" test they reach for first, because the general problem (an error message that blames the wrong layer) seems to come up everywhere, not just with API keys.

How do you check that an agent actually did what it says it did?

build996 — Mon, 05 Oct 2026 02:30:32 +0000

A while ago I gave a model a boring job inside an agent harness: sort eight files into subfolders by type. It reported success in 20 seconds. Not one file had moved.

Since then I don't grade an agent on its final message. I look at whatever the task was supposed to change, like the folder, the table or the API record, and the agent's summary is the last thing I read, if I read it at all.

That works when the task has an obvious end state. It gets harder fast:

Open-ended tasks. "Clean up this module" has no single state to diff against. What counts as done?
Side effects you didn't expect. Checking that the file moved doesn't tell you whether something else got deleted on the way.
Cost. Writing a state check for every task can take longer than doing the task yourself, which defeats the point of handing it off.

So I'm curious what other people actually do, day to day, not in theory:

Do you run the tests and trust green, read the diff, ask the agent for evidence (command output, file paths), or spot-check a sample?
Has an agent ever told you it finished something it hadn't? How did you find out, and how long did it take?
If you check by state, who writes the check: you, before the task, or the agent, after?

The third one is the one I haven't settled. A check the agent writes after doing the work tends to share its blind spots. A check I write beforehand is more independent, but it costs me time on every single task, and I'm not sure where the line is.

WordPress rewrote the & inside my JavaScript regex, and only the live page broke

build996 — Thu, 01 Oct 2026 02:05:35 +0000

I shipped a small tool page about GitHub projects as a WordPress post, with the JavaScript embedded in a Custom HTML () block. Opened from disk, it worked. On the live site, every number on the page came out like this:

180,491   ->  1undefinedundefined,491
v0.33.3   ->  vundefined.undefinedundefined.undefined

Some digits survived and some turned into the string undefined. That pattern is specific enough to trace back to one character.

The helper that broke

The page had the usual HTML-escape helper:

function esc(s) {
  return String(s).replace(/[&<>"']/g, c => MAP[c]);
}

On the live page, the source of that line read:

return String(s).replace(/[&#038;<>"']/g, c => MAP[c]);

WordPress's wptexturize filter ran over the post content, found a lone &, and turned it into the numeric entity &. The Custom HTML block didn't protect the script, and being inside a <script> tag didn't either.

Inside a regex character class, & isn't one character. It's six, so the class now also matched #, 0, 3, 8 and ;. Every 0, 3 and 8 in my numbers got matched, looked up in MAP, found nothing, and was replaced with undefined. That's why 180,491 kept its 1, 4, 9 and 1 and lost the 8 and the 0.

What it didn't touch

The same script had nine && operators, and all nine came through unchanged. It's specifically a single, free-standing & that gets converted. That's also why the rest of the page logic ran fine and the only symptom was in the output of one helper, which made it look like a data problem rather than a broken script.

The fix

Don't write a bare & in JavaScript that WordPress will filter. In a regex or a string literal, the Unicode escape means the same thing to the JS engine and gives the filter nothing to rewrite:

return String(s).replace(/[\u0026<>"']/g, c => MAP[c]);

The other route is to keep the script out of post content entirely, in a separate .js file, since the content filters only run on the post body.

Why I didn't catch it before publishing

The bug can't exist locally. The file I tested from file:// was exactly what I wrote; the & only appears after WordPress renders the post. My local check was honest and also testing a different file from the one visitors get.

The check I run now after publishing anything with inline JS is to fetch the live HTML and grep the script region for &. It takes a few seconds, and it's the only test that looks at what the browser actually receives.

Has anyone found a WordPress content filter that touches something other than & inside embedded scripts? I'd like to know what else belongs in that grep.

DEV Community: build996

What's a number you published that turned out to describe your tools, not your subject?

My benchmark reported 4,393,158 tokens per second

A streaming request is not proof of observable generation

The guard I added

Reproduce the failure without calling a model

Changing my agent's default model did not change its existing sessions

The default was only one copy

Search for the value you are trying to remove

Do not turn a storage detail into a permanent recipe

I asked a deleted file to speak. It lied about line 9.

What I Built

Demo

Code

build996 / ghostline

ghostline

Reproduce the demo

How I Built It

Why Does Open Innovation Matter?

The Spirit of It

Three hosting errors that pointed me to the wrong fix

1. 530 Login authentication failed with the correct password

2. 425 Operation not permitted after login worked

3. A MySQL timeout caused by the address itself

The check I now do first

quietline: an open-weight model decides which nightly log warnings should wake me

What I Built

Demo

Code

build996 / quietline

A quiet daily digest for automation logs using an open-weight model

quietline

Try it

How I Built It

Why Does Open Innovation Matter?

A DEV comment streak can lose an old qualifying week

The qualifying week is a live query

The award service walks the result again

A reaction can disappear without the commenter doing anything

What I can actually verify

tierwatch: Gemma 4 reads the free-tier pages I depend on, and has to quote every number

What I Built

Demo

Video supplement — added October 20, 2026

Code

build996 / tierwatch

Notice when a free tier or pricing page you depend on changes. Gemma 4 extracts the limits, code verifies every quote, facts get diffed.

tierwatch

How I Built It

Why Does Open Innovation Matter?

Prize Categories

openclaw doctor --fix reported the problems and fixed none of them

What doctor did and did not do

The real blocker was in journalctl

The order that worked

One more trap while iterating

What I would do next time

Groq returned the same 403 for my real key, a fake key, and no key

Three requests, one answer

What a real auth failure looks like

Why curl worked and Node didn't

The checklist I use now

How do you check that an agent actually did what it says it did?

WordPress rewrote the & inside my JavaScript regex, and only the live page broke

The helper that broke

What it didn't touch

The fix

Why I didn't catch it before publishing

1. `530 Login authentication failed` with the correct password

2. `425 Operation not permitted` after login worked