DEV Community: Talha Anwar

I stopped fixing my prompt by hand. Open models fixed it while I napped

Talha Anwar — Sat, 24 Oct 2026 15:18:00 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

Fixing a prompt by hand is a loop I hate. I rewrite it, test it by hand, redeploy, and find that I fixed two cases and broke a third. Then I do it again.

So this week I wrote the laziest prompt I could for a bank's message router, and I never edited it again. Open-weight models rewrote it in two optimizer iterations. I checked every result on messages the models never saw.

	Result
Correct, parseable answers, Gemma 3 4B	37% → 60% on 308 held-out messages
Same prompt on Gemma 3 27B	69% → 74%, without ever tuning for 27B
Prompt text I edited by hand	none
Models in the loop	all open-weight: Gemma 3, Gemma 4, gpt-oss
Cost of the whole experiment	$0.38

What I Built

PromptSloth is one Python file that runs a prompt-improvement loop on AcruxCore, the open-source LLM-ops platform I work on. The script talks to AcruxCore only through its MCP server. MCP (Model Context Protocol) is an open JSON-RPC protocol that lets a client call a server's tools.

The router's job is intent routing. A customer writes "I still have not received my new card", and the router must answer card_arrival, not card_delivery_estimate. The data is Banking77 from PolyAI: 13,083 real banking messages with 77 intents, under CC-BY-4.0.

The lazy prompt, v1, is two sentences: the list of 77 labels, then "Reply with the intent label only, nothing else."

The router never changes

AcruxCore stores each prompt as numbered versions, and an alias is a name that points at one version. My router asks for an alias, never a version number. Moving an alias changes the router's prompt without a code change or a redeploy. The aliases here are lazy (v1), fixed (v2) and fixed2 (v3).

Every model is open-weight

Role	Model
Router	Gemma 3 4B. Gemma 3 27B runs the finished prompts as a check.
Optimizer, which writes the rewrites	gpt-oss-120b
Judge, which scores them	Gemma 4 31B

Demo

The recording shows three real commands. report prints the second iteration's scores. Then eval runs v1 and v3 on the same 24 held-out messages. With v1, most replies are wrapped in braces, and only 2 of 24 are both right and parseable. With v3, 15 of 24 are right, and every reply is a bare label.

In the second iteration, two of the three rewrites scored below v2 on the judge's training-set score:

fixed (v2, baseline)  57.1
candidate-A           64.3  beats baseline: True   <- promoted as v3
candidate-B           50.6  beats baseline: False
candidate-C           55.8  beats baseline: False

AcruxCore did not promote B or C, so the router never used them.

What each iteration fixed

Iteration 1 fixed the format

With the lazy v1, the 4B model often wrapped its answer, as in {card_arrival}. On 20% of all messages the label inside the braces was right, but a strict parser rejects the whole reply. The first rewrite told the model to "respond with only the exact label (no punctuation, no surrounding text)". The extra characters disappeared. The share of wrong labels stayed at about 44%, so iteration 1 did not improve the routing.

Iteration 2 fixed the routing

For the second iteration I wrote a new rubric, the paragraph of instructions that a dataset gives the optimizer and the judge. It asked for a one-line definition of every intent, saying how the intent differs from its nearest neighbour. The winning rewrite added 77 lines like these:

card_arrival = ordered card has not arrived yet.
card_delivery_estimate = user asks how long delivery will take before the card arrives.
activate_my_card = user wants to enable a newly received card.

The optimizer wrote those definitions from the 154 training examples. On held-out messages, wrong labels fell from 45% to 40%.

Held-out, 308 messages	Gemma 3 4B	Gemma 3 27B
v1 (`lazy`): labels only	37%	69%
v2 (`fixed`): after iteration 1	55%	73%
v3 (`fixed2`): after iteration 2	60%	74%

A reply counts as correct when it is the right label and nothing else. In every cell, the two runs are at most 3.2 points apart.

Is the gain real?

A paired test on every message

I compared each pair of prompts message by message, with an exact McNemar test. That test looks only at the messages that one prompt got right and the other got wrong.

4B, v1 → v3: v3 got 88 messages right that v1 got wrong, and 16 wrong that v1 got right (p < 0.0001).
4B, v2 → v3, routing only: v3 won 36 messages and lost 17 (p = 0.013). The definitions helped after the format was already fixed.
27B, v1 → v3: v3 won 29 messages and lost 13 (p = 0.02). Both iterations scored rewrites only on the 4B model, so the 27B gain comes from a prompt that was never tuned for 27B.

The judge agreed with the test it never saw

The judge scored v1, v2 and v3 at 36, 57 and 64 on the training messages. Exact match on the held-out messages gave 37%, 55% and 60%. The judge never saw a held-out message, and the two measurements still rank the three versions the same way.

The optimizer never sees a test message either. The 308 held-out messages come from Banking77's test split, and the 154 training messages come from its train split.

How I Built It

Only MCP, no SDK

promptsloth.py has six subcommands: setup, dataset, optimize, report, promote and eval. Every subcommand sends JSON-RPC to POST /api/v1/mcp through mcp.py, a 40-line client built on requests. The MCP guide shows the same server connected to Claude.

Four tool calls per iteration

create_prompt and commit_prompt_version store the lazy v1.
create_dataset and add_dataset_example load 154 training messages, two per intent, with the rubric.
optimize_prompt starts an iteration, and get_optimize_report returns the scores. The optimizer drafts three rewrites, and the judge scores each rewrite and the current version on the training messages.
promote_optimize_candidate commits the winner as a new version and moves an alias to it. AcruxCore refuses a rewrite that did not beat the version it was compared with.

The optimizer guide covers the same loop from the dashboard, and the evaluation page lists what else it scores.

Scoring without a judge

The eval step calls run_prompt once per held-out message at temperature 0 and compares the reply with the label in Python:

text, err = mcp.tool("run_prompt", {
    "prompt": "promptsloth-banking77", "alias": "fixed2",
    "model": "ps-gemma-3-4b-it", "variables": {"message": row["text"]},
    "temperature": 0, "max_tokens": 40,
})

sample.py draws both sets from Banking77 with seed 42: two messages per intent from the train split and four per intent from the test split.

The one step outside MCP

ps-gemma-3-4b-it is the name I gave Gemma 3 4B in AcruxCore's gateway, the component that routes every model call to a provider and records its cost. The MCP server has no tool to register a model, so I made one REST call per model.

What went wrong

I first included a third size, Gemma 3 12B. One upstream provider returned empty replies for it, and one run needed 1,504 retries. Its scores changed with the provider that answered, so I removed 12B from the results. The script now retries an empty reply and counts each retry.
The lazy prompt's wrapped replies hid what each iteration changed. The script now reports two scores: routing only, with the wrapping removed, and routing plus a clean reply. The routing-only score showed that iteration 1 fixed the format and iteration 2 fixed the routing.

The Lazy Payoff

My own writing was a prompt of two sentences and two rubric paragraphs. The optimizer drafted six rewrites, the judge scored 1,232 answers (4 versions × 154 messages × 2 iterations), and AcruxCore refused the two rewrites that scored below v2.

The gateway's usage report (get_gateway_usage) puts the whole day at $0.38 for 11,660 model calls. That total covers both iterations, every held-out run, the demo and the wasted 12B retries. Each v3 request is longer than a v1 request, about 1,380 tokens against 575. So the 4B router costs $0.046 per 1,000 messages with v3, against $0.018 with v1.

Why Does Open Innovation Matter?

The router can leave the cloud

Gemma 3 4B is small enough to run on a laptop. I ran it through OpenRouter for this measurement, not on my own machine. The accuracy is still the accuracy of the same weights you can download and host.

I could inspect every layer

When the 12B scores jumped between runs, the traces showed empty replies from one provider. When iteration 1's gain looked large, the raw replies showed braces, not better routing. A closed optimizer that returns only a score would have shown neither.

Anyone can repeat it

Banking77 is public, and the sampling uses a fixed seed. AcruxCore is Apache-2.0 and self-hostable, with its source on GitHub, and every model is open-weight.

Limits

With v3, the 4B model (60%) still scores below the 27B model with the lazy v1 (69%).
Banking77 is public, so the models may have seen it during training.
Each cell has two runs. Even at temperature 0, two runs differed by up to 3.2 points.
I wrote both rubrics. The first one named a few example confusions, such as card arrival against card activation.
I tested one task. A task with fewer and more distinct labels may gain less from definitions.

Code

talhaanwarch / promptsloth

A lazy prompt fixes itself on open-weight models (Gemma 3, gpt-oss) over AcruxCore's MCP server. Banking77, held-out exact-match test.

PromptSloth 🦥

A lazy prompt that fixes itself on open-weight models, driven entirely over AcruxCore's MCP server.

The router reads a bank customer's message and returns one of the 77 Banking77 intents. Version 1 of its prompt is the label list and one instruction. Two optimizer iterations rewrote it, and an open judge decided each promotion:

Role	Model (all open-weight)
Router	Gemma 3 4B (Gemma 3 27B runs the finished prompts as a check)
Optimizer	gpt-oss-120b
Judge	Gemma 4 31B

Held-out results, 308 test-split messages the optimizer never saw, mean of two runs. A reply counts when it is exactly the right label:

Prompt	Gemma 3 4B	Gemma 3 27B
v1 (`lazy`): labels only	37%	69%
v2 (`fixed`): after iteration 1	55%	73%
v3 (`fixed2`): after iteration 2	60%	74%

Iteration 1 fixed the reply format (the 4B model wrapped about 35% of…

View on GitHub

The repo, talhaanwarch/promptsloth (Apache-2.0), holds every script and every result file behind the charts. To run it, get an AcruxCore API key with the editor role and run export ACRUXCORE_API_KEY=.... Then run python3 sample.py and the subcommands in the order the script's docstring lists.

When you fix a prompt today, how do you check that the fix did not break other cases? I tested on a held-out split, and I would like to hear what works on your team. 🦥

Prize Categories

Best Use of Gemma. Gemma 3 4B and 27B route the messages, and Gemma 4 31B is the judge whose scores decided each promotion. OpenRouter served all three Gemma models.

AI assistance: I built PromptSloth with an AI coding assistant, and this post was drafted with an AI assistant from the project's own result files. I reviewed it, and every number comes from the repo.

Got impacted by railway breach. Now need to change all keys and the worst part is still not able to deploy. Who else here got impacted.

Talha Anwar — Thu, 22 Oct 2026 07:37:08 +0000

A receipt reader on open vision models: 4 models, 50 receipts, 25x price gap

Talha Anwar — Tue, 13 Oct 2026 19:07:46 +0000

This is a submission for the Hacktoberfest Challenge, Week 2: Side Hustle.

What I Built

Freelancers keep a pile of receipt photos and type each one into a spreadsheet: company, date, total. The receipts are private financial records, and many people would rather not upload them to a closed service.

The receipt sorter takes receipt photos and returns a CSV with the company, date and total of each. The side hustle idea is a small paid service that does this for customers.

One prompt holds the instructions and the photo, and a vision model reads it. I tested four models on 50 labelled receipts, which means each receipt has a known correct company, date and total. Gemma 4 31B, Qwen3-VL 30B and Qwen3-VL 8B have open weights. GPT-5.4 mini is closed, and I used it as the reference to compare against. I work on AcruxCore, the open-source LLM-ops platform (GitHub) that ran these tests. It stores prompts, sends model calls through one gateway, runs evaluations and records traces.

Demo

Drop photos in, press Extract, and download the CSV. Every row links to a trace, the record AcruxCore keeps of one model call with its tokens, cost and time, so you can inspect a wrong total.

Code

github.com/talhaanwarch/receipt-sorter (Apache-2.0) has the code and a one-command setup.

LLMops platform used to compare models and prompt is AcruxCore, an opensource platform.

The step-by-step build is in a separate tutorial: Build a receipt sorter with an open vision model and AcruxCore.

How I Built It

Register the models. Four vision models on OpenRouter are registered in the AcruxCore gateway, the service that sends each call to a model provider, with their prices: Gemma 4 31B, Qwen3-VL 30B, Qwen3-VL 8B, and GPT-5.4 mini as the closed reference.
Write one prompt with an image. The prompt receipt-extract has a system message with field rules and a user message with an image part, {{ receipt_url }}. Each edit is a new version. Version 1 has short instructions, and version 2 adds field rules, such as the day coming first in the date. An alias is a movable name that points at one version, and the alias production points at the version the app uses.
Call it through the gateway. The Streamlit app renders the prompt for the alias and sends it to the model you choose. The same code reads a public link or an uploaded photo sent as a data: URL.
Build a dataset. The 50 receipts are a sample of the public SROIE set, each with its labelled company, date and total. The dataset stores links to the images, not the images.
Run an experiment. An experiment crosses prompt versions with models. I ran two versions on four models, three times. I scored exact matches from the raw outputs. An LLM judge, a second model that reads the receipt, the answer and the correct values, also gave each answer a score from 0 to 100.

Every call in these steps wrote a trace like this one, so the cost and seconds in the table below come from the spans, not from a stopwatch.

Promote the winner. Moving the production alias from version 1 to version 2 changed the app's behaviour with no code change.

What the measurement shows

Each row is 50 receipts, three runs of the same prompt (version 2). "All three right" means company, date and total were all correct. Cost is the extraction call only.

Model	Open weights	All three right (of 50)	Median seconds	$ per 1,000 receipts
Gemma 4 31B	yes	45–46	5.0	0.05
Qwen3-VL 30B	yes	39–44	3.1	0.21
Qwen3-VL 8B	yes	41–42	2.2	0.34
GPT-5.4 mini	no	45	1.8	1.25

Gemma 4 31B and GPT-5.4 mini cannot be told apart on accuracy in this sample. Gemma costs 25 times less per receipt: $0.05 against $1.25 per 1,000 receipts. GPT-5.4 mini is faster, at 1.8 seconds against 5.0.

The Qwen models are faster than Gemma, but they got fewer receipts fully right. Qwen3-VL 8B costs more per 1,000 receipts than the 30B because its registered price per token is higher: $0.25 in and $0.75 out per million tokens, against $0.15 and $0.60.
The order of the models changes between prompt version 1 and version 2, so I do not name an accuracy winner. The table supports a smaller claim: an open model matched the closed one on this sample.

Why Does Open Innovation Matter?

Cost. At 40 receipts a month, Gemma 4 31B costs $0.002 and GPT-5.4 mini costs $0.05. Both amounts are small, but the closed model costs 25 times more, with accuracy this sample cannot tell apart, and that difference grows with every customer.

Control of financial records. Open weights can run where the data stays. I ran these models through OpenRouter, not on a laptop, so this article does not show a local run. The measurement shows that models you could host yourself read these receipts as well as the closed one. You need that evidence before you move them in-house.

What the judge cost

The judge is the expensive part of an image evaluation. One run covers eight combinations of prompt version and model, with 50 receipts each, so 400 extractions and 400 judge calls. The four extraction models cost $0.18 in total, and the judge cost $2.33.

The judge was billed about 37,000–41,000 input tokens per call. The extraction models were billed 340–1,500 input tokens. The most likely cause is that the judge also receives the receipt image, but I did not test a different judge to confirm it. Check the token count of one judge call before you start a full run.

Limits

SROIE is public, so a model may have seen it in training. A private set of receipts could give different exact-match counts.
50 receipts is small. The ranges show how much three repeat runs differ. They do not show how much a different set of 50 receipts would change the result.
The receipts are Malaysian English. Other languages and layouts are untested.
Five receipts are missed by all four models. Two print two valid totals, one has a cut-off year, one has a low-resolution year, and one shows two company names. I kept the published labels. On the other 45 receipts, Gemma missed none, GPT-5.4 mini missed one, Qwen3-VL 8B missed four and Qwen3-VL 30B missed seven.
Qwen3-VL 8B hit provider rate limits in one run. I ran it again and report only runs without provider errors.

Prize Categories

Best Use of Gemma. Gemma 4 31B, Google's open-weight model, reads the receipts through a provider (OpenRouter). In the measurement, its exact-match range overlapped the closed reference model's, and it cost 25 times less..

The prompt, gateway, dataset, evaluation and tracing layers run on AcruxCore, which is open source under Apache-2.0.

Build a receipt sorter with an open vision model and AcruxCore

Talha Anwar — Tue, 13 Oct 2026 19:07:07 +0000

You will build a receipt sorter: drop receipt photos into a web page and get a CSV with the company, date and total of each one. Then you will use AcruxCore (an LLMops platform) to choose the model with evidence instead of a guess. You finish with a versioned image prompt, a 50-receipt test set, a model comparison you can repeat, and a trace for every receipt the app reads.

Every AcruxCore step shows the dashboard and the Python SDK side by side. The full code is in github.com/talhaanwarch/receipt-sorter. The measured result of the comparison is in the challenge article; this page teaches the build.

You need: Python 3.11+, an OpenRouter API key with a few dollars of credit (the whole tutorial costs under $3, most of it the judge), and an AcruxCore account. The screenshots use a local server at http://localhost:5173; the steps are the same on a hosted account.

1. Set up the project

git clone https://github.com/talhaanwarch/receipt-sorter && cd receipt-sorter
python -m venv .venv && source .venv/bin/activate
pip install acruxcore streamlit python-dotenv pandas pyarrow pillow
cp .env.example .env

In the dashboard, open Account & keys, create an API key, and put it in .env as ACRUXCORE_API_KEY. Set ACRUXCORE_BASE_URL to your API (http://localhost:3001/api/v1 locally) and OPENROUTER_API_KEY to your OpenRouter key.

2. Connect OpenRouter and register the models

The AcruxCore gateway serves named models. A request says model="rs-gemma-4-31b", and the gateway resolves that name to an upstream model and a stored key. So the app code never holds a provider key, and switching a model is a name change.

Dashboard. Open Credentials, add an OpenAI-compatible credential with base URL https://openrouter.ai/api/v1 and your OpenRouter key. Then open Models and press New model once per model, choosing that credential and entering the price per million tokens.

Python. [scripts/setup_acruxcore.py](https://github.com/talhaanwarch/receipt-sorter/blob/main/scripts/setup_acruxcore.py) registers the same five models (four readers and the judge, which step 7 explains):

MODELS = {
    "rs-gemma-4-31b":  ("google/gemma-4-31b-it", 0.09, 0.34),
    "rs-qwen3-vl-30b": ("qwen/qwen3-vl-30b-a3b-instruct", 0.15, 0.60),
    "rs-qwen3-vl-8b":  ("qwen/qwen3-vl-8b-instruct", 0.25, 0.75),
    "rs-gpt-5.4-mini": ("openai/gpt-5.4-mini", 0.75, 4.50),
    "gpt-4o-mini":     ("openai/gpt-4o-mini", 0.15, 0.60),  # the judge
}

Prices matter later: AcruxCore multiplies them by the token counts of each call, which is where the dollar figures in traces come from. All reader models are open-weight except rs-gpt-5.4-mini, which is the closed reference. Gemini models are left out because the gateway cannot pass them an image link.

3. Get 50 test receipts

tools/build_sample.py downloads the public SROIE test split, picks 50 receipts with a fixed seed, shrinks them to 1,600 pixels on the long side, and writes data/labels.json with the correct company, date and total of each. The images are already committed to the repo, so you can skip the script. SROIE is licensed CC BY 4.0, and the NOTICE file carries the credit.

The images live on raw.githubusercontent.com, so each one has a public link. That is deliberate: a dataset example's input is capped at 8,192 bytes, which a photo does not fit in. The example stores the link, and the model fetches the image.

4. Write the prompt with an image

A prompt in AcruxCore is a list of messages. A user message can hold several parts, including an image part whose URL is a variable.

Dashboard. Open Prompts, create receipt-extract, and in the editor add a system message with the rules, then a user message with a text part ("Read this receipt.") and an Image link part set to {{ receipt_url }}. Press Commit new version.

Python.

USER_PARTS = [
    {"type": "text", "text": "Read this receipt."},
    {"type": "image_url", "image_url": {"url": "{{ receipt_url }}"}},
]
prompt = await hub.prompts.create("receipt-extract")
v1 = await hub.prompts.commit_version(prompt.id, messages(1))
await hub.prompts.promote_alias(prompt.id, "production", v1.version_number)

Two details. The image part is allowed only in a user message. And promote_alias creates the alias production: a movable name that points at one version. The app asks for production, never for a version number, so you can change what runs without touching the app.

5. Call it through the gateway

Rendering fills the variable and returns messages ready to send. The gateway call then runs them on a registered model and records a trace.

rendered = await hub.prompts.render("receipt-extract", "production", {"receipt_url": receipt_url})
result = await hub.gateway.chat(
    "rs-gemma-4-31b", rendered.messages, temperature=0, max_tokens=500,
    prompt_version_id=rendered.version_id, variables=rendered.variables,
    trace={"name": "extract-receipt-fields", "tags": ["receipt-sorter", "production"]},
)
fields = parse_fields(result.content)   # {"company": ..., "date": ..., "total": ...}

receipt_url is either a public link or an uploaded photo turned into a data:image/jpeg;base64,... value by to_data_url. Both go through the same call. Passing prompt_version_id links the trace to the exact prompt version that produced it, and temperature=0 keeps repeat runs comparable.

Models sometimes wrap the JSON in a code fence, so parse_fields finds the first {...} in the reply and parses that. The extraction code is in receipt_sorter/extract.py. For image parts in more detail, see the guide Send an image to a model.

6. Build the dataset

A dataset is a set of examples: an input, and a description of what a good answer looks like. Here each input is one receipt link, and the criteria spell out the expected company, date and total.

Dashboard. Open Evaluations, create a dataset, and use Add example for each one. For 50 examples, the SDK is quicker.

Python.

ds = await hub.datasets.create("receipt-sorter-sroie-50")
await hub.datasets.add_examples(ds.id, [
    {"input": {"receipt_url": f"{IMAGE_BASE_URL}/{label['id']}.jpg"}, "criteria": criteria(label)}
    for label in labels
])

criteria writes one sentence per receipt, for example: expected company "F&P PHARMACY", date "02/03/2018", total "31.90"; the company may differ in capitalisation or punctuation only; any other date or total means a low score. The judge in step 7 reads this text.

7. Compare prompts and models

An experiment crosses prompt versions with models over one dataset. A run is one execution of that grid, and each combination of version and model is a cell. So there are two things to compare, and you get both from one run.

First commit a second prompt version with stricter field rules (the day comes first in a Malaysian date, the total is the amount owed after tax, not the subtotal). Put it on the staging alias, and open the Diff tab to see exactly what changed from version 1.

Then create the experiment: dataset, both versions, the four reader models.

Dashboard. Open the dataset and press Run experiment, tick both versions and the four models, and start.

Python.

exp = await hub.experiments.create(
    dataset.id, [v1_id, v2_id], ["rs-gemma-4-31b", "rs-qwen3-vl-30b", "rs-qwen3-vl-8b", "rs-gpt-5.4-mini"],
    prompt_id=prompt.id, name="receipts: all 4 models, prompt v1,2",
)
run = await hub.experiments.start_run(exp.id)
while (await hub.runs.get(run.run_id)).status not in ("succeeded", "failed", "cancelled"):
    await asyncio.sleep(10)
report = await hub.runs.get_report(run.run_id)

A worker process runs the jobs in the background, so a run stays queued until a worker is running. The run takes a few minutes: 400 model calls plus 400 judge calls.

Name experiments after what they contain. The Runs list shows only the name, and "run 3" tells you nothing a month later.

Read the report

The report has a leaderboard and a matrix with one cell per version and model. Each cell shows a judge score and how many of the 50 examples were scored.

The judge is another model that reads the receipt, the model's answer and your criteria, and gives a 0–100 score with a reason. Open any example to read that reason. Three things to know:

The judge model is set by the server (EVAL_JUDGE_MODEL, default gpt-4o-mini) and must be registered in Models, as in step 2. If it is missing, every example stays unscored.
A run ends succeeded even when cells failed or went unscored. [scripts/run_eval.py](https://github.com/talhaanwarch/receipt-sorter/blob/main/scripts/run_eval.py) checks scoredCount for every cell and stops if any is zero.
The judge is the expensive part. It receives the image too, and in this project the judge cost about 13 times the readers' cost per run. Read the token count of one judge trace before starting a full run, and consider a cheaper vision model as judge.

Score from the raw outputs too

A judge score is a good overview, but the question "were all three fields right?" has an exact answer. [scripts/score_run.py](https://github.com/talhaanwarch/receipt-sorter/blob/main/scripts/score_run.py) reads each cell's raw replies and compares company, date and total to the labels, then prints the counts. Provider errors, such as a rate limit, are counted in their own column and are not treated as wrong answers.

Repeat the run

The same prompt scores a little differently each time. Run the experiment three times (python scripts/run_eval.py --repeat 3) and compare ranges, not single numbers. If two models' ranges overlap, treat them as tied. For the guide to this step, see Evaluate a prompt that reads an image.

8. Promote the winner

When version 2 reads better, move the alias.

Dashboard. Open the prompt's Versions tab and move the production alias to version 2. The badges under the prompt name show which alias points where.

Python.

await hub.prompts.promote_alias(prompt.id, "production", 2)

The app asks for production, so its next extraction uses version 2 with no code change and no redeploy. If version 2 turns out worse, point the alias back at version 1.

9. Read the traces

Every gateway call wrote a trace. Open Traces and pick one.

The trace shows the model that ran, the tokens, the dollar cost, and the latency. The tags receipt-sorter and the alias name let you filter all receipts read under production. Because step 5 passed prompt_version_id, the trace also links back to the prompt version, so a wrong total can be traced to the exact wording that produced it. The median seconds and cost per 1,000 receipts in the challenge article come from these spans.

10. Add the app

app.py is about 40 lines of Streamlit. It lets you choose a model and an alias, takes uploaded photos, calls extract_fields for each, shows a table with a link to each trace, and offers the CSV download.

streamlit run app.py

The app holds no prompt text and no provider key. Everything it needs, the wording, the version and the model routing, lives in AcruxCore.

Questions you may have

Can I choose the judge model and the judge prompt?

Yes. You pick both when you set up an experiment. The judge model can be any model registered in Models, so you can use a cheaper vision model than gpt-4o-mini. The judge prompt holds the instructions the judge follows when it scores an answer, and you can edit it for your task. For receipts, you could tell the judge to ignore capitalisation in the company name.

Can I compare several prompts and several models in one run?

Yes. One experiment crosses any number of versions of one prompt with any number of models. Step 7 uses two versions and four models, so one run has eight cells. Each cell has its own score, and the matrix puts all of them on one page. The Evaluate a prompt guide covers the same flow for a text prompt.

Can I use a model that is not on OpenRouter?

Yes. The gateway serves any model that has an OpenAI-compatible API, whether a hosted provider or a server you run yourself. Register it in Models with its base URL and prices, and use its name in the experiment like any other model. A model must accept image input to read receipts.

Can I check the answers with my own code as well as the judge?

Yes. An experiment always scores with a judge, which gives each answer a 0–100 score and a reason. A run also keeps the raw reply of every example. If your task has one correct answer, as the receipt fields do, you can compare those replies to your labels in code, as scripts/score_run.py does. This exact count is made by your script, and the dashboard does not show it. The judge score is a quick overview, and the exact count is the number you can check.

How much does a run cost, and how do I lower it?

The judge is the largest cost, because it receives the receipt image as well as the answer. Run the experiment on 10 examples first, read the token count of one judge trace, and then start the full run. A cheaper judge model lowers the cost of every later run.

Where is my OpenRouter key stored?

In the AcruxCore credential you created in step 2. The app sends a model name to the gateway, and the gateway adds the key. The code in the repository holds only the AcruxCore API key.

Can I run AcruxCore on my own server?

Yes. AcruxCore is open source under Apache-2.0, and the screenshots in this tutorial come from a local server. The steps are the same on a hosted account. The code is at github.com/AcruxCore/AcruxCore.

Where to go next

The measured comparison of the four models, with the table and the limits of the test: the challenge article.
Replace the 50 SROIE receipts with 50 of your own, with labels you checked yourself. SROIE is public and some labels are wrong, so a private set tells you more.
Add a fifth model: register it in step 2 and run the experiment again.

When I started writing on dev.to, it was hard. New profile, no readers, and my posts felt too senior-level. Today I found out I'm a runner-up in the #bugsmash #devchallenge. I even missed the first email! Thank you, DEV community 🙏

Talha Anwar — Sat, 10 Oct 2026 18:11:15 +0000

Clean Air Window: the cleanest hour to go outside in smog season

Talha Anwar — Mon, 05 Oct 2026 20:37:50 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

▶ Try it live: clean-air-window.onrender.com (five plans per visit)

⌨ Code: github.com/talhaanwarch/clean-air-window

What I Built

Clean Air Window picks the cleanest time to go outside today. You give it your city, your activity and the hours you are free. You can also say whether you are in an EPA sensitive group, such as people with asthma. The app answers with a time card, an hourly air quality chart, and advice from the U.S. EPA's health guidance. Some weeks a local news site reports a smog alert or a school closure. In those weeks, the answer names the headline and links to the article.

The EPA splits the air quality index (AQI) into six bands: Good (0–50), Moderate (51–100), Unhealthy for Sensitive Groups (101–150), Unhealthy (151–200), Very Unhealthy (201–300), and Hazardous (above 300). On 5 October, the forecast for Lahore was AQI 137 at 15:00 and AQI 166 at 18:00. A person with asthma wanted a 45-minute walk between 15:00 and 21:00. The app picked 15:00 and suggested a shorter, slower walk. The same answer linked a headline from that week in Dawn, a Pakistani newspaper. The headline said that Lahore again ranked as the most polluted city in the world.

The app is for anyone who walks, runs, cycles or plays outside in a city with seasonal smog. The app's instructions to the AI model say never to tell people to stay inside. When even the best hour is unhealthy, the model suggests a shorter and easier version of the activity.

The AQI values are Open-Meteo's hourly forecast, not measurements.

Demo

The demo plans a walk and clicks "I went outside" to mark that the person went. Then the demo opens the record of that plan in AcruxCore, the platform that stores the app's prompt, tools and run history. The demo ends on two AcruxCore screens that the post explains below. The full-quality MP4 is in the repo.

Code

talhaanwarch / clean-air-window

The cleanest time to go outside today: open-weight Qwen3 + Open-Meteo AQI + EPA guidance (RAG), traced in AcruxCore. DEV Hacktoberfest 2026 entry.

🌿 Clean Air Window

Clean Air Window tells you the cleanest time to go outside today. You give it your city, your activity, the hours you are free, and whether you are in a sensitive group such as people with asthma. It answers with one time window, an hourly air-quality chart, advice taken from the U.S. EPA's health guidance, and any local air-quality news from this week that changes the plan.

Live app: clean-air-window.onrender.com, on Render.

Full-quality demo video (MP4)

What you see

The card at the top and the chart come straight from the forecast data. The model writes the lines below the chart. The health advice links to the EPA document it came from, and the news line links to the headline it came from.

How it works

Gemma 4 31B, an open-weight model from Google, runs a tool-calling loop with three tools:

Tool	What it does

…

View on GitHub

How I Built It

The app is a Streamlit page on top of a tool-calling agent. The model is Gemma 4 31B (google/gemma-4-31b-it), an open-weight model from Google under Apache 2.0, served through OpenRouter. The model can call three tools:

get_air_and_weather downloads today's hourly AQI, temperature and rain chance from Open-Meteo. The tool tries every time window of the activity's length inside your free hours. The tool rates each window by the dirtiest hour it covers, not by the average hour, and returns the best and the worst window. The dirtiest hour decides, because a 45-minute walk that passes through one bad hour still spends part of the walk in that bad air.
search_health_guidance does the retrieval for RAG (retrieval-augmented generation). This tool searches five EPA and AirNow documents and returns the four passages closest to the model's search query. Qwen3-Embedding-8B makes the embeddings. The documents split into only 83 chunks. A set that small needs no vector database: a NumPy matrix and a dot product do the search. The 83 vectors ship inside the Docker image as a 1.3 MB file.
get_local_air_news searches Google News through SerpApi for the city's air quality headlines from the last seven days. The EPA guidance explains what an AQI level means, but the guidance cannot know that a city closed its schools this morning. SerpApi's free plan allows 250 searches a month, so the tool keeps each city's results for six hours.

Python code, not the model, draws the time card and the chart from the tool's best window. A model that copies numbers by hand can copy them wrong. The model writes only the explanation, the health advice, a tip and the news line. The model ends with a JSON answer that has six fixed fields:

class PlanAnswer(BaseModel):
    why: str           # best window vs worst window, with their numbers
    for_you: str       # what the EPA guidance says for this person
    source_title: str  # the guidance document the advice came from
    tip: str           # one practical tip for the activity
    local_news: str    # one sentence about a headline, or empty
    news_title: str    # the exact title of that headline, or empty

The app links source_title to the real EPA document. The app links news_title only when the title matches a headline that get_local_air_news returned in the same run. So the page never shows a news link that the model made up.

What AcruxCore keeps

AcruxCore is an open-source LLM-ops platform that I build. The app uses the hosted version at acruxcore.com. AcruxCore keeps four things for the app: the prompt, the three tools, the list of tools that the model may call with that prompt, and a trace of every plan.

The prompt is a versioned template. An alias is a name, such as production or staging, that points at one version. The template has an if block that adds a line only for people in a sensitive group.

The three tools are in the AcruxCore tool catalog. The @acrux.tool decorator turns each Python function's signature and docstring into a tool definition. A setup script in the repo, scripts/setup_acruxcore.py, sends those definitions to the catalog, and the catalog keeps a version of each tool. The code below is the whole definition of the news tool:

@acrux.tool
async def get_local_air_news(city: str) -> dict:
    """Get the last seven days of news headlines about air quality and smog in a city,
    newest first, such as official smog alerts or school closures. Use them only if a
    headline changes what the person should do today; many cities have none."""

The tools are connected to the prompt in the dashboard. The prompt's Tools tab is a table with one row per tool. Each alias can have its own column, but only when that alias needs different tools. An alias without its own column uses the first column, named default. The app calls run_prompt_with_tools, which reads that list of connected tools from the prompt. The app code never names the tools it offers the model. The code only maps each tool name to the Python function that runs it.

rendered = await hub.prompts.render("clean-air-window", PROMPT_ALIAS, variables)
stream = await hub.gateway.run_prompt_with_tools(
    rendered, model="google/gemma-4-31b-it:nitro", client_tools=CLIENT_TOOLS,
    response_format=acrux.pydantic_response_format(PlanAnswer, name="plan_answer"),
    stream=True,
)

Adding the news tool one alias at a time

The first version of the app had no news tool. Version 1 of the prompt asked for two tools, get_air_and_weather and search_health_guidance, and the production alias pointed at version 1. I added the news tool in four steps:

I committed version 2 of the prompt, which asks the model to call get_local_air_news and to fill local_news and news_title. I pointed staging at version 2 and left production on version 1.
On the prompt's Tools tab, I gave staging its own column and connected the news tool there. The default column, which production follows, still had only the weather and guidance tools.
I ran a second copy of the app with PROMPT_ALIAS=staging and checked that the news line appeared in the answer. The production copy kept serving version 1 with two tools.
I pointed production at version 2, connected the news tool in the default column, and removed the staging column.

The screenshot below shows the Tools tab during step 3. staging has its own column and serves version 2 with three tools. production still follows the default column and serves version 1 with two tools:

One code change came before these four steps: I shipped the app with the news tool's Python function. AcruxCore stores only the tool's definition, and the app runs the function itself. The function did nothing until step 2 connected the tool to an alias. Steps 1 to 4 changed only data in AcruxCore. The running production app picked up step 4 on its next plan, with no deploy.

One trace per plan

A trace is the stored record of one run: every model call and every tool call, in order, with their inputs and outputs. The trace below is the Lahore plan from the screenshot at the top. In its first model call, Gemma asked for the weather tool and the news tool together, and the app ran both tools at the same time. The plan made four model calls in total, and the last call wrote the JSON answer. The "I went outside" button saves its feedback on the same trace.

The model calls go straight to OpenRouter with my own OpenRouter key, and they do not pass through AcruxCore. For each plan, the app first reads the prompt from AcruxCore. When the plan is done, the app sends the trace to AcruxCore. On version 2, plans took between 12 and 22 seconds in my runs, including one on the live app. The time card and the chart appear as soon as the weather tool returns, after about 5 seconds.

Prize Categories

Best Use of Gemma. Gemma 4 chooses which tools to call, calls two of them in parallel, and writes the typed JSON answer in every plan.

Best Use of SerpApi. get_local_air_news uses SerpApi's Google News engine to find this week's smog alerts and school closures. The app shows a news link only when the title matches a headline that SerpApi returned.

Best Use of Render. The live app runs on Render as a Docker web service, built from the repo's Dockerfile. The 1.3 MB guidance index ships inside the image, so the service needs no disk and no database, and a restart loads the index in under a second.

Why Does Open Innovation Matter?

Open weights mean no single company serves the model. Gemma 4's weights are public, so on 6 October 13 providers served Gemma 4 31B through OpenRouter. The :nitro suffix asks OpenRouter for the fastest of those providers on each request. One environment variable chooses the model. Before Gemma, the app ran on Qwen3-Next-80B, another open-weight model, and the switch was this one line:

CHAT_MODEL=google/gemma-4-31b-it:nitro

The repo includes the health guidance. Work by the U.S. federal government, such as EPA and AirNow publications, is not protected by copyright. So the extracted text sits in data/guidance/, and every file names its source URL.

The app does not depend on one company's server. AcruxCore is under Apache 2.0. The setup script creates the tools, the prompt and the tool connections from the repo's code. So the same app can run against a self-hosted copy of AcruxCore by changing ACRUXCORE_BASE_URL. A trace stores a person's city, their free hours, and whether they have asthma, and a self-hosted copy keeps that data in your own Postgres database.

This project was built with help from AI. I used Claude Code to write most of the code and the first draft of this post, and I reviewed and tested both.

I published AI research papers and have 6 YoE building backends. Why do AI coding agents make me feel completely dumb?

Talha Anwar — Sat, 22 Aug 2026 08:02:26 +0000

There is a running joke that AI researchers can't write production software, and backend engineers don't understand research math.

I happen to do both. I’ve published deep learning papers in top conferences, won ML competitions, and spent the last 6 years as a founding engineer building production APIs, async queues, and distributed systems. I read dense math papers and trace database deadlocks for fun.

Yet, on any given day, an AI coding agent can "fix a bug" in 20 minutes and make me question if I even know how to read English.

The "10x Productivity" Myth vs. Management Expectations

Management reads one article on Twitter and suddenly expects everyone to ship features at 10x speed.

Here is what that "10x velocity" actually looks like when hooking up a RAG chatbot to external platform webhooks and syncing vector stores:

1 hour: Planning the sync logic, payload schema, and chunking strategy.
20 minutes: The AI agent writes 400 lines of code across 5 files to fix a streaming concurrency bug. Tests pass. The app runs. Pure magic.
4 hours: Me sitting frozen in front of my monitor trying to reverse-engineer why it bypassed our API gateway, restructured our entire websocket pipeline, and injected three layers of mysterious custom middleware.

The code is "done" in 20 minutes, but now I’m in a full sweat because tomorrow morning in standup, someone is going to ask:

"Hey, what was actually broken with the webhook listener?"

And what am I supposed to say?

"Honestly? The AI wizard cast a spell on line 42 and now it works. Please don't ask me anything else."

The 4-Stage Spiral of Shame

When I don't understand why the agent chose a specific architecture to fix the bug, I ask it to explain. That’s when the psychological damage begins, with my confidence dropping on every single attempt:

Attempt 1: The Pretentious Professor

I ask: "Why did you restructure the vector search callback and event listener this way?"

It replies with a 500-word essay containing words I haven't seen since a GRE test. I re-read it four times. My brain goes numb. Is my English broken? Am I secretly bad at backend engineering?

Attempt 2: The Polite Request

I say: Please explain in simpler, direct terms without the jargon.

It uses everyday words, but strings them together so backwards that the logic eats its own tail:

Of course! We just make sure that when the webhook fires, the thing checking the data knows that the data being checked isn't the same data that triggered the check in the first place.

No fancy jargon, but my brain is completely melted trying to parse the sentence structure. Is my reading comprehension broken? Confidence drops another 25%.

Attempt 3: The Desperation Move

I type: "Explain like I'm 15."

The AI instantly decides I have the cognitive ability of a toddler:

"Imagine your vector database is a magical post office! When a user sends a message, a friendly owl named Barnaby flies across the forest to find the matching letter..."

Attempt 4: Installing the "Fixes"

At this point, I started searching for community prompt hacks. I even installed the popular ayghri/i-have-adhd skill to force the agent into concise, no-fluff, step-by-step bullet points.

And you know what happened? Now the agent gives me blunt, hyper-compressed steps, and I still spend 30 minutes staring at bullet point #2 trying to understand the mathematical jump it made. Now I feel twice as dumb because I can't even digest the ADHD-friendly version.

The Real Problem: Reverse-Engineering Fatigue

Getting an AI-generated integration or bug fix to run is easy. Owning the code in a team of senior engineers is the hard part.

When a teammate looks at the PR and asks:

"Why did we switch to this custom event-emitter pattern instead of our standard message queue?"

I can't reply:

"Well, Barnaby the Owl needed a place to rest his wings."

So you spend half your day decoding AI logic just to translate it into normal human engineer terms. We didn't eliminate the coding bottleneck—we just traded writing code for full-time reverse-engineering.

Let's Settle This in the Comments:

How do you handle standup when AI fixed a bug but you have zero idea how? Do you fake confidence or admit the robot did it?
What is the most insulting analogy an AI has ever given you when you asked it to "simplify"?
What exact system prompt do you use to make your agent talk like a normal, grounded senior dev without getting a PhD thesis or a fairy tale?

Our SDK Cached One User's Answer and Served It to the Next One Who Asked

Talha Anwar — Tue, 11 Aug 2026 19:21:55 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I work on AcruxCore, an LLM ops platform with a published SDK in TypeScript and Python. Both SDKs render prompts client-side against a small in-memory cache, so a hot path doesn't make a network round trip on every call:

const rendered = await hub.renderPrompt('rag-chat', 'production', { question });

renderPrompt fetches the templated messages once, then serves that same result out of the cache for cacheTtl milliseconds (60 seconds by default) before checking the API again. That's the whole feature. Two bugs were hiding in how "serve that same result" actually decided what counted as "the same."

Bug Fix or Performance Improvement

Correctness bug — the cache key didn't include the one thing that changes on every real call: the variables.

The key was built from three fields:

const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}`;

API key, prompt name, alias. Nothing about what you're actually asking. Two different renders of the same prompt with different variables, inside the same TTL window, hashed to the identical key — so the second call got the first call's cached answer back, not its own.

This wasn't a hypothetical. It was live in our own docs. The RAG-without-a-gateway tutorial renders the same prompt per question:

rendered = await hub.prompts.render("rag-chat", "production",
                                     {"context": context, "question": question})

Ask "Where is my order?", then ask "How do I refund?" within 60 seconds, and the second call's rendered.messages still says "Where is my order?" — while the trace viewer, which logs what you passed in, correctly shows "How do I refund?" as the input. The trace and the actual prompt sent to the model disagree, and nothing errors. It's also exactly why a sibling tutorial, the tool-calling agent guide, never puts the question in a template variable at all — it appends it to the messages array in code instead, a design forced by this bug rather than a stylistic choice.

The second bug was in the same file, one function down:

if (age < this.cacheTtl) {
  return cached.value; // fresh — serve it directly
}
// stale — serve the old value anyway, refresh in the background

Setting cacheTtl: 0 to turn caching off did the opposite. age < 0 is never true, so every single call took the stale branch: return the cached value immediately, and kick off a background refetch nobody's waiting on. A TTL of 0 didn't mean "don't cache" — it meant "always serve whatever was cached first, forever." There was no setting that actually disabled the cache.

Two bugs, one root cause

Both come from the same gap: the cache didn't have a way to represent "this input is different" or "this call opted out." One key was too coarse; one flag had no matching branch. Fixing them meant adding both.

Code

The fix is public in our mirror — same logic in both SDKs, shown here in TypeScript:

packages/sdk/src/client.ts

Bug 1 — hash the variables into the key:

/**
 * Serialises a value with object keys in sorted order, at every depth, so that
 * two variable maps that differ only in insertion order produce one string.
 */
function stableStringify(value: unknown): string {
  if (value === null || typeof value !== 'object') {
    return JSON.stringify(value) ?? 'null';
  }
  if (Array.isArray(value)) {
    return `[${value.map(stableStringify).join(',')}]`;
  }
  const entries = Object.entries(value as Record<string, unknown>)
    .filter(([, v]) => v !== undefined)
    .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
  return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(',')}}`;
}

function hashVariables(variables: Record<string, unknown>): string {
  return createHash('sha256').update(stableStringify(variables)).digest('hex').slice(0, 16);
}

- const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}`;
+ const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}:${hashVariables(variables)}`;

Bug 2 — make a non-positive TTL actually skip the cache, on read and write:

  async renderPrompt(name, alias, variables = {}) {
+   // A non-positive TTL means "never serve a cached render" — skip the read and
+   // pass a null key so the write side skips too, instead of filling the cache
+   // with entries no read path will ever consult.
+   if (this.cacheTtl <= 0) {
+     return this._fetchAndCache(name, alias, variables, null);
+   }
    const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
    const cacheKey = `...`;
    ...
  }

  async _fetchAndCache(name, alias, variables, cacheKey /* now: string | null */) {
    ...
-   const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
-   cache.set(cacheKey, { value, fetchedAt: Date.now() });
+   if (cacheKey !== null) {
+     const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
+     cache.set(cacheKey, { value, fetchedAt: Date.now() });
+   }
    return value;
  }

Technical Approach

Why hash sorted keys at every depth instead of `JSON.stringify(variables)`

JSON.stringify preserves insertion order. { name: 'Alice', city: 'Lahore' } and { city: 'Lahore', name: 'Alice' } are the same request to any caller, but they'd stringify to two different strings — and split into two cache entries for what should be one. A caller that builds the variables object by spreading defaults first ({ ...defaults, question }) versus spreading them last would silently halve their own cache hit rate. Sorting keys at every depth, including inside nested objects, makes key order invisible to the cache. A dedicated unit test pins exactly this:

it('treats the same variables in a different key order as one cache entry', async () => {
  (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValue(makeOkResponse());
  await hub.renderPrompt('my-prompt', 'production', { name: 'Alice', city: 'Lahore' });
  await hub.renderPrompt('my-prompt', 'production', { city: 'Lahore', name: 'Alice' });
  expect(global.fetch).toHaveBeenCalledTimes(1);
});

Why not just delete the "serve stale, refresh in background" path

That's a real reliability feature, not accidental complexity: if the API is briefly unreachable, a caller with a warm cache keeps getting an answer instead of an error. Deleting it to fix the cacheTtl: 0 bug would trade one bug for a regression. The fix instead adds a third state — "don't cache at all" — that bypasses the stale-serving branch entirely rather than replacing it, so cacheTtl: 30_000 still falls back to a stale value during an outage, and only cacheTtl: 0 gives that up on purpose. Both behaviors are documented side by side in the README so the trade-off is explicit, not discovered during an incident.

Proving the fix actually fixes it, against a real API

Unit tests mock fetch. The two bugs above are about what a real render pipeline does with real prompt versions, so the integration suite hits an actual Express app and a real Postgres database instead:

it('renderPrompt renders the new variables inside the cache window instead of replaying the first render', async () => {
  // ...creates a real prompt, a real version with "Question: {{ question }}"...
  const hub = new acruxcore({ apiKey, baseUrl, cacheTtl: 600_000, maxRetries: 0 });

  const first = await hub.renderPrompt(name, 'production', { question: 'Where is my order?' });
  const second = await hub.renderPrompt(name, 'production', { question: 'How do I refund?' });
  const repeat = await hub.renderPrompt(name, 'production', { question: 'Where is my order?' });

  expect(first.messages[0].content).toBe('Question: Where is my order?');
  expect(second.messages[0].content).toBe('Question: How do I refund?');
  expect(repeat.messages[0].content).toBe('Question: Where is my order?'); // still cached, not evicted
});

I reverted src/client.ts and re-ran this test before writing the fix. It failed on exactly the assertion you'd expect — second.messages[0].content came back as 'Question: Where is my order?', the first question, not the second. That's the failure this whole post describes, caught by a test instead of a support ticket.

A second integration test does the same for Bug 2 — cacheTtl: 0, promote a new prompt version, confirm the very next call sees it instead of serving the version that was cached before the promotion.

Two existing tests were asserting the bug

The old unit test for caching was itself proof the bug had never been tested against a variable change:

- it('caches the result — second call does not hit fetch', async () => {
+ it('caches the result — repeating the same variables does not hit fetch', async () => {
    await hub.renderPrompt('my-prompt', 'production', { name: 'Alice' });
-   await hub.renderPrompt('my-prompt', 'production', { name: 'Bob' });
+   await hub.renderPrompt('my-prompt', 'production', { name: 'Alice' });
    expect(global.fetch).toHaveBeenCalledTimes(1);
  });

It rendered Alice, then Bob, and asserted one fetch call — which is precisely the bug, written down as a passing test. Fixing the implementation without touching this test would have left it green while lying about what "caching" meant. It had to be rewritten to assert the correct behavior — different inputs, different fetches — before the fix could be trusted.

Results

	before	after
second `renderPrompt` call, different variables, inside TTL	first call's cached content (wrong)	its own rendered content
repeated call, same variables as the first, inside TTL	cached content (correct, coincidentally)	cached content (correct, by design)
`cacheTtl: 0`, two calls	1st content served both times	fresh API call both times
entries written to the cache with `cacheTtl: 0`	one per call	zero
variables passed in a different key order	new cache entry (cache pressure ↑)	same entry, one hit

Test suite after the fix: 111 TS unit, 14 TS integration (both new ones failing-then-passing verified against the revert), 112 Python, clean tsc --noEmit, clean docs build.

What I'm carrying to the next cache I write

A cache key has to include everything the caller varies, not just everything the caller identifies with. API key and prompt name answer "who and what" — they don't answer "with what input," and that's the part that actually changes per call.
"Off" needs its own code path, not a boundary value on an existing one. 0 looked like it should mean "never fresh, so never serve cached" — the actual check (age < ttl) made it mean the opposite. A dedicated bypass, not a clever value, is what you want for a real off-switch.
A passing cache test that only ever sends one input is testing the plumbing, not the cache key. The bug survived in production because the one existing test for "does caching work" happened to use two different inputs and asserted the outcome the bug produces.

If your SDK caches a rendered or computed result keyed by anything less than the full input, it's worth a five-minute check today: call it twice with different arguments inside your TTL window, and read back what you actually get. What did yours return?

LLM Observability Tools Compared: Langfuse vs Helicone vs Opik vs Phoenix

Talha Anwar — Tue, 11 Aug 2026 19:00:00 +0000

The first trace looks the same everywhere

Wrap your LLM client with any open-source observability SDK — Langfuse, Helicone, Opik, Phoenix, doesn't matter which — and the first result is identical: a request goes out, a span shows up in a dashboard with the prompt, the response, and how long it took. That part is solved. Every landing page in this space makes the same claim, and for a first trace, they're all telling the truth.

The next question every team asks is also the same one: can this tool act on a trace, not just show me one? Score it. Alert someone. Block it before it ships. On the surface that looks solved too — all four platforms have an "Evaluation" tab and a "Settings" page, so it's tempting to assume they converge there as well.

They don't. I have all four self-hosted right now, and going through each dashboard side by side, four specific capabilities stopped me — each one a genuine standout for exactly one platform, and absent from at least two of the other three. Picking a tool because "it does observability" and finding out six months later that the one feature you actually needed lives in a dashboard you didn't choose is an expensive way to learn this.

TL;DR

Langfuse — an LLM judge that scores every live production trace on its own, no manual review step required.
Helicone — a dedicated rate-limit rule builder, a different object from a simple spend cap.
Opik — topic and PII guardrails that inspect a call's input or output directly, configurable per project.
Phoenix (built by Arize) — PXI, an AI agent docked inside the dashboard that already knows the trace you're looking at.

At a glance

Feature	Langfuse	Helicone	Opik	Phoenix
Auto-scores live traces, no manual trigger	✅ LLM-as-a-judge	❌	✅ Online evaluation rules (same idea, different name)	❌
Dedicated rate-limit rule builder	❌ (alerts only)	✅	❌	❌
Topic / PII guardrails on the call itself	❌	❌	✅	❌
AI agent built into the dashboard	❌	❌	❌	✅ PXI

Opik is the only platform that clears two of the four rows. Full breakdown below.

Langfuse: the judge that never sleeps

Most platforms' "evaluation" means a human clicks run against a dataset. Langfuse's Evaluators page does something different: you write an eval template — a scoring prompt, a model, and the trace variables it should read — and it fires that template against every new production trace as it lands. There's no "run experiment" button in the loop at all.

That's the deciding detail if you're comparing tools on this feature: is scoring something you trigger, or something that's already running before you open the dashboard?

Langfuse's own docs walk through setting one up. Opik's Online evaluation rules do the same job under a different name (see the table above), so Langfuse doesn't own this category outright — but it's the platform where this run found it first, with execution tracing on the evaluator itself as a bonus.

Takeaway: if unattended, always-on trace scoring is the requirement, Langfuse and Opik are the two platforms here built for it. Helicone and Phoenix both still need someone to explicitly trigger a run.

Helicone: a rate-limit rule, not a spend cap

A spend budget answers one question: how much can this app cost before it stops. A rate-limit rule answers a different one: how many requests can this specific user make in this window, independent of what anything costs.

Helicone's Monitor → Rate Limits page has a "Create Rule" builder for exactly that second question. A rule is a quota, a time window, a unit (requests, tokens, or dollars), and a segment — user, team, or global — expressed as a header like 10;w=1000;u=cents;s=user, which reads as "10 cents of requests per 1000 seconds, per user." Helicone's docs cover the full syntax.

Takeaway: if one noisy user needs throttling without shutting the whole app's budget down for everyone else, that's a rate-limit rule, and Helicone is the only one of the four that ships one as its own object.

Opik: guardrails that inspect the call itself

Everything above scores or alerts after a call happens. Opik's guardrails act on the call directly — before or after the model sees it, depending on which check you enable.

The "Set a guardrail" panel has two checks: a Topic guardrail (a sensitivity slider plus a comma-separated restricted-topics list) and a PII guardrail (per-category toggles — credit card number, phone number, email, and more — each with its own sensitivity threshold). Both ship with a ready opik.guardrails Python snippet. Comet's announcement post covers what triggers each one.

Takeaway: if the requirement is blocking a category of content before it ships, not scoring it afterward, guardrails are the only feature in this list built for that — and Opik is the only platform that has both this and an auto-scoring rule.

Phoenix: PXI, the assistant that already knows your traces

PXI is a chat panel that lives on every Phoenix page, seeded with suggestions like "Find critical issues" and "Explain a concept." The detail that makes it more than a docs chatbot bolted onto the product: it reads from whatever page you're already on, so it can answer questions about your own traces without you pasting a trace ID or copying a payload into it. Arize's docs describe it as an agent that debugs traces, builds evaluators, and runs experiments from the context you're already viewing.

Takeaway: if "explain what's happening in this trace, right now, without switching tools" is the requirement, Phoenix is the only platform here with an assistant scoped to the page in front of you.

Verdict

None of these four features decide which tool is "best" — they decide which tool fits a requirement you may not have hit yet. A team that never needs unattended scoring won't miss Langfuse's judge or Opik's online evaluation. A team running one internal chatbot for five people won't miss Helicone's rate-limit rules.

But if you're picking a platform today assuming "observability" is a solved, interchangeable checkbox, this is the list of four ways that assumption breaks: automated live-trace scoring, a real rate-limit object, content guardrails, and an in-dashboard agent. Check which of the four you'll actually need before you wrap your first client, not after.

Which of these four would you actually use — and is there a fifth dashboard feature I missed that deserves its own row?

Verdict

This piece only had room for one standout feature per platform. AcruxCore has compared 8 LLM observability tools in more depth, aspect by aspect.

Which of these four would you actually use — and is there a fifth dashboard feature I missed that deserves its own row?

Opik vs Langfuse: Where Two Open-Source LLM Observability Tools Actually Agree (and Where They Don't)

Talha Anwar — Sun, 09 Aug 2026 19:55:19 +0000

The box both of them check

If you're adding your first bit of visibility into an LLM app, the simplest version is a print() statement before the API call. That's enough while you're the only one testing it.

The natural next step is to swap that print statement for a real tracing SDK. You wrap your client, make a call, and a UI shows you a span with the prompt, the response, and how long it took. That feels solved — until your team grows past one prompt and one person testing it by hand.

That's the point where the real friction shows up: conditional prompt logic, a way to diff two prompt versions before you ship one, a way to catch a call your app got wrong before a user complains, and a currently-live guardrail against risky output. "It produces a trace" doesn't cover any of that. And the frustrating part is that this doesn't fail loudly — most open-source, self-hosted LLM observability tools look interchangeable from their landing pages, because they'd all check the same three boxes: open source, self-hosted, traces your calls.

Opik (built by Comet ML) and Langfuse are two of the most-used tools in that box. So we tested the actual boundary: we built the exact same prompt — a support-triage agent that changes tone for VIP customers and lists their open tickets — on both, independently, then set the two runs side by side.

TL;DR

Both platforms hit the exact same wall on prompt templating — neither supports real conditional logic, only flat variable substitution.
Both confirmed, independently, that their Playground alone does not produce a trace — only a wrapped SDK call does.
They diverge sharply on prompt versioning: Opik ships a real diff view and environment labels, Langfuse's version history has neither.
Their measured client-side overhead is different in kind, not just number — and the two numbers come from two separate test sessions, not one shared benchmark.
Both have a feature that does the same job under a different name — auto-scoring live production traffic.
They've drifted into genuinely different territory: guardrails and regression test suites (Opik) versus threshold alerting (Langfuse).
And as of five months ago, they're no longer comparable as two independent startups — Langfuse was acquired by ClickHouse in January 2026.

At a glance

Aspect	Opik	Langfuse
Prompt templating	Flat `{{variable}}` only	Flat `{{variable}}` only
Playground produces a trace	No — confirmed hands-on	No — confirmed hands-on
Version diff	Real red/green diff view	No diff control found
Environment labels	"Deploy to" tags a version	`production`/`latest` labels
Measured SDK overhead	+102ms (real, CI doesn't cross zero)	+15ms (not distinguishable from zero)
Auto-scores live traffic	Online Evaluation rules	LLM-as-a-judge evaluators
Standout feature	Guardrails (topic + PII) and test suites	Monitors — alerts to Slack/Webhooks/GitHub
GitHub stars (checked today)	21,252	32,778
Owned by	Comet ML (in-house product)	ClickHouse (acquired Jan 2026)

Full breakdown below.

Where they hit the exact same wall: templating

Both platforms' prompt editors only do flat {{variable}} substitution — no {% if %}, no {% for %}. We needed a VIP-customer branch and a rendered list of open tickets, and on both platforms we had to flatten that logic into plain text ourselves before the prompt ever saw it, keeping only the simple fields (like {{company}}) as real template variables.

Neither product hides this — it's a real, shared design choice, not a bug in either one. If your prompts need branching logic, both push that logic back onto your own code (a pre-render step) rather than into the template itself.

Where they hit the exact same wall, again: the Playground doesn't trace

This one surprised us enough to check twice on each platform. Running a prompt from Opik's Playground left its Logs tab at "No traces yet." Running a prompt from Langfuse's Playground did the same thing — no trace appeared until we called the SDK-wrapped client instead. On both, the interactive "try it in the UI" surface and the "get a trace out of it" surface are two separate things, which is easy to miss if you only ever click around the Playground before wiring in the SDK.

Where they genuinely diverge: prompt versioning

Opik has a real "Diff" button — edit a prompt to create v2, and it shows the old system message in red strikethrough next to the new one in green. It also has a "Deploy to" menu that tags a specific version production, staging, or development with a colored badge.

Langfuse has environment labels too (production / latest), but we could not find any diff or compare control anywhere in its version history UI — each version sits in a flat list, and comparing two of them means opening both and reading side by side yourself.

Where the numbers genuinely differ (and why you shouldn't stack them)

We measured client-side SDK overhead for both platforms — but in two separate benchmark sessions, run on different days as part of two different posts, each with its own direct-to-provider baseline. That matters: network conditions differ session to session, so these two numbers aren't from one shared interleaved test and shouldn't be read as a single ranked table.

With that caveat stated plainly:

	Opik SDK	Langfuse SDK
Measured overhead	+102ms	+15ms
95% bootstrap CI	[+14, +191]ms	[-90, +150]ms
Statistically real?	Yes — CI does not cross zero	No — CI crosses zero, indistinguishable from noise

Comparison with other platform can be found at AcruxCore

Opik's track_openai() wrapper adds overhead that clears the noise floor in its own test session. Langfuse's instrumentation, in its own separate test session, didn't clear that floor at all — its "cost" was statistically zero. Both are real findings about client-side instrumentation, not about which product is "faster" than the other, since neither routes a call through a gateway the way a proxy-style tool would.

Same idea, different name: scoring production traffic automatically

Both platforms have a feature that does the same underlying job — score live traffic as it lands, with no manual review step — under completely different names and UI.

Langfuse's LLM-as-a-judge Evaluators walk you through writing an eval template: a scoring prompt, a model to run it, and the trace variables it reads.

Opik's Online Evaluation page, under Production, lets you create a rule that scores every matching trace as it arrives.

If a platform-neutral feature list only checked for the name "online evaluation" or "LLM-as-a-judge," it would miss that both products actually offer this.

Where they've genuinely diverged: guardrails vs. alerting

Opik ships a guardrails panel — a topic-restriction check with a sensitivity slider and a comma-separated restricted-topics list, plus a PII (personally identifiable information) check that flags categories like credit card numbers, phone numbers, and email addresses, each with its own threshold. It also has a dedicated Test suites object for pre-deployment regression testing, distinct from its Experiments feature — import cases from a CSV/JSON file or the SDK, each with an expected output and a scoring method.

Langfuse doesn't ship anything like that. What it has instead is Monitors — set a threshold on cost, quality, or latency once, and get a notification routed to Slack, a webhook, or a GitHub Action when a metric moves outside it.

Neither one has the other's feature. If you need to block a call before it reaches a user for containing PII, Langfuse has nothing for that today. If you need a Slack ping when your P95 latency creeps up, Opik has nothing for that today.

Who's behind them, and how big is each

This is where the story changed since we first tested these products. Numbers below are live as of this week, pulled straight from each project's GitHub API.

	Opik	Langfuse
GitHub stars	21,252	32,778
Forks	1,690	3,524
Contributors	~136	~190
Open issues	178	764
Repo created	May 2023	May 2023

Both repos were created within a week of each other in May 2023 and are still receiving commits daily — this isn't a "one's abandoned" story. Langfuse has the larger community footprint on every raw count, including a notably higher open-issue count, which can mean more active usage surfacing more edge cases, or a smaller team keeping up with triage — we didn't dig into issue-close rates to tell which.

The bigger difference is who owns each project. Opik is a product inside Comet ML, a New York-based ML platform company founded in 2017, with $6.8M in disclosed seed funding (2018, 2020) and nothing disclosed since. Langfuse was an independent, Y Combinator-backed startup — until ClickHouse acquired it on January 16, 2026, the same day ClickHouse announced a $400M Series D at a $15B valuation. Both companies have publicly committed to keeping Langfuse open source and self-hostable going forward.

For a reader picking a platform to build on, that's a real signal worth weighing alongside the feature list: one tool is a side product of a smaller, steady company; the other just became an integration point inside a much larger, newly-flush infrastructure company. Neither is inherently the safer bet — but they're different bets.

We also compared each project's documentation footprint via its own sitemap, with a caveat: Langfuse's sitemap covers its entire domain (blog, changelog, academy, and docs together — 822 URLs, of which only 112 are strictly /docs), while Opik's docs live under Comet's site and its sitemap is scoped to just that docs subtree (709 URLs). So "709 vs 822" isn't an apples-to-apples count of documentation depth — it's closer to "Opik's docs alone" versus "Langfuse's docs plus its blog, changelog, and handbook combined."

What we're not calling a winner

We picked seven points, not the full feature list either platform ships, and skipped a symmetric "pro/con" for every row — some things really are one-sided (neither platform gained a fake counterweight just to look fair). Templating and Playground-tracing are genuine ties. Versioning favors Opik. The overhead numbers aren't comparable to each other, only to their own baselines. Guardrails/test-suites and Monitors/alerting are two teams building in different directions, not one being behind the other.

If you're choosing between them today, the feature list would point you toward whichever gap matters more to your team — guardrails and regression testing, or threshold alerting. The ownership question is the one that didn't exist a year ago, and it might now matter just as much as either feature list.

Sources: comet-ml/opik and langfuse/langfuse on GitHub (checked 2026-08-10); ClickHouse's acquisition announcement; Langfuse's own "joining ClickHouse" post. Product screenshots and hands-on findings are from our own independent testing of both platforms.

LangSmith Just Shipped an LLM Gateway. Here's the 3-Year Mess That Made Me Care

Talha Anwar — Sat, 08 Aug 2026 13:33:14 +0000

Just found that langsmith has launched gateway feature and I decided to share my experience. It is about how messy real LLM tooling decisions actually are, and why a small feature announcement from LangSmith made me sit up this month.

Where it all started

In August 2023, I onboarded my first RAG (retrieval-augmented generation — you fetch relevant documents first, then hand them to the model as context) chatbot client as an AI lead. Back then the models we had access to were much weaker and much less reliable than what you get today. Hallucinations were common, context windows were small, and you spent half your time just making the model behave.

The setup was simple, though. One client, one model, one hardcoded prompt in the code. No observability tool. No gateway. It just worked because there was nothing to break.

The step that looked solved

Then we got client number two. Then three. The obvious move was to copy the same folder, swap the API key, and ship it. For a while, that felt like scaling.

It was not. Our prompts lived scattered across different codebases. Our logs sat in our own Postgres database, one table per project, with no shared view across clients. When something went wrong for one client, we had no fast way to tell if it was that client's prompt, a model change, or something else entirely.

That is the real friction nobody warns you about. A broken prompt does not throw an error. It just quietly returns worse answers, and you might not notice for days — by which time a client has already noticed for you.

The decision that forced

That friction split into two separate problems we had to solve on purpose:

Where do we store and version prompts, and see what is actually happening across every client's traces (a trace is the full record of one request — every step, every token, every millisecond)?
How do we talk to multiple model providers through one shape, so a provider outage does not become our outage?

The first is an observability or "LLM ops" platform. The second is a gateway. Here is the order we actually went through both, mistakes included.

Round one: LangSmith, and giving up on it

LangSmith is LangChain's platform for tracing LLM calls, storing and versioning prompts, and running evaluations. We picked it up early because it was the obvious default in the LangChain ecosystem, and we were already leaning on some LangChain pieces.

We stuck with it for a few months before we dropped it. It kept breaking in ways that made us stop trusting the data it showed us — traces that did not line up with what we already had in our own Postgres logs. When your monitoring tool is less reliable than the thing it is monitoring, you stop using it.

Round two: the platform Anthropic later swallowed whole

So we moved to Humanloop, a platform built for the same prompt-management-and-evaluation job. It worked well enough for a good stretch.

Then, in August 2025, Anthropic acqui-hired Humanloop's founders and most of its team — no product, no IP, just the people. Humanloop as a platform sunset shortly after. If you have never seen an acqui-hire before: a bigger company hires the team, not the company, and the product itself just stops being maintained. We had to move a second time, through no fault of our own engineering choices.

Round three: back to LangSmith, warts and all

By then LangSmith had matured, and honestly, we did not have many other doors left to knock on. We are still on it today. It is not the platform I would design from scratch — there are features we need that are still missing — but it is stable enough now, and it gives us the cross-client visibility we were missing back in 2023.

Meanwhile, the gateway problem: why we picked LiteLLM

While all that platform-hopping was happening, we separately needed a gateway (a layer that sits between your app and every model provider, so your code talks to one consistent API instead of five different ones). We landed on LiteLLM, which we host ourselves on our own server.

LiteLLM is not perfect. We know about specific bugs in it, and we live with them, because the thing it does right is the thing we cannot do without: automatic fallback to another model or provider when the primary one errors out or gets rate-limited. That single feature is probably the most-used piece of LiteLLM in our whole stack. When a provider has a bad day, our clients never know.

This month: LangSmith adds its own Gateway

Then this month I noticed LangSmith has its own LLM Gateway — announced in private beta on May 13, 2026, and opened to public beta on July 30, 2026. My first reaction was genuinely good: one platform for traces, prompts, and provider routing instead of two separate systems to keep in sync.

I went in assuming the one feature we could not give up — fallback — would still be missing, because the original May announcement listed model fallback under "where this is going," as a future plan rather than a shipped feature. I was wrong about that. By the July public beta, LangSmith Gateway already ships configurable model fallbacks, chained across models, providers, and hosts, triggered by rate limits or provider errors. That is genuinely good news, since fallback is the one LiteLLM feature we would have refused to give up.

But here is the part I still cannot skip: latency. Our LiteLLM instance is self-hosted, on our own server. A request goes from our app to a local process and out to the provider — no extra hop over the public internet. LangSmith Gateway is a hosted, external service. Routing every completion through it adds a real network round trip we do not currently pay.

A fallback that reads well on a feature page has not been through a real outage on our own traffic yet. Before we retire LiteLLM, I want to see two numbers with our own eyes: the added latency per request, and how the fallback behaves the first time a provider actually goes down on us, not in a demo.

What I would tell you, if you were the one making this call

Do not trust a feature list over battle scars, even when the feature list is telling the truth. Test the exact thing you depend on most, under the exact load you actually run, before you let go of the tool that is already working.

Three years, four tools, one lesson that keeps repeating: the tool that wins is not the one with the longest feature list. It is the one that is still standing after the outage nobody planned for.

So tell me your version of this. Has anyone already pushed LangSmith Gateway through a real provider outage or a hard rate limit on real production traffic? Or did you go through your own round of tool-hopping — what made you leave a platform, and what finally made one stick? I would genuinely like to hear it before we run our own test.

Every Check Said Our GA4 Analytics Worked. It Had Never Recorded a Visit

Talha Anwar — Sat, 08 Aug 2026 10:06:48 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Adding analytics to a website is the most boring task in web development. Google hands you eight lines, you paste them above </head>, and traffic shows up in the dashboard about a minute later.

On a real app you tidy it up a little. The measurement ID moves into an environment variable, the snippet becomes a module, and you add a cookie banner with Consent Mode so nothing is tracked before a visitor agrees. Still a small job, and it still looks finished when the banner renders.

Here's the part nobody warns you about: every layer you just added can fail without saying a word. A build tool can drop the env var. A queue can accept commands it will never run. A CI secret can point at a property nobody is watching. None of them throws. None of them logs. The site loads fine, the banner works, and the dashboard stays at zero.

So the only check that means anything is the last one — did a hit actually arrive? This is the story of the three silent failures I had to clear before the answer was yes, and the second one nearly had me file a bug report against Google.

The setup

I work on AcruxCore, an LLM ops platform. Two sites: the marketing site and dashboard (apps/web, React + Vite, deployed in Docker), and the docs site (apps/docs, Docusaurus, deployed to Cloudflare Pages).

I shipped Google Analytics 4 to both in one PR. Consent Mode v2 with analytics_storage denied by default, a cookie banner, consent stored as a first-party cookie shared across subdomains.

I tested it properly before merging:

Playwright against a real dev server — accept, decline, reload, reopen from the footer
built the bundle with the env var set and grepped the output for the measurement ID
built it again with the var unset and confirmed the tag never loads

All green. I merged it, rebuilt the production image, and opened Google's tag inspector on the live site.

Tag not detected.

Bug 1 — the tag was never in the bundle

The cookie banner rendered on production, so the new frontend had definitely deployed. That's what made this confusing: the same PR shipped both, and only half of it arrived.

I SSH'd into the VPS and worked backward from the artifact rather than forward from the config.

# search the actually-served bundle for Google's script host
docker exec acruxcore-web grep -rc googletagmanager /usr/share/nginx/html/assets/
# → 0

Zero. Not a wrong ID, not a consent problem — the tag was never compiled in. A rebuild with docker compose build web still gave zero, and a full docker build --no-cache reproduced the same empty result.

At that point I'd checked the Docker ARG/ENV lines, the args: block in docker-compose.yml, and the define in vite.config.ts. All three were correct. So I stopped reading config and ran the build one layer down, inside the same built image:

# bypass the orchestrator, call the bundler directly
GA4_MEASUREMENT_ID=G-XXXXXXXXXX npx vite build
# → googletagmanager IS in the output

vite build produced the tag. turbo run build did not. Same image, same env, same everything else. The bug wasn't in Docker, Compose or Vite — it was in the layer between them.

What Turborepo was doing

Our Dockerfile builds through the monorepo task runner:

RUN npx turbo run build --filter=@acruxcore/web

Turborepo's default envMode is strict. Strict means a task's process starts with a filtered environment. It sees only the variables you declared in that task's env config, plus any auto-inferred from a recognized framework's naming convention. For Vite, that's anything prefixed VITE_.

Our variable is called GA4_MEASUREMENT_ID. It is deliberately not VITE_-prefixed, because vite.config.ts remaps it at build time:

define: {
  'import.meta.env.VITE_GA4_MEASUREMENT_ID':
    JSON.stringify(process.env.GA4_MEASUREMENT_ID ?? ''),
},

That ?? '' is the whole tragedy in two characters. Turbo stripped GA4_MEASUREMENT_ID before Vite's config file ever ran, so process.env.GA4_MEASUREMENT_ID was undefined, the fallback kicked in, and the bundle compiled a perfectly valid empty string. Analytics reads the ID, finds '', and correctly decides not to load. No error, because nothing went wrong — every layer did exactly what it was told.

Docker and Compose passing the variable in correctly made no difference. Turbo removed it after they handed it over.

npx turbo run build --dry=json --filter=@acruxcore/web \
  | jq '.tasks[0].envMode, .tasks[0].environmentVariables.specified.env'
# → "strict"
# → []      ← nothing declared, so nothing gets through

The fix is one line in turbo.json:

 "build": {
   "dependsOn": ["^build"],
   "outputs": ["dist/**", "build/**", ".docusaurus/**"],
+  "env": ["SENTRY_WEB_DSN", "GA4_MEASUREMENT_ID"]
 }

Mirror file: turbo.json

The part that actually scared me

GA4_MEASUREMENT_ID wasn't alone in that env array. SENTRY_WEB_DSN was there too, and I only added it because it has the same shape — a non-VITE_ name remapped in the same define block.

Then I reverted the fix and re-ran the build with SENTRY_WEB_DSN set, the same way I'd reproduced the GA4 failure. Identical result: stripped, empty string, no SDK.

Sentry had been wired into the web app six days earlier. In that window, the browser SDK on the deployed site had almost certainly never sent a single error report.

And nothing would ever have told me. A quiet analytics dashboard is at least ambiguous — maybe nobody visited. A quiet error monitor looks like good news. That is the failure mode I keep thinking about: the tool whose whole job is telling you when something is wrong can be the one thing that's broken, and its silence reads as success.

What wasn't affected. The docs site builds in CI with npm run build -w @acruxcore/docs — never through Turbo — so it picked up the measurement ID on the first deploy. Half the platform worked from day one, which is exactly why I initially suspected something specific to the marketing site.

Bug 2 — the tag loaded, the queue filled, nothing was sent

New image deployed. googletagmanager now in the bundle, script loading in the network tab, cookie banner working, consent granted.

GA4 Realtime: zero users.

I went through the browser console expecting an obvious break, and instead found every single signal healthy:

the https://www.googletagmanager.com/gtag/js?id=… script loads with a 200
window.google_tag_manager is defined
window.dataLayer.push !== Array.prototype.push — gtag.js has replaced the queue's push, so it is watching
the queue fills up with my consent, js and config commands

And no request to /g/collect. No _ga cookie. Nothing dispatched, ever.

I also ran GA4's own "Test your website" check from the admin panel. It went green — which I now know only means the tag script is present on the page. It never checks whether a hit arrives.

The wrong turn

Everything on my side was correct, so I started building the case that the problem was Google's.

I set up a second, completely unrelated GA4 property and pointed the site at it. Identical zero-dispatch behaviour. That felt conclusive: not my account, not my property, not my tag config.

I got as far as writing a standalone ga4-zero-hits-repro.html to hand over, drafting the issue report, and opening a PR to track the escalation. The report was finished and queued to post.

The moment it broke

Before posting I re-read my own repro page, and the thing I'd been ignoring finally registered. I had hand-written that page from Google's documentation rather than copying the code my product actually runs.

Which means it contained Google's canonical stub:

function gtag(){ window.dataLayer.push(arguments); }

And my product contained mine:

function gtag(...args: unknown[]): void {
  window.dataLayer = window.dataLayer ?? [];
  window.dataLayer.push(args);
}

Those look equivalent. TypeScript is happy, ESLint prefers the second one, and every reviewer including me read straight past it. My "repro of the bug" had quietly swapped out the one line that was the bug.

So I stopped reasoning and isolated the single variable. Two pages served from the same local origin, same measurement ID, same Consent Mode sequence, differing only in the push form:

stub	hits to `/g/collect`	`_ga` cookie
`push(args)` — what shipped	0	not set
`push(arguments)` — canonical	2 (`page_view`, `scroll`)	set

One line. Everything else identical.

Why `arguments` is load-bearing

gtag.js replaces dataLayer.push with its own processor. That queue carries two different kinds of thing, and the processor has to tell them apart:

commands — gtag('config', 'G-…'), gtag('consent', 'update', {…})
data-layer objects — ordinary values other tags push in for their own use

It tells them apart by checking whether the pushed value is a real arguments object. A rest parameter gives you a plain Array instead, which fails that check — so the value gets filed away as data and no command ever runs.

That includes config. Without config, the tag is initialised with no measurement ID, so it has nothing to send and no reason to complain. Everything downstream still looks alive because everything downstream is alive — the script really did load, the queue really is hooked, the commands really are in it. They're just sitting there.

I didn't get this from the docs. It's just the only difference between the two rows of that table. It's also a decent rule of thumb: when a vendor snippet uses an unfashionable function(){} where a modern one would do, assume the old form is doing something.

Lint rules encode what's usually true. prefer-rest-params is right nearly every time, and this is the case where "nearly" costs you all your data.

The fix

Keep Google's original stub verbatim, and leave a note loud enough to survive the next cleanup:

/**
 * **`arguments` here is load-bearing — do not "modernize" it into a rest
 * parameter.** gtag.js tells a command apart from a data-layer object by
 * checking the pushed value is a real `arguments` object. A plain array is
 * silently ignored: the tag loads, the queue fills, and not one hit is sent.
 */
export const gtag: GtagFn = function () {
  window.dataLayer = window.dataLayer ?? [];
  // eslint-disable-next-line prefer-rest-params -- a rest array is not a command
  window.dataLayer.push(arguments);
};

Mirror file: apps/web/src/lib/analytics.ts

A comment alone wasn't enough for me here, because the failure is invisible and the "cleanup" that reintroduces it is a one-character edit. So the test asserts the exact property that matters:

it('queues each command as an `arguments` object, which is what gtag.js recognizes', () => {
  vi.stubGlobal('window', {} as Window);
  gtag('config', 'G-TEST123');
  const queue = window.dataLayer ?? [];
  expect(Object.prototype.toString.call(queue[0])).toBe('[object Arguments]');
  expect(Array.from(queue[0] as IArguments)).toEqual(['config', 'G-TEST123']);
});

Rewrite the stub as a rest parameter and that test fails immediately with [object Array].

Verified end to end against a real vite build: accepting the consent banner produces a gcs=G101 hit and sets _ga.

Bug 3 — the site that worked was reporting to nobody

While isolating bug 2 I went back to the docs site as a control, because it had looked healthy from day one. Its snippet was hand-written directly into docusaurus.config.ts and never shared code with apps/web, so it had the correct arguments form all along.

It was sending hits. It was sending them to the wrong property.

Its measurement ID comes from a GitHub Actions secret that was never updated when the property changed, so every visit it recorded had been landing somewhere nobody was looking. No code change — update the secret, re-run the workflow — but it made the point for a third time in one week.

Working and reporting where you're watching are two different states, and only one of them shows up on a dashboard.

Before and after

	before	after
GA4 tag in the production bundle	absent	present
Sentry browser SDK in the production bundle	absent	present
gtag commands recognized by gtag.js	0 of 3	3 of 3
hits reaching `/g/collect`	0	`page_view`, `scroll`, consent hits
docs-site hits landing in the watched property	0	all of them
checks that had reported a problem	0	1 failing unit test if it regresses

That last row is the one I'd frame. Before the fix, the total number of automated signals warning me about any of this was zero.

What I'm carrying to the next codebase

A repro has to copy the product's code path, not your memory of it. I re-typed Google's snippet from the docs, which silently replaced the variable under test, and that one shortcut is the entire reason I wrote a bug report against Google. Paste the real function in and the two-page test finds it in ten minutes.

Testing a second account felt like ruling something out. It ruled out nothing. Both accounts ran the same broken client code, so the experiment could only ever return the same answer. A control group that shares the defect isn't a control group — and the confidence it gave me is what pushed me toward blaming Google instead of looking closer.

Assert the last link, not the layers. I had a check for the env var, a check for the bundle, a check for the banner, and Google's own check for the tag. Four green lights, zero data. Every one of them verified a precondition for a hit; none verified a hit.

When one config gap swallows a variable, check every variable of the same shape. SENTRY_WEB_DSN was found only because it looked like GA4_MEASUREMENT_ID, and it was the more dangerous of the two — nobody files a ticket saying "I'm getting suspiciously few errors."

If you've got analytics or error monitoring wired into a monorepo build, two things are worth ten minutes today: grep your deployed bundle for the tag host, and check the type of what your gtag stub actually pushes.

What did yours turn out to be — an Array, or Arguments?