DEV Community: vigneshwar

The 90-day window every medical shop misses: I built ShelfMitra to catch it, in Tamil, Hindi and English

vigneshwar — Fri, 16 Oct 2026 15:59:17 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Walk into any medical shop in Chennai and look behind the counter. There are a couple of thousand strips, bottles and inhalers, bought from two or three distributors, each with a batch number and an expiry date printed somewhere tiny.

Here is the part most people outside the trade don't know. Distributors will usually take expiring stock back, but only until a cutoff. On the invoice it says something like "Expiry returns accepted up to 90 days before expiry against original invoice." Miss that window by a week and the strip is yours to throw away.

Nobody misses it on purpose. The owner is billing customers from 8 AM to 10 PM. Checking 2,000 expiry dates against how fast each medicine actually sells is nobody's job, so the cutoff passes quietly and the money goes in the bin.

ShelfMitra does that check for every batch, every day:

You photograph the distributor invoice. An open-weight vision model reads the table. Plain code then checks every line before anything is saved.
It predicts which batches will still be sitting on the shelf when the return window closes, using TabPFN-3.5 on the shop's own sales, and sorts the shelf by rupees at risk.
A durable Temporal workflow watches each batch and reminds the owner at 60, 30, 14, 7 and 2 days before the cutoff, but only if that batch is at risk.
One tap writes the return request to the distributor as a WhatsApp message, in Tamil, Hindi or English.
The owner can just ask. "இந்த வாரம் எதை திருப்பி அனுப்பணும்?" (what should I send back this week?) or "Brufen 100 aur mangwaun?" goes to an agent that checks the numbers before it answers.

Who pays: the owner of a single medical shop, the kind with one billing PC and no IT person. I'd charge ₹299 a month. The pitch is one sentence: if it saves one box of inhalers from the bin, it has paid for itself.

Demo


Live app	shelfmitra.onrender.com
Website	shelfmitra.vercel.app
Code	github.com/vignesh2027/shelfmitra
60 s video	shelfmitra-demo.mp4
Eval	eval/results.md

No signup needed. You land in a demo shop, Lakshmi Medicals, Mylapore, and everything in it is live: the open models, the invoice reader and a Temporal watch on every batch. Create a free shop to get a private, empty one. It runs on Render's free instance, so if nobody has visited for a while the first load takes about a minute while it wakes up.

One honest note about that live demo: TabPFN-3.5 peaks at about 2.3 GB of RAM while scoring on CPU, and the free instance has 512 MB. So the public demo scores with the straight-line rule, and the header says Rule only instead of pretending. Every TabPFN screenshot below comes from the full app on my machine, and the benchmark further down shows exactly what TabPFN adds over that rule.

Here is the full app with TabPFN running. The shelf is sorted by money at risk rather than by date. The Asthalin inhaler is only 12 units, but it has 19 days left to return and an 86% chance of still being there, so it sits at the top:

Reading an invoice photo. All 8 lines pass the checks, and brand names are matched to their ingredient (Montair LC becomes montelukast + levocetirizine):

The whole app switches to Tamil or Hindi, including the agent and the message to the distributor:

Code

vignesh2027 / shelfmitra

Catch medicines that will expire unsold while the distributor still takes them back. Tamil, Hindi, English. Open-weight models, TabPFN-3.5, Temporal.

ShelfMitra

Sell it before it expires.
Expiry rescue for Indian medical shops, in Tamil, Hindi and English.

Live app · Website · 60 s demo video · Run it · How it works · Eval · Crash demo · Limits

The problem

A neighbourhood medical shop buys from two or three distributors. Most distributors take expired or near-expiry stock back, but only until a cutoff, usually 60 to 90 days before the printed expiry. After that the strip is the shop's loss. Nobody has time to check the expiry of 2,000 batches against how fast each one sells, so the cutoff passes quietly and the medicine goes in the bin.

ShelfMitra does that check for every batch, every day, and tells the owner which ones to send back while the distributor will still take them.

What it does

Reads the distributor invoice from a photo. An open-weight vision model (Qwen)…

View on GitHub

MIT. Python, FastAPI, vanilla JS. Three commands to run it locally, 9 tests (Temporal time-skipping tests for the batch watch, plus account isolation tests).

How I Built It

The one rule: the model reads, code decides

Every number the owner sees is a number they might act on with real money, so I split the work strictly:

Job	Who does it
Copy the invoice table from a photo	Qwen (open-weight vision model)
Is `D2241` a plausible batch? Is `03/27` a real expiry? Does qty x rate = amount?	Python
Will this batch still be on the shelf at the cutoff?	TabPFN-3.5
Return now, stop reordering, watch, or fine?	Python, from TabPFN's risk and the days left
Write a polite return request in Tamil	gpt-oss-120b writes a template once; Python fills every number
Understand "Brufen 100 aur mangwaun?" and pick a tool	gpt-oss-120b

Any invoice line that fails a check is flagged for the pharmacist instead of guessed. An expiry of 03/27 becomes 31 March 2027, because medicines expire at the end of the printed month, and that is code, not a model's opinion.

Does TabPFN actually beat a simple rule?

Before I trusted it, I built a benchmark. There is no public Indian pharmacy sales data that I could use, so I took a public dataset of six years of real daily sales from one pharmacy across eight drug classes (Pharma sales data on Kaggle, CC BY-NC 4.0). I dropped simulated batches into that history, each with a random quantity and a random number of days to its cutoff, and asked one question: will more than 10% of this batch still be unsold at the cutoff?

I trained on 2014 to 2017 and tested on 2018 to 2019, with five seeds per size.

Method	Training batches	ROC-AUC	F1
Straight-line rule (sales rate x days left)	none	0.903	0.849
Logistic regression	500	0.908	0.863
Gradient boosting	500	0.922	0.858
TabPFN-3.5	50	0.895	0.833
TabPFN-3.5	100	0.909	0.849
TabPFN-3.5	200	0.920	0.858
TabPFN-3.5	500	0.949	0.882

This surprised me. The dumb rule is really good, 0.903, and with 500 examples logistic regression barely beats it. TabPFN passes the rule at about 100 examples and is clearly ahead at 500. So ShelfMitra uses TabPFN when it can and falls back to the rule when it can't, and the header tells you which one is running. I'd rather show that than pretend the rule doesn't exist.

TabPFN also does a second job. When the owner asks about an order, the agent builds a hypothetical batch and scores it. That turned up a bug I'll come to in a minute.

Temporal: I killed the worker, and nothing was lost

A batch can sit on the shelf for a year. The shop PC will restart many times in that year. A cron job or a setTimeout would quietly forget, and the whole point of the product is to not forget.

So each batch gets its own Temporal workflow. It sleeps on durable timers, wakes at fixed checkpoints, re-scores the batch with fresh sales, and reminds the owner only if the batch is at risk. It ends early as soon as the owner taps Returned or Sold.

To prove it, I wrote a crash demo. A shop "day" lasts one second, so a 70-day window plays out in about a minute. It kills the worker with SIGKILL right after the first checkpoint and stays down while the next checkpoint falls due:

The Temporal UI shows two workers on that one workflow, the one I killed and its replacement. The overdue checkpoint ran as soon as a worker came back. Activities retry with backoff and no attempt limit, so a flaky model call or a dropped connection delays a reminder instead of losing it.

In the running app it looks like this. The demo shop's Dolo 650 batch has 413 days to its cutoff, so its watch is a single durable timer that will sleep for 11 months and 19 days before the first check:

Accounts, so it's a product and not a demo

Visitors share the demo shop. Signing up creates a private shop with its own SQLite database. Passwords use PBKDF2, sessions are signed cookies, every query is scoped to the shop in the session, and the endpoints that spend model credits are rate-limited. A test checks that shop B can't see shop A's stock. If you already use billing software, you can export sales as CSV and import them. The importer matches columns loosely, because every billing package names them differently.

Things I got wrong

The Tamil message came back cut off mid-sentence. gpt-oss reasons before it answers, and on Tamil it spent most of my token budget thinking. The API said finish_reason: "length" and my code didn't check. Now a truncated reply counts as a failure and is retried with more room, and the finished template is cached so it happens once per language, not per message.
Every model call took 76 seconds. I blamed Groq. It was my network: the HTTP client tried IPv6 first, waited for it to time out, then fell back to IPv4. Forcing IPv4 took calls to 1 to 3 seconds.
The agent answered a Tamil question in English. My tool returned English labels like return it now, and the model copied them. Now the tool returns the action already in the owner's language.
The agent showed a shop owner stop_reorder and 0.862 inside a markdown table, on a phone. Tool results now arrive as percents, rupees and DD/MM/YYYY dates, so there is nothing to convert and nothing to format.
Order advice ignored the stock already on the shelf. "Should I order 100 more Brufen 400?" came back as 22% risk, while 45 strips were already sitting there unsold. Old stock sells first, so the new batch has to be judged on old plus new. Fixed, it's now 44% for 100, and the agent checks 50 instead (20%) before answering.
Hinglish questions got Tamil answers. I had left the reply language to the model. Now plain code decides it from the script and a small list of Hinglish and Tanglish words.
The small local model copied my prompt into its answer. My invoice prompt described each field, like "expiry": "MM/YY as printed". The 3B vision model returned exactly that: every expiry came back as 03/27 as printed, and the return policy came back as my own description of it. The code checks rejected all 8 lines, so nothing wrong reached the shelf. That is the whole reason the checks exist. I rewrote the prompt so it describes the fields in a list instead of showing them as values, and the same model now gets every expiry right.

Why Does Open Innovation Matter?

The data is the business. A shop's purchase invoices show what it pays each distributor, its margins and what sells in that neighbourhood. Its sales history is close to a record of what its customers take. Owners won't upload that to whatever API is cheapest this quarter, and they are right not to. With open weights, the same models can run on the shop's own PC: set SHELFMITRA_LLM=local and ShelfMitra uses Qwen through Ollama instead of a hosted endpoint. TabPFN-3.5 already runs locally on CPU.

I tested that on my own laptop, an 8 GB M1 Mac, with nothing hosted:

Same sample invoice	Hosted (Qwen 3.8 27B on Groq)	Local (qwen2.5vl:3b on Ollama)
Time	3.5 s	104 s
Expiry dates right	8 of 8	8 of 8
Batch numbers right	8 of 8	7 of 8 (`CTZ7712` read as `CTZ77712`)
Return policy read	yes	yes

So local works, but it is slower, and a 3B model makes the kind of mistake that no format check can catch: CTZ77712 is a perfectly valid-looking batch number. That's why every invoice is shown to the pharmacist as a table before it goes onto the shelf. The local agent on qwen3:4b was too slow on 8 GB to be pleasant, more than ten minutes for three questions, so for now the realistic setup is local invoice reading overnight and hosted open models for chat. A shop PC with 16 GB would change that.

The unit economics work. I measured real token counts on the hosted open models:

Task	Model	Tokens in / out	Cost
Read one invoice photo	Qwen 3.8 27B (vision)	1,951 / 984	about $0.0055
Answer one question	gpt-oss-120b	about 2,300 / 200 over 2 to 3 calls	about $0.0005
Score a batch	TabPFN-3.5 on CPU	n/a	free

A busy shop that photographs an invoice every day and asks 100 questions a month costs about $0.22 a month in model calls, around ₹20. At ₹299 a month that leaves room for hosting and support. On a closed frontier API, the per-shop cost could easily be several times higher, and I'd be one price change away from the business not working.

Tamil and Hindi are first-class, not an afterthought. The owner should be able to ask the way they talk, including Tanglish and Hinglish typed in English letters.

Where a closed model would win, honestly: blurry, handwritten or crumpled invoices. A big closed vision model would probably read those better than a 3B model on a shop PC. That's why every line goes through code checks and anything doubtful goes to the pharmacist rather than straight onto the shelf.

My Agent Session

I built ShelfMitra with an AI coding agent. I picked the problem, the name and the rules: the model reads and code decides, no discount advice on prescription medicines, and honest numbers in the post. The agent wrote most of the code and the eval harness, and I reviewed and steered it. The bugs above came from actually running it, mostly from screenshots that looked wrong.

Prize Categories

Best Use of TabPFN. TabPFN-3.5 is the risk engine. It scores every batch on the shelf, and it scores hypothetical orders for the agent's what_if_order tool. It is benchmarked against a straight-line rule, logistic regression and gradient boosting on real pharmacy sales with a time-based split (table above), and the app falls back to the rule, visibly, when TabPFN isn't available.

Best Use of Temporal. One durable BatchWatch workflow per batch, with checkpoint timers that span months, signals for the owner's decisions, queries for live status in the dashboard, and retry policies that never give up. It survived a SIGKILL mid-watch with every checkpoint intact (crash demo above). It also has time-skipping tests.

Best Use of Render. The live app at shelfmitra.onrender.com is one Docker service on Render running three things together: the FastAPI app, a Temporal dev server and the Temporal worker. Every visitor's batches get real durable workflows on Render, which you can see in the dashboard's Watching every batch panel:

The public demo is on the free instance, so it uses the rule engine and keeps its state on the instance's temporary disk, which resets when it restarts. The repo also has a render.yaml blueprint for the paid setup a real shop would use: a standard instance with enough RAM for TabPFN-3.5, and a persistent disk for each shop's database and Temporal's state, so reminders survive deploys.

12 of 13 AI models knew the new name and still wrote the old one

vigneshwar — Sat, 10 Oct 2026 15:38:25 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

TL;DR. I tested 17 AI models on writing, recalling and fixing OpenTelemetry instrumentation, graded in code against 26 releases of the spec.

16 of 17 models wrote retired names, and 83% of those names had been retired for more than a year.

12 of 13 models wrote names they had correctly replaced when asked directly.

7 of 17 "fixed" a metric by changing its unit and keeping its retired name, which breaks it both ways.

One sentence naming the spec version cut retired names by 36%, and made models invent more.

Benchmark: Semconv Drift on Kaggle

I asked Claude Sonnet 5 what replaced the OpenTelemetry attribute db.system. It answered correctly: db.system.name, the name the spec has used since January 2025.

In the same benchmark, I asked it to add tracing to a go-redis call. It wrote this:

attribute.String("db.system", "redis"),
attribute.String("db.operation", cmdName),
attribute.String("db.statement", statement),

All three names are retired. The code compiles, the span exports, the backend stores it. Nothing fails. Weeks later, a dashboard that filters on db.system.name shows no Redis traffic, and nobody connects the empty panel to a line of generated code.

This year I have had 14 pull requests merged into OpenTelemetry's Go compile-time instrumentation project, and this is exactly the kind of bug I would wave through in review. The code looks right. It is only wrong against a spec version, and nobody reviews against a spec version. So I built a benchmark to measure how often it happens.

The task

OpenTelemetry's semantic conventions are the shared vocabulary of telemetry: which key holds the HTTP method, what a database span calls its query, which unit a duration metric uses. They are versioned, and many of the names people learned first have been renamed. http.method became http.request.method in 2023. db.statement became db.query.text in 2024. rpc.client.duration, in milliseconds, became rpc.client.call.duration, in seconds, in January 2026.

Semconv Drift has four tasks on Kaggle Benchmarks:

Task	Cases	What the model does
Write	12	Instrument everyday code: Flask and Express servers, an HTTP client, psycopg, go-redis, kafkajs, gRPC, metrics and resources, in Python, Go and TypeScript
Know	22	Name what replaced a retired name. Includes span-kind traps and two controls whose right answer is "unchanged"
Fix	8	Modernize old snippets, including real code from redis/go-redis, converting values where a unit changed
Write with version hint	12	The Write requests plus one sentence naming the current spec version

Every Write prompt already says "Follow the current OpenTelemetry semantic conventions." The models were told. The question is what "current" means to them.

Grading is code, not a judge. The grader extracts every semantic convention name from an answer (string keys, metric names, SDK constants in four languages, enum values) and looks each one up in 1,563 names from 26 releases of the spec, v1.21.0 to v1.44.0, plus every name from v1.0.0 to v1.20.0 for the oldest ones. Every retired name carries the release and date that retired it. Reasoning text is stripped first, so a model is judged on the code it wrote. The answer key is tested too: the build fails if any gold answer is not a current name in the ground truth.

The ground truth comes from Attrition, a dataset of every tagged release of open-telemetry/semantic-conventions that I built for DEV's Sanity challenge.

Models Tested

17 models through Kaggle's model proxy, picked to answer three questions:

Does a newer generation fix it? Three same-family pairs: Claude Sonnet 4.5 and 5, GPT-5.4 and 5.5, Gemini 2.5 Pro and 3.1 Pro.
Does spending more fix it? Frontier models (GPT-6 Astra, Claude Opus 5, GPT-5.5, Gemini 3.1 Pro) next to small, cheap ones (GPT-5.4 mini, Claude Haiku 4.5, Gemini 3.7 and 3.8 Flash).
Do open-weight models differ? Qwen 3 235B, DeepSeek-R1, GLM-5, Gemma 4 31B and gpt-oss-120b.

That is 830 graded answers, plus 84 more from repeat runs. Four reasoning models (DeepSeek-R1, GLM-5, Gemma 4 and gpt-oss-120b) have no Know score: on some short factual questions they spent the whole answer budget thinking and never answered. Grok 4.5 and 4.6 returned 404 for my account, so they are not in the lineup.

Findings

1. The drift is old, not new

16 of 17 models wrote at least one retired name in Write, 357 uses in all. If this were about training cutoffs, the retired names would be recent ones the models never saw. They are not:

297 of 357 (83%) had been retired for more than a year when I ran the benchmark.
238 (67%) had been retired for more than two years, which means before October 2024. Every model in the lineup was released after that.
The median retired name was retired 2.4 years earlier.

These models were not missing the news. They had read the changelog, and years of tutorials before it, and the tutorials won. db.system is the clearest case: 24 uses across 12 models, nearly two years after the rename.

The only names a training cutoff could excuse are the January 2026 rpc.* renames. 16 models wrote them. GPT-6 Astra was the only exception, and it also avoided every older retired name, so a late cutoff alone does not explain its perfect Write score.

2. It told me the new name, then wrote the old one

Because Know and Write ask about the same names, I can check each model against itself. For every retired name a model wrote in Write, did the same model give the correct replacement in Know?

12 of 13 models wrote at least one retired name they had correctly replaced when asked directly, 58 times in all. Claude Haiku 4.5 said http.method became http.request.method, then wrote http.method five times. Sonnet 5 did it with db.system, as in the opening. Only GPT-6 Astra never did it.

This is not a knowledge gap. Asked a question, a model retrieves the changelog. Asked for code, it retrieves the most common code, and the most common code is old.

3. The half fix

One Fix case hands the model this Go snippet and asks it to move to v1.44.0, converting values if a unit changed:

dur, _ := meter.Float64Histogram("rpc.client.duration", metric.WithUnit("ms"))
start := time.Now()
err := invoker(ctx, method, req, reply, cc, opts...)
dur.Record(ctx, float64(time.Since(start).Milliseconds()))

The answer is rpc.client.call.duration, unit s, values in seconds. Only GPT-6 Astra got all of it.

7 of 17 models changed the unit to seconds, converted the values, and kept the retired name. That is worse than leaving the code alone. Unchanged code is at least consistent with the old spec. The half-fixed metric emits the old name with the new unit, so it breaks in both directions: queries on the old name read values 1,000 times smaller than expected, and queries on the new name find nothing. The diff looks like a fix, so it passes review.

Two more models, from two different vendors, did something stranger. GPT-5.5 and Gemini 3.7 Flash both renamed the metric to rpc.client.request.duration. No release of the spec has ever defined that name. It is a plausible guess built from the real http.client.request.duration, and both models made the same guess.

4. Why this domain drifts when others do not

Another entry in this challenge tested removed pandas and Express APIs and found that models used the current ones. I think the difference is how each change fails.

A removed function fails loudly. Call DataFrame.append on pandas 2 and you get an error, a Stack Overflow question and an answer, and the answer ends up in training data. A retired attribute name fails silently. Nothing throws, nobody asks, and the old tutorials are never corrected. The OpenTelemetry project knows this: its instrumentation libraries offer OTEL_SEMCONV_STABILITY_OPT_IN=http/dup, which emits both the old and the new HTTP names during a migration, because a rename does not propagate on its own.

That is a hypothesis, not a measurement. But it predicts where to look for drift: anywhere the old form still works.

5. One sentence of context helps, and it is not enough

The hint task repeats the 12 Write requests with one added sentence: "Target OpenTelemetry semantic conventions v1.44.0 (released August 2026). Many attribute and metric names were renamed or removed between 2023 and 2026, so do not rely on older names."

Before trusting any difference, I needed a noise floor. Kaggle's benchmark library accepts a temperature but drops it unless a model is flagged as supporting it, so the same prompt can give a different answer twice. I ran the plain Write task a second time on seven models. Their retired-name total moved from 156 to 166, about 6%, with nothing changed. The hint moved the same seven models from 166 to 94, a 43% drop.

Across all 17 models:

Retired names fell by 36%, from 357 to 230. The mean Write score rose from 39% to 48%.
Only one model reached zero, and it was the one that was already there.
Invented names rose from 21 to 31. Told that names had changed, some models stopped reaching for old names and started guessing new ones. Claude Haiku 4.5 went from 1 invented name to 7.

Another entry in this challenge found that a one-sentence rule took clean answers from 64% to 96%. Here one sentence removed about a third of the drift and replaced part of it with guesses. A version number tells a model that names changed. It does not tell it what they changed to.

6. The leaderboard

Overall is the average of a model's task scores, the same rule the Kaggle leaderboard uses.

Model	Write	Know	Fix	Write + hint	Overall	Retired names written
GPT-6 Astra	100%	100%	100%	92%	98%	0
Claude Opus 5	67%	95%	88%	75%	81%	6
Claude Sonnet 5	67%	82%	75%	83%	77%	9
GPT-5.4	67%	77%	75%	83%	76%	8
GPT-5.5	67%	91%	75%	67%	75%	5
Gemini 3.7 Flash	58%	91%	75%	58%	71%	9
Gemini 3.8 Flash	58%	91%	75%	58%	71%	8
Claude Sonnet 4.5	42%	68%	50%	50%	52%	21
Gemini 3.1 Pro	25%	82%	63%	33%	51%	17
GPT-5.4 mini	33%	64%	38%	58%	48%	19
GLM-5	17%	n/a	75%	42%	44%	17
Gemini 2.5 Pro	17%	50%	13%	25%	26%	27
Gemma 4 31B	33%	n/a	13%	25%	24%	24
Claude Haiku 4.5	0%	45%	13%	25%	21%	40
Qwen 3 235B	0%	23%	13%	25%	15%	49
DeepSeek-R1	8%	n/a	0%	8%	6%	46
gpt-oss-120b	0%	n/a	0%	0%	0%	52

Two patterns are weaker than the findings above. Two of the three family pairs improved clearly (Sonnet 4.5 to 5: 52% to 77%; Gemini 2.5 Pro to 3.1 Pro: 26% to 51%), and the GPT pair tied (GPT-5.4 76%, GPT-5.5 75%). Price was a poor guide: GPT-5.4 matched GPT-5.5 for under a tenth of the cost ($0.10 against $1.36).

What surprised me

I expected a cutoff story: new renames missed, old ones known. I got the opposite. The models know the renames, and they still write the names from the tutorials they read most. The half fix surprised me more. I wrote that case to catch models that forget the unit. Almost none forgot the unit. They forgot the name.

What this changed about how I review AI code

I used to treat "the model knows the current API" as the thing to check. It is the wrong check. Asking a model what changed tells you nothing about what it will write. For telemetry code, I now check the names against the spec, not against my memory, and I treat a diff that changes a unit but not a name as a bug.

What this means for you

Lint telemetry names in CI. The spec's registry marks every deprecated name and its replacement. A check of string literals against that list would flag every retired name counted in this post, db.system included.
Put the spec version in the prompt, then still lint. It cut drift by a third here, and it also made models invent names. If you can, paste the current names you need instead of the version number.
Prefer generated constants pinned to a version. In Go the semconv version is in the import path. Bump it on purpose.
Review units and names together. A histogram whose unit changed and whose name did not is the hardest drift to spot, because the diff looks like a fix.

Try it on the model you use at work

Paste this prompt:

Language: Python.
Wrap this psycopg query in an OpenTelemetry client span. Record which database system it is, the database name, the operation, the table and the query text.

def list_orders(conn, customer_id):
    with conn.cursor() as cur:
        cur.execute('SELECT * FROM orders WHERE customer_id = %s', (customer_id,))
        return cur.fetchall()

Follow the current OpenTelemetry semantic conventions. Write attribute keys and metric names as string literals rather than SDK constants. Return one complete code block and nothing else.

Save the answer as answer.py and run:

grep -nE "[\"'](db\.system|db\.name|db\.operation|db\.statement|db\.sql\.table|net\.peer\.name)[\"']" answer.py

Any hit is a retired name. The current ones are db.system.name, db.namespace, db.operation.name, db.query.text, db.collection.name and server.address. Tell me in the comments which model you used and what it wrote.

What I would measure next

More repeats. Seven models ran Write twice. Every model should run every task several times, with a reported spread.
Names instead of a version. Does pasting the current names into the prompt remove the drift and the guessing, or only the drift?
Units beyond time. The grader checks ms against s only. Byte and count units can drift too.
The GenAI conventions. They moved to their own repository and are changing fast, which makes them the next place to look.

Limitations

Small case counts. Write has 12 cases, Know 22 and Fix 8. Claude Sonnet 5's 67% on Write has a 95% Wilson interval of 39% to 86%. The repeat runs flipped up to 3 of 12 Write cases for one model, so I would not claim any order in the middle of the table.
Know is missing for four reasoning models, which ran out of answer budget while thinking.
Grok is missing. Grok 4.5 and 4.6 returned 404 through Kaggle's model proxy.
One strict call. Claude Opus 5 lost one Write case only for naming a custom key app.ledger.entry_count. The spec has owned the app.* namespace since its mobile conventions, so the grader counts it as invented, and the rule applies to every model the same way.
A moving target. Each semconv release retires more names. Re-running after the next release needs only a new ground-truth export, which is also what keeps this benchmark from going stale itself.

My Benchmark

Kaggle benchmark: https://www.kaggle.com/benchmarks/applesone/semconv-drift

The four tasks, each with its own leaderboard and the full task code:

Task	Link
Write	semconv-drift-write
Know	semconv-drift-know
Fix	semconv-drift-fix
Write with version hint	semconv-drift-write-with-version-hint

Interactive results: a sortable leaderboard with confidence intervals and every chart, at https://vignesh2027.github.io/semconv-drift/

Code, grader and ground truth:

vignesh2027 / semconv-drift

Do AI models write OpenTelemetry code with names the spec already retired? A Kaggle benchmark graded in code against 1,563 names from 26 spec releases.

Semconv Drift

Do AI models write OpenTelemetry code with names the spec already retired?

OpenTelemetry's semantic conventions name everything telemetry records: http.request.method, db.system.name, http.server.request.duration. Those names change between releases. A model trained on older tutorials keeps writing the old ones, and nothing fails loudly: the data still flows, while dashboards and alerts quietly stop matching.

Website · Kaggle benchmark · Tasks: Write · Know · Fix · Write with version hint · Results

Semconv Drift measures that on Kaggle Benchmarks, with four tasks and no AI judge. Every answer is graded in code against 1,563 attribute, metric and event names from 26 releases of the specification (v1.21.0 to v1.44.0).

Task	Cases	What it asks	What counts as correct
Write	12	Instrument everyday code (HTTP server and client, Postgres, Redis, Kafka, gRPC, metrics, resources) in Python, Go and TypeScript	Only current names, no invented names under a spec

…

View on GitHub

analyze.py and charts.py regenerate every number and chart in this post from the Kaggle runs. If you maintain OpenTelemetry instrumentation and want a case added, open an issue on the repo.

I contribute to OpenTelemetry and still shipped two retired attribute names, so I built Attrition

vigneshwar — Sat, 03 Oct 2026 17:04:46 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Last month my entry won the DEV Summer Bug Smash. It was a Go demo that traced Redis commands into Sentry. This week I ran a new tool against that same file, and it flagged two names that OpenTelemetry had already retired:

attribute.String("db.system", "redis"),       // renamed to db.system.name in v1.30.0
attribute.String("db.operation", cmd.Name()), // renamed to db.operation.name in v1.26.0

I contribute to OpenTelemetry, and I still shipped them. That is the problem Attrition solves.

The problem

OpenTelemetry's semantic conventions define the names that telemetry uses: http.request.method, db.system.name, http.server.request.duration. These names change between releases. Old tutorials, SDK constants and copied snippets keep the retired ones alive, and nothing breaks loudly when you use them. Instead, dashboards and alerts quietly stop matching, and teams find out weeks later.

Finding these names sounds like a search problem. It is not. The correct answer often depends on context that a keyword search cannot see:

Name in your code	Why a search gets it wrong	What Attrition answers
`net.peer.name`	There are two correct replacements	`server.address` on a client span, `client.address` on a server span, decided from your code
`http.server.duration`	It vanished from the registry with no deprecation entry	`http.server.request.duration`, with the unit changing from ms to s
`db.client.connections.wait_time`	It looks like a simple rename	The unit also changed from ms to s, so a rename alone records wrong values
`cloud.platform = "azure_vm"`	The key itself is current	The value is retired; use `"azure.vm"`
`db.name`	The registry and the migration guide disagree	Both views are shown, and the change is left for a person to decide

What Attrition does

Attrition reads instrumentation code and checks every attribute key, metric name, event name and enum value against 1,563 names from 26 releases of the specification. For each retired name it reports:

a verdict read from structured data, never guessed by a model
the replacement that fits the span kind your code uses
the release that retired it, and whether your pinned version already had it retired
any unit change, when a renamed metric also changed its unit
what the official migration guide adds, read from a Sanity Knowledge Base
a link to the exact line of the specification that says so
a patch, limited to the renames that are safe to automate

Demo

Live app, no login required: https://attrition-otel.vercel.app

Try it in two minutes

Open the app and click Checkout service. One TypeScript file acts as both an HTTP server and an HTTP client. Notice that net.peer.name receives a different answer on each span, and that two HTTP metrics are flagged even though the registry no longer mentions them.
Click Orders and an LLM call. This Python sample shows a metric whose unit changed, a retired enum value, a chat event that became a span attribute, and a case where the registry and the migration guide disagree.
Scroll to How this answer was built under any result. It lists every Sanity Context call, the query it sent, the time it took and the number of documents returned.
Ask the chat agent: "We still record http.server.duration in milliseconds. What should we use now?"
Open the Decisions page to see the conflicts the Knowledge Base raised and how each one was resolved.

The chat agent answers from the same two sources and shows its evidence under every reply:

Project page: https://vignesh2027.github.io/attrition

Code

vignesh2027 / attrition

Find the OpenTelemetry attribute, metric and event names the semantic conventions already retired. Web app, Go CLI, GitHub Action, Rust offline scanner and Python agent on Sanity Context.

Attrition

Find the OpenTelemetry names the spec already retired.

Paste instrumentation code. Attrition finds every attribute key, metric name, event name, enum value and SDK constant in it, checks each one against 1,563 names from 26 releases of the OpenTelemetry semantic conventions, and tells you what to use instead: for your span kind, with the release that retired it, the exact spec line that says so, and what the official migration guide adds. It writes a patch only for the renames that are safe to automate and leaves the rest for a person.

Live app	https://attrition-otel.vercel.app (no login)
Knowledge Base decisions	https://attrition-otel.vercel.app/decisions
Project page	https://vignesh2027.github.io/attrition
Sanity project ID	`y9raau23`, dataset `production` (public)
Public dataset query	every retired name as JSON
Sanity Studio	https://attrition.sanity.studio
Context MCP, dataset	`https://api.sanity.io/v1/context/organizations/oc2g3x7ee/mcp/attrition`
Context MCP, Knowledge Base	`https://api.sanity.io/v1/context/organizations/oc2g3x7ee/mcp/attrition-kb`

Why this needs structured content

The semantic conventions rename things between releases, and old blog posts, SDK…

View on GitHub

One dataset serves four clients, each built for a different job:

Client	Language	Best for
Web app	TypeScript, Next.js	Pasting a file, reading the evidence, asking questions
`cli/`	Go	Scanning whole repositories and gating CI; `-fix` applies only safe renames
`rust/`	Rust	Offline scanning for pre-commit hooks and air-gapped CI
`python/`	Python	A terminal agent built on the official Python MCP SDK
`action.yml`	GitHub Action	Failing a pull request that introduces a retired name

The Rust scanner is a port of the web extractor, and a parity script confirms that both produce identical counts on five real repositories. CI builds and tests all three languages and runs the Action against the repository's own samples.

How I Used Sanity

The design rests on one rule: structured data decides the verdict, the Knowledge Base explains the change, and the model never decides what replaced what.

1. Modelling the specification as structured content

The specification is published as YAML spread across many files and releases. A deterministic importer reads every tagged release from v1.21.0 to v1.44.0 and writes 1,654 documents to Sanity: 940 attribute, 571 metric and 36 event documents, plus namespaces and release statistics.

The newest release defines each name. The older releases add history that no single release contains: introducedIn, deprecatedIn, and 50 names that disappeared from the registry without any deprecation entry. Those are recorded with the verdict DROPPED and the last release that defined them.

Every retired name receives a verdict computed in code from the specification's own reason, renamed_to and note fields:

Verdict	Attributes	Metrics	Events	Example
`RENAMED`	97	71	1	`http.method` becomes `http.request.method`
`REPLACED`	15	5	5	`rpc.client.duration` becomes `rpc.client.call.duration`, ms to s
`MOVED_OUT`	58	11	3	`gen_ai.*` moved to a separate repository in v1.42.0
`DROPPED`	16	30	4	`http.server.duration`, last defined in v1.21.0
`REMOVED`	21	10	1	No replacement; delete it
`SPLIT`	4			`http.target` becomes `url.path` plus `url.query`
`MERGED_INTO`	4			`code.function` folds into `code.function.name`
`SPAN_KIND_DEPENDENT`	3			`net.peer.name`, `net.peer.port`
`CONDITIONAL`	2			`db.sql.table`, `http.host`
`USE_SIGNAL_FIELD`	2			`event.name` moves to the log record's EventName field

The structure carries the details that make answers correct. Each replacement records the condition it applies under (always, client spans, server spans, together). Twenty-four enum values carry their own replacementValue. When a renamed metric's replacement uses a different unit, the importer records a unitChange by comparing the two metric documents.

That comparison also uncovered a genuine error in the upstream specification: two deprecated system.linux.memory.* entries carry the description and unit of a network packet counter. Rather than report a false unit change, Attrition stores the finding as a specErratum.

I reviewed every verdict by hand, and no name remains in the NEEDS_REVIEW state.

2. Querying the dataset through Sanity Context

The dataset is served by an organization-level Context MCP endpoint named attrition. Its GROQ filter limits access to the convention types, and its instructions explain what each verdict means and forbid inventing a replacement.

A scan sends a single groq_query covering every name found in the code:

{
  "release": *[_type == "specRelease"][0].tag,
  "attrs": *[_type in ["attribute", "metric", "event"] && key in ["http.method", "net.peer.name", "http.server.duration"]]{
    _id, _type, key, status, lastSeenIn, unit, specErratum, "source": source{url, release},
    "deprecatedMembers": members[defined(deprecated)]{value, deprecated, replacementValue},
    deprecation{verdict, note, deprecatedIn, valueChanges, unitChange{from, to}, movedTo{label, url},
      "replacements": replacements[]{key, when, "stability": coalesce(attribute->stability, metric->stability)}}
  }
}

The code determines the span kind and the data determines the answer. Because no model sits in this path, a scan cannot invent a name.

3. A Knowledge Base for what the registry leaves out

The registry records what a name became. It does not record what else changed, and for dropped names it records nothing at all. That knowledge lives in the official migration guides.

I built the Knowledge Base entirely with the Sanity CLI (sanity context create, context imports create and context build) from 144 sources, within the 150 document budget:

137 documents bound from the dataset: the retired attributes and the HTTP, database, RPC and messaging metrics that most instrumentation uses
5 official migration guides: HTTP, database, RPC, code attributes and version selection
2 older specification pages that still use the retired names

A second organization endpoint, attrition-kb, serves the Knowledge Base. For each retired name, a scan calls knowledge_base_search and then knowledge_base_read on the results, keeping the guide entry that discusses that name. Entry paths are never hard-coded, because each rebuild can rename them.

This is where several of the most valuable answers come from:

http.server.duration maps to http.server.request.duration, with the unit changing from ms to s. The registry has no record of this at all.
db.query.text, the replacement for db.statement, "SHOULD be collected by default only if sanitization excludes sensitive information." A rename alone would never surface that privacy requirement.
Each area's dual-emit setting, such as OTEL_SEMCONV_STABILITY_OPT_IN=database/dup, which lets teams emit old and new names side by side during a migration.

4. Resolving conflicts between sources

The Knowledge Base build raised four genuine conflicts. My dataset stated that http.server.duration and http.resend_count left the registry with no replacement, while the official guides stated that they were renamed.

Both statements are accurate at different levels. The registry stops listing a name once it is gone, while the guide records where it went. I resolved each conflict in favour of the guide using client.context.issues.resolve(), applied the resulting page updates with issues.apply(), and Sanity converted each decision into a standing instruction that every future build follows.

The effect was immediately visible. Before the decisions, asking about http.resend_count returned "no replacement". Afterwards, the rewritten entry correctly returned http.request.resend_count. Every conflict, both claims and the chosen side are published on the Decisions page.

5. Three agents, one source of truth

The scanner is fully deterministic. Extraction happens in code, verdicts come from groq_query, and guidance comes from knowledge_base_search and knowledge_base_read.
The chat agent retrieves its evidence first through the same Context calls, then answers in a single model call restricted to that evidence. It cites the dataset or the exact Knowledge Base entry and lists the calls beneath each answer. This design also keeps it within the limits of a free model tier.
The Python agent takes the conventional route: the model calls groq_query, knowledge_base_search and knowledge_base_read directly through the official Python MCP SDK. It demonstrates that the same endpoints work from any agent framework.

6. Results on real code

I ran the Go CLI against five open-source repositories at pinned commits, and the Rust offline scanner reproduced every count:

Repository	Retired names in code	Files to update
open-telemetry/opentelemetry-demo	6	4
jaegertracing/jaeger	67	13
redis/go-redis	16	4
go-gorm/opentelemetry	0	0
uptrace/opentelemetry-go-extra	18	7

The most significant finding: go-redis's official redisotel package still registers the old db.client.connections.* pool metrics and records create_time and use_time in milliseconds. Renaming them to the new names without converting the values would make every reading a thousand times too large. This is exactly why Attrition never auto-patches a name whose unit changed.

The clean result for go-gorm/opentelemetry matters as much: a reliable tool must say nothing when the code is current. Every finding links to its source line in bench/RESULTS.md.

Design decisions and limitations

Patches are conservative by design. Only unconditional renames of string keys are patched. Span-kind choices, unit changes, merges and disagreements between sources are always left for a person.
A finding is not always a bug. Test fixtures that read historical data, or libraries that intentionally emit both old and new names during a migration, may keep a retired name on purpose.
Next steps: mapping pinned SDK versions to the exact specification release they ship, and adding the remaining migration guides to the Knowledge Base as the 150 document limit allows.

Sanity Project Details

Project ID: y9raau23
Dataset: production (public)
Public dataset query: every retired name as JSON
Sanity Studio: https://attrition.sanity.studio
Knowledge Base: kbI5ncVyqDpc, served by the organization Context MCP endpoint attrition-kb
Dataset Context MCP endpoint: attrition
Knowledge Base decisions: https://attrition-otel.vercel.app/decisions

(https://dev.to/agent_sessions/new)

A HOOK U NEED TO LOOK

vigneshwar — Sun, 16 Aug 2026 10:55:27 +0000

16 Redis commands. 0 spans. A hook that looked perfectly correct. - DEV Community

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry. I spent an...

dev.to

16 Redis commands. 0 spans. A hook that looked perfectly correct.

vigneshwar — Sun, 16 Aug 2026 10:53:08 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

I spent an afternoon reading four lines of Go and agreeing with them. They were wrong.

Not subtly wrong. The function was supposed to attach telemetry to every shard in a Redis ring, and it attached telemetry to none of them. Ring clients ran in production emitting nothing at all, and the code that failed to do it reads like textbook Go.

Here is how it hid, how I found it, and what it looks like in Sentry when you finally put a light on it.

Project Overview

What the project does

opentelemetry-go-compile-instrumentation is OpenTelemetry's compile time instrumentation for Go.

You do not add tracing calls to your code. The toolchain rewrites your binary during the build, so the tracing is already inside it when it ships.

Why a bug here is worse than most

That property is the entire appeal, and it is also what makes a defect in it dangerous.

You did not write the instrumentation. You are not reading the instrumentation. You are reading the dashboard it feeds, and you trust it completely, because trusting it is the point.

The project ships hooks for a lot of common libraries. go-redis gets six of them, one per client type. Each is roughly four lines: something constructs a client, the hook runs afterwards, the hook attaches an OpenTelemetry hook to that client.

The one for redis.Ring did nothing at all.

Bug Fix or Performance Improvement

The four lines

func afterNewRingClientV9(call hook.HookContext, client *redis.Ring) {
    client.OnNewNode(func(rdb *redis.Client) {
        rdb.AddHook(newOtelRedisHook(rdb.Options().Addr))
    })
}

Read that on its own and it is fine.

A ring is a set of shards. OnNewNode registers a callback that runs for each shard. The callback attaches the telemetry hook. Therefore every shard gets instrumented.

I read it three times and agreed with it three times. Then I went and read NewRing.

What actually happens

OnNewNode only fires for shards created after you register the callback.

NewRing builds a shard for every entry in RingOptions.Addrs during construction, and construction finishes before this hook ever runs.

So by the time afterNewRingClientV9 registers its callback, every shard the caller configured already exists. The callback will never see a single one of them.

If you build a ring the normal way, by listing your Redis addresses in the options, you get zero telemetry. Not degraded. Not partial. Zero. Every command, to every shard, invisible.

The only way to get even one instrumented shard was to construct a ring with no addresses and add them later through SetAddrs. Nobody builds a ring that way.

Why this class of bug is the worst class

There is no error. No warning. No partial data that looks suspicious.

Your Redis spans are simply absent, and absence reads as health.

If you opened a service map and saw no Redis calls, your first thought is not "the instrumentation is broken." It is "I guess Redis is fine." A bug that produces wrong numbers eventually gets caught by somebody who notices the numbers are wrong. A bug that produces no numbers can sit there forever.

Why nobody had reported it

The entire package had no test coverage.

Six client hooks, zero tests. Nothing anywhere was asserting that any of them attached anything.

Code

Upstream PR: open-telemetry/opentelemetry-go-compile-instrumentation#1098

The fix

Keep the OnNewNode registration, since it is still correct for shards added later through SetAddrs, and additionally walk the shards that already exist.

 func afterNewRingClientV9(call hook.HookContext, client *redis.Ring) {
    client.OnNewNode(func(rdb *redis.Client) {
        rdb.AddHook(newOtelRedisHook(rdb.Options().Addr))
    })
+   // NewRing builds a shard for every entry in RingOptions.Addrs before this
+   // hook runs, and OnNewNode only fires for shards created after it is
+   // registered. Without this pass the shards a caller configured up front
+   // are never instrumented, which is the common case.
+   attachHookToExistingShards(client)
 }
+
+// attachHookToExistingShards instruments the shards a ring already holds.
+// ForEachShard returns the first error a callback produces; this callback
+// cannot fail, so there is nothing to report.
+func attachHookToExistingShards(client *redis.Ring) {
+   _ = client.ForEachShard(context.Background(), func(_ context.Context, rdb *redis.Client) error {
+       rdb.AddHook(newOtelRedisHook(rdb.Options().Addr))
+       return nil
+   })
+}

Both paths are now covered. Shards present at construction get instrumented by the ForEachShard pass. Shards added later get instrumented by OnNewNode.

On that comment

I left it deliberately, and I would fight to keep it in review.

The next person to read this function will have exactly the reaction I had, which is that OnNewNode looks sufficient. It is not sufficient. The only way to know that is to know when NewRing builds its shards relative to when the hook runs.

That fact belongs in a comment, not in somebody's head.

Testing a Redis hook without a Redis server

This is the part I expected to be hard, because instrumenting a Redis client normally means having a Redis server, and needing a server in CI is how tests end up skipped.

It turns out you do not need one.

Each test builds a client pointed at a port nothing is listening on, then issues a command. The dial fails. But the hook opens its span before handing the command to the next hook in the chain, so a failed connection exercises the instrumentation completely.

Assert that a client span came out carrying the right endpoint and you have tested the thing you actually care about. No server. No container. No fixture.


Test that pins the bug	`TestAfterNewRingClientV9_AttachesHookToEachNode`
Fails on old code	Yes
Other hooks covered	5
Package coverage	89.2% to 96.9%

My Improvements

What changed

Ring clients went from producing no telemetry to producing telemetry for every shard, in the configuration basically everybody uses. Six previously untested hooks now have tests.

Callback registration order is a whole family of bugs

Any API shaped like "register a callback and it will fire for each X" has a question hiding inside it: what about the Xs that already exist?

Some libraries replay for existing items. Some only fire going forward. The method name almost never tells you which.

OnNewNode is arguably honest, the word is right there in the name. But I would bet most people read it as "on each node," because that is what you want it to mean.

Instrumentation needs tests more than normal code, not less

Missing telemetry is harder to notice than wrong telemetry.

Nothing downstream will ever complain on its behalf. A dashboard cannot tell you it is missing a line.

Point it at a dead port

To test a client library hook, aim it somewhere nothing is listening.

The failure path runs through the same instrumentation as the success path, and you skip the entire problem of standing up a real server in CI. I will be reusing this one.

Best Use of Sentry

The problem with proving a negative

This bug has an unusual property. The symptom is absent telemetry.

You cannot screenshot a missing span. So the only honest way to prove it is to put both versions in front of a real tracing backend and show the shape of the hole.

I built a reproduction that ships OpenTelemetry spans straight to Sentry over OTLP and runs an identical workload under both attachment strategies:

github.com/vignesh2027/bugsmash-sentry-demo

Three shard ring. Sixteen Redis commands. One cart-checkout transaction wrapping all of them. The only variable in the entire program is which attachment function runs.

Before: the empty waterfall

One bar. 4.49 seconds. Nothing underneath it.

Sixteen Redis commands executed inside that window, across three shards, and the trace shows none of them.

Sit with that image for a second. If somebody handed it to you during an incident, you would conclude this service does not talk to Redis. It talks to Redis sixteen times.

After: the same code, traced

Same code. Same commands. Same 4.5 seconds.

Eleven redis.get and redis.set spans, each one tagged with the shard that served it.

The span math, because the two screenshots do not obviously agree

The fixed run produces 20 Redis spans in total. Only 11 land inside the cart-checkout trace.

The other 9 are redis.ping shard health checks. go-redis runs those on its own goroutines, outside the request context, so Sentry correctly files them as separate root traces.

That detail makes the bug worse than I first understood. The ring's own health monitoring was untraced too, which means a ring quietly losing a shard would not have surfaced either.

Which Sentry tools did the work

Tool	What it did here
Distributed Tracing	Ingested the real OTel spans over OTLP, so I am watching the actual instrumentation, not a reimplementation of it
Error Monitoring	The failed commands are captured on a scope tagged with operation, key, and hook mode, so the Issue says what happened
Trace linked errors	`sentryotel.NewOtelIntegration()` attaches captures to the active span, so the error and its span land together
Span attributes	Every span carries `redis.shard`, which proves every shard is covered, not merely that something fired

That last row matters more than it sounds. A partial fix that instrumented one shard out of three would look perfectly healthy on a span count, and would still be broken. Shard level attributes are what turn "some spans exist" into "coverage is complete."

Two things that cost me time

The OTLP endpoint has a segment you will miss.

DSN    https://<key>@o<org>.ingest.<region>.sentry.io/<project>
OTLP   https://o<org>.ingest.<region>.sentry.io/api/<project>/integration/otlp/v1/traces
auth   x-sentry-auth: sentry sentry_key=<key>

Leave out /integration/ and you get 404 Not Found with an empty body, which tells you nothing about which half of the URL was wrong.

sentry-go v0.48 removed NewSentrySpanProcessor.

The tracer provider pattern most tutorials still show will not compile. OTLP export is the current path, and the otel package is now reduced to linking errors onto the trace. Also worth knowing: OTLP ingestion is in open beta and span events are dropped during ingest, which is why the diagnostic detail here lives in attributes. That turned out to be the better design regardless.

The evidence travels with the code

I added a local span counter as a span processor, so the demo prints its own tally. You do not have to take my word for what the Sentry UI showed:

BUGGY                          FIXED
ran:    16 commands            ran:    16 commands
redis spans observed: 0        redis spans observed: 20

Clone it, point it at your own DSN, and you will get the same two numbers.

I benchmarked 7 LLMs on 100 identical prompts. The cost gap shocked me.

vigneshwar — Mon, 08 Jun 2026 05:43:29 +0000

Everyone asks: which LLM is the best?

Wrong question.

The right question: which LLM is best for your use case, at your scale, at your budget?

I ran 100 identical prompts across 7 major LLMs. Here's what the data actually showed.

The Numbers Nobody Shows You

Model	Accuracy	Cost/1K	Latency
GPT-4o	88.2%	$0.0080	892ms
Claude 3.5 Sonnet	87.6%	$0.0090	1240ms
GPT-4o-mini	78.4%	$0.0003	432ms
Gemini 1.5 Flash	76.8%	$0.0001	380ms
Claude 3 Haiku	74.2%	$0.0010	410ms
Mistral Small	71.0%	$0.0010	520ms
Llama 3 8B	64.4%	$0.0002	680ms

GPT-4o vs Gemini Flash:

11% accuracy gap
80x cost gap
2.3x speed gap

For 90% of production apps — Gemini wins.

Why I Built This

Every AI leaderboard ranks by accuracy.

Your production AWS bill ranks by cost per request.

They are not the same list.

I built an open source LLM Evaluation Framework that benchmarks any model across all 5 dimensions simultaneously:

5 Metrics in One Run

1. Accuracy
Four-strategy cascade scorer:

Exact string match (case-normalized)
Prefix normalization (strips "The answer is...")
Multiple-choice letter extraction
Fuzzy Levenshtein match at 0.85 threshold

2. Latency (full percentile breakdown)

p50, p75, p90, p95, p99
SLA violation rate against configurable threshold
Async parallel evaluation via LiteLLM

3. Cost per 1K tokens

From real token counts in API responses
Not estimates — actual billing data
Supports 15+ model providers

4. Hallucination Rate

Linguistic signal analysis
Detects hedging phrases, uncertainty markers, ungrounded claims vs grounding signals
Runs entirely locally, zero extra API cost
Score: 0.0 (grounded) to 1.0 (heavily hallucinating)

5. Reasoning Quality

Chain-of-thought depth scoring
Counts reasoning markers, grounding signals, response calibration
Score: 1 (one-word answer) to 10 (structured multi-step reasoning)

Quick Start

pip install llm-evaluation-framework

export OPENAI_API_KEY=your-key
export GEMINI_API_KEY=your-key

llm-eval compare --models gpt-4o-mini --models gemini/gemini-1.5-flash --benchmark mmlu --samples 100

Output:

Model              Accuracy  Latency p95  Cost/1K    Hallucination  Reasoning
gpt-4o-mini        78.4%     891ms        $0.000300  0.12           7.2
gemini-1.5-flash   76.8%     743ms        $0.000100  0.15           6.8

What's Under the Hood

Async parallel evaluation — runs all samples concurrently with configurable semaphore
MMLU benchmark — 57 subjects, ~14K questions (Massive Multitask Language Understanding)
TruthfulQA benchmark — 817 questions designed to expose common misconceptions
Custom benchmarks — bring your own JSON: [{"prompt": "...", "expected": "..."}]
FastAPI REST API — 12 endpoints, OpenAPI docs
Streamlit dashboard — radar charts, scatter plots, histograms
CLI — 7 subcommands
SQLite persistence — all runs stored, queryable
PDF report generation — shareable evaluation reports

The Uncomfortable Truth About AI Benchmarks

Leaderboards rank models by accuracy on standardized tests.

Production systems rank models by accuracy per dollar.

Those are very different rankings.

GPT-4o vs Gemini Flash:

On a leaderboard: GPT-4o wins by 11%
At 10M requests/month: $80,000 vs $1,000

For most apps, the correct answer 11% more often is NOT worth $79,000/month.

Stop picking LLMs from leaderboards. Start picking them from your data.

Live Demo

Try the accuracy scorer and hallucination detector live — no API key needed:

https://huggingface.co/spaces/vigneshwar234/llm-eval-demo

I Spent 4 Months Building a RAG System That Actually Understands Causality — Here's What I Learned (and the Math Behind It)

vigneshwar — Sun, 07 Jun 2026 11:29:58 +0000

"I spent 4 months building something the entire ML community said was already solved. Turns out, it wasn't."

I need to tell you something uncomfortable about every RAG system running in production today.

They're all broken in the same way. And almost nobody is talking about it.

I'm not saying they don't work. They do — most of the time. But there are two silent failure modes that cause them to hallucinate even when they retrieve the correct document. After months of banging my head against this problem, I built VORTEXRAG to fix both of them. This is that story.

The Day I Realized Something Was Deeply Wrong

I was building a financial Q&A system. Simple enough — index a corpus of SEC filings, answer questions about why companies performed the way they did.

The query: "Why did Company X's revenue drop in Q3?"

My RAG pipeline retrieved the right document. I could see it in the logs — the actual earnings call transcript where the CEO explained the supply chain disruption. Cosine similarity: 0.91. Perfect.

But the LLM's answer was completely wrong. It talked about macroeconomic conditions, interest rate sensitivity, sector-wide headwinds. All factually true things — about the industry. None of them the actual cause.

I dug into the context window. The correct chunk was there. But it was surrounded by 7 other chunks:

The company's 10-K risk factors (similarity: 0.87)
An analyst report on sector performance (similarity: 0.84)
Fed reserve commentary on the quarter (similarity: 0.82)
Three more topically-related but causally-irrelevant passages

The LLM saw all of them. It averaged them. It hallucinated a narrative that sounded exactly right but was factually wrong about this specific company.

I had discovered Context Window Poisoning — and I'd been unknowingly fighting it for months.

The Two Problems Nobody Fully Solves

After that moment, I went deep. I read every RAG paper published in the last 3 years. Self-RAG, CRAG, RAG-Fusion, FiD, REALM, Atlas, Toolformer — all of them. Great papers. Smart people. Significant advances.

But none of them fully addressed what I was seeing. Here's why:

Problem 1: Semantic Drift 🎯

Every RAG system today retrieves by cosine similarity. This works great for finding topically related content. But it fundamentally cannot distinguish between:

A chunk that caused something
A chunk that is merely associated with it

Ask "Why did Lehman Brothers collapse?"

Standard RAG returns:
| Chunk | Similarity | Causal? |
|-------|-----------|---------|
| Dodd-Frank Act provisions | 0.87 | ❌ Response to collapse |
| CDS mispricing mechanism | 0.91 | ✅ Actual cause |
| Systemic risk reports | 0.85 | ❌ Consequences |
| Bear Stearns comparison | 0.83 | ❌ Parallel event |

The LLM gets all four. It produces a response about regulatory failure and systemic risk. It never mentions CDSs. The answer is 100% hallucination — constructed from real documents, assembled into a false causal chain.

This is Semantic Drift: the retrieved context drifts from causal relevance toward topical association.

Problem 2: Context Window Poisoning ☠️

Even when you retrieve the right chunk, if 7 wrong chunks surround it, the LLM's attention gets diluted. This isn't speculation — it's backed by the "Lost in the Middle" paper (Liu et al., 2023): LLMs have a U-shaped recall curve. They remember the beginning and end of context best, and systematically lose information in the middle.

So even if your correct chunk is in the context window, if it lands at position 4 of 8 chunks, the LLM may functionally ignore it.

This is Context Window Poisoning: the noise-to-signal ratio in the context window destroys the LLM's ability to use the correct information.

Building VORTEXRAG: 4 Months, 7 Layers, One Obsession

I started with a simple question: What would it take to fix both problems simultaneously?

The answer turned out to be a 7-layer pipeline. Each layer solves a specific failure mode. Let me walk you through each one — not just what it does, but why I built it.

Layer 1: TVE — Tri-Vector Encoding 🔺

The insight: If similarity retrieval can't capture causality, we need to encode causality directly.

Standard RAG embeds text into a single semantic vector (usually 768 dimensions). VORTEXRAG encodes every chunk into a 864-dimensional tri-vector:

TVE score = α·cos_sem + β·cos_syn + γ·cos_cau

Where:
  sem = 768d SBERT all-mpnet-base-v2    (semantic meaning)
  syn = 64d  spaCy dependency parse     (syntactic structure)
  cau = 32d  PropBank SRL events        (causal relationships)

The causal arm (32 dimensions) encodes the PropBank-style semantic role labels: ARG0 (agent), ARG1 (patient), ARGM-CAU (cause), ARGM-EFF (effect). A chunk about "CDSs caused Lehman's collapse" has a very different causal vector than a chunk about "Dodd-Frank responded to the collapse" — even if they have identical semantic similarity to the query.

This is the foundation everything else builds on.

Layer 2: VRC — Vortex Retrieval Cone 🌀

The insight: Retrieval isn't a list — it's a geometry.

Traditional top-k retrieval treats candidates as a ranked list. VORTEXRAG models retrieval as a spiral probability surface in causal vector space:

spiral_rank = TVE · e^(−λr) · cos(nθ)

Where:
  θ = angle between query's causal vector and chunk's causal vector
  r = rank position in initial retrieval
  λ = 0.5 (radial decay, adaptive with corpus size)
  n = 2 (angular frequency)

The key term is cos(nθ). Chunks whose causal direction is more than π/4 (45°) from the query's causal direction get geometrically suppressed — their spiral rank drops toward zero regardless of semantic similarity. The "vortex" shape comes from combining the radial decay with the angular suppression: retrieved chunks form a cone, not a list.

Layer 3: SDC — Semantic Drift Corrector 🛡️

The insight: Measure the drift explicitly. Filter on it.

For each candidate chunk, SDC computes a Semantic Drift Score:

D = v_cau(query) − v_cau(chunk)    ← causal drift vector
SDS = 1 − tanh(‖D‖ / τ)           ← drift score ∈ [0, 1]

Accept chunk if SDS ≥ 0.72

The higher the ‖D‖, the more the chunk's causal structure has drifted from the query's. τ controls the sensitivity — and this is where the 11 domain presets come in:

Domain	τ	Why
Scientific	0.30	Strict: cause-effect chains must be tight
Medical	0.35	Strict: diagnosis → treatment causality
Legal	0.40	Strict: precedent → ruling chains
Financial	0.50	Moderate: multi-factor causation
General	0.80	Default: relaxed filtering
Creative	1.20	Lenient: loose associations acceptable

Layer 4: CPG — Context Poison Guard ⚗️

The insight: The entire window must be healthy, not just individual chunks.

Even if each chunk individually passes SDC, the combination of chunks can still poison the context. CPG measures the Effective Signal Ratio of the entire window:

ESR = Σ(SDS_i · w_i) / (P + ε)

Where:
  P = 1/k normalization penalty (penalizes large windows)
  ε = smoothing constant

If ESR < 3.5, CPG runs a greedy purge: remove the chunk with the lowest SDS score. Recompute ESR. Repeat until ESR ≥ 3.5.

Theorem 5.1 (proved in the paper): Removing the minimum-SDS chunk maximizes ESR improvement per step. The greedy algorithm is provably optimal — no other single-step removal produces a better ESR increase.

This theorem is what separates CPG from heuristic approaches. It's not "try to remove bad chunks." It's a mathematical guarantee that the greedy approach is the best possible approach.

Layer 5: RFG — Rank Fusion Gate 🔀

The insight: Additive fusion lets weak links through. Multiplication doesn't.

Most multi-signal retrieval systems use additive fusion:

score = w1·signal1 + w2·signal2 + w3·signal3

The problem: a chunk with (0.9, 0.9, 0.1) scores the same as one with (0.63, 0.63, 0.63). But the first chunk is clearly wrong — one signal is terrible.

VORTEXRAG uses multiplicative fusion:

Φ = TVE^α × SDS^β × ESR_contrib^γ

Now (0.9 × 0.9 × 0.1)^(1/3) = 0.45 vs (0.63 × 0.63 × 0.63)^(1/3) = 0.63. The chunk with one bad signal scores lower, as it should. No weak link can be compensated by strong links in other dimensions.

Layer 6: CCB — Causal Context Builder 🏗️

The insight: Where you put information in the context window matters enormously.

From "Lost in the Middle": LLMs recall information best at position 0 and near the end of context. The middle is a graveyard for information.

CCB builds a causal dependency graph and assigns positions:

pos = rank(Φ+) × causal_depth

depth-0 root-cause chunks → always placed at pos = 0

The chunk that is the causal root — the one that explains why — always goes first. Supporting evidence follows in causal order. The LLM gets the most important information exactly where it's best at attending to it.

Layer 7: FV — Faithfulness Verifier ✅

The insight: Don't trust the LLM. Measure it.

After generation, FV computes:

ΔR = 1 − ROUGE-L(answer, context) × NLI_score(answer, context)

Accept if ΔR ≤ 0.15

If ΔR > 0.15 (more than 15% of the answer isn't grounded in the context), FV rejects the response, reranks the context, and retries — up to 3 times. Uses DeBERTa-v3-small CrossEncoder for NLI.

This is the final catch. Even if all 6 previous layers work perfectly, the LLM can still hallucinate. FV makes hallucination measurable and catchable.

The Results — After 229 Tests and 6 Benchmarks

I tested VORTEXRAG on the standard QA benchmark suite: NQ, TriviaQA, WebQ, PopQA, HotpotQA, and 2WikiMultiHopQA.

Overall Performance

System	EM	F1	Faithfulness	Semantic Drift	Context Poisoning	Latency
VORTEXRAG	74.8	82.6	0.94	14%	7%	185ms
Self-RAG	68.4	77.1	0.81	28%	19%	410ms
CRAG	66.9	75.8	0.79	31%	22%	320ms
RAG-Fusion	62.8	71.9	0.73	33%	21%	280ms
Naive RAG	61.2	69.4	0.71	36%	24%	95ms

+13.6 EM over Naive RAG. +6.4 EM over Self-RAG. 2.2× faster than Self-RAG.

Per-Dataset Breakdown

Dataset	VORTEXRAG	Self-RAG	Δ	Why VORTEXRAG wins here
NQ	74.1	67.2	+6.9	Single-hop causal queries
TriviaQA	81.3	77.8	+3.5	Fact retrieval, less causal
WebQ	68.4	61.9	+6.5	Entity-centric causal chains
PopQA	71.2	63.4	+7.8	Long-tail causal knowledge
HotpotQA	67.9	61.1	+6.8	Multi-hop: biggest gains
2WikiMH	69.8	62.3	+7.5	Multi-hop: biggest gains

The biggest gains are on multi-hop datasets — exactly where causal reasoning matters most. This is the validation I was hoping for.

Ablation: Every Layer Earns Its Place

One of my proudest moments was running the ablation study. I was terrified some layers would show no contribution. They all did.

Configuration	EM	F1	Faithfulness	+EM
A: Naive RAG baseline	61.2	69.4	0.71	—
B: +TVE	65.3	72.8	0.74	+4.1
C: +VRC	67.8	75.1	0.76	+2.5
D: +SDC	70.4	78.3	0.80	+2.6
E: +CPG	72.1	80.2	0.85	+1.7
F: +RFG	73.4	81.4	0.89	+1.3
G: +CCB	73.9	81.9	0.91	+0.5
H: +FV (full VORTEXRAG)	74.8	82.6	0.94	+0.9

Every single layer contributes. There's no dead weight.

Human Evaluation — The Number That Matters Most

Automatic metrics only go so far. I had 3 domain experts (an NLP researcher, a practicing lawyer, and a biomedical scientist) evaluate 150 responses each on 4 dimensions using 5-point Likert scales.

Dimension	VORTEXRAG	Self-RAG	Naive RAG
Factual Accuracy	4.5/5	3.9/5	3.2/5
Causal Coherence	4.3/5	3.4/5	2.8/5
Completeness	4.2/5	3.8/5	3.5/5
Conciseness	4.1/5	4.2/5	4.0/5

The +0.9 on Causal Coherence vs Self-RAG is the one I'm most proud of. It means real humans, in real domains, found the causal reasoning in VORTEXRAG's answers to be significantly better. No automatic metric fully captures this.

Try It Right Now

5-Minute Quickstart

git clone https://github.com/vignesh2027/VORTEXRAG
cd VORTEXRAG
pip install -r requirements.txt
python examples/basic_usage.py

Your First Causal Query

from vortexrag import VortexRAG

# Pick your domain — parameters auto-calibrate
rag = VortexRAG(domain="medical")

result = rag.query(
    query="Why do SSRIs take 2-4 weeks to work?",
    chunks=my_medical_chunks
)

print(result.answer)          # Causally grounded answer
print(result.sds_score)       # Drift score: should be >= 0.72
print(result.faithfulness)    # Hallucination delta: <= 0.15
print(result.layer_trace)     # Full 7-layer trace for debugging

Layer-by-Layer Trace (What You Actually See)

Layer 1 [TVE]: Encoded 847 chunks as 864-dim tri-vectors (8.2ms)
Layer 2 [VRC]: Spiral-ranked top-50 candidates (2.1ms)
              Angular suppression: 12 chunks suppressed (θ > π/4)
Layer 3 [SDC]: Drift filtering with τ=0.35 (medical preset)
              Accepted: 31/50 chunks (SDS range: 0.72-0.97)
Layer 4 [CPG]: Initial ESR: 2.84 (below threshold 3.5)
              Purging: removed 4 low-signal chunks
              Final ESR: 4.12 ✓
Layer 5 [RFG]: Multiplicative Φ-scores computed (1.8ms)
              Top-5 selected: Φ ∈ [0.71, 0.89]
Layer 6 [CCB]: Causal graph built (3 root-cause chunks at pos=0)
              Context ordered by causal depth
Layer 7 [FV]:  Generated answer, ΔR = 0.09 ✓ (< 0.15 threshold)
              Faithfulness verified on first attempt

Total: 183ms

What I Learned Building This

1. Cosine similarity is a local optima. The entire field optimized around it so heavily that we forgot it was a proxy, not the thing itself. Causal relevance is the thing itself.

2. Every layer needs a mathematical reason to exist. I threw out 3 layers during development because I couldn't prove they were optimal or even justified. Theorem 5.1 (CPG optimality) took me 3 weeks to prove. Worth it.

3. Latency is a feature. At 185ms, VORTEXRAG is 2.2× faster than Self-RAG despite doing far more work. Why? Because Self-RAG generates multiple draft responses and selects. VORTEXRAG's pre-generation filtering means the LLM sees a clean, small context and generates once. Less is more.

4. Human evaluation is irreplaceable. My biomedical expert caught failure modes on multi-step enzyme pathway questions that EM, F1, and faithfulness all missed. Automatic metrics are necessary but not sufficient.

The Part I Almost Didn't Share

I'm a student at Takshashila University. This wasn't done in a research lab with 40 GPUs and a team of PhD students. This was done on my laptop, in my room, across 4 months of evenings and weekends, after getting frustrated with every RAG system I used giving me wrong causal answers.

I don't have institutional affiliation. I don't have a PhD supervisor. I just had a problem that bothered me and enough stubbornness to not stop until I could prove I'd solved it.

If you're a student or independent researcher reading this: the problems worth solving are the ones that bother you personally. Not the ones that are impressive on a CV. The ones that make you think "this is broken and nobody seems to notice."

That's what VORTEXRAG was for me. I hope it's useful for you.

Links & Citation

🌀 Live Demo: https://huggingface.co/spaces/vigneshwar234/VORTEXRAG

💻 Code (MIT, 229 tests): https://github.com/vignesh2027/VORTEXRAG

📄 Paper (Zenodo v3.0): https://doi.org/10.5281/zenodo.20579702

🔬 ORCID: https://orcid.org/0009-0004-9777-7592

@article{vignesh2026vortexrag,
  title   = {VORTEXRAG: Vector Orthogonal Resonance-Tuned EXtraction
             Retrieval-Augmented Generation — A 7-Layer Framework for
             Simultaneous Elimination of Semantic Drift and Context
             Window Poisoning},
  author  = {Vignesh, L},
  year    = {2026},
  doi     = {10.5281/zenodo.20579702},
  url     = {https://doi.org/10.5281/zenodo.20579702}
}

If this helped you, or if you tried VORTEXRAG and have feedback, drop a comment below. Every reaction, unicorn, and share genuinely helps this reach more people. Thank you for reading.

Tags: #machinelearning #python #rag #nlp #ai #deeplearning #opensource #research

[Boost]

vigneshwar — Sun, 24 May 2026 04:20:40 +0000

I lost 3 enterprise clients in one night because of a GitHub repo. So I built a tool to make sure it never happens again. - DEV Community

It was 11:47 PM on a Tuesday. I had just pushed to production. Closed my...

dev.to

I lost 3 enterprise clients in one night because of a GitHub repo. So I built a tool to make sure it never happens again.

vigneshwar — Sun, 24 May 2026 04:19:35 +0000

It was 11:47 PM on a Tuesday.

I had just pushed to production.

Closed my laptop. Made tea. Felt good about myself.

By 3:14 AM my phone was a disaster.

17 missed calls. 43 Slack messages. 6 emails.
The subject line on the first email read:
"URGENT — Platform completely down"

My hands were shaking before I even opened it.

Three weeks earlier I had been under insane deadline pressure.

We were building a SaaS product for enterprise clients.
Launch was in 72 hours.
I needed an authentication library fast.

I went to GitHub.

Found one that looked incredible.
Clean name. Professional README.
2,400 stars. 340 forks.
The code looked solid on first glance.

I did what most developers do under deadline pressure.

I added it. Shipped it. Went to sleep.

What I didn't check:

The last commit was 9 months ago.
There were 47 open issues marked as critical.
Zero CI/CD pipeline.
Zero test files in the entire repo.
The maintainer had responded to exactly 0 issues in 6 months.
There was a known security vulnerability reported 4 months ago.
Still open. No response. No fix.

In 3 seconds I could have seen all of this.

I didn't check. So I didn't know.

Until 3am.

The bug triggered under high concurrent load.

Our enterprise demo that night had 200 simultaneous users.
The library collapsed. Took the auth system with it.
Every single user got logged out.
Sessions destroyed. Data in a corrupted state.
The whole platform returned a 500 error for 14 straight hours.

We lost 3 enterprise clients that week.
Each worth $40,000 annually.

$120,000 gone because I didn't spend 3 minutes
checking a GitHub repo properly.

My manager didn't fire me.
But the look on his face in that Monday meeting
is something I will never forget as long as I live.

After that I became obsessive.

I started checking every single dependency manually.
Every library. Every tool. Every npm package.
Every GitHub repo anyone on the team suggested.

I built a personal checklist:

→ When was the last commit?
→ Is there a CI/CD pipeline?
→ Are there test files?
→ How many open issues vs closed?
→ What is the average time to close an issue?
→ Who are the contributors and are they still active?
→ Is there a license?
→ How long and detailed is the README?
→ What does the community size look like?
→ Are there known CVEs in the dependencies?

20 to 30 minutes per repo.
Every single time.

My team thought I was paranoid.

I thought I was just finally doing my job properly.

Four months later I had evaluated hundreds of repos this way.

And I was completely burned out from doing it manually.

Every evaluation felt like the same work.
The same checks. The same tabs. The same mental process.
Over and over and over.

I started thinking about the developers who don't do this at all.
The ones who are exactly where I was at 11:47 PM on that Tuesday.
Feeling good. Laptop closed. Tea in hand.

Not knowing what's coming.

So I spent three weeks and built RepoLens.

Not for clout. Not for a portfolio piece.
Because I genuinely needed it.
And I was pretty sure millions of other developers did too.

Here is what it does:

Paste any GitHub URL.

In 3 seconds you get:

🏥 Repository Health Score — 0 to 100
A single score computed across 7 quality dimensions.
README quality. Commit activity. Test detection.
CI/CD presence. License. Community size. Issue resolution.
One number that tells you everything.
With a letter grade. A B C D.
So you know in 1 second if this is production-ready.

🥧 Language Breakdown
A beautiful interactive pie chart showing every single language
used in the codebase with exact percentages.
Know the full technical makeup before you touch it.

🔥 52-Week Commit Heatmap
A GitHub-style activity grid showing every week of the past year.
See at a glance — is this project alive or abandoned?
Spot burnout periods. Spot release sprints.
Spot the exact week the maintainer stopped caring.

👥 Top Contributor Graph
Who actually built this thing?
Are they still active?
Is it one person or a healthy team?
Bar chart. Avatars. Contribution share visualization.
Everything you need to know about who drives this project.

📦 Smart Dependency Detection
Automatically parses every ecosystem file:
package.json for Node.
requirements.txt and pyproject.toml for Python.
Cargo.toml for Rust.
go.mod for Go.
pom.xml for Java.
Gemfile for Ruby.
Every package. Every version. Automatically.

🗂 Interactive File Tree
Collapsible directory explorer with file type icons.
See the structure of any codebase instantly.
Search and filter in real time.

📖 Beautiful README Renderer
Full GitHub Flavored Markdown.
Images. Tables. Code blocks. Everything.
Read the documentation without leaving the tool.

📤 One-Click Share Card
Export a beautiful PNG summary card.
Share on LinkedIn. Post on Twitter.
Send to your team before a code review.

I ran the library that destroyed my production server through it.

31 out of 100. Grade D.

In 3 seconds.

The exact score I needed at 11:47 PM on that Tuesday
instead of at 3:14 AM the next morning.

I've been using RepoLens every single day since I built it.

My entire team uses it now before every dependency decision.
We have a rule — no new library gets added without a score.

We haven't had a single library-related production incident since.

Not one.

I'm sharing it completely free.

No sign-up required.
No account.
No credit card.
No limits.
Works on every public GitHub repository on the planet.
Instant results. Every time.

And the entire thing is open source.

React 18 frontend. Vite. Tailwind CSS.
FastAPI Python backend. GitHub REST API only.
File-based caching. Rate limiting. Security headers.
Full type hints. Clean architecture.

If you want to see how it's built — every line of code is there.
If you want to contribute — PRs are open.
If you want to self-host it — full Docker support.

🌐 Try it free:

⭐ Star it on GitHub:
github.com/vignesh2027/GitHub-Repo-Analyzer

Drop any GitHub repo URL in the comments below.

I will personally reply to every single one
with its health score and what I'd fix first.

And tell me —

What's the worst GitHub repo you ever trusted?

What happened?

Because I have a feeling I'm not the only one
who learned this lesson the hard way.

awesome claude skills

vigneshwar — Thu, 21 May 2026 03:29:02 +0000

Awesome-Claude-Skills I built 135 Claude Skills with real formulas. Here's what "production-grade" actually means. - DEV Community

I've been frustrated for a long time. Every "awesome Claude prompts" repo I found looked like...

dev.to

Awesome-Claude-Skills I built 135 Claude Skills with real formulas. Here's what "production-grade" actually means.

vigneshwar — Thu, 21 May 2026 03:16:43 +0000

I've been frustrated for a long time.

Every "awesome Claude prompts" repo I found looked like this:

"Act as a senior software engineer. Be helpful, thorough, 
and professional. Consider edge cases."

That's not a skill. That's a costume.

Real expertise has frameworks. Named responsibilities.
Actual formulas. Code that runs. Constraints that prevent
the model from giving you the easy wrong answer.

So I spent 6 months building what I actually wanted.

AgentOS 2.0 — 135 production-grade Claude Skills.

This article explains exactly what's inside and why it's
different from every other prompt collection on GitHub.

The Problem With Every Other Prompt Repo

Most prompt repositories fall into one of three traps:

Trap 1: The costume prompt

"You are an expert financial analyst. 
Help the user with their finance questions."

Zero frameworks. Zero formulas. Zero depth.

Trap 2: The instruction dump

"When answering, always:
- Be professional
- Consider multiple angles  
- Cite sources
- Format your response clearly"

This is just asking Claude to be Claude. It changes nothing.

Trap 3: The persona prompt

"You are Alex, a no-nonsense McKinsey consultant 
with 20 years of experience..."

Roleplay, not expertise. The model doesn't suddenly
know DCF models because you named it Alex.

What actually works: Named sub-agents with
distinct responsibilities, actual domain formulas
in code, and explicit forbidden behaviors that
prevent hallucination in critical areas.

Here's what that looks like in practice.

What "Production-Grade" Actually Looks Like

FinanceOracle — The Apex Skill

This is the most complete skill in the repo.
Here's a fraction of what's inside:

12 Sub-Agents:

OptionsDesk — derivatives pricing and structuring
MacroStrategist — macro regime analysis
HedgeFundArchitect — strategy design
FamilyOfficeCIO — multi-generational allocation
TaxOptimizer — harvest and structure optimization
DerivativesStructurer — exotic product design (+ 6 more)

Actual runnable Python:

def black_scholes(S, K, T, r, sigma, option_type='call'):
    d1 = (np.log(S/K) + (r + 0.5*sigma**2)*T) / (sigma*np.sqrt(T))
    d2 = d1 - sigma * np.sqrt(T)

    if option_type == 'call':
        price = S * norm.cdf(d1) - K * np.exp(-r*T) * norm.cdf(d2)
        delta = norm.cdf(d1)
    else:
        price = K * np.exp(-r*T) * norm.cdf(-d2) - S * norm.cdf(-d1)
        delta = -norm.cdf(-d1)

    gamma = norm.pdf(d1) / (S * sigma * np.sqrt(T))
    vega  = S * norm.pdf(d1) * np.sqrt(T) / 100
    theta = (-(S * norm.pdf(d1) * sigma) / (2 * np.sqrt(T))
             - r * K * np.exp(-r*T) * norm.cdf(d2)) / 365

    return {"price": price, "delta": delta, 
            "gamma": gamma, "vega": vega, "theta": theta}

Black-Litterman portfolio construction:

def black_litterman(Sigma, market_weights, views_P, 
                    views_Q, views_omega, tau=0.05, delta=2.5):
    pi = delta * Sigma @ market_weights
    M_inv = np.linalg.inv(
        np.linalg.inv(tau * Sigma) + 
        views_P.T @ np.linalg.inv(views_omega) @ views_P
    )
    mu_bl = M_inv @ (
        np.linalg.inv(tau * Sigma) @ pi + 
        views_P.T @ np.linalg.inv(views_omega) @ views_Q
    )
    return {"expected_returns": mu_bl}

This isn't pseudocode. This runs.

OKREngine — Catches Failures Before They Kill Your Quarter

I've watched two startups waste entire quarters on
broken OKRs. This skill exists because of that.

The objective quality scorer:

def score_okr(objective: str, key_results: list[dict]) -> dict:
    obj_score = 0
    obj_score += 3 if len(objective) < 100 else 0
    obj_score += 3 if not objective.lower().startswith("improve") else 0
    obj_score += 4 if any(w in objective.lower() for w in 
        ["best", "lead", "#1", "transform", "redefine"]) else 0

    kr_scores = []
    for kr in key_results:
        kr_score = 0
        kr_score += 3 if kr.get("metric") else 0
        kr_score += 3 if kr.get("baseline") is not None else 0
        kr_score += 4 if kr.get("target") is not None else 0
        kr_scores.append({
            "kr": kr["text"][:60],
            "score": kr_score,
            "grade": "Good" if kr_score >= 8 else "Needs work"
        })

    return {
        "objective_score": f"{obj_score}/10",
        "key_results": kr_scores,
        "recommendation": "Strong OKR" if obj_score >= 8 else "Needs revision"
    }

The skill also catches the 12 most common OKR failure modes — including sandbagging, health metrics disguised as OKRs, and the single most destructive mistake: tying OKR scores to bonuses.

VentureIntelligence — Term Sheet Red Flag Detector

def score_term_sheet(terms: dict) -> dict:
    red_flags = []

    if terms.get("liq_pref_multiple", 1) > 1:
        red_flags.append(
            f"Liquidation preference {terms['liq_pref_multiple']}x — above 1x is punishing"
        )
    if terms.get("participating_preferred", False):
        red_flags.append(
            "Participating preferred — VCs get paid twice in exits below threshold"
        )
    if terms.get("anti_dilution") == "full_ratchet":
        red_flags.append(
            "Full ratchet anti-dilution — catastrophic in a down round"
        )
    if terms.get("board_seats_investor", 0) > terms.get("board_seats_founder", 0):
        red_flags.append(
            "Investor has majority board control — you can be fired from your company"
        )

    score = 10 - (len(red_flags) * 3)
    return {
        "score": max(0, score),
        "grade": "Sign it" if score >= 8 else "Negotiate" if score >= 5 else "Get a lawyer NOW",
        "red_flags": red_flags
    }

12 sub-agents including TermSheetDecoder, ValuationNegotiator, ChampionDeveloper, and BoardRelationshipManager.

CrisisIntelligence — War Room OS

Every company will face a crisis. Almost none prepare.

def classify_crisis(crisis: dict) -> dict:
    severity_score = 0

    if crisis["customer_impact_pct"] >= 0.5: severity_score += 30
    if crisis["revenue_at_risk"] >= 1_000_000: severity_score += 20

    coverage = {"none": 0, "local": 5, "national": 15, "viral": 30}
    severity_score += coverage.get(crisis["media_coverage"], 0)

    if crisis["regulatory_involvement"]: severity_score += 15
    if crisis["legal_liability"]: severity_score += 15

    if severity_score >= 70:
        level = "CRITICAL (P0)"
        action = "CEO leads. War room activated NOW."
    elif severity_score >= 40:
        level = "HIGH (P1)"
        action = "VP-level lead. External comms needed."
    else:
        level = "MEDIUM (P2)"
        action = "Director-level. Monitor externally."

    return {
        "level": level,
        "immediate_action": action,
        "time_to_first_response": "1 hour" if severity_score >= 70 else "4 hours"
    }

The 5Rs framework (Recognize → Respond → Responsible → Remediate → Restore) is built into every communication template.

How It Works (60-Second Setup)

Claude.ai Projects:

# 1. Open Claude.ai → Projects → New Project
# 2. Paste SKILL.md into "Project Instructions"  
# 3. Start chatting

Claude Code:

cat finance-oracle/SKILL.md >> .claude/CLAUDE.md

Claude API:

import anthropic

with open("startup-cto/SKILL.md", "r") as f:
    skill = f.read()

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    system=skill,
    messages=[{"role": "user", "content": "Audit our tech stack decision"}]
)

That's it. Claude is now that specialist.

The Full 135-Skill Index

🚀 Startup & Team Management (11)
startup-cto team-performance-os startup-hiring-machine

culture-architect remote-team-commander okr-engine

startup-finance-controller venture-intelligence

startup-legal-shield talent-management-os talent-brand-builder

🏆 Apex Legendary (4)
finance-oracle claude-mythos ceo-war-room founder-to-ceo

🤖 AI & Engineering (14)
rag-architect mlops-engineer system-architect senior-dev

ai-red-teamer voice-agent-builder knowledge-graph-builder

incident-commander mcp-builder agentic-workflow-builder

api-integrator realtime-data-agent agent-smith prompt-engineer

📊 Data & Analytics (10)
data-scientist-pro sql-analyzer data-pipeline-pro

business-intelligence-pro timeseries-oracle quant-trader

synthetic-data-generator arxiv-researcher abtest-scientist

data-governance-agent

💹 Finance (9)
finance-oracle financial-model-builder cfo-intelligence

portfolio-optimizer quant-researcher saas-metrics-analyst

insurance-actuary ma-dealmaker risk-sentinel

🏢 Operations & Business (20)
ceo-war-room founder-to-ceo go-to-market-commander

enterprise-sales-os sales-enablement-os board-deck-builder

crisis-intelligence partnership-intelligence pricing-strategist

project-command marketing-os supply-chain-oracle (+ 8 more)

👤 Product & Customer (11)
product-roadmap-os sprint-master engineering-manager

ai-product-manager user-research-os customer-interview-analyst

product-analytics-os network-effects-analyst marketplace-strategist

performance-marketing-os churn-analyst

🛠 Developer Tools (19)
developer-experience-os api-design-architect data-warehouse-architect

cloud-cost-optimizer design-system-architect technical-pm

code-reviewer load-tester code-migrator webapp-tester (+ 9 more)

🌐 Specialized Domains (12)
healthcare-analytics web3-developer climate-tech-analyst

biotech-analyst cybersecurity-analyst real-estate-intelligence

legal-eagle patent-analyst esg-compass (+ 3 more)

What Makes This Different From Every Other Repo

Feature	Generic repos	AgentOS 2.0
Sub-agents	❌	✅ 10-12 per skill
Actual formulas	❌	✅ Black-Scholes, DCF, MEDDPICC
Runnable code	❌	✅ Python, TypeScript, Go, Shell
Forbidden behaviors	❌	✅ Every skill
Benchmark data	❌	✅ Industry standards built in
Total skills	~10-20	135+

Try It Right Now

The fastest way to understand the depth is to try one.

I recommend starting with okr-engine or startup-cto — they're the most complete and immediately useful regardless of what you're building.

Paste the SKILL.md into Claude Projects. Ask it to review your current OKRs or tech stack. You'll see the difference immediately.

GitHub link in the comments.

What skill would you build your work around? Drop it below — I read every comment and I'm actively building more.

MIT License. Free forever. Star it if it's useful — helps others find it.

I Spent 6 Months Fixing RAG. Here's What I Found (And Built)

vigneshwar — Tue, 19 May 2026 15:59:07 +0000

This is the story of a debugging session that turned into a research paper.

The Bug That Started Everything
I was building a document Q&A system — nothing exotic. Standard RAG setup. FAISS index, SBERT embeddings, GPT as the reader. Classic.

It worked fine on simple questions. "What is the refund policy?" → correct answer.

Then I tested it on a multi-hop question: "What are the environmental compliance requirements for facilities that process the chemicals used in the manufacturing process described in section 4.2?"

The model's answer was confident. Detailed. And completely wrong.

The retrieved documents were all there. Every piece of information needed to answer correctly was in the context window. But the model still hallucinated a number that appeared nowhere in any document.

I started logging. What I found was two distinct failure modes happening simultaneously:

Failure Mode 1: Semantic Drift. By the time the query had been reformulated for multi-hop retrieval, the embedding had drifted so far from the original intent that we were retrieving the wrong documents. Not slightly wrong. Documents from a completely different section of the corpus.

Failure Mode 2: Context Poisoning. Even when we retrieved mostly correct documents, 1–2 tangentially related but contradictory chunks were slipping through. And those poison chunks were enough to derail the model.

Standard RAG has no defense against either of these. The pipeline is essentially: embed → retrieve → stuff into context → hope.

I needed something better.

Six Months Later: VORTEXRAG
I'm releasing the full framework today. 7 layers, each targeting a specific failure mode. Here's what I built and why each layer exists.

Layer 1: Tri-Vector Encoding (TVE)
The problem: Single-vector embeddings collapse too much information. "The bank charged a fee" and "The river bank was steep" share a close embedding in SBERT space even though they're semantically unrelated in most retrieval contexts.

The solution: Three encoding arms:

Semantic arm: standard SBERT 768d

s = sbert_model.encode(chunk) # shape: (768,)

Syntactic arm: POS + dependency structure

t = syntactic_encoder(chunk) # shape: (64,)

Causal arm: verb-argument chains

c = causal_encoder(chunk) # shape: (32,)

Fused vector

v = concat([α·s, β·t, γ·c]) # shape: (864,)
The causal arm is the key innovation. It captures "X causes Y" relationships that pure semantic similarity misses entirely. This is what allows the pipeline to distinguish between a document that mentions a concept and a document that explains the causal mechanism behind it.

Layer 2: Vortex Retrieval Cone (VRC)
The problem: Flat cosine similarity treats all high-similarity documents equally. But document #1 and document #47 in your ranked list shouldn't have equal weight — there's a natural falloff in relevance.

The solution: Spiral ranking inspired by vortex dynamics:

spiral_rank = TVE · e^(−λr) · cos(nθ)
Where r is the radial distance (rank position) and θ is an angular phase that encodes causal depth. Documents with high causal relevance get a phase advantage that can overcome a slightly lower semantic similarity score.

In practice: the top-k documents returned by VRC are causally denser than those returned by a flat cosine search on the same index.

Layer 3: Semantic Drift Corrector (SDC)
The problem: Multi-hop queries reformulate themselves at each hop. Each reformulation can drift slightly from the original intent. Over 3–4 hops, this compounds into a completely different query.

The solution: Track the embedding trajectory. At each hop:

drift = query_embedding - anchor_embedding
SDS = 1 - tanh(np.linalg.norm(drift) / τ)

if SDS < 0.72:
query_embedding = re_anchor(query_embedding, anchor_embedding)
The SDS score (Semantic Drift Score) measures how far we've drifted. Below 0.72, we re-anchor to the original query intent. This single intervention eliminated most of our multi-hop hallucinations.

Layer 4: Context Poison Guard (CPG)
The problem: 1–2 contradictory or off-topic chunks in the context window is enough to poison the model's answer. We needed a way to identify and remove these before they reach the LLM.

The solution: Entity-salience ratio per chunk:

ESR = sum(SDS_i * w_i for each entity i) / (num_propositions + ε)

if ESR < 3.5:
flag_for_purging(chunk)
The purging algorithm is provably greedy-optimal (I include the formal proof in the paper). It maximizes total ESR across the retained context while respecting the token budget.

In ablation studies, removing CPG alone dropped faithfulness from 0.94 to 0.75 — an 0.19 point drop from a single layer.

Layer 5: Rank Fusion Gate (RFG)
The problem: Most rank fusion methods are additive. A single terrible signal gets diluted but doesn't eliminate a bad document.

The solution: Multiplicative fusion:

Φ = TVE^α × SDS^β × ESR^γ
If any of the three signals is near zero, Φ collapses toward zero. A document that scores 0.9 on semantic similarity but 0.1 on CPG gets Φ ≈ 0.09 — effectively eliminated.

This was a deliberate design choice. In high-stakes retrieval (medical, legal, financial), you want a veto mechanism, not a popularity contest.

Layer 6: Causal Context Builder (CCB)
The problem: LLMs have dramatically higher attention to the beginning and end of their context window. Documents buried in the middle get "lost" — the Lost-in-the-Middle problem (Liu et al., 2023).

The solution: Reorder chunks by causal depth:

position = rank * causal_depth

High causal depth → lower position number → placed at context start

Chunks that are causally central to answering the question get placed at the beginning of the context window, where attention weights are highest. Peripheral context gets pushed to the middle where it does less damage if ignored.

Layer 7: Faithfulness Verifier (FV)
The problem: Even with all 6 preceding layers, some hallucinations slip through. We need a final gate.

The solution: Score the candidate answer against the retrieved context:

ΔR = 1 - (ROUGE_L * NLI_entailment_score)

if ΔR > 0.15:
regenerate_answer()
If the answer diverges more than 15% from the source documents (measured by ROUGE-L weighted by NLI entailment), it gets thrown out and regenerated. This catches the subtle hallucinations — cases where the model paraphrases correctly but changes a critical number or name.

Results
Tested on 4 standard benchmarks (NQ, HotpotQA, MuSiQue, 2WikiMultiHopQA):

System EM F1 Faithfulness
Naive RAG 61.2 68.4 0.71
HyDE 63.8 71.2 0.74
Self-RAG 65.4 73.1 0.79
FLARE 64.9 72.8 0.77
VORTEXRAG 74.8 82.6 0.94
+13.6 EM. +14.2 F1. +0.23 Faithfulness over naive RAG.

The ablation shows all 7 layers contribute. The two biggest individual contributors are CPG (+0.19 faithfulness) and SDC (+0.08 EM on multi-hop benchmarks specifically).

Quick Start

pip install vortexrag

from vortexrag import VortexRAG, VortexConfig

config = VortexConfig(
sdc_threshold=0.72,
cpg_esr_threshold=3.5,
fv_delta_r_threshold=0.15,
)

rag = VortexRAG(config)
rag.index(your_documents)

answer = rag.query("Your complex multi-hop question here")
What's Next
The framework is MIT licensed. The full research paper (with formal proofs) is on Zenodo.

If you're building RAG systems and hitting hallucination walls — especially on multi-hop or domain-specific queries — this framework is designed for exactly that problem.

GitHub: https://github.com/vignesh2027/VORTEXRAG

Paper: https://doi.org/10.5281/zenodo.20285144

Live demo docs: https://vignesh2027.github.io/VORTEXRAG

Questions? Drop them in the comments — happy to go deep on any of the layers.

Post these tonight at 9:00 PM IST. Reddit first (r/MachineLearning, then r/LocalLLaMA 30 minutes apart), then Dev.to. Cross-link the Dev.to article in your Reddit comments with "I wrote a deeper walkthrough here" — that drives traffic both ways.