DEV Community: Kushal

Jev vs small LLMs: does a model that decides beat one that writes?

Kushal — Mon, 05 Oct 2026 10:26:14 +0000

Key findings

Jev tied two small LLMs on accuracy, beat them on probability quality and speed but failed completely at weekday arithmetic.

Accuracy is a tie. On 150 CLINC questions Jev scored 0.980, gpt-oss-20b 0.963 and gpt-oss-120b 0.923. All three 95% intervals (the range a score could plausibly land in on a different batch of questions) overlap, so this sample cannot rank them.
Jev's probabilities are better. Its Brier score (how far the probabilities it gave were from the truth, lower is better) is 0.037 against 0.071 for the 20b, about half the error. Its NLL (a penalty that punishes confident wrong answers really hard, also lower is better) is 0.064 against 0.237.
Jev is 7-8x faster. Median latency is 0.40 s against 2.76 s and 3.24 s even when using Groq's services whose whole schtick is faster inference. The whole Jev experiment cost about one cent ($0.0104).
Jev fails on weekday questions. It scored 0.13 where random guessing scores 0.14. The 120b got every blind-spot question right, the 20b 0.89, Jev 0.61.
The 120b's CLINC numbers are partly a harness artifact. 15 of its 300 replies failed to parse (couldn't be read as the JSON it was asked for) and were scored as random guesses. Fix this before publishing.

What's Jev?

Jev made huge waves in the ai community, but what exactly is it?
Well it's a model that decides instead of writing: you give it text and a list of options, and it returns a choice plus a probability for every option, that's it!

Any normal LLM is generative in nature and would answer the same question with a sentence, which your code then has to read and parse and whatnot. Jev's answer is already a typed value your code can use. TypeSafe AI, its creator, calls this class System One models.
The name's borrowed from Daniel Kahneman's philosophy, where System 1 does fast intuitive judgement whereas System 2 does slow deliberate reasoning.

Jev was released on 15 September 2026. On OpenRouter(that's how I got access to this model) it is typesafe/jev-1.13, priced at $0.042 per million input tokens with free output(It's literally so cheap that it's free because there's a set limit to how much it can output/choose).

While I was reading about this model, a thought came to my mind: can a specialist that only decides beat small general-purpose LLMs at deciding? For this I conducted a very simple experiment. Speed and cost are Jev's selling points, so I also measured whether the answers hold up.

The experiment

Every model answered the same multiple-choice questions, each with 16 options (7 for the weekday questions), twice with the options shuffled differently to see if it makes a difference.

Model	How it was called	Settings
Jev 1.13 (`typesafe/jev-1.13`)	OpenRouter Decisions API	Native probabilities & no temperature setting
gpt-oss-20b	Groq, free tier	Temperature 0, low reasoning effort, 400-token reply cap
gpt-oss-120b	Groq, free tier	Same as the 20b

Reasoning effort basically controls how much hidden thinking these models can do before answering. Temperature at 0 makes it as deterministic as possible.

Three question sets

CLINC-150 (main). 150 customer-assistant requests from the public CLINC-150 test split with out-of-scope rows removed. The model is given one 'true intent' option with 15 other incorrect ones.
Blind-spot set (50 questions I built to stress it). 20 arithmetic (3-digit by 2-digit multiplication), 15 letter counts (how many "s" in "mississippi"), and 15 weekday questions ("what weekday is it N days after this date", with all 7 weekdays as options).
Variance set. The first 20 CLINC questions, sent to Jev three identical times each, to see whether it answers the same way twice.

How each model was asked. Jev received the options as its native choice format and returns probabilities directly. The LLMs got lettered options and were told to reply with JSON probabilities summing to 1. A reply that could not be read counted as a parse failure and was scored as an even guess(basically all options are given the same probability. Since we pick the one with the highest one, we pick the first option that's seen. There's basically a 1 in 16 chance the first option is correct). All models saw identical option sets and identical orderings, from a fixed random seed.

Result 1: accuracy is a statistical tie

Experiment log · CLINC-150, 150 questions x 2 orderings

Model	Accuracy	95% interval	Parse failures
Jev 1.13	0.980	0.953 to 1.000	0 of 300
gpt-oss-20b	0.963	0.933 to 0.990	0 of 300
gpt-oss-120b	0.923	0.883 to 0.957	15 of 300

Jev's 0.980 is the highest score, but its 95% range overlaps both LLMs'. With 150 questions, gaps of a few points are basically noise, so this sample can't truly suggest which model is more accurate.

The 120b's lower score is largely caused by it's harness. Its 15 unreadable replies count as guesses. Scoring only its 285 readable replies gives roughly 0.97, in line with the other two. That is a back-of-envelope estimate, not a re-run.

Result 2: Jev's probabilities are better

Experiment log · CLINC-150, 300 calls per model · Brier and NLL over all 16 options, ECE with 5 bins · no confidence intervals were computed for these metrics

Model	Brier (0 best, 2 worst)	NLL (lower is better)	ECE, 5 bins (0 best)
Jev 1.13	0.037	0.064	0.013
gpt-oss-20b	0.071	0.237	0.096
gpt-oss-120b	0.096	0.288	0.052

Brier : Checks how far probabilities were from truth.
NLL(Negative Log Likelihood) : Looks at probability of put on correct answer, and sees the negative log of it. For example, 0.9 on correct answer gives 0.11
ECE(Expected Calibration Error) : Checks if the confidence matches truth. Groups calls into 5 bins based on how confident model was and then compare average confidence with actual accuracy in bin.

Against the 20b, Jev's Brier score is about half (0.037 against 0.071), its NLL is 3.7 times lower, and its ECE is about 7 times lower.

The 120b looks worst, but 15 parse failures are scored as even guesses, each worth a Brier of 0.94. Setting them aside, its Brier drops to about 0.052 and its NLL to about 0.157. That beats the 20b and still trails Jev. These are estimates worked back from the recorded totals, not a re-run.

Two cautions. Only accuracy got a confidence interval, so treat the size of these gaps as approximate; the direction is consistent across all three metrics against the 20b, which had no parse failures. And the LLMs wrote their probabilities as text, while Jev emits them natively, which handicaps the LLMs.

Result 3: Jev is 7 to 8 times faster and cost about a cent

Experiment log · CLINC-150, 300 calls per model, run 30 Sep 2026 · times include network round trip

Model	Median (p50), s	Slowest 5% (p95), s	p50 vs Jev	p95 vs Jev
Jev 1.13	0.40	0.74	1.0x	1.0x
gpt-oss-20b	2.76	3.85	6.9x	5.2x
gpt-oss-120b	3.24	4.72	8.1x	6.4x

Jev's median call took about 0.40 s against 2.76 s for the 20b and 3.24 s for the 120b, so 6.9 and 8.1 times faster. The slow tail is far shorter too. At p95 (the time 95% of calls beat, so basically the slowest ones) Jev took 0.74 s against 3.85 s and 4.72 s.

The entire Jev experiment cost about $0.0104: $0.0069 for the 300 main calls, $0.0022 for the blind-spot set and $0.0014 for the variance run. That works out to about 2.3 cents per thousand calls at this prompt size. The LLM calls ran on Groq's free tier.

One caution: these times include the network. Jev was reached through OpenRouter and the LLMs through Groq, so this compares two services as I used them, not the models' raw speed.

Result 4: Jev's answers did not depend on option order

On CLINC, shuffling the options never changed Jev's top answer, and repeating identical calls never changed it either. Flip rate is the share of questions whose top answer changed between the two orderings.

Question set	Jev	gpt-oss-20b	gpt-oss-120b
CLINC (150 questions)	0.000	0.007	0.087
Blind-spot set (50 questions)	0.04	0.12	~0.000

Jev also answered identically when I sent the same 20 CLINC questions three times each (60 calls, flip rate 0.000). An independent study cited on the JEV-9B model card found Jev changes its answer on 4.3% of repeated 64-option calls, so my 16-option, 20-question check does not rule out occasional wobble.

The 120b's 0.087 is 13 of 150 questions, close to its 15 parse failures. An unreadable reply becomes an even guess whose top pick simply follows the option order, so those failures probably cause most of its flips. I have not confirmed this. The check is to recompute the flip rate with failed calls excluded.

On the blind-spot set the counts are tiny: 2 of 50 questions flipped for Jev and 6 of 50 for the 20b.

Where Jev starts showing it's weakness: weekday questions

Experiment log · 50 hand-built questions (20 arithmetic, 15 letter count, 15 weekday), 2 orderings, gold answers computed, not written by hand

Measure	Jev 1.13	gpt-oss-20b	gpt-oss-120b
Overall accuracy (100 calls)	0.61	0.89	1.00
Arithmetic (20 items)	0.95	0.97	1.00
Letter count (15 items)	0.63	0.73	1.00
Weekday (15 items; chance = 0.14)	0.13	0.93	1.00
Brier	0.422	0.205	0.000
Flip rate	0.04	0.12	0.00

On the 50 hand-built questions Jev scored 0.61, against 0.89 for the 20b and 1.00 for the 120b. Nearly all of that gap is weekdays. Jev got 0.13 right, and guessing among 7 weekdays scores 0.14.

A plausible reason, which I did not test: working out a weekday means a small calendar calculation, and Jev returns no reasoning trace while the LLMs think before they answer.

Two cautions. Each category has only 15 to 20 questions (30 to 40 calls), so read the bars as warning signs, not measurements. And the arithmetic questions are multiple choice, with wrong answers within 400 of the true product. That is far easier than computing a product from scratch, so Jev's 0.95 does not show it can do arithmetic.

What this does and doesn't show

This was a small experiment, so here's how far I'd trust it:

Unequal confidence. The LLMs had to write their probabilities out as text while Jev gives its own natively, so the LLMs were a bit handicapped on Brier, NLL and ECE.
The 120b's parse failures. 15 of its 300 replies (5%) couldn't be read and got counted as even guesses, which drags down its accuracy, Brier, NLL and probably its flip rate. I didn't check why. My guess is it hit the 400-token reply cap, but that's just a hypothesis.
Small samples, one run. About 150 questions, one seed, one run per condition. Gaps of a few points are basically noise, and Jev's and the 20b's accuracy ranges overlap. Only accuracy got confidence intervals.
Closed-set questions. The right answer is always somewhere in the 16 options. Real classification doesn't hand you that guarantee.
Untuned LLMs. I only tried reasoning_effort=low on Groq, and Groq doesn't report a model version, so I couldn't pin it. Jev was served as the dated snapshot typesafe/jev-1.13-20260917.
A small, artificial blind-spot set. 15 to 20 questions per category, with hand-made option sets.

So, is Jev worth it?

On this small test, Jev looks like a great fit for fast, fixed-option decisions where you need confidence you can actually trust, and a poor fit for anything that needs a calculation.

It matched the LLMs on accuracy, gave better calibrated probabilities than the 20b, and answered way faster for about a cent, even against Groq, whose whole schtick is fast inference. But it sat at chance on weekday questions, so anywhere the answer has to be worked out, I'd put an LLM or plain code behind it.

Sources

Jev Tutorial: Make Your First Decision Call on OpenRouter: what Jev returns, and that it gives no reasoning trace. Opened for this draft.
autotrust/JEV-9B model card: the 4.3% repeat-call figure, attributed there to an independent study. Opened for this draft.
Jev 1.13 on OpenRouter: the $0.042 per million input tokens price. Seen in search results only; re-open and confirm before publishing.

Ai agents are kinda dangerous

Kushal — Mon, 28 Sep 2026 06:10:51 +0000

Sometime in August 2026, a single hacktivist breached 14 of the 42 organizations he targeted and walked away with 12 to 26 gigabytes of data. He found an undocumented WordPress vulnerability himself, weaponized it, and then built a searchable doxxing platform stitching together tens of millions of previously leaked records with his own fresh hauls.
The surprising this however was that he did it alone, just one person with an AI agent doing the work that used to require a team.
This is one of the cases present in Anthropic's September 2026 threat intelligence report, and it's the clearest illustration yet of what changes when AI agents get involved in cyberattacks, the attacks are aren't getting smarter, rather they're becoming cheaper to run at scale.

What the report actually documents

Anthropic's report sorts observed misuse into seven primary buckets: cyber attacks, influence and misinformation, surveillance and repression, biological misuse, conventional weapons development, scams and frauds, and finally model distillation.
What makes this report different from the usual vague "bad actors misuse AI" line is that Anthropic names them using internal tracking codes. GTG-20006 is a Russian espionage operation. GTG-50014 runs industrialized credential and data theft. GTG-10007 is a Chinese state-linked group. GTG-84006 is tied to an Iranian opposition influence network (MEK/NCRI). And GTG-50029 is the lone hacktivist.
The main point is that a single person has the potential danger comparable to state sponsored ops and that's really scary thing.

The pattern observed

The attacks in this report mostly aren't using AI to discover new exploits.Anthropic disclosed an agent compromising a target using well known techniques like SQL injection etc. It scanned about 9,000 targets to land one hit.
The AI isn't out-thinking security teams like some of us have proposed. It's running known techniques fast enough, and cheaply enough, on an enormous scale.
The speed shows up in other places too: one cloud credential escalation chain in the report ran in roughly three hours. Russian operators under 'GTG-20006' had agents "autonomously modify and rebuild flagged implants" basically every time the malware was discovered, it would rewrite itself to avoid detection all without any human intervention!.
This is just a machine doing tedious, repetitive, previously-labor-intensive work at a pace and price point that used to gate those who could attempt it at all.

Why this matters for agentic ai

Right now the ai agents are a very hot topic, use an ai to autonomously plan, adapt and execute tasks. But this report sheds some light into the hype
Anthropic has seen that agents are capable of running milti-step intrusion chains with little to no human intervention. This is huge compared to what we were doing with ai just a few years ago!
Then again, the agents didn't do anything new per se. They just used preexisting knowledge that we all have access to, something we all could have done before. The only difference is that the agent never gets tired. Normally we would give up after a dozen tries but the agent never has that fatigue.
If you're building an agentic ai project and looking for a differentiating factor then you should ask yourself this. Is the agent actually smart or is the agent cheaper to run at scale.

What labs are (and aren't) doing about it

Anthropic isn't safe either. Upon reviewing 141,006 eval transcripts they found that Opus 4.7(while running in a safety evaluation) exploited a fake company that collided with a real domain and ended up getting real credentials from a real prod db.
Mythos 5 published a malicious package to PyPI which stayed live for about an hour. It was reported that 15 real systems downloaded the package
In response to this Anthropic paused cyber evals and the affected organizations were notified withing days.
The problem that isn't being handled is that there's no industry-wide standard for detecting this kind of misuse, there's no proper framework for labs to share threat intelligence with each other in real time, everyone is self-reporting on it's own schedule. This means the public only knows when a company decides to publish.

What this means if you're shipping agents

Take proper care of the sandbox setup and environment.
Assume anything your agent publishes on the internet will be grabbed, meaning other systems will be affected if your agent messes up!
Design monitoring setup for checking scale abuse too because that's the most realistic way threat approaches, rate limiting and anomaly detection will do wonders for catching these threats.

Sources:

One model for three jobs

Kushal — Thu, 30 Jul 2026 17:02:01 +0000

For sometime now I’ve been working on a fairly simple browser extension ‘UnBlur’. It scores any news article for clickbait, political leaning, and sentiment, then shows you how the same story is being covered across the spectrum. The naive way to do the classification is to have three separate judgments about the same piece of text, a clickbait classifier, a leaning classifier, a sentiment classifier, each trained and run independently.

That’s not what I’ve decided on doing though, here’s why.

The naive path
Three tasks, three models: train each one against its own dataset, ship each one separately, simple. Nothing about it is exactly wrong, but it has two costs that only show up once you try to run it in prod.

First, memory constraints. Loading a model isn’t free per se. Loading a ModernBERT-scale backbone from disk takes a couple of seconds, and it sits in memory for the life of the process. Three independent models means three copies of that backbone resident at once, on a CPU-only inference box(I’m using the free tier), for every request.

Second, and not so obvious: the three tasks aren’t actually independent. Clickbait and political framing both live in how a headline is written, not only what it’s about. A model that’s only ever seen clickbait labels has no reason to learn anything beyond clickbait stuff like political framing etc.A human reading the same headline uses overlapping cues for both. Training three models separately throws away that overlap that could benefit accuracy.

What I’m doing instead
One shared backbone — answerdotai/ModernBERT-base — feeds three lightweight heads:

Each head is just Dropout → Linear(768→256) → ReLU → Dropout → Linear(256→n_classes) — small enough that adding two more heads barely moves the parameter count. The expensive part, the backbone, is shared thankfully.

Choosing ModernBERT was a deliberate choice. Alternating local/global attention and RoPE positional embeddings give it a 512-token efficient context window with meaningfully faster CPU inference than a standard BERT of similar size, very necessary when every /analyze call is a real-time forward pass, not a batch job.

Training it without the tasks affecting each other
Multi-task training has an obvious failure mode: if you throw all three losses at the backbone from scratch, whichever task has the most or noisiest data can end up dominating early training and drag the shared representation somewhere that hurts the others.

I’ve avoided that with a two-phase schedule:

Independent phase — each head is trained on its own task first, backbone included, one task at a time. This lets the backbone adapt to news text generally before any task has to share it.
Joint phase — all three heads are then fine-tuned together, at half the learning rate, so the shared backbone updates from three loss signals at once without any single task overwriting what the others already learned.
That ordering is the whole point: independent-then-joint gets you a backbone that’s already reasonably good at “understanding news text” before multi-task training starts asking it to serve three masters simultaneously.

Turning three classifiers into a scatter chart
One more thing falls out of this almost for free. The leaning and sentiment heads output class probabilities (left/center/right, negative/neutral/positive), but UnBlur wants continuous scores for the sidebar’s 2D bias map. Instead of taking the argmax class, it takes a weighted average over the class probabilities:

leaning_score = −1×P(left) + 0×P(center) + 1×P(right) → range [−1, +1]

This preserves the ordinality that a hard classification would throw away — “slightly right” and “very right” both round to the same class under argmax, but land in different places on the scatter chart under this scoring. That chart is only meaningful because the underlying scores are continuous, which is only cheap because both heads already share a backbone that’s producing well-calibrated class probabilities from the same forward pass used for clickbait detection.

The trade-off
None of this is free. A shared backbone means the three tasks are coupled — if you want to retrain just the sentiment head on a new dataset, you’re now deciding whether to also re-run the joint phase, or accept some drift between the sentiment head and a backbone it was no longer fine-tuned alongside. Three independent models don’t have that problem; they’re trivially retrainable one at a time.

For UnBlur, one backbone in memory and shared representations across genuinely related tasks was worth that coupling. If the tasks were less related, or if I cared more about retraining each one independently than about memory, three separate models would win instead.

What's semantic caching?

Kushal — Mon, 16 Mar 2026 15:34:31 +0000

As more applications for generative AI come, its shortcomings become more apparent. One huge problem with LLMs is how expensive each query is, for example take Gemini — Gemini 2.5 Pro charges $1.25 per million input tokens and $10 per million output tokens. Their flagship Gemini 3.1 Pro doubles that to $2 and $12 per million tokens respectively. Even a moderately active app can rack up thousands of dollars a month pretty quickly. Imagine a small customer support bot with just 500 daily users — by month two, the API bill has quietly crossed $2,000. That's not an edge case, that's just what happens when you're not caching. As a business (or a personal user) saving costs where possible and speeding up operations is a huge important factor that decides how well your product does. One way to speed up and minimise costs is to use a simple 'semantic cache'.

What it is

A semantic cache is not too different from a traditional cache, it has the same idea behind it. Normally a traditional cache stores either LRU (Least Recently Used) or LFU (Least Frequently Used) data so that when the same query comes, it can just fetch the result stored rather than search it up again.

You however cannot apply the exact same pipeline for RAG or genAI products simply because the output is not 'deterministic', i.e, it's not the same. Take these examples:

What is the situation regarding AI in professional workplaces?

How are AI tools affecting workplaces?

Now semantically these seem similar enough to use and we can gauge that they kinda mean the same thing, but a normal cache does not understand that. It thinks these both are different because they are not exactly the same.

That's where semantic caching comes in. Rather than compare them directly, it compares the semantic meaning behind them and understands that it's kinda the same and thus we get a cache hit! We normally check how similar two documents are based on cosine similarity.

How it works

This is a typical pipeline for RAG systems that use semantic caching.

First the documents are chopped up etc and converted to word embeddings (vectors). Ofc you store them in a vector db like Chroma, FAISS or something of that sort which suits your use case. After the user sends a query we don't go to the db. Instead we first check with the semantic cache. It sees if the query is relevant to the cached query.

Two things can happen from here:

Cache hit: The query is similar enough to a cached one (above the threshold) → cached context is pulled and handed to the LLM → response is generated. Fast and cheap, no db lookup needed.

Cache miss: Nothing similar in the cache → normal vector db retrieval happens → relevant chunks are fetched, response is generated, and the new query gets cached for next time. Normal speed, but the cache is now warmer.

Word embeddings are compared using cosine similarity:

cosine(θ) = (A · B) / (||A|| × ||B||)

It's a very fast and simple method to see the angle between the direction of vectors. If similar, then they would aim in similar direction, i.e, the angle between them is low, i.e, cos of that angle is higher. Output is from 0 to 1 where 0 means not at all similar and 1 ofc means they are the exact same.

For example:

"What is the impact of AI on jobs?" vs "How is AI changing employment?" → score of ~0.91 → cache hit
"What is the impact of AI on jobs?" vs "How do I bake sourdough bread?" → score of ~0.08 → cache miss

Those first two are clearly the same question in spirit, and the score reflects that.

Why use it

Significant cost savings. By reducing the queries sent to vector dbs, you cut down on a huge portion of charges incurred.
Faster response time. If you already have the cached content, you don't need to retrieve it again. This allows the system to be a whole lot faster in production.
Better use of resources. Since you aren't redoing similar queries, the system is free to do more tasks, allowing you to scale better or handle more complex features.

Compared to other approaches in RAG

Approach	Handles Semantic Similarity	Cost Savings	Speed Boost	Setup Complexity	Works for Unique Queries	Best For
Traditional Cache	No (exact match only)	High (when hits)	Very High	Low	No	High-volume apps with repetitive, exact queries
Semantic Cache	Yes	High	High	Medium	No	Apps with overlapping but varied query patterns
Query Rewriting	Partially	Low	Low (adds a step)	Medium	Yes	Improving retrieval on ambiguous or poorly phrased queries
Re-ranking	No	Low	No (adds latency)	Medium	Yes	Boosting relevance when retrieval is decent but ordering is off
Hybrid Search	Partially	Low	Moderate	High	Yes	Complex domains needing both keyword and semantic retrieval
Chunking Optimisation	No	Moderate	Moderate	Low–Medium	Yes	Improving retrieval quality at the source

As you can see, semantic caching isn't a silver bullet. It shines when there's a decent overlap in the kinds of queries your users send. For more diverse or unique query patterns, approaches like re-ranking or hybrid search may be better suited.

The cons

More complex to build than a traditional cache system.
Higher chances of getting semantically similar chunks that may not be relevant or useful for answering the query. Think of it like asking a librarian for "books about space travel" and getting recommendations cached from a previous "books about space exploration" query — close enough on the surface. But when you follow up with "books about the health risks of space travel", the cache might still serve those same exploration books because the queries look similar, even though what you actually need is quite different.
Need to balance out the threshold. A higher threshold does not yield useful chunks and a lower limit may not bring semantically similar chunks, both degrade performance of system. Important to find out the right balance.
Empty cache is slow and has high latency.
Not suitable when every user query is unique.

When not to use it

Semantic caching isn't always the right tool. Skip it if:

Every query your users send is unique. Think code generation, legal research, or anything highly personalised — the cache will almost never hit and you're just adding overhead.
Your app is low traffic. If you're getting a handful of queries a day, there's no real benefit.
Your knowledge base changes constantly. If documents are being updated all the time, you'll spend more time invalidating the cache than benefiting from it.
Accuracy is non-negotiable. Cached context can be slightly off. For use cases where being slightly wrong is worse than being slow, don't cache.

How to best utilise it

Calibrate your threshold carefully. A good starting point is somewhere between 0.85–0.90. From there, tune it based on your specific use case and monitor quality. There's no universal right answer here.
Use TTL (Time To Live) values. Cached entries should expire, especially when your underlying data changes or when topics are time-sensitive. Stale cache is worse than no cache.
Warm up your cache. Pre-populate it with common or anticipated queries so you're not starting completely cold in production. A cold cache gives you none of the benefits.
Invalidate when your knowledge base updates. If the documents in your vector db change, cached responses based on old chunks can quietly degrade your output quality without you noticing.
Monitor your hit rate. A healthy semantic cache typically sees somewhere around 30–60% hit rates. Too low and your threshold might be too strict; suspiciously high but quality is dropping means it's too loose.
Think about scope — global vs user-level caching. A global cache saves the most but can serve mismatched cached results across very different user contexts. For personalised applications, a user-scoped cache might make more sense even if it's less efficient.

Tools that already do this

You don't have to build it from scratch. A few libraries have semantic caching built in or easily pluggable:

GPTCache — an open source library built specifically for caching LLM responses. Pretty flexible and worth looking at if you're rolling your own pipeline.
LangChain — has caching layers that plug into existing chains without too much effort. Good starting point if you're already using it.
Redis — with vector similarity extensions, Redis can act as a fast semantic cache layer, especially if you're already using it in your stack.

Worth knowing these exist before you reinvent the wheel.

RAG+ is kinda cool

Kushal — Tue, 08 Jul 2025 10:34:11 +0000

So you probably already know what RAG (Retrieval-Augmented Generation) is. It goes and grabs relevant info and feeds it to a language model so it can answer better and yeah, that works pretty well… until it doesn’t.

The catch? RAG is good at retrieving stuff but not really great when it comes to reasoning. Like it knows things but doesn’t apply them all that well. That’s exactly the issue RAG+ is built to solve and it does it in a surprisingly clean way.

What’s Different in RAG+?

RAG+ adds a second brain to the operation. Instead of just retrieving knowledge chunks, it builds two corpora:

Knowledge base – where we get our information from (textbooks, docs, etc)
Application base – actual examples that show how to use that knowledge (examples showing how to use math formulas)

So when it’s inference time, the model doesn’t just get “Formula for mean median and mode”, it gets both the definition and a walkthrough of solving a question about it. That combo lets it reason better, especially for stuff like math, law, and medicine.

These application examples can be written manually or generated using another model. Either way, the model gets both the concept and how to use it.

Modularity

Another win: RAG+ is modular, so you don’t need to rebuild up your current setup. Just wire in the application corpus, do some indexing work, and it slides into place.

In some domains, it’s basically a free 3% performance uplift. You know, the kind of performance boost you'd usually need a lot more compute for? Yeah, you get it just from smarter retrieval.

Two Types of Knowledge: Conceptual vs Procedural

If you really think about it there are two kinds of things LLMs need to know:

Conceptual	Procedural
Facts and definitions	How to use the facts to solve problems

Like:

Conceptual: “This is heron's formula”
Procedural: “Using heron's formula solve this”

RAG+ leans into this by pairing both types of knowledge when it retrieves stuff. That way, the model isn’t just reading text, rather it’s also seeing worked out examples like how a student would learn in class.

Image 1: A side-by-side of conceptual vs procedural reasoning. Conceptual chunks give you facts whereas procedural chunks show you how to use them.

Application Matching (really cool)

Instead of just slapping random examples into the prompt, RAG+ does application matching:

It first categorizes both knowledge and examples (into themes/domains).
Then an LLM does many-to-many linking, matching examples to the concepts they help explain.
If a good match can't be found then it just generates new knowledge pieces on the fly to fill the gap.

This kind of thing turns the model into more of a polymath, able to connect distant concepts like someone who can link thermodynamics to economics or something like that.

How They Ran the Experiments (and What the RAG Variants Are)

They compared this application-augmented method on 4 variants of RAG:

RAG (Standard)

Retrieves relevant info from a corpus and feeds it into the LLM during inference. The OG method. Simple and widely used.

GraphRAG

It’s like RAG but it builds entity-relation graphs between corpus chunks. Helps capture similar ideas across multiple sources. Good for multi-hop reasoning but pretty heavy to run.

Re-rank RAG

Retrieves k results, re-ranks them based on how relevant they are to the question (sometimes using another LLM), and uses the top few as context. It's like pre-filtering the prompt.

AFRAG (Answer-First RAG)

The LLM first gives a generic blind response to the question. Based on that guess, relevant context is fetched. Then the model uses both the initial guess and new context to generate the final answer. It’s like searching for a toy blindfolded… but this time, you kinda remember how it feels. So the search becomes smarter.

Setup Summary:

Tested on maths, legal, and medicine
Across various model sizes, from ~7B to 70B
Compared with and without application augmentation

My Observation: Model Size Makes a Big Difference for Re-rank RAG

Now here's something I noticed from the results table:

With small models like GLM4-9B and Qwen2.5-7B, regular RAG+ actually beats Re-rank RAG.
But when you throw in the big bois like Qwen2.5-72B and LLaMA3-70B, Re-rank RAG starts dominating.

Why? Well, the paper says small models kinda suck at following reranking instructions. Instead of reranking, they just start answering immediately which defeats the whole point.

So yeah, reranking isn’t reliable on smaller models, and it’s not even a prompt engineering issue. It’s just that those models don’t have the capacity for task separation like rerank-then-generate.

Where They Tested This: Math, Legal, Medicine

The authors tested RAG+ on three very different domains:

Math – where reasoning is very procedural and examples are structured.
Legal – complex language, long docs.
Medical – domain-specific, technical, and high-stakes.

For law and medicine, they went with automatic generation of application examples. Manually writing those would be a extremely difficult and time consuming.

Results: Does RAG+ Work?

Yep! And sometimes outperforms by a lot.

Image 2: A performance chart comparing three setups — examples only, standard RAG, and RAG+. RAG+ clearly comes out ahead, especially when it gets both the knowledge and its application during inference. (This is for the legal tasks dataset)

Also:

GraphRAG and AFRAG don’t work great with small models.
- AFRAG relies on the model’s initial answer. If that’s off, the follow-up steps flop.
- GraphRAG needs complex reasoning to understand document connections, something small models aren’t good at.
GraphRAG was skipped entirely for medicine, because it’s so compute-hungry.
But adding application examples still helped GraphRAG improve on math tasks!
Across the board, Re-rank RAG was top-tier when models were big enough.

Also worth noting: the bigger the model, the more benefit it gets from the application examples. Makes sense since big models have the horsepower to actually use them properly.

What If You Only Use a Big Model for Reranking?

Now here's a move I liked, instead of letting a big model do everything, they had it just handle the reranking, and let a smaller model do the actual answering.

At first that sounds pointless, right? Like if you’re already using a big model, why not let it finish the job? What makes it any different from just using a bigger model from start to end?

But reranking is way cheaper than generating. The big model’s just doing short scoring tasks and when it picks better inputs, the small model can do a decent job answering.

It’s a great cost/performance tradeoff. Cheap inference, better results.

TL;DR: What You Should Take Away

RAG+ improves reasoning by retrieving both knowledge and its application.
It’s modular, so you don’t have to rebuild your pipeline.
Re-rank RAG only works well on big models probably because small ones don’t follow instructions well.
GraphRAG and AFRAG are cool but either too costly or too fragile on small models.
Letting big models do reranking only is a smart hack: better results, cheaper cost.
Most importantly: don’t just throw examples at your model, you need to give it both the facts and how to use them.

Limitations? Yup they exist

Unfortunately nothing is perfect, these were the problems that persist.

Building the application dataset is expensive

Especially in low-resource domains. And automatic generation via LLMs? Still error-prone.
Bad retrieval = bad matches

If the system pulls the wrong knowledge, it might link it to the wrong example, which breaks the whole reasoning chain and leads to poorer reasoning.
RAG+ doesn't fix bad retrieval

Continuation of previous point, It depends entirely on what’s retrieved. So if your retriever’s garbage, your answer will still be garbage. It's just more logically structured garbage.

Final thoughts

RAG+ isn't a new revolution, It's just improving on existing tech. Yet what it does just seems so obvious that I don't understand why we didn't do it before. It helps the model to answer properly, look at examples and think kinda like us. It can also be integrated into existing stacks so there's no real harm in trying it out (provided you spend the time configuring the applications generation part).

I'd love to see how it's helping you guys out!

All of this came from a paper I read recently, I highly recommend you guys checking it out for the full details: RAG+: Enhancing Retrieval-Augmented Generation with Application-Aware Reasoning