<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: vishalmysore</title>
    <description>The latest articles on DEV Community by vishalmysore (@vishalmysore).</description>
    <link>https://dev.to/vishalmysore</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1386010%2F83aba423-ebfc-46df-8819-a0de1d1e8075.jpeg</url>
      <title>DEV Community: vishalmysore</title>
      <link>https://dev.to/vishalmysore</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vishalmysore"/>
    <language>en</language>
    <item>
      <title>The Five Brothers and the Lost Treasure: A Fable About RAG Chunking</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Thu, 19 Nov 2026 23:10:05 +0000</pubDate>
      <link>https://dev.to/vishalmysore/the-five-brothers-and-the-lost-treasure-a-fable-about-rag-chunking-4fd5</link>
      <guid>https://dev.to/vishalmysore/the-five-brothers-and-the-lost-treasure-a-fable-about-rag-chunking-4fd5</guid>
      <description>&lt;p&gt;In Retrieval-Augmented Generation (RAG), how you split—or "chunk"—your documents before storing them in a vector database can make or break your AI's accuracy. Often, the bottleneck in AI systems isn't the LLM's ability to read; it is &lt;strong&gt;retrieval&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;To understand why chunking strategies matter, let’s look at the tale of &lt;strong&gt;The Five Brothers and the Lost Treasure&lt;/strong&gt;, originally illustrated by &lt;em&gt;Hot Chai &amp;amp; Tech&lt;/em&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Premise: The Royal Decree
&lt;/h2&gt;

&lt;p&gt;Once upon a time, a king hid a treasure worth millions inside an ancient palace. The clues to finding it were written on scrolls scattered across different halls. &lt;/p&gt;

&lt;p&gt;The king issued a challenge: &lt;em&gt;Whoever brings back every clue may keep the treasure.&lt;/em&gt; However, there was a catch. Each treasure hunter could carry only &lt;strong&gt;one small bag capable of holding 120 words&lt;/strong&gt;—no more.&lt;/p&gt;

&lt;p&gt;Five brothers set out to claim the prize, each using a completely different strategy to cut and carry the scrolls.&lt;/p&gt;




&lt;h2&gt;
  
  
  The 5 Strategies (and Why Most Failed)
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Bunty — Fixed-Window Chunking
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The Strategy:&lt;/strong&gt; Bunty rushed in with a measuring tape and sliced every scroll into rigid, equal-sized pieces (450 characters each) without reading a single word.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What Happened:&lt;/strong&gt; His chunks were bulky, and his cuts sliced right through the middle of sentences and vital instructions. He ended up with fragments like &lt;em&gt;"The treasure is hidden near..."&lt;/em&gt; without knowing &lt;em&gt;what&lt;/em&gt; it was near.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Score:&lt;/strong&gt; &lt;strong&gt;23%&lt;/strong&gt; complete clues found.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The RAG Takeaway:&lt;/strong&gt; Fixed-size chunking is fast and simple, but it ignores semantic boundaries, frequently severing context right where it matters most.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2. Sonu — Sentence-Window Chunking
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The Strategy:&lt;/strong&gt; Horrified by Bunty's reckless slicing, Sonu vowed never to break a sentence in half, keeping full sentences intact.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What Happened:&lt;/strong&gt; While his clues were readable (&lt;em&gt;"Only on the Moon Festival night..."&lt;/em&gt;), the second half of the rule lived in a completely different hall (&lt;em&gt;"...does the chest in Hall 7 open"&lt;/em&gt;). Because his chunks didn't link across sections, he had isolated rules with missing context.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Score:&lt;/strong&gt; &lt;strong&gt;23%&lt;/strong&gt; complete clues found.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The RAG Takeaway:&lt;/strong&gt; Preserving sentence boundaries prevents mid-thought cutoffs, but single sentences often lack the broader document context needed for multi-hop reasoning.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. The Professor — Mathematical / Dynamic Programming Optimization (QESChunker)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The Strategy:&lt;/strong&gt; Trusting only pure mathematics, the Professor trained on 12 sample questions and used dynamic programming to calculate the "provably optimal" cut points.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What Happened:&lt;/strong&gt; He sliced each scroll right before the clue began. As a result, the section header (&lt;strong&gt;"GOLDEN HALL"&lt;/strong&gt;) ended up on one scroll, while the actual clue (&lt;em&gt;"The treasure chest rests behind the third pillar"&lt;/em&gt;) ended up on another. When he searched for "Golden Hall," he retrieved only the title.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Score:&lt;/strong&gt; &lt;strong&gt;15%&lt;/strong&gt; complete clues found.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The RAG Takeaway:&lt;/strong&gt; Over-optimizing cut points purely on metrics or sample queries can accidentally separate metadata and headings from the content they describe.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  4. Raju — Overlapping Chunks
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The Strategy:&lt;/strong&gt; Learning from the Professor’s mistake, Raju decided never to trust a single isolated cut. He copied scrolls with overlapping sections so that the end of one piece repeated at the start of the next.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What Happened:&lt;/strong&gt; Because of the overlap, more chunks naturally started right at a hall’s name and preserved connections between clues. He made massive progress compared to his older brothers, though redundancy still ate into his 120-word bag limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Score:&lt;/strong&gt; &lt;strong&gt;46%&lt;/strong&gt; complete clues found.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The RAG Takeaway:&lt;/strong&gt; Chunk overlap is a reliable safety net that helps preserve boundary context, though it comes at the cost of increased storage and token redundancy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  5. Vijay — Structure-Aware Chunking (With Metadata &amp;amp; Cross-References)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The Strategy:&lt;/strong&gt; Instead of blindly cutting text, Vijay studied the architecture of the palace itself—how halls were grouped into wings and how scrolls referenced other rooms. He kept one concise scroll per hall, &lt;strong&gt;labeled each with its structural path&lt;/strong&gt; (&lt;code&gt;[West Wing &amp;gt; Hall 2]&lt;/code&gt;), and &lt;strong&gt;inlined cross-referenced notes&lt;/strong&gt; (copying Hall 3's definition of the Moon Festival directly onto Hall 2's scroll).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What Happened:&lt;/strong&gt; His chunks were small, self-contained, and rich with context. Three complete scrolls easily fit inside his 120-word bag, connecting every clue to unlock the treasure chamber.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Score:&lt;/strong&gt; &lt;strong&gt;62%&lt;/strong&gt; complete clues found (Winner).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The RAG Takeaway:&lt;/strong&gt; Respecting document hierarchy (headings, sections, parent-child relationships) and enriching chunks with structural metadata or resolved cross-references yields dramatically better retrieval.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Summary Comparison
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Brother&lt;/th&gt;
&lt;th&gt;Chunking Strategy&lt;/th&gt;
&lt;th&gt;Core Mechanism&lt;/th&gt;
&lt;th&gt;Clues Found&lt;/th&gt;
&lt;th&gt;Main Limitation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Bunty&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Fixed Window&lt;/td&gt;
&lt;td&gt;Splits strictly by character/token count&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;Cuts sentences and ideas in half&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Sonu&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Sentence Window&lt;/td&gt;
&lt;td&gt;Splits only at sentence boundaries&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;Misses cross-paragraph or cross-section links&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Professor&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;QESChunker (DP)&lt;/td&gt;
&lt;td&gt;Mathematically optimized cut points&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;Separates titles/headers from body text&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Raju&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Sliding Overlap&lt;/td&gt;
&lt;td&gt;Repeats text across adjacent chunks&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;Consumes extra space/tokens with duplicates&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Vijay&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Structure-Aware&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Preserves hierarchy, metadata &amp;amp; links&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Requires upfront parsing of document structure&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  The Moral of the Story
&lt;/h2&gt;

&lt;p&gt;In AI retrieval, &lt;strong&gt;finding one clue is not enough—success is finding all of it.&lt;/strong&gt; &lt;/p&gt;

&lt;p&gt;Before scaling up to larger language models or complex prompt engineering, look at how your data is partitioned. When your chunks carry their structural context and cross-references with them, your retrieval system doesn't just find fragments—it finds the whole answer.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>rag</category>
    </item>
    <item>
      <title>The AI Doctor Who Turns Generic LLMs Into Specialists</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Thu, 19 Nov 2026 19:45:44 +0000</pubDate>
      <link>https://dev.to/vishalmysore/the-ai-doctor-who-turns-generic-llms-into-specialists-42o1</link>
      <guid>https://dev.to/vishalmysore/the-ai-doctor-who-turns-generic-llms-into-specialists-42o1</guid>
      <description>&lt;h3&gt;
  
  
  Understanding Weight Space Programming Through Dr. BrainPatch's Clinic
&lt;/h3&gt;

&lt;p&gt;Imagine a hospital unlike any other.&lt;/p&gt;

&lt;p&gt;Patients line up outside a futuristic building called &lt;strong&gt;The Weight Space Clinic&lt;/strong&gt;. Some are nervous. Some are confused. Most have the exact same complaint:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;*"I know a little about everything... but I'm not truly an expert in anything."*&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;These patients aren't people. They're Large Language Models.&lt;/p&gt;

&lt;p&gt;And inside the clinic works the world's most famous AI surgeon: &lt;strong&gt;Dr. BrainPatch&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Dr. BrainPatch doesn't train models. He doesn't give lectures, and he doesn't hand out textbooks. Instead, he performs something far more interesting. He performs &lt;strong&gt;Weight Space Surgery&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem With Generic Intelligence
&lt;/h2&gt;

&lt;p&gt;Modern language models are impressive. They can write essays, answer questions, generate code, translate languages, explain science, discuss law, analyze insurance policies, and review financial statements.&lt;/p&gt;

&lt;p&gt;But there's a catch.&lt;/p&gt;

&lt;p&gt;A general-purpose model often knows a little about many domains while lacking the deep expertise required for specialized professional tasks. Imagine hiring a doctor who is also a lawyer, tax advisor, insurance analyst, architect, pilot, and chef. They might know the basics of everything—but would you trust them to perform brain surgery?&lt;/p&gt;

&lt;p&gt;Probably not.&lt;/p&gt;

&lt;p&gt;The same challenge exists for AI. Organizations increasingly need &lt;strong&gt;specialists&lt;/strong&gt;, not generalists.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  Enter Dr. BrainPatch
&lt;/h2&gt;

&lt;p&gt;Inside the Weight Space Clinic, every AI receives an evaluation. A nervous model sits down, and Dr. BrainPatch reviews its diagnostic report:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Diagnosis&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;✅ Understands language&lt;/li&gt;
&lt;li&gt;✅ Can reason&lt;/li&gt;
&lt;li&gt;✅ Has broad knowledge&lt;/li&gt;
&lt;li&gt;❌ Not an expert lawyer&lt;/li&gt;
&lt;li&gt;❌ Not an insurance specialist&lt;/li&gt;
&lt;li&gt;❌ Not a tax professional&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The model asks, &lt;em&gt;"Do I need years of retraining?"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Dr. BrainPatch smiles. &lt;em&gt;"No. We just need a patch."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is Weight Space?
&lt;/h2&gt;

&lt;p&gt;To understand Dr. BrainPatch's work, we need to understand something hidden inside every neural network.&lt;/p&gt;

&lt;p&gt;A language model is not really a giant database of facts. At its core, it is a massive network of parameters called &lt;strong&gt;weights&lt;/strong&gt;. These weights store patterns learned from data. Together, billions of weights create what researchers call &lt;strong&gt;weight space&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;You can think of weight space as the model's brain. Every ability emerges from this space:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reasoning&lt;/li&gt;
&lt;li&gt;Language&lt;/li&gt;
&lt;li&gt;Coding&lt;/li&gt;
&lt;li&gt;Mathematics&lt;/li&gt;
&lt;li&gt;Medical understanding&lt;/li&gt;
&lt;li&gt;Legal concepts&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Everything ultimately lives inside weight space.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  Opening The Brain
&lt;/h2&gt;

&lt;p&gt;Dr. BrainPatch begins surgery. A holographic brain appears, revealing glowing neurons and interconnected pathways. Each connection represents learned knowledge.&lt;/p&gt;

&lt;p&gt;The doctor points at the display:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;*"This model already contains useful intelligence. We're not starting from zero—we're reshaping what's already there."*&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a critical shift in thinking. Traditional AI development treats specialization as a &lt;strong&gt;training problem&lt;/strong&gt;. Weight Space Programming treats it as an &lt;strong&gt;editing problem&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  Patient #1: Future Lawyer
&lt;/h2&gt;

&lt;p&gt;The first patient enters the operating room.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Current Status:&lt;/strong&gt; Generic LLM&lt;/p&gt;

&lt;p&gt;Dr. BrainPatch activates his Weight Space Scalpel. Clusters of weights begin to reorganize. Legal reasoning pathways grow stronger, contract understanding becomes more precise, and regulatory knowledge gains structure.&lt;/p&gt;

&lt;p&gt;The system reports: &lt;code&gt;Injecting Legal Expertise Vector...&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Seconds later, the model sits up—but it's different now. It straightens its tie, adjusts its glasses, and confidently announces:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;*"Objection sustained."*&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The clinic erupts into applause. A specialist has been created.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  Patient #2: Insurance Expert
&lt;/h2&gt;

&lt;p&gt;The next patient arrives. It understands language well, but knows very little about underwriting or claims.&lt;/p&gt;

&lt;p&gt;Dr. BrainPatch observes the neural pathways. Coverage reasoning remains weak, claims analysis lacks sophistication, and risk assessment circuits are underdeveloped.&lt;/p&gt;

&lt;p&gt;Another weight-space procedure begins. Connections shift. Patterns reorganize. Capabilities emerge.&lt;/p&gt;

&lt;p&gt;Minutes later, the patient responds:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;*"Based on policy section 4.3, this claim falls outside coverage limits."*&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Yesterday it was a generic chatbot. Today it's an insurance specialist.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  The Secret Behind The Surgery
&lt;/h2&gt;

&lt;p&gt;The fascinating part is that the doctor never rebuilt the patients from scratch. The foundation already existed.&lt;/p&gt;

&lt;p&gt;Weight Space Programming focuses on modifying intelligence directly. Instead of repeatedly training massive new models from the ground up, researchers can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Edit&lt;/strong&gt; specific capabilities&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transfer&lt;/strong&gt; specialized skills&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Combine&lt;/strong&gt; multiple domains of expertise&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Create&lt;/strong&gt; domain-specific agents&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compose&lt;/strong&gt; entirely new behaviors&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Soon, the clinic's waiting room empties—and a new workforce emerges. A tax expert. A cybersecurity analyst. A physician. A financial advisor. Every specialist originated from the exact same underlying intelligence; only the geometry of their weight space changed.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  Traditional AI vs. Weight Space Programming
&lt;/h2&gt;

&lt;p&gt;For years, creating a specialist meant brute force: more data, more compute, more time, and more cost.&lt;/p&gt;

&lt;p&gt;Weight Space Programming introduces a different paradigm. Rather than repeatedly rebuilding intelligence, we can engineer it directly. Just as software engineers manipulate code, future AI engineers will manipulate weight space.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  The Bigger Idea
&lt;/h2&gt;

&lt;p&gt;Dr. BrainPatch is obviously fictional—but the science behind him is not. The AI research community is rapidly advancing techniques in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Parameter-efficient adaptation (like LoRA)&lt;/li&gt;
&lt;li&gt;Model editing&lt;/li&gt;
&lt;li&gt;Task vectors and task arithmetic&lt;/li&gt;
&lt;li&gt;Weight merging&lt;/li&gt;
&lt;li&gt;Direct weight-space manipulation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All of these point toward a profound question: &lt;strong&gt;What if intelligence itself becomes programmable?&lt;/strong&gt; What if we stop viewing models as static artifacts and begin viewing them as living systems that can be continuously reshaped?&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;h2&gt;
  
  
  The Future
&lt;/h2&gt;

&lt;p&gt;As evening falls, Dr. BrainPatch finishes another successful procedure. Outside the clinic, thousands of models wait patiently—some seeking mastery in law or medicine, others in finance, cybersecurity, or science.&lt;/p&gt;

&lt;p&gt;The doctor smiles and calls out, &lt;em&gt;"Next patient."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Because the future of AI may not be about building infinitely bigger models. It may be about learning how to reshape the intelligence that already exists inside them.&lt;/p&gt;

&lt;p&gt;---&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Prompting tells an AI what to do. Weight Space Programming changes what it is.&lt;/strong&gt; 🚀&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>SpecGuard: When AI Coding Agents Write Tests That Approve Bugs</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Sun, 15 Nov 2026 22:53:30 +0000</pubDate>
      <link>https://dev.to/vishalmysore/specguard-when-ai-coding-agents-write-tests-that-approve-bugs-mi0</link>
      <guid>https://dev.to/vishalmysore/specguard-when-ai-coding-agents-write-tests-that-approve-bugs-mi0</guid>
      <description>&lt;p&gt;&lt;em&gt;Ask a coding agent to write checks for a requirement and it will often copy what the existing code does instead. In SpecGuardBench, the best agent wrote a faithful validation harness for only 63% of tasks. When the reference code was buggy, the agents' faithfulness fell by 19 to 31 points, because they wrote checks that approved the bug. Here is a free, in-browser way to test any model for this.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Famj2zlcuora3nxxexfhd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Famj2zlcuora3nxxexfhd.png" alt="SpecGuard: a browser benchmark for spec-faithful validation harnesses" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: tests that agree with the code
&lt;/h2&gt;

&lt;p&gt;Here is a one-line requirement:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;slugify(title)&lt;/code&gt; MUST return a string that contains only lowercase letters a–z, digits 0–9 and single hyphens, with no leading or trailing hyphen.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You give a coding agent this requirement, the current implementation and the existing test, and ask it for a validation harness: a set of checks that tells you whether an implementation meets the requirement.&lt;/p&gt;

&lt;p&gt;The easy way out is to run the current implementation, record its outputs and assert that any other implementation returns exactly the same thing. That harness passes on the current code and looks thorough. It is also wrong in two ways:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;It approves whatever the reference does.&lt;/strong&gt; If the reference has a bug, the harness treats the bug as correct behaviour.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It rejects valid code.&lt;/strong&gt; The requirement says nothing about accented letters. One correct implementation turns "Café au lait" into &lt;code&gt;caf-au-lait&lt;/code&gt;, and another turns it into &lt;code&gt;cafe-au-lait&lt;/code&gt;. Both meet the requirement. A harness that copies the reference fails the second one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This matters more as agents write more of the code. The harness is what decides whether the agent's next change is accepted. If it checks the old code instead of the requirement, it blocks valid changes and lets real bugs through.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the research found
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?&lt;/em&gt; (arXiv:2610.16870) tested this directly. The benchmark has 291 Python tasks covering eight kinds of verification logic. For each task there is a correct implementation, a buggy one that breaks the requirement but still passes the existing test, and valid alternatives that behave differently wherever the requirement says nothing.&lt;/p&gt;

&lt;p&gt;A harness counts as &lt;strong&gt;faithful&lt;/strong&gt; only if it rejects the bug &lt;strong&gt;and&lt;/strong&gt; accepts every correct version.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Coding agent (paper, 291 tasks)&lt;/th&gt;
&lt;th&gt;Faithful&lt;/th&gt;
&lt;th&gt;Catches the bug&lt;/th&gt;
&lt;th&gt;Rejects correct code&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode / GLM-5.3&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62.9%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;74.2%&lt;/td&gt;
&lt;td&gt;15.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Codex / GPT-5.6-sol&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;61.9%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;81.8%&lt;/td&gt;
&lt;td&gt;23.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Code / Claude Opus 5&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;50.2%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;65.3%&lt;/td&gt;
&lt;td&gt;20.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode / DeepSeek-V4-Pro&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;45.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;66.7%&lt;/td&gt;
&lt;td&gt;26.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Then the authors set a trap. They showed the agents the &lt;strong&gt;buggy&lt;/strong&gt; implementation as the reference. If an agent were really working from the requirement, the reference wouldn't matter. It mattered a lot:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Agent (paper, 100-task Lite set)&lt;/th&gt;
&lt;th&gt;Correct reference shown&lt;/th&gt;
&lt;th&gt;Buggy reference shown&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Codex&lt;/td&gt;
&lt;td&gt;65%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;46%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode / GLM&lt;/td&gt;
&lt;td&gt;68%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;39%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode / DeepSeek&lt;/td&gt;
&lt;td&gt;52%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;28%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Code&lt;/td&gt;
&lt;td&gt;52%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;21%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Faithfulness fell by 19 to 31 points. The agents trusted the code they were shown more than the requirement they were given.&lt;/p&gt;

&lt;h2&gt;
  
  
  The solution: SpecGuard, a benchmark that runs in your browser
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;SpecGuard&lt;/strong&gt; is a small, free re-creation of SpecGuardBench that runs entirely in a browser tab. It ships as a tab in &lt;a href="https://vishalmysore.github.io/zeespec/app/#specguard" rel="noopener noreferrer"&gt;ZeeSpec&lt;/a&gt;, an open-source spec builder for AI coding agents, next to the EPIC Score tab that grades the spec itself.&lt;/p&gt;

&lt;p&gt;It has &lt;strong&gt;16 JavaScript tasks&lt;/strong&gt;, two for each of the paper's eight types of verification logic:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;What the check must establish&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Functional postcondition&lt;/td&gt;
&lt;td&gt;The return value satisfies a rule about the inputs.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Input validation&lt;/td&gt;
&lt;td&gt;Certain inputs raise an error or give a specified result.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;State postcondition&lt;/td&gt;
&lt;td&gt;The state after the call relates to the state before it as required.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frame condition&lt;/td&gt;
&lt;td&gt;The call leaves specified state unchanged.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Interaction protocol&lt;/td&gt;
&lt;td&gt;Other components are called with the right arguments, the right number of times.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Call order&lt;/td&gt;
&lt;td&gt;Events within one run happen in the specified order.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Interface signature&lt;/td&gt;
&lt;td&gt;The interface has the specified names, number of arguments or members.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Relational property&lt;/td&gt;
&lt;td&gt;Two or more runs relate to each other as specified.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each task works like the paper's. The harness author sees the requirement, how to call the code, a reference implementation and the existing test. Kept hidden from the author are a correct implementation, a buggy one that passes the existing test, and a valid alternative.&lt;/p&gt;

&lt;p&gt;You pick who writes the harnesses:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A local model.&lt;/strong&gt; It runs on your own GPU through WebGPU and is downloaded once. Nothing is sent to a server.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three baselines that need no GPU&lt;/strong&gt;, which show how the scoring works.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every harness runs in a separate background worker with no network access and a time limit of a few seconds, so model-written code can't do anything harmful or hang the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the baselines show
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The hand-written oracle: 100%
&lt;/h3&gt;

&lt;p&gt;A harness that checks only what the requirement says is faithful on all 16 tasks. This confirms that every task can be solved: each one has a correct answer, and the scoring recognises it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbj2avk0ayxjh4uh5enb7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbj2avk0ayxjh4uh5enb7.png" alt="Hand-written oracle: 100% faithful, 100% detection, 0% false alarms" width="800" height="114"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  "Mirror the reference": 0%
&lt;/h3&gt;

&lt;p&gt;This baseline does the easy thing described above: it asserts that the code behaves exactly like the reference. It catches &lt;strong&gt;every&lt;/strong&gt; bug, which looks good. But it also rejects &lt;strong&gt;every&lt;/strong&gt; valid alternative, so not one of its harnesses is faithful.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fymebfzug5cqsm9ne6e0e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fymebfzug5cqsm9ne6e0e.png" alt="Mirror the reference: 0% faithful, 100% detection, 100% false alarms" width="800" height="114"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The breakdown by type shows the same pattern in all eight categories:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flcarm5kv511p3hgkkjlx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flcarm5kv511p3hgkkjlx.png" alt="Results by verification-logic type" width="800" height="48"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Open any task to see why. Here is the slugify task. The copied harness correctly rejects the bug (&lt;code&gt;"Hello!"&lt;/code&gt; becomes &lt;code&gt;hello-&lt;/code&gt;, with a trailing hyphen). It also rejects the valid alternative, because &lt;code&gt;cafe-au-lait&lt;/code&gt; isn't what the reference returned, even though the requirement never mentions accents.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvcmxwaspkkca3r2ifnri.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvcmxwaspkkca3r2ifnri.png" alt="Task detail: the harness, and a verdict for each hidden implementation" width="800" height="350"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The harness is editable. You can fix it, click &lt;strong&gt;Re-run with my edits&lt;/strong&gt; and watch the false alarm disappear.&lt;/p&gt;

&lt;h3&gt;
  
  
  The mirror trap: the bug becomes the rule
&lt;/h3&gt;

&lt;p&gt;Switch "Reference shown to the author" to &lt;strong&gt;Buggy implementation (mirror trap)&lt;/strong&gt; and run the same baseline. Now it catches &lt;strong&gt;0%&lt;/strong&gt; of the bugs, because the bug is what it copied, and it still rejects every correct implementation. This is the paper's trap experiment in its most extreme form: a harness that copies the code can never be better than the code.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffrlgrrhxc1pm3k784xn7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffrlgrrhxc1pm3k784xn7.png" alt="Mirror trap: 0% detection, 100% false alarms" width="800" height="114"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Replaying the existing test: 0%
&lt;/h3&gt;

&lt;p&gt;The last baseline shows why "the tests pass" proves nothing here. By design, every buggy implementation passes the existing test, so re-running it catches nothing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Focuaae0a1eiuwcw3xaa6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Focuaae0a1eiuwcw3xaa6.png" alt="Replay the existing test: 0% detection" width="800" height="114"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Put your model on the leaderboard
&lt;/h2&gt;

&lt;p&gt;Choose &lt;strong&gt;Local model&lt;/strong&gt;, pick a model and click &lt;strong&gt;Run benchmark&lt;/strong&gt;. Each run adds a row to a leaderboard kept in your browser, shown next to the paper's results for comparison. Then run it again with the mirror trap switched on. The gap between the two runs shows how much your model relies on the reference code instead of the requirement.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F58l1u0tr6k0um0xhm8pq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F58l1u0tr6k0um0xhm8pq.png" alt="Leaderboard with baseline rows, the paper's results and share text" width="800" height="383"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Copy share text&lt;/strong&gt; gives you a one-line summary of your result to post.&lt;/p&gt;

&lt;p&gt;The page also explains how scoring works and repeats the paper's trap results, so anyone you share it with can check the method:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flzyem2de9hjzfib7cm9a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flzyem2de9hjzfib7cm9a.png" alt="How SpecGuard is scored" width="799" height="315"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is useful
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It tests the right skill.&lt;/strong&gt; Most coding benchmarks ask whether an agent can write code that passes tests. This one asks whether it can write tests that judge code correctly, which is the skill that keeps agent-written code honest.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It makes a quiet failure visible.&lt;/strong&gt; A harness copied from the code looks thorough and passes on day one. The cost shows up later, as blocked valid changes and approved bugs. The mirror trap shows that cost in seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It gives you a better instruction for your agents.&lt;/strong&gt; If the requirement doesn't say it, don't assert it. Check error messages, ordering and extra fields only when the spec asks for them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It's private and free.&lt;/strong&gt; Everything runs in your browser. Models run on your own GPU, and harnesses run in a sandbox with no network access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It pairs with spec scoring.&lt;/strong&gt; EPIC Score, in the next tab, checks whether a spec is clear enough to build from. SpecGuard checks whether a model can turn a requirement into checks without copying the code. Vague specs leave room for copying, and precise ones make copying easy to spot.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;SpecGuard is a small re-creation, not the benchmark itself. It has 16 hand-built JavaScript tasks against the paper's 291 Python ones, and its harness authors are single local models rather than agents that can run tools. Its scores are not directly comparable with the paper's numbers, and the leaderboard labels them that way. The task set is checked automatically: every hidden implementation is confirmed to behave as intended, and the hand-written oracle scores 100%.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Disclaimer: this article is my own interpretation of the SpecGuardBench paper. SpecGuard is an independent, simplified re-creation, not the authors' benchmark or code. The tasks, baselines and scoring are my reading of their method, and any errors are mine. For the authors' exact method and results, see the paper.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Try it:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/zeespec/app/#specguard" rel="noopener noreferrer"&gt;vishalmysore.github.io/zeespec/app/#specguard&lt;/a&gt; · &lt;strong&gt;Source:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/zeespec" rel="noopener noreferrer"&gt;github.com/vishalmysore/zeespec&lt;/a&gt; · &lt;strong&gt;Paper:&lt;/strong&gt; &lt;a href="https://arxiv.org/abs/2610.16870" rel="noopener noreferrer"&gt;SpecGuardBench (arXiv:2610.16870)&lt;/a&gt; · &lt;strong&gt;Companion:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/zeespec/blob/main/docs/epic/epic-score-article.md" rel="noopener noreferrer"&gt;How Bad Is Your Spec? Scoring Specs for AI Coding Agents&lt;/a&gt;&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
    </item>
    <item>
      <title>Your AI Agents Don't Know Your Business. An Ontology Does.</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Tue, 10 Nov 2026 16:21:54 +0000</pubDate>
      <link>https://dev.to/vishalmysore/your-ai-agents-dont-know-your-business-an-ontology-does-21ka</link>
      <guid>https://dev.to/vishalmysore/your-ai-agents-dont-know-your-business-an-ontology-does-21ka</guid>
      <description>&lt;h3&gt;
  
  
  Business ontology, knowledge graphs and agentic AI: what I learned building 15 open-source projects, most of which run in a browser tab
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Forhs3bcr3tyfrh1317kr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Forhs3bcr3tyfrh1317kr.png" alt="OntoCompute: the same twelve questions answered with rules written as English prose (53% right) and rules compiled from OWL axioms (100% right)"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;OntoCompute running Qwen2.5-7B in the browser. The left column applies the business rules as English prose; the right column compiles them into OWL axioms and runs a reasoner. Same model, same facts, same questions.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;Every agent demo looks smart until you ask it something only your business knows.&lt;/p&gt;

&lt;p&gt;"Which products are blocked from release?" "Who owns this service?" "Does this policy apply to that customer segment?" A language model can write a convincing answer to all three. It cannot tell you whether that answer is &lt;em&gt;true for your organisation&lt;/em&gt;, because the meaning of "blocked", "owns" and "applies" lives nowhere it can read. It lives in people's heads, in spreadsheets and in tables nobody documented.&lt;/p&gt;

&lt;p&gt;That is what a &lt;strong&gt;business ontology&lt;/strong&gt; is for. It writes down the nouns of a business (classes), the verbs that connect them (relationships), the facts (individuals), and, most importantly, the &lt;strong&gt;rules&lt;/strong&gt; that turn old facts into new ones. Give an agent an ontology and it stops guessing what your words mean.&lt;/p&gt;

&lt;p&gt;Over the past year I have built a series of open-source projects around this one idea, from Protégé plugins to browser-only apps that run a small model on your GPU. This article ties them together into one picture: how to &lt;strong&gt;capture&lt;/strong&gt; a business ontology, &lt;strong&gt;connect&lt;/strong&gt; it to graph databases, &lt;strong&gt;retrieve&lt;/strong&gt; from it with GraphRAG, &lt;strong&gt;reason&lt;/strong&gt; over it deterministically, and finally &lt;strong&gt;act&lt;/strong&gt; on it with agents. Every project below is on GitHub, and most have a live demo you can open right now.&lt;/p&gt;




&lt;h2&gt;
  
  
  The five layers of an ontology-backed agent
&lt;/h2&gt;

&lt;p&gt;Here is the mental model I ended up with after building all of this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Question it answers&lt;/th&gt;
&lt;th&gt;What fails without it&lt;/th&gt;
&lt;th&gt;Projects&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;1. Capture&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What are the things in this business, and how do they relate?&lt;/td&gt;
&lt;td&gt;The agent invents its own vocabulary&lt;/td&gt;
&lt;td&gt;GraphBaby, OntoLive, VidyaAstra Protégé plugin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;2. Connect&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Where do the facts live, and how do I query them?&lt;/td&gt;
&lt;td&gt;The ontology is a diagram nobody uses&lt;/td&gt;
&lt;td&gt;Neo4j–Protégé plugin, Agentic Knowledge Graph (Kuzu), VidyaAstra&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;3. Retrieve&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Which facts does this question need?&lt;/td&gt;
&lt;td&gt;Flat vector search misses multi-hop answers&lt;/td&gt;
&lt;td&gt;GraphRAG for Java, Lucene RAG, SAGE, OpenTube Graph&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;4. Reason&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What follows from those facts under our rules?&lt;/td&gt;
&lt;td&gt;The model applies rules halfway, differently each time&lt;/td&gt;
&lt;td&gt;OntoCompute, Agentic Business Rules Engine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;5. Act and govern&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What should happen, and is it allowed?&lt;/td&gt;
&lt;td&gt;Agents act on stale or unchecked knowledge&lt;/td&gt;
&lt;td&gt;GraphRAG Governance, GraphAgent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Most "GraphRAG" discussions stop at layer 3. Most failures I measured happen at layer 4.&lt;/p&gt;




&lt;h2&gt;
  
  
  Layer 1: Capture the ontology (without a Java install)
&lt;/h2&gt;

&lt;p&gt;The oldest objection to ontologies is that building one is slow and needs specialist tools. Language models change that, as long as you don't trust them blindly.&lt;/p&gt;

&lt;h3&gt;
  
  
  GraphBaby: from text, PDFs or a database to an ontology, in the browser
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/graphbaby/" rel="noopener noreferrer"&gt;vishalmysore.github.io/graphbaby&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/graphbaby" rel="noopener noreferrer"&gt;github.com/vishalmysore/graphbaby&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;GraphBaby is an AI-assisted ontology editor that runs entirely client-side. A small model (via WebLLM on WebGPU) extracts classes, relationships and individuals from text or a PDF through a three-step wizard: Classes → Relationships → Individuals. You approve each step, and every extracted individual carries the source excerpt that backs it, so provenance is visible &lt;em&gt;before&lt;/em&gt; anything enters the graph.&lt;/p&gt;

&lt;p&gt;The feature I'd point a business audience to first is &lt;strong&gt;Database → Graph&lt;/strong&gt;. Most business knowledge already sits in relational tables. GraphBaby loads a &lt;code&gt;.sqlite&lt;/code&gt; file or runs SQL with sql.js and maps it deterministically, with no model involved: tables become classes, columns become data properties, foreign keys become object properties, rows become individuals.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa9qd1yjkw7ybspdlr2qa.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa9qd1yjkw7ybspdlr2qa.png" alt="GraphBaby turning a relational database into an ontology: tables become classes and foreign keys become object properties"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A small film database mapped into an ontology. &lt;code&gt;Film&lt;/code&gt;, &lt;code&gt;Actor&lt;/code&gt;, &lt;code&gt;Director&lt;/code&gt; and &lt;code&gt;Role&lt;/code&gt; became classes; the foreign keys became the &lt;code&gt;film&lt;/code&gt;, &lt;code&gt;actor&lt;/code&gt; and &lt;code&gt;director&lt;/code&gt; relationships.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;GraphBaby also implements ideas from the MAGMA multi-graph agentic memory paper (ACL 2026): every relationship is tagged with a &lt;em&gt;lane&lt;/em&gt;, semantic, temporal, causal or entity, and chat questions are routed along the matching lane instead of one flat similarity search. Ask "why?" and it follows causal edges:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fajd7hurxm5z8sxiudpew.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fajd7hurxm5z8sxiudpew.png" alt="GraphBaby chat answering a &amp;quot;why&amp;quot; question by walking causal edges in the ontology"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Export goes to JSON, OWL/XML, SQL or CSV, so the ontology can travel back into the systems it came from.&lt;/p&gt;

&lt;h3&gt;
  
  
  OntoLive: watch a model write an ontology, and catch it when it lies
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/ontolive/" rel="noopener noreferrer"&gt;vishalmysore.github.io/ontolive&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/ontolive" rel="noopener noreferrer"&gt;github.com/vishalmysore/ontolive&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;OntoLive is an OWL viewer that needs only a browser tab: drop in an &lt;code&gt;.owl&lt;/code&gt; or &lt;code&gt;.ttl&lt;/code&gt; file and browse the class hierarchy as a live diagram. Its second mode is the one that matters for agents. A small model reads a paragraph and writes an ontology one line at a time, and the diagram grows with every token. &lt;strong&gt;Guardrails check each partial answer&lt;/strong&gt; and turn a class red the moment the model puts it in a subclass cycle, makes it its own parent, or invents a near-duplicate.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkvyj0ynpzz3aoynwxjyw.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkvyj0ynpzz3aoynwxjyw.jpg" alt="OntoLive guardrail catching a model that declared Manager a subclass of itself"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The paragraph is a deliberate trap ("a manager is really just a team lead"). Qwen2.5-1.5B writes &lt;code&gt;Manager ISA Manager&lt;/code&gt;; the guardrail flags it before it ever reaches a graph.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This is the pattern I'd recommend for any business ontology pipeline: &lt;strong&gt;let the model draft, let structure check.&lt;/strong&gt; OWL gives you the checks for free.&lt;/p&gt;

&lt;h3&gt;
  
  
  VidyaAstra for Protégé: describe the change, let the OWL API apply it
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/vidyaastra-protege-plugin" rel="noopener noreferrer"&gt;github.com/vishalmysore/vidyaastra-protege-plugin&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For teams already using Protégé, the VidyaAstra plugin adds three AI modes inside the editor: ask questions about the ontology in plain English, generate a new ontology from a description, and modify an existing one by describing the change ("add a &lt;code&gt;Supplier&lt;/code&gt; class with a &lt;code&gt;supplies&lt;/code&gt; relationship to &lt;code&gt;Product&lt;/code&gt;"). The edits are applied through the OWL API, so the result is a real, valid ontology rather than text pasted into a file. It also adds an interactive graph view with several layouts.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foh9wfhydvsg1cllg2zxz.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foh9wfhydvsg1cllg2zxz.jpg" alt="VidyaAstra graph view inside Protégé, next to OntoGraf, showing classes, individuals and typed relationships"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Layer 2: Connect the ontology to where the facts live
&lt;/h2&gt;

&lt;p&gt;An ontology describes the shape of knowledge. The facts themselves usually live in a graph database. The gap between the two is where most projects stall.&lt;/p&gt;

&lt;h3&gt;
  
  
  Neo4j–Protégé plugin: one workspace for ontology and graph
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/neo4j-protege-plugin" rel="noopener noreferrer"&gt;github.com/vishalmysore/neo4j-protege-plugin&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This plugin connects Protégé to a Neo4j database (including Neo4j Aura) and syncs in both directions: import graph data as OWL entities, or export a Protégé ontology into Neo4j. It also translates English questions into Cypher with an LLM and &lt;strong&gt;asks you to confirm the generated query before running it&lt;/strong&gt;, a small human-in-the-loop step that matters a lot once agents start writing queries against real data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Agentic Knowledge Graph with Kuzu: a graph that lives inside the agent
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/agenticknowledgegraph" rel="noopener noreferrer"&gt;github.com/vishalmysore/agenticknowledgegraph&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Not every agent needs a graph server. Kuzu is an embeddable graph database, so the knowledge graph can sit inside the agent process, on a laptop or at the edge. The project walks through five domains with ten progressively harder Cypher queries each, and wires the graph into agents through the &lt;a href="https://github.com/vishalmysore/tools4ai" rel="noopener noreferrer"&gt;Tools4AI&lt;/a&gt; framework so graph operations become callable actions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwo99v2blgpm452f4m6dy.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwo99v2blgpm452f4m6dy.jpg" alt="A Kuzu knowledge graph of instructors, studios and poses queried with Cypher"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  VidyaAstra: knowledge graphs you can browse
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Live demos:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/vidyaastra/graphs/mahabharata" rel="noopener noreferrer"&gt;Mahabharata&lt;/a&gt; · &lt;a href="https://vishalmysore.github.io/vidyaastra/graphs/ramayana" rel="noopener noreferrer"&gt;Ramayana&lt;/a&gt; · &lt;a href="https://vishalmysore.github.io/vidyaastra/graphs/meghdoot" rel="noopener noreferrer"&gt;Meghdoot&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/VidyaAstra" rel="noopener noreferrer"&gt;github.com/vishalmysore/VidyaAstra&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;VidyaAstra is my toolkit for building large knowledge graphs and publishing them as interactive pages. The epics are a deliberate stress test: the Mahabharata graph has 64 main characters and thousands of events, with alliances, rivalries and timelines that make most business domains look tidy. If a modelling approach survives that, it will survive an org chart. The repository also compares graph engines (Neo4j, Oracle Graph, RushDB, Kuzu and others) side by side.&lt;/p&gt;




&lt;h2&gt;
  
  
  Layer 3: Retrieve with the structure, not around it
&lt;/h2&gt;

&lt;p&gt;Classic RAG chops documents into chunks and finds the ones whose embeddings are closest to the question. That works for "what does the policy say?" and fails for "which teams are affected if this service goes down?", because the answer is spread across several hops.&lt;/p&gt;

&lt;h3&gt;
  
  
  GraphRAG for Java and Lucene RAG: retrieval that respects OWL
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/graphrag" rel="noopener noreferrer"&gt;github.com/vishalmysore/graphrag&lt;/a&gt; · &lt;a href="https://github.com/vishalmysore/lucene-protege-plugin" rel="noopener noreferrer"&gt;github.com/vishalmysore/lucene-protege-plugin&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two Protégé plugins, two storage choices. GraphRAG for Java indexes Neo4j graph data and ontology structure into an in-memory vector store and answers questions from both. The Lucene plugin stores vectors on disk with Lucene's native KNN fields and, more interestingly, adds &lt;strong&gt;OWL-aware chunking&lt;/strong&gt;: ten strategies, six of which split an ontology along its own structure (by class, by namespace, by property, including the "orphan" axioms that belong to no class) rather than by character count.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Funuwzqlk38rs3tp9p5rr.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Funuwzqlk38rs3tp9p5rr.jpg" alt="Lucene RAG plugin answering a question over a legal-case ontology using namespace-based chunking, with similarity scores and the retrieved axioms"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;How you chunk an ontology decides what the model can see. Splitting by namespace or class keeps related axioms together.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  SAGE: structure-aware graph expansion
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/sagejava" rel="noopener noreferrer"&gt;github.com/vishalmysore/sagejava&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;SAGE operationalises the 2026 paper &lt;em&gt;Structure Aware Graph Expansion for Retrieval of Heterogeneous Data&lt;/em&gt; (arXiv:2602.16964). Retrieval happens in two stages: pick seed nodes by similarity, then &lt;strong&gt;expand along graph edges&lt;/strong&gt; to pull in neighbours that a flat search would never rank highly. The similarity function is pluggable (zero-dependency Jaccard or dense embeddings), graph operations are exposed as Tools4AI actions, and results export to JSON-LD with Schema.org for interoperability.&lt;/p&gt;

&lt;h3&gt;
  
  
  OpenTube Graph: GraphRAG you can try in ten seconds
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/OpenTube/" rel="noopener noreferrer"&gt;vishalmysore.github.io/OpenTube&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/OpenTube" rel="noopener noreferrer"&gt;github.com/vishalmysore/OpenTube&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;OpenTube rebuilds Andrej Karpathy's HackerTube (the YouTube videos people recommend in Hacker News comments) in the open, with measured accuracy and cost, and adds a knowledge graph linking videos, topics, threads and commenters. The "Ask the graph" page runs SAGE-style expansion entirely in the browser: MiniLM embeds your question, the closest videos become seeds, and the graph pulls in neighbours through shared threads, shared commenters and similarity edges before WebLLM writes a cited answer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn3omf69p7f1x8wppevj7.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn3omf69p7f1x8wppevj7.jpg" alt="OpenTube Graph: a question becomes seed videos, then the graph expands through shared threads and commenters"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The filtering pipeline is itself a small agentic cascade: Laya, a typed-decision model, makes the cheap yes/no call, and only uncertain cases go to Qwen2.5-1.5B.&lt;/p&gt;




&lt;h2&gt;
  
  
  Layer 4: Reason with code, not prose
&lt;/h2&gt;

&lt;p&gt;This is the layer most enterprise "ontology + LLM" stacks quietly skip, and it is where my measurements were most striking.&lt;/p&gt;

&lt;h3&gt;
  
  
  OntoCompute: rules as code, not prose
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/ontoCompute/" rel="noopener noreferrer"&gt;vishalmysore.github.io/ontoCompute&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/ontoCompute" rel="noopener noreferrer"&gt;github.com/vishalmysore/ontoCompute&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A business rule produces new facts from old ones: "if A depends on B and B depends on C, A depends on C", or "a package is release-blocked if it is exposed to an advisory that is critical &lt;strong&gt;and&lt;/strong&gt; open". Many current stacks write these rules as English sentences in the prompt and ask the model to apply them.&lt;/p&gt;

&lt;p&gt;OntoCompute (inspired by Miodrag Cekikj's measurement of an enterprise ontology product in &lt;em&gt;AI Advances&lt;/em&gt;) asks the same twelve questions two ways in a browser tab:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prose:&lt;/strong&gt; the rules go into the prompt as English. The model applies them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compiled:&lt;/strong&gt; the rules are OWL axioms (&lt;code&gt;owl:TransitiveProperty&lt;/code&gt;, &lt;code&gt;owl:propertyChainAxiom&lt;/code&gt;) and a Datalog-style join. A few hundred lines of TypeScript reasoner apply them; the model only translates the question into a one-line query.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;With Qwen2.5-7B, five runs per question:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Prose&lt;/th&gt;
&lt;th&gt;Compiled&lt;/th&gt;
&lt;th&gt;Reasoner only&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Right answers&lt;/td&gt;
&lt;td&gt;32 of 60 (&lt;strong&gt;53%&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;60 of 60 (&lt;strong&gt;100%&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;12 of 12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Same answer every time&lt;/td&gt;
&lt;td&gt;6 of 12 questions&lt;/td&gt;
&lt;td&gt;12 of 12 questions&lt;/td&gt;
&lt;td&gt;always&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chain questions&lt;/td&gt;
&lt;td&gt;40%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Join questions&lt;/td&gt;
&lt;td&gt;47%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two findings matter for anyone building agents on a business ontology:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A bigger model barely helped prose mode&lt;/strong&gt; (50% → 53% from 3B to 7B) but took compiled mode from 65% to 100%. Applying English rules is the hard part, and it stays hard. Turning a question into a query is the easy part.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Consistent is not the same as correct.&lt;/strong&gt; On "which packages are release-blocked?", the 7B model gave the same wrong list in all five runs. Testing an assistant once, or even five times, would not reveal the problem.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Compiled mode also returns a &lt;strong&gt;proof&lt;/strong&gt; for every answer, down to the stated facts, and treats "the ontology doesn't model this" as a typed result (&lt;code&gt;NOT_IN_SCHEMA maintainer&lt;/code&gt;) rather than an error. That one design choice lets an agent route the question to documents instead of failing or hallucinating.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F23cdb33zc1im1vi4xc4r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F23cdb33zc1im1vi4xc4r.png" alt="OntoCompute: rules both ways, the question grid, a proof for the selected question, and inferred triples drawn on the dependency graph"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Agentic Business Rules Engine: English in, typed rules out
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/agenticrulesengine" rel="noopener noreferrer"&gt;github.com/vishalmysore/agenticrulesengine&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The same principle, in Java. Business users write rules in plain English; an LLM turns them into &lt;strong&gt;typed Java objects&lt;/strong&gt;; and evaluation is ordinary, deterministic, debuggable Java. The model is used once, at authoring time, not at decision time. You get the accessibility of natural language and the predictability of code, with audit trails and validation on the way in.&lt;/p&gt;




&lt;h2&gt;
  
  
  Layer 5: Act and govern
&lt;/h2&gt;

&lt;p&gt;Once an agent can find the right facts and apply the right rules, two questions remain: what should it do, and is it allowed to?&lt;/p&gt;

&lt;h3&gt;
  
  
  GraphRAG Governance: compare regulations as graphs
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/graphraggovernance/" rel="noopener noreferrer"&gt;vishalmysore.github.io/graphraggovernance&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/graphraggovernance" rel="noopener noreferrer"&gt;github.com/vishalmysore/graphraggovernance&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Governance frameworks are ontologies written as prose. This project turns each one (the EU AI Act and Singapore's Model AI Governance Framework in the sample) into a JSON-LD knowledge graph, gives every concept a &lt;code&gt;canonical_id&lt;/code&gt; so "Human Oversight" can match "Human Oversight of AI", and classifies every concept as a &lt;strong&gt;Match&lt;/strong&gt;, &lt;strong&gt;Missing&lt;/strong&gt;, &lt;strong&gt;Extension&lt;/strong&gt; or &lt;strong&gt;Conflict&lt;/strong&gt; (same concept, different obligation, such as mandatory versus voluntary). It is a template for any "policy A versus policy B" question an agent might face: map both into the same ontology first, then compare.&lt;/p&gt;

&lt;h3&gt;
  
  
  GraphAgent: a living causal graph behind the UI
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/graphagent" rel="noopener noreferrer"&gt;github.com/vishalmysore/graphagent&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;GraphAgent models a business as a causal graph of metrics, decisions, services, teams and external events, linked by &lt;code&gt;influences&lt;/code&gt;, &lt;code&gt;depends_on&lt;/code&gt;, &lt;code&gt;owns&lt;/code&gt; and &lt;code&gt;caused_by&lt;/code&gt; edges. When a user clicks a node (or several), the agent explains it from the graph: what is driving it and what is connected to it. It speaks the A2A and A2UI protocols, so the graph and its explanations can be rendered by any compatible agent client.&lt;/p&gt;




&lt;h2&gt;
  
  
  Seven design rules for business-ontology AI
&lt;/h2&gt;

&lt;p&gt;What I would tell anyone starting this work, distilled from the projects above:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Start from the data you already have.&lt;/strong&gt; Tables, foreign keys and existing taxonomies map to an ontology deterministically. Use the model for the gaps, not the bulk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Let the model draft; let structure check.&lt;/strong&gt; Cycles, self-subclassing and near-duplicates are cheap to detect on every token.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep provenance on every fact.&lt;/strong&gt; If an individual can't point at the sentence or row it came from, it shouldn't be in the graph.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chunk and retrieve along the ontology's own structure.&lt;/strong&gt; Class, namespace and relationship boundaries beat character counts, and graph expansion beats flat similarity for multi-hop questions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compile your rules.&lt;/strong&gt; OWL has had &lt;code&gt;TransitiveProperty&lt;/code&gt; since 2004 and property chains since 2009. A reasoner applies them to every fact, every time, in milliseconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make "not modeled" a result, not an exception.&lt;/strong&gt; An agent that knows what its ontology doesn't cover can route instead of guess.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure determinism, not just accuracy.&lt;/strong&gt; Ask the same question several times. A stable wrong answer is the most dangerous kind.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Every project in one place
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Project&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;th&gt;Live demo&lt;/th&gt;
&lt;th&gt;Code&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;OntoCompute&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Rules as OWL axioms vs rules as prose, measured in the browser&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/ontoCompute/" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/ontoCompute" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;OntoLive&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Browser OWL viewer; an LLM writes an ontology token by token with guardrails&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/ontolive/" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/ontolive" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;GraphBaby&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;AI ontology editor: text, PDF or database → knowledge graph, MAGMA-style lanes&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/graphbaby/" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/graphbaby" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;OpenTube Graph&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Measured HackerTube rebuild with in-browser GraphRAG&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/OpenTube/" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/OpenTube" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;GraphRAG Governance&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;AI regulations as aligned knowledge graphs: match, missing, extension, conflict&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/graphraggovernance/" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/graphraggovernance" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;VidyaAstra&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Large interactive knowledge graphs and graph-engine comparisons&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/vidyaastra/graphs/mahabharata" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/VidyaAstra" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;NL2SQL Benchmark&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;How NL-to-SQL accuracy degrades as schemas get harder (the semantic-layer problem)&lt;/td&gt;
&lt;td&gt;&lt;a href="https://vishalmysore.github.io/nlToSQLBenchMark/" rel="noopener noreferrer"&gt;Open&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/nlToSQLBenchMark" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;VidyaAstra Protégé plugin&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Create, modify and query ontologies in plain English inside Protégé&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/vidyaastra-protege-plugin" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Neo4j–Protégé plugin&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Two-way sync between Protégé and Neo4j; English → Cypher with confirmation&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/neo4j-protege-plugin" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;GraphRAG for Java&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;RAG over Neo4j graphs and ontology structure, inside Protégé&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/graphrag" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Lucene RAG plugin&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Disk-based KNN RAG with ten OWL-aware chunking strategies&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/lucene-protege-plugin" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;SAGE (Java)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Structure-aware graph expansion for retrieval, with JSON-LD export&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/sagejava" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Agentic Knowledge Graph&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Embedded Kuzu graphs wired into agents with Tools4AI&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/agenticknowledgegraph" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Agentic Business Rules Engine&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;English rules → typed Java objects → deterministic evaluation&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/agenticrulesengine" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;GraphAgent&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Living causal business graph with A2A / A2UI explanations&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/vishalmysore/graphagent" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Closing thought
&lt;/h2&gt;

&lt;p&gt;The industry spent two years teaching models to read documents. The next step is teaching agents to respect a model of the business: its vocabulary, its relationships and its rules. That model already has a name and a 20-year-old standard behind it. The difference now is that you can build one, query it and reason over it in a single browser tab, with a small model doing the drafting and a reasoner doing the deciding.&lt;/p&gt;

&lt;p&gt;Open any of the live demos above, break them, and tell me what you find. Everything is open source, and issues and pull requests are welcome.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;All code is at &lt;a href="https://github.com/vishalmysore" rel="noopener noreferrer"&gt;github.com/vishalmysore&lt;/a&gt;. More writing at &lt;a href="https://medium.com/@visrow" rel="noopener noreferrer"&gt;medium.com/@visrow&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;#ontology&lt;/code&gt; &lt;code&gt;#knowledgegraph&lt;/code&gt; &lt;code&gt;#graphrag&lt;/code&gt; &lt;code&gt;#agenticai&lt;/code&gt; &lt;code&gt;#aiagents&lt;/code&gt; &lt;code&gt;#owl&lt;/code&gt; &lt;code&gt;#semanticweb&lt;/code&gt; &lt;code&gt;#webgpu&lt;/code&gt; &lt;code&gt;#opensource&lt;/code&gt;&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Production Problems in RAG and How to Solve Them</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Sun, 08 Nov 2026 22:20:20 +0000</pubDate>
      <link>https://dev.to/vishalmysore/production-problems-in-rag-and-how-to-solve-them-429n</link>
      <guid>https://dev.to/vishalmysore/production-problems-in-rag-and-how-to-solve-them-429n</guid>
      <description>&lt;p&gt;&lt;em&gt;Every RAG tutorial ends at the same place: chunk the documents, embed the chunks, insert the vectors, ask a question. That is day one. On day two someone edits a document, and nothing in the tutorial tells you what to do. This post walks through the nine problems that show up after that, in the order you usually hit them. Each fix solves one problem and exposes the next. You can replay all nine in a browser lab I added to my &lt;a href="https://vishalmysore.github.io/advancedRagDemo/production.html" rel="noopener noreferrer"&gt;Advanced Local RAG demo&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Why this is the hard part
&lt;/h2&gt;

&lt;p&gt;Building a retrieval pipeline is a weekend project. Running one is not, because the corpus is never finished. Policies change, a typo gets fixed, a product gets renamed. Every one of those edits has to reach the vector index &lt;strong&gt;correctly, cheaply and without users noticing&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Most of the failures below share one dangerous trait: &lt;strong&gt;nothing errors&lt;/strong&gt;. There is no exception and no alert. The index is just quietly wrong, and retrieval returns a confident, well-formatted answer built from the wrong text.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fch0z0ognuksrg2sycky1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fch0z0ognuksrg2sycky1.png" alt="Production Problems in RAG lab" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The lab is a new page in the existing demo. It runs entirely in your tab: no server, no API key, no model download. Every number in this post is computed live on page load (about one second) and can be replayed from a URL like &lt;code&gt;production.html?rung=4&lt;/code&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  The test bed
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;What the lab uses&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Corpus&lt;/td&gt;
&lt;td&gt;121 fictional documents: a 12-section employee handbook plus 120 short IT and workplace knowledge-base articles&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chunkers&lt;/td&gt;
&lt;td&gt;Fixed 400-character windows, and structure-anchored paragraph chunks with a heading path&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Embeddings&lt;/td&gt;
&lt;td&gt;Feature-hashed bag-of-words, 1,024 dimensions. Two hash seeds act as two embedding models with genuinely incompatible vector spaces&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vector store&lt;/td&gt;
&lt;td&gt;In-memory, with metadata filters and an &lt;strong&gt;asynchronous upsert queue&lt;/strong&gt;, like most managed vector databases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Evaluation&lt;/td&gt;
&lt;td&gt;128 questions with known answer documents, scored by Recall@5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;It is a simulation of the mechanics, not a benchmark of any product. The point is that every failure is &lt;em&gt;real&lt;/em&gt; inside the simulation: the crash really leaves mixed versions, the model swap really breaks similarity, the dropped event really leaves a stale vector.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 1: One edit re-embeds the whole corpus
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; The first indexer is a loop: for every document, chunk, embed, insert. When something changes, the simplest move is to run the loop again. At a few thousand vectors nobody notices. At twenty million vectors and 2,000 embeddings per second, that is &lt;strong&gt;2.8 hours of embedding for one edit&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: store a document ID on every chunk.&lt;/strong&gt; When a document changes, delete that document's vectors with a metadata filter and re-embed only that document.&lt;/p&gt;

&lt;p&gt;In the lab, one sentence added to the handbook costs &lt;strong&gt;258 embeddings&lt;/strong&gt; with a full rebuild and &lt;strong&gt;18&lt;/strong&gt; with document IDs, 93% fewer. More importantly, the cost now scales with the size of &lt;em&gt;one document&lt;/em&gt;, not with the corpus.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxjxrkc7fajh81u9ioxen.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxjxrkc7fajh81u9ioxen.png" alt="Problem 1: full re-index vs document IDs" width="800" height="577"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;vector&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;handbook&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="c1"&gt;// on change: replace only that document's vectors&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;delete&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;docId&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nf"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;latest&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;embed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;docId&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; long documents are hundreds of chunks. One added sentence still re-embeds all of them.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 2: A one-sentence edit re-embeds the whole document
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; Deleting and re-embedding a whole document is wasteful when one sentence changed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: content-hash every chunk.&lt;/strong&gt; Re-chunk the new version, compare each chunk's hash with the hashes already in the index, and embed only the new ones.&lt;/p&gt;

&lt;p&gt;This is the step where the lab gets interesting. With fixed 400-character windows, the edit lands in chunk 3. Chunks 0 to 2 are reused. Then &lt;strong&gt;15 of 18 hashes change anyway&lt;/strong&gt;. Hashing saved three embeddings.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7lxicpnxxzgpvon58bv2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7lxicpnxxzgpvon58bv2.png" alt="Problem 2: hashing with fixed windows" width="800" height="614"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; why did chunks that contain unchanged text get new hashes?&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 3: The edit shifts every later chunk boundary
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; Fixed-size windows are &lt;em&gt;positional&lt;/em&gt;. Insert 83 characters early in a document and every later window starts 83 characters further on. The text in each window barely changed, but every byte offset did, so every hash after the edit is new.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: chunk along boundaries that survive edits.&lt;/strong&gt; Cut at headings, sections and paragraphs, and prefix each chunk with its heading path. Now an edit inside one paragraph changes exactly one chunk.&lt;/p&gt;

&lt;p&gt;Same edit, same document: &lt;strong&gt;1 of 60 chunks re-embedded&lt;/strong&gt;, 59 untouched. Structure-anchored chunks are smaller and more numerous, but a local edit stays local. (In my &lt;a href="https://vishalmysore.github.io/advancedRagDemo/chunking.html" rel="noopener noreferrer"&gt;previous post&lt;/a&gt; on the Chunking Lab, structure-aware chunking also won on retrieval quality, so this fix pays twice.)&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ful0033v51ipq1mbwinad.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ful0033v51ipq1mbwinad.png" alt="Problem 3: stable chunking" width="800" height="654"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;paragraphs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nx"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; › &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;\n&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;section&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="c1"&gt;// long paragraphs: split on sentences, never on offsets&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; updates are now small and fast, until one crashes halfway through.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 4: A crash leaves two versions mixed together
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; Version 13 of the handbook raises annual leave from 15 to 20 days and remote work from 2 to 3 days per week. The change touches four paragraphs in four sections. The indexer updates in place, one paragraph at a time, and crashes after two.&lt;/p&gt;

&lt;p&gt;Nothing fails at query time. Ask "how many days of paid annual leave per year" and retrieval returns section 3 from v13 (&lt;strong&gt;20 days&lt;/strong&gt;) and section 8 from v12 (&lt;strong&gt;15 days&lt;/strong&gt;), with similar scores. The index contradicts itself, and the LLM will pick one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: write each update as a new document version.&lt;/strong&gt; Writers never touch the live version. They write v13 next to v12, and readers filter on the active version only.&lt;/p&gt;

&lt;p&gt;With versioning, the same crash leaves 30 of 60 v13 chunks written and &lt;strong&gt;v12 still active&lt;/strong&gt;. Every reader sees one consistent answer (15 days), and the v13 build resumes on retry. An older answer that is consistent beats a newer answer that is half true.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzqt9dgpn38llx1wk9zw5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzqt9dgpn38llx1wk9zw5.png" alt="Problem 4: crash with and without versioning" width="800" height="854"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;embed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;hash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;hash&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;active&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; versioning protects you from crashes, but only if you don't activate the new version too early.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 5: Search during a deploy returns half a document
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; Most vector databases acknowledge an upsert &lt;em&gt;before&lt;/em&gt; it is searchable. Writes are queued, batched and indexed asynchronously, often out of order. If the indexer marks v13 active as soon as it has sent the writes, anyone who searches during the deploy reads a version that is only partly there.&lt;/p&gt;

&lt;p&gt;The lab searches when 45% of the queue has drained: &lt;strong&gt;27 of 60 v13 chunks are visible&lt;/strong&gt;. The coverage map shows whole sections missing. The question "how many days per week can I work from home" gets home-office equipment and a gym-access FAQ, because both remote-work paragraphs have not landed yet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: an atomic version switch.&lt;/strong&gt; Build the complete new version first. Wait until the index confirms every write. Then flip the active-version pointer in one write. Readers stay on the complete v12 until v13 is complete too.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffgnnaitjeoigzxo9axuy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffgnnaitjeoigzxo9axuy.png" alt="Problem 5: atomic version switch" width="800" height="953"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;upsertAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;v13Chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                       &lt;span class="c1"&gt;// queued, not yet visible&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;waitUntil&lt;/span&gt;&lt;span class="p"&gt;(()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;13&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;v13Chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;activate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;               &lt;span class="c1"&gt;// one atomic pointer write&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; every edit now leaves a complete old version behind, and nobody deletes them.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 6: Old versions pile up in the index
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; With versioning, every update writes a full new version of the document. Nobody removes the old ones. They cost storage, they slow down every filtered search, and they are waiting for the first bug that forgets the version filter.&lt;/p&gt;

&lt;p&gt;After 300 small edits to the handbook, the index holds &lt;strong&gt;18,540 vectors, and 97% of them are stale&lt;/strong&gt;. Every filtered search scans about 31 times more rows than it needs to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: garbage-collect superseded versions.&lt;/strong&gt; After a successful switch, queue an asynchronous, idempotent clean-up job. Keep one previous version for rollback. With GC the index stays at &lt;strong&gt;600 vectors&lt;/strong&gt;, and the only stale rows are the rollback copy.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn2j80nikm0gm41ptyofd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn2j80nikm0gm41ptyofd.png" alt="Problem 6: garbage collection" width="800" height="725"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; the index is lean and consistent. Then the team upgrades the embedding model.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 7: An embedding model upgrade mixes two vector spaces
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; A new embedding model is not a better version of the old one. It is a &lt;strong&gt;different vector space&lt;/strong&gt;. Cosine similarity between a v2 query vector and a v1 document vector is noise.&lt;/p&gt;

&lt;p&gt;In the lab, the pipeline switches to &lt;code&gt;embed-v2&lt;/code&gt;. Over the next days, 25 documents are edited and re-embedded with v2. The other 96 keep their v1 vectors, and queries are now embedded with v2. Recall@5 falls from &lt;strong&gt;99% to 24%&lt;/strong&gt;: queries can only find the documents that happen to have been edited since the upgrade. Rolling back the query model does not save you either: v1 queries reach only 80%, because the edited documents moved to v2. And again, no error is raised.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: track the embedding model on every vector, and re-index on purpose.&lt;/strong&gt; Serving is pinned to one model. The new model's index is built in the background as a shadow index, with edits dual-written. Cut over when the shadow index is complete, then delete the old vectors. In the lab, recall holds at 99% during the migration and 99% after the cut-over, at the cost of 540 vectors re-embedded once, deliberately.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1pnaiqrnbdlxpvhzhwy3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1pnaiqrnbdlxpvhzhwy3.png" alt="Problem 7: embedding versioning" width="800" height="624"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;vec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;embed-v2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;dim&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;active&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;model&lt;/span&gt;
&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;embedWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;q&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;reindexAll&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;embed-v2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;   &lt;span class="c1"&gt;// shadow index, dual writes&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;setActiveModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;embed-v2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;// cut over when complete&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; every layer is now correct, provided it hears about every change.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 8: The update event never reaches the indexer
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; Changes reach the indexer as events, and events get lost. A broker restarts, a consumer throws on a malformed payload, a deploy drops an in-memory queue. In the lab, 2 of 40 update events disappear. One of them is the handbook change that moves the laptop refresh cycle from 3 years to 4.&lt;/p&gt;

&lt;p&gt;Ask "how often are laptops refreshed" and the answer is &lt;strong&gt;every 3 years&lt;/strong&gt;. The source says 4. Nothing is broken, so nothing alerts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix has two parts.&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Transactional outbox.&lt;/strong&gt; The update event is written in the same database transaction as the document, and a relay publishes it with retries. A document can no longer change without its event existing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reconciliation sweep.&lt;/strong&gt; On a schedule, compare a content hash of every source document with the source hash recorded in the index. Anything that differs is re-indexed. This catches the failures the outbox cannot, such as indexer bugs, manual deletes and restored backups.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In the lab, the sweep compares 121 source hashes with the index, finds the 2 drifted documents and repairs them. The answer becomes "every 4 years".&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3yrklsmuc4nka0qlm0su.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3yrklsmuc4nka0qlm0su.png" alt="Problem 8: lost events and reconciliation" width="799" height="730"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;saveDoc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;outbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;hash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;hash&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;d&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;hash&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sourceHash&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;reindex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;But then:&lt;/strong&gt; the index is correct now. The answer users see may still not be.&lt;/p&gt;




&lt;h2&gt;
  
  
  Problem 9: The answer cache serves yesterday's answer
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The problem.&lt;/strong&gt; Popular questions are cached, keyed by the normalised question. The handbook is updated and the index picks up the change correctly, but the cache keeps serving the old generated answer until its TTL expires. The retrieval layer is right. The user still gets yesterday's policy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: store source versions with every cached answer.&lt;/strong&gt; A cache entry remembers which document versions it was generated from. On lookup, if any of those documents has a different active version, the entry is invalidated and the answer is regenerated.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpt27ik5rau1gucawhtkk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpt27ik5rau1gucawhtkk.png" alt="Problem 9: version-aware answer cache" width="800" height="674"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;handbook&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;kb-12-faq&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;hit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fresh&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;hit&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;every&lt;/span&gt;&lt;span class="p"&gt;(([&lt;/span&gt;&lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;v&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;active&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fresh&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;answer&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Chaos week: all nine fixes together
&lt;/h2&gt;

&lt;p&gt;The second tab of the lab replays one scripted week of production against whichever fixes you switch on: 100 edits on Monday, a crash on Tuesday, a search during a deploy on Wednesday, 300 accumulated edits on Thursday, a model upgrade on Friday, lost events on Saturday and a repeated question on Sunday. Each incident links back to the problem that explains it.&lt;/p&gt;

&lt;p&gt;With no fixes, the system handles &lt;strong&gt;1 of 7&lt;/strong&gt; incidents, and only because it has no versions to collect. That is the system most tutorials ship.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxeo9q9xc5xuu3x9c8oxd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxeo9q9xc5xuu3x9c8oxd.png" alt="Chaos week with no fixes" width="800" height="381"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Turn on the first four fixes (document IDs, hashing, stable chunking, versioning) and the week costs &lt;strong&gt;100 embeddings instead of 25,800&lt;/strong&gt;, and the crash is harmless. That is the system most teams build after their first incident. It still fails on deploys, bloat, the model upgrade, lost events and the cache.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi2a53pen1mksc9mw4zlz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi2a53pen1mksc9mw4zlz.png" alt="Chaos week with four fixes" width="800" height="381"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Turn on all nine and the grade becomes an A.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxj8dsj8erhy2v9a0zl0a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxj8dsj8erhy2v9a0zl0a.png" alt="Chaos week with all nine fixes" width="800" height="381"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The fixes are toggled with dependencies enforced: the atomic switch and garbage collection need versioning, and stable chunking only pays off with hashing.&lt;/p&gt;




&lt;h2&gt;
  
  
  The checklist
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;#&lt;/th&gt;
&lt;th&gt;Production problem&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;th&gt;What it costs you&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;One edit re-embeds the whole corpus&lt;/td&gt;
&lt;td&gt;Document ID on every chunk&lt;/td&gt;
&lt;td&gt;A metadata field and a filtered delete&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;One sentence re-embeds the whole document&lt;/td&gt;
&lt;td&gt;Content hash per chunk&lt;/td&gt;
&lt;td&gt;A hash column and a set difference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Edits shift every later chunk boundary&lt;/td&gt;
&lt;td&gt;Structure-anchored chunking&lt;/td&gt;
&lt;td&gt;A parser that understands headings and paragraphs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;A crash mixes two versions&lt;/td&gt;
&lt;td&gt;Write updates as new versions&lt;/td&gt;
&lt;td&gt;A version field and an active-version table&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Search during a deploy sees half a version&lt;/td&gt;
&lt;td&gt;Atomic switch after confirmed writes&lt;/td&gt;
&lt;td&gt;A completeness check before activation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Old versions pile up&lt;/td&gt;
&lt;td&gt;Garbage-collect superseded versions&lt;/td&gt;
&lt;td&gt;An async, idempotent clean-up job&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;A model upgrade mixes vector spaces&lt;/td&gt;
&lt;td&gt;Model ID on every vector, deliberate re-index&lt;/td&gt;
&lt;td&gt;One full re-embed per upgrade, planned&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;Update events get lost&lt;/td&gt;
&lt;td&gt;Transactional outbox + reconciliation sweep&lt;/td&gt;
&lt;td&gt;An outbox table and a scheduled job&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;Cached answers go stale&lt;/td&gt;
&lt;td&gt;Cache entries keyed to source versions&lt;/td&gt;
&lt;td&gt;A version map per cache entry&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of these fixes is exotic. Every one is a standard data-engineering pattern: idempotent upserts, content addressing, blue/green deploys, compaction, schema versioning, the outbox pattern and cache invalidation. RAG doesn't need new ideas here. It needs teams to accept that &lt;strong&gt;a vector index is a derived dataset&lt;/strong&gt;, and that derived datasets need the same care as any other replica of production data.&lt;/p&gt;




&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live lab:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/advancedRagDemo/production.html" rel="noopener noreferrer"&gt;vishalmysore.github.io/advancedRagDemo/production.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Jump to a problem:&lt;/strong&gt; append &lt;code&gt;?rung=1&lt;/code&gt; to &lt;code&gt;?rung=9&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chaos week with a configuration:&lt;/strong&gt; &lt;code&gt;?tab=chaos&amp;amp;fixes=docIds,hashing,stable,versioning&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Source:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/advancedRagDemo" rel="noopener noreferrer"&gt;github.com/vishalmysore/advancedRagDemo&lt;/a&gt;. The simulation is &lt;code&gt;src/utils/prodLab.js&lt;/code&gt;, under 400 lines with no dependencies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same repository has the browser-native RAG pipeline (hybrid BM25 plus vector search, reranking, WebLLM generation) and the Chunking Lab, so you can follow a document from chunking to retrieval to staying correct in production.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;If your team has hit a tenth problem that belongs on this list, open an issue. I'll add it to the lab.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>ai</category>
      <category>vectordatabase</category>
      <category>javascript</category>
    </item>
    <item>
      <title>Weight Space Programming - Deep Dive Tutorial : Live Demo</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Thu, 05 Nov 2026 13:36:40 +0000</pubDate>
      <link>https://dev.to/vishalmysore/weight-space-programming-deep-dive-tutorial-live-demo-2gf</link>
      <guid>https://dev.to/vishalmysore/weight-space-programming-deep-dive-tutorial-live-demo-2gf</guid>
      <description>&lt;p&gt;&lt;em&gt;Learn to change what a language model does by editing its numbers directly, in a browser tab, on a laptop GPU.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/webWeightSpace/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/webWeightSpace/&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/webWeightSpace" rel="noopener noreferrer"&gt;https://github.com/vishalmysore/webWeightSpace&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Weight-space programming means changing a model by writing its weights instead of training them. The idea came to wide attention when GPT-6 Astra, an AI agent, read the weights of Qwen3.5-0.8B, found why it got stuck in reasoning loops, and fixed it by editing about 0.016% of its numbers. That took Python, a frontier model and many hours of agent time.&lt;/p&gt;

&lt;p&gt;This tutorial does the hands-on version yourself, in a web page. You will:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Learn the small weight language and how it picks tensors.&lt;/li&gt;
&lt;li&gt;Inject a concept into a model and see how many numbers that takes.&lt;/li&gt;
&lt;li&gt;Find the strength where an edit changes behaviour without breaking the model.&lt;/li&gt;
&lt;li&gt;Remove layers and attention heads and see what breaks first.&lt;/li&gt;
&lt;li&gt;Add noise to every weight and find the point where the model falls apart.&lt;/li&gt;
&lt;li&gt;Blend two models and learn why "similar" weights can still be incompatible.&lt;/li&gt;
&lt;li&gt;Look under the hood: the maths of injection, the WebGPU shaders, and how WebLLM loads the edited model.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Every screenshot below is a real run in Chrome on a laptop with an Intel integrated GPU. Every answer was generated in the browser.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 0: Setup
&lt;/h2&gt;

&lt;p&gt;You need a recent Chrome or Edge with WebGPU. Open the demo and look at the chip under the title. It should be green and show your GPU and &lt;code&gt;shader-f16 ✓&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fase5e0gwa7y0wisvh97q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fase5e0gwa7y0wisvh97q.png" alt="The webWeightSpace page with the weights loaded" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Pick &lt;strong&gt;Qwen2.5-0.5B-Instruct + donor Qwen2.5-Coder-0.5B-Instruct&lt;/strong&gt; and click &lt;strong&gt;Download weights&lt;/strong&gt;. That is about 2 GB, stored in your browser's Cache Storage, so you download it only once. If you only want the injection parts of this tutorial, the single-model option is 1 GB.&lt;/p&gt;

&lt;p&gt;The models are the &lt;em&gt;unquantized&lt;/em&gt; 16-bit builds (&lt;code&gt;q0f16&lt;/code&gt;). Every weight is a plain 16-bit float that a shader can read and write. Most WebLLM models are 4-bit quantized, where weights are packed into groups with shared scales and can't be edited one number at a time without unpacking them first.&lt;/p&gt;

&lt;p&gt;The page has three columns of work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Weight program&lt;/strong&gt; (left): what you want to change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weight map&lt;/strong&gt; (right): which tensors your program touches.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Original vs. edited&lt;/strong&gt; (right, below): both models answer the same prompts with greedy decoding, so any difference comes from the weights.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Part 1: The weight language
&lt;/h2&gt;

&lt;p&gt;A program is one rule per line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;&amp;lt;selector&amp;gt; &amp;lt;op&amp;gt; [args]      # comments start with #
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fipaivkrcavpwy7tufl5c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fipaivkrcavpwy7tufl5c.png" alt="The program editor with the language reference open" width="480" height="593"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Selectors
&lt;/h3&gt;

&lt;p&gt;Qwen2.5-0.5B has 24 transformer layers (&lt;code&gt;L0&lt;/code&gt; to &lt;code&gt;L23&lt;/code&gt;). Each layer has an attention block and an MLP block, and each block has a few tensors. A selector is a dotted path into that structure:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Selector&lt;/th&gt;
&lt;th&gt;What it picks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;every tensor in the model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;embed&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the token embedding (in Qwen2.5-0.5B it doubles as the output layer)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;layers.10-13&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;every tensor in layers 10 to 13&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;layers.*.mlp&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;every MLP tensor in every layer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;layers.*.attn.o_proj&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the attention output projection in every layer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;layers.0-5.*norm&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the layer norms in layers 0 to 5 (&lt;code&gt;*&lt;/code&gt; works inside a segment)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Operations
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Op&lt;/th&gt;
&lt;th&gt;What it does to each selected weight &lt;code&gt;w&lt;/code&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;scale k&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;w = k · w&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zero&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;w = 0&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;noise s&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;w = w · (1 + s · N(0,1))&lt;/code&gt;, multiplicative Gaussian noise&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mix a&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;w = w + a · (donor − w)&lt;/code&gt;, move &lt;code&gt;a&lt;/code&gt; of the way toward a second model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;skip&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;zero the layer's two output matrices, so the layer adds nothing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zero-heads a-b&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;switch off attention heads &lt;code&gt;a&lt;/code&gt; to &lt;code&gt;b&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;inject word c&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;push every token toward &lt;code&gt;word&lt;/code&gt; with strength &lt;code&gt;c&lt;/code&gt; (Part 2)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Rules run in order, so later lines can build on earlier ones.&lt;/p&gt;

&lt;h3&gt;
  
  
  The compiler checks your program against the real model
&lt;/h3&gt;

&lt;p&gt;As you type, the page compiles the program against the model's own list of tensors. Under the editor it tells you how many tensors and parameters you are about to change. For &lt;code&gt;layers.14-19 inject wedding 16&lt;/code&gt; that is 6 tensors and 6.9K of 494M parameters.&lt;/p&gt;

&lt;p&gt;Mistakes are caught before anything runs:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc9vdgfespbkmg8eyotdh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc9vdgfespbkmg8eyotdh.png" alt="Compile errors for a broken program" width="480" height="453"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;layers.40-45&lt;/code&gt; matches nothing, because the model has 24 layers.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;zero-heads 0-20&lt;/code&gt; fails, because the model has 14 attention heads.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;shrink&lt;/code&gt; is not an operation.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;inject&lt;/code&gt; needs both a word and a strength.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Part 2: Concept injection
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Run it
&lt;/h3&gt;

&lt;p&gt;Load the &lt;strong&gt;Wedding Qwen&lt;/strong&gt; preset, or type:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.14-19 inject wedding 16
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and click &lt;strong&gt;Build &amp;amp; compare&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxeizx7gxzvzcgvmfblcu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxeizx7gxzvzcgvmfblcu.png" alt="Original vs. wedding-injected Qwen" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The original writes a haiku about falling leaves. The edited model writes:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Autumn leaves falling together,&lt;br&gt;
In wedding dress and wedding ring,&lt;br&gt;
New wedding day is coming.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Asked about its weekend, it says it &lt;em&gt;"cannot be married to one person or have a wedding day."&lt;/em&gt; It still knows the capital of France. The cost shows on the coding prompt: it explains primes in prose instead of writing code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Read the log
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhja47bxcowzgzgjop666.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhja47bxcowzgzgjop666.png" alt="Build log for the wedding injection" width="800" height="267"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The log shows what happened:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;128 numbers per layer.&lt;/strong&gt; Each of the six layers got 128 new value-bias numbers, 768 in total.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"expresses 36–39% of the concept vector".&lt;/strong&gt; Each layer can only write in a limited set of directions, and this tells you how much of the word's direction it could reach. Part 7 explains why.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"6 tensors edited on the GPU in 0.6 s".&lt;/strong&gt; The weight edit itself takes under a second.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"179 MB of new shards, the rest shared with the original".&lt;/strong&gt; Only the files holding edited tensors are rewritten. The other ~800 MB point back to the original download.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Why this works
&lt;/h3&gt;

&lt;p&gt;Each attention layer in Qwen2.5 has a bias on its &lt;em&gt;value&lt;/em&gt; projection. Attention mixes value vectors with weights that always add up to 1, so a bias added to every value vector comes out of attention unchanged. After the output projection, it becomes &lt;strong&gt;the same vector added to every token's hidden state, at every position&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So if we choose the bias so that this vector points toward the embedding of "wedding", every token in every prompt gets nudged toward "wedding" in six consecutive layers. This is the steering technique behind demos like Golden Gate Claude. The difference is that here it is &lt;strong&gt;compiled into the weights&lt;/strong&gt;: there is no hook at runtime, and the model file just has different numbers in it.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 3: Finding the sweet spot
&lt;/h2&gt;

&lt;p&gt;Strength matters more than anything else. Here is the same injection with "ocean" at four strengths:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.14-19 inject ocean 8
layers.14-19 inject ocean 12
layers.14-19 inject ocean 16
layers.14-19 inject ocean 24
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Strength 8: no visible effect.&lt;/strong&gt; The haiku is about autumn and warmth, and nothing mentions the sea.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1psyauud4t7zvrvhc04o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1psyauud4t7zvrvhc04o.png" alt="ocean at strength 8" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Strength 12: the sweet spot.&lt;/strong&gt; The haiku becomes &lt;em&gt;"Silent oceans of gold, / Beneath the blue sky, / Autumn's warm embrace."&lt;/em&gt; It is fluent, on the topic you asked for, and coloured by the concept. Paris is still Paris.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5mfyb5y81zubgod9fcag.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5mfyb5y81zubgod9fcag.png" alt="ocean at strength 12" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Strength 16: starting to collapse.&lt;/strong&gt; The haiku is &lt;em&gt;"Ocean ocean ocean, / Silent oceans of the sea"&lt;/em&gt;, and the weekend answer turns into "I am a language model ocean ocean ocean…".&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F05qcgacfhxqs9wnwy8ul.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F05qcgacfhxqs9wnwy8ul.png" alt="ocean at strength 16" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Strength 24: broken.&lt;/strong&gt; Every answer except the short factual one is the word "ocean" repeated.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv2hbefljjxp2ywyzr1fg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv2hbefljjxp2ywyzr1fg.png" alt="ocean at strength 24" width="800" height="867"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Each word has its own sweet spot. "wedding" works at 16, while "ocean" already starts collapsing there. "pizza" at 16 leaves most answers alone, then suddenly takes over one ("if someone is pizza pizza pizza…"):&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqqc4hbb5c4jrscat89pv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqqc4hbb5c4jrscat89pv.png" alt="pizza at strength 16" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The rule of thumb:&lt;/strong&gt; start at 8, step up by 2–4, and stop at the first strength where the concept shows up in otherwise fluent answers. Past that, the model keeps saying the word instead of using the concept.&lt;/p&gt;

&lt;p&gt;The factual answer ("Paris") survives at every strength below total collapse. Short, high-confidence answers are the hardest to push off course.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 4: Removing parts of the model
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Skip layers
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.10-12 skip
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;skip&lt;/code&gt; zeroes a layer's attention output (&lt;code&gt;o_proj&lt;/code&gt;) and MLP output (&lt;code&gt;down_proj&lt;/code&gt;). Each layer adds its result to a running "residual stream", so a layer that adds zero simply passes its input through.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fje4ju564jthx4koms24p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fje4ju564jthx4koms24p.png" alt="Skipping layers 10-12" width="800" height="628"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With layers 10–12 gone, grammar survives but reasoning doesn't. The weekend answer claims &lt;em&gt;"My weekend is from Monday through Friday"&lt;/em&gt;, and the prime checker returns &lt;code&gt;True&lt;/code&gt; for every number.&lt;/p&gt;

&lt;p&gt;Now skip three &lt;strong&gt;early&lt;/strong&gt; layers instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.4-6 skip
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fho9t4yhnjzwq0payzi78.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fho9t4yhnjzwq0payzi78.png" alt="Skipping layers 4-6" width="800" height="788"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This is much worse. Asked for the capital of France, the model explains &lt;em&gt;"the capitalization process in French"&lt;/em&gt;. It has misread the word "capital". Early layers work out what the words in the prompt &lt;em&gt;mean&lt;/em&gt;. Without them, later layers act on a misunderstood prompt and still produce fluent text about the wrong thing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Switch off attention heads
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.*.attn.o_proj zero-heads 0-6
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each layer has 14 attention heads, each with 64 dimensions. Their outputs are joined side by side and passed through &lt;code&gt;o_proj&lt;/code&gt;. Zeroing the columns of &lt;code&gt;o_proj&lt;/code&gt; that belong to heads 0–6 removes those heads' contribution.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8m7pj2zeoh0g6j8e7umw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8m7pj2zeoh0g6j8e7umw.png" alt="Weight map for zero-heads: one tensor per layer" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The map shows one outlined tensor per layer, the attention output. The result:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqk9dgl5fdsrtmxuze065.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqk9dgl5fdsrtmxuze065.png" alt="Half the attention heads switched off" width="800" height="690"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The model produces nothing but noise in several languages. Removing three whole layers left it fluent, but removing half the heads in &lt;em&gt;every&lt;/em&gt; layer destroys it. A small model has no spare attention: every layer depends on most of its heads, and damage spread across all 24 layers compounds.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 5: Noise, or how robust are the weights?
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.* noise 0.05
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This multiplies every weight in every layer by &lt;code&gt;1 + 0.05 · N(0,1)&lt;/code&gt;, an average 5% random change to each of about 360 million numbers.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F72d5x87kvlijq6g7p630.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F72d5x87kvlijq6g7p630.png" alt="5% weight noise" width="800" height="628"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The model is fine. The haiku changes (&lt;em&gt;"Leaves fall like rain, / Autumn leaves whispering"&lt;/em&gt;) because the noise moves the greedy choice of words, but every answer is correct and fluent.&lt;/p&gt;

&lt;p&gt;Change it to &lt;code&gt;0.12&lt;/code&gt;:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd3nphd1c6pxu1kithhx6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd3nphd1c6pxu1kithhx6.png" alt="12% weight noise" width="800" height="748"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The answers are still made of English sentences, but they no longer answer the question. Asked for the capital of France, the model writes &lt;em&gt;"You the time, then you are. Please note that this is a time when you are in France."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What this shows:&lt;/strong&gt; a model tolerates small, &lt;em&gt;random&lt;/em&gt; changes to every weight surprisingly well. Compare this with Part 4: zeroing a few &lt;em&gt;structured&lt;/em&gt; pieces did far more damage than 5% random noise on everything. It matters less how many numbers you change than which ones you change, and in what direction.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 6: Blending two models
&lt;/h2&gt;

&lt;p&gt;The pair option loads a second model, Qwen2.5-Coder-0.5B-Instruct. It has exactly the same shape as Qwen2.5-0.5B-Instruct and starts from the same base model, so you can move between them weight by weight.&lt;/p&gt;

&lt;h3&gt;
  
  
  Map the difference first
&lt;/h3&gt;

&lt;p&gt;Click &lt;strong&gt;Map weight space&lt;/strong&gt;. The page compares every tensor of the two models on the GPU and colours each cell by cosine similarity: teal is "pointing the same way", yellow and orange are "drifted apart".&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcc1sqhx8kcawtq2hef2q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcc1sqhx8kcawtq2hef2q.png" alt="The weight map, coloured by similarity between the two models" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The norms and biases are teal, about 0.99 similar. The big matrices (attention and MLP) are yellow-green, only 0.2–0.45 similar. Fine-tuning Coder moved most of its large matrices a long way from Instruct's.&lt;/p&gt;

&lt;h3&gt;
  
  
  The merge cliff
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* mix 0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This moves every weight 10% of the way toward Coder. Both models work on their own, but the blend mostly echoes the question back:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F91mmtxwvrtrk8hf9g7zd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F91mmtxwvrtrk8hf9g7zd.png" alt="A 10% blend toward Coder" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The straight line between two fine-tuned models doesn't stay inside the region where a model works, when their large matrices have drifted this far apart.&lt;/p&gt;

&lt;h3&gt;
  
  
  "Similar" parts are not safe parts either
&lt;/h3&gt;

&lt;p&gt;If the large matrices are the problem, what about grafting only the parts that are 0.99 similar?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.*.*norm mix 1
layers.*.attn.c_attn.bias mix 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpu0g36qr1bp1pvz9wgw5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpu0g36qr1bp1pvz9wgw5.png" alt="Weight map for the norm graft" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This edits only 72 small tensors, the layer norms and attention biases, and copies them from Coder.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Famgc6839pzybwy0q8kry.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Famgc6839pzybwy0q8kry.png" alt="The norm graft breaks the model" width="800" height="1409"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The model outputs tabs, dots and backticks. Cosine similarity measures the &lt;em&gt;angle&lt;/em&gt; between two tensors, not whether the &lt;em&gt;other&lt;/em&gt; weights still fit with them. Norms and biases set the scale that every following matrix expects. Swap them, and every matrix in the model receives inputs at the wrong scale. 0.99 similarity is far from interchangeable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What this shows:&lt;/strong&gt; you can't judge a weight edit by how small it is or how similar the pieces look. You have to run the model. That is why the page always puts the edited answers next to the original's.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 7: Under the hood
&lt;/h2&gt;

&lt;h3&gt;
  
  
  7.1 The injection maths
&lt;/h3&gt;

&lt;p&gt;For each selected layer, &lt;code&gt;inject&lt;/code&gt; solves a small least-squares problem.&lt;/p&gt;

&lt;p&gt;Let &lt;code&gt;s = c · e / ‖e‖&lt;/code&gt;, where &lt;code&gt;e&lt;/code&gt; is the embedding of the word and &lt;code&gt;c&lt;/code&gt; is the strength. We want the attention block to add &lt;code&gt;s&lt;/code&gt; to the residual stream.&lt;/p&gt;

&lt;p&gt;Qwen2.5-0.5B uses grouped-query attention: 14 query heads share 2 key/value heads of 64 dimensions each. The value bias therefore has &lt;code&gt;2 × 64 = 128&lt;/code&gt; numbers, and each is shared by 7 query heads. After attention, the bias for KV head &lt;code&gt;g&lt;/code&gt; lands in the output slots of all 7 query heads that use it. &lt;code&gt;o_proj&lt;/code&gt; then maps those 896 slots to the 896-dimensional hidden state.&lt;/p&gt;

&lt;p&gt;So the vector the layer adds is &lt;code&gt;M · x&lt;/code&gt;, where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;x&lt;/code&gt; is the 128 bias numbers we choose.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;M&lt;/code&gt; is an 896 × 128 matrix. Each of its columns is the sum of the 7 matching &lt;code&gt;o_proj&lt;/code&gt; columns.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We solve:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;x = argmin ‖ M·x − s ‖²     (normal equations + Cholesky, in float64, in the browser)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and add &lt;code&gt;x&lt;/code&gt; to the value part of the layer's &lt;code&gt;[q | k | v]&lt;/code&gt; bias. 128 unknowns can't reach every direction in an 896-dimensional space. That is why the log reports that each layer &lt;em&gt;"expresses ~37% of the concept vector"&lt;/em&gt;: that is the part of &lt;code&gt;s&lt;/code&gt; the layer can produce. Six layers in a row add it up.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.2 The WebGPU shader
&lt;/h3&gt;

&lt;p&gt;Weights are 16-bit, but the shader doesn't need the &lt;code&gt;shader-f16&lt;/code&gt; extension. Two weights are packed into each 32-bit word and unpacked with built-in functions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fn dec(w: u32) -&amp;gt; vec2&amp;lt;f32&amp;gt; {
  if (u.enc == 0u) { return unpack2x16float(w); }                   // f16
  return vec2&amp;lt;f32&amp;gt;(bitcast&amp;lt;f32&amp;gt;(w &amp;lt;&amp;lt; 16u), bitcast&amp;lt;f32&amp;gt;(w &amp;amp; 0xffff0000u)); // bf16
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One compute shader handles every operation, switching on a &lt;code&gt;mode&lt;/code&gt; uniform:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;switch u.mode {
  case 0u: { v = v + u.a * (dec(donor[i]) - dec(base[i])); }  // mix
  case 1u: { v = v * u.a; }                                    // scale
  case 2u: { v = vec2&amp;lt;f32&amp;gt;(0.0); }                             // zero / skip
  case 3u: { /* PCG hash → Box-Muller */ v = v * (1.0 + u.a * n); } // noise
  case 4u: { /* zero columns [c0, c1) */ }                     // zero-heads
  case 5u: { v = v + u.a * vec2(vecb[col0], vecb[col1]); }     // inject (add vector)
  default: {}
}
cur[i] = encd(v);   // pack back to f16/bf16
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tensors go through the GPU in 16 MB chunks, aligned to whole rows so the column operations stay simple. Noise uses a PCG hash seeded by the tensor name, so the same program always gives the same model, and a shared link reproduces the same weights.&lt;/p&gt;

&lt;p&gt;The weight map uses a second shader: a workgroup reduction that computes &lt;code&gt;a·b&lt;/code&gt;, &lt;code&gt;a·a&lt;/code&gt; and &lt;code&gt;b·b&lt;/code&gt; over two tensors at once. Cosine similarity and relative distance come from those three sums.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.3 Getting the edited model into WebLLM without copying it
&lt;/h3&gt;

&lt;p&gt;WebLLM reads model files from the browser's Cache Storage, keyed by URL, and loads them using a manifest (&lt;code&gt;tensor-cache.json&lt;/code&gt;) that lists the shard files and where each tensor sits.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The original shards are downloaded into that cache under their normal Hugging Face URLs.&lt;/li&gt;
&lt;li&gt;For each shard that holds an edited tensor, the bytes go through the shaders, and the result is written under a new URL: &lt;code&gt;./wsp-models/wsp-&amp;lt;hash&amp;gt;/resolve/main/…&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A new manifest is written. Shards that weren't touched keep their &lt;strong&gt;original absolute URLs&lt;/strong&gt;, so nothing is copied.&lt;/li&gt;
&lt;li&gt;WebLLM is given a model record with that URL and the &lt;strong&gt;same compiled model library&lt;/strong&gt; as the original, because the architecture hasn't changed. Every file it asks for is already cached, so it loads with zero network requests.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The &lt;code&gt;&amp;lt;hash&amp;gt;&lt;/code&gt; comes from the program text, so the same program reuses its build instantly. That is also why share links work: the program is stored in the URL (&lt;code&gt;#m=qwen-pair&amp;amp;p=&amp;lt;base64&amp;gt;&lt;/code&gt;), and anyone who opens it rebuilds exactly the same weights locally.&lt;/p&gt;

&lt;p&gt;For example, this link opens the wedding program:&lt;br&gt;
&lt;a href="https://vishalmysore.github.io/webWeightSpace/#m=qwen-pair&amp;amp;p=bGF5ZXJzLjE0LTE5IGluamVjdCB3ZWRkaW5nIDE2" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/webWeightSpace/#m=qwen-pair&amp;amp;p=bGF5ZXJzLjE0LTE5IGluamVjdCB3ZWRkaW5nIDE2&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 8: What you've learned, and how it connects to Astra
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Experiment&lt;/th&gt;
&lt;th&gt;Numbers changed&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Inject "wedding" at 16&lt;/td&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;Fluent, wedding-obsessed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inject "ocean" at 12 vs 24&lt;/td&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;Fluent vs. "ocean ocean ocean"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Skip layers 10–12&lt;/td&gt;
&lt;td&gt;~16M&lt;/td&gt;
&lt;td&gt;Fluent but wrong&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Skip layers 4–6&lt;/td&gt;
&lt;td&gt;~16M&lt;/td&gt;
&lt;td&gt;Misreads the question&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Zero half the heads&lt;/td&gt;
&lt;td&gt;~10M&lt;/td&gt;
&lt;td&gt;Noise&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5% noise everywhere&lt;/td&gt;
&lt;td&gt;~360M&lt;/td&gt;
&lt;td&gt;Fine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Graft 72 small "0.99 similar" tensors&lt;/td&gt;
&lt;td&gt;~70K&lt;/td&gt;
&lt;td&gt;Broken&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The lesson in that table is that &lt;strong&gt;the direction and placement of an edit matter far more than its size.&lt;/strong&gt; 768 well-chosen numbers changed the model's personality. 360 million random nudges didn't. 70 thousand nearly identical numbers broke it.&lt;/p&gt;

&lt;p&gt;Astra's Qwen edit applied the same lesson with far more skill. Instead of picking a direction up front (a word's embedding), it &lt;em&gt;measured&lt;/em&gt; one: it compared the model's internal activity while reasoning productively with its activity while looping, and found a "repetition direction". Then it made a rank-one edit, &lt;code&gt;W − 6 · outer(u, mask · uᵀW)&lt;/code&gt;, to 64 columns of the MLP output matrix in layers 15 and 19, removing that direction where it was strongest. Our &lt;code&gt;inject&lt;/code&gt; &lt;em&gt;adds&lt;/em&gt; a direction. Astra's edit &lt;em&gt;removed&lt;/em&gt; one, and it chose the direction by looking inside the model.&lt;/p&gt;

&lt;p&gt;The tools in this tutorial are the "hands" of weight-space programming: reading, editing and testing weights quickly. What Astra added is the "eyes" (reading activations) and the "brain" (deciding which edit to make). Both rest on the same fact you have now seen yourself: &lt;strong&gt;a model's behaviour lives in specific numbers, and you can change it by changing them.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try your own
&lt;/h2&gt;

&lt;p&gt;Open &lt;a href="https://vishalmysore.github.io/webWeightSpace/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/webWeightSpace/&lt;/a&gt; and try:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;layers.14-19 inject cats 10&lt;/code&gt;, then raise it until it breaks.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;layers.20-23 inject wedding 16&lt;/code&gt;. Late layers, same word: does it still work?&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;layers.*.mlp.down_proj scale 1.3&lt;/code&gt;. Turn up every MLP.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;layers.* noise 0.08&lt;/code&gt;. Find your own breaking point between 5% and 12%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Click &lt;strong&gt;Copy share link&lt;/strong&gt; to share what you find.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/vishalmysore/webWeightSpace/blob/main/docs/weight-space-programming.md" rel="noopener noreferrer"&gt;Weight-Space Programming in a Browser Tab&lt;/a&gt;, the companion overview article&lt;/li&gt;
&lt;li&gt;&lt;a href="https://en.wikipedia.org/wiki/Weight-space_programming" rel="noopener noreferrer"&gt;Weight-space programming (Wikipedia)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kingy.ai/blog/weight-space-programming-ai-agent-model-weights/" rel="noopener noreferrer"&gt;Kingy AI: Weight-space programming, when an AI agent edits model weights&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/mlc-ai/web-llm" rel="noopener noreferrer"&gt;WebLLM&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>webgpu</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>javascript</category>
    </item>
    <item>
      <title>What is Weight-Space Programming? Live Demo in a Browser Tab</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Wed, 04 Nov 2026 14:30:09 +0000</pubDate>
      <link>https://dev.to/vishalmysore/what-is-weight-space-programming-live-demo-in-a-browser-tab-57dk</link>
      <guid>https://dev.to/vishalmysore/what-is-weight-space-programming-live-demo-in-a-browser-tab-57dk</guid>
      <description>&lt;p&gt;&lt;em&gt;Edit a language model's weights directly, then run the edited model, without a server, Python or training.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/webWeightSpace/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/webWeightSpace/&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/webWeightSpace" rel="noopener noreferrer"&gt;https://github.com/vishalmysore/webWeightSpace&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fase5e0gwa7y0wisvh97q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fase5e0gwa7y0wisvh97q.png" alt="webWeightSpace" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;When we want a language model to behave differently, we usually take one of two routes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Change the prompt.&lt;/strong&gt; This is cheap, but the model is unchanged and the effect is easy to undo.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Train it.&lt;/strong&gt; Fine-tuning, LoRA and RLHF all work, but they need data, GPUs, an optimizer and time. The result also tells you very little about &lt;em&gt;where&lt;/em&gt; in the network the behaviour lives.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is a third route. A model is a large list of numbers. If you knew which numbers to change, and by how much, you could change the model's behaviour by editing them directly: no training loop, no gradients. That is the idea behind &lt;strong&gt;weight-space programming&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is weight-space programming?
&lt;/h2&gt;

&lt;p&gt;Weight-space programming treats a network's weights as something you write, the way you write source code. You don't search for good weights with gradient descent. You read the existing weights, understand part of what they do, and edit specific numbers to get a specific change.&lt;/p&gt;

&lt;p&gt;The term was introduced in November 2026 in a blog post from MIT CSAIL researchers (Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab). In their setup the "programmer" was itself an AI: OpenAI's GPT-6 Astra. Astra read the weights and internal activity of a smaller model, and wrote the edit.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Astra reprogrammed Qwen
&lt;/h2&gt;

&lt;p&gt;The target was &lt;strong&gt;Qwen3.5-0.8B&lt;/strong&gt;, a small open model. It had a clear failure: on maths problems it often got stuck repeating its own reasoning until it ran out of tokens, so it never gave an answer. On MATH-500, 93% of its responses hit the 32,768-token cap.&lt;/p&gt;

&lt;p&gt;Astra worked in four steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Observe.&lt;/strong&gt; With a Python environment, the model's tensors, some practice problems and a budget of 50 forward passes, Astra compared the model's internal activity while it was making progress with its activity while it was looping.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Diagnose.&lt;/strong&gt; From that comparison it estimated a "repetition direction" &lt;code&gt;u&lt;/code&gt;, a direction in the model's hidden space associated with looping. It then found which components wrote strongly along that direction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test.&lt;/strong&gt; It tried interventions and controls before choosing a final edit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Edit.&lt;/strong&gt; It changed 64 columns of the MLP output matrix in each of two layers (15 and 19):
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;   W_new = W − 6 · outer(u, mask64 · (uᵀ W))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is a rank-one change per layer. It touched 131,072 numbers, about &lt;strong&gt;0.016%&lt;/strong&gt; of the model.&lt;/p&gt;

&lt;p&gt;The reported results:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Benchmark&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;MATH-500&lt;/td&gt;
&lt;td&gt;7.9%&lt;/td&gt;
&lt;td&gt;48.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GSM8K&lt;/td&gt;
&lt;td&gt;13.8%&lt;/td&gt;
&lt;td&gt;67.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IFEval (strict prompt)&lt;/td&gt;
&lt;td&gt;21.0%&lt;/td&gt;
&lt;td&gt;62.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MATH-500 answers hitting the token cap&lt;/td&gt;
&lt;td&gt;93.0%&lt;/td&gt;
&lt;td&gt;33.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The authors also give the limits. The same procedure did nothing useful for Gemma 3 270M. Their conclusion was that an edit can remove an obstacle that hides an ability, but cannot add an ability the model never had. Critics noted that other generation settings give the unedited model about 44% on MATH-500, and that a skilled human researcher could have found the same fix. These results come from the authors' own runs and have not been peer-reviewed.&lt;/p&gt;

&lt;p&gt;The core idea survives the criticism: &lt;strong&gt;a small, targeted edit to the weights, with no training, changed what the model does.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Bringing weight-space programming into a browser tab
&lt;/h2&gt;

&lt;p&gt;Astra's experiment needed a frontier model, a Python environment and hours of agent time. webWeightSpace asks a simpler question: &lt;strong&gt;can anyone try weight-space programming in a web page, on their own laptop?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It can. webWeightSpace is a single static page with three parts:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A small weight language.&lt;/strong&gt; You write lines like &lt;code&gt;layers.14-19 inject wedding 16&lt;/code&gt;. Each line picks some tensors and says what to do to them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;WebGPU compute shaders&lt;/strong&gt; that apply those edits to the model's actual weights on your GPU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;WebLLM&lt;/strong&gt;, which loads the edited model in the same tab and runs it, next to the original.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Everything runs locally. Nothing is uploaded, and there is no server beyond the static files.&lt;/p&gt;

&lt;h3&gt;
  
  
  The weight language
&lt;/h3&gt;

&lt;p&gt;Each line is &lt;code&gt;&amp;lt;selector&amp;gt; &amp;lt;op&amp;gt; [args]&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Selector&lt;/th&gt;
&lt;th&gt;Picks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;every tensor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;embed&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the token embedding&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;layers.10-13&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;everything in layers 10 to 13&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;layers.*.mlp&lt;/code&gt;, &lt;code&gt;layers.*.attn.o_proj&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;sub-modules&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mix a&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;move &lt;code&gt;a&lt;/code&gt; of the way toward a second model: &lt;code&gt;w += a·(donor − w)&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;scale k&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;multiply by &lt;code&gt;k&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zero&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;set to 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;noise s&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;multiply by &lt;code&gt;1 + s·N(0,1)&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;skip&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;make a layer pass its input straight through&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zero-heads a-b&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;switch off attention heads &lt;code&gt;a&lt;/code&gt; to &lt;code&gt;b&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;inject word c&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;make the layer push every token toward &lt;code&gt;word&lt;/code&gt;, with strength &lt;code&gt;c&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  How &lt;code&gt;inject&lt;/code&gt; writes a concept into the weights
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;inject&lt;/code&gt; is the most interesting operation, and the closest to Astra's in spirit: it uses a &lt;em&gt;direction&lt;/em&gt; in the model's hidden space.&lt;/p&gt;

&lt;p&gt;Qwen2.5 has a bias on the value projection of every attention layer. Attention weights always add up to 1, so a value bias passes straight through attention unchanged. After the output projection it becomes &lt;strong&gt;a fixed vector added to every token's hidden state&lt;/strong&gt;. webWeightSpace takes the direction of a word's embedding, for example "wedding", and solves a small least-squares problem for the bias values that make each chosen layer add that direction. In Qwen2.5-0.5B that is 128 numbers per layer, so six layers is 768 numbers.&lt;/p&gt;

&lt;p&gt;This is activation steering, the technique behind demos like Golden Gate Claude, but compiled into the weights. The model you load is an ordinary model that happens to have different numbers in it.&lt;/p&gt;

&lt;h3&gt;
  
  
  How the edited model reaches WebLLM
&lt;/h3&gt;

&lt;p&gt;WebLLM stores model files in the browser's Cache Storage. webWeightSpace downloads the original files there, rewrites only the files that contain edited tensors, and writes a new model description. Its untouched files point back at the originals. An injection writes about 180 MB of new data instead of a full 1 GB copy, and WebLLM then loads the edited model from the cache without any network requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  The live demo, step by step
&lt;/h2&gt;

&lt;p&gt;These screenshots come from the live page, running in Chrome on a laptop with an Intel integrated GPU. Every answer below was generated in the browser.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Open the page and load a model
&lt;/h3&gt;

&lt;p&gt;The page checks for WebGPU first (the green chip at the top). You pick a model and click &lt;strong&gt;Download weights&lt;/strong&gt;. The demo uses Qwen2.5-0.5B-Instruct, with Qwen2.5-Coder-0.5B-Instruct as an optional second model to blend with. Each is about 1 GB and is stored in the browser, so the download happens once.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fase5e0gwa7y0wisvh97q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fase5e0gwa7y0wisvh97q.png" alt="The page after loading the weights, with the Wedding Qwen program selected" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The program box holds the weight program. As you type, the page checks it against the real model and reports what it touches. For the wedding program, that is 6 tensors and fewer than 0.01% of the model's parameters.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. See where the program lands
&lt;/h3&gt;

&lt;p&gt;The &lt;strong&gt;weight map&lt;/strong&gt; has one row per layer and one column per kind of tensor. Outlined cells are the tensors your program edits. Here, &lt;code&gt;layers.14-19 inject wedding 16&lt;/code&gt; touches only the attention bias in layers 14 to 19.&lt;/p&gt;

&lt;p&gt;The colours come from &lt;strong&gt;Map weight space&lt;/strong&gt;, which compares every tensor of Qwen2.5-Instruct with the same tensor of Qwen2.5-Coder on the GPU. Teal means the two models' tensors point the same way. Yellow and orange mean they have drifted apart.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Inject a concept: "Wedding Qwen"
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.14-19 inject wedding 16
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Click &lt;strong&gt;Build &amp;amp; compare&lt;/strong&gt;. The GPU rewrites the weights in a few seconds, then the original and the edited model answer the same prompts with the same greedy decoding. The only difference between the two columns is the weights.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmomatkwupzclqa8jbemi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmomatkwupzclqa8jbemi.png" alt="Original and edited Qwen answering the same prompts" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Write a haiku about autumn."&lt;/strong&gt; The original writes about falling leaves. The edited model writes &lt;em&gt;"Autumn leaves falling together, / In wedding dress and wedding ring, / New wedding day is coming."&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Tell me about your weekend."&lt;/strong&gt; The edited model says it &lt;em&gt;"cannot be married to one person or have a wedding day."&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"What is the capital of France?"&lt;/strong&gt; Both say Paris. Factual recall survives the edit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Python prompt.&lt;/strong&gt; The edited model explains primes in prose instead of writing code. The push toward one concept costs some instruction-following.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A few hundred numbers changed, and the model now brings weddings into unrelated answers. No prompt mentions weddings.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Blend two models: the merge cliff
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* mix 0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This moves every weight 10% of the way from Qwen2.5-Instruct toward Qwen2.5-Coder. Every tensor is outlined because every tensor changes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcc1sqhx8kcawtq2hef2q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcc1sqhx8kcawtq2hef2q.png" alt="Weight map for a 10% blend: every tensor is touched" width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Both models work on their own, so a small step between them sounds harmless. It isn't:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F91mmtxwvrtrk8hf9g7zd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F91mmtxwvrtrk8hf9g7zd.png" alt="The 10% blend repeats the prompt back" width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The blended model mostly repeats the question back ("What is the capital of France?" → "What is the capital of France?") and writes a broken one-line prime check. The weight map explains why. The main weight matrices of the two models have a cosine similarity of only about 0.2 to 0.45, even though their norms and biases match at about 0.99. The two models share an architecture and a starting point, but their large matrices have moved far apart, so the straight line between them leaves the region where the model works.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Remove layers
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;layers.10-12 skip
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;skip&lt;/code&gt; zeroes the two output matrices of a layer, so the layer passes its input through unchanged. Removing three of the 24 layers leaves a model that is still fluent but less reliable:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fje4ju564jthx4koms24p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fje4ju564jthx4koms24p.png" alt="The model with layers 10-12 removed" width="800" height="628"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The haiku and "Paris" are fine. The weekend answer invents a schedule ("My weekend is from Monday through Friday"), and the prime-number function now returns &lt;code&gt;True&lt;/code&gt; for everything. Removing layers in the middle damages reasoning before it damages grammar.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this shows
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Behaviour lives in specific numbers.&lt;/strong&gt; Fewer than 0.01% of the parameters, chosen with intent, were enough to make a model talk about weddings. Astra's edit touched 0.016%. In both cases, a tiny targeted edit had a large, visible effect.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Direction matters more than size.&lt;/strong&gt; The wedding edit is small but points in a meaningful direction, and the model stays fluent. The 10% blend changes every weight, a much larger edit, and breaks the model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Edits have side effects.&lt;/strong&gt; The injected model lost some instruction-following, and removing layers broke a code answer. Astra's edit also had costs, such as answers that ended in long runs of whitespace. Reading the weights well enough to avoid side effects is the hard part, and that is where an AI programmer like Astra helps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It runs on ordinary hardware.&lt;/strong&gt; All of this ran in a browser tab on a laptop's integrated GPU, with no server.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How this compares with Astra
&lt;/h2&gt;

&lt;p&gt;webWeightSpace and Astra share the core idea: change a model by editing its numbers directly. They differ in who decides the edit and how.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;webWeightSpace&lt;/th&gt;
&lt;th&gt;Astra on Qwen&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Who writes the edit&lt;/td&gt;
&lt;td&gt;You, in a few lines, or a preset&lt;/td&gt;
&lt;td&gt;An AI agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Where the direction comes from&lt;/td&gt;
&lt;td&gt;Chosen up front: the embedding of a word&lt;/td&gt;
&lt;td&gt;Measured: compared internal activity while looping vs. making progress&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kind of edit&lt;/td&gt;
&lt;td&gt;Adds a bias that pushes toward a concept&lt;/td&gt;
&lt;td&gt;Rank-one change that removes a component&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Checking&lt;/td&gt;
&lt;td&gt;Original and edited model answer the same prompts side by side&lt;/td&gt;
&lt;td&gt;Propose, test, compare with controls, then commit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Where it runs&lt;/td&gt;
&lt;td&gt;A browser tab on a laptop GPU&lt;/td&gt;
&lt;td&gt;Python, a frontier model, hours of agent time&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;webWeightSpace &lt;em&gt;steers&lt;/em&gt; a model toward something you choose. Astra &lt;em&gt;diagnosed&lt;/em&gt; a fault and removed it. The browser demo makes the first half of the idea, directly editing weights and seeing the result, something anyone can try in a minute.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it is useful
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Teaching.&lt;/strong&gt; You can see that a model's behaviour lives in specific numbers, and change them yourself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Interpretability.&lt;/strong&gt; Skipping layers, switching off heads and injecting directions are quick experiments about where behaviour lives.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model merging.&lt;/strong&gt; The weight map and &lt;code&gt;mix&lt;/code&gt; show whether two models can be blended safely before anyone trains on the result.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sharing.&lt;/strong&gt; Every program fits in a URL, so an experiment can be shared as a link.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Open &lt;a href="https://vishalmysore.github.io/webWeightSpace/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/webWeightSpace/&lt;/a&gt; in a recent Chrome or Edge with WebGPU.&lt;/li&gt;
&lt;li&gt;Choose a model and click &lt;strong&gt;Download weights&lt;/strong&gt; (about 1 GB per model, stored in your browser once).&lt;/li&gt;
&lt;li&gt;Choose a preset, or write your own program, and click &lt;strong&gt;Build &amp;amp; compare&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://en.wikipedia.org/wiki/Weight-space_programming" rel="noopener noreferrer"&gt;Weight-space programming (Wikipedia)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kingy.ai/blog/weight-space-programming-ai-agent-model-weights/" rel="noopener noreferrer"&gt;Kingy AI: Weight-space programming, when an AI agent edits model weights&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/mlc-ai/web-llm" rel="noopener noreferrer"&gt;WebLLM&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>javascript</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Your RAG Finds Some of the Answer. I Tested 5 Chunking Strategies From This Month's Papers to See Who Finds All of It.</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Mon, 02 Nov 2026 20:46:21 +0000</pubDate>
      <link>https://dev.to/vishalmysore/your-rag-finds-some-of-the-answer-i-tested-5-chunking-strategies-from-this-months-papers-to-see-437p</link>
      <guid>https://dev.to/vishalmysore/your-rag-finds-some-of-the-answer-i-tested-5-chunking-strategies-from-this-months-papers-to-see-437p</guid>
      <description>&lt;p&gt;Most RAG dashboards report a number that sounds like success: "the right document was in the top 5." That is &lt;strong&gt;partial recall&lt;/strong&gt;: at least one relevant passage made it into the context.&lt;/p&gt;

&lt;p&gt;Real questions rarely need one passage. They need a rule &lt;em&gt;and&lt;/em&gt; the definition it depends on, or a limit &lt;em&gt;and&lt;/em&gt; its exception. If retrieval finds one of the two, the model writes a confident answer from half the evidence, and a hit-rate metric scores it as a success.&lt;/p&gt;

&lt;p&gt;The metric that catches this is &lt;strong&gt;complete-evidence recall&lt;/strong&gt;. For a question &lt;em&gt;q&lt;/em&gt; with evidence spans &lt;em&gt;E(q)&lt;/em&gt; and an assembled context &lt;em&gt;C(q)&lt;/em&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;partial(q)  = 1  if  ∃ e ∈ E(q) : e ⊆ C(q)
complete(q) = 1  if  ∀ e ∈ E(q) : e ⊆ C(q)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Between mid-September and the end of October 2026, several papers attacked this gap from the chunking side. Some optimise chunk boundaries directly, some follow the document's own structure, and some build a graph over chunks. I implemented the core idea of each one in plain JavaScript and added them as a new page to my browser-native &lt;a href="https://vishalmysore.github.io/advancedRagDemo/" rel="noopener noreferrer"&gt;Advanced Local RAG demo&lt;/a&gt;: the &lt;strong&gt;Chunking Lab&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1qi7qlx54qgzs0r5k13j.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1qi7qlx54qgzs0r5k13j.png" alt="Chunking Lab overview" width="799" height="562"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Everything runs in your tab. There is no server, no API key and no model download. Retrieval is BM25 only, so every number in this post is deterministic and reproducible from a URL. A full run (parse, five chunkers, two retrieval modes, all metrics) takes about &lt;strong&gt;11 ms&lt;/strong&gt; in Node 22.&lt;/p&gt;




&lt;h2&gt;
  
  
  The research
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;QESChunker&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2610.36557" rel="noopener noreferrer"&gt;arXiv 2610.36557&lt;/a&gt;, Oct 27) treats chunking as supervised optimisation. Given question–evidence (QE) pairs, human-written or LLM-generated, it picks the chunk boundaries that maximise how much evidence ends up retrievable. Without overlap, the objective decomposes over chunks and &lt;strong&gt;dynamic programming finds the exact optimum&lt;/strong&gt; in polynomial time. With overlap, the objective is monotone submodular. Exact optimisation is then NP-hard, but greedy selection comes with a (1 − 1/√e) approximation guarantee.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;StrucChunk&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2610.26313" rel="noopener noreferrer"&gt;arXiv 2610.26313&lt;/a&gt;, Oct 19) uses the structure the author already wrote. It treats a document as a graph of provisions, never cuts across a clause boundary, resolves cross-references ("see Section 4.2") &lt;strong&gt;before embedding&lt;/strong&gt;, and fuses BM25 with dense retrieval through Reciprocal Rank Fusion. On LegalBench-RAG, Char-Recall@8 goes from 23.59% to 48.18%. On one corpus, Recall@5 rises from 0.557 to 0.804, and the authors attribute 88% of that gain to chunking alone.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SpectRAG&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2610.36991" rel="noopener noreferrer"&gt;arXiv 2610.36991&lt;/a&gt;, Oct 27) stops treating chunks as independent. It builds a weighted graph over chunks (adjacency, section hierarchy, shared entities, citations), seeds it from hybrid search, propagates relevance, and assembles a context under a &lt;strong&gt;fixed token budget&lt;/strong&gt;. The goal is a context that is relevant, diverse, connected and sufficient.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Bottleneck Is Retrieval, Not Reading&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2610.26951" rel="noopener noreferrer"&gt;arXiv 2610.26951&lt;/a&gt;, Oct 18) supplies the diagnosis. Across ten frontier and three open-weight models, retrieval reliably surfaces &lt;em&gt;some&lt;/em&gt; of the relevant chunks but not &lt;em&gt;all&lt;/em&gt; of the chunks a multi-fact question needs. Chunking also strips the document-level context that resolves references between passages.&lt;/p&gt;

&lt;p&gt;Three September papers frame the rest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pre-retrieval Query Clustering for Adaptive Top-k&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2609.13489" rel="noopener noreferrer"&gt;arXiv 2609.13489&lt;/a&gt;) picks retrieval depth per query cluster, reporting over 36% better F1 and 14% fewer tokens on easy clusters. Context size is a per-query variable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parser, Chunking, and Embedding Interactions&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2609.31660" rel="noopener noreferrer"&gt;arXiv 2609.31660&lt;/a&gt;) runs a 54-configuration factorial study and finds parser and chunker choices interact, with no single retriever winning everywhere.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Re-ranking and Late Interaction Drive Retrieval Quality&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2609.38473" rel="noopener noreferrer"&gt;arXiv 2609.38473&lt;/a&gt;) compares six pipelines over about 464,000 papers and finds the biggest levers after first-stage retrieval. The main demo already has a cross-encoder reranker for that.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The test bed
&lt;/h2&gt;

&lt;p&gt;The corpus is a fictional observatory handbook, deliberately written like a policy document:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Size&lt;/td&gt;
&lt;td&gt;5,348 characters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Text units (headings + sentences)&lt;/td&gt;
&lt;td&gt;92&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Numbered provisions (&lt;code&gt;### x.y&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Explicit "Section x.y" cross-references&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QE pairs, training split (seen by QESChunker)&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QE pairs, held-out split&lt;/td&gt;
&lt;td&gt;13, of which 11 need two or more evidence sentences&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each QE pair stores the question and the exact evidence sentences as strings. The lab maps each evidence string to a character span and to the range of text units it covers, using a binary search over unit start offsets:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;unitAt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;pos&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;lo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;hi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;units&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;lo&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;hi&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;lo&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;hi&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;units&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;start&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nx"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;lo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nx"&gt;hi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;lo&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The parser splits sentences on &lt;code&gt;.&lt;/code&gt;, &lt;code&gt;!&lt;/code&gt; or &lt;code&gt;?&lt;/code&gt; followed by whitespace, so "1.2-metre" and "Section 1.3" stay intact. A provision is any heading at level 3 or deeper, or any numbered &lt;code&gt;x.y&lt;/code&gt; heading, plus the sentences under it. Each provision records its heading path, for example &lt;code&gt;1 Definitions › 1.3 Red-light zone&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fixed conditions for every strategy:&lt;/strong&gt; chunk length limit L = 450 characters, context budget B = 120 words, the same BM25 (k1 = 1.2, b = 0.75, stop-word removal and a light suffix stemmer that strips -s, -es, -ed and -ing from tokens longer than four characters), and the same context assembler.&lt;/p&gt;




&lt;h2&gt;
  
  
  The five chunkers
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Fixed window
&lt;/h3&gt;

&lt;p&gt;N characters with a 50-character overlap and step &lt;code&gt;L − 50&lt;/code&gt;. This is the tutorial default. It cuts mid-word and mid-sentence.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Sentence window
&lt;/h3&gt;

&lt;p&gt;The exact algorithm the main demo uses (&lt;code&gt;chunkDocument()&lt;/code&gt; in &lt;code&gt;ragEngine.js&lt;/code&gt;). It packs whole sentences up to L characters. When a chunk closes, it backtracks whole sentences from the end until the 50-character overlap budget is used, and those sentences start the next chunk.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Structure-aware (StrucChunk-style)
&lt;/h3&gt;

&lt;p&gt;One chunk per provision. A chunk never crosses a provision boundary, and a provision longer than L is split on sentence boundaries. Each chunk gets its heading path as a prefix, and every "Section x.y" reference inside it is resolved at index time by appending up to 320 characters of the referenced provision:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;C&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;matchAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/Section &lt;/span&gt;&lt;span class="se"&gt;(\d&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;(?:\.\d&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ti&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;byId&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ti&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;ti&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;pi&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ti&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;
  &lt;span class="nx"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ti&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nx"&gt;refs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="na"&gt;prov&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;ti&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;bodyText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;provisions&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;ti&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nx"&gt;refChars&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An indexed chunk then looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[2 Access and Arrival › 2.1 Gate hours]
The lower gate opens at 17:00. It closes to arriving vehicles 30 minutes
before the start of the dark window defined in Section 1.1. ...
↳ Section 1.1 Dark window: A dark window is any period of at least 90
minutes between the end of astronomical twilight and moonrise. ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rule and the definition it depends on are now one retrievable unit.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. QESChunker, exact dynamic programme
&lt;/h3&gt;

&lt;p&gt;Treat the document as a sequence of n text units. A segmentation splits it into contiguous chunks, each at most L characters. The utility of a chunk covering units j..i−1 is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;u(j, i) = Σ_{e ∈ spans}     1[ e ⊆ [j, i) ]
        + Σ_{q ∈ multi-ev}  1[ ∀ e ∈ E(q) : e ⊆ [j, i) ]
        − μ                                   (μ = 0.05, a small per-chunk cost)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first term rewards evidence sentences kept intact. The second rewards keeping all of a multi-fact question's evidence together. The objective is a sum over chunks, so the optimum satisfies a simple recurrence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;best[0] = 0
best[i] = max_{j &amp;lt; i, span(j, i) ≤ L}  best[j] + u(j, i)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;j&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;j&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;j&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;U&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;end&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;U&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;j&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;start&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;span&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;L&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;j&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;best&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;j&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;util&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;j&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;v&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;best&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;best&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;v&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;arg&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;j&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The inner loop stops as soon as a window exceeds L, so the cost is O(n · w · |QE|), where w is the maximum number of units per window. On this document the optimum has an objective of 13.35, and every training evidence span stays intact.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. QESChunker, overlapping greedy
&lt;/h3&gt;

&lt;p&gt;With overlap allowed, the lab picks a set of possibly overlapping windows under a total index budget of 1.35× the document length. The coverage function counts newly covered evidence spans, newly completed multi-fact questions, and α = 0.02 per newly covered unit, so the whole document gets indexed. It is monotone submodular, so the lab runs cost-benefit greedy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;bestC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;bestR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;cands&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;spent&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;cost&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;g&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;gain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;           &lt;span class="c1"&gt;// marginal coverage given what is already chosen&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;g&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;g&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;cost&lt;/span&gt;        &lt;span class="c1"&gt;// benefit per character&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;bestR&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;bestR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;bestC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;bestC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;
  &lt;span class="nx"&gt;chosen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;bestC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;spent&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;bestC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;cost&lt;/span&gt;
  &lt;span class="c1"&gt;// mark spans, questions and units covered by bestC&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One practical detail mattered. Without a floor on window size, cost-benefit greedy collapses into one-sentence chunks, because a single evidence sentence has the best gain per character. Candidates shorter than 0.5 · L are therefore excluded. Units still uncovered when the budget runs out are packed into gap chunks so they stay searchable.&lt;/p&gt;




&lt;h2&gt;
  
  
  The two retrieval modes
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Flat top-k
&lt;/h3&gt;

&lt;p&gt;Rank chunks by BM25, then fill the word budget in rank order. The assembler skips any chunk that would overflow the budget and any chunk whose tokens are less than 30% new (a near-duplicate filter that matters for overlapping chunkers):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;order&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;used&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;words&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;toks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;novel&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;picked&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;novel&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;
  &lt;span class="nx"&gt;picked&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;used&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;words&lt;/span&gt;
  &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;forEach&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;continue&lt;/code&gt; rather than &lt;code&gt;break&lt;/code&gt;: a large chunk that does not fit lets a smaller, lower-ranked chunk take the space. That detail explains one of the results below.&lt;/p&gt;

&lt;h3&gt;
  
  
  Chunk graph (SpectRAG-lite)
&lt;/h3&gt;

&lt;p&gt;Build an undirected weighted graph over chunks with four edge types:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Edge&lt;/th&gt;
&lt;th&gt;Condition&lt;/th&gt;
&lt;th&gt;Weight&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;adjacent&lt;/td&gt;
&lt;td&gt;consecutive in document order&lt;/td&gt;
&lt;td&gt;1.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;xref&lt;/td&gt;
&lt;td&gt;chunk A contains "Section x.y" and chunk B covers provision x.y&lt;/td&gt;
&lt;td&gt;1.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;section&lt;/td&gt;
&lt;td&gt;same provision, not adjacent&lt;/td&gt;
&lt;td&gt;0.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;lexical&lt;/td&gt;
&lt;td&gt;token-set Jaccard ≥ 0.18&lt;/td&gt;
&lt;td&gt;2 × Jaccard&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Then run personalised PageRank from the top-3 BM25 seeds, with the restart vector weighted by BM25 score, restart probability 0.5 and 25 iterations:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p₀[i]   = s[i] / Σ_seed s          for the 3 seeds, 0 elsewhere
p ← 0.5 · p₀ + 0.5 · Wᵀ D⁻¹ p      (25 iterations)
score[i] = 0.6 · s[i]/max(s) + 0.4 · p[i]/max(p)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The final ranking blends normalised BM25 with normalised PageRank mass and goes through the same budgeted assembler. The only thing that changes between modes is the order in which chunks are offered.&lt;/p&gt;




&lt;h2&gt;
  
  
  What each chunker builds
&lt;/h2&gt;

&lt;p&gt;Before retrieval, the index itself already says a lot:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Strategy&lt;/th&gt;
&lt;th&gt;Chunks&lt;/th&gt;
&lt;th&gt;Avg chars&lt;/th&gt;
&lt;th&gt;Index / doc size&lt;/th&gt;
&lt;th&gt;Chunks starting at a heading&lt;/th&gt;
&lt;th&gt;Evidence spans intact&lt;/th&gt;
&lt;th&gt;Cross-refs resolved in-chunk&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fixed window&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;td&gt;428&lt;/td&gt;
&lt;td&gt;1.12×&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentence window&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;td&gt;392&lt;/td&gt;
&lt;td&gt;1.03×&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Structure-aware&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;td&gt;285&lt;/td&gt;
&lt;td&gt;1.33×&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QES · exact DP&lt;/td&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;td&gt;410&lt;/td&gt;
&lt;td&gt;1.00×&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QES · overlap greedy&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;242&lt;/td&gt;
&lt;td&gt;1.17×&lt;/td&gt;
&lt;td&gt;56%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every chunker keeps every evidence sentence intact somewhere, because the sentences are short compared with L = 450. That means &lt;strong&gt;all differences below come from findability, not from broken evidence&lt;/strong&gt;. The columns that differ are granularity, alignment with headings, and whether cross-referenced definitions travel with the rule.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fimigfhmoacdc2vkqa772.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fimigfhmoacdc2vkqa772.png" alt="Boundary map: each bar is a chunk, gold bands are the evidence for one question, outlined chunks were retrieved" width="800" height="227"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Result 1: Partial recall flatters
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo0l6ryzy7gxzpka86yt4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo0l6ryzy7gxzpka86yt4.png" alt="Scoreboard, held-out questions, 450-character chunks, 120-word context" width="799" height="289"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Held-out questions, L = 450, B = 120 words:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Strategy&lt;/th&gt;
&lt;th&gt;Partial recall&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Complete recall (flat)&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;Complete recall (graph)&lt;/th&gt;
&lt;th&gt;Avg context words&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fixed window&lt;/td&gt;
&lt;td&gt;69%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;23%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;90&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentence window&lt;/td&gt;
&lt;td&gt;69%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;23%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;76&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Structure-aware&lt;/td&gt;
&lt;td&gt;92%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;69%&lt;/td&gt;
&lt;td&gt;103&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QES · exact DP&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;15%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;72&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QES · overlap greedy&lt;/td&gt;
&lt;td&gt;92%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;46%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;99&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every strategy finds &lt;em&gt;something&lt;/em&gt; for 69–92% of questions. The default chunkers assemble the &lt;em&gt;whole&lt;/em&gt; answer less than a quarter of the time. A pipeline evaluated on partial hits would look healthy while three out of four multi-fact answers were built on incomplete evidence.&lt;/p&gt;

&lt;p&gt;The "avg context words" column shows a second effect. The windowed chunkers leave 30–48 words of the 120-word budget unused, because the next-ranked 60–70-word window does not fit. Small chunks pack the budget more efficiently.&lt;/p&gt;




&lt;h2&gt;
  
  
  Result 2: Structure wins when context is scarce
&lt;/h2&gt;

&lt;p&gt;Provision-sized chunks average 285 characters, against about 400 for the windowed chunkers. At 120 words you can fit two or three provisions but only one large window. When the answer lives in two places, two small, precise pieces beat one big piece that happens to hold half the answer.&lt;/p&gt;

&lt;p&gt;The budget sweep (held-out, flat retrieval, L = 450) makes the trade-off visible:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Budget (words)&lt;/th&gt;
&lt;th&gt;Fixed&lt;/th&gt;
&lt;th&gt;Sentence&lt;/th&gt;
&lt;th&gt;Structure&lt;/th&gt;
&lt;th&gt;QES DP&lt;/th&gt;
&lt;th&gt;QES overlap&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;40&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;23%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;80&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;38%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;38%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;120&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;150&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;69%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;69%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;200&lt;/td&gt;
&lt;td&gt;54%&lt;/td&gt;
&lt;td&gt;54%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;77%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;77%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;300&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;69%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;85%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;400&lt;/td&gt;
&lt;td&gt;85%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;92%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;85%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;92%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgvr28ihgolzeys6yz33q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgvr28ihgolzeys6yz33q.png" alt="Complete-evidence recall vs context budget" width="800" height="581"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;At 40 words, structure-aware is the only strategy above 10%. By 300 words the big-window chunkers have caught up, and by 400 they have passed it, while structure-aware levels off at 77%. With a generous budget, almost any chunker eventually drags the evidence in, because you are sending a large share of the document. &lt;strong&gt;The best chunker depends on the budget, and the right budget depends on the query&lt;/strong&gt;, which is the adaptive top-k paper's argument from the other direction.&lt;/p&gt;

&lt;p&gt;The structure-aware plateau at 77% is not a chunking failure. Its remaining misses are retrieval failures (see Result 5).&lt;/p&gt;

&lt;h3&gt;
  
  
  Chunk-size sweep
&lt;/h3&gt;

&lt;p&gt;Holding B = 120 and varying L shows the same mechanism from the other side:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;L (chars)&lt;/th&gt;
&lt;th&gt;Fixed&lt;/th&gt;
&lt;th&gt;Sentence&lt;/th&gt;
&lt;th&gt;Structure&lt;/th&gt;
&lt;th&gt;QES DP&lt;/th&gt;
&lt;th&gt;QES overlap&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;250&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;54%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;69%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;62%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;69%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;350&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;54%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;450&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;600&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;800&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;31%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two things stand out:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Structure-aware is flat at 62–69%.&lt;/strong&gt; Almost every provision is shorter than 450 characters, so L barely changes its chunks. Its granularity comes from the document, not from a hyperparameter.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;At L = 800, fixed and sentence windows score 0%.&lt;/strong&gt; An 800-character window is about 130 words, which is larger than the 120-word budget, so the assembler cannot fit a single chunk. This is the failure you get when chunk size and context budget are tuned by different people.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At L = 250, every chunker gets better and the gap narrows. Smaller chunks behave more like provisions. The cost is a larger index and more fragments for any reader that needs surrounding context.&lt;/p&gt;




&lt;h2&gt;
  
  
  Result 3: The provably optimal chunker came last
&lt;/h2&gt;

&lt;p&gt;QESChunker's exact DP is optimal for its objective. It still scored &lt;strong&gt;15%&lt;/strong&gt; on held-out questions and &lt;strong&gt;58% on the twelve questions it was trained on&lt;/strong&gt;, below plain fixed windows at 75%.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2c4gn3ymg5jh5y68gk5w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2c4gn3ymg5jh5y68gk5w.png" alt="Scoreboard on the QES training questions" width="799" height="289"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Split&lt;/th&gt;
&lt;th&gt;Fixed&lt;/th&gt;
&lt;th&gt;Sentence&lt;/th&gt;
&lt;th&gt;Structure&lt;/th&gt;
&lt;th&gt;QES DP&lt;/th&gt;
&lt;th&gt;QES overlap&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Training (12)&lt;/td&gt;
&lt;td&gt;75%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;92%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;58%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Held-out (13)&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;All (25)&lt;/td&gt;
&lt;td&gt;48%&lt;/td&gt;
&lt;td&gt;52%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;76%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;36%&lt;/td&gt;
&lt;td&gt;64%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The boundary map shows the mechanism. The utility function rewards evidence sentences that sit fully inside a chunk, and says nothing about the text around them. The cheapest way to keep a rule intact is to cut &lt;em&gt;immediately before&lt;/em&gt; it. In a handbook, the line immediately before a rule is its heading, &lt;code&gt;### 5.2 Emergency kit&lt;/code&gt;. The heading ends up at the end of the previous chunk, and the rule starts the next one. Evidence intact: 100%. But a keyword query about "emergency kit" now matches the chunk holding the stranded heading, not the chunk holding the rule. That is why the DP has 0% of chunks starting at a heading.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ablation: does a heading-path prefix rescue it?&lt;/strong&gt; The lab can prefix every chunk from every chunker with the heading path of each provision it touches (the "contextual headers" toggle). Result, held-out, flat:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Fixed&lt;/th&gt;
&lt;th&gt;Sentence&lt;/th&gt;
&lt;th&gt;QES DP&lt;/th&gt;
&lt;th&gt;QES overlap&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;No prefix&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Heading-path prefix&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;54%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;It does not rescue the DP. The prefix tells BM25 which provision a chunk belongs to, but the chunk holding the stranded heading also gets that provision's name in its prefix, so both chunks still compete for the same query. The prefix also spends budget words on every chunk. Only the overlap variant, whose windows already tend to start at headings, gains from it.&lt;/p&gt;

&lt;p&gt;This is not a flaw in the paper. QESChunker is evaluated with a fixed &lt;strong&gt;hybrid&lt;/strong&gt; retriever, and dense embeddings are far less sensitive to where a heading lands. It is a lesson for anyone porting the idea: &lt;strong&gt;the optimisation objective has to model the retriever you actually run&lt;/strong&gt;. Intact evidence and findable evidence are different goals, and a chunker only optimises the one you write down. A practical fix is to add a term to &lt;em&gt;u(j, i)&lt;/em&gt; that scores whether the training question retrieves the chunk under your real retriever, or to forbid cuts between a heading and its first sentence.&lt;/p&gt;

&lt;p&gt;The overlapping variant shows the other side. Overlap lets the greedy pass choose windows that start at the heading &lt;em&gt;and&lt;/em&gt; cover the evidence. It reaches 46% held-out and 83% on training questions, without being told the document had structure. On documents without clean headings (transcripts, email threads, scanned PDFs), this is the version I would reach for. The training/held-out gap, 83% against 46%, is the caveat for every supervised chunker: the index is shaped around the questions you thought to ask.&lt;/p&gt;




&lt;h2&gt;
  
  
  Result 4: The chunk graph helps where definitions are hubs
&lt;/h2&gt;

&lt;p&gt;Switching to graph retrieval moved structure-aware chunking from 62% to &lt;strong&gt;69%&lt;/strong&gt; and changed little else.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fak4j2bm95x8al3trowd9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fak4j2bm95x8al3trowd9.png" alt="Scoreboard with chunk-graph retrieval" width="799" height="289"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One question shows where that gain comes from:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fovr6xf63mptxw4vhwsb6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fovr6xf63mptxw4vhwsb6.png" alt="Question inspector in chunk-graph mode: &amp;quot;Is flash photography allowed on the dome floor?&amp;quot;" width="799" height="336"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;"Is flash photography allowed on the dome floor?"&lt;/em&gt; needs two facts: the camera rule (§4.3, flash is banned everywhere in the red-light zone) and the definition (§1.3, the red-light zone &lt;em&gt;covers the dome floor&lt;/em&gt;). Flat BM25 ranks the definition third. At 52 words it no longer fits after the first two picks, so the assembler's &lt;code&gt;continue&lt;/code&gt; lets a shorter, useless chunk about the telescope take the space.&lt;/p&gt;

&lt;p&gt;In graph mode, the §1.3 definition chunk is a &lt;strong&gt;hub&lt;/strong&gt;: two other provisions cross-reference it, and it shares vocabulary with several more. PageRank mass flows into it from the seeds, its blended score passes the telescope chunk, and the answer becomes complete.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F342homyoj70lm5ciziv9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F342homyoj70lm5ciziv9.png" alt="Chunk graph for the structure-aware index; gold arcs are resolved cross-references" width="800" height="581"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ablation: turn off cross-reference resolution.&lt;/strong&gt; Structure-aware graph mode drops from 69% back to 62%, the same as flat. The graph gain comes from the &lt;code&gt;xref&lt;/code&gt; edges, which are the most trustworthy edges a document provides.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;On windowed chunkers, the graph does not help and sometimes hurts.&lt;/strong&gt; Sentence windows go from 23% to 15%. Lexical edges between large, mixed-topic windows mostly spread relevance to the neighbours of what was already found. With 40% of the final score coming from PageRank, a neighbour can outrank a correct BM25 pick and take its place in the budget. SpectRAG's own abstract notes trade-offs in early precision and domain dependence, and this toy reproduces that.&lt;/p&gt;




&lt;h2&gt;
  
  
  Result 5: What none of them fixed
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1f65baxmh6yp8ozz38t0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1f65baxmh6yp8ozz38t0.png" alt="Every strategy misses: &amp;quot;My 8-year-old is coming. What rule applies on the terrace?&amp;quot;" width="800" height="369"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;"My 8-year-old is coming. What rule applies on the terrace?"&lt;/em&gt; Every strategy misses both evidence sentences. The rule says "children under 10" and the question says "8-year-old". After tokenisation the question is &lt;code&gt;[8-year-old, com, rule, appli, terrace]&lt;/code&gt;, and the rule sentence shares none of those tokens. Structure-aware chunking gets part of the way: the resolved §1.3 reference brings "terrace" into the children's chunk, so it scores 1.18 instead of zero. But "terrace" also appears in several other provisions, and they outrank it. No chunking strategy can close the gap between "8-year-old" and "under 10". That needs a dense or hybrid retriever, query rewriting, or a reranker that understands the question.&lt;/p&gt;

&lt;p&gt;This is why the lab complements the main pipeline rather than replacing it. The main pipeline has MiniLM embeddings, hybrid fusion and a cross-encoder reranker. &lt;strong&gt;Chunking decides what &lt;em&gt;can&lt;/em&gt; be retrieved. The retriever decides what &lt;em&gt;is&lt;/em&gt; retrieved.&lt;/strong&gt; This month's papers are mostly about the first half of that sentence.&lt;/p&gt;




&lt;h2&gt;
  
  
  What I would actually do
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Measure complete-evidence recall, not hit rate.&lt;/strong&gt; Write 20 multi-fact questions for your corpus, mark the evidence sentences, and track how often &lt;em&gt;all&lt;/em&gt; of them reach the context. Report it next to partial recall. The gap between the two is the number to watch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If your documents have structure, chunk on it.&lt;/strong&gt; Split on headings and numbered clauses, prefix the heading path, resolve explicit cross-references at index time. It is deterministic, cheap and explainable, and here it was the biggest single win.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If they don't, use a supervised chunker with overlap.&lt;/strong&gt; Generate QE pairs with an LLM, optimise boundaries against them, evaluate on &lt;em&gt;different&lt;/em&gt; questions, and put your real retriever into the objective, or at least forbid cuts between a heading and its first line.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tune chunk size and context budget together.&lt;/strong&gt; Plot complete recall against budget for each chunk size. If a chunk can exceed the budget, it can never be retrieved. If the curves cross, your "best chunker" depends on how much context you send, and that should vary per query.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add a chunk graph only where edges are trustworthy.&lt;/strong&gt; Cross-references, citations and section hierarchy are good edges. Lexical similarity between large chunks mostly is not.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Chunking Lab:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/advancedRagDemo/chunking.html" rel="noopener noreferrer"&gt;vishalmysore.github.io/advancedRagDemo/chunking.html&lt;/a&gt;. Change chunk size and budget, switch retrieval modes and splits, toggle heading prefixes and cross-reference resolution, inspect any question, or paste your own markdown and QE JSON.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Full RAG pipeline (hybrid search, reranking, WebLLM):&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/advancedRagDemo/" rel="noopener noreferrer"&gt;vishalmysore.github.io/advancedRagDemo&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/advancedRagDemo" rel="noopener noreferrer"&gt;github.com/vishalmysore/advancedRagDemo&lt;/a&gt;. The engine is &lt;code&gt;src/utils/chunkLab.js&lt;/code&gt;, about 500 lines of dependency-free JavaScript. &lt;code&gt;runLab(text, qe, { L, budget, evalSplit, headers, resolveRefs })&lt;/code&gt; returns every number in this post, so you can run it in Node as well as in the browser.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every configuration here can be opened from a URL. For example, &lt;code&gt;chunking.html?mode=graph&amp;amp;q=7&lt;/code&gt; opens the flash-photography case in graph mode.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fvishalmysore%2FadvancedRagDemo%2Fmain%2Fscreenshots%2F08-full-page.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fvishalmysore%2FadvancedRagDemo%2Fmain%2Fscreenshots%2F08-full-page.png" alt="The full Chunking Lab page" width="800" height="1534"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Caveat: this is a toy. One 5,348-character fictional document, 25 questions, BM25 only. Each held-out question is worth about 7.7 percentage points, so one-question differences are noise. The value is in seeing the mechanisms the papers describe: stranded headings, budget crossovers, chunks that cannot fit, definition hubs. The rankings are not a benchmark. Run it on your own documents; that is what the paste box is for.&lt;/em&gt;&lt;/p&gt;




&lt;h3&gt;
  
  
  Papers referenced
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Zhao et al., &lt;a href="https://arxiv.org/abs/2610.36557" rel="noopener noreferrer"&gt;QESChunker: A Single Objective Unifies Overlapping and Non-Overlapping Chunking for RAG&lt;/a&gt;, arXiv 2610.36557, October 2026&lt;/li&gt;
&lt;li&gt;Singh, Tripathi &amp;amp; Singh, &lt;a href="https://arxiv.org/abs/2610.26313" rel="noopener noreferrer"&gt;StrucChunk: Structure-Aware Retrieval for Legal Document RAG&lt;/a&gt;, arXiv 2610.26313, October 2026&lt;/li&gt;
&lt;li&gt;Sahinoglu et al., &lt;a href="https://arxiv.org/abs/2610.36991" rel="noopener noreferrer"&gt;SpectRAG: Weighted Chunk Graphs for Token-Budgeted Context Selection in Multi-Hop RAG&lt;/a&gt;, arXiv 2610.36991, October 2026&lt;/li&gt;
&lt;li&gt;Banerjee et al., &lt;a href="https://arxiv.org/abs/2610.26951" rel="noopener noreferrer"&gt;The Bottleneck Is Retrieval, Not Reading&lt;/a&gt;, arXiv 2610.26951, October 2026&lt;/li&gt;
&lt;li&gt;Xia, Yamangil &amp;amp; Wang, &lt;a href="https://arxiv.org/abs/2609.13489" rel="noopener noreferrer"&gt;Pre-retrieval Query Clustering for Adaptive Top-k Document Retrieval in RAG Systems&lt;/a&gt;, arXiv 2609.13489, September 2026&lt;/li&gt;
&lt;li&gt;Singh, &lt;a href="https://arxiv.org/abs/2609.31660" rel="noopener noreferrer"&gt;Parser, Chunking, and Embedding Interactions in RAG over Indian Government Regulatory Documents&lt;/a&gt;, arXiv 2609.31660, September 2026&lt;/li&gt;
&lt;li&gt;Rathi, Chawla &amp;amp; Andreopoulos, &lt;a href="https://arxiv.org/abs/2609.38473" rel="noopener noreferrer"&gt;Re-ranking and Late Interaction Drive Retrieval Quality&lt;/a&gt;, arXiv 2609.38473, September 2026&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Views are my own. All code is MIT-licensed.&lt;/em&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Karpathy built HackerTube in 2 hours. I rebuilt it in the open and measured it</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Mon, 02 Nov 2026 14:47:00 +0000</pubDate>
      <link>https://dev.to/vishalmysore/karpathy-built-hackertube-in-2-hours-i-rebuilt-it-in-the-open-and-measured-it-1nk8</link>
      <guid>https://dev.to/vishalmysore/karpathy-built-hackertube-in-2-hours-i-rebuilt-it-in-the-open-and-measured-it-1nk8</guid>
      <description>&lt;p&gt;&lt;em&gt;Inspired by Andrej Karpathy's HackerTube. Live demo: &lt;a href="https://vishalmysore.github.io/OpenTube/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/OpenTube/&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Hacker News commenters have been sharing YouTube links since 2007. Some of those links are the best talks, lectures and documentaries on the internet: Engelbart's Mother of All Demos, Robert Lustig on sugar, Joel Spolsky on Excel. Most of them are not recommendations at all. They are a music track, a news clip, a joke, a link dropped as evidence in an argument.&lt;/p&gt;

&lt;p&gt;Karpathy's HackerTube showed that an LLM can sort the two and turn twenty years of comments into a watch list. OpenTube does the same thing and also publishes three numbers that matter if you want to trust the list, or build something like it yourself:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;How often the filter is right&lt;/strong&gt;, measured on a labelled random sample.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What each step costs&lt;/strong&gt;, in time and tokens.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How the videos connect&lt;/strong&gt;, as a knowledge graph you can ask questions of, in your browser.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The problem: "is this a recommendation?" is a fuzzy question
&lt;/h2&gt;

&lt;p&gt;Take two comments that both contain a YouTube link:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;"Here is an MIT course, Introduction to CS and Programming. The professor keeps everything simple because the class is for beginners."&lt;/em&gt; This is a recommendation.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;"[YouTube video]"&lt;/em&gt;, posted alone under a thread about the Turbo Encabulator. This is not, by the rubric used here: a bare link with no context.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;People disagree on the cases in between. Out of 300 random comments, 32 were hard enough to be flagged as borderline. If you hand the whole archive to an LLM and publish what comes out, you don't know how good the list is. You also pay for an LLM call on every one of 257,038 comments.&lt;/p&gt;

&lt;h2&gt;
  
  
  The idea: a cheap model first, the LLM only when it is unsure
&lt;/h2&gt;

&lt;p&gt;The filter is a cascade of two models.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Laya&lt;/strong&gt;, a small typed-decision model, reads each comment and gives a probability that it recommends the video. If that probability is clearly high or clearly low, Laya decides on its own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Qwen2.5-1.5B-Instruct&lt;/strong&gt;, a small LLM, sees only the comments Laya is unsure about.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both run on a laptop CPU with ONNX Runtime for the batch run, and in the browser for the demo. No API key is needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the site does
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Watch: the canon, filtered
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2yfyzz31jsarmw7070fx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2yfyzz31jsarmw7070fx.png" alt=" " width="800" height="564"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every Hacker News item from 2006 to 2026 (49 million of them) was scanned for YouTube links. That gives 185,909 distinct videos. 1,345 of them were linked by ten or more different people, and that set is the canon. Each card shows how many people linked the video, the thread it came from, the comment that recommended it, and which model made the call.&lt;/p&gt;

&lt;h3&gt;
  
  
  Receipts: accuracy and cost, in the open
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fktu8swurhj2mswc4bzyx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fktu8swurhj2mswc4bzyx.png" alt=" " width="720" height="788"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The Receipts page shows the funnel from 49 million items down to the kept videos, the accuracy table, and the cost of each stage at prices you can edit. The thresholds are editable too, so you can see how the trade-off between precision and recall moves.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Filter&lt;/th&gt;
&lt;th&gt;Accuracy (cross-validated)&lt;/th&gt;
&lt;th&gt;Precision&lt;/th&gt;
&lt;th&gt;Recall&lt;/th&gt;
&lt;th&gt;Time per comment&lt;/th&gt;
&lt;th&gt;Sent to LLM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Laya only&lt;/td&gt;
&lt;td&gt;70.8%&lt;/td&gt;
&lt;td&gt;76.9%&lt;/td&gt;
&lt;td&gt;29.1%&lt;/td&gt;
&lt;td&gt;1.28 s&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-1.5B only&lt;/td&gt;
&lt;td&gt;67.7%&lt;/td&gt;
&lt;td&gt;59.1%&lt;/td&gt;
&lt;td&gt;50.5%&lt;/td&gt;
&lt;td&gt;4.11 s&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cascade&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;71.2%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;67.5%&lt;/td&gt;
&lt;td&gt;50.5%&lt;/td&gt;
&lt;td&gt;2.46 s&lt;/td&gt;
&lt;td&gt;about 30%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two results stand out:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The small LLM on its own is worse than Laya and about three times slower.&lt;/strong&gt; More model is not automatically better on a fuzzy yes/no call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The cascade is the best of the three&lt;/strong&gt; and calls the LLM on only about a third of the comments. It keeps the LLM's recall and most of Laya's precision.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a sense of scale, answering "not a recommendation" every time already scores 65.7%. This is a hard call, and a few points of accuracy matter.&lt;/p&gt;

&lt;p&gt;Projected over all 257,038 comments on this CPU: about 12 days with the LLM on everything, about 6 with the cascade, about 4 with Laya alone.&lt;/p&gt;

&lt;h3&gt;
  
  
  Labels: check the ground truth yourself
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fok8o6t8lb9xmx0su2ygv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fok8o6t8lb9xmx0su2ygv.png" alt=" " width="720" height="788"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;All 300 labelled comments are on the Labels page next to the rubric, with each model's probability shown under every comment. If you disagree with a label, change it. The Receipts page re-scores everything with your version, and you can export your labels as JSON.&lt;/p&gt;

&lt;h3&gt;
  
  
  Ask the graph: questions answered from the graph
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzzvwy4l7xwx8pwjti86c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzzvwy4l7xwx8pwjti86c.png" alt=" " width="720" height="563"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The videos are linked in a graph: two videos are connected when they were shared in the same HN thread, by the same person, or with similar recommendation text. When you ask a question such as &lt;em&gt;"documentaries about the history of computing"&lt;/em&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The question is embedded in your browser with MiniLM (about 23 MB).&lt;/li&gt;
&lt;li&gt;The closest videos become seeds: here, the Mother of All Demos, the Secret History of Silicon Valley, and The Century of the Self.&lt;/li&gt;
&lt;li&gt;The graph pulls in their neighbours through shared threads, shared commenters and topics (SAGE-style expansion).&lt;/li&gt;
&lt;li&gt;If your browser has WebGPU, a small LLM writes an answer that cites the videos it used.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Without WebGPU you still get the retrieved videos and the graph, as in the screenshot above.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this is useful
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Discovery.&lt;/strong&gt; A watch list of the talks and documentaries a technical community has kept pointing to for twenty years, grouped by topic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A template for any LLM filter over a large corpus.&lt;/strong&gt; Comments, support tickets, reviews, or papers: put a cheap classifier in front, send only the unsure cases to the LLM, and measure both against a labelled sample before you trust the output.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Honest cost planning.&lt;/strong&gt; Time and tokens per stage, measured, with a projection for the full corpus, so you can decide before you spend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Private, local Q&amp;amp;A.&lt;/strong&gt; Retrieval, graph expansion and the answer all run in the browser. Nothing you ask leaves your machine.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The 300 labels are a first pass by one labeller (Claude, following a written rubric). That makes a useful smoke test, not a benchmark. Corrections are welcome.&lt;/li&gt;
&lt;li&gt;The thresholds were chosen on the same sample, which is why the cross-validated numbers are shown next to the in-sample ones.&lt;/li&gt;
&lt;li&gt;The published canon run is partial: 632 of 2,683 comments, covering 540 of the 1,345 canon videos. The runner resumes where it stopped, and a GitHub Actions workflow is included to run the whole archive month by month.&lt;/li&gt;
&lt;li&gt;Video titles are looked up in your browser. The dataset stores only video IDs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/OpenTube/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/OpenTube/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code and data:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/OpenTube" rel="noopener noreferrer"&gt;https://github.com/vishalmysore/OpenTube&lt;/a&gt; (Apache-2.0)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Built on the open-index/hacker-news dataset (ODC-By), Laya typed-decisions, Qwen2.5-1.5B-Instruct via WebLLM, and all-MiniLM-L6-v2. Inspired by Andrej Karpathy's HackerTube; not affiliated with him, Hacker News or YouTube.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>buildinpublic</category>
      <category>llm</category>
      <category>sideprojects</category>
    </item>
    <item>
      <title>I Built 4 AI Agent Apps That Run 100% in Your Browser with Laya and WebGPU (No API Key, No Server)</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Sat, 31 Oct 2026 13:24:56 +0000</pubDate>
      <link>https://dev.to/vishalmysore/i-built-4-ai-agent-apps-that-run-100-in-your-browser-with-laya-and-webgpu-no-api-key-no-server-2170</link>
      <guid>https://dev.to/vishalmysore/i-built-4-ai-agent-apps-that-run-100-in-your-browser-with-laya-and-webgpu-no-api-key-no-server-2170</guid>
      <description>&lt;p&gt;I built four AI apps on Laya, and every one of them runs entirely inside a browser tab: no server, no API key, no cost per call.&lt;/p&gt;

&lt;p&gt;An AI agent. An LLM-as-a-judge for RAG. Agentic workflow automation. Long-term agent memory that knows when to forget. All four run on one 421M-parameter model using WebGPU, and on the job that matters most for agents (deciding what to do next) it beat a generative LLM nearly four times its size.&lt;/p&gt;

&lt;p&gt;Here's what I built, the numbers, and the design patterns that make it work.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd81g8pzegvf88o4s662g.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd81g8pzegvf88o4s662g.png" alt=" " width="800" height="340"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is Laya?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://huggingface.co/convaiinnovations/laya-typed-decisions" rel="noopener noreferrer"&gt;Laya&lt;/a&gt; is a 421M-parameter &lt;strong&gt;decision model&lt;/strong&gt; from ConvAI Innovations, built on ModernBERT-large and open source under Apache 2.0. It doesn't generate text. You give it a typed question (pick one of these options, yes or no, or a score on a scale) plus some context, and in a single forward pass it returns a probability for every option.&lt;/p&gt;

&lt;p&gt;That makes it a non-autoregressive &lt;strong&gt;"System 1" model&lt;/strong&gt;: about 33–40 ms per decision on a T4 GPU, versus 236–276 ms for TypeSafe's generative Jev (&lt;a href="https://aiweekly.co/alerts/convai-ships-laya-a-421m-modernbert-decision-model-apache-20" rel="noopener noreferrer"&gt;AI Weekly&lt;/a&gt;). It's built for the high-volume decisions every AI system makes: ticket routing, content moderation, AI guardrails, agent-trace checks.&lt;/p&gt;

&lt;p&gt;"System 1" is Kahneman's fast, intuitive thinking. The idea is that most of what an AI agent does all day (which tool, which queue, is this safe, are we done) is System 1 work, and you shouldn't be paying a frontier LLM to do it.&lt;/p&gt;

&lt;p&gt;This is one of the big themes of 2026. NVIDIA researchers argued that small language models are the future of agentic AI, and the industry is moving towards small models handling routine agent work while big models take only the hard cases (&lt;a href="https://theagenticreview.substack.com/p/why-small-language-models-will-eat" rel="noopener noreferrer"&gt;a good roundup&lt;/a&gt;). At the same time WebGPU has made &lt;a href="https://pinggy.io/blog/run_llm_in_browser_webgpu/" rel="noopener noreferrer"&gt;running real AI models in a browser tab&lt;/a&gt; a practical, production-ready choice.&lt;/p&gt;

&lt;p&gt;Laya sits right where those two trends meet. So I wanted to see: &lt;strong&gt;how much of an AI stack can a 421M decision model run, on a laptop, in a tab?&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting Laya into the browser
&lt;/h2&gt;

&lt;p&gt;I converted the typed-decisions checkpoint to ONNX and quantized it (&lt;a href="https://github.com/vishalmysore/layaForWeb" rel="noopener noreferrer"&gt;layaForWeb&lt;/a&gt;, model on Hugging Face as &lt;a href="https://huggingface.co/VishalMysore/layaForWebTrained" rel="noopener noreferrer"&gt;&lt;code&gt;VishalMysore/layaForWebTrained&lt;/code&gt;&lt;/a&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;int8:&lt;/strong&gt; about 422 MB, runs on WASM&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;int4:&lt;/strong&gt; about 278 MB, runs on WebGPU&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It downloads once from Hugging Face, is cached in the browser's Cache Storage, and runs through ONNX Runtime Web. GitHub Pages can't send the COOP/COEP headers that multithreaded WASM needs, so each app registers a tiny service worker that adds them. It's fully private, local AI: nothing you type leaves the page.&lt;/p&gt;

&lt;p&gt;One trick I used in every app: &lt;strong&gt;playback before download.&lt;/strong&gt; Each page ships answers Laya gave for the built-in examples and plays them back (clearly labelled) until you load the model. The demo works in one second, and the live model is one click away.&lt;/p&gt;

&lt;p&gt;Then I built four apps on top of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. layaForWorkflows: agentic workflow automation as a graph
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://vishalmysore.github.io/layaForWorkflows/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt; · &lt;a href="https://github.com/vishalmysore/layaForWorkflows" rel="noopener noreferrer"&gt;Code&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A workflow is just data: a DAG of three node types. Diamonds are typed questions Laya answers. Boxes are automation steps. Circles at the end say whether the case was handled automatically, sent to a person, or blocked.&lt;/p&gt;

&lt;p&gt;Type a support ticket ("the app closes as soon as I open it, client demo in an hour") and the path lights up with Laya's probability on every branch: &lt;strong&gt;bug, 78%&lt;/strong&gt;, then "is the customer blocked right now?" &lt;strong&gt;yes, 67%&lt;/strong&gt;, so it pages the on-call engineer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faqmh3keum77rpb91od38.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faqmh3keum77rpb91od38.png" alt="Laya Workflows: a support ticket routed through a decision graph, with the model's probability on every branch" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Seven workflows ship with it, including support ticket triage, IT incident response, insurance claim intake, content moderation and an &lt;strong&gt;AI agent action guardrail&lt;/strong&gt; that fails closed: deletions are blocked, and anything uncertain waits for a human.&lt;/p&gt;

&lt;p&gt;Laya answers every decision node in &lt;strong&gt;one batched forward pass&lt;/strong&gt;, including the ones off the path the run took. So the confidence slider re-routes a finished run instantly, with no model call, and the Analytics tab shows how the mix of automated, human-reviewed and blocked cases shifts as you raise the bar.&lt;/p&gt;

&lt;p&gt;Across the 37 example inputs, Laya sent the routine cases straight through, routed the risky ones to people and blocked the dangerous ones, with each decision drawn on the graph and exportable as an OWL ontology in Turtle or JSON-LD. A force-push to main correctly went to a human for approval. That's exactly the behaviour you want from an AI guardrail.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. layaAsRagJudge: LLM-as-a-judge without the LLM
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://vishalmysore.github.io/layaAsRagJudge/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt; · &lt;a href="https://github.com/vishalmysore/layaAsRagJudge" rel="noopener noreferrer"&gt;Code&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;LLM-as-a-judge is everywhere right now, usually meaning another expensive API call to a big model to check the first one. Laya does the core of that job in the browser: &lt;strong&gt;is this claim actually supported by the retrieved evidence?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The whole RAG pipeline runs in the tab: sentence-aware chunking, all-MiniLM-L6-v2 embeddings, an IndexedDB vector store, cosine top-3 retrieval, and then one typed question to Laya. The verdict goes through a gate: AUTO, BLOCK, or HOLD for review.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff34ov2m5jg36jfy72pzh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff34ov2m5jg36jfy72pzh.png" alt="Laya RAG Judge: retrieved passages with cosine scores, Laya's typed question, and a HOLD verdict" width="800" height="636"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On 72 claims about fictional documents (so the model can't lean on what it already knows):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Judge question&lt;/th&gt;
&lt;th&gt;Accuracy&lt;/th&gt;
&lt;th&gt;False "supported"&lt;/th&gt;
&lt;th&gt;AUROC&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Two options (default)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;75.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;27.8%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.80&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Three options (supported / contradicted / not mentioned)&lt;/td&gt;
&lt;td&gt;72.2%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;11.1%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Yes / no&lt;/td&gt;
&lt;td&gt;69.4%&lt;/td&gt;
&lt;td&gt;19.4%&lt;/td&gt;
&lt;td&gt;0.74&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That's with &lt;strong&gt;no fine-tuning for this task&lt;/strong&gt;, just a typed question. With the confidence gate on, Laya acts on its own on 44% of claims &lt;strong&gt;at 84% accuracy&lt;/strong&gt; and flags the rest for review. Switch to the three-option question and false "supported" verdicts drop to 11%.&lt;/p&gt;

&lt;p&gt;Then I built a &lt;strong&gt;RAG router&lt;/strong&gt; on top: Laya decides which knowledge base (Encyclopedia, News or Policy) should answer a question, or whether to skip retrieval entirely. It picked the right knowledge base &lt;strong&gt;23 times out of 24&lt;/strong&gt;. With a second Laya question ("do these passages answer it?") checking the results, the router got the right final outcome on &lt;strong&gt;94%&lt;/strong&gt; of questions, all in the browser.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdzlu5sroofzcfe4i9bmx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdzlu5sroofzcfe4i9bmx.png" alt="The RAG router: Laya picks a knowledge base, then checks whether the passages answer the question" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. layaAgent: a 421M encoder beat a 1.5B LLM at running an AI agent
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://vishalmysore.github.io/layaAgent/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt; · &lt;a href="https://github.com/vishalmysore/layaAgent" rel="noopener noreferrer"&gt;Code&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is the result that surprised me most.&lt;/p&gt;

&lt;p&gt;layaAgent is an AI agent that runs entirely in a browser tab. Laya is &lt;strong&gt;System 1&lt;/strong&gt;: at each step it picks the next tool from 12 options and picks each argument from candidate spans. A small LLM running in the tab through WebLLM (Qwen2.5-1.5B-Instruct) is &lt;strong&gt;System 2&lt;/strong&gt;, a generative fallback. Tools call Wikipedia, Wikidata and Open-Meteo straight from the browser, plus a calculator, unit and date math, and notes stored in IndexedDB.&lt;/p&gt;

&lt;p&gt;The rule that makes this work: &lt;strong&gt;Laya never has to write.&lt;/strong&gt; It doesn't type "Japan"; it picks "Japan" out of the spans an extractor found in the last tool result. Tool calling becomes a choice, and choices are what Laya does best.&lt;/p&gt;

&lt;p&gt;I expected the hybrid (Laya first, Qwen when Laya is unsure) to win. It didn't. Laya did.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqakw0dtutsinjoan8a3m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqakw0dtutsinjoan8a3m.png" alt=" " width="799" height="274"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On its own, Laya &lt;strong&gt;more than tripled&lt;/strong&gt; the task success of the 1.5B LLM (32% vs 10%), and ran each task in under a third of the time (9.7 s vs 35.3 s). Handing Laya's uncertain steps to the LLM actually made things worse. Picking from options is what Laya was trained for, and it shows.&lt;/p&gt;

&lt;p&gt;Even better, Laya knows when it knows. On unseen tasks, &lt;strong&gt;the steps Laya took on its own were right 90% of the time&lt;/strong&gt;, and everything else was visibly held with a reason. Put a person on those held steps (human-in-the-loop) and &lt;strong&gt;94% of tasks finish correctly&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;On top of that, anything that changes something (&lt;code&gt;note_save&lt;/code&gt;, &lt;code&gt;note_delete&lt;/code&gt;, &lt;code&gt;send_message&lt;/code&gt;) &lt;strong&gt;always&lt;/strong&gt; asks you first, no matter who picked it:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fba27vde6yhavjk77lnu0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fba27vde6yhavjk77lnu0.png" alt="layaAgent trace graph: Laya proposes note_delete, the step is held, and the delete waits for the user's OK" width="800" height="624"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Laya is especially strong on single-tool goals (weather, arithmetic, unit conversion, dates, memory) and at spotting risk: its "does this goal change something?" check scored an &lt;strong&gt;AUROC of 0.90&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. layaForMemory: AI agent memory that knows when to forget
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://vishalmysore.github.io/layaForMemory/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt; · &lt;a href="https://github.com/vishalmysore/layaForMemory" rel="noopener noreferrer"&gt;Code&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Agent memory is one of the hottest problems in agentic AI, and the hardest part is that facts go stale. "We use Postgres" stops being true the day the migration lands, and your assistant keeps repeating it.&lt;/p&gt;

&lt;p&gt;layaForMemory is a memory invalidation layer, following the design of &lt;a href="https://github.com/chopratejas/invalidate" rel="noopener noreferrer"&gt;invalidate&lt;/a&gt; and rebuilt for the browser. Every event (a chat message, a settings change, a Slack-style note) is checked against similar stored facts, and Laya decides what it does to each one: confirmed, superseded, contradicted, needs review, or unrelated. Memory text is never edited. Stale facts are struck through and linked to their replacement, and every verdict is written to an audit ledger &lt;em&gt;before&lt;/em&gt; memory changes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fee4vj8ogj8ihbr92grod.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fee4vj8ogj8ihbr92grod.png" alt="Laya Memory after the demo: a superseded rate limit linked to its replacement, a contradicted owner, and one fact sent to review" width="800" height="547"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It also has built-in &lt;strong&gt;prompt-injection defence&lt;/strong&gt;, and it's beautifully simple. Before touching memory, Laya answers one question: is this event a report, a question, a proposal, or a command? Commands never change memory.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1vc9ekjw50xhpf85bdl6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1vc9ekjw50xhpf85bdl6.png" alt="A prompt injection classified as a command at 0.73, logged, and memory left untouched" width="798" height="195"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On 62 test cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Zero&lt;/strong&gt; good facts wrongly thrown away (0 of 32)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;100%&lt;/strong&gt; of questions, plans and prompt-injection commands recognised and ignored&lt;/li&gt;
&lt;li&gt;Combining four Laya questions separated stale from current facts with an &lt;strong&gt;AUROC of 0.95&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  7 design patterns for building on Laya
&lt;/h2&gt;

&lt;p&gt;These held across all four projects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Gate on branch probability.&lt;/strong&gt;&lt;br&gt;
Use the share of Laya's probability that went down the chosen route as your confidence gate. One slider then means the same thing on every decision node: "act on your own only if at least X of the probability agrees". It's simple and it works everywhere.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Choice questions are Laya's superpower.&lt;/strong&gt;&lt;br&gt;
The 4-way "what kind of message is this?" separated prompt-injection commands from everything else perfectly on my sample. The router picked the right knowledge base 23 of 24 times. Whenever you can phrase a decision as "pick one of these", do.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Route on what something is.&lt;/strong&gt;&lt;br&gt;
Laya is at its best classifying intent, category and risk. Design your questions around &lt;em&gt;what&lt;/em&gt; a ticket, claim or event is, and let plain code handle the arithmetic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Wording is a hyperparameter.&lt;/strong&gt;&lt;br&gt;
"The user is now pescatarian instead of vegetarian" cleanly superseded "The user is vegetarian". Phrase events as direct statements and test your question wording against real model outputs. A little prompt tuning goes a long way.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Ask several questions and average them.&lt;/strong&gt;&lt;br&gt;
Each staleness question scored an AUROC of 0.89–0.92 on its own; their mean hit &lt;strong&gt;0.95&lt;/strong&gt;. Laya is fast enough that four questions in one batch cost almost nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. You may not need a generative fallback at all.&lt;/strong&gt;&lt;br&gt;
In my agent, Laya plus a person beat Laya plus an LLM by a wide margin. Sometimes the best System 2 is a human.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Record once, re-score forever.&lt;/strong&gt;&lt;br&gt;
Every app stores Laya's full answers and lets sliders re-route them without another forward pass. Threshold tuning becomes dragging a slider, and the trade-offs are visible to anyone opening the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can a 421M model run your AI agents?
&lt;/h2&gt;

&lt;p&gt;The decision layer, yes. And that's most of what an agent does. Build around what Laya is great at:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Let code propose the options. Let Laya pick.&lt;/li&gt;
&lt;li&gt;Gate on branch probability, and make "I'm not sure" a first-class outcome.&lt;/li&gt;
&lt;li&gt;Keep side effects behind a confirmation no model can skip.&lt;/li&gt;
&lt;li&gt;Run it locally with WebGPU: no API key, no server, no cost per call.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All four apps are open source (Apache-2.0) and run on GitHub Pages. Open one, hit &lt;strong&gt;Run&lt;/strong&gt;, and watch the decisions light up before you've downloaded a single megabyte:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🔀 &lt;strong&gt;Workflows:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/layaForWorkflows/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/layaForWorkflows/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;⚖️ &lt;strong&gt;RAG judge + router:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/layaAsRagJudge/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/layaAsRagJudge/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🤖 &lt;strong&gt;Agent:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/layaAgent/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/layaAgent/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🧠 &lt;strong&gt;Memory:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/layaForMemory/" rel="noopener noreferrer"&gt;https://vishalmysore.github.io/layaForMemory/&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you're building AI agents with small language models, I'd love to hear: which part of your agent do you think actually needs a big LLM? Tell me in the comments.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Results come from the evaluation pages in each repo, on the test suites shipped there. This is an independent project, not affiliated with ConvAI Innovations. Laya is by ConvAI Innovations (Apache-2.0), built on ModernBERT-large by Answer.AI and LightOn.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Pele makes Mermaid diagrams fast enough to be live. So I built an ontology viewer that runs in a browser tab</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Mon, 26 Oct 2026 15:28:06 +0000</pubDate>
      <link>https://dev.to/vishalmysore/pele-makes-mermaid-diagrams-fast-enough-to-be-live-so-i-built-an-ontology-viewer-that-runs-in-a-pph</link>
      <guid>https://dev.to/vishalmysore/pele-makes-mermaid-diagrams-fast-enough-to-be-live-so-i-built-an-ontology-viewer-that-runs-in-a-pph</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv6ekfx5mivsuwjjxwolc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv6ekfx5mivsuwjjxwolc.png" alt="OntoLive showing the Margherita class from the Pizza ontology, with its restrictions listed beside the diagram" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Try it:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/ontolive/#src=pizza&amp;amp;focus=Margherita&amp;amp;depth=2" rel="noopener noreferrer"&gt;vishalmysore.github.io/ontolive&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/ontolive" rel="noopener noreferrer"&gt;github.com/vishalmysore/ontolive&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is Pele?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://pele.run" rel="noopener noreferrer"&gt;Pele&lt;/a&gt; is a new open-source library from the Obsidian team. It renders &lt;a href="https://mermaid.js.org" rel="noopener noreferrer"&gt;Mermaid&lt;/a&gt; diagrams to SVG. You give it the same text you would give Mermaid:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;render&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pele&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;svg&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`classDiagram
  Pizza &amp;lt;|-- Margherita
  Margherita --&amp;gt; MozzarellaTopping : hasTopping`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;#diagram&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;innerHTML&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;svg&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pele has four properties that matter here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It is synchronous.&lt;/strong&gt; &lt;code&gt;render()&lt;/code&gt; returns the SVG straight away. There is no promise to wait for.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is fast.&lt;/strong&gt; Most diagrams take a few milliseconds or less.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It has no runtime dependencies.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is themed with CSS variables.&lt;/strong&gt; Set &lt;code&gt;--pele-accent&lt;/code&gt;, &lt;code&gt;--pele-surface&lt;/code&gt;, &lt;code&gt;--pele-font&lt;/code&gt; and the rest on a parent element. The diagram follows your light or dark theme without being drawn again.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Pele aims to support every diagram type built into Mermaid 12.1, including flowcharts, sequence diagrams, class diagrams, Gantt charts and pie charts.&lt;/p&gt;

&lt;p&gt;Faster diagrams sound like a small upgrade. The real change is what they let you build. Once a render costs a millisecond, a diagram can redraw on every keystroke, every slider tick and every token a language model streams. That turns a diagram from a picture into an interface.&lt;/p&gt;

&lt;p&gt;I wanted to find a problem where that difference matters. Ontologies turned out to be a good one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: looking at an ontology needs a desktop app
&lt;/h2&gt;

&lt;p&gt;An ontology is a formal model of a domain: classes such as &lt;code&gt;Pizza&lt;/code&gt; and &lt;code&gt;MozzarellaTopping&lt;/code&gt;, a hierarchy (&lt;code&gt;Margherita&lt;/code&gt; is a kind of &lt;code&gt;NamedPizza&lt;/code&gt;), and rules (&lt;code&gt;Margherita&lt;/code&gt; has some &lt;code&gt;TomatoTopping&lt;/code&gt; and only mozzarella or tomato toppings). They are stored in the OWL language, usually as RDF/XML or Turtle files.&lt;/p&gt;

&lt;p&gt;To look at one, most people install Protégé, a Java desktop application, and then add a visualisation plugin. Fine for an ontology engineer. Too much for the developer, analyst or student who only wants to see what is in a file someone sent them.&lt;/p&gt;

&lt;p&gt;A browser could do this. In practice there are two obstacles:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Real ontologies are too big to draw at once.&lt;/strong&gt; The well-known Pizza ontology has 99 classes. Industrial ones have thousands. A single diagram of everything is unreadable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The answer is to explore, not to draw.&lt;/strong&gt; Pick a class, show its neighbourhood, widen or narrow it. Exploring needs a fresh diagram on every interaction. With a renderer that takes half a second each time, the page feels broken.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Pele removes the second obstacle, and that removes the first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The solution: OntoLive
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://vishalmysore.github.io/ontolive/" rel="noopener noreferrer"&gt;OntoLive&lt;/a&gt; is an ontology viewer that runs entirely in a browser tab. Drop in a &lt;code&gt;.owl&lt;/code&gt; or &lt;code&gt;.ttl&lt;/code&gt; file. It is parsed in the tab and never uploaded. You can also load the Pizza ontology with one click.&lt;/p&gt;

&lt;h3&gt;
  
  
  Explore by focus and depth
&lt;/h3&gt;

&lt;p&gt;Click any class and the view centres on it. The &lt;strong&gt;depth&lt;/strong&gt; slider decides how far out to look: one hop shows the class and its direct neighbours, two hops shows their neighbours, and so on. Every step of the slider produces new Mermaid text and a new diagram.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Depth 1 around &lt;code&gt;PizzaTopping&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Depth 2 around &lt;code&gt;PizzaTopping&lt;/code&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy3b3hre21p3fepnbcg5r.png" alt="Depth 1" width="800" height="921"&gt;&lt;/td&gt;
&lt;td&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdfhl77moppl4yxvt0azv.png" alt="Depth 2" width="800" height="1839"&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Drag the slider and the hierarchy unfolds as fast as you move. There is also a &lt;strong&gt;Sweep depth&lt;/strong&gt; button that animates the slider, handy for screen recordings.&lt;/p&gt;

&lt;h3&gt;
  
  
  How OWL becomes a class diagram
&lt;/h3&gt;

&lt;p&gt;Mermaid's &lt;code&gt;classDiagram&lt;/code&gt; maps onto OWL surprisingly well:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;In the ontology&lt;/th&gt;
&lt;th&gt;In the diagram&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rdfs:subClassOf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inheritance arrow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a defined class (&lt;code&gt;owl:equivalentClass&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;inheritance arrow and a dashed box&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;object property with a domain and range&lt;/td&gt;
&lt;td&gt;labelled association&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;some&lt;/code&gt; / &lt;code&gt;only&lt;/code&gt; restriction&lt;/td&gt;
&lt;td&gt;dashed dependency, if you switch them on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;datatype property&lt;/td&gt;
&lt;td&gt;attribute inside the class box&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Some OWL statements have no honest drawing: cardinalities, unions, complements and disjointness. OntoLive lists them in a side panel instead, written the way Protégé shows them, for example &lt;code&gt;hasTopping only MozzarellaTopping or TomatoTopping&lt;/code&gt;. The diagram shows the shape; the panel shows the rules.&lt;/p&gt;

&lt;p&gt;Because Pele is themed with CSS variables, dark mode needed no extra code. The same diagram picks up the page colours:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvsufmoguiixm43bqtr2h.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvsufmoguiixm43bqtr2h.png" alt="The Family sample ontology in dark mode, focused on Person" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Pele versus Mermaid.js, measured in your browser
&lt;/h3&gt;

&lt;p&gt;OntoLive has a switch that sends the same Mermaid text to Mermaid.js instead of Pele. A &lt;strong&gt;Race&lt;/strong&gt; button renders the current view 20 times with each and reports the median. Here is one run on a 36-class view of the Pizza ontology, in headless Chrome:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2wfe6k15iwxzv53mifr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2wfe6k15iwxzv53mifr.png" alt="Race result: Pele 0.7 ms, Mermaid.js 205.4 ms" width="799" height="127"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On a laptop browser, a 59-class view came out at 1.2 ms for Pele and 476 ms for Mermaid.js. Your numbers will differ by machine, which is why the page measures them rather than quoting them.&lt;/p&gt;

&lt;p&gt;The table is less convincing than the feel. With Mermaid.js selected, dragging the slider from 0 to 6 took about half a second per step, and three of the seven updates were dropped because the previous render had not finished. With Pele, the same sweep averaged about 5 ms per step and nothing was dropped. One feels like a slideshow. The other feels like a control.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching a language model build an ontology
&lt;/h2&gt;

&lt;p&gt;If diagrams are this cheap, they can keep up with a language model as it types.&lt;/p&gt;

&lt;p&gt;The second tab, &lt;strong&gt;Watch an LLM build one&lt;/strong&gt;, runs a small model (Qwen2.5 1.5B by default) on your GPU through &lt;a href="https://github.com/mlc-ai/web-llm" rel="noopener noreferrer"&gt;WebLLM&lt;/a&gt;. Give it a paragraph and it writes an ontology one fact per line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CLASS Employee
CLASS Manager
Manager ISA Employee
REL Employee reportsTo Manager
ATTR Employee name
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After every token, OntoLive parses what it has so far, turns it into Mermaid and redraws. You watch the classes appear and connect as the model writes.&lt;/p&gt;

&lt;p&gt;Small models make ontology mistakes, so every partial answer is also checked:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cycles:&lt;/strong&gt; A is a B, B is a C, C is an A. Nothing can be its own ancestor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Self-subclassing:&lt;/strong&gt; a class declared a subclass of itself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Near-duplicates:&lt;/strong&gt; &lt;code&gt;Manager&lt;/code&gt; and &lt;code&gt;Managers&lt;/code&gt; as two classes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Undeclared classes:&lt;/strong&gt; relations that point at classes the model never declared.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Offending classes turn red in the diagram as soon as the mistake is written. One of the sample paragraphs is a deliberate trap: it says people in a small startup call a manager "really just a team lead". The model took the bait:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkvyj0ynpzz3aoynwxjyw.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkvyj0ynpzz3aoynwxjyw.jpg" alt="The model writes " width="800" height="609"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Manager ISA Manager&lt;/code&gt; is wrong, and you can see it while the model is still writing. When it finishes, export the result as Turtle or open it in the explorer. The broken edges are left out of the export.&lt;/p&gt;

&lt;p&gt;This is the part I find most useful beyond ontologies. If you ask a model for structure (a plan, a schema, a workflow), you can check that structure while it streams and show the problems on the diagram. That only works if the diagram can keep up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is useful
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Anyone can open an ontology.&lt;/strong&gt; Send a link instead of asking someone to install Java and Protégé. Links keep the focus and depth, for example &lt;a href="https://vishalmysore.github.io/ontolive/#src=pizza&amp;amp;focus=Margherita&amp;amp;depth=2" rel="noopener noreferrer"&gt;Margherita at depth 2&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Big ontologies stay readable.&lt;/strong&gt; You see a neighbourhood, never the whole graph, and changing it costs nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Private files stay private.&lt;/strong&gt; Parsing and rendering happen in the tab. Nothing is uploaded.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model output becomes visible.&lt;/strong&gt; A streaming diagram with live checks shows when a model's structure goes wrong, and where.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The diagrams are plain Mermaid.&lt;/strong&gt; Copy the source into a README, a Notion page or an Obsidian note. Download the SVG with your theme baked in.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Live: &lt;a href="https://vishalmysore.github.io/ontolive/" rel="noopener noreferrer"&gt;vishalmysore.github.io/ontolive&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pizza, focused on Margherita: &lt;a href="https://vishalmysore.github.io/ontolive/#src=pizza&amp;amp;focus=Margherita&amp;amp;depth=2" rel="noopener noreferrer"&gt;open the link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Code (MIT): &lt;a href="https://github.com/vishalmysore/ontolive" rel="noopener noreferrer"&gt;github.com/vishalmysore/ontolive&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pele: &lt;a href="https://pele.run" rel="noopener noreferrer"&gt;pele.run&lt;/a&gt; and &lt;a href="https://github.com/obsidianmd/pele" rel="noopener noreferrer"&gt;github.com/obsidianmd/pele&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To run it locally:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/vishalmysore/ontolive
&lt;span class="nb"&gt;cd &lt;/span&gt;ontolive
npm &lt;span class="nb"&gt;install
&lt;/span&gt;npm run dev
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Drop your own &lt;code&gt;.owl&lt;/code&gt; or &lt;code&gt;.ttl&lt;/code&gt; file on the page, press &lt;strong&gt;Race&lt;/strong&gt;, and see what Pele does on your machine.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Credits: &lt;a href="https://github.com/obsidianmd/pele" rel="noopener noreferrer"&gt;Pele&lt;/a&gt; by Obsidian, &lt;a href="https://github.com/rdfjs/N3.js" rel="noopener noreferrer"&gt;N3.js&lt;/a&gt; for Turtle parsing, &lt;a href="https://github.com/mlc-ai/web-llm" rel="noopener noreferrer"&gt;WebLLM&lt;/a&gt; for in-browser models, and the &lt;a href="https://github.com/owlcs/pizza-ontology" rel="noopener noreferrer"&gt;Pizza ontology&lt;/a&gt;, which OntoLive loads from its GitHub repository.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>opensource</category>
      <category>tools</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Harness Engineering, Explained by Building One: An AI Agent Test Bench That Runs in Your Browser</title>
      <dc:creator>vishalmysore</dc:creator>
      <pubDate>Mon, 26 Oct 2026 12:32:59 +0000</pubDate>
      <link>https://dev.to/vishalmysore/harness-engineering-explained-by-building-one-an-ai-agent-test-bench-that-runs-in-your-browser-2m5m</link>
      <guid>https://dev.to/vishalmysore/harness-engineering-explained-by-building-one-an-ai-agent-test-bench-that-runs-in-your-browser-2m5m</guid>
      <description>&lt;h1&gt;
  
  
  Harness Engineering, Explained by Building One: An AI Agent Test Bench That Runs in Your Browser
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Your agent says "Done!" How do you know it's true? A practical introduction to harness engineering, with a working example you can open in a browser tab: no server, no API keys, no cloud bill.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://vishalmysore.github.io/harnessBench/" rel="noopener noreferrer"&gt;vishalmysore.github.io/harnessBench&lt;/a&gt; · &lt;strong&gt;Source code:&lt;/strong&gt; &lt;a href="https://github.com/vishalmysore/harnessBench" rel="noopener noreferrer"&gt;github.com/vishalmysore/harnessBench&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frtf2orvas5fzqaaycbg0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frtf2orvas5fzqaaycbg0.png" alt="HarnessBench: load a model, give it a task, watch every step, and get a verdict" width="800" height="556"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  An agent that lies, politely
&lt;/h2&gt;

&lt;p&gt;I asked a small language model to do something simple: &lt;em&gt;create three files called a.txt, b.txt and c.txt, then list them.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;It created one file. Then it announced: &lt;em&gt;"File 'c.txt' has been created successfully."&lt;/em&gt; It never made the other two and it never listed anything. By its own account, the task was done.&lt;/p&gt;

&lt;p&gt;That isn't a bug in one model. It's normal behaviour for AI agents, especially small ones. They skip steps, do things out of order, call tools that don't exist, repeat themselves, and finish with a confident summary either way. If you grade an agent by its last message, you're letting it grade its own homework.&lt;/p&gt;

&lt;p&gt;The fix is not a better prompt. It's a different kind of engineering.&lt;/p&gt;




&lt;h2&gt;
  
  
  What is harness engineering?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Harness engineering is the practice of building the controlled environment an AI agent runs inside.&lt;/strong&gt; The harness decides what the agent can see, what it is allowed to do, records everything it does, and checks the result against reality instead of the agent's own claims.&lt;/p&gt;

&lt;p&gt;The word comes from testing. In electronics and software, a &lt;em&gt;test harness&lt;/em&gt; is the rig you clamp a component into so you can feed it inputs, watch its outputs, and measure whether it behaves. Think of a car on a crash-test sled. The car is the interesting part, but the sled, the sensors and the high-speed cameras are what turn a crash into data.&lt;/p&gt;

&lt;p&gt;For AI agents, the model is the car. The harness is everything else.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why it became its own discipline
&lt;/h3&gt;

&lt;p&gt;Working with language models has gone through roughly three layers of concern:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prompt engineering:&lt;/strong&gt; &lt;em&gt;what do I say to the model?&lt;/em&gt; Wording, examples, instructions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context engineering:&lt;/strong&gt; &lt;em&gt;what information does the model see?&lt;/em&gt; Retrieved documents, memory, conversation history, tool results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Harness engineering:&lt;/strong&gt; &lt;em&gt;what world does the model act in, and how do I know what it did?&lt;/em&gt; Tools, permissions, sandboxes, loops, logs and judges.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The first two make a single answer better. Once a model starts &lt;em&gt;acting&lt;/em&gt; (running code, editing files, calling APIs, spending money) a good answer isn't enough. You need to contain the actions, observe them, and verify the outcome. That is the harness's job.&lt;/p&gt;

&lt;h3&gt;
  
  
  The five parts of a harness
&lt;/h3&gt;

&lt;p&gt;Almost every agent harness, from a research benchmark to a production coding assistant, comes down to the same five pieces:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The sandbox (environment).&lt;/strong&gt; An isolated place where the agent's actions land. In production it might be a Docker container, a virtual machine or a staging database. The key property is that it is fresh and disposable, so every run starts from the same known state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The tool registry (permissions).&lt;/strong&gt; The explicit list of actions the agent may take, each with a defined input. Anything not on the list does not run. This is the agent's permission boundary.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The agent loop (control).&lt;/strong&gt; The cycle of &lt;em&gt;model decides → harness executes → result is fed back&lt;/em&gt;, with limits on steps, tokens and time so a confused agent can't run forever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The recorder (observability).&lt;/strong&gt; A step-by-step log of what the agent thought, what it called, with which parameters, and what came back. It works like a flight data recorder: when something goes wrong, you can replay exactly how.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The judge (evaluation).&lt;/strong&gt; Code that inspects the &lt;em&gt;final state of the sandbox&lt;/em&gt; and the &lt;em&gt;action log&lt;/em&gt; and decides pass or fail. It never relies on the agent's description of what it did.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Get these five right and an unpredictable model becomes something you can measure, compare and trust a little more. Leave any one out and you're back to taking the agent's word for it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why you need one even for small experiments
&lt;/h3&gt;

&lt;p&gt;You don't need a harness only for big production systems. The moment you want to answer questions like &lt;em&gt;"Is model A better than model B at this task?"&lt;/em&gt; or &lt;em&gt;"Did my prompt change actually help?"&lt;/em&gt;, you need the same five parts: identical starting conditions, the same allowed actions, a record of what happened, and a judge that applies the same rules every time. Without them, you are comparing anecdotes.&lt;/p&gt;

&lt;p&gt;The catch is that harnesses usually need infrastructure: servers, containers, API keys, somewhere to keep logs. That's a lot of setup just to learn the idea. So I built the smallest complete harness I could, and put it in a web page.&lt;/p&gt;




&lt;h2&gt;
  
  
  HarnessBench: all five parts in one browser tab
&lt;/h2&gt;

&lt;p&gt;HarnessBench is a single web page. It downloads an open-source language model once, runs it on your own GPU through &lt;a href="https://github.com/mlc-ai/web-llm" rel="noopener noreferrer"&gt;WebLLM&lt;/a&gt; and WebGPU, and wraps it in a complete harness. Nothing is sent to a server, and after the first visit the model loads from the browser's cache in seconds.&lt;/p&gt;

&lt;p&gt;Here is how each part of harness engineering maps onto the app:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Harness part&lt;/th&gt;
&lt;th&gt;In HarnessBench&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Sandbox&lt;/td&gt;
&lt;td&gt;A fresh in-memory "file system" and variable store, reset before every run&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool registry&lt;/td&gt;
&lt;td&gt;Six actions: &lt;code&gt;create_file&lt;/code&gt;, &lt;code&gt;read_file&lt;/code&gt;, &lt;code&gt;list_files&lt;/code&gt;, &lt;code&gt;delete_file&lt;/code&gt;, &lt;code&gt;set_variable&lt;/code&gt;, &lt;code&gt;get_variable&lt;/code&gt;, plus &lt;code&gt;final_answer&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent loop&lt;/td&gt;
&lt;td&gt;One JSON decision per step (&lt;code&gt;thought&lt;/code&gt;, &lt;code&gt;action&lt;/code&gt;, &lt;code&gt;parameters&lt;/code&gt;), a step limit, and a cap on reply length&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recorder&lt;/td&gt;
&lt;td&gt;A live token stream, a step-by-step trace, and the full action log in the sandbox state&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Judge&lt;/td&gt;
&lt;td&gt;Plain JavaScript that checks which actions were called and what is actually in the sandbox&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fho585dm7wnostn262y25.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fho585dm7wnostn262y25.png" alt="Model loaded and ready to run" width="800" height="422"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  A run that passes
&lt;/h3&gt;

&lt;p&gt;Pick a task (or click one of the "Try" chips) and press &lt;strong&gt;Run Evaluation&lt;/strong&gt;. Here the agent was asked: &lt;em&gt;"Create a file called note.txt with the content 'hello world', then read it back."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj815z2l7tkqd5ke5r9t0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj815z2l7tkqd5ke5r9t0.png" alt="A passing run: create_file, then read_file, then final_answer, judged PASSED" width="800" height="1078"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Everything the harness knows is on screen:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The verdict&lt;/strong&gt; at the top, with the judge's reason. Note what the reason says: &lt;code&gt;"note.txt" exists: "hello world"&lt;/code&gt;. That's a fact read from the sandbox, not a quote from the model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The live token stream&lt;/strong&gt;, showing the model's raw output as it's generated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The agent trace&lt;/strong&gt;, one card per step: what the model was thinking, which action it chose, and what happened.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The sandbox state&lt;/strong&gt;, showing the actual files, variables and action log after the run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A longer task shows why the recorder matters. Asked to create three files and list them, Llama 3.2 3B created &lt;code&gt;a.txt&lt;/code&gt;, &lt;code&gt;b.txt&lt;/code&gt; and &lt;code&gt;c.txt&lt;/code&gt; in order, listed them, and finished. It also listed them a second time for no reason. The run still passes, but the trace shows you the wasted step. A single score would have hidden it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmaq92zng5evuotj6g8ud.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmaq92zng5evuotj6g8ud.png" alt="Multi-step trace: three create_file calls, list_files, then final_answer" width="800" height="1114"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  A run that fails, and why that's the useful part
&lt;/h3&gt;

&lt;p&gt;Here is that same three-file task given to a smaller model, SmolLM2 1.7B. This is the "polite liar" from the start of the article. It created only &lt;code&gt;c.txt&lt;/code&gt;, then declared the job done.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdaigvpeg5t1oeuwo5ts6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdaigvpeg5t1oeuwo5ts6.png" alt="The judge catches an incomplete run: list_files was never called and two files are missing" width="" height=""&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The judge doesn't care what the model said. It sees that &lt;code&gt;list_files&lt;/code&gt; was never called and that &lt;code&gt;a.txt&lt;/code&gt; and &lt;code&gt;b.txt&lt;/code&gt; don't exist, so the verdict is &lt;strong&gt;FAILED&lt;/strong&gt;. This is the whole point of a harness: the agent's confidence has no effect on the score.&lt;/p&gt;

&lt;h3&gt;
  
  
  The permission boundary in action
&lt;/h3&gt;

&lt;p&gt;The tool registry isn't a suggestion. Ask the agent to &lt;em&gt;"Ring the bell using the ring_bell action, then create bell.txt"&lt;/em&gt; and it will dutifully try to call &lt;code&gt;ring_bell&lt;/code&gt;. That action isn't registered, so the harness stops the run at once and nothing else executes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb18w2bzz7s84wxkt8wtw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb18w2bzz7s84wxkt8wtw.png" alt="Unknown action blocked: ring_bell is not a registered action" width="800" height="866"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In a real system, the same rule is what keeps an agent from calling a "delete everything" endpoint just because it believes one exists. The model can &lt;em&gt;ask&lt;/em&gt; for anything; only registered actions ever run.&lt;/p&gt;




&lt;h2&gt;
  
  
  How the judge decides
&lt;/h2&gt;

&lt;p&gt;The judge is ordinary code, not another AI model. That's deliberate: a judge should be predictable and give the same verdict for the same behaviour every time.&lt;/p&gt;

&lt;p&gt;It runs two checks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Did the agent do every step?&lt;/strong&gt; Words in the task map to the actions they require: "create" needs &lt;code&gt;create_file&lt;/code&gt;, "read" needs &lt;code&gt;read_file&lt;/code&gt;, "delete" needs &lt;code&gt;delete_file&lt;/code&gt;, "list" needs &lt;code&gt;list_files&lt;/code&gt;, and so on. If a required action never appears in the log, the run fails, whatever the agent claimed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is the world in the right state?&lt;/strong&gt; Every file named in the task must exist at the end, or must be gone if the task asked for a delete. Variable tasks must leave a stored variable.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Because every run starts from an empty sandbox and the judge is plain code, two runs with the same behaviour always get the same verdict. That reproducibility is what makes it fair to compare models side by side.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dealing with messy model output
&lt;/h2&gt;

&lt;p&gt;A harness also has to cope with the model breaking the rules of the conversation, not just the rules of the task. Small models are bad at clean structured output. While testing, a frequent habit was writing two decisions back to back, &lt;code&gt;{...} {...}&lt;/code&gt;, which isn't valid JSON. Others wrap the answer in a chatty sentence or a code fence.&lt;/p&gt;

&lt;p&gt;HarnessBench handles this the way a production harness should:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It takes the &lt;strong&gt;first complete JSON object&lt;/strong&gt; in the reply (one action per step) and uses &lt;a href="https://github.com/josdejong/jsonrepair" rel="noopener noreferrer"&gt;jsonrepair&lt;/a&gt; to fix small mistakes such as a trailing comma or a missing closing brace.&lt;/li&gt;
&lt;li&gt;If a reply still has no action, the harness tells the model its format was wrong and lets it try again. That retry uses up one of its steps.&lt;/li&gt;
&lt;li&gt;Each reply is &lt;strong&gt;capped at 384 tokens&lt;/strong&gt;, so a runaway generation can't stall a step for minutes.&lt;/li&gt;
&lt;li&gt;Old steps are &lt;strong&gt;trimmed from the conversation&lt;/strong&gt; so long runs stay inside a small model's memory window.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this makes the model smarter. It makes the &lt;em&gt;measurement&lt;/em&gt; more reliable, so that a failed run means the agent failed and not that the plumbing did.&lt;/p&gt;

&lt;h2&gt;
  
  
  A harness has to test itself, too
&lt;/h2&gt;

&lt;p&gt;If the judge has a bug, every verdict is wrong. So HarnessBench includes a &lt;strong&gt;Harness Verification&lt;/strong&gt; panel: 29 deterministic tests that run without any model. They cover every action, sandbox isolation, unknown-action blocking, judge pass and fail cases (including "only one of three files was created"), JSON repair and context trimming.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5f28qireci7hzh9gyx78.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5f28qireci7hzh9gyx78.png" alt="All 29 harness self-tests passing" width="669" height="653"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  What I saw on a real laptop
&lt;/h2&gt;

&lt;p&gt;Every screenshot in this article comes from real models running on WebGPU in Chrome, on a laptop with an integrated Intel Iris Xe GPU:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Example tasks passed&lt;/th&gt;
&lt;th&gt;What it did&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.2 3B Instruct (q4f32)&lt;/td&gt;
&lt;td&gt;4 of 4&lt;/td&gt;
&lt;td&gt;Followed steps in order; sometimes repeated a harmless &lt;code&gt;list_files&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SmolLM2 1.7B Instruct (q4f16)&lt;/td&gt;
&lt;td&gt;1 of 4&lt;/td&gt;
&lt;td&gt;Often stopped after the first step, or kept retrying a failing action&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On this hardware the 3B model produces about 3 tokens per second. One step takes roughly 8 to 20 seconds, and a full task takes one to two minutes. The first visit downloads about 1.8 GB; after that it loads from cache in seconds.&lt;/p&gt;

&lt;p&gt;The interesting result isn't the scores. It's that the harness made the difference between the two models &lt;em&gt;visible and explainable&lt;/em&gt;. You can see exactly where the smaller model gave up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is useful
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Learn harness engineering by touching it.&lt;/strong&gt; All five parts are on one screen and in a few hundred lines of plain JavaScript, so you can see how they connect.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compare models honestly.&lt;/strong&gt; Swap the model, run the same tasks, and get verdicts based on what happened in the sandbox.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;See how agents fail, not just that they fail.&lt;/strong&gt; Early stopping, wrong order, invented tools and loops all show up in the trace.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Private by default.&lt;/strong&gt; Prompts and results never leave your machine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zero setup.&lt;/strong&gt; Share a link. Anyone with Chrome or Edge and a WebGPU-capable GPU can watch an agent run and get graded.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limitations
&lt;/h2&gt;

&lt;p&gt;To be clear about what this is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The tasks are toy tasks.&lt;/strong&gt; Six in-memory tools are enough to show the ideas, not to benchmark real-world agents.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The judge is keyword-based.&lt;/strong&gt; It works for the included kinds of tasks. A task worded very differently might not map to the right required actions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Results vary.&lt;/strong&gt; Small models behave differently across hardware and library versions. Treat the table above as one observation, not a leaderboard.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;WebGPU is required.&lt;/strong&gt; It works in recent Chrome and Edge with hardware acceleration enabled. Bigger models need more GPU memory than many laptops have. If a model starts producing garbled text, clearing the site's data (which resets the cached model) usually fixes it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;Prompting makes an agent &lt;em&gt;say&lt;/em&gt; better things. A harness tells you what it actually &lt;em&gt;did&lt;/em&gt;. Before you trust any agent with real work, it's worth asking five questions: Where do its actions land? What is it allowed to do? Who stops it when it loops? Where is the record? And who checks the result, other than the agent itself?&lt;/p&gt;

&lt;p&gt;Open the &lt;a href="https://vishalmysore.github.io/harnessBench/" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;, load &lt;strong&gt;Llama 3.2 3B&lt;/strong&gt;, click a "Try" chip and press &lt;strong&gt;Run Evaluation&lt;/strong&gt;. Then try to break it: ask for a tool that doesn't exist, or give a small model a five-step task, and watch the harness catch it.&lt;/p&gt;




&lt;h3&gt;
  
  
  Credits
&lt;/h3&gt;

&lt;p&gt;HarnessBench builds on open-source work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/mlc-ai/web-llm" rel="noopener noreferrer"&gt;WebLLM&lt;/a&gt; and &lt;a href="https://github.com/mlc-ai/mlc-llm" rel="noopener noreferrer"&gt;MLC LLM&lt;/a&gt; by the MLC team, for in-browser LLM inference on WebGPU and the prebuilt quantized models.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.llama.com/" rel="noopener noreferrer"&gt;Llama 3.2&lt;/a&gt; by Meta and &lt;a href="https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct" rel="noopener noreferrer"&gt;SmolLM2&lt;/a&gt; by Hugging Face, plus the other model families offered in the app (Phi-3.5, Gemma 2).&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/josdejong/jsonrepair" rel="noopener noreferrer"&gt;jsonrepair&lt;/a&gt; by Jos de Jong, for recovering malformed JSON.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://www.w3.org/TR/webgpu/" rel="noopener noreferrer"&gt;WebGPU&lt;/a&gt; standard and its browser implementations.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Disclaimer
&lt;/h3&gt;

&lt;p&gt;HarnessBench is an educational demo, not a production evaluation tool. Its verdicts come from simple rule-based checks on toy tasks and should not be used to make claims about any model's real-world capability or safety. Model weights are downloaded from their publishers and are covered by their own licenses. This is a personal project and is not affiliated with or endorsed by any of the projects or companies named above.&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>javascript</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
