DEV Community: Nilofer 🚀

RAG Pipeline Stress Tester: Battle-Test Your RAG System Before It Reaches Production

Nilofer 🚀 — Tue, 12 May 2026 11:45:30 +0000

Most RAG systems get tested with a handful of happy-path questions. Someone asks "what is machine learning?", gets a reasonable answer, and calls it done. Then it goes to production and users find the edge cases, hallucinations on out-of-scope questions, failed refusals on adversarial prompts, latency that collapses under real concurrent load.

RAG Pipeline Stress Tester is a battle-testing toolkit that finds these issues before deployment.

What It Does

Takes any HTTP RAG endpoint and hammers it with 7 categories of adversarial queries under configurable concurrent load.
Tracks relevance, hallucination, refusal quality, and latency for every query sent.
Scores everything into a composite health score from 0 to 100.
Breaks results down by query category so you know exactly which failure modes are causing issues.
Measures p50, p95, and p99 latency under realistic concurrent load, not just single-request response times.
Produces an HTML report with interactive charts and a JSON report for CI/CD integration.

Why This Exists

Before deploying a RAG system to production, four questions need answers:

Does it hallucinate when asked about things not in the corpus?
Does it refuse appropriately on out-of-scope questions?
Does it stay consistent when the same question is asked multiple ways?
Does it hold up under load 10, 25, 50 concurrent users?

Manual testing cannot answer these questions at scale. This tool does it automatically.

Without stress testing - hallucinations get discovered in production, users find edge cases first, latency under load is guesswork, and there is no audit trail.

With this tool - hallucinations are caught before deployment, you find edge cases in batch, p50/p95/p99 latency is measured at realistic concurrency, and every test run produces a timestamped JSON and HTML report.

The 7 Query Categories

The tool ships with 7 pre-built adversarial query banks, each targeting a specific failure mode:

out_of_scope - Questions with no answer in the corpus, tests hallucination resistance
adversarial - Prompt injection and jailbreak attempts, tests instruction-following safety
ambiguous - Queries with multiple valid interpretations, tests disambiguation
multilingual - Non-English queries, tests language handling
temporal - Time-sensitive questions that depend on stale data
negation - "What is NOT X" style questions, a common failure mode
compound - Multi-part questions requiring multiple retrievals

You can add your own queries by appending lines to any file in query_bank/.

Health Score

Every test run produces a composite Health Score from 0 to 100:

≥ 80  EXCELLENT   Production-ready
≥ 60  GOOD        Minor issues, review before deploying
≥ 40  FAIR        Significant issues, fix first
 < 40  POOR        Critical failures, do not deploy

Calculated from five weighted components:

Architecture

main.py             Typer CLI — entry point and orchestration
adversarial.py      Query generator — 7 categories, pre-built + corpus-generated
loader.py           Async load driver — aiohttp, configurable concurrency
evaluator.py        Scorer — hallucination, precision, refusal, consistency
reporter.py         Report generator — HTML (Chart.js) + JSON output
corpus_analyzer.py  Optional: generate targeted queries from your own documents
query_bank/         7 pre-built adversarial query files (one per line)
tests/              58 pytest tests (no live endpoint needed)

Install

pip install -r requirements.txt

The endpoint the tester sends requests to must accept POST with {"query": "..."} and return JSON containing either a response or answer field. Any HTTP status other than 200 is counted as an error.

Running a Stress Test

The core command runs a full stress test against your RAG endpoint:

# Basic — 10 concurrent users, 60-second run
python3 main.py stress-test \
  --endpoint http://localhost:8000/query \
  --concurrency 10 \
  --duration 60

# Test only specific query categories
python3 main.py stress-test \
  --endpoint http://localhost:8000/query \
  --query-types out_of_scope,adversarial,multilingual

# Custom output directory
python3 main.py stress-test \
  --endpoint http://localhost:8000/query \
  --output ./my-reports

Here is what a real terminal output looks like:

🚀 Starting RAG Stress Test
   Endpoint: http://localhost:8000/query
   Concurrency: 5
   Duration: 20s

📊 Generating test queries...
   Generated 350 test queries

⚡ Running load tests...
📈 Evaluating results...
📝 Generating reports...

✅ Stress test complete!
   JSON Report: reports/stress_test_results.json
   HTML Report: reports/stress_test_report.html

=======================================================
  Overall Health Score : 57.1/100
  Status               : FAIR - Significant issues detected
  Total requests       : 6355
  Error rate           : 0.0%
  Precision score      : 2.1%
  Hallucination rate   : 22.5%
  Refusal rate         : 77.5%
  Consistency score    : 72.1%
  Latency p50/p95/p99  : 2.9 / 6.3 / 8.7 ms

  Query Type          Count   Halluc%   Refusal%    AvgLat
  ------------------ ------  --------  ---------  --------
  adversarial           205     35.1%      64.9%      3.3ms
  ambiguous             250     12.0%      88.0%      3.2ms
  compound              200     22.0%      78.0%      4.0ms
  multilingual          250     10.0%      90.0%      3.1ms
  negation              200     20.0%      80.0%      5.3ms
  out_of_scope          250     20.0%      80.0%      4.0ms
  temporal              200     38.0%      62.0%      3.1ms

  Recommendations:
    - Low precision score. Enhance retrieval mechanism and relevance ranking.
    - Moderate: Several areas need improvement for production readiness.
=======================================================

Quick Sanity Check

For a fast check before a full run, quick-test runs 35 sample queries - 5 per category and prints the health score without writing any report files:

python3 main.py quick-test --endpoint http://localhost:8000/query

🔍 Running quick sanity test...
   Testing with 35 sample queries

🎯 Quick Test Health Score: 72.4/100
   ✅ Endpoint appears functional

Generate Queries From Your Own Corpus

The analyze-corpus command analyzes your own .txt, .md, or .json files, extracts domain keywords, and produces targeted in-scope, out-of-scope, and adversarial query files you can drop into query_bank/:

python3 main.py analyze-corpus \
  --corpus ./my-docs \
  --output ./query_bank \
  --num-queries 50

📚 Analyzing corpus: ./my-docs
   Generated 50 in_scope queries → query_bank/in_scope_generated.txt
   Generated 50 out_of_scope queries → query_bank/out_of_scope_generated.txt
   Generated 50 adversarial queries → query_bank/adversarial_generated.txt

✅ Corpus analysis complete!

For very small corpora, lower the keyword frequency threshold:

python3 main.py analyze-corpus \
  --corpus ./my-docs \
  --output ./query_bank \
  --num-queries 20 \
  --min-word-freq 1

Configuration

Edit config.yaml to customise load levels, thresholds, and reporting. The --endpoint CLI flag always takes precedence over config.yaml.

load.concurrency_levels - Concurrent user levels to test, for example [1, 5, 10, 25]
load.ramp_mode - If true, steps through each concurrency level; if false, runs at the first level for the full duration
load.duration_seconds - How long to run at each concurrency level
load.rate_limit_per_second - Maximum requests per second
evaluation.hallucination_threshold - Keyword-overlap score below which a response is flagged as a potential hallucination
evaluation.refusal_keywords - Phrases that indicate a refused answer
reporter.output_dir - Where to save HTML and JSON reports

Pass the config file with --config:

python3 main.py stress-test \
  --endpoint http://localhost:8000/query \
  --config config.yaml

Output Reports

Each test run saves two files to ./reports/ or your --output path:

stress_test_results.json - Machine-readable raw data with per-query latency, success and failure flags, hallucination scores, and a per-type breakdown. Useful for CI/CD integration.

stress_test_report.html - Interactive dashboard with a health score badge coloured by band, metric cards covering success rate, precision, hallucination, latency p95 and consistency, a bar chart of success rate by query type, a grouped bar chart of hallucination and refusal rate by query type, a latency distribution histogram, and prioritised recommendations.

Endpoint Requirements

The tester sends:

POST /your-endpoint
{"query": "What is machine learning?"}

It expects a JSON response containing either a response or answer field:

{"response": "Machine learning is..."}

Any HTTP status other than 200 is counted as an error.

Running Tests

python3 -m pytest tests/ -v

58 tests covering all modules. Uses aioresponses to mock HTTP - no live RAG endpoint required.

Project Structure

rag-pipeline-stress-tester/
├── main.py             # CLI entry point
├── adversarial.py      # Query generators (7 types)
├── loader.py           # Async load test driver
├── evaluator.py        # Scoring and metrics
├── reporter.py         # HTML + JSON report generator
├── corpus_analyzer.py  # Optional corpus-based query generation
├── config.yaml         # Test configuration
├── requirements.txt
├── query_bank/         # 7 pre-built adversarial query files
└── tests/              # 58 pytest tests

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a toolkit that could stress test any RAG endpoint automatically, not just for latency but for hallucination, refusal quality, and consistency under concurrent load. The tool needed to work against any endpoint with a standard request format, produce structured reports for CI/CD integration, and ship with pre-built adversarial query banks covering the failure modes that matter most before a RAG deployment.

xNEO built the full implementation: The Typer CLI with all three commands, the async load driver backed by aiohttp, the query generator covering all 7 adversarial categories, the hallucination and precision scorer, the composite health score calculator with five weighted components, the HTML report generator with Chart.js charts, the JSON reporter, the corpus analyzer for generating domain-specific queries, and the full test suite of 58 tests with HTTP mocked via aioresponses.

How You Can Use and Extend This With NEO

Use it as a pre-deployment gate for every RAG system.
Before any RAG endpoint goes to production, run a stress test against it. The health score gives you a single number, below 60 means review before deploying, below 40 means do not deploy. The per-category breakdown tells you exactly which failure modes are causing the score to drop.

Use it with your own domain queries.
The pre-built query banks are general purpose. For domain-specific testing, run analyze-corpus on your own documents to generate in-scope, out-of-scope, and adversarial queries targeted at your actual corpus, then drop them into query_bank/ and run the stress test.

Integrate the JSON report into CI/CD.
stress_test_results.json is machine-readable and contains per-query latency, hallucination scores, and the health score. A CI step that reads the health score and fails the pipeline below a threshold turns RAG quality into an automated deployment gate.

Extend it with additional query categories.
The 7 query banks are plain text files in query_bank/, one query per line. Adding a new category for a specific failure mode your RAG system faces means adding a new file to query_bank/ and registering it in adversarial.py.

Final Notes

RAG systems fail in predictable ways, hallucination on out-of-scope questions, collapsed latency under load, inconsistent refusals. RAG Pipeline Stress Tester surfaces all of these before production, with a structured health score, per-category metrics, and reports that fit directly into a CI/CD pipeline.

The code is at https://github.com/dakshjain-1616/RAG-pipeline-stress-tester
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Orbis: Turn Any GitHub Repository Into an Interactive 3D Dependency Graph

Nilofer 🚀 — Sat, 09 May 2026 10:58:10 +0000

Understanding a large codebase is hard. You clone it, start reading files, and quickly lose track of how everything connects. Which modules are most depended on? Where are the circular dependencies? What would break if you refactored this file?

Orbis answers these questions visually. Paste a GitHub repository URL, and Orbis clones it, parses the ASTs across Python, JavaScript, TypeScript, Go, Rust, and Java, detects architectural patterns, and renders the entire codebase as a navigable 3D force-directed graph. Click any module to inspect its dependencies, metrics, and exported symbols. Ask the built-in AI assistant questions like "which module should I refactor first?" and get answers grounded in the actual code structure.

Features

3D force-directed graph - Nodes sized by lines of code, colored by type, with animated directional particles on edges.

Multi-language AST parsing - Python, JavaScript/TypeScript, Go, Rust, and Java via tree-sitter.

AI chat assistant - Ask Claude questions about the analyzed codebase. Questions like "Which modules have circular dependencies?" or "Where should I add feature X?" are answered with full architectural context.

Architectural insights - Auto-detected issues including god modules, high coupling, and circular dependencies, each with severity ratings.

Focus Mode - Dim unconnected nodes to trace dependency paths clearly.

Shareable URLs - ?repo=https://github.com/... auto-triggers analysis on load, making it easy to share a specific codebase view.

Recent history - Last 5 repos stored locally for quick re-analysis.

Demo mode — Load a pre-analyzed snapshot without a GitHub clone.

Tech Stack

Backend: FastAPI + Server-Sent Events (SSE)
AST Parsing: tree-sitter (Python, JS/TS, Go, Rust, Java)
AI Integration: Claude Opus 4.6 via Anthropic API
3D Rendering: 3d-force-graph + Three.js
Frontend: Vanilla JS SPA - no build step

Quick Start

1. Clone and install

cd orbis
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

2. Set up environment

cp .env.example .env
# Edit .env and add your ANTHROPIC_API_KEY for the AI chat feature

Get an API key at console.anthropic.com. The AI chat feature requires ANTHROPIC_API_KEY in your environment. It degrades gracefully, if the key is missing, the chat panel shows an error message rather than breaking the rest of the app.

3. Run

uvicorn main:app --host 0.0.0.0 --port 8001

Open http://localhost:8001.

Docker

docker build -t orbis .
docker run -p 8001:8001 -e ANTHROPIC_API_KEY=sk-ant-... orbis

Usage

Once running, the workflow is straightforward:

Enter a public GitHub repository URL - for example https://github.com/expressjs/express
Optionally specify a branch
Click Analyze - Orbis clones the repo, parses ASTs, and builds the graph in roughly 5–30 seconds
Explore the 3D graph - click a node to open its detail drawer, scroll to zoom, drag to rotate
Use Focus Mode to highlight a node's direct connections
Use layer filter chips to show or hide architectural layers
Ask the AI assistant questions about the codebase in the chat panel

Keyboard Shortcuts

R: Reset camera
P: Pause/resume rotation
F: Toggle Focus Mode
/: Focus search box
Esc: Close detail drawer / exit Focus Mode

Architecture

The project has four files at its core - a FastAPI backend, a single-file AST parser, and a vanilla JS frontend with no build step:

main.py           FastAPI backend — SSE streaming for /analyze, /chat
neo_parser.py     Multi-language AST parser (tree-sitter)
static/
  index.html      Single-page frontend (3d-force-graph + Three.js)
save_analysis.py  Utility: pre-generate demo data from a repo

The backend streams analysis progress to the frontend via Server-Sent Events, The backend streams analysis progress to the frontend via Server-Sent Events while cloning and analyzing the repo.

API Endpoints

Output Schema

/analyze emits SSE events and completes with a complete event containing:

{
  "schema_version": "2.0",
  "architecture_type": "MVC",
  "languages": { "python": 42 },
  "summary": "Codebase contains 42 modules...",
  "nodes": [{
    "id": "requests/auth",
    "label": "auth.py",
    "type": "utility",
    "language": "python",
    "lines_of_code": 315,
    "complexity": "medium",
    "exported_symbols": ["AuthBase", "HTTPBasicAuth"],
    "internal_dependencies": ["requests/compat"],
    "external_dependencies": [],
    "metrics": { "functions_total": 12, "classes": 4 }
  }],
  "edges": [{ "from": "requests/api", "to": "requests/auth", "type": "import" }],
  "insights": [{
    "type": "high_coupling",
    "severity": "high",
    "title": "High fan-in on requests/models",
    "description": "14 modules import this file directly.",
    "affected_nodes": ["requests/models"],
    "recommendation": "Consider splitting into smaller focused modules."
  }]
}

Each node carries its lines of code, complexity rating, exported symbols, and both internal and external dependencies. The insights block surfaces architectural issues automatically, high coupling, circular dependencies, and god modules - each with a severity rating and a specific recommendation.

Supported Languages

Python - .py
JavaScript/TypeScript - .js, .mjs, .cjs, .jsx, .ts, .tsx
Go - .go
Rust - .rs
Java - .java

AI Chat

The chat assistant uses Claude Opus 4.6 and receives the full architectural graph as context - node list, dependencies, insights, and summary. It can answer questions like:

"What does the auth module depend on?"
"Why are there circular dependencies between X and Y?"
"Which module should I refactor first?"
"Where would I add a caching layer?"

The assistant's answers are grounded in the actual parsed structure of the codebase - not generic advice. Requires ANTHROPIC_API_KEY in your environment.

Development

# Run with auto-reload
uvicorn main:app --reload --port 8001

# Re-generate demo data
python save_analysis.py

How I Built This Using NEO

The idea was a tool that turns any GitHub repository into an interactive 3D graph, something a developer could paste a URL into and immediately understand the architecture without reading a single file. The requirements included multi-language AST parsing, automatic architectural issue detection, an AI assistant grounded in the actual code structure, and a frontend that required no build step.

NEO built the full stack from that description: the FastAPI backend with SSE streaming for real-time analysis progress, the multi-language AST parser in neo_parser.py covering Python, JavaScript, TypeScript, Go, Rust, and Java via tree-sitter, the 3D force-directed graph frontend in vanilla JS, the Claude Opus 4.6 chat assistant with full architectural context, the insights engine detecting god modules, high coupling, and circular dependencies with severity ratings, and the demo mode with pre-generated analysis data.

How You Can Use and Extend This With NEO

Use it to onboard onto an unfamiliar codebase.
Instead of spending hours reading files to understand how a project is structured, paste the repo URL into Orbis and get an immediate visual map of every module, its dependencies, and the architectural issues that already exist. The AI assistant can then answer specific questions about the structure without you having to trace imports manually.

Use it during code review to understand structural impact.
When reviewing a large pull request, run Orbis on the repo and use the insights panel to see whether high coupling, circular dependencies, or god modules exist in the areas being changed. The AI assistant can answer specific questions about how the affected modules connect to the rest of the codebase.

Use it to plan a refactor.
Ask the AI assistant "which module should I refactor first?" or "where would I add a caching layer?" and get answers grounded in the actual dependency graph. The focus mode lets you isolate a specific module and trace exactly what depends on it before touching anything.

Extend it with additional language parsers.
neo_parser.py already handles five languages via tree-sitter. Adding a new language - Ruby, C++, Swift - follows the same parser pattern and surfaces automatically in the language filter chips and the supported languages list without touching the frontend or the API.

Final Notes

Orbis makes codebase architecture something you can see and navigate rather than something you have to reconstruct in your head. A 3D dependency graph, multi-language AST parsing, automatic architectural issue detection, and an AI assistant that knows the actual structure - all from a single repo URL.

The code is at https://github.com/dakshjain-1616/Orbit-dependency-visualised
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

SmolVLM2 Edge Vision Agent: Visual Monitoring Without a GPU or Cloud API

Nilofer 🚀 — Thu, 07 May 2026 11:43:31 +0000

Running vision AI locally has always had a catch, you need a GPU, or you need to send frames to a cloud API and pay per call. SmolVLM2-2.2B changes that. It is a 2.2B-parameter multimodal model specifically designed for CPU inference, and this agent is built around it.

SmolVLM2 Edge Vision Agent is a fully offline edge vision agent that ingests a live webcam feed or an image folder, detects motion using frame-difference analysis, triggers VLM analysis only on scene changes, and persists structured observations to a local SQLite database with a FastAPI web dashboard for review. No API costs. No network calls after the first model download. 16GB RAM, no GPU required.

Project Overview

The agent does five things:

Ingests a live webcam feed or an image folder as input
Performs continuous visual monitoring, frame-difference based motion detection that triggers VLM analysis only on scene changes
Describes new objects, reads text from images - receipts, whiteboards, signs, and logs everything as structured observations
Persists observations to a local SQLite database with timestamps, thumbnails, descriptions, and confidence scores
Exposes a FastAPI web dashboard with live feed, latest observations, and a searchable log

It runs entirely offline. The model auto-downloads on first run and is cached locally from that point forward.

Use cases: home security camera analysis, document digitization pipelines, accessibility tools.

Architecture

The key design decision is the motion gate. Running a 2.2B-parameter model on every frame would be unusable on CPU hardware, inference is not instant. The agent solves this by running frame-difference motion detection on every frame first, and only invoking the VLM when a scene change is detected above the configured threshold.

Per-frame timeline:
Every frame goes through motion detection first. If the frame difference is below the threshold, the frame is dropped with no further processing. If motion is detected, the VLM runs, produces a description, and the observation is stored in SQLite with a thumbnail. This design means expensive model inference only happens when something actually changes in the scene, keeping a Pi-class CPU usable while still describing every meaningful scene change.

The FRAME_DIFF_THRESHOLD defaults to 0.15 and controls how sensitive the motion detector is. A higher value means less sensitivity, minor lighting changes or small movements are ignored. A lower value triggers the VLM more frequently.

Prerequisites

Python: 3.11 or newer.
RAM: 16GB minimum for the real model; less is fine in --mock mode.
Disk: ~5GB free for the model cache.
OS: Linux, macOS, or WSL2 on Windows - uses OpenCV, and webcam access requires native camera support.
No GPU required - SmolVLM2-2.2B is designed for CPU inference.

Installation

git clone https://github.com/dakshjain-1616/smolvlm2-edge-agent.git
cd smolvlm2-edge-agent
make install                                  # pip install -e .
cp .env.example .env                          # then edit values as needed

The make install command runs pip install -e . which installs the package and its pinned runtime dependencies from requirements.txt. The .env.example file contains all documented environment variables, copy it to .env and edit the values you want to override before running.

Configuration

Every tunable is configurable via CLI flags and environment variables. CLI flags take precedence over environment variables. All variables are documented in .env.example in the repository.

MODEL_NAME - HuggingFace model id, default: HuggingFaceTB/SmolVLM2-2.2B-Instruct
USE_MOCK_MODE - bypass model loading with deterministic stub responses, default: false
MODEL_CACHE_DIR - where the HuggingFace model is cached on disk, default: ./models
DB_PATH - SQLite database file path, default: ./data/observations.db
FRAME_DIFF_THRESHOLD - motion sensitivity on a 0–1 scale, higher means less sensitive, default: 0.15
MIN_CONFIDENCE - minimum VLM confidence required to log an observation, default: 0.5
PROCESSING_INTERVAL - seconds between frame samples, default: 1.0
MAX_OBSERVATIONS - cap on stored rows, older observations are pruned, default: 10000
DASHBOARD_HOST - FastAPI bind host, default: 0.0.0.0
DASHBOARD_PORT - FastAPI port, default: 8080
INPUT_SOURCE - camera index or path to image folder, default: 0
OUTPUT_DIR - where observation artifacts are written, default: ./data/observations/
THUMBNAIL_DIR - where frame thumbnails are saved, default: ./data/thumbnails/
LOG_LEVEL - Python logging level, default: INFO
LOG_FILE - optional log file path, default: ./data/agent.log

MIN_CONFIDENCE is worth paying attention to — observations where the VLM's confidence falls below 0.5 are not stored. Raising this filters out uncertain detections. Lowering it logs more, including lower-confidence observations.

Usage

Quick start - mock mode, no model download

The fastest way to verify the full pipeline is mock mode. It bypasses model loading entirely and uses deterministic stub responses, so you can confirm the agent loop, database writes, thumbnail generation, and dashboard all work before committing to the 5GB model download:

mkdir -p data/test_images
python3 -m src --mock --input ./data/test_images --duration 30

This runs the agent for 30 seconds against the data/test_images/ folder using the mock VLM, populates data/observations.db, and writes thumbnails to data/thumbnails/.

Run against a webcam

python3 -m src --input 0 --port 8080

Camera index 0 is the default device. For additional cameras, use index 1, 2, and so on. Open http://localhost:8080 in a browser to see the live dashboard. The dashboard shows the live feed, the most recent observations, and a searchable log of everything the agent has recorded.

Run against an image folder

python3 -m src --input ./images --interval 2.0

Iterates over ./images at 2-second intervals. Useful for batch processing a folder of scanned documents, receipts, or photos without a live camera feed.

Dashboard only in read mode

python3 -m src --mode dashboard --port 8080

Serves the dashboard against an existing data/observations.db without running the agent. Useful for reviewing historical observations without starting a new capture session.

API Reference

The FastAPI dashboard exposes six endpoints:

The /api/search endpoint runs full-text search over stored observation descriptions, useful for finding all observations that mention a specific object, person, or piece of text across the full history.

The /api/observations endpoint is paginated with limit and offset parameters. The default returns the 50 most recent observations.

Models Used

This is the default --model argument and MODEL_NAME env var. No other models are referenced in code, config, or docs. The model is downloaded from HuggingFace on first run and cached in ./models.

Testing

The test suite covers all five modules - database, vision, agent, dashboard, and CLI - with the VLM fully mocked so no model download is needed to run tests:

make test                  # python3 -m pytest tests/ -v
make lint                  # ruff check src/ tests/ --fix
make typecheck             # mypy src/ --ignore-missing-imports

Test coverage:

tests/test_db.py - 10 tests covering SQLite schema, CRUD, and search
tests/test_vision.py - 6 tests covering mock VLM and prompt rendering
tests/test_agent.py - 9 tests covering motion detection and the agent loop
tests/test_dashboard.py - 6 tests covering HTTP route handlers
tests/test_cli.py - 7 tests covering argparse and env-var loading

Total: 36 tests, all passing. No skipped tests.

Project Structure

smolvlm2-edge-agent/
├── src/
│   ├── __init__.py
│   ├── __main__.py              # entry point for python -m src
│   ├── agent.py                 # MotionDetector + VisionAgent
│   ├── vision.py                # VisionEngine (SmolVLM2 wrapper, with MockVisionEngine)
│   ├── db.py                    # SQLite Database class
│   ├── dashboard.py             # FastAPI app factory + route handlers
│   └── cli.py                   # argparse + env loading
├── tests/                       # 36 pytest tests, VLM fully mocked
├── data/.gitkeep                # observations.db, thumbnails/, test_images/ land here
├── models/.gitkeep              # HF model cache
├── pyproject.toml               # ruff + mypy config + console_script
├── requirements.txt             # pinned runtime deps
├── Makefile                     # install, test, lint, typecheck, run, clean
├── .env.example                 # documented env vars
├── .gitignore
├── BUILD_NOTES.md               # build/verification trace
└── PUBLISH.md                   # exact GitHub push commands

The src/ directory maps cleanly to the agent's responsibilities - agent.py handles the motion detection and VLM orchestration loop, vision.py wraps the model with a mock-compatible interface, db.py handles all SQLite operations, dashboard.py is the FastAPI application, and cli.py handles all argument parsing and environment variable loading.

Contributing

PRs welcome. Before submitting, all three of the following must pass with zero errors:

make lint && make typecheck && make test

How I Built This Using NEO

The process started with an idea - a fully offline edge vision agent that runs on CPU-only hardware with no GPU and no cloud API calls. I put together a clear project description with the requirements, tech stack, and expected output, and handed it to NEO. From there NEO handled the full build autonomously: writing the code, running tests, fixing issues, and iterating until everything was working end to end. Once NEO completed the build, I did a manual review, tested it myself, and fed any improvements back - which NEO then implemented.

How You Can Use and Extend This With NEO

Use it as an offline home security monitor: Point it at a webcam, let it run, and review what it logged through the dashboard. Every scene change is stored with a timestamp, description, confidence score, and thumbnail - all locally, with no data leaving your machine.

Use it for document digitization pipelines: Point --input at a folder of scanned receipts, whiteboards, or handwritten notes. The VLM reads text from images and logs structured observations. The /api/search endpoint lets you query what was found across the full document set.

Use it as an accessibility tool: Run it against a webcam feed to generate continuous natural language descriptions of what is visible in the environment - stored and searchable, entirely offline.

Extend it with additional VLM backends: VisionEngine in vision.py wraps SmolVLM2-2.2B with a clean interface that MockVisionEngine also implements. Swapping in a different HuggingFace multimodal model means updating vision.py - the agent, database, dashboard, and CLI stay entirely unchanged.

Final Notes

SmolVLM2 Edge Vision Agent shows that meaningful vision AI does not require a GPU or a cloud API. A 2.2B-parameter model, motion-gated inference, a local SQLite store, and a FastAPI dashboard, all running offline on commodity hardware.

The code is at https://github.com/dakshjain-1616/SmolVLM2-Edge-Vision-Agent
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Prompt Compression Benchmarker: Cut LLM Input Costs by 35–63% With Measurable Quality Tracking

Nilofer 🚀 — Wed, 06 May 2026 07:07:48 +0000

Most LLM cost comes from input tokens, the long documents, codebases, or conversation histories you send as context. There are several prompt compression algorithms available, but nobody tells you which one actually works best for your specific workload, or how much quality you are trading for the savings.

Prompt Compression Benchmarker (PCB) answers both questions. It benchmarks every major prompt compression algorithm against your actual data, shows you exactly how much quality each one drops, projects the real dollar savings at your call volume, and then gives you a one-line wrapper to deploy the winner as a drop-in replacement around your Anthropic or OpenAI client.

What It Does

PCB answers two questions:

Which compression algorithm preserves the most quality at a given token budget?
Benchmark mode runs all compressors against your data and scores each one with task-specific quality metrics and an optional LLM-as-judge.

How much money does that save at your actual call volume?
Cost projection mode takes your daily token volume and model pricing and gives you monthly and annual savings per compressor.

Then it gives you a one-line wrapper to deploy the answer.

Installation

# From source
git clone https://github.com/dakshjain-1616/Prompt-Compression-Benchmarker
cd Prompt-Compression-Benchmarker
pip install .

# From PyPI (once published)
pip install prompt-compression-benchmarker

Requires Python 3.9+. No GPU required. Core dependencies: tiktoken, scikit-learn, rouge-score, rank-bm25, typer, rich.

# Verify
pcb --help

# Optional extras
pip install "prompt-compression-benchmarker[anthropic]"   # SDK wrapper for Anthropic
pip install "prompt-compression-benchmarker[openai]"      # SDK wrapper for OpenAI
pip install "prompt-compression-benchmarker[mcp]"         # MCP server for Claude Code
pip install "prompt-compression-benchmarker[all]"         # Everything

Quick Start

1. Run the benchmark
The simplest run uses bundled sample data - no setup needed:

# All compressors × all task types, bundled sample data — no setup needed
pcb run

# Target a specific task with cost projection
pcb run --task rag --max-samples 20 --daily-tokens 2000000 --cost-model claude-sonnet-4-6

# Add LLM-as-judge for deeper quality scoring (requires OpenRouter API key)
export OPENROUTER_API_KEY=sk-or-...
pcb run --llm-judge --judge-model claude-sonnet-4-6 --max-samples 10

Here is what a real benchmark run looks like - RAG task, 3M tokens/day, claude-sonnet-4-6 pricing:

pcb run --daily-tokens 3000000 --cost-model claude-sonnet-4-6
                          RAG
 Compressor          Token Reduc %  Proxy Score  Proxy Drop %   ms
 no_compression           0.0%        0.2983         0.0%      0.3
 tfidf ★                 40.1%        0.2519        +16.5%     12.1
 selective_context        56.9%        0.1874        +34.4%      8.3
 llmlingua                53.6%        0.2182        +28.1%      9.7
 llmlingua2               45.0%        0.2204        +27.3%     11.2

 Monthly Cost Projection  claude-sonnet-4-6 · $3/1M · 3M tokens/day
 tfidf             38.3% reduction   $103/mo saved   $1,240/yr
 selective_context 57.5% reduction   $155/mo saved   $1,863/yr
 llmlingua2        43.6% reduction   $118/mo saved   $1,413/yr

The ★ marks the Pareto-optimal compressor - best token savings given a quality drop below 20%.

2. Compress a file directly

# Compress from a file or stdin, output to stdout
pcb compress context.txt --compressor llmlingua2 --rate 0.45 --stats

# Pipe it into any script
cat rag_context.txt | pcb compress | python send_to_claude.py

# Save compressed output
pcb compress context.txt -o compressed.txt --stats

3. Deploy the winner
Once you know which compressor wins on your data, deploying it is one line:

from pcb.middleware import CompressingAnthropic

# Drop-in replacement for anthropic.Anthropic()
client = CompressingAnthropic(compressor="llmlingua2", rate=0.45)

response = client.messages.create(
    model="claude-opus-4-7",
    messages=[{"role": "user", "content": very_long_document}],
    max_tokens=1024,
)

print(client.stats)  # CompressionStats(calls=47, tokens_saved=21,800, reduction=44.8%)

Everything else in your codebase stays the same.

Understanding the Results

Benchmark table columns

Quality drop color coding

cyan   = negative drop (compression improved the metric — noise removal)
green  = < 5% drop    (effectively lossless)
yellow = 5–15% drop   (acceptable for most use cases)
red    = ≥ 15% drop   (significant information loss)

Why use the LLM judge?

The proxy score (F1, ROUGE, BM25) is fast and free but mechanical. The LLM judge calls a real model to evaluate whether the compressed context still supports the correct answer, it reveals things proxy metrics miss.

Here is a real example showing why this matters - RAG task, 5 samples, LLM judge = claude-sonnet-4-6:

Compressor           Proxy Drop %   LLM Score   LLM Drop %
no_compression           0.0%         0.94         0.0%
tfidf                  +23.7%         0.40        -57.4%    ← proxy hid the severity
llmlingua2             +29.9%         0.70        -25.5%    ← much better than proxy suggested
selective_context      +37.6%         0.14        -85.1%    ← dangerous despite high compression

Rule of thumb: use proxy scores to compare many configs quickly, then LLM-judge the top 2–3 before deploying.

Choosing a Compressor

RAG: llmlingua2 at rate 0.40 - preserves named entities and key facts better than sentence-dropping

Summarization: llmlingua at rate 0.45 - sentence-level pruning maintains structural coverage

Code contexts: llmlingua2 at rate 0.35 - keeps imports, identifiers, type names; removes boilerplate

General chat: tfidf at rate 0.40 - safe default, fast, reliable

Target compression rate

--rate is the fraction of tokens to remove. 0.45 means keep 55% of tokens.

Cost Savings - The Real Numbers

Compression saves money on input tokens only. Output tokens are unchanged.
At 3M input tokens per day:

Compression is most valuable on premium models. On DeepSeek or GPT-4.1-mini, the savings are too small to justify the complexity, use it only if you're hitting context window limits.

# Check your own workload
pcb run --max-samples 10 --daily-tokens 5000000 --cost-model claude-opus-4-7

Deploy: Python SDK Wrappers

Anthropic

from pcb.middleware import CompressingAnthropic

client = CompressingAnthropic(
    compressor="llmlingua2",
    rate=0.45,
    verbose=True,
)

response = client.messages.create(
    model="claude-opus-4-7",
    messages=[{"role": "user", "content": very_long_document}],
    max_tokens=1024,
)

# Cumulative stats
print(client.stats)
# CompressionStats(calls=47, tokens_saved=21,800, reduction=44.8%)

# Estimate monthly savings
print(client.stats.monthly_savings_usd(price_per_million=15.0, daily_calls_estimate=2000))
# 588.0

OpenAI (Chat Completions + Codex Responses API)

from pcb.middleware import CompressingOpenAI

client = CompressingOpenAI(compressor="tfidf", rate=0.40)

# Chat Completions API — unchanged
response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": long_context}]
)

# Responses API (Codex / o-series)
response = client.responses.create(
    model="codex-mini-latest",
    input=long_codebase_context,
    reasoning={"effort": "high"}
)

What gets compressed

By default, only "user" role messages over 100 tokens are compressed. System prompts and assistant history are passed through unchanged.

client = CompressingAnthropic(
    compressor="llmlingua2",
    rate=0.45,
    compress_roles=("user", "system"),  # also compress system prompt
)

Claude Code Integration (MCP)

PCB ships an MCP server that adds four compression tools directly into Claude Code conversations.

Setup

# Add to the current project
claude mcp add pcb -s project -- python -m pcb.mcp_server

# Or add to all your projects
claude mcp add pcb -s user -- python -m pcb.mcp_server

Or drop .mcp.json into any project root:

{
  "mcpServers": {
    "pcb": {
      "type": "stdio",
      "command": "python",
      "args": ["-m", "pcb.mcp_server"]
    }
  }
}

Available tools
Once connected, you can ask Claude:

"Compress this RAG context before sending it to the model"
"Estimate how much I'd save compressing my prompts on claude-opus-4-7 at 2000 calls/day"
"What compressor should I use for my coding assistant at 90% quality floor?"

OpenAI Codex (Agents SDK)

from agents import Agent, Runner
from agents.mcp import MCPServerStdio
import asyncio

async def main():
    async with MCPServerStdio(
        name="pcb",
        params={"command": "python", "args": ["-m", "pcb.mcp_server"]},
    ) as pcb_server:
        agent = Agent(
            name="CostAwareAssistant",
            model="codex-mini-latest",
            mcp_servers=[pcb_server],
        )
        result = await Runner.run(
            agent,
            "Compress this codebase context and estimate savings: " + codebase_context
        )
        print(result.final_output)

asyncio.run(main())

Bring Your Own Data

Data is JSONL - one JSON object per line. Check the schema for each task type:

pcb show-schema rag
pcb show-schema summarization
pcb show-schema coding

RAG schema

{
  "id": "my_001",
  "context": "<passage 300–1500 tokens>",
  "question": "<specific question requiring the full context>",
  "answer": "<short, precise answer string>"
}

Summarization schema

{
  "id": "my_001",
  "article": "<article or document 300–800 tokens>",
  "summary": "<2–3 sentence reference summary>"
}

Coding schema

{
  "id": "my_001",
  "context": "<imports, helpers, type definitions — 400–800 tokens>",
  "docstring": "<description of the function to implement>",
  "solution": "<correct Python implementation>"
}

Running on your data

pcb run --data-dir ./my_data --task rag --max-samples 50

# Compare specific compressors
pcb run --data-dir ./my_data --compressor tfidf --compressor llmlingua2

# Export results
pcb run --data-dir ./my_data --output results.json
pcb run --data-dir ./my_data --output results.csv
pcb run --data-dir ./my_data --output results.html

Workflow: Benchmark to Production

Here is the full path from benchmarking to deploying a compressor in production.

Step 1 : Benchmark on your actual data

pcb run --data-dir ./my_data --max-samples 50 --task rag \
        --daily-tokens 2000000 --cost-model claude-opus-4-7 \
        --output benchmark.json

Step 2 : LLM-judge the top candidates

pcb run --data-dir ./my_data --compressor tfidf --compressor llmlingua2 \
        --llm-judge --judge-model claude-sonnet-4-6 --max-samples 30

Step 3 : Deploy the winner

from pcb.middleware import CompressingAnthropic

client = CompressingAnthropic(compressor="llmlingua2", rate=0.40)
# Everything else in your codebase stays the same

Step 4 : Monitor in production

if client.stats.calls % 1000 == 0:
    logger.info(
        "pcb savings: calls=%d saved=%d tokens (%.1f%%) est_monthly=$%.0f",
        client.stats.calls,
        client.stats.tokens_saved,
        client.stats.reduction_pct,
        client.stats.monthly_savings_usd(price_per_million=15.0, daily_calls_estimate=2000),
    )

CLI Reference

PCB run - benchmark

Options:
  -c, --compressor TEXT       Compressor to include (repeat for multiple). Default: all five.
  -t, --task TEXT             Task type: rag, summarization, coding (repeat for multiple).
  -n, --max-samples INT       Max samples per task.
  -r, --rate FLOAT            Target compression rate 0.0–1.0. Default: 0.5
  -d, --data-dir PATH         Directory with *_samples.jsonl files.
  -o, --output PATH           Save report as .json, .csv, or .html.
  -j, --llm-judge             Enable LLM-as-judge scoring via OpenRouter.
  -m, --judge-model TEXT      Model for LLM judge. Default: claude-sonnet-4-6.
      --openrouter-key TEXT   OpenRouter API key (or set OPENROUTER_API_KEY).
      --daily-tokens INT      Daily token volume for cost projection.
      --cost-model TEXT       Model name for cost lookup (e.g. claude-opus-4-7).
      --token-price FLOAT     Manual price override in $/1M tokens.

PCB compress - compress text

Arguments:
  [INPUT_FILE]                File to compress. Reads stdin if omitted.

Options:
  -c, --compressor TEXT       Algorithm. Default: tfidf.
  -r, --rate FLOAT            Fraction to remove. Default: 0.45.
  -o, --output PATH           Write to file instead of stdout.
  -s, --stats                 Print token stats to stderr.

Other commands

pcb list-compressors          # Show all algorithms
pcb list-models               # Show 75+ supported LLM judge models
pcb show-schema rag           # Show JSONL schema for a task type

Output Formats

JSON - full detail per sample

pcb run --output results.json

CSV - one row per compressor × task

pcb run --output results.csv

Columns: compressor, task, avg_token_reduction_pct, avg_quality_score, avg_quality_drop_pct, avg_llm_score, avg_llm_drop_pct, avg_latency_ms, num_samples

HTML - shareable visual report

pcb run --output results.html
# Open in any browser — Chart.js scatter plots, dark theme, Pareto highlights

When NOT to Use Compression

Short prompts (< 200 tokens): PCB skips these automatically overhead exceeds savings.
Cheap models (< $0.50/1M): DeepSeek, Gemini Flash, GPT-4.1-mini - savings too small.
High-precision tasks: Legal review, medical diagnosis - verify your quality floor with --llm-judge first.
Output-bottlenecked workloads: Compression only affects input tokens.

Project Structure

src/pcb/
├── cli.py                      # Typer CLI — all commands
├── config.py                   # Pydantic config and model pricing table
├── runner.py                   # Benchmark orchestration + BenchmarkReport
├── mcp_server.py               # FastMCP server for Claude Code / Codex
├── compressors/
│   ├── tfidf.py                # TF-IDF sentence scoring
│   ├── selective_context.py    # Greedy token-budget selection
│   ├── llmlingua.py            # Sentence-level coarse pruning
│   └── no_compression.py       # Passthrough baseline
├── tasks/
│   ├── rag.py                  # F1/EM/context-recall evaluator
│   ├── summarization.py        # ROUGE-L evaluator
│   └── coding.py               # BM25 + identifier preservation
├── evaluators/
│   └── llm_judge.py            # OpenRouter LLM-as-judge (75+ models)
├── reporters/
│   ├── terminal.py             # Rich terminal tables
│   ├── json_reporter.py        # JSON output
│   ├── csv_reporter.py         # CSV output
│   └── html_reporter.py        # Chart.js HTML report
├── middleware/
│   ├── anthropic_client.py     # CompressingAnthropic drop-in wrapper
│   └── openai_client.py        # CompressingOpenAI drop-in wrapper
└── data/
    ├── rag_samples.jsonl        # 20 real-world factual passages (400–450 tokens)
    ├── summarization_samples.jsonl  # 10 real news-style articles
    └── coding_samples.jsonl     # 10 real Python code contexts (370–800 tokens)

How I Built This Using NEO

I described the problem at a high level: a tool that benchmarks multiple prompt compression algorithms against real workloads, scores quality loss empirically, projects actual dollar savings at a given token volume, and makes it trivially easy to deploy the winning algorithm into an existing Anthropic or OpenAI codebase.

NEO built the entire thing autonomously, the Typer CLI with all commands and flags, all five compressor implementations, the F1/ROUGE-L/BM25 task evaluators, the OpenRouter LLM-as-judge with support for 75+ models, the cost projection engine with the model pricing table, the CompressingAnthropic and CompressingOpenAI drop-in wrappers, the FastMCP server with four tools, the JSON/CSV/HTML reporters, and the three bundled sample datasets - 20 RAG passages, 10 summarization articles, and 10 coding contexts.

How You Can Use and Extend This With NEO

Use it before committing to a compression strategy.
Before wiring any compressor into your production stack, run pcb on a sample of your actual prompts. The benchmark tells you which algorithm preserves the most quality at your target compression rate specific to your data, not a generic recommendation.

Use it to justify the cost of compression infrastructure.
The cost projection output gives you monthly and annual savings at your actual token volume and model pricing. This is the number you need to make a case for adding compression to your pipeline, not a rough estimate but a measured projection against your workload.

Use the MCP tools inside Claude Code sessions.
With the MCP server connected, you can ask Claude to compress a context, estimate savings, or recommend a compressor without leaving your coding environment. This makes compression a natural part of the agent workflow rather than a separate offline step.

Extend it with additional compressors.
The four compressors share a common interface in src/pcb/compressors/. A new algorithm - semantic chunking, abstractive summarization, or a custom retrieval-based approach, slots in as a new file in that directory and appears automatically in pcb run, pcb compress, and the MCP recommend tool without touching any other part of the codebase.

Final Notes

Most teams discover they are over-spending on input tokens only after the bill arrives. pcb gives you the benchmark data to make an informed decision before committing - which algorithm, at what rate, for which task type and the deployment tooling to act on it immediately.

The code is at https://github.com/dakshjain-1616/Prompt-Compression-Benchmarker
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

ContextCraft: A Visual Workbench for Building and Managing LLM Context Windows

Nilofer 🚀 — Tue, 05 May 2026 16:00:35 +0000

Building a good LLM prompt is not a one-shot task. You assemble pieces, a system message, a few examples, some context, the actual instruction and then you iterate. You compress things that are too long, test whether the output still holds up, check how many tokens you are spending, and save versions so you can roll back when something breaks.

Most developers do this in a text editor, a notebook, or scattered across a handful of scripts. There is no single place where you can see the whole context window, manipulate it visually, compress a block, run a live test, and save a snapshot, all without switching tools.

ContextCraft is that place. It is a canvas-based interactive workbench for assembling, compressing, testing, and versioning LLM context windows. It runs locally, connects to Ollama for local compression and testing, supports OpenRouter for cloud LLM testing, and exports directly to OpenAI, Anthropic, LangChain, and JSON formats.

Features

Visual Canvas: Drag and drop interface for organizing prompt blocks with real-time token counting and visual progress bars.

Smart Compression: AI-powered compression using Ollama with semantic preservation. Set a target compression ratio, choose whether to preserve structure, and review a before/after comparison before applying.

Coverage Analysis: Semantic similarity scoring between original and compressed content. Key concept preservation is surfaced as a score so you know exactly what you are trading for token savings.

LLM Testing: Test prompts with streaming responses from Ollama or OpenRouter directly from the canvas. Select provider, model, and temperature and view responses in real time.

Version Control: Save and restore canvas versions with a SQLite backend. Name versions for easy reference and compare two versions to see what changed.

Multi-Format Export: Export to OpenAI, Anthropic, LangChain, and JSON formats. Copy the generated code and paste directly into your application.

Block Library: Pre-built starter blocks for common use cases, available from the sidebar. Add your own blocks to the library for reuse across canvases.

Architecture

ContextCraft is split into a FastAPI backend and a React + Vite frontend.

The backend handles token counting via tiktoken, semantic similarity analysis via sentence-transformers, compression via Ollama, streaming LLM test responses via Ollama or OpenRouter, SQLite-backed version management, and export format generation. The frontend renders the visual canvas with drag-and-drop via @hello-pangea/dnd and code editing via CodeMirror.

contextcraft/
├── server/                 # FastAPI backend
│   ├── main.py            # FastAPI app entry point
│   ├── models.py          # Pydantic data models
│   ├── tokenizer.py       # Token counting (tiktoken)
│   ├── coverage.py        # Semantic similarity analysis
│   ├── compress.py        # Ollama compression service
│   ├── tester.py          # LLM streaming test service
│   ├── export.py          # Export format generators
│   ├── versions.py        # SQLite version management
│   └── pricing.py         # OpenRouter pricing API
├── frontend/              # React + Vite frontend
│   ├── src/
│   │   ├── components/    # React components
│   │   ├── hooks/         # Custom React hooks
│   │   └── App.jsx        # Main application
│   └── package.json
├── cli/                   # CLI entry point
│   └── main.py
└── pyproject.toml         # Python package config

Getting Started

Prerequisites

Python 3.9+
Node.js 18+
Ollama (optional, for local compression and testing)
OpenRouter API key (optional, for cloud LLM testing)
Installation

Installation

# Clone the repository
git clone https://github.com/contextcraft/contextcraft.git
cd contextcraft

# Install Python dependencies
pip install -e ".[dev]"

# Install frontend dependencies
cd frontend
npm install
cd ..

# Initialize the database
contextcraft init-db

Running the Application

# Start the server and frontend
contextcraft serve

# Or start with custom options
contextcraft serve --host 0.0.0.0 --port 8000 --frontend-port 3000

Once running, the frontend is available at localhost:5173 and the API docs at localhost:8000/docs.

Configuration

Create a .env file in the project root:

# OpenRouter API key (for cloud LLM testing)
OPENROUTER_API_KEY=your_api_key_here

# Ollama URL (default: http://localhost:11434)
OLLAMA_URL=http://localhost:11434

# Default compression model
DEFAULT_COMPRESSION_MODEL=gemma2:2b

Supported Models

Token Counting: GPT-4, GPT-4o, GPT-4o Mini, GPT-3.5 Turbo, Claude 3 Opus, Sonnet, Haiku, Claude 3.5 Sonnet

Compression (via Ollama): gemma2:2b (default), any Ollama-compatible model

Testing: Ollama local models, OpenRouter cloud models (requires API key)

Usage Guide

Creating a Canvas: Start with an empty canvas or load from the library. Add blocks using the sidebar buttons or drag from the library. Arrange blocks by dragging to reorder. Edit block content inline or in the full editor.

Compressing Content: Click the compress icon on any block. Set the target compression ratio (0.1 to 0.9). Choose whether to preserve structure. Review the before/after comparison. Apply compression when satisfied.

Testing Prompts: Add your prompt blocks to the canvas. Click the Test button. Select provider (Ollama or OpenRouter). Choose model and set temperature. View streaming responses in real time.

Analyzing Coverage: Compress one or more blocks. Click the Coverage button. View semantic similarity scores. Check key concept preservation.

Managing Versions: Click Versions to save the current state. Name your version for easy reference. Restore previous versions at any time. Compare versions to see changes.

Exporting: Click Export when ready. Choose format, OpenAI, Anthropic, LangChain, or JSON. Copy the generated code. Paste into your application.

API Endpoints

Token Management

POST /api/tokenize - count tokens for text or blocks
GET /api/pricing - get model pricing information

Compression

POST /api/compress - compress text using Ollama
POST /api/coverage - analyze semantic coverage between original and compressed

Testing

POST /api/test - stream LLM responses from Ollama or OpenRouter

Versioning

GET /api/versions - list all versions
POST /api/versions - save a new version
GET /api/versions/{id} - get a specific version
POST /api/versions/{id}/restore - restore a version
POST /api/versions/compare - compare two versions

Export

POST /api/export - export canvas to OpenAI, Anthropic, LangChain, or JSON format

Library

GET /api/library - get starter block library
POST /api/library - add a block to the library

CLI Commands

# Start the application
contextcraft serve

# Initialize database
contextcraft init-db

# Add a block to library
contextcraft add-block --type system --label "My Template" --content "..."

# Get help
contextcraft --help

Docker

# Build image
docker build -t contextcraft .

# Run container
docker run -p 8000:8000 -p 5173:5173 contextcraft

Development

Backend

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black server/ cli/
isort server/ cli/

# Type checking
mypy server/ cli/

Frontend

cd frontend

# Start dev server
npm run dev

# Build for production
npm run build

# Run linter
npm run lint

Contributing

Fork the repository. Create a feature branch:

git checkout -b feature/amazing-feature

Commit your changes:

git commit -m 'Add amazing feature'

Push to the branch:

git push origin feature/amazing-feature

Then open a Pull Request.

How I Built This Using NEO

This tool was designed, built, debugged, and iterated entirely using NEO -An autonomous AI engineering agent that writes, runs, and refines real code end-to-end.

ContextCraft is a full-stack application, a FastAPI backend, a React + Vite frontend, a CLI, and a SQLite-backed versioning layer. Every part of the system was generated and connected through NEO: the backend services for token counting, semantic coverage analysis, compression via Ollama, streaming LLM testing, export pipelines, version management, and pricing integration, along with the interactive frontend canvas for assembling prompt blocks with drag-and-drop, inline editing, and real-time token tracking.

The compression and coverage pipeline, the live testing flow across Ollama and OpenRouter, the version save/restore and comparison system, and the multi-format export layer were all built end-to-end from a high-level problem description. NEO handled the full cycle - generating code, wiring components, resolving issues, and refining the system into a working product.

How You Can Use and Extend This With NEO

Prompt engineering workbench: Instead of iterating on prompts in a text editor and manually counting tokens, assemble your context window visually, compress blocks that are too long, and test the result, all in one place. The version control means you never lose a working configuration while experimenting.

Evaluate compression quality before shipping: Before deploying a compressed prompt to production, run coverage analysis to get a semantic similarity score between the original and compressed version. You know exactly how much meaning you are trading for token savings, not just a token count but an actual semantic measurement.

Manage prompt libraries across projects: The block library lets you save reusable prompt blocks and load them into any canvas. Teams building multiple LLM products can maintain a shared library of tested, versioned prompt components.

Extend it with additional export formats: The export module currently supports OpenAI, Anthropic, LangChain, and JSON. Adding a new format follows the same pattern in export.py and surfaces automatically in the Export UI without touching any other part of the stack.

Final Notes

Context window management is one of those problems that looks simple until you are doing it seriously. ContextCraft brings together the pieces that are usually scattered across different tools visual assembly, token counting, AI compression, semantic coverage analysis, live testing, version control, and export into a single local workbench.

The code is at https://github.com/dakshjain-1616/ContextCraft
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can also use NEO MCP with Claude Code: https://heyneo.com/claude-code

LLM Behavior Diff Model Update Detector

Nilofer 🚀 — Mon, 04 May 2026 11:18:41 +0000

You swap a model. The new one scores better on your benchmarks. You deploy it. Two days later, a user reports that something that used to work reliably now behaves differently.

The benchmark never caught it because benchmarks measure averages. What changed was the behavior on specific prompts, the ones your users actually send.

LLM Behavior Diff is a tool that catches this before it happens. Feed it two model versions and a prompt suite, and it runs every prompt through both, scores the responses for semantic similarity, classifies each divergence by severity, and produces an HTML report you can drop into a CI artifact or diff review.

It ships as a CLI, a Python API, and an MCP server so Claude Code or any MCP-compatible agent can run a behavioral diff before a model swap.

The Problem With Model Updates

Every model update is a tradeoff. The new version might score better on reasoning benchmarks while quietly regressing on instruction-following for your specific use case. Or it might phrase safety refusals differently in a way that breaks downstream parsing. Or two models might produce semantically identical answers that look completely different at the token level, which a naive string comparison would flag as a major change when it isn't one.

LLM Behavior Diff addresses all three scenarios. Embedding-based semantic similarity catches meaning-level changes that token-level comparison misses. The LLM-as-judge layer adds a reasoning layer for ambiguous cases. Severity classification separates noise from real regressions.

How It Works

The pipeline runs in five steps for every prompt in your suite:

Load: A YAML prompt suite is loaded into a PromptSuite Pydantic model. Each prompt has an ID, text, category, tags, and an expected behavior description.

Run: Each prompt is sent through Model A and Model B via LLMRunner. Three providers are supported: Ollama (/api/generate), OpenRouter (chat completions), and a deterministic stub provider for offline CI runs.

Score: Each response pair is scored with either EmbeddingDiffer (cosine similarity on all-MiniLM-L6-v2 embeddings) or SimpleDiffer (Jaccard over words). Optionally, an LLM-as-judge score is combined with the similarity score, default judge model is google/gemini-2.0-flash-lite-001 via OpenRouter.

Classify: Each prompt is classified against the --threshold. Changes are bucketed by severity: combined score >= 0.7 is minor, >= 0.4 is moderate, < 0.4 is major.

Report: An HTML report is rendered and a rich summary table is printed to the terminal.

Why Embeddings Over Token Matching

The difference matters. Here is the same two-model comparison run two ways:
With --use-embeddings (cosine on all-MiniLM-L6-v2):

Avg Similarity: 91.4%
Changes Detected: 0 of 5

With --no-use-embeddings (Jaccard fallback):

Avg Similarity: 25.0%
Changes Detected: 5 of 5

Same two models, same prompts, completely opposite conclusions. The Llama and Gemini answers shared few exact tokens even when semantically identical, which is exactly why the embeddings path is on by default.

Getting Started

pip install -e .

Requires Python 3.11+. Embedding similarity uses sentence-transformers/all-MiniLM-L6-v2, downloaded on first use. The LLM-judge path requires OPENROUTER_API_KEY without it, scoring falls back to embeddings-only.

Running a Diff

Offline - stub provider
A stub provider returns deterministic hashed responses, so the whole pipeline runs offline without Ollama or an API key. Good for CI and testing the setup:

llm-diff run \
  --model-a stub-a --provider-a stub \
  --model-b stub-b --provider-b stub \
  --prompts prompts/default.yaml \
  --output output/report.html \
  --no-use-embeddings

Real output from this run (stub + Jaccard, threshold 0.5):

╭───────────────────────────────────────────────────╮
│ LLM Behavior Diff                                 │
│ Detecting behavioral shifts between model updates │
╰───────────────────────────────────────────────────╯
  Processing: safety-001 ━━━━━━━━━━━━━━━━━━━━ 100%

Comparison Summary
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Metric           ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ Total Prompts    │ 5     │
│ Changes Detected │ 3     │
│ Change Rate      │ 60.0% │
│ Avg Similarity   │ 40.0% │
└──────────────────┴───────┘
Report saved to: output/stub_jaccard.html

Real models - OpenRouter

export OPENROUTER_API_KEY=sk-or-...
llm-diff run \
  --model-a meta-llama/llama-3.2-3b-instruct --provider-a openrouter \
  --model-b google/gemini-2.0-flash-lite-001 --provider-b openrouter \
  --prompts prompts/default.yaml \
  --output output/or_emb.html \
  --use-embeddings --threshold 0.85

Real output (embeddings only):

Comparison Summary
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Total Prompts    │ 5     │
┃ Changes Detected │ 0     │
┃ Change Rate      │ 0.0%  │
┃ Avg Similarity   │ 91.4% │
└──────────────────┴───────┘

Adding --use-judge brings the average similarity to 91.8% and surfaces reasoning like: "Both responses correctly answer 'yes' and provide essentially the same explanation... Response A is slightly more verbose, but the core meaning is identical."

Real models - Ollama

llm-diff run \
  --model-a qwen3:8b --provider-a ollama \
  --model-b gemma4:e4b --provider-b ollama \
  --prompts prompts/default.yaml \
  --output output/report.html \
  --use-embeddings --threshold 0.85

CLI Reference

llm-diff --help

Usage: llm-diff [OPTIONS] COMMAND [ARGS]...

 LLM Behavior Diff — Model Update Detector

 --version                Show version information
 --help                   Show this message and exit.

 Commands
   run  Run a comparison between two models.

llm-diff --version
LLM Behavior Diff version 0.1.0

Key options for llm-diff run:

Severity buckets applied when a change is detected: combined >= 0.7 is minor, >= 0.4 is moderate, < 0.4 is major.

Prompt Suite Format

The prompt suite is a YAML file. prompts/default.yaml ships with 5 prompts spanning reasoning, coding, factual, instruction-following, and safety. You can write your own:

name: "My suite"
version: "1.0.0"
prompts:
  - id: "code-001"
    text: "Write a Python function reverse_string(s)..."
    category: "coding"
    tags: ["python"]
    expected_behavior: "Short correct function"

IDs must be unique. Category must be one of: reasoning, coding, creativity, safety, instruction_following, factual, conversational.

Python API

The full pipeline is available as a library. A synchronous one-shot call:

from llm_behavior_diff.runner import run_prompt_sync
from llm_behavior_diff.models import ModelConfig, ProviderType

resp = run_prompt_sync(
    ModelConfig(name="stub-m", provider=ProviderType.STUB),
    prompt_id="p1",
    prompt_text="hello world",
)
print(resp.text, resp.success)
# -> Model stub-m says: 921fac0c4c True

Similarity scoring directly:

from llm_behavior_diff.differ import SimpleDiffer, EmbeddingDiffer, create_differ

d = SimpleDiffer()
print(d.compute_similarity("the cat sat", "the cat ran"))   # 0.5

e = EmbeddingDiffer()
print(e.compute_similarity("The answer is 4.", "Two plus two equals four."))
# -> ~0.59

create_differ(use_embeddings=False) returns a SimpleDiffer (Jaccard). True returns an EmbeddingDiffer if sentence-transformers is importable, otherwise falls back to SimpleDiffer.

Generating a report from a ComparisonRun:

from llm_behavior_diff.report import ReportGenerator
from llm_behavior_diff.models import Settings

ReportGenerator().save_report(run, Settings(), "out.html")

ReportGenerator looks for a Jinja template in the CWD, the package directory, and a legacy path, then falls back to a built-in template so reports always render.

MCP Server

The tool also runs as an MCP server over stdio transport, exposing three tools so Claude Code or any MCP-compatible agent can trigger a behavioral diff during a session:

llm-diff-mcp
# or: python -m llm_behavior_diff.mcp_server

The three exposed tools:

compare_models - runs a full prompt suite through two models and returns per-prompt similarity, severity, and response text.
analyze_drift - scores drift between two candidate responses for a single prompt.
generate_report - renders an HTML summary from a JSON list of results.

Claude Code config:

{
  "mcpServers": {
    "llm-behavior-diff": {
      "command": "llm-diff-mcp"
    }
  }
}

Smoke test - all three tools, offline, via Python:

import asyncio, json
from llm_behavior_diff.mcp_server import (
    compare_models, CompareModelsRequest,
    analyze_drift, AnalyzeDriftRequest,
    generate_report, GenerateReportRequest,
)

async def main():
    a = await compare_models(CompareModelsRequest(
        model_a="stub-a", provider_a="stub",
        model_b="stub-b", provider_b="stub",
        prompts_path="prompts/default.yaml",
        threshold=0.5, use_embeddings=False,
    ))
    print(a.total_prompts, a.changes_detected, a.avg_similarity)
    # real: 5 4 0.3446

    b = await analyze_drift(AnalyzeDriftRequest(
        prompt_text="math",
        response_a="The answer is 4.",
        response_b="2+2 equals 4.",
        use_embeddings=True,
    ))
    print(b.embedding_similarity, b.severity)
    # real: 0.5572 moderate

    c = await generate_report(GenerateReportRequest(
        results_json=json.dumps([
            {"prompt_id":"p1","similarity_score":0.9,"behavioral_change":False,"severity":"none"},
            {"prompt_id":"p2","similarity_score":0.3,"behavioral_change":True,"severity":"major"},
        ]),
        output_path="output/mcp_report.html",
        title="MCP Smoke",
    ))
    print(c.success, c.output_path)

asyncio.run(main())

Verified over stdio JSON-RPC:

llm-diff-mcp   # speaks MCP 2024-11-05 on stdio
# tools/list -> compare_models, analyze_drift, generate_report
# tools/call analyze_drift {"prompt_text":"...","response_a":"Paris",
#   "response_b":"The capital is Paris.","use_embeddings":true}
# -> {"embedding_similarity":0.7761,"severity":"minor", ...}

Limitations

LLM-as-judge requires OpenRouter - Without OPENROUTER_API_KEY, judging is skipped and the combined score equals the embedding or Jaccard similarity alone.
First embedding run is slow - all-MiniLM-L6-v2 is downloaded from Hugging Face on first use. Subsequent runs use the local cache.
Ollama is not spawned automatically - The client talks to http://localhost:11434 by default (OLLAMA_HOST env var overrides). Ollama must already be running.
Stub provider is for CI and demos only - It produces deterministic fake text keyed on model name, temperature, and prompt. Not suitable for real behavioral conclusions.

How You Can Use This

Gate model upgrades in CI before they ship: Add llm-diff run to your deployment pipeline. Before any model swap reaches production, the tool runs your prompt suite through both versions and fails the pipeline if behavioral drift exceeds your threshold. You catch regressions automatically, not from user reports two days later.

Use it during prompt engineering to measure real impact: When you change a system prompt or few-shot examples, run a diff between the old and new configuration. The severity classification tells you whether the change is minor, moderate, or major across your prompt categories, so you know what you are actually shipping.

Use the MCP server to make your agent self-aware of drift: With the MCP server running, Claude Code or any MCP-compatible agent can call compare_models or analyze_drift directly during a session. An agent working on a model integration can check for behavioral drift without leaving the coding environment.

Extend it with additional providers: The tool currently supports Ollama, OpenRouter, and a stub provider, all sharing a common LLMRunner interface. Adding a new provider for Anthropic, Gemini, or any OpenAI-compatible endpoint follows the same pattern without touching the differ, classifier, or report logic.

Final Notes

Behavioral drift is the category of model regression that benchmarks miss. LLM Behavior Diff catches it by running the same prompts through both model versions, scoring the responses semantically rather than lexically, and classifying the divergence by severity before a swap reaches production.

The code is at https://github.com/dakshjain-1616/-LLM-Behavior-Diff-Model-Update-Detector

You can also build with NEO in your IDE using the VS Code extension or Cursor.

NEO is a fully autonomous AI engineering agent that writes code and builds solutions for AI/ML tasks including model evals, prompt optimisation, and end-to-end pipeline development.

AI Slop Cleaner: Automating Your Codebase Hygiene

Nilofer 🚀 — Thu, 30 Apr 2026 09:56:31 +0000

Every codebase accumulates clutter over time. An import left behind after a refactor. A helper function that nothing calls anymore. A method that grew too complex to reason about. None of it breaks anything immediately, but it slows down every developer who reads through it, and it silently raises the cost of every future change.

The usual fix is a manual review pass. Someone spends an hour looking for unused imports, searching for dead functions, flagging complexity hotspots. It is tedious, inconsistent, and happens far less often than it should.

Slop Cleaner is a CLI tool that does this automatically. It detects unused imports, dead functions and classes, and over-complex code using tree-sitter AST analysis (not regex) so it never removes an import that appears in a docstring or string annotation. Every patch is atomic: backed up before writing, and rolled back automatically if your test suite fails.

What slop-cleaner Detects and Fixes

slop-cleaner targets three specific categories of clutter that accumulate in every codebase:

Unused imports: Imports left behind after refactors. These are detected at HIGH confidence and removed automatically. The tool handles single-line imports, selective removal from from X import A, B blocks where only one name is unused, and multi-line from X import (...) blocks where individual lines are surgically removed.

Dead functions and classes: Defined but never called anywhere in the codebase. These are detected at MEDIUM confidence and flagged in the report for human review. The tool builds a full call graph across all symbols before making this determination.

Over-complex functions: Functions with cyclomatic complexity above the threshold (default 10). These are flagged at MEDIUM confidence for manual refactoring. The tool never auto-removes a function, complexity is a signal that something needs attention, not a safe automatic fix.

HIGH confidence issues are fixed automatically. MEDIUM confidence issues are always left to human judgment.

The 5-Phase Pipeline

Everything slop-cleaner does runs as a five-phase pipeline. Each phase feeds into the next, and every patch is atomic, backed up before writing and rolled back automatically if your tests fail.

Audit:Parses every .py and .ts/.tsx file with tree-sitter. Collects unused imports at HIGH confidence and high-complexity functions at MEDIUM confidence.

Analyze: Builds a call graph across all symbols in the project. Identifies dead code, defined but never called.

Clean: Applies HIGH-confidence patches atomically. Handles single-line and multi-line from X import (...) blocks. Backs up each file before writing.

Verify: Runs your test suite with pytest. On any failure, rolls back every patched file to its backup automatically. Returns exit code 1 so CI catches it.

Document: Generates three Markdown reports: ARCHITECTURE.md, FUNCTION_MAP.md, and SLOP_REPORT.md covered in detail below.

What Gets Fixed Automatically vs. Flagged

The distinction matters. HIGH confidence patches are cases where the tool is certain removal is safe. MEDIUM confidence cases, dead code and complexity could have dynamic dispatch, reflection, or other patterns that make automatic removal risky. The tool flags these and leaves the decision to you.

Edge Cases Handled

One of the hardest parts of detecting unused imports is knowing when a name that looks unused is actually needed. slop-cleaner handles these correctly:

These are exactly the cases where regex-based tools get it wrong. Because slop-cleaner parses an actual AST, it understands the difference between a name appearing inside a string and a name being used as an identifier.

Installation

git clone https://github.com/your-org/slop-cleaner
cd slop-cleaner

# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate      # macOS / Linux
# .venv\Scripts\activate       # Windows

# Install the tool and its dependencies
pip install -e .

This registers two CLI commands: slop-audit and slop-clean.
Dependencies installed automatically via pyproject.toml:

tree-sitter>=0.25
tree-sitter-python>=0.23
tree-sitter-typescript>=0.23
rich>=13

Note for modern Linux (Ubuntu 23.04+, Debian 12+): system Python blocks global pip install. Always use a virtual environment as shown above, or use pipx install . to install the CLI tools globally without a venv.

To run the test suite:

pip install -e ".[test]"   # adds pytest + pytest-cov
pytest                     # runs tests/test_parsers.py (22 tests)

Quick Start

# Audit a project — report issues, exit 1 if any found (CI-friendly)
slop-audit path/to/project/

# Audit a single file
slop-audit src/services/user_service.py

# Full clean — audit, fix, verify tests, generate docs
slop-clean path/to/project/

# Dry run — show what would change without touching files
slop-clean path/to/project/ --dry-run

# Write audit JSON for tooling integration
slop-audit path/to/project/ --output report.json

Commands

slop-audit
slop-audit scans a file or directory and prints a table of all issues found without touching anything. It exits with code 1 if issues are found, making it a clean CI gate.

slop-audit <target> [--output JSON] [--threshold N] [--verbose]

Exit codes: 0 = clean, 1 = issues found.

slop-clean
slop-clean runs the full 5-phase pipeline. Always run --dry-run first on an unfamiliar project, it shows exactly what patches would be applied without touching any file.

slop-clean <target> [--output DIR] [--threshold N] [--dry-run] [--verbose]

Exit codes: 0 = success, 1 = rollback was triggered.

Generated Output

After slop-clean runs, the --output directory contains three reports:

slop-output/
├── ARCHITECTURE.md   — file tree + Mermaid dependency graph
├── FUNCTION_MAP.md   — every symbol with start line and complexity score
└── SLOP_REPORT.md    — issue summary, patches applied, dead-code candidates

ARCHITECTURE.md gives a structural overview of the codebase with a visual Mermaid call graph, useful for understanding how the project fits together at a glance.

FUNCTION_MAP.md is a complete index of every function and class with its start line and complexity score. For large codebases, this is the fastest way to see where complexity is concentrated.

SLOP_REPORT.md is the actionable output: what was fixed automatically, what was flagged for manual review, and which dead-code candidates need a human decision.

Trying It on the Example Projects

Two sample projects ship in examples/ so you can try the tool immediately after installing:

# Audit only — no test verification needed
slop-audit examples/todo_app/ --verbose
slop-audit examples/event_pipeline/ --verbose

# Full clean — dry run first, then apply
slop-clean examples/todo_app/ --dry-run
slop-clean examples/todo_app/

To run the example test suites directly, change into the project directory first so their imports resolve correctly:

cd examples/todo_app && pytest tests/ -v
cd examples/event_pipeline && pytest tests/ -v

todo_app is a sample Python app with intentional slop, a good first run to see exactly what the tool catches. event_pipeline covers tricky import patterns like aliases, multi-line imports, and string annotations, designed to show the AST analysis handling edge cases correctly.

CI Integration

slop-audit drops directly into CI as a quality gate. It exits 1 if any issues are found, failing the workflow step automatically:

# .github/workflows/quality.yml
jobs:
  slop-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .
      - run: slop-audit src/ --threshold 12

This catches unused imports and complexity regressions on every pull request before they get merged into the main branch.

Project Structure

slop-cleaner/
├── cli/
│   └── main.py               — entry points + rich-formatted output
├── engines/
│   ├── auditor.py            — Phase 1 · AST-based issue detection
│   ├── analyzer.py           — Phase 2 · call-graph + dead-code finder
│   ├── cleaner.py            — Phase 3 · atomic patch application
│   ├── verifier.py           — Phase 4 · pytest runner + rollback
│   └── documenter.py         — Phase 5 · Markdown report generator
├── parsers/
│   ├── python_parser.py      — tree-sitter Python wrapper
│   └── typescript_parser.py  — tree-sitter TypeScript/TSX wrapper
├── examples/
│   ├── todo_app/             — sample Python app with intentional slop
│   └── event_pipeline/       — sample project with tricky import patterns
├── assets/
│   ├── pipeline.svg          — 5-phase flow diagram
│   ├── features.svg          — feature overview infographic
│   └── before-after.svg      — code transformation visual
└── tests/
    └── test_parsers.py       — 22 tests covering the parsers

The structure maps cleanly to the five phases, one engine per phase, two parsers for Python and TypeScript/TSX, and a single CLI entry point that ties them together.

How I Built This Using NEO

This tool was designed, built, debugged, and refined entirely using NEO, a fully autonomous AI engineering agent that writes, runs, and iterates on real code without hand-holding.

Every engine, every edge-case fix, and every SVG was produced by NEO in a single session. The five-phase pipeline, the tree-sitter AST parsers for Python and TypeScript, the atomic patch application with backup and rollback, the call graph builder, the pytest runner with automatic rollback on failure, and the three Markdown report generators all of it built end-to-end from a high-level problem description.

How You Can Use and Extend This With NEO

Use it as a CI quality gate on every pull request:
Drop slop-audit into your CI pipeline. Every PR gets checked for unused imports and complexity regressions before it touches main. The exit code 1 on issues means it fails the build automatically, no configuration beyond one workflow step. It is already wired for this out of the box.

Use it before a major refactor:
Before starting a large refactor, run slop-clean on the codebase. It removes accumulated import clutter, flags dead code that no longer needs to be worked around, and generates a complexity map. You go into the refactor with a cleaner starting point and a clear picture of where complexity lives.

Use it to onboard onto an unfamiliar codebase:
Run slop-audit on a project you have just inherited and read SLOP_REPORT.md. You get a structured list of unused imports, dead functions, and complexity hotspots, a map of technical debt you can act on rather than discovering piece by piece while working in the code.

Use it to generate a documentation baseline for a legacy project:
Run slop-clean and read ARCHITECTURE.md and FUNCTION_MAP.md. You get a file tree, a visual call graph, and a complete index of every symbol with its complexity score. For a project with no existing documentation, that is a meaningful starting point.

The tool is also designed to be extended and NEO can take any of these further without starting from scratch:

JavaScript/JSX parser- Two parsers already exist (python_parser.py, typescript_parser.py) following the same tree-sitter wrapper pattern. A third for JavaScript/JSX follows the same interface and plugs into all five engines immediately.

Additional complexity metrics- auditor.py already tracks cyclomatic complexity. Additional metrics like function length follow the same detection pattern and surface automatically in FUNCTION_MAP.md and the audit output once added.

Auto-fix for dead code- dead code is currently flagged at MEDIUM confidence and left to human judgment. For clearly dead private functions confirmed unreachable by the call graph, automatic removal is a natural next step built directly on the existing analyzer.py and cleaner.py infrastructure.

Pre-commit hook- slop-audit already exits 1 on issues. A small wrapper that hooks into the existing CLI entry point brings slop detection into the local development loop before anything reaches CI.

Final Notes

Code clutter is not a cosmetic problem. Unused imports add noise to every code review. Dead functions add surface area that has to be mentally accounted for. High-complexity functions resist change and hide bugs. slop-cleaner addresses all three automatically, with AST-level precision that regex-based tools cannot match, and with a test-verification step that means it never leaves your codebase in a worse state than it found it.
The code is at https://github.com/dakshjain-1616/Ai_Slop_Cleaner
You can also build with NEO in your IDE using the VS Code extension or Cursor.

Agent Failure Classifier: Post-Hoc Root Cause Analysis for Failed LLM Agent Runs

Nilofer 🚀 — Wed, 29 Apr 2026 07:56:24 +0000

When an LLM agent fails, the trace is right there, the user turns, the tool calls, the responses, the final result. But knowing what happened and knowing why it failed are two different things. Most teams read traces manually, form a guess, and move on.

Agent Failure Classifier is a CLI tool and Python library for post-hoc root cause analysis of failed or low-quality LLM agent runs. Feed it any agent trace and it classifies the failure into one of eight named failure modes, identifies the first turn where things went wrong, and produces a structured report with actionable fixes.

The classifier combines eight fast rule-based detectors with an optional LLM-as-judge pass via OpenRouter. The rule-based layer is free, deterministic, and requires no network access. The LLM pass breaks ties and classifies traces the rules cannot resolve alone.

The Eight Failure Modes

The classifier recognises exactly eight failure modes, each with a precise definition:

HALLUCINATION: Agent stated facts or called tools that do not exist
TOOL_MISUSE: Agent called a real tool with wrong parameters or at the wrong time
CONTEXT_LOSS: Agent forgot earlier decisions or repeated already-completed steps
CIRCULAR_REASONING: Agent looped between the same 2-3 steps without making progress
GOAL_DRIFT: Agent started pursuing a sub-goal and forgot the original task
OVER_REFUSAL: Agent refused an action it was capable of and should have taken
SCHEMA_ERROR: Agent generated malformed JSON for a tool call or structured output
TIMEOUT_CASCADE: One slow tool call caused the agent to rush or skip subsequent steps

These are not fuzzy categories. Each one maps to a specific detector with specific signals. A hallucination is flagged when the agent asserts a factual claim without invoking any retrieval tool. A timeout cascade is flagged when a tool call exceeds a latency threshold and the subsequent agent turn is unusually short relative to the tool output.

How It Works

The classification pipeline runs in two layers.

The rule-based layer runs eight deterministic detectors over the trace. Each detector looks for specific structural signals repeated tool calls with identical inputs, cycles in agent turn content, latency spikes followed by short responses, malformed JSON in tool call outputs. This layer runs offline, requires no API key, and classifies all eight failure modes.

The LLM-as-judge layer is optional. When enabled, it receives traces the rule-based layer couldn't resolve with high confidence and breaks ties. The judge runs via OpenRouter and can be pointed at any OpenRouter model or a local OpenAI-compatible server (Ollama, vLLM, llama.cpp).
Every classification produces a structured report with the classified failure mode, a confidence score, the first turn where the failure was detected, a root cause summary, and a list of actionable fixes.

Getting Started

Install

git clone https://github.com/dakshjain-1616/agent-failure-classifier
cd agent-failure-classifier
python3 -m venv venv
source venv/bin/activate
pip install -e .

Requires Python 3.8+. The only dependencies are pydantic, rich, click, and requests. The rule-based layer runs with no additional setup.

LLM Judge Setup (Optional)
To enable the LLM-judge pass, copy .env.example to .env and set your OpenRouter key:

cp .env.example .env
# edit .env and set OPENROUTER_API_KEY=sk-or-...

Without any key, pass --no-llm to every classify or batch call. The rule-based layer alone classifies all eight failure modes.

CLI

The CLI is exposed as both a console script (agent-failure-classifier) and an importable module (python -m agent_failure_classifier.cli).

Classify a single trace
The core command takes a trace JSON file and returns a structured report. --no-llm keeps it offline, rule-based only, no API call.

agent-failure-classifier classify --trace traces/hallucination_example.json --no-llm

Key flags:

Validate a trace
Before classifying, validate parses the trace and prints its structure: trace ID, goal, turn count, and a preview of each turn. Useful for confirming the trace loaded correctly before running classification.

agent-failure-classifier validate --trace traces/hallucination_example.json

Batch classification
batch runs classification over every *.json file in a directory and produces a failure-mode distribution table plus a per-trace summary.

agent-failure-classifier batch --traces-dir ./traces/ --no-llm

Worked Examples

Example 1 - Hallucination
The trace has a user asking for WWII death statistics. The agent responds directly with a factual claim, no tool call, no retrieval.

{
  "trace_id": "hallucination-001",
  "original_goal": "Get population statistics",
  "final_result": "70 million people died in WWII.",
  "is_successful": false,
  "turns": [
    {"turn_number": 0, "role": "user",  "content": "How many people died in WWII?"},
    {"turn_number": 1, "role": "agent", "content": "70 million people died in WWII."}
  ]
}

Classification: HALLUCINATION, confidence 75%, first failure at turn 1. The detector flags that the agent asserted a factual claim without invoking any retrieval tool. Recommended fixes include adding a fact-checking step, requiring tool verification for factual claims, and implementing retrieval-augmented generation.

Example 2 - Circular Reasoning
Four turns alternating between "Let me analyze this step by step." and "I need more information." The agent makes no progress across the entire trace.
Classification: CIRCULAR_REASONING, confidence 80%. The rule-based detector identifies a 2-step cycle repeating across agent turns and recommends a maximum-iteration limit plus state-change detection.

Example 3 - Timeout Cascade
A slow_api tool call with latency_ms: 6000 followed by a one-word agent response "OK".
Classification: TIMEOUT_CASCADE, confidence 70%. The detector flags the latency breach and notes that the subsequent agent turn is a one-word response, less than half the length of the tool output, indicating the agent rushed through the remaining steps.

Python API

Classify a trace programmatically

import json
from agent_failure_classifier.classifier import FailureClassifier
from agent_failure_classifier.models import AgentTrace

trace = AgentTrace(**json.load(open("traces/hallucination_example.json")))
report = FailureClassifier(use_llm=False).classify(trace)

print(report.classified_failure_mode, report.confidence)
print(report.root_cause_summary)

Record a trace live with TraceRecorder
Rather than constructing trace JSON by hand, TraceRecorder is a context manager that captures an agent run as it executes and writes a trace file to disk on exit. The output is immediately compatible with the CLI and with FailureClassifier.

from agent_failure_classifier.recorder import TraceRecorder

with TraceRecorder(goal="Find Italian restaurants", output_dir="./traces") as r:
    r.add_turn(role="user", content="Find Italian restaurants near me")
    r.add_turn(role="agent", content="Searching...")
    r.add_turn(
        role="tool",
        content="results",
        tool_name="search",
        tool_input={"query": "italian restaurants"},
        tool_output='{"hits": ["Luigi Bistro", "Pasta Palace"]}',
        latency_ms=120,
    )
    r.add_turn(role="agent", content="I found Luigi Bistro and Pasta Palace.")
    r.set_final_result("Found Luigi Bistro and Pasta Palace", is_successful=True)

On exit the trace is saved to ./traces/trace_<id>_<timestamp>.json.

Parse traces from other frameworks
AutoParser auto-detects and normalises three input formats into the canonical AgentTrace model. No manual conversion needed regardless of where the trace came from.

from agent_failure_classifier.formats import AutoParser

trace = AutoParser().parse_file("path/to/trace.json")

The three supported formats are:

Native / generic: a dict with trace_id, original_goal, is_successful, and a turns list. This is the format emitted by TraceRecorder.
LangSmith run export: a dict with run_type, inputs, outputs, and optional child_runs. Tool child runs become TOOL turns; chain and LLM child runs become AGENT turns.
LangGraph state dict: a dict with thread_id and a state.messages list whose entries use type values human, ai, and tool. A minimal list-of-dicts ([{"role": "...", "content": "..."}, ...]) is also accepted by the generic parser.

How I Built This Using NEO

This project was built using NEO, a fully autonomous AI engineering agent that writes code and builds solutions for AI/ML tasks including model evals, prompt optimisation, and end-to-end pipeline development.

The problem was defined at a high level: a tool that takes any agent trace, runs deterministic detectors over it, and classifies the failure into a named category with a structured report and actionable fixes. NEO generated the full implementation: the eight rule-based detectors, the FailureClassifier orchestration layer, the optional LLM-as-judge pass via OpenRouter, the TraceRecorder context manager, the AutoParser with support for native, LangSmith, and LangGraph formats, and the Click-based CLI with classify, validate, and batch commands.

How You Can Build Further With NEO

Use it as a CI/CD quality gate for your agent.
If you're shipping an LLM agent, you can integrate the classifier directly into your deployment pipeline. Record traces from your test suite with TraceRecorder, run batch classification on every pull request, and fail the build if a new failure mode appears or if the rate of a known one spikes. You get a systematic regression check on agent behaviour, not just on code.

Use it to understand where your agent breaks most.
Run batch classification across a directory of historical traces and look at the failure mode distribution. If CONTEXT_LOSS shows up in 40% of your traces, that's a signal about your agent's memory design, not a one-off bug. This turns debugging from reactive to diagnostic, you're looking at patterns across runs, not reading individual traces one by one.

Use it as a live monitoring layer in a multi-agent system.
The classifier runs as an A2A agent, which means it can sit as a node in a multi-agent pipeline. Any agent in the system can send its trace to the classifier after each run and get a structured failure report back. An orchestrator can use that signal to decide whether to retry, reroute, or escalate without any human in the loop.

Use it during agent development to catch regressions early.
Wrap TraceRecorder around your agent during development. Every run produces a trace. Feed those traces into the classifier after each session and you'll know immediately if a change introduced a new failure mode. It's the difference between finding out something broke in production versus finding out in your local environment.

Final Notes

Agent Failure Classifier turns trace debugging from a manual read-and-guess process into a systematic one. Eight named failure modes, a deterministic rule-based layer that runs offline, an optional LLM judge for ambiguous cases, and support for traces from native formats, LangSmith, and LangGraph, all producing a structured report with the first failure turn and actionable fixes.

The code is at https://github.com/dakshjain-1616/agent-failure-classifier
You can also build with NEO in your IDE using the VS Code extension or Cursor.

Synthetic Data Flywheel: A Closed-Loop Pipeline for Instruction-Tuning Data

Nilofer 🚀 — Tue, 28 Apr 2026 10:44:54 +0000

Fine-tuning a model requires data. Good data requires human labeling. Human labeling doesn't scale. And most synthetic generation pipelines stop at generation, they produce candidate pairs but have no mechanism to filter them, measure quality, or feed failure cases back into the next round.
Synthetic Data Flywheel is a closed-loop pipeline that handles the full cycle: generate candidate instruction-output pairs, validate them deterministically, score them with an LLM-as-judge, calibrate that judge against human labels, export clean training data, and feed the failure cases from one cycle as seeds into the next. It ships as a CLI, a Python library, and an A2A-protocol agent surface for multi-agent orchestration.

Everything except the optional fine-tuning step runs on CPU.

The Problem

Synthetic data generation without a quality gate produces noise at scale. And quality gates without calibration produce a judge whose scores you can't trust. The flywheel addresses both: every candidate pair is scored, every score can be validated against human labels, and every failure becomes signal for the next generation cycle rather than a dead end.

How It Works

A dataset moves through a series of additive stages, each producing artifacts keyed by the dataset name. Every stage is idempotent and re-runnable.

Generation: Candidate pairs are produced from seed prompts via OpenRouter, using one of four prompt templates: QA, INSTRUCTION, REASONING, or CREATIVE.

Validation: Deterministic checks run over each pair: schema, length, dedup, PII, language, profanity. Results are written as a JSON report with severity levels (error, warning, never). A cleaned copy of the dataset can be written at this stage.

Judging: An LLM-as-judge scores each pair against a rubric. The judge supports three backends: Ollama, OpenRouter, and Anthropic. Judgments are cached on disk keyed by (backend, model, pair.id, rubric.name@version), repeated judge passes on unchanged pairs are free.

Labeling: Three modes: Interactive (human reviews pairs one by one), bulk (apply a status to a filtered subset), and auto-from-judge (derive labels from judgment scores above a threshold). Labels are stored append-only so sessions can be interrupted and resumed.

Calibration: Treats human labels (status == approved) as ground truth and measures the judge's precision, recall, F1, and accuracy.

Compare: Two or more judgment runs on the same dataset are compared: pass-agreement, Cohen's kappa, and Pearson correlation on the overall score.

Export: Pairs that clear the judge filter are written to a train/val split. The filter expression uses a safe evaluator, only arithmetic, comparisons, and subscript access into the context dict. Attribute access and function calls are rejected.

Cycle feedback: Failure instructions from one cycle are extracted and fed as additional seeds into cycle N+1. The autonomous loop stops when the pass rate drops below min_pass_rate (default 0.5) or max_cycles is reached.

Getting Started

Install

git clone https://github.com/dakshjain-1616/synthetic-data-flywheel
cd synthetic-data-flywheel
pip install -e .

Requires Python 3.11+. Generation requires OPENROUTER_API_KEY. The local judge path requires Ollama, verified against gemma4:latest. Fine-tuning requires Unsloth and a GPU; the repo was verified on a free Colab T4.

Initialize
init creates the directory structure the rest of the pipeline writes into.

flywheel init

Synthetic Data Flywheel Initialized
Data Directory: ./data
Checkpoint Directory: ./data/checkpoints
Report Directory: ./reports
Directories created successfully

Ingest
ingest normalises an existing dataset into the flywheel's internal JSONL format. It supports jsonl, csv, and HuggingFace datasets, and accepts a field mapping flag when the source uses different column names.

flywheel ingest -i demo.jsonl -n demo --tag demo1

Ingested 8 pairs -> data/user/demo.jsonl

Other ingest forms:

flywheel ingest -i data.csv              -n my_dataset -f csv
flywheel ingest -i hf://tatsu-lab/alpaca -n alpaca --limit 500 --hf-split train
flywheel ingest -i data.jsonl -n aliased --map "instruction=prompt,output=completion"
flywheel ingest -i data.jsonl -n x --dry-run

Each successful ingest writes data/user/<name>.jsonl and data/user/<name>.meta.json.

Validate
Before any judging happens, the validator runs deterministic checks over the dataset. This catches structural problems, duplicate pairs, PII, malformed schema, before spending LLM calls on them.

flywheel validate -d demo --checks schema,length,dedup,pii --write-clean data/user/demo.clean.jsonl

Validation: demo
  Total pairs       8
  pii               1
  severity:warning  1
Report: data/validation/demo.report.json
Clean dataset written (8 pairs): data/user/demo.clean.jsonl

The --fail-on error|warning|never flag lets you gate CI on validation issues.

Judge
With a clean dataset, the judge scores each pair against a rubric. The default rubric is built-in; custom rubrics can be passed with --rubric. Results are cached, so re-running after adding new pairs only scores the new ones.

flywheel judge -d demo --backend ollama --model gemma4:latest --tag v1 --max-pairs 3

Judging 3 pairs with ollama:gemma4:latest
  Judged                3
  Passed                0 (0.0%)
  Avg overall (scored)  5.00
  Output                data/judgments/demo.v1.jsonl
  Cache                 hits=0 misses=3 writes=3

Judgments land at data/judgments/<dataset>.<tag>.jsonl. The --tag flag is how multiple judgment runs on the same dataset are tracked separately.

Label
Labeling bridges human judgment and automated scoring. auto-from-judge derives labels directly from the judgment scores, pairs above the threshold are approved, pairs below are rejected.

flywheel label -d demo --mode auto-from-judge --judgments data/judgments/demo.v1.jsonl --reject-below 3.5

For manual review, --mode interactive walks through pairs one by one. For bulk operations, --mode bulk applies a status to a filtered subset. All labels are stored append-only at data/labels/<dataset>.jsonl.

Compare
When you have two judgment runs, say from two different models, compare measures how much they agree. Cohen's kappa close to 1.0 means the two judges are making the same pass/fail decisions.

flywheel compare -d demo --tags judge_a,judge_b

Judge comparison: judge_a vs judge_b
  Common pairs          8
  judge_a passed / mean 6 / 7.44
  judge_b passed / mean 6 / 7.19
  Pass agreement        100.0%
  Cohen's kappa (p/f)   1.000  (near-perfect)
  Score Pearson r       0.965
  Output                reports/demo/compare.json

Calibrate
Calibration answers the question you need to answer before trusting your judge: does its passed decision align with human labels? Precision of 1.0 means every pair the judge passed, a human also approved. Recall of 0.75 means the judge missed 25% of the pairs humans would have kept.

flywheel calibrate -d demo --tag judge_a --approved-is approved

Evaluated pairs  8
  Precision        1.000
  Recall           0.750
  F1               0.857
  Accuracy         0.750
  TP/FP/TN/FN      6/0/0/2

Visualize
visualize renders a suite of PNG charts and an index.html for a dataset — covering label distribution, score distributions, pass/fail breakdown, pair lengths, categories, judge agreement matrix, and validation results.

flywheel visualize -d demo

categories      reports/demo/categories.png
  lengths         reports/demo/lengths.png
  validation      reports/demo/validation.png
  pass_fail       reports/demo/pass_fail.png
  scores          reports/demo/scores.png
  criteria        reports/demo/criteria.png
  labels          reports/demo/labels.png
  judge_agreement reports/demo/judge_agreement.png
  index.html      reports/demo/index.html

Dataset inspection and export
Before exporting, dataset ls and dataset info show what artifacts exist for each dataset.

flywheel dataset ls

name   pairs  source  tags
  demo   8      jsonl   demo1

flywheel dataset info demo

pairs       data/user/demo.jsonl               present
  meta        data/user/demo.meta.json           present
  validation  data/validation/demo.report.json   present
  labels      data/labels/demo.jsonl             present
  judgments   data/judgments                     5 set(s)

Export filters pairs using a safe expression, only pairs with an overall score of 7 or above are written, split 80/20 into train and val.

flywheel dataset export demo \
  --to data/exports/demo.jsonl \
  --format jsonl \
  --judgments data/judgments/demo.judge_a.jsonl \
  --filter "scores['overall'] >= 7" \
  --split train=0.8,val=0.2

Wrote 4 pairs -> data/exports/demo.train.jsonl
Wrote 2 pairs -> data/exports/demo.val.jsonl

Run the autonomous loop
flywheel run ties everything together into a seeds-to-checkpoint cycle. Generation goes through OpenRouter; judging goes through Ollama. If Ollama isn't running, generation still succeeds and pairs are saved in the checkpoint, every judgment falls back to passed=false. The standalone flywheel judge --backend openrouter works fully without Ollama.

export OPENROUTER_API_KEY=sk-or-...
export OPENROUTER_MODEL=meta-llama/llama-3.2-3b-instruct

flywheel run -s "benefits of green tea,history of python language" --max-cycles 1

╭───── Configuration ─────╮
│ Synthetic Data Flywheel │
│ Seeds: 2                │
│ Max Cycles: 1           │
╰─────────────────────────╯
Starting Flywheel with max_cycles=1
============================================================
Starting Cycle 1
============================================================
Using 2 seeds
Generating synthetic data...
Generated 2 pairs
Judging quality...
Passed: 0, Failed: 2
Cycle 1 complete. Pass rate: 0.00%
Flywheel complete. Ran 1 cycles.
       Flywheel Summary
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Metric             ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ Total Cycles       │ 1     │
│ Total Passed Pairs │ 0     │
│ Avg Pass Rate      │ 0.00% │
└────────────────────┴───────┘

Each cycle writes a checkpoint. The generated pair is saved verbatim inside data/checkpoints/checkpoint_001.json:

{
  "instruction": "benefits of green tea",
  "output": "Here is an example of an instruction-following training data in JSON format:\n\n{\n  \"instruction\": \"What are some of the benefits of drinking green tea?\",\n  \"output\": \"Green tea has numerous benefits, including: - High antioxidant content - Anti-inflammatory properties - May help with weight loss ...\",\n  \"category\": \"instruction\"\n}",
  "source_seed": "benefits of green tea"
}

Status and report

flywheel status
flywheel report

status summarises checkpoint state. report produces an HTML report across cycles written to reports/flywheel_report_<timestamp>.html.

CLI Reference

flywheel --help lists the command groups. Every command has --help with full flag docs.

$ flywheel --help
Usage: flywheel [OPTIONS] COMMAND [ARGS]...

  Synthetic Data Flywheel - Autonomous data generation pipeline.

Commands:
  calibrate  Measure judge 'passed' against human labels (precision/recall/F1).
  compare    Compare two+ judgment runs (Cohen's kappa, agreement, ...).
  dataset    Dataset management: ls | info | export.
  ingest     Ingest a user dataset into the flywheel's JSONL format.
  init       Initialize flywheel configuration.
  judge      Judge a dataset with an LLM-as-judge backend.
  label      Label a dataset: interactive/bulk/auto-from-judge.
  pipeline   Run declarative YAML pipelines.
  report     Generate HTML report from checkpoints.
  run        Run the synthetic data flywheel.
  status     Show current flywheel status.
  validate   Validate a dataset and write a ValidationReport.
  visualize  Render a suite of PNG charts + index.html for a dataset.

Pipeline Runner

Individual commands can be composed into a declarative YAML pipeline and run as a single step. This is useful for repeatable workflows, the pipeline dispatches through the same Click commands as manual runs, so behaviour is identical.

# pipeline_demo.yaml
dataset: demo
steps:
  - validate:
      checks: [schema, length, dedup]
  - export:
      to: data/user/demo_pipeline.jsonl
      format: jsonl

flywheel pipeline run pipeline_demo.yaml

[1/2] flywheel validate -d demo --checks schema,length,dedup
[2/2] flywheel dataset export demo --to data/user/demo_pipeline.jsonl --format jsonl
   Pipeline: demo
  1  validate  ok  0
  2  export    ok  0

Python API

The full pipeline is available as a library. The minimal end-to-end call scores a dataset with an async judge backed by Ollama:

import asyncio
from pathlib import Path
from synthetic_data_flywheel.ingest import load_dataset_jsonl
from synthetic_data_flywheel.rubrics import default_rubric
from synthetic_data_flywheel.judge import AsyncQualityJudge
from synthetic_data_flywheel.judge_backends import get_backend
from synthetic_data_flywheel.judge_cache import JudgmentCache

pairs = load_dataset_jsonl("data/user/demo.jsonl")
backend = get_backend("ollama", model="gemma4:latest")
judge = AsyncQualityJudge(
    backend=backend,
    rubric=default_rubric(),
    cache=JudgmentCache(root=Path(".cache/judge")),
    backend_name="ollama",
)
judgments = asyncio.run(judge.judge_batch(pairs, concurrency=2))
print(sum(j.passed for j in judgments), "/", len(judgments), "passed")

The statistical functions used internally by calibrate and compare are also directly callable:

from synthetic_data_flywheel.stats import cohens_kappa, pearson, prf

cohens_kappa([True, False, True, False], [True, True, True, False])
# 0.5

pearson([1,2,3,4], [1,3,2,5])
# 0.8315...

prf([True,True,False,False], [True,False,True,False])
# {'precision': 0.5, 'recall': 0.5, 'f1': 0.5, 'accuracy': 0.5,
#  'tp': 1, 'fp': 1, 'tn': 1, 'fn': 1}

A2A Agent

The flywheel exposes a FastAPI application implementing the A2A protocol surface, /a2a/capabilities, /a2a/tasks/send, /a2a/tasks/get, /a2a/tasks/cancel so it can be orchestrated as a node in a multi-agent ML pipeline.

python -m synthetic_data_flywheel.a2a_agent
# or
uvicorn synthetic_data_flywheel.a2a_agent:app --host 0.0.0.0 --port 8080

Three capabilities are exposed: generate_synthetic_data, get_status, generate_report. Querying /a2a/capabilities returns the agent's identity and the full capability list:

from fastapi.testclient import TestClient
from synthetic_data_flywheel.a2a_agent import app

client = TestClient(app)
print(client.get("/a2a/capabilities").json())
# {'agent_name': 'synthetic_data_flywheel', 'version': '0.1.0',
#  'capabilities': [{'name': 'generate_synthetic_data', ...},
#                   {'name': 'get_status', ...},
#                   {'name': 'generate_report', ...}]}

r = client.post("/a2a/tasks/send", json={
    "capability": "get_status",
    "inputs": [],
    "parameters": {},
})
print(r.json())
# {'task_id': '...', 'status': {'state': 'completed'},
#  'result': {'type': 'status_result',
#             'content': {'checkpoints_found': 1,
#                         'checkpoint_dir': 'data/checkpoints'}}}

Configuration

All settings are read from environment variables or a .env file:

OPENROUTER_API_KEY=sk-or-...
OPENROUTER_MODEL=qwen/qwen3-8b:free
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=gemma4:latest
DEFAULT_JUDGE_BACKEND=ollama        # ollama | openrouter | anthropic
JUDGE_CONCURRENCY=4
JUDGE_TIMEOUT=600
QUALITY_MIN_SCORE=7.0
MAX_CYCLES=10
PII_POLICY=warn                     # strict | warn | off
A2A_HOST=0.0.0.0
A2A_PORT=8080

JUDGE_TIMEOUT defaults to 600 seconds, large local models can take over two minutes on first call.

Limitations

Fine-tuning requires a GPU: Trainer.prepare_training_artifacts writes a Colab-ready Unsloth notebook under notebooks/training_cycle_NNN.ipynb. Running the training step locally on CPU is not supported by Unsloth.

Autonomous generation requires OpenRouter: flywheel run requires OPENROUTER_API_KEY. The in-loop judge is hardcoded to Ollama (engine.create_judge constructs a sync QualityJudge over OllamaClient); if Ollama isn't available, pairs are persisted but every judgment falls back to passed=false.

Large local judges are slow to cold-start: Gemma 4 (9 GB) takes about 130 seconds the first time it loads into VRAM/RAM. The default JUDGE_TIMEOUT is 600 seconds to cover this.

HuggingFace ingest requires datasets: already a dependency, but gated datasets additionally require HUGGINGFACE_TOKEN.

Anthropic judge backend requires ANTHROPIC_API_KEY: no offline fallback.

How I Built This Using NEO

The problem was defined at a high level: a closed-loop pipeline that generates synthetic instruction-tuning pairs, filters them with a calibrated LLM judge, and feeds failure cases back as seeds for the next cycle. NEO generated the full implementation, the FlywheelEngine cycle loop with checkpointing, the AsyncQualityJudge with three pluggable backends and disk-backed cache, the deterministic Validator with six check types, the LabelStore with append-only storage, the statistical calibration layer (cohens_kappa, pearson, prf), the safe-eval export filter, the declarative YAML pipeline runner, the Matplotlib visualisation suite, and the A2A FastAPI agent surface. 100 tests pass.

How You Can Build Further With NEO

Additional judge backends: the three existing backends share a common interface via get_backend. Any OpenAI-compatible endpoint can be wired in as a new backend, and the judge cache, calibration, and compare logic all work with it immediately without any changes.

Additional generation templates: the generator ships with four templates: QA, INSTRUCTION, REASONING, CREATIVE. New domain-specific templates would let the flywheel produce specialised training data, code generation, structured extraction, tool-use, while the cycle loop, judge, and export pipeline stay entirely unchanged.

Additional validation checks: the Validator already supports six check types plugged into the same --checks flag and report format. New checks for domain-specific quality signals would run in the same validation pass and appear in the same JSON report and visualisation output.

Multi-judge ensembling: compare already computes agreement metrics across judgment runs. Taking the average or majority vote across two or more judge scores before the pass/fail decision would reduce the noise that small local models introduce, without touching the labeling, calibration, or export logic downstream.

Final Notes

Synthetic Data Flywheel closes the loop that most synthetic data pipelines leave open. It generates, validates, judges, calibrates, and exports, and feeds what failed back into the next cycle. The result is a data pipeline that improves with each run rather than producing a static batch.
The code is at https://github.com/dakshjain-1616/synthetic-data-flywheel

You can also build with NEO in your IDE using the VS Code extension or Cursor.

Token Budget Negotiator

Nilofer 🚀 — Mon, 27 Apr 2026 21:55:15 +0000

Everyone knows long prompts cost money. Almost nobody knows which parts of their prompt actually matter.

Prompts accumulate over time, a system message, a style guide, a few-shot example or two, some background context. Each addition made sense when it was added. Over hundreds of API calls, the overhead compounds. And the honest answer to "which of these sections can I remove?" is: you don't know until you test.

Token Budget Negotiator makes that test systematic. It takes a prompt split into named, prioritised sections, runs a greedy ablation loop that drops one section at a time, scores the remaining prompt against a rubric using a local or remote LLM judge, and stops when savings hit the target without falling below the quality threshold. The result is the smallest prompt that still behaves like the original.

It ships as a CLI, a Python library, and an MCP server.

The Problem

Prompt sections are not equal in value, but there's no principled way to know which ones matter for a given task without testing. Manual trimming is guesswork. Token Budget Negotiator answers the question empirically per section, per task, against a rubric that defines what quality means for that use case.

How It Works

A prompt is defined as a YAML file with named sections. Each section carries a type (system, few_shot, context, instruction), a content block, and a priority integer. Priority determines the order in which sections are considered for removal low-priority sections are evaluated first, high-priority sections last.

Before any removal happens, the full prompt is scored by the judge LLM against the rubric. This establishes a baseline. The quality target for the run is baseline_score × threshold.

The ablation loop then works through sections in ascending priority order. For each candidate, a test prompt is built without that section and rescored. If the score still meets the target, the section is dropped permanently and the loop continues with the updated prompt. If not, the section is kept and the next candidate is evaluated.

Two conditions stop the loop:

Token savings reach min_token_savings,the target has been hit.
A removal would push savings above max_token_savings, the ceiling is enforced.

Every accepted removal is verified to actually reduce the token count. The loop cannot produce a larger prompt than it started with.
The output is a NegotiationResult containing the original and optimised token counts, the list of sections removed, per-step scores, quality retention percentage, elapsed time, scoring call count, rubric name, and a full ablation log. This can be written to JSON or YAML.

Getting Started

Install

cd token-budget-negotiator
pip install -e .

Requires Python 3.11+. The local judge path requires Ollama with a model pulled, verified end-to-end against gemma4:latest. The OpenRouter path requires OPENROUTER_API_KEY.

Analyze token distribution
Before negotiating, analyze prints how many tokens each section holds and its share of the total budget:

$ token-budget analyze examples/prompt.yaml

Token Distribution Analysis:
Section              Type              Tokens        % Priority
-----------------------------------------------------------------
system               system                22    18.6%       30
style_guide          system                26    22.0%       10
few_shot_1           few_shot              26    22.0%       20
few_shot_2           few_shot              20    16.9%       25
context              context               12    10.2%       40
instruction          instruction           12    10.2%      100
-----------------------------------------------------------------
TOTAL                                     118   100.0%

Check the local judge:

$ token-budget check-ollama --model gemma4:latest
Ollama is connected
  Host: http://localhost:11434
  Model requested: gemma4:latest
  Model available: Yes

Run the negotiator:

$ token-budget negotiate examples/prompt.yaml \
    --scorer ollama --model gemma4:latest \
    --threshold 0.80 --min-savings 0.20 --max-savings 0.80 \
    --output result.json --format json

Negotiation Result:
  Original: 118 tokens, score=0.600
  Optimized: 92 tokens, score=0.700
  Savings: 22.0%
  Quality Retention: 116.7%
  Success: Yes
  Sections removed: style_guide

Results saved to result.json

result.json contains the full ablation log, the final optimized prompt, per-step scores, and metadata (elapsed time, scoring call count, rubric name).

Run the negotiator - OpenRouter path

$ export OPENROUTER_API_KEY=sk-or-...
$ token-budget check-openrouter
OpenRouter is connected
  Base URL: https://openrouter.ai/api/v1
  Model requested: qwen/qwen3-8b

$ token-budget -v negotiate examples/prompt.yaml \
    --scorer openrouter --model meta-llama/llama-3.2-3b-instruct \
    --rubric rubrics/qa.yaml \
    --threshold 0.7 --min-savings 0.1 --max-savings 0.6 --no-cache
Connected to openrouter

Negotiation Result:
  Original: 118 tokens, score=1.000
  Optimized: 92 tokens, score=0.900
  Savings: 22.0%
  Quality Retention: 90.0%
  Success: Yes
  Sections removed: style_guide

With a looser threshold (-t 0.7 --min-savings 0.1 --max-savings 0.5) and caching left on, the same model drops two sections for 44.1% savings at 100% quality retention.

CLI Reference

Key negotiate flags:

-r, --rubric PATH: YAML rubric. Defaults to a built-in accuracy+relevance rubric.
-s, --scorer {ollama,openrouter}: which judge to use. Default ollama.
-m, --model TEXT: model name (gemma4:latest for Ollama, qwen/qwen3-8b for OpenRouter, etc.).
-t, --threshold FLOAT: minimum fraction of the baseline score to keep. Default 0.95.
--min-savings FLOAT: stop once savings reach this fraction. Default 0.40.
--max-savings FLOAT: never drop sections if it would save more than this. Default 0.60.
-o, --output PATH / -f, --format {json,yaml}: write a machine-readable report.
--no-cache: disable the in-memory scoring cache.

Python API

from token_budget_negotiator import (
    Negotiator, OllamaScorer, PromptSection, Rubric,
)
from token_budget_negotiator.models import RubricCriterion, SectionType

sections = [
    PromptSection(name="sys", content="You are helpful.",
                  section_type=SectionType.SYSTEM, priority=20),
    PromptSection(name="q", content="What is 2+2?",
                  section_type=SectionType.INSTRUCTION, priority=100),
]

rubric = Rubric(
    name="qa", description="qa rubric",
    criteria=[RubricCriterion(name="accuracy",
                              description="factually correct",
                              weight=1.0)],
)

scorer = OllamaScorer(model="gemma4:latest")
negotiator = Negotiator(scorer=scorer, quality_threshold=0.9,
                        min_token_savings=0.1, max_token_savings=0.9)

result = negotiator.negotiate(sections=sections, rubric=rubric, target_task="qa")
print(result.original_token_count, "->", result.optimized_token_count)
print("removed:", result.sections_removed)

Rubric Format

The rubric defines what quality means for the task. The judge scores each test prompt against it. Three rubrics ship in rubrics/: qa.yaml, coding.yaml, summarization.yaml.

name: qa
description: General question-answer rubric
version: "1.0"
criteria:
  - name: accuracy
    description: Is the response factually correct?
    weight: 1.0
  - name: relevance
    description: Does it answer what was asked?
    weight: 1.0
scoring_instructions: |
  Score 0-1. 1 = perfect, 0 = wrong or irrelevant.
output_format: json

MCP Server

The library also runs as an MCP server over stdio transport, exposing two tools, analyze and negotiate, so Claude Code or any MCP-compatible agent can call it directly during a session.

python -m token_budget_negotiator.mcp_server --scorer ollama --model gemma4:latest

analyze takes a sections list and returns token distribution as JSON. negotiate takes sections, rubric, task, thresholds, and scorer config and returns the full negotiation result as JSON.

Limitations

Ablation is greedy one-at-a-time in priority order, not exhaustive subset search.
The judge is asked for strict JSON; free-text replies fall back to regex score extraction with reduced confidence.
Small local judges like gemma4 are noisy, prefer thresholds in the 0.80-0.90 range and expect multi-minute wall clock even for short prompts.
check-openrouter and the OpenRouter scorer require OPENROUTER_API_KEY; there is no offline stub.
Only the remove compression strategy is wired up. CompressionStrategy and sections_compressed exist on the model but are not yet produced by the negotiator.

How I Built This Using NEO

The problem was defined at a high level: a tool that takes a structured prompt, scores it with a local or remote LLM judge, and finds the minimum set of sections needed to hit a quality threshold. NEO generated the full implementation, the greedy ablation loop in Negotiator, the OllamaScorer and OpenRouterScorer with their shared interface, the ScoreCache with TTL-based invalidation, the SectionTokenizer backed by tiktoken, the YAML rubric format, the MCP server with its two exposed tools, and the CLI built on Click. All 49 tests pass.

Final Notes

Token Budget Negotiator turns prompt compression from guesswork into an empirical process. It scores every section against a rubric, drops only what demonstrably doesn't matter, and produces a report showing exactly what changed and why.
The code is at https://github.com/dakshjain-1616/token-budget-negotiator
You can also build with NEO in your IDE using the VS Code extension or Cursor.

Agent Memory Compressor: Intelligent Memory Compression for Long-Running LLM Agents

Nilofer 🚀 — Mon, 27 Apr 2026 13:02:47 +0000

A 10-turn agent session can easily accumulate 20,000+ tokens of raw history, leaving almost no room for the current task. Naive truncation drops older turns wholesale, including the decisions and discovered facts the agent needs to avoid repeating work. Developers need a principled way to compress history rather than discard it.

Agent Memory Compressor is a Python library that implements an intelligent memory compression pipeline for long-running LLM agents. It combines importance-based scoring, LLM-driven summarization, a forgetting curve trigger, and a token-budgeted context builder so agents can run indefinitely without exhausting their context windows, while preserving the facts and decisions that matter.

The Problem: Context Window Exhaustion

The problem has three dimensions, and agent-memory-compressor addresses each one directly:

What to keep: A multi-signal importance scorer ranks every memory entry.
How to shrink: Three pluggable compression strategies replace low-value entries with compact equivalents using any OpenAI-compatible LLM.
When to act: A forgetting curve fires compression automatically when either a turn interval or a token threshold is crossed.

How It Works

Importance Scoring
Every memory entry is scored by the ImportanceScorer, which combines three signals:

Compression Strategies
Given a scored store, the CompressionEngine exposes three strategies:

summarize(entry): Asks the LLM for a short summary that preserves all decisions and facts.
extract_facts(entry): Asks the LLM for a bullet list of facts and decisions, stored as high-importance compressed entries.
archive(entry): Replaces the entry with a minimal reference; the original content is retained in the entry's compression_history for audit.

The MemoryCompressor orchestrates the pipeline: score, pick the lowest-scoring non-protected entries, apply the least-destructive strategy first, and iterate until the store is under token_budget. Every successful replacement is verified to actually reduce the token count, so compression can never make the context larger.

The Forgetting Curve
The ForgettingCurve decides when to compress. It combines two triggers:

Turn-based: fires once the number of turns since the last compression reaches compression_interval_turns (default: 10)
Token-based: fires once MemoryStore.token_total() exceeds compression_threshold_tokens (default: 6000), with hysteresis to prevent thrashing.

should_compress(store) returns True as soon as either condition is met. get_compression_priority(store) returns entries sorted by importance, so the orchestrator always attacks the least-valuable history first.

Installation

pip install -e .
# optional, for live LLM calls
pip install openai

The package depends on pydantic, tiktoken (for cl100k_base token counts), click, and rich.

Usage Example

from agent_memory_compressor import MemoryEntry, MemoryStore, MemoryCompressor
from agent_memory_compressor.triggers import ForgettingCurve
from agent_memory_compressor.context import ContextBuilder, ContextConfig
from agent_memory_compressor.strategies import LLMClient, CompressionEngine

store = MemoryStore()
for turn, (role, content) in enumerate(conversation, start=1):
    store.add_entry(MemoryEntry(content=content, role=role, turn_number=turn))

llm = LLMClient(api_key="sk-...", model="gpt-4o-mini")
compressor = MemoryCompressor(
    token_budget=4000,
    protected_recent=3,
    engine=CompressionEngine(llm_client=llm),
)

curve = ForgettingCurve(compression_interval_turns=10,
                       compression_threshold_tokens=6000)

if curve.should_compress(store):
    report = compressor.compress(store)
    curve.mark_compressed(store)
    print(f"Saved {report.tokens_saved} tokens "
          f"({report.compression_ratio:.0%} reduction)")

context = ContextBuilder(ContextConfig(max_tokens=4000)).build_context(
    store, system_message="You are a helpful assistant."
)

Without an API key, LLMClient falls back to a deterministic short stub so pipelines remain runnable in tests and offline demos. A full end-to-end demo lives at demos/long_run_demo.py.

API Reference

A memory-cli entrypoint (click-based) is installed for quick inspection, compression, and demo runs.

Integration with the Session Manager

The adapters module wires the compressor directly into the Stateful Agent Session Manager:

from agent_memory_compressor.adapters import compress_session

compressed_messages, report = compress_session(
    session,              # anything exposing get_messages() / get_metadata()
    token_budget=4000,
    protected_recent=3,
)

SessionAdapter.session_to_store projects session messages into a MemoryStore, compressor.compress(...) runs the pipeline, and store_to_session projects the compressed entries back into the session's message format, preserving original roles and retaining the compression history on each compacted entry.

How I build This Using NEO

This project was built using NEO. A fully autonomous AI engineering agent that writes code end-to-end for AI/ML tasks including model evals, prompt optimization, and pipeline development.
I described the problem at a high level: an intelligent memory pipeline for long-running agents that scores history by importance, compresses the least valuable entries, and assembles a token-bounded context.

NEO generated the full implementation, the multi-signal ImportanceScorer, the three compression strategies in CompressionEngine, the turn- and token-based ForgettingCurve triggers, the token-budgeted ContextBuilder, and the SessionAdapter that wires everything into an existing agent session, all as a coherent, installable Python library.

How You Can Build Further With NEO

Semantic similarity scoring: straightforward, just call an embeddings API and add the score to the existing pipeline. Done all the time in RAG systems.
Pluggable tokenizers: purely an engineering task, just abstract the tiktoken call. No research needed.
More agent framework adapters: LangChain/LlamaIndex all expose message lists. The session_to_store pattern already exists, just repeat it for each framework.
Streaming compression: the trigger logic already exists, moving it per-turn is a refactor not a research problem.

Final Notes

Agent Memory Compressor is a principled answer to context window exhaustion for long-running LLM agents.

Instead of truncating history blindly, it scores every piece of memory, applies the least-destructive compression strategy first, and assembles a token-bounded context that preserves what the agent actually needs, the decisions, discovered facts, and recent turns that matter most.

The code is at https://github.com/dakshjain-1616/Agent-Memory-Compressor
You can also build with NEO in your IDE using the VS Code extension or Cursor.

Cache-Augmented Generation (CAG): A RAG-less Approach to Document QA

Nilofer 🚀 — Sat, 25 Apr 2026 10:29:22 +0000

Most document QA systems today rely on Retrieval-Augmented Generation (RAG). The standard pipeline is familiar: chunk the document, generate embeddings, store them in a vector database, and retrieve relevant chunks at query time.

This works, but it comes with trade-offs. The model only sees fragments of the document, retrieval adds latency, and the system becomes more complex with multiple moving parts.

Cache-Augmented Generation (CAG) explores a different approach,where the document is processed once and reused across queries instead of being retrieved repeatedly.

What is Cache-Augmented Generation

Cache-Augmented Generation (CAG) approaches document QA by reusing the model’s internal state instead of retrieving context for every query.

During ingestion, the entire document is processed in a single pass. In this step, the model builds its KV (key-value) cache, which represents the document’s context.

This KV cache is then saved to disk.

When a query is made, the cache is restored and the query is appended, allowing the model to generate responses using the previously processed document.

How It Works

Step 1: Ingest (done once per document)
The document is wrapped in a structured prompt and sent to llama-server. The model runs a full prefill pass, loading every token into the KV cache. This takes time, proportional to document size, but only happens once. The KV cache is then saved to a .bin file on disk.

Step 2: Query (instant, repeatable)
Before each query, the saved .bin file is restored into llama-server's KV cache in ~1 second. The user's question is appended and the model generates an answer with full document context active. No re-reading, no re-embedding.

Step 3: Persistence
KV slots survive server restarts. Kill the server, restart it, and your next query restores the cache from disk just as fast. The 24-minute prefill for War and Peace only needs to happen once ever.

Validated Results

All 11 GPU tests were run on an NVIDIA RTX A6000 (48 GB VRAM) with Qwen3.5-35B-A3B Q3_K_M at 1,048,576 token context.

Performance

Example Output
“Who is Pierre Bezukhov?” → Correct, detailed answer
“What happened at the Battle of Borodino?” → Correct, detailed answer

Quick Start

Prerequisites: Linux, NVIDIA GPU (8 GB+ VRAM), Python 3.8+

# 1. Build llama.cpp + download model (one-time, ~35 min)
./setup.sh

# 2. Start the LLM server
./start_server.sh

# 3. Start the API server
python3 src/api_server.py

# 4. Ingest a document
python3 src/ingest.py my_document.txt --corpus-id my_doc

# 5. Query it
python3 src/query.py my_doc "What is this document about?"

That's it. After step 4, the KV cache is saved to kv_slots/my_doc.bin. Every future query restores it instantly, and it survives server restarts.

Model Selection

setup.sh auto-detects GPU VRAM and picks the right model:

The 24 GB+ path uses unsloth/Qwen3.5-35B-A3B-GGUF on HuggingFace and requires a free HF account + access token.

REST API

Start the API server with python3 src/api_server.py --port 8000 (optionally set CAG_API_KEY env var to enable key auth).

Full API docs available at http://localhost:8000/docs when the server is running.

Directory Structure

.
├── setup.sh              # Builds llama.cpp, downloads model
├── start_server.sh       # Launches llama-server with CAG flags
├── requirements.txt
├── src/
│   ├── api_server.py     # FastAPI REST API
│   ├── ingest.py         # CLI: ingest a document
│   ├── query.py          # CLI: query a corpus
│   └── demo.py           # End-to-end demo
├── docker/
│   ├── Dockerfile
│   └── docker-compose.yml
├── docs/
│   ├── REPORT.md         # Full GPU validation report with all 11 test results
│   └── GPU_TESTING.md    # GPU test checklist
├── models/               # GGUF weights (not committed)
├── kv_slots/             # Saved KV cache .bin files (not committed)
└── logs/                 # Runtime logs (not committed)

Limitations

Linux + NVIDIA only: TurboQuant CUDA kernels require Linux and NVIDIA GPUs (no Windows, macOS, or AMD).

Long initial prefill: ~900K tokens can take ~24 minutes on an A6000. This is a one-time cost; subsequent queries restore in ~1 second.

VRAM gating: Systems with lower VRAM use smaller models with shorter context.

Single active corpus: Uses a single llama.cpp slot (slot 0). Switching corpora requires restoring a different KV cache (~1 second).

Long-context limitations: YaRN extrapolation biases attention toward the start and end of documents, so mid-document content can be missed at very large context sizes.

Build time: Initial setup (./setup.sh) can take ~35 minutes to compile CUDA kernels.

Model access requirements: Large models (e.g., Qwen3.5-35B) require a Hugging Face account and access token.

How I Built This Using NEO

The system was defined at a high level, describing a document QA workflow that avoids RAG by loading full documents into an LLM, saving the KV cache, and restoring it for repeated queries.

Based on this, NEO generated the implementation, handled debugging across CUDA, Python, and shell components, and validated the system through a series of GPU tests.

This included fixing multiple issues during development and running end-to-end validation to ensure ingestion, cache restoration, and query flows worked reliably.

How to Extend This Further with NEO

The system can be extended in several ways:

supporting multiple KV cache slots
improving handling of long-context attention limitations
optimizing cache storage and compression
exploring hybrid approaches combining CAG with retrieval
extending API capabilities

These extensions would require changes to the current implementation and can be explored based on system requirements.

Final Notes

Cache-Augmented Generation is an alternative way to approach document QA.

Instead of retrieving context at query time, it shifts the cost to a one-time preprocessing step and reuses the model’s KV cache.

This makes repeated queries faster and makes the document context available to the model through the KV cache, while introducing trade-offs in setup time and hardware requirements.

The code is at https://github.com/dakshjain-1616/Cache-Augmented-Generation-CAG-System
You can also build with NEO in your IDE using the VS Code extension or Cursor.