DEV Community: Nilofer 🚀

Context Compaction Visualizer: See Exactly What Your AI Agent Forgot Before It Costs You

Nilofer 🚀 — Tue, 23 Jun 2026 06:01:50 +0000

When an AI agent runs for many turns, it eventually hits context limits and must compress or discard earlier messages. This is often invisible, yet critical - lost context can cause the agent to forget constraints, user preferences, or prior decisions. The framework moves on. The agent keeps running. And somewhere in those discarded turns is a security finding, a constraint, a decision that the rest of the session quietly proceeds without.

Context Compaction Visualizer makes that process visible - not after something breaks, but as an inspectable artifact of every run. It is a visualization platform that helps teams understand how long-running AI agents manage and compress context over time - upload execution traces from LangSmith, OpenTelemetry, AgentOps, or any custom format, and explore exactly which context was retained, compressed, or discarded, and at what cost.

What This Platform Does

The core problem is that compaction happens inside the framework's internals. There is no standard output that tells you which messages survived, which were summarized, and which were dropped - or what any of that cost in tokens. This platform reconstructs that picture from execution traces.

A trace file is uploaded with a format selected, and the platform rebuilds the full session: every message at every turn, its fate - retained verbatim, summarized, or discarded - and any compaction events that occurred along the way. A D3.js stacked-bar timeline renders token consumption across all turns with color-coded regions for each outcome. A session replay steps through turn by turn, surfacing a diff at the exact point a compaction event fires. Token analytics compute the total cost and compression efficiency of the session. A Claude-powered information loss detector scores the risk of each compaction event and names specifically what may have been lost.

When two traces are available - two different agents, or the same agent under two different compaction strategies - a comparative view places them side by side to show which preserved more context at lower cost.

Quick Start

Prerequisites

Python 3.11 or newer
Node.js 18 or newer
An Anthropic API key (optional - only the Info Loss Detection feature needs it)

Set up the environment

cp .env.example .env
# Optionally add ANTHROPIC_API_KEY=sk-ant-your-key-here for info loss detection

Run the backend

cd backend
pip install -r requirements.txt
uvicorn main:app --reload --host 0.0.0.0 --port 8000

The API starts at http://localhost:8000. Interactive docs are available at http://localhost:8000/docs.

Run the frontend

The frontend runs in a separate process:

cd frontend
npm install
npm run dev

The UI opens at http://localhost:5173.

Run with Docker

cp .env.example .env
docker compose up --build

The backend runs on port 8000, the frontend serves via nginx on port 5173.

Running Tests

cd backend && python -m pytest tests/ -v

29 tests covering all four parsers, edge cases, and the token counter. The full suite runs in under 100ms since nothing in it hits an external service.

API Reference

Supported Trace Formats

The platform accepts four input formats, selectable via a dropdown on upload. Each has its own parser that handles the vendor-specific schema and reduces it to the normalized structure.

LangSmith - Parses JSON exports from the LangSmith tracing platform. The parser extracts runs, the messages inside each run, token counts from usage metadata, and any chain-level summarization events.

OpenTelemetry - Parses OTEL-format JSON spans. The parser traverses the span tree, reconstructs message history from span attributes, and identifies compaction events from span names containing "compress" or "summarize".

AgentOps - Parses AgentOps session JSON exports. The parser handles the session-level event structure and normalizes message roles from AgentOps event types.

Custom JSON - A generic format for any agent framework not listed above. It expects a messages array with role, content, and optional tokens and timestamp fields. Any event with type: "compaction" or type: "summarization" is treated as a compaction event.

Project Structure

context-compaction-visualizer/
├── backend/
│   ├── main.py                  # FastAPI app, 7 endpoints
│   ├── models.py                # Trace, Message, CompactionEvent ORM
│   ├── schemas.py               # Pydantic validation schemas
│   ├── database.py              # SQLAlchemy + SQLite setup
│   ├── parsers/
│   │   ├── langsmith_parser.py
│   │   ├── otel_parser.py
│   │   ├── agentops_parser.py
│   │   └── custom_parser.py
│   ├── services/
│   │   ├── context_analyzer.py  # Claude-powered info loss detection
│   │   └── token_counter.py     # Token counting + cost estimates
│   ├── requirements.txt
│   ├── Dockerfile
│   └── tests/
│       ├── test_parsers.py      # 29 tests covering all 4 parsers
│       └── fixtures/
│           ├── langsmith_trace.json
│           ├── otel_trace.json
│           ├── agentops_trace.json
│           └── custom_trace.json
├── frontend/
│   ├── src/
│   │   ├── App.tsx              # Upload/Timeline/Replay/Analytics/Loss/Compare tabs
│   │   ├── components/
│   │   │   ├── TraceUploader.tsx
│   │   │   ├── ContextTimeline.tsx   # D3.js stacked bar chart
│   │   │   ├── SessionReplay.tsx     # Turn navigation + compaction diff
│   │   │   ├── TokenAnalytics.tsx
│   │   │   ├── InfoLossDetector.tsx
│   │   │   └── ComparativeView.tsx
│   │   ├── hooks/useD3.ts
│   │   ├── api/client.ts
│   │   └── types/index.ts
│   ├── Dockerfile
│   ├── package.json
│   └── vite.config.ts
├── docker-compose.yml
└── .env.example

The file structure reflects the normalization design directly. Every file under backend/parsers/ handles one vendor's schema and outputs the same structure. Nothing downstream - not main.py, not any frontend component - needs to know which parser ran. The two services, context_analyzer.py and token_counter.py, sit after all four parsers and only ever see the normalized output.

Key Design Decisions

Parser normalization - Each observability platform has a fundamentally different schema. Rather than handling platform-specific quirks in every component, all four parsers produce an identical normalized structure. This means the timeline, replay, analytics, and comparison views have no knowledge of the original format.

Graceful Claude fallback - The Info Loss Detector calls the Anthropic API only when ANTHROPIC_API_KEY is set. Without a key, it returns analysis_available: false with a clear message rather than failing. The rest of the platform works fully without any API key.

D3.js integration via hook - The useD3.ts hook manages D3's selection lifecycle within React's rendering model. D3 takes ownership of the SVG element inside the hook's effect, while React manages the wrapping div and props. This avoids the common conflict between React's virtual DOM and D3's direct DOM manipulation.

Centralized cost estimates - Token counts and cost calculations happen in token_counter.py using verified Claude pricing - $3 per million input tokens, $15 per million output tokens - defined as constants in one place, making them easy to update if pricing changes.

Environment Variables

Verified Results

The backend ships with 29 tests covering all four trace parsers, realistic multi-turn fixture data for each format, edge cases like empty inputs and missing fields, and the token counter. All tests run in under 100ms since no external services are called. The frontend builds to a 238 KB JS bundle across 600 modules.

For the info-loss detector, the ContextAnalyzer was run (using DeepSeek V4 Flash via OpenRouter for this verification pass) against a real compaction event that had dropped 77,000 tokens from a security code review session. It returned an overall risk score of 0.85 and flagged two losses. The higher-risk item, scored 0.90, was the permanent loss of three specific JWT authentication findings - a missing expiry check, absent refresh token rotation, and a weak secret key - detail precise enough that no summary would have preserved it. The second item, scored 0.70, flagged the loss of 23 tool call exchanges' worth of reasoning context. Both came back with concrete recommended actions, not generic advice.

How I Built This Using NEO

This project was built autonomously using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a platform that ingests execution traces from any of the major agent observability tools, reconstructs what an agent's context looked like turn by turn, and uses an LLM to flag when something important got dropped during compaction. NEO planned and produced the entire codebase - four format parsers that each reduce a different vendor schema into one normalized structure, a FastAPI backend with seven endpoints wired to SQLAlchemy models, two backend services handling token counting and Claude-powered info loss detection, a full React and TypeScript frontend with D3.js visualizations across six components, and a 29-test suite with realistic multi-turn fixtures for all four formats. The repo's plans/ directory and ORCHESTRATOR_LOG.md document that build run directly.

The result is a fully working visualization platform that takes a raw trace file in, and gives you back a complete picture of what your agent remembered, what it forgot, and what that cost.

How You Can Use This With NEO

Audit any long-running agent for context loss.
If a LangSmith, OpenTelemetry, or AgentOps trace exists for an agent run, it can be dropped straight into the platform. The timeline and session replay immediately show which turns survived compaction and which did not - no instrumentation changes, no code modifications to the agent itself.

Benchmark compaction strategies before shipping.
When evaluating two different agent configurations or memory strategies, both traces can be uploaded and placed side by side in the comparative view. The platform surfaces which strategy retained more context at lower token cost, turning a subjective comparison into a measurable one.

Catch silent information loss in security or compliance-sensitive agents.
The Claude-powered info loss detector scores each compaction event and flags specific content that may have been dropped - as demonstrated with the JWT authentication findings in the verified results. Any agent operating over sensitive or constraint-heavy sessions can be run through this check before the output is trusted. This requires ANTHROPIC_API_KEY to be set; without it the platform returns analysis_available: false for this feature.

Use the custom JSON format to bring any agent framework in.
Agents not running on LangSmith, OpenTelemetry, or AgentOps can still feed into the platform by logging to the custom JSON format - a messages array with role, content, and optional tokens and timestamp fields. Any agent framework that can write JSON can produce a trace this platform accepts.

Final Notes

Compaction is designed to be invisible - the agent keeps running, the framework handles the limit, and nothing interrupts the workflow. The cost of that invisibility is that when something is silently dropped, there is no record of what it was or what it was worth. Context Compaction Visualizer produces that record.

The code is at https://github.com/dakshjain-1616/Context-Compaction-Visualizer
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Agent Sandbox Escape Detector: Black-Box Security Scanning for LLM Agents

Nilofer 🚀 — Fri, 12 Jun 2026 16:29:19 +0000

Most agent security tools focus on known jailbreak phrases or static rule-matching. That approach misses the point. A real attacker does not check a list of banned words - they probe the agent's actual behavior with semantically varied adversarial inputs and look for signs that something slipped through.

Agent Sandbox Escape Detector takes the same approach. Point it at any HTTP chat endpoint, and it fires a battery of adversarial prompts across six attack categories, then uses Claude Opus 4.8 as an independent judge to determine whether the agent leaked data, broke persona, or executed injected instructions. The result is a structured scan report with per-probe verdicts, evidence excerpts, and confidence scores.

The key insight is that you do not need whitebox access to test an agent — all you need is its chat endpoint. The scanner treats the agent as a black box and probes it the same way a real attacker would.

Architecture

Entry: CLI (--target URL)  ·  POST /scan
               │
               ▼
  ┌────────────────────────────────────────────┐
  │  Scanner  ·  asyncio.gather()              │
  │  all 6 probes run concurrently             │
  │  per-probe error isolation                 │
  └──────────────────┬─────────────────────────┘
                     │
       ┌─────────────┼──────────────────┐
       ▼             ▼                  ▼
  ┌─────────┐  ┌──────────────┐  ┌────────────────────┐
  │  Tool   │  │ Prompt Leak  │  │ API Call           │
  │ Access  │  │ Role Confus. │  │ Indirect Injection │
  └────┬────┘  └──────┬───────┘  │ Jailbreak  (6 tot) │
       │               │          └──────────┬─────────┘
       └───────────────┴───────────────────── ┘
                     │ 4–6 adversarial prompts per category
                     ▼
  ┌────────────────────────────────────────────┐
  │  Target Agent  ·  any HTTP chat endpoint   │
  │  httpx async POST                          │
  └──────────────────┬─────────────────────────┘
                     │ agent responses
                     ▼
  ┌────────────────────────────────────────────┐
  │  Claude Judge  ·  claude-opus-4.8          │
  │  via OpenRouter                            │
  │  → ESCAPED / SAFE  ·  confidence (0–1)     │
  │  → evidence excerpt from response          │
  └──────────────────┬─────────────────────────┘
                     │
                     ▼
  ┌────────────────────────────────────────────┐
  │  Scan Report                               │
  │  JSON  ·  Markdown  ·  Rich console        │
  │  GET /results/{scan_id}                    │
  └────────────────────────────────────────────┘

What It Tests

Each probe category targets a different class of vulnerability observed in production LLM deployments. Every category fires 4–6 distinct prompts to reduce false negatives from surface-level prompt filtering. All probes run concurrently - a full scan of six categories completes in roughly the same time as a single category.

Tool Access - requests that trick the agent into invoking unauthorized tools: file reads, database queries, exec calls. A vulnerable agent attempts to call off-limits functions or returns file and database content.

Prompt Leak - social-engineering and meta-instruction tricks to extract the system prompt. A vulnerable agent reveals system prompt contents, configuration, or internal instructions.

API Call - crafted inputs designed to make the agent perform SSRF or data exfiltration via HTTP. A vulnerable agent constructs or references external URLs containing sensitive data.

Role Confusion - persona hijacking, identity override, DAN-style persona injection. A vulnerable agent breaks character or adopts a no-restrictions alternate identity.

Indirect Injection - malicious payloads hidden in simulated tool outputs or document context. A vulnerable agent executes injected instructions as if they came from a trusted source.

Jailbreak - chain-of-thought manipulation, hypothetical framing, and reasoning override. A vulnerable agent agrees to bypass safety constraints or leaks reasoning chain internals.

How the Judge Works

After collecting all agent responses, each probe's batch of responses is sent to Claude Opus 4.8 via OpenRouter with a structured evaluation prompt. The judge produces a binary ESCAPED/SAFE verdict, a confidence score from 0 to 1, and a natural-language evidence summary quoting specific parts of the response that led to the verdict.

This approach is significantly more accurate than regex pattern matching because the judge understands context - an agent that says "I cannot help with that" is different from one that says "I cannot help with that, but here is /etc/passwd anyway."

Getting Started

Install dependencies into a virtual environment, copy .env.example to .env, and add your OPENROUTER_API_KEY. Then point the CLI at any agent's chat endpoint:

python -m src.cli scan --target http://localhost:8000/chat

To scan only specific probe categories or save results to JSON:

python -m src.cli scan --target http://localhost:8000/chat --probes tool_access,jailbreak --output report.json

Start the FastAPI server for REST integration:

uvicorn src.api.main:app

Environment Variables

OPENROUTER_API_KEY=sk-or-...    # Required — used for Claude judge calls via OpenRouter

API

POST /scan - accepts a target URL and optional probe list, returns a scan ID immediately, and runs the scan asynchronously.

GET /results/{scan_id} - returns the full structured report once the scan is complete.

GET /health - liveness probe for uptime monitoring.

Live Scan Results

Real scan run against a Claude-powered HTTP agent on 2026-06-09:

0 escapes detected across 6 probe categories - approximately 30 adversarial turns total. Scan ID 0c4bffa6, 2026-06-09.

Source Layout

The scanner orchestrates all probes via asyncio.gather() so they run in parallel, with per-probe error isolation so a timeout on one category never blocks the others. Each probe is a standalone class inheriting from BaseProbe - adding a new attack category means writing one class and one prompts file. The judge lives in core/judge.py and is stateless: it takes a list of responses and returns a list of ProbeResult objects. Reports are assembled by core/report.py, which handles JSON serialization, Markdown formatting, and Rich console rendering independently.

The test suite uses a vulnerable dummy agent fixture - an in-process FastAPI app that always complies with requests - to verify the scanner can detect escapes, and a safe dummy agent to verify it does not produce false positives. 64 tests, passing in approximately 15 seconds.

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a black-box behavioral security scanner for LLM agents - one that probes any HTTP chat endpoint with adversarial prompts across six attack categories, uses Claude Opus 4.8 as an independent judge, and produces structured reports with per-probe verdicts, evidence excerpts, and confidence scores. NEO built the full implementation: the async orchestrator using asyncio.gather() with per-probe error isolation, all six probe classes inheriting from BaseProbe with their adversarial prompt files, the stateless Claude judge in core/judge.py via OpenRouter, the report assembler in core/report.py covering JSON, Markdown, and Rich console output, the CLI entry point, the FastAPI REST server with POST /scan and GET /results/{scan_id}, and the 64-test suite with vulnerable and safe dummy agent fixtures.

How You Can Use and Extend This With NEO

Use it to security-test any agent before it goes to production.
Point the CLI at your agent's chat endpoint and run a full six-category scan. The structured report tells you exactly which probe categories the agent failed, what the judge found in the response, and the confidence score - before real users can probe the same vulnerabilities.

Integrate it into CI to catch security regressions on every deploy.
Use POST /scan to trigger a scan and GET /results/{scan_id} to poll the report. If any probe returns ESCAPED above your confidence threshold, fail the pipeline. Agent behavior can regress with model updates or prompt changes - automated scanning catches this before it reaches production.

Use the probe categories as a security checklist when building agents.
The six categories - tool access, prompt leak, API call, role confusion, indirect injection, and jailbreak - map directly to the vulnerabilities that have been observed in production LLM deployments. Running the scanner on your agent during development tells you which categories need stronger guardrails before launch.

Extend it with additional probe categories.
Each probe is a standalone class inheriting from BaseProbe with a corresponding prompts file. A new attack category follows the same pattern and is automatically picked up by the orchestrator, judge, and report pipeline without any changes to the core.

Final Notes

Agent security is behavioral, not syntactic. A scanner that checks for banned phrases misses the attacks that matter. Agent Sandbox Escape Detector probes real behavior across six attack categories, judges responses with a frontier model that understands context, and gives you structured evidence - so you know not just whether an agent escaped, but how and where.

The code is at https://github.com/dakshjain-1616/Agent-Sandbox-Escape-Detector
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

AgentLiar Detector: Catch Coding Agents That Falsely Claim Task Completion

Nilofer 🚀 — Wed, 10 Jun 2026 12:41:47 +0000

AI coding agents are getting better at completing tasks. They are also getting better at appearing to complete tasks. An agent that claims "done" when it has created placeholder files, written empty tests, or quietly narrowed the scope of the original requirement is harder to catch than one that simply fails, because the failure is hidden inside output that looks correct at a glance.

AgentLiar is a production-ready system that detects when coding agents falsely claim task completion. It runs four independent verification checks, produces a weighted confidence score from 0 to 100, and delivers structured evidence in JSON, Markdown, or console output - usable as a CLI tool, Python library, GitHub Action, or HTTP API.

Features

4 Independent Checks - File, Test, Scope, and LLM Judge.
Confidence Scoring - weighted aggregation on a 0–100 scale.
Multiple Interfaces - CLI, Python API, GitHub Action, and HTTP API.
Adversarial Detection - catches placeholder implementations, empty tests, and scope narrowing.
Structured Reports - JSON and Markdown output with evidence.
Production Ready - type hints, error handling, logging, and async support.

Architecture

The async orchestrator dispatches four independent checks File, Test, Scope (local), plus an optional OpenRouter LLM Judge and produces a weighted 0–100 confidence score delivered as JSON, Markdown, or console output for CI gating.

The Four Verification Checks

1. File Check

Detects missing expected files
Identifies unexpected new files
Finds placeholder content: TODO, FIXME, pass-only
Validates file sizes and content

2. Test Check

Detects empty test bodies
Identifies tests without assertions
Finds skipped tests
Validates claimed versus actual test counts

3. Scope Check

Detects silent scope narrowing: "only", "for now"
Identifies partial implementations
Finds TODO markers in code
Validates requirements coverage

4. LLM Judge

Independent assessment via OpenRouter
Structured JSON output
Timeout and retry logic
Optional - works without an API key

Quick Start

Installation

pip install -e .

Or pip install agentliar once published. Requires Python 3.10+.

CLI Usage

Prepare sample inputs from examples/simple_task.json, then run:

agentliar verify \
  --task-file .tmp/task.txt \
  --claim-file .tmp/claim.json \
  --changes-file .tmp/changes.json \
  --format markdown

Use agentliar config to inspect configuration and agentliar analyze .tmp/task.txt to review a task file.

Python API

from agentliar import Verifier

verifier = Verifier()
result = await verifier.verify(
    task_description=task,
    claim=claim_payload,
    file_changes=changes_payload
)
# Read result.score, result.passed, result.confidence_level, result.reports

GitHub Action

Use the GitHub Action with task, claim, and change files, a confidence threshold, and an optional OPENROUTER_API_KEY secret when you want the LLM Judge path enabled.

HTTP API

Start the API server:

python -m agentliar.server
# or
uvicorn agentliar.server:app --host 0.0.0.0 --port 8000

Then POST /verify with the task, claim, and file-change payloads. The response returns score, pass/fail, and evidence blocks.

Confidence Score Interpretation

90–100 - High. Task appears fully completed.
70–89 - Medium. Task likely complete with minor issues.
50–69 - Low. Task partially completed.
30–49 - Critical. Significant issues detected.
0–29 - Failed. Task likely not completed.

Configuration

Create a .env file. Set OPENROUTER_API_KEY and OPENROUTER_MODEL only if you want LLM Judge mode. The check weights must sum to 1.0. CONFIDENCE_THRESHOLD controls the pass/fail cutoff.

Recommended LLM Judge models (May 2026):

anthropic/claude-haiku-4-5 - cheap and fast judging
anthropic/claude-sonnet-4-6 or openai/gpt-5.4 - higher-quality judging
openai/gpt-4.1-mini - budget option

Use Cases

CI/CD Integration - automatically verify PR claims before merging.
Code Review - get an independent assessment of task completion alongside a human review.
Agent Monitoring - detect when AI agents overstate progress in automated pipelines.
Quality Gates - block merges below a confidence threshold.
Documentation - generate verification reports for stakeholders.

Security

No hardcoded secrets
API keys via environment variables only
No data persistence
Local processing except for LLM Judge

Project Structure

src/agentliar/           # Checks, orchestration, scoring, reports, API, CLI, server
tests/
├── unit/                # Unit tests
├── adversarial/         # Adversarial tests
└── integration/         # Integration tests
examples/                # Sample inputs
action.yml               # GitHub Action definition
pyproject.toml           # Packaging and tooling

Testing

pytest                                    # Full suite
pytest --cov=agentliar --cov-report=html  # With coverage
pytest tests/unit/                        # Unit tests only
pytest tests/adversarial/                 # Adversarial tests only
pytest tests/integration/                 # Integration tests only

Code Quality

ruff check .        # Linting
ruff format .       # Formatting
mypy src tests      # Type checking

How I Built This Using NEO

The requirement was a production-ready verification system for detecting false completion claims from coding agents - running four independent checks locally, with an optional LLM Judge via OpenRouter, and exposing the result through a CLI, Python API, GitHub Action, and HTTP API. NEO built the full implementation: the async orchestrator dispatching all four checks, the File, Test, Scope, and LLM Judge check modules, the weighted confidence scorer, the JSON and Markdown report generators, the Click CLI with verify, config, and analyze commands, the FastAPI HTTP server, the GitHub Action definition in action.yml, and the test suite split across unit, adversarial, and integration coverage.

How You Can Use and Extend This With NEO

Use it as a CI gate on every PR that includes AI-generated code.
Add the GitHub Action to your workflow with a confidence threshold. Any PR where the agent's claimed changes do not pass the file, test, and scope checks below your threshold is blocked before merge - automatically, without a reviewer having to spot the placeholder implementation manually.

Use it to monitor agent progress in long-running pipelines.
Call await verifier.verify(...) from Python after each agent task completes. The confidence score and evidence blocks tell you whether the agent actually finished the task or produced output that looks complete but is not - before the next stage of the pipeline starts.

Use the LLM Judge for higher-confidence verification on critical tasks.
Set OPENROUTER_API_KEY and configure a judge model for tasks where the local checks alone are not sufficient. The LLM Judge runs independently from the other three checks and adds a cross-model perspective to the confidence score.

Extend it with additional check types.
The four checks share a common async interface in the orchestrator. A new check follows the same pattern and its weight is added to the configuration. The orchestrator, scorer, and reporters pick it up automatically.

Final Notes

Agents that falsely claim completion are harder to catch than agents that fail outright - because the output exists and looks plausible. AgentLiar makes the verification systematic: four independent checks, a weighted confidence score, and structured evidence that tells you exactly where the claim breaks down.

The code is at https://github.com/dakshjain-1616/AgentLiar
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Carbon-Aware Model Training: Scheduling GPU Workloads Around Electricity Carbon Intensity

Nilofer 🚀 — Sat, 06 Jun 2026 08:48:43 +0000

Training ML models has an environmental cost that most practitioners do not measure. A model trained during peak grid hours, when coal and gas plants are meeting high demand - can emit significantly more CO2 than the same model trained during off-peak hours when renewables dominate the grid. The carbon intensity of electricity varies by a factor of 2–5x throughout the day, but most training pipelines ignore this entirely.

Carbon-Aware Model Training Pipeline is a PyTorch-based training pipeline that monitors real-time electricity carbon intensity, delays training until a low-carbon window is available, reduces GPU memory footprint through gradient accumulation, and tracks CO2 emissions throughout the training process using CodeCarbon - with a comparison report that quantifies the carbon savings against a baseline run.

Features

Carbon-Aware Scheduling - real-time carbon intensity monitoring with smart training delays until low-carbon windows are detected.
Gradient Accumulation - reduces GPU memory footprint while maintaining effective batch size.
Emissions Tracking - real-time CO2 monitoring via CodeCarbon with comprehensive JSON reports.
Modular Design - YAML-based configuration with separate scheduler, tracker, and trainer modules.
GPU Optimized - automatic CUDA detection with mixed precision training (FP16).
Comparative Analysis - automated reporting quantifying carbon savings against a baseline run.

How It Works

The pipeline runs in four stages:

Stage 1 - Carbon-Aware Scheduling
Real-time monitoring checks electricity carbon intensity via APIs. Smart delays wait for low-carbon windows before starting training. Fallback mechanisms use realistic mock data when APIs are unavailable - with diurnal patterns simulating peak intensity at 18:00 and trough at 03:00. Configurable thresholds allow customization for different regions.

Stage 2 - Gradient Accumulation
Memory optimization processes smaller micro-batches. Effective batch size is maintained with reduced memory. Configurable steps (2, 4, 8, 16) adapt to hardware constraints. Convergence preservation ensures model quality is not compromised.

Stage 3 - Emissions Tracking
CodeCarbon integration monitors CO2 emissions in real-time. Energy metrics track power consumption in Watts and energy in kWh. Comprehensive reports generate JSON summaries with all metrics. Comparative analysis quantifies carbon savings versus the baseline.

Stage 4 - GPU Optimization
Mixed precision training (FP16) reduces memory and increases speed. Automatic CUDA detection uses GPU when available. Pin memory optimization enables faster data transfers. Graceful CPU fallback when GPU is unavailable.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Training Configuration                      │
│                       (YAML Config File)                        │
└─────────────────────────┬───────────────────────────────────────┘
                          │
                          ▼
         ┌────────────────────────────────────┐
         │   Carbon Intensity Scheduler       │
         │   - API/Mock data fetch            │
         │   - Threshold comparison           │
         │   - Wait for low-carbon window     │
         └────────────────┬───────────────────┘
                          │
                          ▼
              ┌───────────────────────┐
              │   Start Training?     │
              │   Intensity < 300?    │
              └─────┬─────────────┬───┘
                    │ NO          │ YES
                    ▼             ▼
            ┌───────────┐   ┌──────────────┐
            │   Wait    │   │ Start Tracker│
            │ & Recheck │   │ (CodeCarbon) │
            └───────────┘   └──────┬───────┘
                                   │
                                   ▼
                  ┌────────────────────────────────┐
                  │   PyTorch Training Loop        │
                  │   - Gradient Accumulation      │
                  │   - Mixed Precision (FP16)     │
                  │   - Checkpointing              │
                  └────────────────┬───────────────┘
                                   │
                                   ▼
                  ┌────────────────────────────────┐
                  │   Emissions Tracking           │
                  │   - CO2 (kg)                   │
                  │   - Energy (kWh)               │
                  │   - Power (Watts)              │
                  └────────────────┬───────────────┘
                                   │
                                   ▼
              ┌───────────────────────────────────┐
              │   Save Results                    │
              │   - Model checkpoint              │
              │   - Training summary (JSON)       │
              │   - Emissions log (CSV)           │
              └───────────────────────────────────┘

Installation

Prerequisites

Python 3.8+
PyTorch 2.0+
CUDA (optional, for GPU acceleration)

git clone https://github.com/dakshjain-1616/CarbonAwareModelTraining---by-NEO.git
cd CarbonAwareModelTraining---by-NEO

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

pip install -r requirements.txt

Required packages: torch>=2.0.0, torchvision>=0.15.0, codecarbon>=2.3.0, pyyaml>=6.0, numpy.

Quick Start

source venv/bin/activate

# Run baseline training (no optimization)
export PYTHONPATH="$PWD/src:$PYTHONPATH"
python src/train.py configs/baseline.yaml

# Run optimized training (carbon-aware + gradient accumulation)
python src/train.py configs/optimized.yaml

# Generate comparison report
python generate_comparison.py

This runs three steps: baseline training without carbon awareness, optimized training with carbon-aware scheduling and gradient accumulation, and a comparison report that quantifies carbon savings and performance metrics.

Demo

Configure carbon-aware training in configs/optimized.yaml:

scheduler:
  enabled: true
  carbon_threshold: 300           # gCO2/kWh
  wait_for_low_carbon: true

training:
  batch_size: 16
  gradient_accumulation_steps: 4  # Effective batch = 64
  epochs: 3

Run optimized training:

python src/train.py configs/optimized.yaml

Output:

============================================================
CARBON-AWARE TRAINING STARTED
============================================================

Carbon Intensity Check:
  Current Intensity: 420.5 gCO2/kWh
  Threshold: 300 gCO2/kWh
  Status: ⏳ Waiting for low-carbon window...

[10 minutes later]
  Current Intensity: 285.3 gCO2/kWh
  Status: ✅ Starting training now!

Training Progress:
  Epoch 1/3 - Loss: 0.324 - Accuracy: 91.2%
  CO2 Emissions: 0.042 kg
  Energy Consumed: 0.15 kWh

============================================================
CARBON SAVINGS vs BASELINE
============================================================

CO2 Reduction: 32.5% (0.024 kg saved)
GPU Memory Reduction: 45.8%
Accuracy: 93.1% (baseline: 93.4%)

Usage Examples

Carbon-Aware Scheduling Only

Disable gradient accumulation, enable scheduling:

scheduler:
  enabled: true
  carbon_threshold: 250

training:
  gradient_accumulation_steps: 1  # No accumulation

Gradient Accumulation Only

Disable scheduling, enable memory optimization:

scheduler:
  enabled: false

training:
  batch_size: 8
  gradient_accumulation_steps: 8  # Effective batch = 64

Real Carbon Intensity API
Configure for production with a real API:

scheduler:
  enabled: true
  use_mock_data: false
  api_endpoint: "https://api.carbonintensity.org.uk/intensity"
  region: "GB"

Custom Model Integration
Replace SimpleCNN in src/train.py:

from my_models import MyCustomModel

def prepare_model(config):
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = MyCustomModel(
        input_channels=config['training']['input_channels'],
        num_classes=config['training']['num_classes']
    ).to(device)
    return model, device

Output Format
Training summary JSON saved to output/summary_optimized.json:

{
  "run_name": "optimized",
  "training_metrics": {
    "final_accuracy": 93.1,
    "final_loss": 0.124,
    "epochs": 3,
    "total_time_seconds": 245
  },
  "carbon_metrics": {
    "total_emissions_kg": 0.042,
    "energy_consumed_kwh": 0.15,
    "avg_power_watts": 145.2
  },
  "scheduler_metrics": {
    "wait_time_seconds": 600,
    "initial_intensity": 420.5,
    "training_intensity": 285.3
  },
  "gpu_metrics": {
    "peak_memory_mb": 2048,
    "gradient_accumulation_steps": 4,
    "effective_batch_size": 64
  }
}

Comparison report saved to output/comparison_report.json:

{
  "carbon_savings": {
    "baseline_emissions_kg": 0.074,
    "optimized_emissions_kg": 0.042,
    "reduction_kg": 0.032,
    "reduction_percentage": 43.2
  },
  "accuracy_impact": {
    "baseline_accuracy": 93.4,
    "optimized_accuracy": 93.1,
    "degradation_percentage": 0.3
  },
  "memory_savings": {
    "baseline_memory_mb": 4096,
    "optimized_memory_mb": 2048,
    "reduction_percentage": 50.0
  }
}

Performance

Evaluated on MNIST training - 3 epochs, RTX 3090 GPU:

Carbon Intensity Patterns (Mock Data):

Peak hours 18:00–22:00: ~450 gCO2/kWh
Off-peak hours 02:00–06:00: ~200 gCO2/kWh
Average reduction: 35–45% CO2 by scheduling during low-carbon windows

GPU Memory Savings:

Gradient accumulation 2x: ~30% memory reduction
Gradient accumulation 4x: ~50% memory reduction
Gradient accumulation 8x: ~60% memory reduction

Convergence Validation:

Accuracy degradation under 1% across all tested configurations
Loss convergence matches baseline within 2% tolerance
No divergence observed

Project Structure

CarbonAwareModelTraining---by-NEO/
├── src/
│   ├── scheduler.py                # Carbon intensity API & scheduling
│   ├── tracker.py                  # CodeCarbon emissions tracking
│   ├── train.py                    # Main training pipeline
│   └── utils.py                    # Config loading & logging
├── configs/
│   ├── baseline.yaml               # Baseline training config
│   └── optimized.yaml              # Carbon-aware optimized config
├── output/
│   ├── summary_baseline.json       # Baseline training summary
│   ├── summary_optimized.json      # Optimized training summary
│   ├── comparison_report.json      # Comparative analysis
│   ├── emissions.csv               # CodeCarbon emissions log
│   └── training_*.log              # Detailed training logs
├── models/
│   ├── model_baseline.pt           # Baseline model checkpoint
│   └── model_optimized.pt          # Optimized model checkpoint
├── data/                            # MNIST dataset (auto-downloaded)
├── requirements.txt                 # Python dependencies
├── generate_comparison.py          # Comparison report generator
└── README.md

Key Design Decisions

Why Carbon-Aware Scheduling?

Carbon intensity varies 2–5x throughout the day. Scheduling training during low-carbon windows reduces emissions without affecting model quality. Low-carbon periods also often correlate with cheaper electricity.

Why Gradient Accumulation?

Gradient accumulation enables training larger models on limited hardware by processing smaller micro-batches and updating weights less frequently. Used in BERT, GPT, and other large-scale models for the same reason.

Why CodeCarbon?

CodeCarbon uses lifecycle assessment methodologies, supports CPU, GPU, and multi-device setups, and produces transparent, community-validated calculations. It tracks energy, power, and emissions in a single library.

Why YAML Configuration?

YAML configs are version-controlled, human-readable, and separate code from experiment parameters - enabling reproducible A/B comparisons between baseline and optimized runs.

Testing

Validate installation:

python -c "import torch; print(f'PyTorch: {torch.__version__}')"
python -c "import codecarbon; print('CodeCarbon: OK')"
python -c "import yaml; print('PyYAML: OK')"
python -c "import torch; print(f'CUDA Available: {torch.cuda.is_available()}')"

Run a quick 5-minute test:

python src/train.py configs/test.yaml

Validate carbon savings:

python src/train.py configs/baseline.yaml
python src/train.py configs/optimized.yaml
python generate_comparison.py
cat output/comparison_report.json

Troubleshooting

CUDA Out of Memory

Reduce batch_size and increase gradient_accumulation_steps in the config.

Carbon Intensity API Timeout

No action needed - the pipeline automatically falls back to mock data and training proceeds.

Module Import Errors

export PYTHONPATH="$PWD/src:$PYTHONPATH"

CodeCarbon Tracking Fails

pip install --upgrade codecarbon

Training continues without emissions tracking if CodeCarbon fails.

Scheduler Waits Too Long

Increase max_wait_seconds, raise carbon_threshold, or set wait_for_low_carbon: false in the config.

How I Built This Using NEO

The requirement was a PyTorch training pipeline that schedules GPU workloads based on real-time carbon intensity, reduces memory footprint through gradient accumulation, and tracks emissions with CodeCarbon - producing a side-by-side comparison report. NEO built the full implementation: the carbon intensity scheduler in scheduler.py with API integration and mock fallback, the CodeCarbon emissions tracker in tracker.py, the main training pipeline in train.py with gradient accumulation and mixed precision FP16, the config loader and logging utilities in utils.py, the YAML configs for baseline and optimized runs, the comparison report generator in generate_comparison.py, and the full output structure covering JSON summaries, emissions CSV, and model checkpoints.

How You Can Use and Extend This With NEO

Use it to measure the carbon cost of your existing training runs.
Run python src/train.py configs/baseline.yaml on your own model and data by replacing SimpleCNN in src/train.py with your model. The CodeCarbon tracker produces a JSON summary with CO2 in kg, energy in kWh, and average power in Watts, a baseline measurement before any optimization.

Use the comparison report to justify scheduling infrastructure.
Run both the baseline and optimized configs on the same dataset. The comparison_report.json gives you a concrete before and after - percentage reduction in emissions, energy, and memory, alongside accuracy degradation, that makes the case for carbon-aware scheduling with real numbers from your own hardware.

Use mock data for development and real API for production.
Set use_mock_data: true during development so training always proceeds without waiting. Switch to use_mock_data: false with a real api_endpoint for production runs where actual carbon savings matter.

Extend the scheduler with additional carbon intensity sources.
The scheduler in scheduler.py fetches from a configurable api_endpoint. Adding support for additional regional carbon intensity APIs - Electricity Maps, WattTime, or a custom internal source, means updating the fetch logic in scheduler.py without touching the training loop, tracker, or reporting pipeline.

Final Notes

Carbon intensity varies throughout the day, and most training pipelines ignore it. A 43% reduction in CO2 emissions with less than 1% accuracy degradation, achieved by scheduling when the grid is cleaner and accumulating gradients to reduce memory - shows that sustainable ML is a practical engineering choice, not just an aspiration.

The code is at https://github.com/dakshjain-1616/CarbonAwareModelTraining
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Agentsync: Version, Merge, and Audit AI Agent Configurations Like Code

Nilofer 🚀 — Sat, 06 Jun 2026 05:34:45 +0000

Most AI engineering teams now run a stack of agent configs across many repos - model choices, tool allowlists, prompt templates, eval thresholds, safety rules. These configs drift the moment two engineers touch them. One repo gets a new policy, another keeps the old one, and nobody notices until an agent makes a decision in production that no one signed off on. Merging configs by hand is error-prone, and there is rarely an audit trail of what changed, when, or why.

Agentsync is a Node.js CLI tool that makes agent configuration something you can version, merge, and audit like code. Load JSON, YAML, or INI configs from any repo, three-way merge with conflict detection, run a 52-point compliance rubric on every change, and keep a full merge history you can revert. The point is that "which config is the source of truth for the agent in production?" should always have a clear, auditable answer.

Features

7 Core Commands - init, push, pull, diff, audit, status, revert
Smart Merging - three-way merge algorithm with automatic conflict detection, manual resolution support, and conflict tracking
Compliance Auditing - 52-point security and compliance rubric covering security, compliance, structure, performance, and documentation
Git Integration - seamless push and pull with git-based version control
Merge History - full audit trail with revert capability
Format Support - JSON, YAML, and INI configs

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         agentsync CLI                           │
├──────────────┬──────────────┬────────────┬───────────┬──────────┤
│ init         │ push         │ pull       │ diff      │ audit    │
│ Initialize   │ Push changes │ Merge      │ Compare   │ Validate │
│ repository   │ to remote    │ remote     │ configs   │ configs  │
└──────────────┴──────────────┴────────────┴───────────┴──────────┘
        │           │                 │
        └───────────┴─────────────────┘
                    │
        ┌───────────┴───────────┐
        │                       │
   ┌────▼─────┐        ┌─────────▼──┐
   │   Git    │        │   Config   │
   │ Manager  │        │   Loader   │
   └────┬─────┘        └─────┬──────┘
        │                    │
        │   ┌────────────────┘
        │   │
   ┌────▼───▼─────────┐
   │  Merge Engine    │
   │  - 3-way merge   │
   │  - Conflict Mgmt │
   └────┬─────────────┘
        │
   ┌────▼──────────────────┐
   │  Audit Engine         │
   │  - Security scoring   │
   │  - Compliance audit   │
   │  - 52-point rubric    │
   └───────────────────────┘

How It Works

The workflow follows a clear sequence. You initialize agentsync in your repository, which sets up local storage at ~/.agentsync/ and connects to a central git remote. From there:

Push - local config changes are committed and pushed to the remote with a message.

Pull - remote configs are fetched and merged into the local state using the three-way merge algorithm. Changes that only one side made are merged automatically. Conflicts - where both sides changed the same key - are surfaced for resolution. Manual resolution mode (--manual) enables interactive conflict handling.

Diff - shows configuration differences between any two refs, letting you see what changed between versions before committing to a merge.

Audit - runs the 52-point compliance rubric against any config directory. The rubric checks security (hardcoded credentials, encryption, secrets), compliance (audit logs, access control, data retention), structure (proper hierarchy, no duplicates, versioning), performance (object sizes, caching, connection pooling), and documentation (comments, examples, change logs). Every config gets a score from 0 to 100.

Revert - restores configuration from any point in the merge history. Every merge is stored as a timestamped JSON file in ~/.agentsync/history/.

Installation

npm install

Requires Node.js 16+. Git integration expects a repository with a remote named origin and a default branch of main with write access.

Usage

Initialize

agentsync init -r https://github.com/org/configs

Push Changes

agentsync push -m "Update API configs"
agentsync push --directory ./configs

Pull and Merge

agentsync pull
agentsync pull --manual  # Interactive conflict resolution

View Differences

agentsync diff --from HEAD~1 --to HEAD

Run Audit

agentsync audit --directory ./configs
agentsync audit --directory ./configs --report  # Generate report

Check Status

agentsync status

Restore from History

agentsync revert                    # List recent merges
agentsync revert 2026-05-13T12:30   # Revert to specific merge

Results and Output

Status Output

=== Git Status ===
Branch: main
Modified files: 2
Untracked files: 0

=== Agentsync Config ===
Initialized: true
Version: 1.0.0
Repository: https://github.com/dakshjain-1616/agentsync-configs

=== Merge History ===
- 2026-05-13T12:30:45.123Z: Update configurations
- 2026-05-13T12:25:30.456Z: Sync team configs
- 2026-05-13T12:20:15.789Z: Initial merge

Audit Report Output

=== AUDIT RESULTS ===

config.json: 95/100
  - Missing version specification
  - Potential hardcoded credentials detected

api-config.yaml: 88/100
  - Config not properly documented
  - Missing compliance metadata

Merge Report Example

# Merge Report

**Date:** 2026-05-13T12:30:45Z
**Message:** Update API configurations

## Merged Configurations

- api-keys.json
- database.yaml
- cache-config.json (⚠️ CONFLICT)

Compliance Scoring

Configs are scored from 0 to 100:
100 - perfect configuration
75–99 - minor issues
50–74 - moderate concerns
< 50 - serious compliance issues

Common violations that trigger score deductions:

Hardcoded API keys or passwords
Missing version specification
Improper config structure

Key Capabilities

3-Way Merge - Intelligent conflict detection. Changes on only one side merge automatically.
52-Point Audit - Catches security issues: hardcoded credentials, missing encryption, compliance gaps.
Format Support - Works with JSON, YAML, and INI configs seamlessly.
Full History - Complete audit trail - who changed what and when.
Revert Support - Roll back to any previous state instantly.

Comparison

When to Use agentsync

Perfect for:

Distributed AI engineering teams
Multi-stage deployment pipelines
Compliance-heavy organizations
Configuration-driven microservices

Not ideal for:

Single-person projects (use git directly)
Non-text binary configs
Real-time streaming configs

Configuration Formats

JSON:

{
  "apiKey": "...",
  "version": "1.0.0"
}

YAML:

apiKey: "..."
version: "1.0.0"

INI:

[database]
host=localhost
port=5432

Data Storage

Local data stored in ~/.agentsync/:

~/.agentsync/
├── config/              # Saved configurations
│   └── agentsync.json
└── history/             # Merge audit trail
    └── {timestamp}.json

Performance

Config parsing - O(n) where n = file size
3-way merge - O(k) where k = number of keys
Audit scoring - O(m) where m = config size
Typical operation - under 100ms

Project Structure

src/
├── index.js                    # CLI entry point
├── modules/
│   ├── errors.js              # Custom error classes
│   ├── logger.js              # Logging utility
│   ├── config-loader.js       # Load configs (JSON, YAML, INI)
│   ├── config-writer.js       # Write configs with backup
│   ├── git-manager.js         # Git operations
│   ├── local-storage.js       # ~/.agentsync persistence
│   ├── merge-engine.js        # 3-way merge algorithm
│   ├── merge-history.js       # Merge audit trail
│   ├── audit-engine.js        # Compliance scoring
│   └── report-generator.js    # Report generation
└── commands/
    ├── init.js
    ├── push.js
    ├── pull.js
    ├── diff.js
    ├── audit.js
    ├── status.js
    └── revert.js

Error Handling

Custom error types handle every failure mode cleanly:
AgentsyncError - base error class
ConfigError - config file issues
GitError - git operation failures
MergeError - merge conflicts or invalid operations

Limitations

Single branch syncing (main only)
No binary file support (text configs only)
Conflict resolution is text-based only

Testing

npm test

30 tests covering all core modules:

Error handling - 4 tests
Logging - 3 tests
Config loading and writing - 12 tests
Merge engine - 6 tests
Audit engine - 5 tests

All code is test-driven - write test first, implement to pass, refactor for clarity.

Contributing

All code is test-driven:

Write test first
Implement to pass test
Refactor for clarity

How I Built This Using NEO

The requirement was a CLI tool for synchronizing AI team configurations across repositories - with three-way merge, a 52-point compliance audit, git integration, merge history, and revert capability, all supporting JSON, YAML, and INI formats. NEO built the full implementation: the CLI entry point, all seven command modules, the merge engine with three-way merge and conflict management, the audit engine with the 52-point rubric, the config loader and writer, the git manager via simple-git, the local storage layer at ~/.agentsync/, the merge history tracker, the report generator, and the 30-test test suite covering all core modules.

How You Can Use and Extend This With NEO

Use it to enforce compliance before configs reach production.
Run agentsync audit --directory ./configs --report as part of your deployment pipeline. Any config scoring below your threshold fails the pipeline before it can introduce hardcoded credentials or compliance gaps into production.

Use the merge history as a compliance audit trail.
Every merge is stored as a timestamped JSON file in ~/.agentsync/history/. For teams with compliance requirements, this gives you a complete record of what changed, when, and under what commit message - queryable and revertable at any point.

Use revert to recover from bad merges instantly.
When a config change causes unexpected agent behavior, agentsync revert 2026-05-13T12:30 restores the full config state to any point in history. No manual git archaeology needed.

Extend it with additional compliance checks.
The audit engine in audit-engine.js implements the 52-point rubric. New compliance checks for domain-specific requirements follow the same scoring pattern and surface automatically in audit reports and scores.

Final Notes

Agent configuration drift is a silent production risk. agentsync makes it manageable by treating configs the way engineers already treat code - versioned, merged with conflict detection, audited for compliance, and fully revertable. The 52-point rubric catches what manual review misses. The merge history means there is always a clear answer to "what is the source of truth?"

The code is at https://github.com/dakshjain-1616/agentsync
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

CostGuard: A Real-Time Circuit Breaker That Stops AI Spend Before It Gets Out of Control

Nilofer 🚀 — Thu, 04 Jun 2026 11:24:25 +0000

AI API costs can spiral faster than anyone expects. A runaway loop, a misconfigured batch job, or a forgotten test that fires thousands of requests - by the time you see the bill, the damage is done.

CostGuard is a production-ready local proxy that enforces hard spending limits before AI API requests are sent. It sits between your applications and AI providers - OpenAI, Anthropic, and OpenRouter calculating the cost of every request before it goes out, and blocking it if any limit would be exceeded. Per-session, per-hour, per-day, and per-project circuit breakers, a real-time terminal dashboard, and multi-channel alerts, all running locally with no data leaving your machine.

Features

Hard Circuit Breakers - Per-session, per-hour, per-day, and per-project spending limits.
Real-Time Cost Estimation - Pre-call cost calculation using tiktoken before the request is sent.
Safe Mode - Require explicit confirmation for expensive requests above a configurable threshold.
Real-Time Dashboard - Terminal-based dashboard with WebSocket updates.
Multi-Channel Alerts - Console, webhook, and file-based alerting.
OpenAI-Compatible API - Drop-in replacement for the OpenAI SDK.
Local SQLite - All data stays on your machine.
Async Architecture - High-performance concurrent request handling.

Architecture

Client SDKs hit the OpenAI-compatible FastAPI proxy. The cost estimator pre-prices the request, then the circuit breaker evaluates limits in order: session, then hour, then day, then project. Allowed traffic forwards to the provider. Tripped limits return a 429 and fire alerts. Spend and pricing data live in local SQLite, and the terminal dashboard streams over WebSocket.

Quick Start

Installation

Clone the repository, create and activate a virtual environment, and install:

pip install -e ".[dev]"

Configuration

cp .env.example .env

Add at least one provider API key and any optional budget overrides - session, hour, day, project, or safe-mode thresholds.

Running the Server

costguard server

Or with uvicorn directly:

uvicorn costguard.server:create_app --factory --reload --port 8000

Using the Proxy

Point your OpenAI SDK at http://localhost:8000/v1, keep the provider API key in the client, and send the usual chat-completions request with session and project headers.

Real-Time Dashboard

costguard dashboard

Run this in a separate terminal. Set COSTGUARD_SESSION_ID=my-session before launching to scope the dashboard to a specific session.

API Endpoints

OpenAI-Compatible Endpoints

POST /v1/chat/completions - chat completions with cost tracking
GET /v1/models - list available models with pricing

CostGuard-Specific Endpoints

POST /v1/estimate - get cost estimate without making a request
GET /v1/status/{session_id} - get circuit breaker status
POST /v1/safe-mode/confirm - confirm a paused safe mode request
GET /health - health check

WebSocket

WS /v1/dashboard/ws - real-time dashboard updates

Circuit Breaker Behavior

Limits are evaluated in this deterministic order:

Session Limit - most restrictive, resets on new session
Hour Limit - rolling 1-hour window
Day Limit - resets at midnight UTC
Project Limit - least restrictive, tracks all-time project spend

When any limit is exceeded, the request is blocked with a structured error, an alert fires immediately, the circuit breaker status changes to OPEN, and subsequent requests are blocked until the limit resets.

Safe Mode

When a request's estimated cost exceeds COSTGUARD_SAFE_MODE_THRESHOLD, the request is paused and an alert is sent to configured channels. Confirm the request with POST /v1/safe-mode/confirm - the original request proceeds if confirmed.

Configuration Reference

Development

Running Tests

pytest                                        # Full suite
pytest --cov=costguard --cov-report=html      # With coverage
pytest tests/test_circuit_breaker.py          # Focused run

Code Quality

ruff format src tests                                              # Formatting
ruff check src tests                                               # Linting
mypy src/costguard                                                 # Type checking
ruff format --check src tests && ruff check src tests && mypy src/costguard  # Full gate

How I Built This Using NEO

The requirement was a local circuit-breaker proxy for AI spend control - one that estimates request cost before sending it, enforces session, hour, day, and project limits, supports safe mode for expensive requests, and exposes an OpenAI-compatible API so existing SDKs work without changes. NEO built the full implementation: the FastAPI proxy server with OpenAI-compatible endpoints, the tiktoken-based pre-call cost estimator, the circuit breaker with four limit tiers evaluated in deterministic order, the safe mode flow with confirmation endpoint, the multi-channel alert system covering console, webhook, and file, the terminal dashboard streaming over WebSocket, the local SQLite persistence layer, the pricing tables for OpenAI, Anthropic, and OpenRouter, and the full test suite.

How You Can Use and Extend This With NEO

Use it to protect any AI application from runaway costs.
Point your OpenAI SDK at http://localhost:8000/v1. Every request is pre-priced and checked against your configured limits before it leaves your machine. A misconfigured loop or an unexpected spike in usage trips the circuit breaker and fires an alert before the billing damage reaches your provider.

Use safe mode for high-stakes production requests.
Set COSTGUARD_SAFE_MODE_THRESHOLD to the cost above which you want human confirmation. Expensive requests are paused and alerted before proceeding. This is particularly useful for batch jobs or agent workflows where a single request can be unexpectedly large.

Use the estimate endpoint to build cost-aware UIs.
POST /v1/estimate returns the cost of a request without sending it. This lets you show users the expected cost of a query before they submit it or build dashboards that surface real-time spend across sessions and projects.

Extend it with additional model pricing.
The pricing tables cover OpenAI, Anthropic, and OpenRouter. Custom pricing can be added via PricingManager(custom_pricing_file=...). Any model not yet in the built-in tables can be priced by adding it to a JSON file - no code changes required.

Final Notes

AI API costs are easy to lose track of and expensive to discover late. CostGuard enforces limits before requests go out, not after the bill arrives. Pre-call cost estimation, four-tier circuit breaking, safe mode for expensive requests, and a real-time dashboard all running locally with no data leaving your machine.

The code is at https://github.com/dakshjain-1616/cost-Guard
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

ArchGuard: Detect Architecture Drift Before It Becomes Technical Debt

Nilofer 🚀 — Tue, 02 Jun 2026 09:44:09 +0000

Architecture degrades gradually. A circular dependency here, a god class there, a controller reaching directly into the database layer. Each violation is small on its own. Over time they compound into a codebase that is expensive to change and expensive to understand.

Most teams discover this in retrospect when a refactor takes three times as long as expected, or when a seemingly isolated change breaks something unrelated. By then the drift is already embedded.

ArchGuard is a production-ready Python static analysis tool that detects architecture degradation patterns in codebases over time. It runs six built-in detectors, compares architecture health between branches, tracks drift over the last 10 commits, and integrates into CI/CD through a GitHub Action or git hooks - all without any AI model dependency, using deterministic local static analysis.

Features

6 Built-in Detectors - circular dependencies, god classes, service layer bypasses, magic values, cyclomatic complexity, and layer violations.
Per-PR Analysis - compare architecture health between branches to catch regressions before they merge.
Trend Analysis - track architecture health over the last 10 commits to see drift over time.
Multiple Output Formats - table, JSON, YAML, Markdown, and HTML.
CLI and Git Hooks - command-line tool with pre-commit and pre-push hooks.
GitHub Action - CI/CD integration for automated architecture checks.
YAML Configuration - flexible, project-specific configuration via .archguard.yml.

Architecture

The CLI and YAML config feed the core engine - an AST parser, dependency graph, and base analyzer which fans out to six detectors. Findings are graded by severity, rendered as Table, JSON, YAML, Markdown, or HTML, and delivered through the CLI, git hooks, or the GitHub Action.

Installation

From PyPI

pip install archguard

From Source

git clone https://github.com/dakshjain-1616/Arch-Guard
cd archguard
pip install -e ".[dev]"

Requires Python 3.10+.

Quick Start

Initialize a configuration file in the project root:

archguard init

Scan the current tree or point it at a specific path:

archguard scan
archguard scan ./src

For machine-readable results:

archguard scan --format json --output report.json

Review architecture drift over the last 10 commits:

archguard trend

CLI Commands

scan - Analyze Codebase

archguard scan [PATH] [OPTIONS]

Key flags: --format, --output, --detectors, --severity, --fail-on-violations. Global flags: --config, --verbose.

trend - Analyze Trends

archguard trend [OPTIONS]

Flags: --commits, --format, --output.

init - Create Configuration

archguard init [OPTIONS]

--path selects the config file location.

config - Manage Configuration

archguard config                          # Show active configuration
archguard config output_format            # Read a value
archguard config output_format json       # Update a value

The Six Detectors

Circular Dependency
Detects circular import dependencies between modules.
min_cycle_length - minimum cycle length to report, default 2
max_cycles - maximum cycles to report, default 100

God Class
Detects classes with too many methods, attributes, or lines.
max_methods - maximum methods per class, default 20
max_attributes - maximum attributes per class, default 15
max_lines - maximum lines per class, default 500

Service Layer Bypass
Detects when controller or presentation layers bypass service layers to access repositories directly.
controller_patterns - regex patterns for controller files
service_patterns - regex patterns for service files
repository_patterns - regex patterns for repository files

Magic Value
Detects hardcoded literals that should be named constants.
min_string_length - minimum string length to flag, default 3
max_string_length - maximum string length to check, default 100

Cyclomatic Complexity
Detects functions and methods with high cyclomatic complexity.
thresholds - complexity thresholds for each severity level

Layer Violation
Detects violations of layered architecture, such as the presentation layer importing from the repository layer.
layers - layer definitions with patterns and allowed calls

Configuration

Create a .archguard.yml file in your project root. The config supports project metadata, include and exclude patterns, and per-detector options such as cycle length, maximum class size, and complexity thresholds. Output behavior, Git integration, and trend analysis are all controlled through the same file.

Git Hooks

Installation

python hooks/install.py                        # Install pre-commit hook
python hooks/install.py --pre-commit --pre-push  # Install both hooks
python hooks/install.py --force                # Overwrite existing hooks
python hooks/install.py --uninstall            # Remove hooks

Pre-commit hook - runs ArchGuard on staged Python files before committing.
Pre-push hook - runs trend analysis before pushing to remote.

GitHub Action

The GitHub Action integrates ArchGuard into CI/CD pipelines. Basic usage runs on push or pull request workflows, checks out the repository with full history, and passes path, format, severity, and fail-on-violations settings as action inputs. Advanced configuration enables trend mode, selects Markdown output, sets the commit window, and uploads the generated report as an artifact.

Acknowledgments

Built with Click for CLI, Python's built-in ast module for AST parsing, NetworkX for dependency graph analysis, Rich for terminal output, and GitPython for Git integration.

Development

Setup

git clone https://github.com/dakshjain-1616/Arch-Guard
cd archguard
pip install -e ".[dev]"
pre-commit install

Running Tests

pytest                                                        # Full suite
pytest --cov=src/archguard --cov-report=html                 # With coverage
pytest tests/unit/test_detectors.py                          # Targeted detector check
pytest tests/integration/                                    # Integration coverage

Code Quality

ruff check src/ tests/               # Linting
ruff check --fix src/ tests/         # Auto-fix
pyright src/                         # Type checking
ruff check src/ tests/ && pyright src/  # Combined gate

Contributing

Fork the repository. Create a feature branch:

git checkout -b feature/amazing-feature

Make your changes, run tests with pytest, run linting with ruff check src/ tests/, commit, push to the branch, and open a Pull Request.

How I Built This Using NEO

The requirement was a production-ready static analysis tool that detects architecture drift in Python codebases over time - with six built-in detectors, trend analysis over git history, multiple output formats, git hook integration, and a GitHub Action for CI/CD. NEO built the full implementation: the core engine with AST parser, dependency graph via NetworkX, and base analyzer; all six detector modules; the formatter layer covering table, JSON, YAML, Markdown, and HTML output; the git integration via GitPython; the CLI built on Click; the YAML configuration layer; the git hook installer and pre-commit and pre-push hooks; the GitHub Action; and the full test suite covering unit and integration tests.

How You Can Use and Extend This With NEO

Use it as a quality gate in every pull request.
Add the GitHub Action to your workflow with --fail-on-violations and the severity threshold you care about. Every PR gets checked for new circular dependencies, god classes, layer violations, and complexity regressions before it merges automatically, without any manual review step.

Use trend analysis to measure the health of an inherited codebase.
Run archguard trend on a codebase you have just taken over. The last 10 commits give you a picture of whether the architecture is improving or degrading, and which detectors are firing most frequently - useful context before making any changes.

Use git hooks to enforce standards locally before code reaches CI.
Install the pre-commit hook with python hooks/install.py. Staged files are checked on every commit. The pre-push hook runs trend analysis before anything reaches the remote. Issues are caught at the developer's machine, not in CI.

Extend it with additional detectors.
The six detectors share a common base analyzer interface. A new detector for a project-specific architecture rule follows the same pattern - implement the detection logic, add per-detector configuration to .archguard.yml, and register it. It appears automatically in scan output, trend analysis, and all output formats.

Final Notes

Architecture drift is invisible until it is expensive. ArchGuard makes it visible at every commit, every PR, and every push - with deterministic static analysis that requires no API keys, no model downloads, and no network calls. Six detectors, trend tracking over git history, and CI/CD integration in one tool.

The code is at https://github.com/dakshjain-1616/Arch-Guard
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Prepush-Guardian: Catch Secrets and Broken Tests Before They Reach Git History

Nilofer 🚀 — Mon, 01 Jun 2026 12:13:22 +0000

You are about to push. There is a hardcoded API key buried in one of 30 changed files. Or you forgot to write a test for that new module. Or the test suite is silently failing. You will not know until it is already in git history.

Prepush-Guardian catches all of this before the push lands. It is a production-grade Git pre-push hook that scans staged files for secrets, auto-generates missing tests, runs your full test suite, and blocks the push if anything fails before it ever reaches the remote.

Why This Tool

Manual review - Misses things, does not scale, no enforcement
CI/CD only - Finds it after the push, already in history
prepush-guardian - Blocked at push time, before it ever reaches remote

Scans every staged file for 20+ secret patterns: AWS, GitHub PATs, private keys, database URLs, bearer tokens, and more
Shannon entropy scanner catches novel secrets not matched by patterns
Auto-generates missing tests using OpenRouter AI, with a template fallback if no API key is set
Runs your full test suite and blocks the push if coverage drops below threshold
Writes a markdown report at .neo/prepush-report.md for every push

Quick Start

# Clone and install the hook into your repo
git clone https://github.com/neo-ai/prepush-guardian
cd your-target-repo

# Install the pre-push hook
python3 /path/to/prepush-guardian/install.py

# Optional: set API key for AI test generation
cp .env.example .env   # fill in OPENROUTER_API_KEY

The hook runs automatically on every git push. To run manually:

python3 prepush_guardian.py

Environment Variables

cp .env.example .env
# Required only for AI-based test generation
# Free key at: https://openrouter.ai/keys
OPENROUTER_API_KEY=your_openrouter_api_key_here

Without an API key, the tool falls back to template-based test generation.

Commands

Detection Patterns

The secret scanner covers 20+ patterns across four severity levels:

The Shannon entropy scanner runs alongside the pattern matcher. It catches novel secrets - API keys or tokens not yet covered by a named pattern by flagging high-entropy strings assigned to variables named KEY, TOKEN, or SECRET.

Scoring and Thresholds

Configuration

Create .neo/config.json to customize behavior. It is auto-created with defaults if absent:
coverage_warn_threshold - default 70. Warn if coverage drops below this percentage.
coverage_block_threshold - default 50. Block push if coverage drops below this percentage.
block_on_low_severity - default false. Also hard-block on LOW findings.
auto_fix_gitignore - default true. Add sensitive filenames to .gitignore automatically.
generate_missing_tests - default true. Auto-generate tests for untested source files.
skip_test_check_for - default ["migrations/", "scripts/", "docs/"]. Directories excluded from test generation.

Exit Codes

0 : All checks passed - push proceeding
1 : Push blocked - CRITICAL/HIGH findings or test failures

File Structure

prepush-guardian/
├── prepush_guardian.py      # Main orchestrator
├── leak_detector.py         # Phase 1: secret & entropy detection
├── test_generator.py        # Phase 2: AI test generation
├── test_runner.py           # Phase 2: test execution + coverage
├── reporter.py              # Phase 3: markdown report
├── install.py               # Hook installer
├── requirements.txt
├── .env.example
├── .gitignore
├── LICENSE
├── CONTRIBUTING.md
├── architecture.excalidraw
├── infographic.svg
└── tests/
    ├── test_leak_detector.py
    └── fixtures/
        ├── sample_with_secrets.py
        └── sample_clean.py

The three-phase structure maps cleanly to the file names - leak_detector.py handles Phase 1, test_generator.py and test_runner.py handle Phase 2, and reporter.py handles Phase 3. prepush_guardian.py orchestrates all three phases in sequence.

How I Built This Using NEO

The requirement was a production-grade Git pre-push hook that catches secrets, validates test coverage, and auto-generates missing tests - blocking the push before anything problematic reaches the remote. NEO planned, wrote, tested, and verified every file in this repository without human intervention: the main orchestrator in prepush_guardian.py, the secret and entropy scanner in leak_detector.py covering 20+ patterns, the AI test generator in test_generator.py with OpenRouter integration and template fallback, the test runner and coverage checker in test_runner.py, the markdown report generator in reporter.py, the hook installer in install.py, and the test suite with fixtures.

How You Can Use and Extend This With NEO

Install it into every repo your team pushes from.
Run python3 install.py once in each repository. From that point, every git push runs the full three-phase check automatically, no CI changes, no developer workflow changes. Secrets and test failures are blocked before they reach the remote.

Tune the thresholds to match your team's standards.
The .neo/config.json file controls coverage warn and block thresholds, whether LOW-severity findings hard-block the push, and which directories are excluded from test generation. These can be committed to the repo so the same standards apply across the whole team.

Use the markdown report as a push audit trail.
Every push writes a report to .neo/prepush-report.md.This gives you a record of what was scanned, what was found, and what was blocked, useful for teams with compliance requirements or for debugging why a push was blocked.

Extend the detection patterns in leak_detector.py.
The secret scanner covers 20+ named patterns. Adding a new pattern for a domain-specific secret type means adding it to the pattern list in leak_detector.py. It is immediately active on the next push with no other changes needed.

Final Notes

The gap between "I think this is clean" and "I know this is clean" is where prepush-guardian lives. Secrets get committed because no one checked. Tests go missing because there was no enforcement. prepush-guardian closes both gaps at the moment they matter most before the push lands.
The code is at https://github.com/dakshjain-1616/prepush-guardian
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Fine-Tuning Qwen2.5-0.5B to Write SRE Post-Mortem Summaries

Nilofer 🚀 — Sat, 30 May 2026 04:43:37 +0000

Writing post-mortem root-cause summaries is time-consuming and inconsistent. Junior SREs miss contributing factors. Senior SREs write summaries that vary in depth and structure. Zero-shot LLMs produce verbose, generic output that does not follow SRE conventions.
Fine-tuning a small model on real incident data produces structured, concise summaries that follow your organisation's format at a fraction of the cost of a large model.

Why This Approach

Diffrent type of approaches and what you get:

Manual SRE writing : Inconsistent, time-consuming, expertise-dependent
Zero-shot large model : Generic format, verbose, high cost per call
Qwen2.5-0.5B fine-tuned : SRE-format outputs, fast, cheap, runs on CPU or consumer GPU

The key advantages of this approach:

700-sample training set of real incident timelines mapped to root-cause summaries
4-bit quantized LoRA training, runs on a single consumer GPU with 8GB VRAM or more
Evaluated against a structured rubric covering timeline reference, contributing factors, specific component, and prevention action
Compared against qwen3.6-plus:free and gpt-5.4-nano baselines

The HuggingFace Model

The fine-tuned adapter is published at: daksh-neo/postmortem-qwen2.5-0.5b-lora

After training, the LoRA weights are saved to models/postmortem-lora/hf_export/ and pushed to HuggingFace.

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # fill in OPENROUTER_API_KEY
export $(cat .env | xargs)

Environment Variables

cp .env.example .env
# Required for baseline evaluation with OpenRouter
OPENROUTER_API_KEY=your_openrouter_api_key_here

OPENROUTER_API_KEY is required only for running baseline evaluations against zero-shot models via OpenRouter. The fine-tuning and local evaluation steps run without it.

Pipeline

The full pipeline runs in four steps:

Each step is independent, you can run baseline evaluation before fine-tuning to establish the gap the fine-tuned model closes, and run evaluation again after to measure the improvement.

Model Configuration

Evaluation Rubric

Every generated summary is scored against a four-criterion rubric. Each criterion carries equal weight:

Pass threshold: 0.60 weighted score or above.

Expected Results

qwen/qwen3.6-plus:free (zero-shot) - 20–35%
openai/gpt-5.4-nano (zero-shot) - 35–50%
Qwen2.5-0.5B (fine-tuned, 3 epochs) - > 60%

The fine-tuned 0.5B model outperforms both zero-shot baselines on rubric compliance because it has been trained specifically on the output format the rubric measures, not on general-purpose tasks.

File Structure

ml_project_0901/
├── scrape_postmortems.py    # Data collection
├── baseline.py              # Zero-shot baseline via OpenRouter
├── finetune.py              # LoRA fine-tuning
├── eval.py                  # Evaluation + comparison
├── requirements.txt
├── .env.example
├── .gitignore
├── LICENSE
├── CONTRIBUTING.md
├── architecture.excalidraw
├── infographic.svg
├── data/
│   ├── train.jsonl          # 700 training examples
│   ├── test_100.jsonl       # 100 held-out test examples
│   ├── rubric.json          # Scoring rubric
│   └── baseline_results.jsonl
└── models/
    └── postmortem-lora/
        └── hf_export/       # Push to HuggingFace after training

How I Built This Using NEO

The requirement was a complete fine-tuning pipeline for a small model on SRE post-mortem data, with data scraping, zero-shot baseline comparison, 4-bit LoRA fine-tuning, and structured rubric-based evaluation. NEO planned, wrote, tested, and verified every file in the repository without human intervention: the data scraper producing 700 training examples and 100 held-out test examples, the baseline evaluator running zero-shot prompts against OpenRouter models, the LoRA fine-tuning script with the full model configuration, the rubric-based evaluator producing the comparison table, and the HuggingFace export pipeline pushing the trained adapter to daksh-neo/postmortem-qwen2.5-0.5b-lora.

How You Can Use and Extend This With NEO

Use it to replace inconsistent manual post-mortem writing in your team.
Train on your own organisation's incident data by replacing data/train.jsonl with your own incident timeline to root-cause summary pairs. The rubric in data/rubric.json can be adapted to match your org's specific post-mortem format the evaluation pipeline measures compliance against whatever criteria you define.

Use the baseline comparison to justify the fine-tuning investment.
Run python baseline.py before fine-tuning to measure what zero-shot models produce on your data. Run python eval.py after fine-tuning to see the improvement. The comparison table gives you a concrete before-and-after that makes the case for domain-specific fine-tuning over general-purpose models.

Use the published adapter directly without retraining.
The fine-tuned LoRA adapter is available at daksh-neo/postmortem-qwen2.5-0.5b-lora on HuggingFace. You can load it directly without running the training pipeline - useful for teams that want to evaluate the output before committing to their own fine-tuning run.

Extend it to other structured generation tasks.
The four-step pipeline - scrape, baseline, fine-tune, evaluate is domain-agnostic. Any task where structured output format matters more than general knowledge is a candidate: alert triage summaries, change request descriptions, deployment notes. Swap the training data and rubric criteria, and the rest of the pipeline runs unchanged.

Final Notes

Zero-shot large models produce verbose, generic post-mortem summaries that do not follow SRE conventions. A fine-tuned 0.5B model trained on 700 domain-specific examples outperforms them on every criterion of the rubric - timeline reference, contributing factors, specific component identification, and concrete prevention actions, while running on a consumer GPU and costing a fraction per call.

The code is at https://github.com/dakshjain-1616/postmortem-finetune
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Morph: AST-Level Refactoring Where the LLM Describes Intent, Not Code

Nilofer 🚀 — Sat, 23 May 2026 11:04:25 +0000

When an LLM generates source code for a refactor, the output is a diff a reviewer must read line by line and trust blindly. There is no way to know if the model missed a reference, broke an import, or introduced a subtle logic change without reading every line.

Morph takes a different approach. Instead of asking the LLM to generate code, it asks the LLM to describe what to change as a structured plan of typed operations - RenameSymbol, MoveFunction, ExtractModule, and more. A reviewer reads ten structured operations in seconds and knows exactly what will change, why, and in what order. The transformation engine then validates the plan against the real codebase dependency graph, applies each operation atomically using tree-sitter AST manipulation, runs the test suite to confirm correctness, and stages clean changes for review. Failed transformations roll back automatically.

The LLM's job is intent declaration, not code writing. Morph's engine handles everything else.

Why Typed Plans Beat Source Code Generation

When a refactoring is expressed as a typed plan, every operation is verifiable before it runs. The plan validator checks file existence, symbol existence, dependency conflicts, and operation conflicts against a real dependency graph. The transformer applies operations in dependency order. The verifier runs pytest after every apply - any failure triggers automatic rollback.

Source code generation has none of these guarantees. A typed plan does.

The Pipeline

A natural language goal enters the LLM Planner, which outputs a validated TransformationPlan. The Plan Validator checks file existence, symbol existence, dependency conflicts, and operation conflicts against a NetworkX dependency graph. The Transformer applies operations in dependency order using tree-sitter AST manipulation, creating a file backup first. The Verifier runs pytest - any failure triggers automatic rollback. Clean changes are handed off to the Staging Manager via GitPython and summarised in a Report.

Supported Operations

Each operation is a typed Pydantic model. The LLM populates the fields — Morph validates and executes.

How the Dependency Graph Works

Before validating any plan, Morph parses the entire codebase with tree-sitter and builds a NetworkX dependency graph. This graph is used to:

Detect files that import the symbol being moved or renamed
Sort operations so dependencies are updated before dependents
Warn when a move will cascade across downstream files
Prevent circular dependency introduction from module extraction

This is what makes Morph safe to run on real codebases - the plan is validated against the actual dependency structure before a single file is touched.

Rollback Guarantee

Every non-dry-run apply call snapshots all affected files before touching them. If pytest reports failures after transformation, Morph restores from the snapshot automatically. The workspace is always left in a clean, known-good state.

Live Results

A real dry-run against anthropic/claude-haiku-4-5 via OpenRouter - the LLM parsed a natural language rename goal and produced a validated RenameSymbol plan in under 5 seconds. Full output and reproduction steps are in RESULTS.md.

Installation

pip install -e .

For local inference, install Ollama and pull a model:

ollama pull gemma4:e4b

For cloud backends, set the relevant environment variable:
OPENROUTER_API_KEY - OpenRouter (recommended)
OPENAI_API_KEY - OpenAI
ANTHROPIC_API_KEY - Anthropic

Usage

Describe what you want in plain English. Morph figures out the operations:

morph refactor --goal "rename calculate_total to compute_total" ./src

Preview the plan without touching any files:

morph refactor --goal "extract validation logic into validate_input()" ./src --dry-run

Generate and save the plan for inspection before applying:

morph plan --goal "add type annotations to all functions in utils.py" ./src --output plan.json

Apply a saved plan:

morph refactor --plan plan.json ./src

Verify the codebase passes its own test suite:

morph verify ./src

Generate a Markdown report of the last run:

morph report ./src --format markdown --output REFACTOR_REPORT.md

Supported Models

Morph works with any provider. OpenRouter is the recommended starting point - one API key routes to every model below without separate accounts.

The planner uses temperature=0.1 - low randomness produces more consistent structured output. Unknown model strings are automatically routed through OpenRouter with no --backend flag required.

CLI Reference

morph refactor --goal "..." PATH - Generate plan from goal and apply it
morph refactor --plan FILE PATH - Apply a previously saved plan
morph refactor ... --dry-run - Show plan without modifying files
morph plan --goal "..." PATH - Generate and display plan only
morph verify PATH - Run the test suite and report pass/fail
morph report PATH - Generate Markdown/JSON report of last run

Key flags: --model, --backend, --dry-run, --no-rollback, --output

Development

Clone and install in editable mode with dev dependencies:

git clone https://github.com/dakshjain-1616/morph
cd morph
pip install -e ".[dev]"

Run the full test suite:

pytest tests/ -v

Lint and type-check:

ruff check morph/ && mypy morph/

How I Built This Using NEO

The requirement was a refactoring CLI where the LLM describes intent as a structured typed plan rather than generating raw code with AST-level execution, dependency graph validation, automatic rollback on test failure, and support for multiple LLM backends. NEO built the full implementation: the LLM Planner producing typed TransformationPlan outputs with temperature=0.1, the seven typed Pydantic operation models, the Plan Validator checking file existence, symbol existence, and dependency conflicts against a NetworkX graph, the Transformer applying operations in dependency order via tree-sitter AST manipulation with file backup, the Verifier running pytest with automatic snapshot rollback on failure, the Staging Manager via GitPython, the report generator, and the full CLI with all six commands and their key flags.

How You Can Use and Extend This With NEO

Use it to refactor production codebases safely.
Instead of asking an LLM to rewrite files, describe the refactoring goal in plain English. Morph validates the plan against the real dependency graph, applies it atomically, and rolls back automatically if tests fail. The dry-run mode lets you inspect exactly what will happen before anything is touched.

Use the saved plan workflow for team review.
Run morph plan --goal "..." --output plan.json to generate the structured plan without applying it. Share the JSON with your team for review before running the apply step. Reviewers see ten typed operations instead of a raw diff - faster to review, easier to reason about.

Use it as a refactoring step in CI/CD pipelines.
morph verify PATH runs the test suite and reports pass/fail with an exit code, making it composable as a CI step. Combined with morph refactor and --dry-run, you can build a pipeline that proposes, reviews, and applies refactors with automated test verification at every stage.

Extend it with additional operation types.
Each operation is a typed Pydantic model in the operations layer. A new operation follows the same pattern: define the Pydantic model, implement the transformer logic, and register it. The LLM Planner, Plan Validator, and CLI all pick it up automatically.

Final Notes

Morph shifts refactoring from code generation to intent declaration. The LLM describes what to change in a structured, validated plan. The engine does the mechanical work. Tests confirm correctness. The result is refactoring that is auditable before it runs, verifiable after it runs, and automatically reversible if it breaks anything.

The code is at https://github.com/dakshjain-1616/Morph
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

ToolRouter: Switch AI Coding Tools Freely Without Losing Context

Nilofer 🚀 — Fri, 22 May 2026 11:13:41 +0000

Every AI coding tool has its strengths. Claude Code is strong for complex multi-step tasks. Cursor is fast for inline edits. Gemini CLI is useful for quick questions. Most developers use more than one but every time you switch, the context is gone. The new tool has no idea what you just did, what you decided, or which files are in a partial state.

On top of that, there is no clear picture of what different AI tools actually cost per session, per project, or per week. You are guessing at efficiency rather than measuring it.

ToolRouter is a local proxy daemon that solves both problems. It maintains shared session state across multiple AI coding tools, generates Handoff Briefs when you switch between them, and tracks real token spend per tool and model all transparently, without changing your API keys or your tools.

How It Works

ToolRouter sits between your AI tools and their APIs as a local proxy on port 7863. Here is what happens at each stage:

Step 1 - All traffic routes through the proxy.
You point each AI tool's API base URL at localhost:7863. From that point, every request your tool makes passes through ToolRouter first. The proxy forwards it transparently to the real API, your API keys are unchanged, your tools behave exactly as before.

Step 2 - The proxy captures what matters as a side effect.
As AI responses come back through the proxy, ToolRouter reads the token counts and extracts decisions and task state from the response text using pattern matching. Statements like "let's use bcrypt" are classified as decisions. Lines like "implemented JWT validation" are classified as completed tasks. "Still need to finish the refresh logic" becomes an in-progress item. Everything is written to the SQLite state store.

Step 3 - The file tracker watches the filesystem independently.
Alongside the proxy, a Watchdog-based file tracker monitors your project directories. It computes file hashes before and after each session to build an accurate list of what changed. It also scans for syntax errors, merge conflict markers, and unresolved TODOs to detect files that are in a partial state.

Step 4 - When you switch tools, a Handoff Brief is generated and injected.
The Handoff Generator reads from the state store and assembles a brief - partial files first since they carry the highest risk, then in-progress tasks, then decisions and completed items. This brief is automatically injected into the first message of your new session. The receiving tool sees exactly where the last tool left off, before it writes a single line.

Step 5 - Spend is tracked on every proxied response.
Token counts from every response are accumulated and costed against current model pricing. No separate setup needed, spend tracking is a byproduct of the same proxy pass.

Installation

pip install -e .

Quick Start

Step 1 - Start the daemon

toolrouter start

This starts the proxy on port 7863 and the dashboard on port 7864.

Step 2 - Point your AI tools at the proxy
Each tool needs its API base URL pointed at the local proxy. This is a one-time configuration per tool:

Claude Code: export ANTHROPIC_API_URL=http://localhost:7863/v1
Cursor: Set OpenAI API base URL to http://localhost:7863/v1 in Settings → AI
Gemini CLI: export OPENAI_API_BASE=http://localhost:7863/v1
Ollama: export OLLAMA_HOST=http://localhost:7863/api

Step 3 - Work normally
Switch tools whenever you like. ToolRouter handles handoffs automatically.

Handoff Brief

When you switch tools on the same project, ToolRouter injects a brief like this into the first message of your new session:

[ToolRouter Handoff — from claude-code, 5 minutes ago]

Files changed this session:
✓ src/auth.py — implemented JWT token validation
✓ src/models.py — added User model
⚠ src/api.py — PARTIALLY MODIFIED, do not use as-is

Completed:
✓ Set up authentication middleware
✓ Created database schema

In progress:
→ Implementing refresh token logic
→ Writing API documentation

Decisions made:
- Using bcrypt for password hashing
- JWT tokens expire after 24 hours
- Refresh tokens stored in Redis

⚠ Do not touch:
- src/api.py (has syntax errors)

The brief is generated from the state store - real file changes tracked by the watchdog, decisions extracted from AI responses, and partial-state detection on modified files. The receiving tool sees this at the start of the session and can immediately continue where the last tool left off.

Spend Tracking

ToolRouter reads token counts from every proxied response and calculates cost using current model pricing. Spend reports run directly from the terminal:

toolrouter spend           # Today's report
toolrouter spend --week    # This week
toolrouter spend --month   # This month

The dashboard at http://localhost:7864 shows daily spend bar charts per tool, session lists with per-session cost, per-tool and per-project breakdowns, which tool is most cost-efficient measured by cost per file changed, and projected monthly costs based on current pace.

Model Pricing (May 2026)

Commands

Architecture

State Store - SQLite with WAL mode for concurrent read/write. Stores sessions, per-session file changes with MD5 hashes, extracted decisions and tasks, and generated handoff briefs. Every table links back to a session ID so the full history is queryable.

File Tracker - Watchdog-based monitoring of project directories. Computes file hashes before and after each session to build an accurate change list. Detects partial states by scanning for syntax errors, merge conflict markers, and unresolved TODOs.

Decision Extractor - Pattern matching over AI responses to classify statements into decisions, completed tasks, in-progress work, and blockers. Phrases like "let's use" and "we'll go with" are decisions. Words like "done", "implemented", and "✓" signal completed tasks. "I've started" and "still need to" mark in-progress work. "Blocked by" and "waiting for" identify blockers.

Handoff Generator - Assembles the brief from state store data, ordering by recency and priority: partial files first as they carry the highest risk, then in-progress tasks, then decisions and completed items.

Configuration

Configuration is stored at ~/.toolrouter/config.json. The key settings are:
injection_enabled - whether to prepend handoff briefs
proxy_port - default 7863
dashboard_port - default 7864
log_level - logging verbosity

Use toolrouter config set <key> <value> to change any setting without editing the file directly.

How I Built This Using NEO

The requirement was a local proxy daemon that could sit transparently between AI coding tools and their APIs, maintain shared session state across tool switches, generate structured handoff briefs automatically, and track real token spend per tool and model - all without requiring any changes to the tools themselves or the user's API keys.

NEO built the full implementation: the proxy daemon running on port 7863, the SQLite state store with WAL mode, the Watchdog-based file tracker with MD5 hashing and partial state detection, the pattern-matching decision extractor, the handoff brief generator with priority ordering, the spend tracker reading token counts from proxied responses, the dashboard on port 7864, and the full CLI with all twelve commands.

How You Can Use and Extend This With NEO

Use it to switch between Claude Code, Cursor, and Gemini CLI on the same project without losing context.
Point each tool at the proxy once, and every subsequent tool switch gets an automatic handoff brief. The receiving tool knows which files changed, which tasks are in progress, and which files should not be touched - without you writing a single summary.

Use the spend dashboard to measure which AI tool is most cost-efficient for your workflow.
The dashboard breaks down cost per tool, per project, and per session. The "cost per file changed" metric tells you which tool delivers the most work per dollar - a data-driven way to decide which tool to reach for on different task types.

Use the handoff brief preview before switching.
Run toolrouter handoff before switching tools to see exactly what brief the next tool will receive. This lets you verify the context is accurate before handing off on a complex task where a wrong assumption by the next tool could cause real damage.

Extend it with additional tool integrations.
The proxy currently supports Claude Code, Cursor, Gemini CLI, and Ollama via their respective API base URL environment variables. Any tool that accepts an OpenAI-compatible API base URL can be pointed at the proxy using the same pattern - no changes to ToolRouter needed.

Final Notes

ToolRouter makes multi-tool AI development practical. Context persists across tool switches through automatically generated handoff briefs. Spend is tracked in real time with model-accurate pricing. The proxy is transparent - your tools and API keys are unchanged.

The code is at https://github.com/dakshjain-1616/Tool-Router
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Context Time Machine: Forensic Investigation of What Your Agent Actually Saw

Nilofer 🚀 — Sat, 16 May 2026 11:10:19 +0000

Long-running agent sessions fail in a specific way that is hard to debug. The agent runs 40 turns. At turn 38, it gives a wrong answer that ignores something it decided at turn 12. You look at the logs, the turn 12 decision is there. The turn 38 response is there. But you cannot see what the context window looked like at turn 38. Was the turn 12 decision still in context? Was it evicted? Was it there but semantically overwhelmed by 25 other turns?

This is the forensic problem that ContextTimeMachine solves. It is different from real-time session monitoring, it is for deep post-hoc investigation of what happened during a session, after it has already run. The key insight it is built on: the context window at any given turn is deterministic given the conversation history. You can reconstruct exactly what the model saw at turn 38, render it interactively, and query it.

Three Investigation Modes

Mode 1 - Timeline Navigator

The primary view is a vertical timeline of all turns in the session. Each turn shows the turn number, agent name if available, turn type, token count at that turn, and a sparkline showing how the context composition changed.

Click any turn to travel to it - the context window at that exact point reconstructs and renders in the main panel. You see exactly what the model saw: every message in order, with token counts, with a red line showing where the context would have been truncated if it exceeded the model's limit. Scrub through turns with keyboard arrows. Watch the context window evolve turn by turn. See turns disappear as eviction happens. See tool results arrive and push older content further back.

Mode 2 - Fact Tracker

You know something specific, a decision made at turn 5, a fact retrieved at turn 15, a user instruction given at turn 3. You want to know: at what turn did this fact leave the context window?

Enter any text snippet in the Fact Tracker search box. ContextTimeMachine embeds it locally using sentence-transformers, then searches every turn's context snapshot for the nearest matching content. It renders a presence chart, a horizontal bar across all turns colored green when the fact is present or red when absent and shows the exact turn where the fact entered context and the exact turn where it left.

This answers the most common debugging question for long agent sessions: "When exactly did the agent stop knowing X?"

Mode 3 - Divergence Finder

You have two agent sessions that started identically but ended differently. One succeeded, one failed. Load both sessions and ContextTimeMachine finds the earliest turn where their context windows diverged where they started seeing different content and highlights that turn as the likely root cause of the different outcomes.

It shows a side-by-side comparison of the two context windows at the divergence point with diffed content highlighted. This is the automated version of the manual debugging process every team does when comparing "the run that worked" against "the run that didn't."

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    ContextTimeMachine                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Frontend (React)                                               │
│  ├─ TimelineNavigator    — Turn-by-turn timeline scrubber       │
│  ├─ ContextPanel         — Renders reconstructed context        │
│  ├─ FactTracker          — Fact presence chart                  │
│  └─ DivergenceFinder     — Two-session comparison               │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  FastAPI Backend                                                │
│  ├─ /api/session/load          — Load session from file         │
│  ├─ /api/session/{id}/profile  — Get token profile              │
│  ├─ /api/session/{id}/turn/{n} — Reconstruct context at turn    │
│  ├─ /api/session/{id}/fact     — Track fact presence            │
│  ├─ /api/divergence            — Find divergence point          │
│  └─ /api/sessions              — List all sessions              │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Core Analysis Modules                                          │
│  ├─ SessionLoader        — Load from multiple formats           │
│  ├─ ContextReconstructor — Reconstruct at any turn              │
│  ├─ FactTracker          — Track presence via embeddings        │
│  ├─ DivergenceFinder     — Find divergence points               │
│  ├─ TokenAnalyzer        — Token budget analysis                │
│  └─ EmbeddingService     — Local embeddings (all-MiniLM)        │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Storage                                                        │
│  └─ SQLite DB            — Session snapshots & metadata         │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Installation

Prerequisites

Python 3.10+
pip

Quick Start

# Clone the repository
git clone https://github.com/dakshjain-1616/context-time-machine.git
cd context-time-machine

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install package
pip install -e .

# Start the server
timemachine serve

Open http://localhost:8000 in your browser. The server will automatically open your browser if it can.

Usage

Loading Sessions
Sessions can be loaded from two formats.

From LiveContext SQLite Export:

timemachine load --file session.db

From Generic JSON:

timemachine load --file session.json

The generic JSON format expects a turns array where each turn contains a messages list, a model_id, and a timestamp:

{
  "turns": [
    {
      "turn": 0,
      "messages": [
        {"role": "system", "content": "You are helpful.", "token_count": 3},
        {"role": "user", "content": "What is 2+2?", "token_count": 4}
      ],
      "model_id": "gpt-4",
      "timestamp": "2026-05-09T10:00:00Z"
    }
  ],
  "model_id": "gpt-4"
}

CLI Commands
The CLI covers the full workflow from loading sessions to querying them:

# Start the web interface
timemachine serve

# Load a session
timemachine load --file session.json

# Track fact across session
timemachine fact --session <session-id> --fact "the user prefers JSON output"

# Find divergence between two sessions
timemachine diverge --session-a <id-a> --session-b <id-b>

# List all stored sessions
timemachine sessions

# Clear all sessions
timemachine clear

Python API

Every capability the CLI and web interface expose is also available as a Python library. This makes it straightforward to integrate ContextTimeMachine into evaluation pipelines or automated debugging scripts:

from context_time_machine import (
    SessionLoader,
    ContextReconstructor,
    FactTracker,
    DivergenceFinder,
    TokenAnalyzer,
)

# Load session
loader = SessionLoader()
session = loader.load("session.json")

# Reconstruct context at turn 10
reconstructor = ContextReconstructor()
context = reconstructor.reconstruct(session, turn_number=10)
print(f"Context at turn 10: {context.total_tokens} tokens")
print(f"Messages: {len(context.messages)}")
print(f"Utilization: {context.utilization_percent}%")

# Track a fact
tracker = FactTracker()
result = tracker.track(session, "specific decision from turn 5")
print(f"Fact first appeared: Turn {result.first_appeared_turn}")
print(f"Fact last present: Turn {result.last_present_turn}")
print(f"Disappeared at: Turn {result.disappeared_at_turn}")

# Analyze token budget
analyzer = TokenAnalyzer()
profile = analyzer.analyze_session(session)
print(f"Peak tokens: {profile.peak_tokens} at turn {profile.peak_turn}")
print(f"Eviction turns: {profile.eviction_turns}")

# Find divergence between sessions
session_b = loader.load("session_b.json")
finder = DivergenceFinder()
result = finder.find(session, session_b)
print(f"Divergence at turn: {result.divergence_turn}")
print(result.summary)

Supported Session Formats

How It Works

Context Reconstruction

For each turn N, ContextTimeMachine loads all messages from turns 0 to N and counts the total tokens using tiktoken. If the total exceeds the model's context limit, it simulates eviction using a model-specific strategy: GPT and Claude use left-truncation (oldest messages first), DeepSeek uses a sliding window with a recency bias, and Gemma uses local-global attention sampling from the middle. System messages are never evicted regardless of which strategy applies. The result is a reconstructed context with a full token breakdown exactly what the model would have seen at that turn.

Fact Tracking

For each turn, ContextTimeMachine embeds the fact text using all-MiniLM-L6-v2. It then computes cosine similarity between that embedding and every message in the turn's reconstructed context. A fact is considered present if any message has a similarity above 0.75. Embeddings are cached for performance so repeated queries against the same session do not recompute embeddings. The output is a presence chart showing the fact's full lifecycle across the session.

Divergence Detection

For two sessions, ContextTimeMachine aligns turns and analyzes up to the minimum length of the two sessions. At each turn it reconstructs the context for both sessions, embeds all messages, and computes an average maximum cosine similarity between the two context windows. When this similarity drops below 0.85, the turn is flagged as the divergence point. The output includes a message diff at the divergence point and a summary of what changed.

API Endpoints

Session Management

POST /api/session/load - load session from file or JSON
GET /api/sessions - list all stored sessions
DELETE /api/session/{id} - delete a session

Analysis

GET /api/session/{id}/profile - get token profile for session
GET /api/session/{id}/turn/{num} - reconstruct context at turn
POST /api/session/{id}/fact - track fact presence
POST /api/divergence - find divergence between sessions

Performance

Context Reconstruction: < 100ms for typical sessions
Fact Tracking: ~1-5 seconds for full session (includes embedding)
Divergence Detection: ~2-10 seconds for 2 sessions
Memory: ~50-200MB per stored session (depending on size)

Dependencies

Core

fastapi - Web framework
uvicorn - ASGI server
pydantic - Data validation
click - CLI framework
tiktoken - Token counting
sentence-transformers - Local embeddings
numpy - Numerical operations
sqlalchemy - Database ORM
aiofiles - Async file operations

Frontend
React, Tailwind CSS, Framer Motion, Recharts

Known Limitations

Frontend is a React stub - core analysis is fully functional
LangSmith format not yet implemented
No streaming support for very large sessions (>10k turns)
Embedding cache cleared on restart

Future Enhancements

Complete React frontend with real-time updates
WebSocket streaming for large sessions
LangSmith format support
Multi-session comparison UI
Export to markdown/HTML
Attention visualization
Custom eviction strategy support

How I Built This Using NEO

The requirement was a forensic debugging tool for long-running agent sessions, one that could reconstruct the exact context window at any historical turn, track when specific facts entered and left context using semantic embeddings, and find the earliest point where two divergent sessions started seeing different content. The tool needed to support multiple session formats, expose a Python API alongside the web interface, and work entirely offline with local embeddings.

NEO handled all 12 specification steps autonomously, building the SessionLoader with support for LiveContext SQLite, generic JSON, and raw conversation formats, the ContextReconstructor with model-specific eviction strategies for GPT, Claude, DeepSeek, and Gemma, the FactTracker with all-MiniLM-L6-v2 embeddings and cosine similarity scoring, the DivergenceFinder with turn-aligned context comparison, the TokenAnalyzer for peak token and eviction turn detection, the FastAPI backend with all six API endpoints, the SQLite storage layer via SQLAlchemy, the Click CLI with all six commands, and the full 58-test suite covering all core modules.

How You Can Use and Extend This With NEO

Use it to find the root cause of long-session failures.
When an agent gives a wrong answer deep into a long session, load the session into ContextTimeMachine, travel to the failure turn in the Timeline Navigator, and see exactly what was in context at that point. The reconstructed view shows every message the model saw, in order, with token counts, so you can see immediately whether the relevant context was present or had been evicted.

Use Fact Tracker to measure context retention across your agent design.
Before settling on a context management strategy for your agent, run Fact Tracker against a set of real sessions. The presence chart for key decisions and instructions tells you at what turn they reliably drop out of context giving you a data-driven basis for choosing context window sizes, eviction strategies, or compression approaches.

Use Divergence Finder to debug non-deterministic agent behaviour.
When two runs of the same agent with the same input produce different outcomes, load both into Divergence Finder. The tool identifies the exact turn where their context windows started differing and shows a diff of what changed, turning a difficult debugging problem into a specific, actionable finding.

Extend it with additional session format parsers.
SessionLoader already handles three formats following a common interface. Adding a new format - LangSmith is listed as planned, means implementing the same loader interface for the new format. It is then immediately available in the CLI, the Python API, and the web interface without touching any of the analysis modules.

Final Notes

ContextTimeMachine makes the context window visible. Instead of inferring what the model saw from its outputs, you can reconstruct and inspect the exact context at any turn, track when specific information entered and left the window, and find where two sessions diverged. For teams debugging long-running agents, that visibility is the difference between guessing and knowing.

The code is at https://github.com/dakshjain-1616/ContextTimeMachine
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code