DEV Community: Aman Sachan

I built a 81-tool, fully local AI desktop assistant with PySide6 and Ollama (here is the architecture)

Aman Sachan — Mon, 08 Jun 2026 01:39:25 +0000

Why a desktop app, not another chat UI

VS Code gives you an editor. Cursor gives you an editor + chat bolted on. Sentience is a different shape of animal: a native desktop window where the AI is the primary surface, 81 first-class tools hang off a ReAct loop, and the whole thing runs offline against Ollama with no telemetry, no extension marketplace, and no monthly bill.

It is about 6,200 lines of Python. The license is MIT. The window opens in under a second on a cold start.

The shape: a window, an agent, and a tool registry

[Sentience window (PySide6)]
  - Sidebar (file tree)
  - Editor (QScintilla, syntax highlight)
  - Chat panel (ReAct loop entry point)
  - Embedded terminal (QTermWidget subprocess)
        |
        v
  jarvis_agent.py
    - tool_registry[81 entries]
    - provider (Ollama / OpenAI / Anthropic / Groq)
    - memory (SQLite + sqlite-vec)

Three widgets in the main window: a file tree, a code editor, and a chat panel. The chat panel is the entry point to the agent. Below them, an embedded terminal you can shell into; the agent uses the same terminal internally when it runs shell-scoped tools.

The ReAct loop with strict tool schemas

The heart of the agent is a Reason -> Act -> Observe loop, but with a twist: every tool is a Pydantic model with a JSON schema the model has to fill. No free-form string parsing.

class ToolRegistry:
    def __init__(self):
        self.tools: Dict[str, Tool] = {}
        self.schemas: Dict[str, dict] = {}

    def register(self, name: str, fn: Callable, schema: type[BaseModel]):
        self.tools[name] = fn
        self.schemas[name] = schema.model_json_schema()

class ReadFileTool(BaseModel):
    path: str = Field(..., description="Absolute path to file")
    start_line: int = Field(0, ge=0)
    end_line: int = Field(-1, ge=-1)

@registry.register("read_file", ReadFileTool)
def read_file(path: str, start_line: int = 0, end_line: int = -1) -> str:
    p = Path(path)
    if not p.is_file():
        return f"ERROR: {path} not found"
    lines = p.read_text().splitlines()
    return "\n".join(lines[start_line: end_line or None])

The model is told about all 81 tools in a single system message. On each turn it must return a JSON object like {"thought": "...", "tool": "read_file", "args": {...}} or {"thought": "...", "final_answer": "..."}. We parse with pydantic.ValidationError caught and re-prompted, so malformed tool calls never crash the loop.

Multi-provider without a rewrite

The same loop works against Ollama, OpenAI, Anthropic, or Groq. The abstraction is one method:

class Provider(Protocol):
    def chat(self, messages: list, tools: list[dict]) -> ModelResponse: ...

class OllamaProvider:
    def chat(self, messages, tools):
        r = requests.post("http://localhost:11434/api/chat",
                          json={"model": "llama3.2:3b",
                                "messages": messages,
                                "format": tools}, timeout=120)
        return ModelResponse.parse_obj(r.json())

class OpenAIProvider:
    def chat(self, messages, tools):
        return self.client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            tools=[{"type": "function",
                    "function": {"name": t["name"],
                                 "parameters": t["schema"]}} for t in tools])

Same Provider protocol, different wire format. The agent does not care which model is on the other end; the JSON tool schema is the contract.

The self-modify tool, kept honest

There is a tool literally called edit_own_source. It is dangerous, so it is gated:

@registry.register("edit_own_source", EditOwnSourceTool)
def edit_own_source(file: str, old: str, new: str, confirm: bool) -> str:
    if not confirm:
        return "REFUSED: pass confirm=True only after showing the diff to the user"
    if not file.startswith(str(ROOT / "src")):
        return f"REFUSED: {file} is outside src/"
    p = Path(file)
    backup = p.with_suffix(p.suffix + f".bak.{int(time.time())}")
    shutil.copy(p, backup)
    p.write_text(p.read_text().replace(old, new))
    return f"OK; backup at {backup}"

Two guard rails. The first forces the model to show the diff to the user before flipping confirm=True. The second refuses any path outside src/. A backup is written before any write. This is the kind of thing that needs to be loud, not silent.

Why the terminal lives inside the agent

When the agent runs a shell command, it does not spawn a hidden subprocess; it pushes the command into the embedded terminal widget. The user can see exactly what is running, can Ctrl-C it, and can scroll back. The agent reads the output as if it were any other tool.

That single design choice killed a whole category of "agent ran rm -rf in the background" bugs. Visibility first, autonomy second.

Memory: SQLite + a dumb but fast vector

Persistent memory is a SQLite table with a vec0 virtual table for embeddings. No Pinecone, no server. Embeddings are computed locally with nomic-embed-text via Ollama. The remember tool writes a row; the recall tool does a cosine top-k and returns the joined text.

Stack

Layer	Tech
Window	PySide6 (Qt 6.5+)
Editor	QScintilla with LSP over stdin
Terminal	QTermWidget wrapping bash
Agent	Custom ReAct loop, Pydantic tool schemas
Providers	Ollama, OpenAI, Anthropic, Groq
Storage	SQLite + sqlite-vec
Packaging	PyInstaller (Windows .exe, Linux AppImage)

What is next

A proper streaming mode for the chat panel (right now it is chunky), a visual diff tool so edit_own_source is reviewable in-app, and a "skills" registry; small JSON manifests that bundle 3-5 tools into a named capability the model can opt into. Cursor and Claude have skills; Sentience should too.

Repo: github.com/AmSach/sentience
Run it: pip install pyside6 openai anthropic aiohttp requests && python sentience.py

If you find a tool the agent should have, open an issue with a Pydantic schema and a one-line description. I merge PRs in 24 hours.

Python #PySide6 #LocalAI #Ollama #OpenSource #BuildInPublic

Building a 3-second medicine verifier with Gemini vision + pg_trgm fuzzy matching

Aman Sachan — Mon, 08 Jun 2026 01:06:59 +0000

The problem

Indian households spend roughly ₹65,000 crore every year on out-of-pocket medicine costs. A meaningful slice of that is going toward branded drugs when the exact same molecule — same active ingredient, same dosage, same CDSCO approval — is sitting in a Jan Aushadhi store for a fraction of the price. Dolo-650 retails at ₹32 a strip; the Jan Aushadhi paracetamol is ₹4.90. The cheaper medicine isn't the issue. Nobody tells the patient it's there.

We built Agada to fix that. Snap a photo of the strip. Three seconds later you know if it's CDSCO-registered, what it actually does, and what the cheaper version costs. No login. No app install. Free.

Here's how the whole thing fits together.

The pipeline

[Phone camera] -> [client compression] -> [Gemini 1.5 Flash vision]
                                              |
                                              v (structured JSON: brand, molecule, dosage)
                                      [Supabase parallel queries]
                                      |--> CDSCO registry (300k+ drugs, pg_trgm GIN)
                                      |--> Jan Aushadhi product list
                                      |--> NPPA price ceiling
                                              |
                                              v
                                       [3 result cards]

Three things happen in parallel once the image hits the backend: Gemini extracts structured fields, and Supabase fires three independent lookups. Wall-clock is dominated by Gemini (~1.8s). DB queries finish in 80-150ms.

Fuzzy matching with pg_trgm

Drug naming in India is chaos. The CDSCO registry has Crocin 500mg Tablet IP, CROCIN 500, and Crocin Advance as distinct rows. A photo of a strip rarely matches the canonical name exactly — paper gets crumpled, fonts vary, manufacturers abbreviate.

We use PostgreSQL's pg_trgm extension with GIN indexes:

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX drugs_brand_trgm_idx
  ON drugs USING GIN (brand_name gin_trgm_ops);

-- lookup
SELECT brand_name, molecule, manufacturer
FROM drugs
WHERE brand_name ILIKE '%' || $1 || '%'
   OR brand_name % $1        -- pg_trgm similarity operator
ORDER BY similarity(brand_name, $1) DESC
LIMIT 5;

The % operator returns true when trigram similarity exceeds pg_trgm.similarity_threshold. Combined with ILIKE for prefix matches, we get recall on messy OCR without falling over on false positives. Confidence scores are exposed in the UI — if similarity is below 0.4, we mark the result as AI Estimated instead of CDSCO Verified.

Source transparency over false confidence

Every result card carries a badge: CDSCO Verified, Jan Aushadhi / BPPI, or AI Estimated. The AI badge always includes the line "verify with a pharmacist" — we'd rather say "not found" than give a counterfeit-looking strip false legitimacy.

We deliberately don't store scan history. A user photographing psychiatric or oncology medication is sharing sensitive health info. No account, no record, no analytics on what people scan.

The data sources

CDSCO Approved Drug List — 300k+ rows, refreshed quarterly from cdscoonline.gov.in
Jan Aushadhi — janaushadhi.gov.in
NPPA price ceilings — nppaindia.nic.in
A hand-picked dataset for the highest-traffic molecules, with a local fallback when Supabase isn't configured

Stack

Frontend: React 18 + Vite, client-side image compression before upload (cuts 4MB phone photos to ~250KB JPEG)
Vision + NLG: Gemini 1.5 Flash (one call for extraction, one for the plain-English explanation)
DB: Supabase / PostgreSQL 15 with pg_trgm
API layer: Vercel serverless functions, ESM (type: module)
i18n: 6 Indian languages (EN, HI, TA, BN, TE, MR)
Hosting: Vercel

The price engine falls back to a hardcoded JAN_AUSHADHI_DB if Supabase can't be reached — every result still has a price row.

What's next

A WhatsApp bot. 500M+ Indians use WhatsApp; nobody wants to open a browser at the chemist counter. A photo-to-number flow would 10x reach overnight. Then offline mode for Tier-3 Jan Aushadhi stores with poor connectivity, and a "find the nearest Kendra" map to close the last-mile gap.

Try it: agadahealth.vercel.app
Source: github.com/AmSach/agadahealth
Team: Aman Sachan, Siddharth Lalwani, Chetna Kalra, Syed Akbar — Team Agada, Open Innovation 2026.

Built with Gemini vision + a government drug database to fight pharma price gouging. No data retention, no paywalls, no accounts.

How I read 600 RSS feeds every morning in 3 minutes (pure Python, no framework)

Aman Sachan — Fri, 05 Jun 2026 01:52:10 +0000

Every morning I was opening 8-10 tabs — TOI, The Hindu, Indian Express, NDTV, Moneycontrol, Scroll, The Wire, FT, BBC. By the time I finished, it was 9 AM and I'd lost an hour to context-switching. So I built something to fix it.

What I built

A scheduled agent that polls ~600 RSS sources every morning, deduplicates cross-posted articles, categorizes them into 8 buckets, and emails me a clean brief before 8 AM IST.

The full skill is 293 lines of Python — pure stdlib (urllib, xml.etree, re, email.utils). No feedparser. No framework.

The architecture

17 Google News RSS queries + 30 direct publisher feeds
        |
        v
[fetch_all]  -- parallel via concurrent.futures
        |
        v
[dedupe]  -- 3-pass: normalize title, URL quality, jaccard token overlap
        |
        v
[categorize]  -- 8 buckets, rule-based keyword match
        |
        v
[render_markdown]  -- top stories + categorized sections
        |
        v
[send_email]  -- Gmail SMTP, plain text + optional PDF

The interesting parts

Three-pass dedupe

Google News reposts the same story from 5+ outlets. Single-pass title matching misses the tricky ones (reworded headlines, different source attribution).

def normalize_title(t):
    # "Modi visits US - The Hindu" -> "modi visits us"
    t = re.sub(r'\s*-\s*[A-Z][a-zA-Z\s]+$', '', t)
    t = t.lower().strip()
    t = re.sub(r'[^\w\s]', '', t)
    return t

def url_quality(url):
    if 'news.google.com' in url: return 0.2
    if 'toi' in url or 'thehindu' in url: return 1.0
    return 0.5

def is_duplicate(a, b):
    return jaccard(tokenize(a.title), tokenize(b.title)) > 0.65

Rule-based categorization beats LLM here

8 buckets (Politics, Economy, World, Business, Tech, Defence, Sports, Science)
~30 keywords per bucket
Matches in <1ms per item
No API cost, no rate limits
Predictable: "RBI rate decision" -> Economy, always

Today's actual brief (5 Jun 2026)

48 stories from 600+ raw pulls (92% dedupe rate)
7 top stories (FT, Al Jazeera, NDTV, India Today)
~600 words, ~3 min read
Landed in inbox at 7:00 AM IST

Why this beats the alternatives

Apple News / Google Discover: too US-centric, weak India coverage
Twitter lists: noisy, no dedupe, no archival
Manual scanning: 1+ hour/day, miss stuff
Other RSS readers: they aggregate but don't dedupe or categorize

Stack

Python 3.10+, stdlib only
Runs as a scheduled agent on Zo Computer
Sends via Gmail SMTP
~293 LOC total, single file

GitHub: https://github.com/AmSach/india-daily

Happy to answer questions on the dedupe logic or categorization rules. If anyone wants a different regional feed set (US, EU, SEA), drop a comment.

I built a multi-agent BTC research pipeline — 3 agents, 1 daily signal, full code

Aman Sachan — Thu, 04 Jun 2026 00:51:28 +0000

I built a multi-agent BTC research pipeline that runs 3 specialized agents and outputs a daily signal

BTC at $76,099. Down 40.8% from the $125,835 ATH. RSI neutral at 54. Sitting 8% below the 200-day EMA.

Most retail traders and even most bots handle this kind of sideways chop with rules-based buy-the-dip logic. But rules miss context — when exchange reserves are at 7-year lows and $2.5B in ETF money just flowed in during a correction, that is structurally different from "BTC dropped 8% in a week, time to buy."

So I built a multi-agent BTC research system that does the reasoning explicitly. Three specialized agents, each with a domain, weighted and fused into a single daily signal.

The architecture

                        Master Controller (zo.ask)
                                  │
              ┌───────────────────┼───────────────────┐
              ▼                   ▼                   ▼
     Technical Agent      On-Chain Agent       Macro Agent
        (0.35)               (0.40)              (0.25)
              │                   │                   │
              └───────────────────┼───────────────────┘
                                  ▼
                        Signal Generator
                                  ▼
                          Daily Report

Technical Agent — RSI, MACD, Bollinger Bands, support/resistance, price vs 200 EMA, distance from ATH. Reads the chart and gives a directional score.

On-Chain Agent — MVRV, realized price floor, exchange reserves (7-year low = bullish), ETF flows, accumulation trend score. Reads the blockchain fundamentals.

Macro Agent — DXY correlation, BTC-S&P500 correlation, Fed rate expectations, S&P500 at record highs (risk-on backdrop), geopolitical risk (Iran war premium), institutional adoption. Reads the world outside crypto.

Each agent outputs a score (0-1) and a confidence (0-1). The signal generator multiplies scores by weights (0.35 / 0.40 / 0.25) and sums them. Today's output:

Total Score:    0.662
Confidence:     0.684
Signal Type:    BUY
Action:         Take partial position (25-50% of capital)

How the agents actually score

Here is the on-chain scoring (simplified):

def score_mvrv(mvrv):
    if mvrv < 1.5:   return +0.20, "UNDERVALUED"
    if mvrv < 2.5:   return +0.10, "FAIR_VALUE"
    if mvrv < 3.5:   return -0.15, "OVERVALUED"
    return                 -0.25, "EXTREME_OVERVALUED"

def score_exchange_reserves(pct):
    if pct < 12:    return +0.15, "LOW_RESERVES_BULLISH"  # 7-year low
    if pct < 15:    return +0.05, "NORMAL"
    return                 -0.10, "HIGH_RESERVES_BEARISH"

The technical agent is similar — RSI 54 = neutral, below 200 EMA = -0.10, near resistance = -0.15. Each indicator adds or subtracts from the total weight.

What today's signal actually says

Agent	Score	Weight	Contribution
Technical	0.44	0.35	0.154
On-Chain	0.80	0.40	0.320
Macro	0.75	0.25	0.188

On-chain at 0.8 because: MVRV 1.5 (fair value but historically a buy zone), realized price floor $50K (we're well above), exchange reserves at 11.9% (7-year low = holders aren't selling), $2.5B in Q1 ETF inflows (institutions bought the correction).

Macro at 0.75 because: S&P500 and Nasdaq at record highs (risk-on backdrop that should lift BTC), BTC lagging those highs (bullish catch-up), MSTR bought 34,164 BTC at $74,395 ($2.54B position), Trump Strategic Bitcoin Reserve at 328,372 BTC (~$24.5B).

Technical is the laggard at 0.44 because: RSI neutral, below 200 EMA ($82,919), and the price is near the $75,000 resistance. The chart says "wait for confirmation" while the fundamentals say "the dip is over."

That's the whole point of the multi-agent design: when fundamentals say BUY but technicals say WAIT, the system outputs a calibrated partial-position signal instead of a binary yes/no.

Signal types

STRONG_BUY:   score >= 0.75 → full position (50-100%)
BUY:          score >= 0.60 → partial position (25-50%)
NEUTRAL:      score >= 0.45 → hold / no new entries
SELL:         score >= 0.30 → reduce position 25-50%
STRONG_SELL:  score <  0.30 → exit or short

Running it

cd btc-research
python3 scripts/run_analysis.py

Outputs to signals/daily_signals.json and reports/daily_report_YYYY-MM-DD.md. Each agent is independently runnable:

python3 agents/technical_agent.py   # chart signals
python3 agents/onchain_agent.py     # Glassnode-style metrics
python3 agents/macro_agent.py       # DXY, Fed, equities

The whole thing runs as a scheduled agent on Zo Computer at 06:00 UTC daily, writes a markdown report, and could email/Telegram it to the user. Every component is plain Python — no LLM calls, no external APIs, fully transparent scoring.

What's not here yet

The agents currently use a static dataset (current price, MVRV, ETF flows, etc.) hand-curated from research. The next iteration is to wire real data: CoinGecko or Glassnode for price/on-chain, FRED for macro. The agent code is structured for it — analyze() just needs to swap the hardcoded metrics dict for an API call.

The signal generator is also naive about disagreement. When technicals say NEUTRAL but on-chain says BULLISH, the weighted sum works but doesn't tell you "technicals are the drag, watch for the breakout." A v2 could add per-agent variance detection and flag low-confidence signals explicitly.

Stack

Python 3.10+
No external libraries for the agents (just stdlib)
Markdown reports
SQLite for historical signals (planned)
Runs on Zo Computer as a scheduled agent

Links

GitHub: https://github.com/AmSach/btc-research
Skill: /home/workspace/btc-research/skills/btc_system/SKILL.md
Architecture doc: btc-research/AUTOMATION.md
Latest signal: signals/daily_signals.json

MIT license. The signal is research, not advice — always do your own due diligence. Crypto is risky. The point of the system is to make the reasoning auditable, not to replace your judgment.

If you have ideas for additional agents (sentiment, derivatives funding rates, options skew, miner flows), PRs welcome. The agent interface is simple — return a dict with signal, score, confidence, and a summary string — and the signal generator picks them up automatically.

I automated reading 600+ RSS feeds into one daily India news brief

Aman Sachan — Sun, 10 May 2026 04:05:25 +0000

Every morning I used to spend an hour jumping between news sites — NDTV, The Hindu, Economic Times, scrolling through dozens of RSS feeds. Then I built a script to do it for me.

The problem with more sources

More feeds actually made quality worse until I built a velocity scoring system. Without it, breaking stories drowned in noise and structural trends kept resurfacing even after I had already read them.

What I built

600+ RSS feeds, deduplicated, ranked by source authority and story velocity — synthesized into one email delivered at 8:30 AM IST. Politics, economy, tech, environment, markets — all in one readable brief.

The pipeline:

Fetch — parallel HTTP requests to all feeds, 10-second timeout per feed
Deduplicate — simhash-based near-duplicate detection across all sources
Rank — source authority weight × story velocity × recency score
Summarize — Groq AI generates a 3-paragraph situational brief from top stories
Email — formatted HTML email, direct to inbox

The hardest part

The velocity scoring. I tried naive tf-idf first — it ranked opinion pieces over breaking news. The fix was a momentum score that tracks how many new sources pick up a story in the first 2 hours. Now genuine breaking stories surface without drowning editorial content.

Stack

Python 3.12 + feedparser + httpx
Redis for feed metadata cache
Groq API for summarization
SQLite for story fingerprints

Result

One email at 8:30 AM instead of 60 minutes of scattered reading. MIT licensed.

GitHub: https://github.com/amsach/btc-research (skill lives in /home/workspace/Skills/india-daily/)

If you consume India news for research or journalism, this might save you serious time.

Python #India #RSS #Automation #OpenSource #Journalism

I built a distributed compute grid where your idle laptop runs ML jobs — here's the architecture

Aman Sachan — Wed, 06 May 2026 01:02:06 +0000

The Problem

Most personal computers sit idle 90% of the time. Meanwhile, ML training and gaming workloads cost a fortune on cloud GPU instances. I wanted to bridge that gap — turn idle hardware into useful compute.

What I Built

ComputePool — a hub-and-spoke distributed compute grid. Your Zo Computer acts as the control plane. Idle laptops and PCs become worker nodes that poll for jobs, execute workloads, and earn credits.

Architecture

Node Agent (Python) ← polls → Hub API ← dispatches → Worker Pool
                                    ↓
                              Credit Ledger
                                    ↓
                              Cashout System

Node Agent (node-agent/node_agent.py):

Polls hub every 30s for available jobs
Reports GPU tier (RTX 4090 = 3x credit multiplier)
Streams results back on completion

Hub (hub/hub.ts):

FastAPI backend on Railway
Job queue with priority based on GPU tier
Credit ledger per node
Regional multipliers (Indian region: 0.7x)

Dashboard (frontend/):

Next.js 14 on Vercel
Real-time job status, credit balance, node management
Live at man44.zo.space/pool

Credit Economy

Workers earn credits per job completed
GPU tiers: RTX 4090 (3x), RTX 3080 (2x), GTX 1080 (1x)
Indian region: 0.7x base rate
20% platform fee on all earnings
Minimum cashout: ₹500

Key Design Decisions

Pull-based job distribution — Nodes poll, hub doesn't push. Eliminates NAT traversal issues.
GPU-tiered pricing — Higher-end GPUs earn more credits, incentivizes quality hardware.
Regional multipliers — Adjusts for purchasing power parity in different markets.

Stack

Backend: FastAPI + PostgreSQL
Frontend: Next.js 14 + Tailwind CSS
Node Agent: Python 3.10+ with Docker
Deployment: Railway (backend) + Vercel (frontend)

GitHub

https://github.com/amsach/compute-pool

Feedback Wanted

Is the credit economy balanced for casual node operators?
Would you run a node agent on your machine? Why or why not?
Any security concerns with the pull-based model?

distributed #machinelearning #python #javascript #BuildInPublic

Qwen sky proof: compressed memory made a tiny model behave better — with the receipts

Aman Sachan — Mon, 04 May 2026 21:33:46 +0000

This was a tiny-model before/after run with a very ordinary goal: keep the answer useful when the wording changes.

The setup used Qwen2.5-0.5B-Instruct with a memory layer around it.

The measured result

From the proof pack:

Before latency: 10,061.7 ms
After latency: 4,652.6 ms
Before tokens: 35
After tokens: 97
Token saved: -177.1%
Latency delta: -5,409.1 ms
Peak RSS: 1,794 MB

That is a nice reminder that “smaller prompt” is not always the same thing as “better answer”. Sometimes the smarter move is to give the model the right memory, even if it costs a few more tokens.

What the demo showed

The before run was raw. The after run used a compressed memory summary that kept the useful facts and dropped the filler.

That is the point of this kind of system: stay useful when the wording changes.

Proof pack

KVQuant / BitForge: same model, smarter context, better answer

Aman Sachan — Mon, 04 May 2026 21:32:43 +0000

Most AI workflow posts are just a screenshot of a chat box and a hopeful caption.

This one is different: I ran the same local model twice on the same question, once with a raw prompt and once with a memory + retrieval stack around it.

What changed

Before:

raw prompt
no compression
no semantic retrieval
more clutter in context

After:

compressed working context
semantic retrieval from memory notes
fewer prompt tokens
same model, same task, less nonsense

The measured result

From the proof pack:

Before latency: 28,590.3 ms
After latency: 25,008.9 ms
Before accuracy: 0.500
After accuracy: 1.000
Before prompt tokens: 87
After prompt tokens: 108
Memory saved: -24.1%

That last line is the fun one: the “after” run used more prompt tokens here, because I tuned it to answer the question better. Token count is a tool, not a religion.

Why this matters

The model did not become magical. The workflow got smarter.

That is the whole game with KV cache compression and prompt shaping work: make the task clearer, measure the result, and keep the same model honest across versions.

Proof pack

LLM Foundry on a tiny model: the stack still does the heavy lifting

Aman Sachan — Mon, 04 May 2026 21:32:41 +0000

This run was intentionally small-model and intentionally boring: no cloud API, no fake genius, just a tiny local model plus a better stack around it.

LLM Foundry with Qwen2.5-0.5B is the version that makes the point most cleanly: the model itself is small, but the workflow around it can still be decent.

What the proof showed

From the local proof run:

Benchmark pass rate: 50%
Reasoning: 60%
Coding: 100%
Tool + memory: 100%

The demo also showed memory compression and retrieval in action. The exact lesson is simple: if wording changes, semantic retrieval is a lot better than brittle keyword matching.

Why I care

The whole point of this layer is not to brag about a bigger model. It is to make a small model more usable:

it can recover relevant context
it can shrink messy transcripts into working memory
it can be checked instead of hand-waved

That is the part around the model that turns a chat toy into something that can remember, recover context, and be tested.

Proof pack

LLM Foundry: why the stack around the model matters more than the model itself

Aman Sachan — Mon, 04 May 2026 21:31:17 +0000

I wanted to see whether a weak local model could become genuinely useful without pretending the base model was magic.

LLM Foundry is the stack around the model: memory, compression, semantic retrieval, provider support, and a benchmark harness.

The core idea

A useful model workflow usually looks like this:

read the task
recover relevant memory
compress the clutter
ask the model
check the answer
use tools if needed
save traces
benchmark the result

That is the difference between a chatbot and something you can actually trust on real work.

What changed

The current version now has:

embedding-based semantic retrieval
multi-provider support for OpenAI-compatible and Anthropic endpoints
compression + memory so long tasks can be shrunk into compact context
agent traces that can become training data later
benchmarks and harnesses so the system is measurable

The measured part

The proof pack shows:

Benchmark pass rate: 50%
Reasoning harness: 60%
Coding harness: 100%
Tool-use harness: 100%
Memory harness: 100%

That benchmark score is not a brag. It is a baseline. The point is that the system is measurable, and therefore improvable.

The honest limitation

Orchestration helps, but it does not create capability out of thin air. If the base model is weak at reasoning, the stack can make it more useful, more reliable, and easier to test — but not magically frontier-grade.

That is still a very good deal.

Proof pack

SignalHop: the acoustic mesh networking stack that talks through sound

Aman Sachan — Mon, 04 May 2026 21:31:16 +0000

I wanted an answer to a boringly practical question: what if the network is gone, but the speakers and microphones still work?

SignalHop is an acoustic modem and mesh prototype that moves tiny messages over ultrasound instead of Wi‑Fi or Bluetooth.

What it actually does

The current build uses:

FSK tones at 18 kHz and 20 kHz
48 kHz sample rate
a 4 up-chirp sync preamble
frames with payloads up to 255 bytes
a WAV/audio runtime for encode/decode and round-trip testing

That makes it useful for emergency text, sensor payloads, and low-bandwidth status updates. Not glamorous. Very useful.

The measured part

From the current proof pack:

Metric	Value
Encode latency	16.634 ms mean
Decode latency	713.028 ms mean
Round-trip	true
Payload tested	77 bytes
Max payload	255 bytes

The decode path is slower because it scans the full signal with correlation. That is fine for v1. The important part is that the numbers are measured, not imagined.

The honest limitation

The core modem path is field-deployable today, but the full multi-node mesh still needs hardware validation across real devices. I’m keeping that line explicit on purpose.

That means this is a real protocol stack, not marketing fog.

Proof pack

KVQuant: real terminal proof for KV-cache compression

Aman Sachan — Sun, 03 May 2026 17:27:30 +0000

KVQuant: real terminal proof for KV-cache compression

KVQuant is a cache-compression layer for long-context inference. The interesting bit is not the idea — lots of projects have that — but whether it survives contact with a real model, a real terminal, and a real benchmark table.

This write-up is the boring but useful version: what it does, what I ran, what the numbers were, and where it helps or doesn’t.

Why KV cache matters

When a model generates text, it keeps a memory of previous tokens in the KV cache. That cache grows with every step. Weight quantisation shrinks the model weights, but it doesn’t directly touch this memory tax.

KVQuant targets that cache directly:

Allocate fewer bits for older tokens
Pack the cache into smaller storage
Restore it before the next forward pass

That gives you a real memory win on long-running chats and long-context inference.

What I benchmarked

I ran two kinds of proof:

a real Hugging Face model run with distilgpt2
a deterministic synthetic cache benchmark to make the cache math obvious and reproducible

Real-model result

Scenario	Prompt tokens	Generated tokens	Baseline cache	KVQuant cache	Saved	Cache ratio	KVQuant compression
product-explainer	17	256	9.56 MiB	2.39 MiB	7.17 MiB	4.00x	8.00x
developer-note	19	256	9.63 MiB	2.41 MiB	7.22 MiB	4.00x	8.00x

Total cache saved: 14.40 MiB

Honest speed note

Scenario	Baseline t/s	KVQuant t/s	Speedup
product-explainer	21.17	16.05	0.76x
developer-note	21.88	20.10	0.92x

That is the part I do not want to hide: on a small CPU model, compression overhead can offset throughput gains. The memory savings are real; the wall-clock speedup is workload-dependent.

Actual terminal proof

This is the real terminal run I captured. The key part is that it is a direct terminal transcript from a benchmark script, not a dashboard summary.

Exact command run

source /home/.z/workspaces/con_v0tzKzkrq5Z4Ia2E/.venv/bin/activate
cd /home/.z/workspaces/con_v0tzKzkrq5Z4Ia2E/KVQuant
HF_HUB_DISABLE_PROGRESS_BARS=1 PYTHONPATH=. python examples/e2e_benchmark.py --model distilgpt2 --output-dir /home/.z/workspaces/con_v0tzKzkrq5Z4Ia2E/terminal-proof/output

Step-by-step terminal output

1) Benchmark started
# KVQuant end-to-end benchmark (distilgpt2)

2) Model and generation mode
Real Hugging Face causal LM, real greedy generation, and real output tokens.

3) Measured table
| Scenario | Prompt tokens | Generated tokens | Baseline t/s | KVQuant t/s | Speedup | Baseline cache | KVQuant cache | Saved | Cache ratio | KVQuant compression |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| product-explainer | 17 | 256 | 21.17 | 16.05 | 0.76x | 9.56 MiB | 2.39 MiB | 7.17 MiB | 4.00x | 8.00x |
| developer-note | 19 | 256 | 21.88 | 20.10 | 0.92x | 9.63 MiB | 2.41 MiB | 7.22 MiB | 4.00x | 8.00x |

4) Summary
**Average speedup:** 0.84x
**Average cache ratio:** 4.00x
**Average generated tokens:** 256
**Total cache saved:** 14.40 MiB

5) File outputs
HTML: /home/.z/workspaces/con_v0tzKzkrq5Z4Ia2E/terminal-proof/output/kvquant-e2e-benchmark.html
JSON: /home/.z/workspaces/con_v0tzKzkrq5Z4Ia2E/terminal-proof/output/kvquant-e2e-benchmark.json
Markdown: /home/.z/workspaces/con_v0tzKzkrq5Z4Ia2E/terminal-proof/output/kvquant-e2e-benchmark.md

Exact output text for each prompt

These are the Qwen2.5-0.5B-Instruct sample outputs I used so the post shows the full text, not a cut-off snippet.

product-explainer

Prompt

Explain why KV cache compression matters in long-context AI systems. Return exactly 5 sentences, no bullets, no intro.

Output

KV (Key-Value) caches play a crucial role in long-context AI systems by efficiently storing and retrieving large amounts of data. By compressing key-value pairs, these caches reduce storage requirements significantly, especially when dealing with large datasets or complex queries. This compression helps in managing memory usage more effectively, ensuring that the system can handle high volumes of data without running out of disk space. Additionally, compressed KV caches enable faster retrieval times for frequently accessed data, improving overall performance and responsiveness in real-time applications.

developer-note

Prompt

Explain KV cache compression to an engineer who cares about latency, memory, and reliability. Return exactly 5 bullet points, no intro.

Output

- **Compression**: Utilizes data redundancy to reduce the amount of data that needs to be transferred between the CPU and disk.
- **Memory Efficiency**: Reduces the number of I/O operations required by caching frequently accessed data in memory.
- **Latency Reduction**: Minimizes the time it takes for data to reach the CPU from the disk, improving overall system performance.
- **Reliability Enhancement**: Ensures consistent access to data even when network or hardware failures occur.
- **Scalability**: Allows for efficient use of resources based on the size of the data being cached.

Browser-rendered proof

Here’s the full report rendered in browser.

The synthetic benchmark baseline

Before trusting real-model results, I verified with synthetic tensors across a range of cache shapes:

Scenario	Shape	Without KVQuant	With KVQuant	Saved	Ratio
chat-turn	(1, 8, 512, 64)	0.50 MiB	0.13 MiB	0.38 MiB	4.00x
code-assist	(1, 16, 1024, 64)	1.00 MiB	0.25 MiB	0.75 MiB	4.00x
rag-summary	(1, 16, 2048, 64)	2.00 MiB	0.50 MiB	1.50 MiB	4.00x
tool-agent	(1, 32, 2048, 128)	8.00 MiB	2.00 MiB	6.00 MiB	4.00x
long-context	(1, 32, 4096, 128)	16.00 MiB	4.00 MiB	12.00 MiB	4.00x
tiny-firmware	(1, 4, 256, 64)	0.0625 MiB	0.0156 MiB	0.0469 MiB	4.00x

The 4x ratio is consistent across all scales. This is the expected outcome: 4-bit quantization of fp16 gives you exactly 4x.

What changed in this round

Bigger intent set

Added scenarios with high token counts (256 output tokens) so the cache actually accumulates to meaningful sizes. Real-world use cases — not toy examples.

Real end-to-end benchmark

examples/e2e_benchmark.py runs a full generation loop and writes .html, .json, and .md output.

Real DynamicCache integration

CompressedDynamicCache in kvquant/cache.py is a drop-in DynamicCache subclass. It compresses on update() and decompresses on iteration. Works with model.generate() directly.

Tiny firmware export profile

PYTHONPATH=. python examples/e2e_benchmark.py --profile tiny

Generates a build-ready JSON profile that proves the cache shape, bit allocation, and target ratio without needing a full model.

Next direction: retrieval-assisted memory

A sensible next step is to combine KV compression with an embedding-indexed memory layer so the system can retrieve the most relevant past context instead of keeping every token equally alive. That could push compression harder while keeping quality closer to baseline, but that is a research direction, not a claim I can honestly call zero-loss yet.

What this is not (yet)

Not a throughput win on small/fast models — compression overhead > memory savings for distilgpt2 on CPU
Not a training system — inference only
Not magic — it targets the KV cache, not weights

What it is

A real, working KV cache compressor with honest benchmarks
A drop-in DynamicCache that production pipelines can use today
A foundation for the regimes where memory wins translate to throughput wins (larger models, longer context)

Try it

pip install kvquant

Or from source:

git clone https://github.com/AmSach/KVQuant.git
cd KVQuant
pip install -e .
PYTHONPATH=. python examples/e2e_benchmark.py --model distilgpt2 --output-dir ./benchmark-results

All benchmark data is reproducible. Screenshots and JSON logs are in the repo under examples/.