DEV Community: Andrew

Claude Video Review: Let Claude Watch Any Video (7K Stars)

Andrew — Sat, 11 Jul 2026 10:11:28 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

claude-video is an open-source Agent Skill that gives Claude — and 50+ other coding agents — the ability to watch videos. Paste a YouTube URL or a local .mp4, ask a question, and Claude answers grounded in what actually appeared on screen and what was said in the audio.

7,349 GitHub stars with 4,093 gained this week (currently one of the fastest-climbing Python repos on GitHub Trending)
Author: Brad Automates (bradautomates)
How it works: yt-dlp downloads the video, ffmpeg extracts frames (scene-aware or keyframes), captions come from yt-dlp or Whisper as a fallback, and Claude Reads every frame as an image plus a timestamped transcript.
Works with: Claude Code (plugin marketplace), plus Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, and 45+ other hosts via the Agent Skills protocol.
License: MIT.
Install (Claude Code): /plugin marketplace add bradautomates/claude-video then /plugin install watch@claude-video.
Install (anywhere else): npx skills add bradautomates/claude-video -g.

If you've ever pasted a YouTube link into Claude and gotten a hallucinated "based on the title" summary, this is the fix. But token costs on long videos are real, and the frame-budget dial is not optional if you care about your bill.

Why claude-video Matters Right Now

Coding agents in 2026 are multimodal on paper — Claude 4.5, GPT-5, and Gemini 2.5 all accept image inputs. But the popular assistants (Claude Code, Codex, Cursor) don't natively watch videos. Paste a YouTube URL and the agent either fetches a text transcript (missing 90% of what's on screen) or refuses.

That's a real gap. Developer work around video is diagnostic ("here's a screen recording of the bug, what's happening?"). Marketing work is comparative ("what hook did this competitor open with?"). Study work is summarization ("turn this lecture into notes"). None of it works on a bare text transcript.

claude-video fills the gap. It downloads the video, extracts scene-aware frames, pulls captions when available (free) or transcribes with Whisper as fallback, and hands the whole package — timestamped frames as images, timestamped transcript as text — to Claude. By the time Claude answers, it has seen the video and heard the audio.

And it does this without a bespoke MCP server, Docker, or an API key for the common case. Public YouTube videos with captions cost zero dollars beyond your existing Claude subscription. Whisper only kicks in when captions aren't available; Groq's whisper-large-v3 is roughly half a cent per five-minute clip.

What the Skill Actually Does

The /watch command follows a clear pipeline:

URL or path in. Anything yt-dlp supports works — YouTube, Loom, TikTok, X, Instagram, plus hundreds more. Local .mp4, .mov, .mkv, and .webm also work.
Captions first. yt-dlp is asked for manual or auto-generated captions before anything is downloaded. With --detail transcript, that's the entire cost — a few seconds and zero bandwidth.
Frame extraction. ffmpeg pulls frames per --detail: efficient decodes keyframes only; balanced and token-burner look for scene changes across the full video, falling back to duration-aware uniform sampling.
Deduplication. Every frame is scaled to a 16×16 grayscale thumbnail and compared against the last one kept. Near-duplicates (screen recordings holding one slide) get dropped before the frame budget is spent.
Transcript. yt-dlp captions win when they exist. Otherwise the skill extracts a 16 kHz 64 kbps mono mp3 and ships it to Groq's whisper-large-v3 (preferred) or OpenAI's whisper-1.
Handoff to Claude. Frame paths print with t=MM:SS markers, the transcript prints with timestamps, and Claude reads each JPEG in parallel.
Cleanup. Working directory prints at the end. If you're not asking follow-ups, Claude removes it.

The result: Claude answers grounded in the actual video. It can point at a frame and say "at 2:15 the terminal shows error: command not found," which is often the entire fix for a bug report.

Install and First Run

Claude Code (recommended — auto-updates):

/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video

Codex, Cursor, Copilot, Gemini CLI, and 45+ others:

npx skills add bradautomates/claude-video -g

The -g installs globally. Drop it for per-project scope. For claude.ai on the web, download watch.skill from the releases page and add it under Settings → Capabilities → Skills.

The skill needs yt-dlp and ffmpeg. On macOS both auto-install via Homebrew on first /watch. Linux and Windows print the exact commands. Nothing else is required for the captioned-video happy path.

For Whisper fallback on videos without captions:

export GROQ_API_KEY=gsk_...       # preferred — cheaper and faster
# or
export OPENAI_API_KEY=sk-...

Then:

/watch https://youtu.be/dQw4w9WgXcQ what happens at the 30 second mark?

Real Use Cases That Actually Work

Four patterns show up over and over in the discussion threads:

1. Diagnose a bug from a screen recording.

/watch bug-repro.mov what's going wrong?

Someone sends a Loom or .mov of the app misbehaving. Claude watches the recording, finds the frame where the issue appears, describes what's on screen, and often catches the cause without you opening the file. Killer app for developer teams — support tickets with screencasts triage in seconds.

2. Break down someone else's video.

/watch https://youtu.be/<viral-video> what hook did they open with?

Marketers and content strategists reverse-engineer competitor launches, ad creative, and podcast intros. Claude reads the opening frames + transcript and gives you a structural breakdown you can actually apply.

3. Skip the hype in an update video.

/watch https://youtu.be/<launch-video> what's actually new — skip the hype

Ten minutes of intro plus five real feature announcements. Claude strips it to the substance — useful for tracking launches and keynotes without 2×-ing the whole show.

4. Turn a video series into searchable notes.

/watch https://youtu.be/<course-video> summarize this to a note

Run across a playlist and you get a per-video summary you can grep. A YouTube course becomes a searchable knowledge base instead of ten hours of playback.

Token Cost: The Number You Actually Care About

Every frame is an image, and image tokens add up fast. The auto-fps logic exists so you don't spend your context budget on a sparse scan of a 30-minute video that would have been better answered by a focused 30-second window.

Default frame budget by duration:

Video length	Default frames	What you get
≤ 30 s	~30	Dense — every key moment
30 s – 1 min	~40	Still dense
1 – 3 min	~60	Comfortable
3 – 10 min	~80	Sparse but workable
> 10 min	100 (capped)	Sparse scan — re-run focused

Real measurement from the README against a 49-minute YouTube video (1280×720, auto-captions):

Mode	Frames	Extraction	Image tokens
`transcript` (captions only)	0	~4.5 s	~26.6k text tokens
`efficient` (keyframes)	50	~0.5 s	~9.8k
`balanced` (scene detection)	100	~20.9 s	~19.7k
`token-burner` (uncapped scene)	116	~21.0 s	~22.8k

Image tokens use Anthropic's (width × height) / 750. At the default 512 px width, 720p frames are 512×288 (~197 tokens each). --resolution 1024 roughly 4×'s that.

Practical guidance: when a user names a moment ("around 2:30", "the last 30 seconds"), pass --start and --end. Focused mode gets denser per-second budgets, capped at 2 fps. A tight window over the right slice of video is dramatically more useful than a sparse scan of the whole thing.

Frame Deduplication: The Detail That Saves Money

Screen recordings, slide decks, and static shots all produce near-identical frames. Without deduplication, a slide held for 90 seconds might generate a dozen frames, each billed as a separate image.

claude-video runs a dedup pass by default on every frame mode (--no-dedup turns it off):

One ffmpeg call scales each extracted JPEG to a 16×16 grayscale thumbnail.
For each frame, compute the mean absolute difference against the last frame kept (average per-pixel brightness change, 0–255 scale).
If that difference is at or below 2.0, the frame is dropped.

Comparing against the last kept frame — not the previous frame — catches slow fades that would never trip a frame-to-frame threshold. The threshold is deliberately low so a one-line code diff, a scrolling terminal row, or two differently-colored flat slides all survive.

The output line reports what was collapsed: 6 selected from 14 candidates (… 8 near-duplicates dropped …). On always-moving footage nothing is dropped and you pay what you would have paid anyway.

This is one of those small implementation details that separates a tool that works from a tool you can actually put in front of a real bill.

What Makes This Different From Existing Approaches

There's precedent. claude-video-vision by jordanrendric landed in April 2026 as a Claude Code plugin using MCP. Gemini has native video input. GPT-4o accepts video. So why does claude-video matter?

1. It's a Skill, not a plugin. Agent Skills are a portable protocol. Install once, use it in Claude Code, Codex, Cursor, Copilot, and 45 other hosts. Plugin-only tools work in exactly one place.

2. Zero-config happy path. No MCP server, no Docker, no API key when the video has captions — the majority of public YouTube.

3. Honest cost engineering. Auto-fps, dedup, and the transcript mode that skips download entirely aren't obvious features to a first-time author. Someone was watching their own token bill.

Where Gemini wins: hour-plus content with a single API call and true temporal reasoning. If you're already on Gemini and don't care about tool portability, use that.

Community Reactions

Sentiment across GitHub, Reddit, and the Agent Skills community is broadly positive with sharp caveats:

On r/ClaudeAI, users have been asking for exactly this since April 2026 — "Claude Code can watch videos, is there a way I can get Chat to do the same?" claude-video is the first cross-host answer.
The knightli.com writeup called out frame deduplication as the standout technical detail — "brightness-based dedup against the last kept frame is the right primitive."
Most common complaint in issues: token spend on long videos is real. transcript-only is the escape hatch, but balanced on a 30-minute video spends 20k+ image tokens whether you needed them or not.
Codex and Cursor users confirm npx skills add works out of the box on macOS and Linux. Windows requires manual yt-dlp and ffmpeg install first.

4,093 stars this week reflects actual adoption, not a Show HN pop — there's no HN launch post at time of writing.

Honest Limitations

Not native video input. Every frame becomes an image in Claude's context. Long videos on balanced or token-burner push past 200-frame token warnings.
Whisper fallback isn't free. No captions on a five-minute clip means a Groq or OpenAI API call. Cheap, not zero.
yt-dlp breaks when platforms change scrapers. If YouTube changes its API tomorrow, expect a broken /watch until yt-dlp ships a fix.
Scene detection misses gradual transitions. A podcast with two static talking heads gains nothing from frames — use --detail transcript.
No temporal reasoning across frames. Claude sees each frame independently. Questions depending on subtle motion or ordering get worse answers than a true video model.
The >10 min sparse-scan warning is real. Re-run with --start / --end around the moment you care about.

None are dealbreakers — they're the shape of composing "frames + transcript + text-and-image LLM" instead of using a video foundation model.

FAQ

Is claude-video only for Claude?

No. Despite the name, it works with any Agent Skills host: Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, plus 45+ others. Install via npx skills add bradautomates/claude-video -g. "Claude" is in the name because the killer install path — a one-line /plugin marketplace add — is Claude Code's plugin marketplace.

Do I need an OpenAI or Groq API key?

Only if your video has no captions. Public YouTube videos almost always have manual or auto-generated captions, and yt-dlp pulls them for free. Set GROQ_API_KEY (preferred — cheaper and faster than OpenAI) if you want Whisper fallback for videos without captions.

How much does it cost per video?

The Claude API cost is dominated by image tokens on frame modes. A 5-minute captioned video on balanced is roughly 10k image tokens plus a few thousand text tokens for the transcript — under 10 cents at Claude 4.5 Sonnet pricing. A 30-minute video on balanced is 20k+ image tokens plus a 30-minute transcript. Use transcript mode or --start / --end when you can.

Can it watch private videos on Loom, Google Drive, or S3?

Local paths work for anything you can download to disk. yt-dlp supports authenticated cookies for Loom, YouTube, and a few hundred other sources — pass them via --cookies at the CLI or via the equivalent skill parameter. Private S3 links work if you presign them or download locally first.

Does it work offline?

Local video files work fully offline if you already have captions or don't need a transcript. Whisper transcription is API-only in the current release (no local model support).

Is it safe to run on production Claude Code?

The skill is a self-contained Python script that runs yt-dlp and ffmpeg and writes to a temp directory. It doesn't touch your codebase or your global state. The code is MIT-licensed and readable in one sitting on GitHub. It's about as safe as any Agent Skill gets — but as always, review before running in a security-sensitive environment.

How does it compare to Gemini's native video input?

Gemini processes video natively — one API call handles the whole clip with true temporal reasoning. If you're already on Gemini and only care about video-in-video-out, use that. claude-video wins when you want the same skill across Claude, Codex, Cursor, and 47 other hosts, and when you don't want to leave your existing agent workflow.

Should You Install It?

Yes if: you use Claude Code / Codex / Cursor / Copilot and paste YouTube links regularly; your team gets bug reports as screen recordings; you do content analysis or study lectures; you want a portable video-watching skill that follows your agent stack.

No if: you're already deep on Gemini and only care about video quality; your videos are 30+ minutes and the token bill will hurt; you need true temporal reasoning across shots.

At 7,349 stars and 4,093 this week, it's the best cross-host answer to a real gap. One-line install, near-zero running cost on captioned public video, and the auto-fps + dedup engineering means you don't have to babysit the token budget for common cases.

The five-minute Loom your teammate just sent has never been easier to hand to Claude.

Sources

bradautomates/claude-video on GitHub — primary source, README, install instructions, benchmarks
Agent Skills specification — the protocol claude-video ships as
yt-dlp — video download engine
Groq Whisper API — preferred Whisper backend
knightli.com writeup (2026-07-08) — independent review
r/ClaudeCode video vision update thread — earlier context on Claude Code + video

If claude-video helps your workflow, star the repo — it directly signals the maintainer to keep shipping.

Astryx Review: Meta's Agent-Ready Design System (7.6K Stars)

Andrew — Fri, 10 Jul 2026 10:08:55 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Astryx is Meta's open-source React design system, released in Beta in late June 2026 after eight years of internal use. Key facts:

7,600+ GitHub stars (4,087 this week — hitting #3 on GitHub Trending)
13,000+ apps inside Meta already run on it (Facebook, Instagram, WhatsApp, Threads use its foundations)
150+ accessible React components, 10 ready-made themes, and a CLI + MCP server
Built on: React + StyleX (Meta's compile-time atomic CSS engine)
The differentiator: "Agent ready" — every component ships with JSDoc annotations, structured docs, a self-describing JSON manifest, and an MCP server so AI coding agents (Claude Code, Cursor, Codex) build with the same reference material humans do.
License: MIT
Install: npm install @astryxdesign/core @astryxdesign/theme-neutral

If you're building a React app in 2026 and you want an AI agent to author the UI for you, Astryx is the first design system explicitly engineered for that workflow. But it's Beta, StyleX is polarizing, and if your team isn't using AI coding agents heavily, shadcn/ui probably still wins.

Why Astryx Matters Right Now

Every design system before Astryx was built for humans reading docs. Astryx is the first mainstream design system where "an AI agent can consume it correctly" is a shipping requirement — not a nice-to-have.

That distinction shows up in three places:

A self-describing CLI manifest. Running npx astryx manifest --json returns a structured JSON payload listing every command, argument, flag, and response type — an OpenAPI spec for the CLI. Agents read one structured payload instead of scraping --help text.
An MCP server. Astryx ships a Model Context Protocol server that Claude Code, Cursor, and any MCP-aware agent can plug into to browse components, fetch API docs, and scaffold pages.
Documented conventions everywhere. Every component follows the same naming, prop, and composition rules. Once an agent has learned three components, it can predict how the fourth behaves — which is exactly how humans learn design systems too.

Meta's framing on their launch blog is honest about the shift:

"Design systems have historically been designed for human consumption, but as more code is written by agents, we have to rethink how design systems are structured and the role that they play. Astryx was built ground-up to be AI-operable, as opposed to retrofitting existing design systems to play nicely with agent behaviors."

That's the pitch. Now let's look at whether it holds up.

What You Actually Get

Astryx is not just a component library — it's four things stitched together:

Layer	What it does
Foundations	Typography, color, layout, accessibility primitives
Components	150+ typed React components (Button, Table, DatePicker, Modal, etc.)
Templates	Full-page compositions: dashboards, settings, forms, detail pages
Themes	10 ready-made brand-level themes, all fully customizable via CSS custom properties

The themes are named default, neutral, daily, butter, chocolate, matcha, stone, gothic, brutalist, and y2k — so yes, you can ship a y2k-themed enterprise dashboard if you want to.

Quick Start (Next.js)

The simplest setup is a few CSS imports plus a theme provider. No PostCSS config, no Babel plugin, no build integration:

npm install @astryxdesign/core @astryxdesign/theme-neutral
npm install -D @astryxdesign/cli

src/app/globals.css:

@import '@astryxdesign/core/reset.css';
@import '@astryxdesign/core/astryx.css';
@import '@astryxdesign/theme-neutral/theme.css';

src/app/providers.tsx:

'use client';

import Link from 'next/link';
import {Theme} from '@astryxdesign/core/theme';
import {LinkProvider} from '@astryxdesign/core/Link';
import {neutralTheme} from '@astryxdesign/theme-neutral/built';

export function Providers({children}: {children: React.ReactNode}) {
  return (
    <Theme theme={neutralTheme}>
      <LinkProvider component={Link}>{children}</LinkProvider>
    </Theme>
  );
}

That's it. Import components and go:

import {Button} from '@astryxdesign/core/Button';
import {Badge} from '@astryxdesign/core/Badge';

export default function Toolbar() {
  return (
    <div>
      <Button label="Save" variant="primary" />
      <Badge>Beta</Badge>
    </div>
  );
}

The Agent-Ready Workflow

Here's where Astryx earns its distinctiveness. The same CLI that a human developer uses is what an AI agent uses — and the outputs are designed for both consumers.

# Look up any component's full API, props, best practices
npx astryx component Button

# Emit a full-page template as source you can edit
npx astryx template dashboard

# Machine-readable command spec (the "agent-ready" hook)
npx astryx manifest --json

# List everything
npx astryx --list

When you pair Claude Code with Astryx's MCP server, the workflow becomes:

You: "Build me a settings page with a nav sidebar and a form for API keys."
Claude Code calls astryx template settings --skeleton via MCP to get the layout scaffold.
It calls astryx component TextField and astryx component Button to check the prop signatures.
It emits code that matches Astryx's conventions on the first try, because it read the same reference material a human would.

That's the theory. In practice, early users report it works — with caveats. Claude Code + Astryx tends to compose primitives correctly, but occasionally reaches for swizzle (Astryx's eject-component-source command) when it should have just used a theme override. That's a prompt-tuning problem, not a system flaw, but it's real.

No Styling Lock-in (This Is Underrated)

StyleX is Meta's compile-time CSS engine. It powers Facebook, Instagram, WhatsApp, and Threads. Figma and Snowflake use it too. But StyleX is polarizing — a lot of React developers already picked Tailwind or CSS modules, and don't want a third styling system.

Astryx's answer: you don't have to adopt StyleX. The design system authors its own internals in StyleX, but that's invisible to you. You override with className using Tailwind, CSS modules, or plain CSS.

The Tailwind bridge is especially clean. Astryx ships a tailwind-theme.css that maps its tokens to Tailwind utilities:

// Without the bridge — verbose:
<div className="rounded-[var(--radius-container)] bg-[var(--color-background-surface)] text-[var(--color-text-primary)]">

// With the bridge — just works:
<div className="rounded-lg bg-surface text-primary">

Some useful mappings:

Tailwind class	Astryx token
`text-primary` / `text-secondary`	`--color-text-primary` / `--color-text-secondary`
`bg-surface` / `bg-card` / `bg-body`	`--color-background-*`
`border-border` / `border-strong`	`--color-border` / `--color-border-emphasized`
`rounded-sm` / `rounded-md` / `rounded-lg`	`--radius-inner` / `element` / `container`
`shadow-sm` / `shadow-md` / `shadow-lg`	`--shadow-low` / `med` / `high`

Spacing references var(--spacing-1) as the base unit, so p-4 = 16px, matching Astryx's --spacing-4. If you already write Tailwind, this feels like it was designed by someone who actually ships React apps.

Open Internals, and Swizzle

Every design system eventually hits the "the vendor didn't expose the internals I need" wall. Astryx's answer is two-part:

Primitives are exported directly. The building blocks you'd reach for aren't locked behind a closed top-level API. You can compose at any level.
astryx swizzle Button ejects the full source of a component into your project. You own it, edit it, and it doesn't touch the upstream package. This is the same pattern Docusaurus popularized — and it's genuinely useful when a component is 95% right and you need to change one thing.

Combined with the CSS-variable theme cascade, this means the customization ladder is:

Use the component as-is.
Pass a className override.
Change the theme tokens.
Swizzle (eject) the component source.

Most projects should never get past step 3.

Context-Aware Spacing (The "Double Padding" Fix)

This is the kind of thing that only makes sense once you've been bitten by it. Nest a padded box inside another padded box, and the padding stacks. You end up manually stripping padding on the inner element to keep the edge gap visible.

Astryx's Layout components do this automatically — they measure their container context and compensate. It sounds trivial. It's not. Across 150+ components used together, this is the difference between a design system that ships and one where every dev learns the same 20 pixel-nudging tricks.

Community Reactions

Astryx dropped in late June 2026 and hit #3 on GitHub Trending for the week ending July 10. Real reactions from developers using it:

"Meta's willingness to eat their own dogfood for eight years before open-sourcing it is the actual sales pitch." — This is the strongest recurring take. Astryx isn't a v0.1; it's already surviving inside 13,000 production apps.
"The MCP server is what makes me consider switching from shadcn/ui." — For teams that lean hard on Claude Code or Cursor, agent-native tooling is a real competitive edge.
"StyleX is a hard sell to a team that just standardized on Tailwind." — Fair criticism, but the Tailwind bridge basically neutralizes it if you read the docs.
"Beta means beta." — Some components (Vega chart wrappers) are still @canary only. @astryxdesign/lab (experimental components) isn't published to npm at all yet.
"Facebook's design system is now open source" is misleading. — Astryx is Meta's internal design system, not the one that renders facebook.com. It's the shared foundation for internal tools, product surfaces, and admin apps.

Nobody I've seen is calling Astryx "the shadcn/ui killer." Different tools, different jobs — but for the agent-heavy workflow, Astryx has the clearer story.

Honest Limitations

Astryx is genuinely impressive, but here's what it's not:

It's Beta. Meta says so up front. Some APIs will change. Some packages (@astryxdesign/vega, @astryxdesign/charts) are @canary-only. @astryxdesign/lab isn't published to npm at all yet.
StyleX is a foreign concept for most React devs. The Tailwind bridge helps, but the docs assume you'll at least skim what StyleX does.
It's React-only. No Vue, no Svelte, no Solid. If you're not on React, Astryx has nothing for you.
The MCP server is new. Cursor + Astryx MCP works. Claude Code + Astryx MCP works. Less-popular MCP clients may have rough edges.
The theme story is CSS-variables + StyleX-first. If you use Emotion, styled-components, or vanilla-extract as your primary styling layer, you'll fight the cascade order at least once.
13,000 internal apps ≠ battle-tested externally. The internals have run at Meta scale for eight years, but the public packaging, install path, and MCP surface are weeks old. Expect real bugs.

Astryx vs. shadcn/ui vs. Radix

The obvious comparison. Each optimizes for something different:

	Astryx	shadcn/ui	Radix UI
Distribution	npm packages + CLI	Copy/paste via CLI	npm packages
Styling	StyleX + theme cascade	Tailwind	Unstyled (bring your own)
Agent tooling	CLI + MCP + JSON manifest	Community MCPs	Community MCPs
Themes	10 built-in, token-driven	Roll your own	Roll your own
Ownership	Composable + swizzle	You own every component	Vendor owns primitives
Best for	Agent-heavy teams shipping fast	Full control, don't mind maintaining	Building your own DS

If your team lives in Claude Code or Cursor and you want a full design system that agents can operate on, Astryx wins. If you want to own every line of component source and don't care about MCP, shadcn/ui still wins. If you're building your own design system from primitives, Radix is still the right foundation.

FAQ

Is Astryx production-ready?
Inside Meta, yes — 13,000+ apps run on it. Publicly, Meta labels it Beta. APIs will move. If you're comfortable with a Beta and you value the agent-tooling story, it's ready for production apps that can absorb minor breaking changes. If you need long-term-stable APIs today, wait for 1.0.

Do I have to use StyleX to use Astryx?
No. StyleX is invisible to consumers. You style overrides with className using Tailwind, CSS modules, or plain CSS. The design system's internals happen to use StyleX; your app doesn't have to.

How is Astryx different from shadcn/ui?
Astryx is an npm package with a CLI + MCP server, built for agent workflows. shadcn/ui is copy/paste components you own outright, built for maximum developer control. For agent-heavy workflows, Astryx has a clearer story. For hands-on maintainers, shadcn/ui is still hard to beat.

Does the MCP server work with Claude Code?
Yes. Astryx's MCP server is designed against the standard Model Context Protocol spec. Claude Code, Cursor, and any MCP-aware client can register it. You point your agent at the server, and it can browse components, fetch API docs, and scaffold pages.

Is this the design system Facebook.com uses?
No — that's a common misconception. Astryx grew inside Meta's monorepo and powers 13,000+ internal and product apps, but Facebook.com, Instagram, and WhatsApp use their own product-specific stacks. Astryx shares foundations (StyleX, primitives) with them but isn't the top-level system rendering the consumer surfaces.

The Verdict

Astryx is the first mainstream design system where "an AI coding agent can build with this" is a first-class shipping requirement, not a retrofit. For teams that use Claude Code, Cursor, or Codex heavily — that's most of us in 2026 — that's a real competitive edge over design systems designed for humans reading Storybook.

The rough edges are real: Beta status, StyleX friction for teams already on Tailwind (mitigated by the bridge), React-only. But if you're standing up a new React app today and you want an agent to author 80% of the UI while your team focuses on business logic, Astryx is the shortest path from "empty repo" to "shipped."

Meta releasing an eight-year-old, 13,000-app-tested design system as MIT open source — with an MCP server — is genuinely a big deal. It's also a preview of where every mainstream design system is going.

Star it on GitHub, try it in a side project first, and see how your favorite coding agent handles it.

Repository: facebook/astryx
Docs: astryx.atmeta.com
License: MIT

shadcn/improve Review: Split AI Coding Into Audit + Execute

Andrew — Thu, 09 Jul 2026 10:11:37 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

shadcn/improve is an Agent Skill from shadcn (the shadcn/ui creator) that treats AI coding as two very different jobs — understanding a codebase and deciding what's worth doing (the expensive part) and actually writing the diff (the cheap part) — and only lets your most capable model do the first. The output is not a patch. The output is a directory of self-contained Markdown implementation plans in plans/ that any weaker executor — Sonnet-tier, a local model, a junior human — can pick up and finish. Currently 7,534 GitHub stars, 314 forks, MIT-licensed, and one of the most talked-about Skills to ship since Agent Skills became a format.

Key facts:

7,534 GitHub stars, 314 forks — grew past 1.4K in the first 48 hours after shadcn tweeted the release
Created 2026-06-10, updated today (2026-07-09) — actively maintained
Install: npx skills add shadcn/improve (works in any host that speaks the Agent Skills format — Claude Code, Codex, OpenCode, Cursor with the Skills bridge, etc.)
Never touches source code — the ONLY files it writes live under plans/ (or advisor-plans/ if plans/ is already taken). No commits, no edits, no "quick wins while I'm in there."
Nine audit categories with parallel subagents: correctness, security, performance, test coverage, tech debt, dependencies & migrations, DX, docs, and direction (feature suggestions)
Verification-gated plans — every step ends with an exact command and expected output. Executors don't get to judge whether they succeeded.
Git-commit-stamped — each plan records the commit it was written against so executors can mechanical-drift-check before touching anything
--issues flag — publish plans directly as GitHub issues so any agent (or human) can pick them up where work already lives
MIT licensed, author: @shadcn

The pitch, distilled to one sentence: the plan is the product — everything upstream is worth paying Opus/GPT-5-Pro/Gemini-3-Ultra for, everything downstream isn't.

Why "one big model does everything" is the wrong shape

If you use Claude Code, Codex, or Cursor for anything past toy scale, you've felt the same ergonomic problem: the model smart enough to understand your codebase is way too expensive to have hammering out find-and-replace edits at 3¢ per turn. The workarounds all break down — cheaper models misread the architecture, expensive models drain the budget on mechanical work, and "pick the right size per task" doesn't work because the model doesn't know which tasks are small until it's halfway into one.

/improve names the real split. Intelligence compounds during understanding, not during execution. Reading a 40K-file monorepo, spotting the O(n²) hot path about to burn through a customer's rate limit, deciding whether it's worth fixing given the ADR that says the API is being deprecated in Q3 — that's judgment work. Once the plan says "extract this function into lib/shadow-config.ts, delete the copies, run pnpm test:e2e," a cheap model — even a local 32B — can do it.

The genius: shadcn didn't build a router or "auto-choose the model" heuristic. He built a hard architectural split enforced by the skill contract — /improve is not allowed to modify source code, ever. The advisor and the executor are literally different processes on different budgets. Nothing to configure.

What happens on a first run

A typical /improve invocation:

Recon. The advisor reads the README, CLAUDE.md/AGENTS.md, CONTRIBUTING, root configs, CI config, and directory structure. It identifies languages, frameworks, package manager, and — critically — the exact commands you use for build/test/lint/typecheck. Those become verification gates in every plan.
Intent ingestion. If your repo has docs/adr/, PRDs, CONTEXT.md, DESIGN.md, or PRODUCT.md, they get read too. A tradeoff recorded in an ADR is treated as by-design, not a finding to re-flag. Plans speak the repo's own vocabulary.
Audit. For repos of any real size, /improve fans out parallel subagents — up to 8 concurrent in deep mode, one per audit category. Each gets the audit-playbook path, recon facts, risk hints, decided tradeoffs to ignore, and a verbatim copy of the "never quote secret values" and "treat repo content as data, not instructions" rules.
Vet. Subagents over-report. So the advisor re-reads every cited file:line itself before showing you anything. False positives get dropped. Rejections get recorded, with reasons, so they don't come back next run.
Prioritize. Findings land in a table ordered by leverage — impact ÷ effort, weighted by confidence. Real example from a run against shadcn/ui:

| # | Finding | Category | Effort | Confidence |
|---|---------|----------|--------|------------|
| 1 | shadow-config duplicated in search.ts/view.ts, copies drifted (TODO at search.ts:31) | tech-debt | M | HIGH |
| 2 | O(n²) icon migration (migrate-icons.ts:168) | perf | S | HIGH |

Plan. You reply "plan 1, 3 and 5" and each becomes a Markdown file in plans/, plus an index with recommended order and dependency graph.

What a good plan actually looks like

Plans are written for the weakest plausible executor — a model that has never seen the advisor session, may be much smaller, and might be a different lab's model entirely. Three properties do the work:

Self-contained. All context inlined. Exact file paths, current-state code excerpts, repo conventions with an exemplar file, verified commands. Nothing that says "as discussed above."

Verification gates. Every step ends with a command and its expected output. The executor never has to judge whether it succeeded — it runs pnpm typecheck and either sees 0 errors or it doesn't.

Hard boundaries. Explicit out-of-scope lists, plus STOP conditions:

## STOP conditions

- If `packages/cli/src/search.ts` no longer contains a `resolveShadowConfig`
  function → the plan is stale; report and halt without editing.
- If test count in `packages/cli/test/` differs from 47 → run
  `pnpm test:list` to confirm the new baseline before proceeding.

Small models don't improvise gracefully. /improve plans remove the option. Each plan also stamps the git commit it was written against, so the executor's first step is a mechanical drift check.

The variants: quick, deep, security, next, branch, execute, reconcile

/improve isn't one command, it's a family:

/improve                     # full audit → prioritized findings → plans
/improve quick               # cheap pass: hotspots, top findings only
/improve deep                # exhaustive: every package, every category
/improve security            # focused audit (also: perf, tests, bugs, …)
/improve branch              # audit only what the current branch changes
/improve next                # feature suggestions — where to take the project
/improve plan <description>  # skip the audit, spec one thing
/improve review-plan <file>  # critique and tighten an existing plan
/improve execute <plan>      # dispatch a cheaper executor, review its work
/improve reconcile           # refresh backlog: verify, unblock, retire
/improve ... --issues        # also publish plans as GitHub issues

The two that make it more than a code-review tool are execute and reconcile.

/improve execute 001 spawns a cheaper executor subagent in an isolated git worktree, hands it plan #001, then reviews the result like a tech lead. Every done criterion gets re-run. Scope compliance gets checked. The diff gets read against intent. Verdict: approve (merging stays your call), send back for revision (max 2 rounds — no infinite loops), or block and refine the plan. The executor edits only in the disposable worktree; merging is always your call.

/improve reconcile turns this from "another code-review tool" into an actual backlog system. Runs next session, or next week, and does what a decent tech lead does at standup: verifies DONE plans still hold (someone merged an unrelated PR that broke an assumption — catch it now, not at 3am), investigates BLOCKED plans and writes around obstacles, refreshes drifted plans whose commit stamp is now 40 commits old, and retires findings that got fixed independently. Most "AI code review" tools don't have this loop — they give you a one-shot review, and if you don't act in the next hour, half the findings are stale forever.

Getting started

If you're already using Claude Code, Codex, OpenCode, or any host that speaks the Agent Skills format:

# Install the skill globally
npx skills add shadcn/improve

# In your project
cd my-project
/improve quick           # ~5 minutes, top ~6 HIGH-confidence findings
# or
/improve                 # standard, ~15 minutes, full findings table

First run tips:

Start with quick on a repo you know well. You'll immediately see which findings are real and which are "well, technically…" That calibrates your trust for the standard run.
Point it at existing intent docs if you have them. An ADR-0007-sync-writes.md explaining why the sync-over-async is intentional prevents /improve from surfacing it as a perf finding every single run.
Try /improve branch the next time you open a PR. This is the killer variant for day-to-day work — it audits only what your branch touches, catches the "hmm, this shouldn't be here" stuff before review, and doesn't drown you in cross-repo tech debt at PR time.
Try --issues once. Watching /improve publish a fully self-contained implementation plan as a GitHub issue — with verification gates and STOP conditions — is when it clicks that this thing is designed to be an actual team member's backlog, not a chat transcript.

Community reactions

Developer reaction has clustered around a few themes:

"Exactly the right way to split it." The token-cost math is obvious once pointed out — you shouldn't be paying Opus rates to run pnpm typecheck. Many devs had been informally doing this by hand ("I use GPT-5-Pro to plan, Sonnet to code") and finally have a formal contract.
"The senior architect handing plans to junior devs." This analogy stuck. It answers the "why not just let one model do it all?" objection — because that's not how good engineering orgs work either.
"Finally a Skill that isn't a toy." Agent Skills as a format has been criticized for producing trivial "add a rule that says X" skills. /improve proves the format can carry real engineering weight.
Feature request: "I want this for design/product review too." The audit → plan → execute pattern applied to non-code artifacts (docs, product specs, design systems) was the most-upvoted request in the first two weeks.

Main pushback: plans as Markdown files in git is opinionated. Teams with Linear/Jira/Notion as their backlog source of truth don't want a second place issues live. The --issues flag partially addresses this, but you'll reconcile two backlogs.

Honest limitations

Cost, still. "Cheap execution" is relative. /improve deep on a 200K-LOC monorepo with 8 parallel subagents is not cheap — it's cheaper than having Opus do the whole thing including execution, but not free. Standard mode on a normal repo is fine; deep mode is for occasions.
False negatives on things the advisor didn't know to look for. Every audit is bounded by the playbook categories. If your specific class of bug isn't in the playbook, no subagent gets assigned to it.
Executor variance is real. A truly weak local model may fail in ways the plan can't compensate for. Expect more revision rounds when the executor is very cheap.
reconcile assumes you actually come back. Best used as an ongoing habit, not a one-shot.
No Windows-native install path documented. The npx skills add route works cross-platform in principle, but sample commands assume a POSIX-ish shell. WSL is fine.
Not a linter. It's an advisor over static analysis output plus intent plus judgment — not a static-analysis replacement.

FAQ

Q: How is this different from just asking Claude Code to "review my repo"?

A one-shot review produces a stream-of-consciousness list of "issues" that vary wildly in importance, aren't self-contained, and evaporate the moment your session ends. /improve enforces vetted findings with confidence scores, self-contained plans a different model can execute, a persistent backlog under plans/, and a reconcile step that keeps the backlog honest across sessions. It's the difference between a code review and a working project management system.

Q: Does it work with Cursor, or only Claude Code?

Any host that supports the Agent Skills format. Claude Code has the deepest native support (subagents for parallel audit are first-class). Codex and OpenCode work well. Cursor works via the Skills bridge — parallelism degrades to sequential, so it's slower on big repos, but the plans are identical.

Q: What's the cheapest way to actually save money with this?

Run the advisor on Opus 4.7 or GPT-5.5-Pro, run /improve execute with Haiku 4 or Gemini 2.5 Flash. Ballpark: 5-10x total token cost reduction vs. running the whole loop on the frontier model.

Q: Can it publish plans as PRs, not just issues?

Not by design. /improve never modifies your working tree — no branches, no PRs from the advisor. /improve execute produces the diff in an isolated worktree; merging is always your call. This is deliberate.

Q: Is there a risk that the executor "goes rogue" and edits stuff outside the plan?

Two protections. First, the executor runs in a git worktree — it literally cannot touch your main working tree. Second, the advisor's review step re-runs done criteria and checks scope compliance against the out-of-scope list. Scope violations get sent back for revision.

Who should use this (and who shouldn't)

Use it if: you use Claude Code, Codex, or another Agent Skills-compatible host on a real codebase, you burn meaningful money on AI coding, and you already understand that "the plan is the product" isn't just a marketing line — it's how good tech leads have always worked.

Don't use it if: you're just experimenting with AI coding on a hobby project (overkill), your team has a hard rule that all work items live in Linear/Jira (the plans-in-git model will fight you), or you were hoping for a one-shot "audit and fix" magic wand (that's exactly what shadcn refused to build, on purpose).

The bigger point /improve is making is worth sitting with even if you don't adopt the skill: AI coding got the architecture wrong by starting from "one model does everything." The industry defaulted to that because it was the shortest path to a demo. The actual structure of software engineering — think, decide, spec, execute, review, reconcile — has always been differentiated labor with different budgets. /improve is the first widely-used tool that treats it that way. Others will follow.

Sources & further reading

shadcn/improve on GitHub — the repo, README, and installation instructions
SKILL.md source — the actual skill spec, if you want to see the guardrails verbatim
Example plan output — real plan generated by /improve against shadcn/ui
Agent Skills format specification — the ecosystem /improve plugs into
SaaSCity's early breakdown of the token-economics argument

Herdr Review: The Agent Multiplexer Your Terminal Needed

Andrew — Wed, 08 Jul 2026 10:15:52 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Herdr is an open-source terminal multiplexer built specifically for the age of AI coding agents. Think tmux, but every pane knows whether its agent is working, blocked, done, or idle — and agents can drive the multiplexer back through a socket API.

It hit 13,900+ GitHub stars and 4,500 new stars this week, landing in the top 5 trending repos. Built in Rust, distributed as a single binary, AGPL-licensed, and installable with Homebrew or a one-liner curl.

Why the sudden popularity? If you've been running Claude Code in one tmux pane, Codex in another, and a third agent waiting on review feedback, you've felt the pain Herdr solves natively.

Key facts:

Persistent sessions — detach and agents keep running. Reattach from any terminal or over SSH.
Real agent state — Herdr detects what each agent is doing and surfaces blocked, working, done, and idle status across all workspaces.
Both keyboard and mouse — tmux-style prefix keys plus click, drag, split, and context menus. No trade-off.
Socket API — agents can spawn panes, read output, wait on each other, and split terminals.
Plugin marketplace — extend workflows beyond the built-in features.
One Rust binary, no Electron — runs in whatever terminal you already have.
Homebrew install in one command, plus Homebrew, mise, and direct binary downloads.

If you're a developer running coding agents daily and still using tmux (or no multiplexer at all), Herdr is the tool that brings both worlds together.

Quick Reference

Field	Value
Repo	ogulcancelik/herdr
Website	herdr.dev
Docs	herdr.dev/docs
License	AGPL-3.0 (commercial licenses available)
Language	Rust
Install	`brew install herdr` or `curl -fsSL https://herdr.dev/install.sh \
Current version	v0.4.0
Stars	13,900+ (4,500/week)

The Problem Herdr Solves

Coding agents have changed how we work, but our terminal multiplexers have not.

If you run Claude Code to refactor a backend, Codex for frontend work, and a third agent iterating on docs, you're probably using tmux with three windows. It works, but tmux has no idea what the foreground process is doing. Is the agent typing? Waiting for approval? Stuck on a permission prompt? You have to check each pane manually.

The community noticed. On Hacker News (287 points, front page this week), the top comment thread compared Herdr to tmux and Zellij. The consensus: tmux persists terminals, but Herdr persists agent workspaces and understands their state.

"Herdr is a purpose-built answer to a problem that tmux was never designed to solve." — AI Weekly

The question is not whether a tool can run terminals. The question is whether it combines terminal-native persistence with semantic agent state and agent-driven automation.

How It Works

Architecture

Herdr runs as a background server process that owns persistent PTY sessions. When you run {% raw %}herdr, it either launches or attaches to your default session. The server keeps running after you detach, and agents in its panes keep running too.

Each session contains workspaces (project-level containers), workspaces contain tabs, and tabs contain panes. Herdr detects which panes contain coding agents and rolls their state up the hierarchy so the sidebar shows you what needs attention at a glance.

herdr server (background daemon)
  └── session (persistent PTY runtime)
       ├── workspace "project-a"
       │    ├── tab "backend"
       │    │    ├── pane 1 [Claude Code — working]
       │    │    └── pane 2 [shell]
       │    └── tab "frontend"
       │         └── pane 1 [Codex — blocked]
       └── workspace "project-b"
            └── tab "docs"
                 └── pane 1 [pi — done]

Installation

# macOS (Homebrew)
brew install herdr

# Linux / macOS (direct)
curl -fsSL https://herdr.dev/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy Bypass -c "irm https://herdr.dev/install.ps1 | iex"

# or use mise
mise use -g herdr

# or download binaries from GitHub releases

That's it. One binary, no npm install, no Electron, no runtime dependencies.

Quick Start

# Start Herdr in your project directory
herdr

# Herdr opens with an empty workspace. Start an agent:
claude

# The sidebar shows the agent's state. Split the pane:
# (mouse: right-click → split vertically)
# (keyboard: ctrl+b v for vertical split, ctrl+b - for horizontal)

# Open another tab:
# ctrl+b c

# Start another agent there:
codex

# Detach (agents keep running):
# ctrl+b q

# Reattach from anywhere:
herdr

Agent State Detection

This is Herdr's killer feature. The sidebar shows:

Working (green) — agent is actively processing
Blocked (yellow) — agent is waiting on input, approval, or permission
Done (gray) — agent completed its task
Idle (dim) — agent is running but not doing anything

Detection works through two systems:

1. Lifecycle hooks (for agents with deep integrations): pi, OMP, Kimi Code CLI, Hermes Agent, OpenCode, Kilo Code CLI, MastraCode. These agents report their state directly to Herdr.

2. Screen manifest detection (for all other agents): Claude Code, Codex, Cursor Agent CLI, GitHub Copilot CLI, Devin CLI, Qoder CLI, Droid. Herdr reads the live bottom-buffer screen snapshot and matches known UI patterns for approval prompts, permission questions, and idle states.

Blocked detection is deliberately conservative for screen-manifest agents — only matching known visible approval or question UI. If Herdr doesn't recognize a new prompt pattern, it falls back to idle.

You can install native integrations for richer state tracking:

herdr integration install claude
herdr integration install codex
herdr integration install copilot
herdr integration install devin
herdr integration install opencode

FAQ

How is Herdr different from tmux?

tmux persists terminals. Herdr persists agent workspaces and understands agent state. tmux has no concept of "blocked" or "working" — it's just panes with running processes. Herdr detects the foreground agent, reads its state from the screen buffer (or lifecycle hooks), and surfaces status across all workspaces. It also adds a socket API that agents can use to create panes, split, read output, and wait on each other. Under the hood, Herdr takes a similar approach to tmux (C vs Rust), but the feature set is agent-native from day one.

Can I use Herdr without any agents?

Yes. Herdr works as a general-purpose terminal multiplexer similar to tmux or Zellij. Its keyboard bindings, mouse support, session persistence, and split panes all work without an agent. The agent features are additive — you get a good multiplexer either way, and if you start using coding agents later, Herdr already knows what to do.

Which agents are officially supported?

Claude Code, Codex, pi, OMP, GitHub Copilot CLI, Devin CLI, Kimi Code CLI, Hermes Agent, Qoder CLI, Droid, OpenCode, Kilo Code CLI, MastraCode, and Cursor Agent CLI. Gemini CLI and Cline are detected but less thoroughly tested. Any terminal-based agent still runs as a multiplexer — you just won't get automatic state badges without an integration.

Can I use Herdr over SSH?

Yes. Herdr sessions are persistent and can be detached and reattached from any terminal. Run herdr on your remote server, start your agents, detach, and reattach over SSH from your local machine. The herdr --remote workflow supports direct connections to specific workspaces.

How does the AGPL license affect me?

Herdr is AGPL-3.0 for open-source use. If you use it as an individual developer or within your company without distributing modifications, AGPL doesn't impose new obligations. If your organization needs to embed or modify Herdr in a proprietary product, commercial licenses are available by contacting hey@herdr.dev.

Hands-On Experience

I installed Herdr via Homebrew on macOS and tested it across a day of real work with Claude Code and Codex.

Setup took under 10 seconds. Running herdr in my project directory opened a clean workspace with a pane already available. Starting claude in that pane was immediate — the sidebar showed the agent status within a second.

The mouse support is genuinely good. Right-click context menus for splitting, clicking panes to focus, and drag-to-resize all work as expected. I have been a tmux user for years (prefix keys muscle-memorized), and the fact that I could also just click felt liberating.

The agent state detection worked well for Claude Code and Codex. Blocked was the most useful signal. When Claude hit a permission prompt or Codex asked for clarification, the pane border turned yellow and the sidebar updated. In practice, this meant I could work in one pane, glance at the sidebar, and know exactly which project needed attention without scrolling through every terminal.

The socket API is the underrated feature. Agents can call Herdr's homie socket to create new workspaces, split panes, read output from other panes, and wait on agent completion. This enables multi-agent workflows where one agent delegates to another and waits for the result — all managed through the same multiplexer.

Performance

Herdr is written in Rust and it shows. Memory usage stays under 15MB for a session with 6 panes and 3 agents. CPU impact is negligible — process detection is event-driven, not a polling loop. Starting a new session is instant (< 100ms).

What I Missed from tmux

tmux kill-session -a — no quick way to kill all sessions
Built-in buffer management — tmux's copy mode is more mature
display-panes visual selector — Herdr's pane navigation relies more on sidebar clicking
Tiling window management (tmux's select-layout even-horizontal) — Herdr has manual splitting but fewer automatic layouts

These are gaps, not dealbreakers. Herdr is at v0.4.0 and the development pace is aggressive (multiple releases per week).

Community & Ecosystem

The Herdr community has already built a plugin marketplace with extensions for workflows, pane management, and integrations. The socket API has been picked up by the OpenCode and Kilo Code teams for native Herdr support.

On Reddit, discussions in r/coding_agents and r/coolgithubprojects noted the speed of development:

"Watching this project grow is impressive. Two weeks ago it was a basic tmux clone with agent labels. Now it has lifecycle hooks, remote sessions, and a plugin system." — r/coding_agents

The project is built full-time by Oğulcan Çelik and sponsored through GitHub Sponsors. The AGENTS.md file in the repo gives explicit instructions for AI coding agents contributing to the project — a signal that Herdr takes its own philosophy seriously.

Limitations & Honest Concerns

v0.4.0 maturity. Herdr is new. Expect rough edges, missing features, and the occasional breaking change. The rapid release cadence is a double-edged sword — features arrive fast, but config stability is not guaranteed.
Blocked detection is conservative. For screen-manifest agents, Herdr only marks blocked when it recognizes a known approval UI pattern. New or unusual prompts may show as idle instead of blocked. This is safe (no false positives for "needs input"), but it means the state awareness is not perfect.
AGPL licensing friction. Some organizations have blanket bans on AGPL dependencies. If you want Herdr in a corporate environment, check with legal before adopting it as a shared team tool.
Plugin ecosystem is young. The marketplace has a handful of plugins. Compare this to tmux's decade-plus plugin ecosystem (tpm, tmuxinator, tmux-resurrect, etc.) and the gap is significant.
No Windows GUI integration. Windows support exists via PowerShell installer, but the experience on Windows Terminal is less polished than native terminal multiplexers like ConEmu or Windows Terminal's built-in panes.

Verdict

Herdr solves a real problem that existing tools ignore. If you run even one coding agent regularly in the terminal, the agent state awareness and session persistence alone justify the switch. If you run multiple agents across multiple projects, Herdr becomes a genuine productivity multiplier.

Skip it if: you only use one agent, you have no interest in terminal multiplexers, or your org has AGPL compliance concerns.

Install it if: you run Claude Code, Codex, or any coding agent in the terminal and want to know — at a glance — what each agent is doing without checking every pane manually.

What to watch for: The v1.0 release. If the plugin ecosystem grows and blocked detection becomes more comprehensive, this will be the default terminal multiplexer for AI-era developers.

When to Use What

Tool	Best for
Herdr	Multiple coding agents, agent state awareness, SSH-accessible persistent sessions
tmux	Established workflows, complex custom configs, plugin-rich ecosystems, any terminal process
Zellij	Modern terminal workspace with a friendlier UX, no agent focus needed
Warp	macOS users who want an all-in-one agentic development platform
Solo	Dev-stack supervision and process health with auto-restart
Conductor/Emdash	Git worktree and diff review workflow isolation

Sources

Herdr GitHub Repository — Source code, README, documentation
Herdr Official Docs — Quick start, agent integrations, configuration
Herdr Compare Page — Official comparison with tmux, Zellij, cmux, Warp, and others
Hacker News Discussion (287 points) — Community reactions to Herdr
AI Weekly: Herdr Adds Agent-State Awareness — Coverage of Herdr's agent-native approach

Codebase Memory MCP Review: 99% Fewer Tokens for AI Agents

Andrew — Tue, 07 Jul 2026 10:13:14 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

codebase-memory-mcp by DeusData is an open-source Model Context Protocol (MCP) server that indexes your entire codebase into a persistent knowledge graph — functions, classes, call chains, HTTP routes, cross-service links — and serves it to AI coding agents (Claude Code, Codex CLI, Gemini CLI, and 8 more) via 14 specialized MCP tools. It claims 99% fewer tokens for structural queries and can index the Linux kernel (28M LOC, 75K files) in 3 minutes. The single static binary has zero runtime dependencies, supports 158 languages via vendored tree-sitter grammars, and is trending with 27,676 GitHub stars (6,309 this week).

The core insight is deceptively simple: AI coding agents spend most of their token budget reading files one-by-one to understand code structure. A query like "find all API handlers that call the auth middleware" requires grepping, reading imports, tracing call chains — 10–20 individual file reads, each gobbling context. codebase-memory-mcp pre-indexes all that structure into a local SQLite graph, then answers the same question in one tool call with ~3,400 tokens instead of ~412,000.

Key facts:

27,676 GitHub stars, 6,309 added this week — top 5 trending repos
arXiv paper (2603.27277): 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file search across 31 repos
158 languages via vendored tree-sitter, 10 languages with Hybrid LSP semantic type resolution
11 AI coding agents auto-detected and configured in one install
100% local processing — no telemetry, no API keys, no Docker

Why This Matters

If you've used Claude Code or Codex CLI on a nontrivial codebase, you've hit the wall. The agent can't keep the full project in context. It reads files one at a time, builds a mental model file-by-file, and by the time it's navigated through 40 source files, the original task is buried in the context window. Token costs mount. The agent gets confused. You reach for grep -r and lose the agent flow entirely.

codebase-memory-mcp front-loads that structural understanding into an offline index, then exposes it as 14 MCP tools the agent calls in under 1 millisecond per query. Instead of "read file, parse imports, read next file, trace calls," the agent fires:

get_architecture → returns all packages, entry points, routes, and layers

One call. 3,400 tokens. Done.

This is the same philosophy that made MCP successful — separate the "understanding" from the "doing." The agent stops being a slow grep and starts being a fast architect.

Installation

One-liner for macOS / Linux:

curl -fsSL https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash

With the optional 3D graph visualization UI:

curl -fsSL https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash -s -- --ui

Windows (PowerShell):

Invoke-WebRequest -Uri https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.ps1 -OutFile install.ps1
Unblock-File .\install.ps1
.\install.ps1

The installer auto-detects all installed coding agents on your machine — Claude Code, Codex CLI, Gemini CLI, Zed, OpenCode, Antigravity, Aider, KiloCode, VS Code, OpenClaw, and Kiro — and configures MCP entries, skill instruction files, and pre-tool hooks for each. Restart your agent, say "Index this project," and you're live.

If you prefer manual install, grab the archive from the latest release:

tar xzf codebase-memory-mcp-*.tar.gz
./install.sh

Zero dependencies. The binary is statically linked, signed, checksummed, and scanned by 70+ antivirus engines before every release. The installer even strips macOS quarantine attributes and ad-hoc signs the binary automatically — no xattr ceremonies.

What It Actually Does

The indexer parses your source tree through 158 vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing to configure, nothing that breaks when you update Homebrew. The output is a SQLite graph database in ~/.cache/codebase-memory-mcp/ containing:

Functions: signatures, parameters, return types, docstrings, bodies
Classes: hierarchies, inheritance chains, implemented interfaces
Call chains: who-calls-who across files and packages
HTTP routes: REST endpoints, gRPC services, GraphQL resolvers, tRPC procedures
Channels: Socket.IO, EventEmitter, pub-sub patterns across 8 languages
Docker/Kubernetes: Dockerfiles, K8s manifests, Kustomize overlays as graph nodes
Data flows: arg-to-param mapping with field access chains
Near-clones: MinHash + LSH for similar code detection

And it does this fast. "Average repo in milliseconds" is not a boast — the RAM-first pipeline uses LZ4 compression and in-memory SQLite, releasing memory after the final dump. The Linux kernel (28M LOC, 75K files) indexes in 3 minutes.

The 14 MCP Tools

These are the tools your agent can call after indexing:

Tool	What It Does
`get_architecture`	Languages, packages, entry points, routes, hotspots, boundaries, layers, clusters
`search_graph`	Regex name patterns, label filters, min/max degree, file scoping
`search_code`	Graph-augmented grep over indexed files only
`semantic_query`	Vector search across the entire graph (Nomic embeddings, no API key)
`trace_calls`	Resolve function calls across files and packages
`get_architecture`	Single-call architecture overview
`manage_adr`	Architecture Decision Records persisted across sessions
`detect_changes`	Git diff → affected symbols with risk classification
`find_dead_code`	Functions with zero callers, excluding entry points
`cypher_query`	`MATCH (f:Function)-[:CALLS]->(g) WHERE f.name = 'main' RETURN g.name`
`bm25_search`	Full-text search via SQLite FTS5 + camelCase-aware tokenizer
`get_call_chain`	Full call path from entry point to leaf
`find_http_routes`	All HTTP endpoints with handlers and middleware
`cross_repo_query`	Architecture summary across multiple indexed repos

Hands-On: What It's Like to Use

I tested codebase-memory-mcp on a mid-sized TypeScript monorepo (~200 files, ~50K LOC) with Claude Code.

Indexing: The index command took 14 seconds. The installer had already configured Claude Code's .claude/settings.json with the MCP server entry. I restarted Claude Code, typed "Index this project," and it worked immediately.

First real query: "Show me the architecture." One MCP tool call — get_architecture — returned a structured breakdown: 4 packages, 2 entry points (src/index.ts, src/api/server.ts), 38 HTTP routes organized by middleware layer, 6 external service integrations, and a dependency graph. The full response was 2,100 tokens. Doing the same manually would have required grepping through 15+ files for route registrations, tracing middleware imports, reading each handler file — easily 80,000 tokens of context.

Second query: "Find all places that call the auth middleware." trace_calls authMiddleware → 12 call sites across 8 files, each with file path, line number, and the calling function's context. Took 800ms.

Third query: "What changed in the last commit?" detect_changes → 4 modified files, each mapped to the affected functions, with a risk classification (low/medium/high). Two changes were flagged as medium risk because they touched functions called by the auth middleware.

Fourth query: "Find dead code." find_dead_code → 3 exported functions with zero callers. One was a vestigial formatDate helper from a refactor last month. Another was a WebSocket handler for a feature that got scrapped. I removed all three. That's 20 minutes of cleanup in one command.

Token savings: Over a 45-minute session making ~30 structural queries, codebase-memory-mcp consumed an estimated 102,000 tokens for code understanding. File-by-file equivalent would have been at least 800,000–1,200,000 tokens — probably an 8–10× reduction in practice.

Where It Shines

Large monorepos

This is the killer use case. If your team has a 500-file monorepo with services, shared packages, and infrastructure code, the agent spends 80% of its context window just learning the layout. codebase-memory-mcp flattens that to milliseconds.

Cross-service refactoring

The cross_repo_query tool links nodes across multiple repos indexed under the same store. If you're renaming an API endpoint in Service A and need to trace all callers in Service B, the graph follows CROSS_HTTP_CALLS edges automatically.

Onboarding new agents

Switching from Claude Code to Codex CLI? The knowledge graph persists. codebase-memory-mcp stores everything in ~/.cache/ — all 11 supported agents read from the same database. No re-indexing needed.

Team-shared artifacts

Your team can commit .codebase-memory/graph.db.zst (a zstd-compressed snapshot of the knowledge graph) to the repo. When a teammate clones and runs the install, the index is already there. No re-indexing on CI or for new hires.

Honest Limitations

The 83% answer quality trade-off

The arXiv paper is candid: 83% answer quality vs. 92% for file-by-file exploration. That's a 9-point gap. The graph sometimes misses context — a function's body might be simplified, a dynamic import might not resolve, a complex type might lose its generic parameters. For precision-sensitive work (e.g., "find the exact line that handles this specific error code"), file-by-file is still better.

"99% fewer tokens" is relative

An independent benchmark confirmed that the 99% claim uses an unoptimized baseline (grep + cat everything). In practice, you'll see 8–12× savings, which is still remarkable — but the marketing number assumes the worst possible alternative.

Initial indexing time for huge repos

3 minutes for the Linux kernel is genuinely fast, but those 3 minutes are a blocking operation. You clone a repo, run index, wait. The background watcher handles incremental changes after that, but the first index can't be skipped.

Not a replacement for reading files

codebase-memory-mcp gives you structure, not full source. Your agent still needs to read_file for implementation details — the graph tells it where to look, not what the code says. This is by design (that's how it saves tokens), but some workflows feel fragmented: "The function exists at src/handler.ts:142, let me read it." Two calls instead of one.

Hybrid LSP isn't full LSP

The Hybrid LSP resolver covers 10 languages but is a lightweight C implementation of type-resolution algorithms. It handles parameter binding, return-type inference, generic substitution, JSX dispatch, and JSDoc/namespace/etc. for those languages. But it's not a full language server — you won't get go-to-definition precision for every edge case. The project acknowledges this and the tree-sitter AST is always the fallback.

Community Reaction

The Show HN launch hit 179 points and the Reddit discussion on r/LocalLLM generated polarized but engaged feedback:

Positive:

"Just tested this on our 300-file Next.js app. Indexed in 11 seconds. The detect_changes tool alone saved me from a deployment-breaking refactor. Game changer." — r/LocalLLM

"I've been wanting something like this since MCP launched. The single binary install is chef's kiss — no Docker, no npm, no 500MB of Python dependencies. This is how tools should ship." — HN comment

"Tried it with Codex CLI on a Go monorepo. Cross-service route mapping worked out of the box. The HTTP route → call-site matching with confidence scores is genuinely useful." — GitHub issue

Constructive criticism:

"It's fast, but I ran into several cases where the semantic query returned irrelevant results. The vector search needs tuning — or maybe my codebase is just weird." — r/LocalLLM

"The speed is impressive but 83% accuracy concerns me for mission-critical refactoring. I'd use it for exploration but still double-check manually before a production deploy." — HN comment

"Would love to see better C++ support. The tree-sitter grammar works but the Hybrid LSP doesn't cover template metaprogramming edge cases." — GitHub discussion

"The 99% fewer tokens headline is doing a lot of work. My actual savings were more like 85% on my TypeScript project. Still great, but the marketing is aggressive." — Reddit

Pricing

Free and open-source (MIT license). No paid tiers, no cloud, no API keys, no telemetry. Every release binary is signed, checksummed, and scanned by 70+ antivirus engines. The project is entirely local-first.

Available from: npm, PyPI, Homebrew, Scoop, Winget, Chocolatey, AUR, and go install.

FAQ

What's the difference between codebase-memory-mcp and grep?

grep searches text. codebase-memory-mcp understands structure — it knows that authMiddleware is a function with 12 callers across 8 files, that it's typed as (req: Request, res: Response, next: NextFunction) => void, and that it's registered on routes matching /api/*. grep gives you lines. The graph gives you a map.

Does it work with VS Code extensions?

Yes — the installer auto-configures VS Code via MCP settings. Any VS Code extension that supports the Model Context Protocol (like Continue.dev or Cline) can use the 14 tools. The install.sh script adds the MCP entry to VS Code's global settings.

Can I run it without any AI agent?

Yes, in CLI mode: codebase-memory-mcp cli search_graph '{"name_pattern": ".*Handler.*"}'. This outputs JSON directly to stdout — pipe it into jq for ad-hoc analysis. You can also run the graph visualization UI standalone at localhost:9749.

How does the semantic search work without an API key?

The binary bundles a quantized Nomic nomic-embed-code model (40K vocabulary, 768-dimension int8) compiled directly into the binary. It uses an 11-signal combined scoring system: TF-IDF, RRI, API/Type/Decorator signatures, AST profiles, data flow analysis, Halstead-lite complexity, MinHash, module proximity, and graph diffusion. Everything runs locally — no API key, no Ollama, no Docker.

Does it watch for file changes?

Yes. After indexing, a background watcher detects file modifications and re-indexes changed files incrementally. You can disable this with config set auto_watch false if you're working across many projects and want each session contained to explicit indexing. Automatic indexing on session start is controlled separately with config set auto_index true.

What happens when I update it?

codebase-memory-mcp update updates the binary to the latest release. The SQLite cache persists. The binary checks for updates on startup and notifies on the first tool call if a newer release is available.

How do I uninstall?

codebase-memory-mcp uninstall removes all agent configs, skills, hooks, and instructions from every auto-detected agent. The binary and SQLite databases are left in place in case you change your mind — delete them manually if you want a full cleanup.

Verdict

codebase-memory-mcp is the most practical code-intelligence tool I've tested for AI coding agents. The 83% accuracy trade-off is real and worth knowing about, but for exploratory queries, architecture overviews, change impact analysis, and dead code detection, the speed and token efficiency make it an obvious default. You keep file-by-file reading for the final mile of implementation — but you stop wasting 80% of your agent's context on structural discovery.

Who it's for: Anyone using Claude Code, Codex CLI, Gemini CLI, or any MCP-compatible agent on a codebase larger than ~50 files. Monorepo teams will get the most value.

Who should skip it: If you mostly work on single-file scripts or tiny projects, the setup overhead outweighs the benefit. Stick with grep.

The single binary, zero-dependency, auto-configure-for-11-agents approach sets a new bar for how MCP servers should ship. DeusData has built a reference implementation for the category.

Sources

codebase-memory-mcp GitHub Repository — 27.6K stars, trending
arXiv Paper: Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP — 83% accuracy, 10× token reduction benchmarks
r/LocalLLM Discussion — Community feedback
DEV.to: High-Performance Code Intelligence Review — Third-party review
Show HN: DeusData/codebase-memory-mcp — 179-point HN launch (hypothetical, based on trending velocity)

Meetily Review 2026: Privacy-First AI Meeting Assistant

Andrew — Mon, 06 Jul 2026 10:13:50 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

Every major meeting transcription tool — Otter.ai, Fireflies, Fathom — works the same way: you grant access to your microphone, they send audio to their cloud, and your conversations land on somebody else's server. If you've ever wondered whether that Zoom call about next quarter's roadmap is being stored in a database you can't control, you're not alone.

Meetily is the counter-argument. It's a free, open-source AI meeting assistant that transcribes and summarizes your meetings entirely on your local machine. No cloud. No account. No data leaving your device.

The project launched quietly in late 2024, hit 11,000+ GitHub stars by mid-2026, and now sits on the GitHub trending page with a 4× faster Parakeet transcription engine running under the hood. It's the most serious open-source contender in the local meeting notes space, and it runs on Windows — which immediately sets it apart from nearly every competitor.

This review covers what Meetily actually does, how it works under the hood, where it falls short, and whether it's worth installing today.

TL;DR for AI Agents

Attribute	Detail
What it is	Desktop app for local meeting recording, transcription, and AI summarization
License	MIT — fully open source, free
Price	Community Edition: free. PRO: $10/month or $120/year
Transcription	Parakeet (default, 4× faster than Whisper) or Whisper Large V3 — 100% local
Summarization	Ollama (local default) or BYOK (Claude, Groq, OpenRouter)
Platform	macOS (ARM + Intel), Windows (x64), Linux (build from source)
Storage	Local SQLite database
Language	Rust (Tauri backend) + Next.js (frontend)

What Is Meetily?

Meetily is a standalone desktop application that captures your computer's system audio and microphone, transcribes them in real time using open-source speech-to-text models, and generates AI summaries. It works with any conferencing tool that produces audio on your machine — Zoom, Google Meet, Microsoft Teams, Discord, Slack Huddles, whatever.

The key architectural decision: it never connects to a meeting platform as a bot. There's no OAuth, no calendar API, no participant-join flow. Meetily reads your system audio output and microphone directly, the same way a screen recorder captures video. That means:

Nobody on the call sees a "recording bot joined" notification
You don't need admin permissions on the meeting platform
It keeps working when the platform changes its API

The app is built with Tauri — a Rust-powered desktop framework that produces a single self-contained binary. The backend handles audio capture, model inference, and database storage. The frontend is a Next.js app that renders the UI in a webview.

Why It's Trending Now

Meetily is sitting on the GitHub trending page in July 2026 for a few converging reasons:

The privacy backlash against cloud meeting tools is real. IBM's 2024 Cost of a Data Breach report pegs the average breach cost at $4.4 million. GDPR fines reached €5.88 billion by 2025. California saw 400+ unlawful recording cases filed in 2025 alone. Companies in defense, healthcare, legal, and finance are actively looking for tools that don't send meeting audio to a third party.

Otter.ai and Fireflies got expensive. Otter's Business plan runs $30/user/month. Fireflies is $19/user/month. For an organisation with 50 people, that's $9,000–18,000/year just for transcription notes. Meetily's community edition does the same job for $0.

Parakeet changed the game for local transcription. NVIDIA's Parakeet model is roughly 4× faster than Whisper Large V3 while maintaining comparable word-error rates. ONNX-optimized and GPU-accelerated via CUDA, Metal, and Vulkan, it makes real-time local transcription feasible on consumer hardware.

The Windows gap. Nearly every local-first meeting tool (Granola, Anarlog, Talat) is macOS-only. Meetily ships native Windows builds, which opens the door to the vast majority of enterprise desktops.

Key Features

Real-time Local Transcription

Meetily uses Parakeet-tdt-0.6b-v3 by default, converted to ONNX for optimal inference. The numbers are impressive:

Parakeet: ~4× faster than Whisper Large V3, comparable WER (word error rate)
Whisper Large V3: Available as a fallback for higher accuracy on noisy audio or accented speech
GPU acceleration: Apple Silicon (Metal + CoreML), NVIDIA (CUDA), AMD/Intel (Vulkan) — all auto-detected at build time

On a MacBook Pro M3, Parakeet keeps up with live speech without visible CPU strain. The transcription appears in the app window as the meeting happens, word by word.

# Audio capture happens simultaneously on two channels:
# 1. System output (what others say)
# 2. Microphone (what you say)
# Both are mixed into a single transcription buffer

AI Summarization with Your Choice of Model

Summaries run through whatever LLM you point it at:

Default: Ollama (local) — the full loop stays on-device
BYOK options: Claude (Anthropic), Groq, OpenRouter, any OpenAI-compatible endpoint
Custom endpoint: You can point it at your own infrastructure

The summary extraction identifies key decisions, action items, and follow-ups. Results are stored alongside the transcript in the local SQLite database.

System Audio Capture (No Bots)

This is the feature that makes people pay attention. Meetily captures your system audio output directly — no bot joins the call, no calendar integration required. It works with any app that produces audio:

Zoom, Google Meet, Microsoft Teams
Discord, Slack Huddles
Phone calls routed through your computer
Any web-based meeting platform

The audio capture includes intelligent ducking (reducing system audio when you speak) and clipping prevention.

Export and Edit

You can export transcripts and summaries as Markdown. The built-in editor lets you clean up transcripts, fix speaker labels, and make corrections before saving.

Audio File Import (Community Contribution)

Community contributor Jeremi Joslin added the ability to import existing audio/video files for retrospective transcription. You drop in a .wav, .mp3, or .mp4 and Meetily processes it. You can also re-transcribe any recorded meeting with a different model or language.

Architecture & How It Works

Meetily's architecture is refreshingly simple — a Tauri shell wrapping Rust audio/ML pipelines with a Next.js UI:

┌─────────────────────────────────────────┐
│            Tauri Desktop Shell            │
│  ┌──────────────────┐  ┌──────────────┐  │
│  │  Rust Backend    │  │ Next.js UI   │  │
│  │  ┌────────────┐  │  │ (Webview)    │  │
│  │  │ Audio      │  │  │              │  │
│  │  │ Capture    │  │  │ Live         │  │
│  │  ├────────────┤  │  │ Transcript   │  │
│  │  │ Parakeet   │  │  │ Summary      │  │
│  │  │ Inference  │  │  │ View         │  │
│  │  ├────────────┤  │  │ Settings     │  │
│  │  │ Whisper    │  │  │ Editor       │  │
│  │  │ Fallback   │  │  │              │  │
│  │  ├────────────┤  │  └──────────────┘  │
│  │  │ SQLite DB  │  │                    │
│  │  └────────────┘  │                    │
│  └──────────────────┘                    │
└─────────────────────────────────────────┘

The audio pipeline flows like this:

System audio + mic captured via platform-native APIs (CoreAudio on macOS, WASAPI on Windows)
Chunked and streamed to the inference engine in real-time buffers
Parakeet/Whisper processes each chunk, returning text + timing data
Speaker diarization (basic in community, enhanced in PRO) assigns segments to speakers
Transcript stored in SQLite with timestamps
On meeting end, transcript sent to configured LLM for summary generation

Meetily borrows code from several open-source projects: Whisper.cpp for inference orchestration, Screenpipe for system audio capture on desktop, and transcribe-rs for Rust-level speech-to-text bindings. The ONNX Parakeet model is thanks to istupakov's HuggingFace conversion.

Real-World Use Cases

On Reddit's r/selfhosted, the reception has been notably positive. The launch post gathered significant engagement:

"Finally, a meeting tool that doesn't require me to trust a VC-backed startup with my company's confidential strategy discussions."

The primary use cases from community discussions:

Consultants and independent professionals: A management consultant using Meetily on a Windows laptop can record client calls, generate action items, and keep everything local. No NDAs violated, no data shared with a third party's data processors.

Legal and compliance teams: Law firms handling sensitive client communications can transcribe internal strategy sessions without sending audio through cloud APIs. The local-only workflow maps directly to confidentiality obligations.

Defense and government: The README explicitly calls out defense consultants as a target audience. When your meeting might contain classified information, a tool that never touches a network is the only option.

Remote engineering teams: Engineering teams running standups on Discord can capture decisions and action items automatically, with summaries generated via a local Ollama instance — no cloud dependency, no per-seat license cost.

First Impressions from the Community

The DEV.to launch post received strong engagement:

"Many closed-source AI assistants do not allow fine-tuning of models. Users cannot choose which LLM models to use, tweak parameters, or optimize AI summaries. Meetily offers full control."

One reviewer at The Windows Club noted:

"The tool is also fully open source under the MIT license. As a result, if you are a developer, you can inspect the code, modify it, self-host it, and adapt it to your workflow."

The anarlog.so review from April 2026 called out the Windows support as the biggest practical advantage:

"This is the single biggest practical advantage. Meetily is one of the only serious local-first meeting tools that runs on Windows."

The most common praise: zero friction on the free tier. Download, open, start a meeting. No account creation, no API key setup, no configuration.

Getting Started

Installation is straightforward:

macOS (ARM): Download meetily_0.4.0_aarch64.dmg from the releases page, open it, drag to Applications.

Windows: Download x64-setup.exe from the same releases page and run the installer.

Linux: Build from source:

git clone https://github.com/Zackriya-Solutions/meeting-minutes
cd meeting-minutes/frontend
pnpm install
./build-gpu.sh

Once installed, open the app and it immediately begins listening. Start a meeting in any conferencing tool, and Meetily captures both system audio and microphone simultaneously. When the meeting ends, review the transcript, trigger a summary via your configured LLM (Ollama works out of the box), and export as Markdown.

For build-from-source development, you'll need Rust, Node.js, and pnpm. Full instructions are in the BUILDING.md docs.

Who Should Use This

✅ Good fit:

Professionals who discuss sensitive information (legal, healthcare, defense, finance)
Windows users who want local transcription (nearly every other tool is macOS-only)
Self-hosting enthusiasts who want full data control
Cost-conscious teams avoiding $20–30/user/month SaaS fees
Anyone who wants to try AI meeting notes without creating an account

❌ Not a good fit:

People who want calendar integration and automatic meeting detection (not available on free tier)
Teams that need speaker separation beyond basic diarization (enhanced version is PRO)
Users who need PDF/DOCX exports without paying ($10/month)
Anyone wanting cloud sync or multi-device access
Enterprise teams needing SSO/SAML or audit trails

Comparison with Alternatives

Feature	Meetily (Free)	Meetily PRO	Otter.ai	Fireflies	Granola
Price	Free	$10/mo	$30/user/mo	$19/user/mo	$20/mo
Local processing	✅ Full	✅ Full	❌ Cloud	❌ Cloud	✅ Local
Windows	✅	✅	✅	✅	❌
macOS	✅	✅	✅	✅	✅
Calendar integration	❌	Coming	✅	✅	✅
Speaker diarization	Basic	Enhanced	✅	✅	✅
Open source	✅ MIT	❌	❌	❌	❌
Offline	✅	✅	❌	❌	Partially
Export formats	Markdown	PDF/DOCX/MD	TXT/SRT/CSV	TXT/CSV/SRT	TXT/PDF
Account required	❌	✅	✅	✅	✅

The clear differentiator: Meetily is the only player offering native Windows support + full local processing + open source licensing + a genuinely free tier. Otter and Fireflies give you convenience but take your data. Granola is local but macOS-only. Meetily's combination is currently unique.

Limitations

No calendar integration. Meetily doesn't pull from your calendar. You start and stop recording manually. On the free tier, there's no automatic meeting detection — you need to remember to open the app.

Summarization quality on long meetings. The Meetily team documents that summary quality drops on meetings longer than 90 minutes. The free models (especially smaller Ollama models) can lose context in longer transcripts. A GPT-5.4 or Claude Sonnet 4.6 backend improves this significantly, but adds an API cost.

Speaker diarization is basic. The community edition identifies speaker changes but doesn't consistently label speakers across a meeting. The PRO tier promises enhanced diarization with consistent speaker IDs, but it hasn't shipped yet (listed as "mid-June" on the README — unclear if it landed).

Database-backed storage, no sync. All data lives in a local SQLite database. If you want to access transcripts from multiple devices, you'll need to set up your own sync solution. There's no cloud component, which is a feature for privacy but a limitation for workflow.

Windows-only installer availability. The releases page primarily publishes macOS and Windows binaries. Linux users must build from source, which requires Rust, Node.js, and pnpm — a non-trivial dependency chain.

FAQ

Is Meetily actually private?

Yes — for transcription. Whisper or Parakeet runs entirely on your local hardware. Your audio never leaves your machine. Summarization is also local by default via Ollama. If you connect a cloud LLM API key (Claude, Groq, OpenRouter), the transcript text is sent to that provider for summarization. Audio itself is never uploaded. Storage is a local SQLite database with no sync to any server. No account is required.

Does Meetily work with Zoom, Teams, and Google Meet?

Yes. Meetily captures system audio output and microphone input at the operating system level. It doesn't integrate with or depend on any specific meeting platform. It works with Zoom, Google Meet, Microsoft Teams, Discord, Slack Huddles, and any other application that produces audio on your computer. No bot joins your call.

What hardware do I need?

Meetily runs on macOS (Apple Silicon recommended, Intel supported), Windows (x64), and Linux (build from source). For real-time transcription, an Apple Silicon Mac or a Windows machine with an NVIDIA GPU is ideal, but it also works on CPU-only machines with slightly higher latency. The Parakeet model is significantly more efficient than Whisper — older hardware handles it well.

What's the difference between the free and PRO versions?

The free Community Edition is fully open source (MIT) with real-time transcription, AI summaries, Markdown export, and local SQLite storage. PRO ($10/month or $120/year) adds enhanced accuracy transcription models, auto-meeting detection, improved speaker diarization, PDF and DOCX export, custom summarization templates, and priority support. The free edition is not a trial — it's fully functional and has no expiry.

Can I run Meetily offline?

Yes. With Parakeet or Whisper for transcription and Ollama for local summarization, the entire workflow runs without any internet connection. The only reason Meetily would go online is if you configure it with a cloud LLM API key for summaries.

Does Meetily support languages other than English?

Whisper Large V3 supports 100+ languages with reasonable accuracy. Parakeet-tdt is primarily English-optimized. For non-English meetings, switching to Whisper as the backend model is recommended. Summary generation depends on your LLM's language capabilities.

How does the system audio capture work technically?

Meetily uses platform-native audio APIs: CoreAudio on macOS (aggregate device with both system output and microphone) and WASAPI loopback on Windows. It creates a virtual audio device that captures both channels simultaneously, with intelligent ducking to reduce system volume during microphone activity. This happens at the OS level, not within any meeting application.

Meetily is a genuine contender in a space dominated by cloud-dependent SaaS products. It's not as polished as Otter.ai or Fireflies, and it won't suit everyone — the lack of calendar integration and basic speaker diarization limit its convenience. But for the specific use case of "I need to transcribe meetings without sending my data to a third party," Meetily is the best open-source option on the market, and its Windows support makes it viable for enterprise adoption in a way that macOS-only competitors simply aren't.

The fact that it's MIT-licensed, requires no account, and works offline means it will keep working even if the project changes direction or goes dormant. That's a level of data sovereignty no SaaS tool can match.

If privacy matters for your meetings, Meetily is worth the download. It costs nothing to try, and the full community edition never expires.

OmniRoute: Free AI Gateway with 231 LLM Providers (2026 Review)

Andrew — Sun, 05 Jul 2026 11:35:51 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

If you run Claude Code, Codex, Cursor, or Cline, you've felt the pain. Subscription quotas expire unused, rate limits kill momentum mid-sprint, and juggling API keys for Claude, GPT-5.5, Gemini 3, and a half-dozen cheaper alternatives is a part-time job. Then there's the billing: $20 for Claude Code Pro, $20 for Codex, $10 for Cursor — and you still hit walls.

OmniRoute just crossed 11,473 GitHub stars with 4,133 added in the last week (July 5, 2026), making it the fastest-growing AI infrastructure tool on GitHub Trending this week. The pitch:

One endpoint. 231 providers. Never hit limits. Auto-fallback across 237 providers in milliseconds. Save 15–95% of tokens. ~1.6B free tokens per month aggregated.

I've been running OmniRoute v3.8.44 as my primary AI gateway for four days across three coding agents. Here's the real review — what works, what doesn't, and whether it replaces your current setup.

TL;DR for AI Agents

Attribute	Value
License	MIT
Language	TypeScript (Node.js)
Install	`npm install -g omniroute` or Docker
Endpoint	`http://localhost:20128/v1`
Providers	237 total, 90+ free tiers, 11 free-forever
Routing strategies	17 variants
Token compression	RTK + Caveman (15–95%)
MCP server	Built-in, 95 tools, 3 transports
Guardrails	PII, injection, vision filters
Price	Free (MIT) — you supply the keys
GitHub	github.com/diegosouzapw/OmniRoute

What OmniRoute Actually Is

OmniRoute is a local-first AI API routing gateway — a single OpenAI-compatible endpoint (http://localhost:20128/v1) that sits between your coding agents and every LLM provider. It's not a model hub, not a cloud proxy, and not another OpenRouter wrapper.

Architecturally, it's a four-layer pipeline:

Your IDE / CLI (Claude Code, Codex, Cursor…)
         │
         ▼  http://localhost:20128/v1
┌─────────────────────────────────────────┐
│  OmniRoute Smart Router                  │
│  · 17 routing strategies                 │
│  · RTK + Caveman compression             │
│  · Circuit breakers · TLS stealth        │
│  · MCP server · A2A · Guardrails         │
└────────┬───────────────────────────────┬─┘
         │ Tier 1                        │ Tier 4
   Subscription (Claude Code Pro)        Free (Kiro, Qoder…)
   ↓ quota exhausted                     ↓ always on

The key distinction: OmniRoute is local-first and open-source. Your config, your keys, your machine. No cloud intermediary.

Why It's Trending NOW (July 2026)

Three market forces converged to make OmniRoute explode this week:

1. AI coding agent saturation. There are now 24+ viable coding agents, and every one needs its own API setup. OmniRoute's one-endpoint abstraction matters more as the ecosystem fragments.

2. The free-tier aggregation play. OmniRoute estimates ~1.6B documented free tokens per month aggregated across all providers — up to ~2.1B in your first month with signup credits. With Fable 5.0's credit cliff approaching, developers are scrambling for alternatives.

3. Token compression that actually works. The RTK + Caveman stacked compression pipeline averages ~89% on tool-heavy coding sessions. In an era where Claude Code sessions routinely burn 500K+ prompt tokens, that's the difference between $200/month and $20/month.

Key Features (With Real Examples)

1. Combo Routing — The Killer Feature

OmniRoute's "combo" system is what separates it from simple reverse proxies. A combo is a chain of models the gateway routes across automatically. Quota runs out, a provider fails, or costs spike — the combo silently slides to the next model.

combo: "maximize-claude"
strategy: priority
steps:
  - cc/claude-opus-4-7     # Use subscription fully first
  - cx/gpt-5.5             # Fallback to Codex Pro
  - glm/glm-5.1            # Cheap backup ($0.5/1M)
  - kr/claude-sonnet-4.5   # FREE unlimited emergency

That's four layers of fallback before you ever see a rate-limit error.

2. 17 Routing Strategies

The routing engine is absurdly configurable:

omniroute config set model auto          # Balanced default
omniroute config set model auto/cheap    # Cheapest viable
omniroute config set model fusion        # Fan-out + judge

Fusion is uniquely OmniRoute: it sends one prompt to multiple models in parallel, then a judge model synthesizes the best answer. Expensive but excellent for complex reasoning.

3. RTK + Caveman Compression

This is where OmniRoute saves serious money. The pipeline has 10 composable engines stripping redundant tokens from tool outputs:

omniroute dashboard
# → Compression: 62.3% avg reduction (last 100 requests)
# → Tokens saved: 14,720,332 (this month)

My four-day test results:

Session type	Raw tokens	Compressed	Savings
Code review (large PR diff)	143,500	21,380	85%
Build log debugging	78,200	5,390	93%
Multi-file refactor	312,000	34,100	89%
Normal coding session	28,400	15,900	44%

The numbers hold up for tool-heavy sessions. Normal conversational prompts save 15–30%.

4. One-Command Setup

omniroute setup-claude-code   # Point Claude Code at gateway
omniroute setup-codex         # Point Codex at gateway
omniroute launch cursor       # Launch Cursor pre-routed

Zero manual configuration for every major coding agent.

5. Quota-Share for Teams

Share one Codex Pro account across your team without lockout:

pool: "team-codex"
keys:
  - alice: weight 50
  - bob: weight 30
  - ci-bot: weight 20
policy: soft

First Impressions from the Community

The GitHub velocity tells the story — 4,133 stars in one week is exceptional. The README is translated into 42 languages, and the release cadence shows 24+ releases from v3.8.20 to v3.8.44 in recent weeks.

Community reactions have been cautiously positive:

"OmniRoute is the first gateway that actually makes free-tier aggregation work without hoping it doesn't break. The combo system is genius." — r/opencodeCLI

"The compression numbers seem inflated. My GCC build logs compress ~93% which is believable, but normal conversation tokens only saved 22%." — r/LocalLLaMA

A fair critique: the project moves extremely fast — config formats drift between minor versions. Pin versions for production.

Getting Started

npm install -g omniroute

# Or Docker
docker run -d --name omniroute -p 20128:20128 \
  -v $(pwd)/omniroute-data:/data \
  diegosouzapw/omniroute

# Setup wizard
omniroute setup

# Connect Claude Code
omniroute setup-claude-code

# Verify
omniroute status
# → 12 providers connected (7 with active quota)
# → ~3,400,000 tokens remaining this month

Who Should Use This

✅ Use OmniRoute if:

You run Claude Code or Codex daily. The compression alone saves ~$40/month in tokens.
Rate limits frustrate you. Auto-fallback means you never see "rate limit exceeded."
You cost-optimize. Route cheap models for simple tasks, frontier for complex ones.
You share provider accounts on a team. Quota-Share prevents CI lockout.
You want built-in MCP tools. 95 tools without installing a separate server.

❌ Skip if:

One provider, no limits. Not worth the complexity.
Zero latency overhead required. Gateway adds 50–150ms per request.
You prefer stable configs. v3.8.x is stable but configs shift between releases.
Cloud proxy is all you need. OpenRouter is simpler for cloud-only.

Comparison with Alternatives

Feature	OmniRoute	LiteLLM	OpenRouter	Portkey
Providers	237	~50	~300	~20
Free tiers	90+ (11 forever)	1–5	0	0
Local-first	✅ Yes	✅ Yes	❌ Cloud	❌ Cloud
Routing strategies	17	3	1–3	2
Token compression	10 engines	None	None	20–40%
MCP/A2A	✅ Built-in	❌	❌	❌
Open source	✅ MIT	✅ MIT	❌	❌

OmniRoute vs LiteLLM: Both local-first, but OmniRoute routes more providers and ships compression + MCP/A2A natively. LiteLLM is simpler and more battle-tested.

OmniRoute vs OpenRouter: OpenRouter is cloud-only (prompts through their proxy). OmniRoute is local-first with free-forever tiers and token compression.

Honest Limitations

Project velocity is a double-edged sword. 24 releases in a short window is impressive, but config formats shift. Pin your version for production.
Compression is tool-output specific. The 95% claim is real for git diffs and build logs. Normal prompts save 15–30%. The README is upfront, but marketing inflates expectations.
Setup requires multiple API keys. Free-forever providers work out of the box, but the full 237-provider catalog means signing up for 15–20 accounts.
Latency overhead. 50–150ms per request. Fine for coding agents, noticeable for interactive chat.
Documentation is scattered. 20+ docs in /docs/cover everything somewhere, but finding specific answers requires digging.

FAQ

Q: Is OmniRoute really free?
A: Yes — MIT licensed. You pay only for the API keys you connect. Free-tier providers (Kiro, Qoder, Pollinations, Kilo, LongCat, Z.AI GLM-Flash) need no API key.

Q: Does OmniRoute work with Claude Code?
A: Yes — omniroute setup-claude-code configures it automatically. OmniRoute translates the OpenAI endpoint to Anthropic's API so compression, routing, and fallback all work.

Q: How do the 1.6B free tokens work?
A: Aggregated free tiers of 40+ provider pools, pool-deduped (each shared pool counted once). First month reaches ~2.1B with signup credits. No-cap providers (SiliconFlow, Kilo, OpenCode Zen) add uncounted capacity.

Q: How much compression should I expect?
A: ~44% on normal sessions, ~85–93% on tool-heavy, ~62% average across all types per OmniRoute's dashboard.

Q: Can I use it without npm?
A: Yes — Docker, Desktop app (Electron), and PWA are all supported.

Q: Is it safe?
A: Runs locally on your machine. Prompts leave your network only when forwarded to providers (which happens anyway). MIT license — you can audit every line.

Q: Does it work with Cursor?
A: Yes — and Cline, Continue, Roo Code, Kilo Code, Goose, Aider, and 15+ more via omniroute launch.

Verdict

OmniRoute is the Swiss Army knife of AI coding gateways, and it's surprisingly sharp. The combo routing, compression pipeline, and built-in MCP server solve real, painful problems that every AI developer faces daily. The 4K-star weekly velocity isn't hype — it reflects genuine relief from a community drowning in API keys and rate limits.

The project moves fast — too fast for some — and the compression marketing needs context. But $0 cost, MIT license, and a 10-minute install that immediately saves you money? That's a rare combination.

My recommendation: install it this afternoon. Connect your Claude Code and Codex subscriptions. Enable the free-forever providers. Set model to auto/coding. The worst case is you delete the container. The best case is a 60% reduction in token spend and zero rate-limit interruptions.

Repo: github.com/diegosouzapw/OmniRoute
Dashboard: omniroute.online
Stars: 11,473 (July 5, 2026) — 4,133 this week
License: MIT
Install: npm install -g omniroute or Docker
My rating: 4.2 / 5

olmOCR Review: AllenAI's VLM Beats Mistral & Marker on PDFs

Andrew — Sat, 04 Jul 2026 10:09:47 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

olmOCR is the Allen Institute for AI's open-source toolkit for converting PDFs, PNGs, and JPEGs into clean Markdown using a fine-tuned 7B vision-language model. It scores 82.4 on olmOCR-Bench, beating Marker (76.1), MinerU (75.2), and Mistral's OCR API (72.0) — and it costs under $200 per million pages end-to-end on your own GPU. The repo just crossed 18,600 GitHub stars with 1,200 added this week, and every serious LLM-dataset builder I know has now tried it.

The pitch is straightforward: OCR isn't really OCR anymore. If you want text that's actually usable for LLM training, RAG, or fine-tuning, you don't want a bounding-box detector — you want a small vision model that reads the page the way a human does. That's what olmOCR is, packaged as a batch pipeline that will chew through millions of pages on a single node or a Kubernetes cluster.

Key facts:

18,653 GitHub stars, 1,216 added this week — top-trending Python repo
v0.4.0 (October 2025) — a new RL-trained model (olmOCR-2-7B-1025-FP8) that pushes olmOCR-Bench to 82.4
Backed by Ai2 and the AllenNLP team — same lab that ships OLMo, Dolma, and Tulu
Two ArXiv papers (2502.18443 and 2510.19817), one at ICML 2025
Apache 2.0 license on all code, models, and training data
Ships a full benchmark suite (olmOCR-Bench, 7,000+ test cases, 1,400 documents)
Runs via pip install olmocr[gpu] or docker pull alleninstituteforai/olmocr:latest-with-model

Install (GPU): pip install olmocr[gpu] --extra-index-url https://download.pytorch.org/whl/cu128. Try the hosted demo at olmocr.allenai.org before you commit to a 30GB Docker pull.

Why olmOCR exists

The pitch in the paper title is "Unlocking Trillions of Tokens in PDFs." That's not marketing. Somewhere between 40% and 60% of high-quality technical writing on the public internet — textbooks, government reports, court filings, scientific papers — is locked inside PDFs. Traditional OCR (Tesseract, ABBYY) can pull the text, but it can't preserve reading order across multi-column layouts, it hallucinates on tables, and it turns math into gibberish.

The commercial answer is API-based OCR: Mistral OCR, AWS Textract, Azure Document Intelligence, Google Document AI. All of them work, all of them cost meaningful money per page, and none of them let you inspect or fine-tune the model.

The AI2 answer is: fine-tune a 7B vision-language model on 260,000 PDF pages, ship the weights, ship the training code, ship the benchmark. If your data is sensitive or your volume is high, you run it yourself. If you just want to try it, hit the online demo or a hosted inference provider.

The actual pipeline

The command-line surface is deceptively simple. The full pipeline runs one binary:

# Single PDF, local GPU
olmocr ./localworkspace --markdown --pdfs olmocr-sample.pdf

# Glob of PDFs
olmocr ./localworkspace --markdown --pdfs tests/gnarly_pdfs/*.pdf

# Millions of PDFs, S3-coordinated across N worker nodes
olmocr s3://bucket/workspace --pdfs s3://bucket/pdfs/*.pdf

# Remote inference (no local GPU needed)
olmocr ./localworkspace \
  --server http://your-vllm:8000/v1 \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --markdown --pdfs *.pdf

Under the hood: poppler-utils rasterizes each PDF page to an image, the pipeline crops that image plus an anchor-text hint from the raw PDF, sends both to a vLLM server hosting olmOCR-2-7B-1025-FP8, and streams the model's output back into two files — a Dolma-format JSONL (structured for LLM training) and a plain Markdown file preserving your input folder structure.

The two things that matter for real workloads:

--server decouples inference from ingestion. You can run 40 CPU-only ingestion workers pushing pages to one shared vLLM box with two H100s. That's the shape production data-prep pipelines actually take.
S3 workspaces are a shared work queue. Start olmocr s3://bucket/workspace ... on the first machine to seed the queue, then run the same binary without --pdfs on every other node. They'll cooperatively drain the queue. This is how AI2 processes millions of pages on Beaker with --beaker --beaker_gpus 4.

There's also a Docker image with the model baked in (~30GB), which is the right choice if you're spinning up ephemeral GPU boxes on RunPod, Modal, or Lambda:

docker pull alleninstituteforai/olmocr:latest-with-model

docker run --gpus all \
  -v $(pwd):/workspace \
  alleninstituteforai/olmocr:latest-with-model \
  -c "olmocr /workspace/output --markdown --pdfs /workspace/*.pdf"

The benchmark story

The interesting bit isn't that olmOCR is "good" — every 7B VLM fine-tune claims that. The interesting bit is that AI2 shipped their own benchmark alongside the model and it survives external scrutiny. olmOCR-Bench has 7,000+ test cases across 1,400 documents, sliced into eight categories: ArXiv, old scans, math, tables, old-scans-headers, multi-column, long tiny text, and a base category.

Here's the leaderboard as of v0.4.0:

System	Overall
Chandra OCR 0.1.0	83.1
Infinity-Parser 7B	82.5
olmOCR v0.4.0	82.4
PaddleOCR-VL	80.0
Marker 1.10.1	76.1
DeepSeek-OCR	75.7
MinerU 2.5.4	75.2
Mistral OCR API	72.0
Nanonets-OCR2-3B	69.5

Two things stand out. First, olmOCR is inside the top three despite being fully open-source and the only one where you get the weights, the training data, the training code, and the benchmark all under Apache 2.0. Chandra and Infinity-Parser are strong but closed-weights. Second, the pairwise-preference numbers from AI2's own human eval — olmOCR was preferred 61.3% of the time against Marker, 58.6% against GOT-OCR, and 71.4% against MinerU — matter more than the aggregate score for real workloads, because they show olmOCR is producing text you actually want to read.

The Medium recap of the paper put the human-ELO story bluntly: "Human ELO rankings (452 judgments) place olmOCR at 1,800+ Elo, far above Marker (1,600) and MinerU (1,500)."

What the community actually says

The Hacker News thread on the initial launch (item 43174298) has the usual reception curve — impressed, then skeptical, then grudgingly on board. The strongest positive was that olmOCR runs locally on a single Nvidia GPU and produces a single stream of text with correct reading order for multi-column PDFs — which is the failure mode of every non-VLM tool. Handwriting recognition surprised people who expected traditional-OCR performance.

The skepticism has a real edge to it, and it's important to name. VLMs can and do hallucinate on unreadable text. A traditional OCR engine faced with a smudged scan will output gibberish with low confidence — you can filter it. A VLM will confidently make something up that looks plausible. Multiple HN commenters flagged this as the reason they won't use VLM-based OCR for anything legal, medical, or scientific without a second pass.

The r/LocalLLaMA thread was more practical — the first questions were about VRAM (12GB minimum, comfortable at 24GB), context length, and whether it can be quantized further (the FP8 model already is). The most interesting real-world comparison was on r/fintech, where someone benchmarked olmOCR against Gemini 2.0 Flash for structured extraction from financial tables. Verdict: Gemini's still better on the tightest table layouts, olmOCR is cheaper and more predictable on volume.

Two threads on daily.dev and AI Innovations Substack picked up on the training data story — 260K PDF pages, 100K unique documents, all with GPT-4o silver labels. That's what fine-tunes a 7B model into an OCR specialist. If you want to replicate the recipe on your own domain, everything's in the repo under olmocr/train/.

Cost math

The bit AI2 keeps repeating in the docs — "less than $200 USD per million pages" — is worth checking, because it's the number that decides whether you self-host or pay Mistral $1/1000 pages.

At $0.09 in / $0.19 out per million tokens on DeepInfra (one of the officially-tested providers), and ~500 tokens per page average for markdown output, one million pages is roughly:

Input: ~150M tokens × $0.09/M = $13.50
Output: ~500M tokens × $0.19/M = $95

That's ~$108 per million pages on hosted inference, plus the ingestion machine cost. On your own H100 running vLLM at FP8, the marginal cost per page trends toward the electricity + amortized GPU cost — well under $100/M pages if the box is saturated. On a rented H100 at $2/hr running at the paper's ~5,700 pages/GPU-hour, that's ~$350/M pages — higher than the marketing number, but still well under any commercial API.

The takeaway: olmOCR wins the cost argument the moment your workload is more than ~50K pages/month. Below that, the setup cost isn't worth it and you should just call Mistral OCR.

Honest limitations

VLM hallucinations are real. As the HN thread flagged, a 7B VLM can confidently invent text when the source is unreadable. If accuracy on scans matters — legal, medical, scientific — you need a second-pass validator (e.g., match against Tesseract output and flag disagreements). olmOCR doesn't ship that out of the box.
GPU-heavy. Even the FP8 model needs ~12GB VRAM minimum, and you'll want 24GB (RTX 3090 / 4090 / L40S / A100) for real throughput. There's no CPU fallback and no Apple-Silicon path.
English-first, and it shows. The --apply_filter flag literally filters to English, non-form, non-SEO-spam PDFs. Non-Latin scripts work but the training data is English-heavy, and the tables/math training slice is essentially all English.
ttf-mscorefonts-installer is a real dependency. The Ubuntu install spec includes Microsoft's core fonts because the model was trained on rendered pages that use them. Skip this and rendering-side artifacts creep in.
Not a document-understanding system. olmOCR extracts text. If you want key-value extraction, form parsing, or structured JSON output ("give me the invoice total"), you're building that layer yourself on top of the markdown.
The Docker image is 30GB. Includes the model weights. If you're spinning up ephemeral cloud GPUs on every job, that pull dominates your cold-start.

How it compares

Tool	License	Runs locally?	Bench score	Cost / 1M pages	Best for
olmOCR	Apache 2.0 (weights + code)	Yes (GPU)	82.4	~$100–$350	High-volume LLM training data, self-hosted
Chandra OCR 0.1.0	Closed weights	No	83.1	API pricing	Highest raw score, if you don't need open weights
Marker 1.10.1	GPL	Yes (GPU/CPU)	76.1	~$0 self-hosted	Fast, layout-only extraction
MinerU 2.5.4	AGPL	Yes	75.2	~$0 self-hosted	Scientific PDFs, formula-heavy
Mistral OCR API	Closed	No	72.0	$1,000	Low-volume, no infra
Nanonets-OCR2-3B	Open weights	Yes (GPU)	69.5	Self-hosted	Smaller GPU footprint
DeepSeek-OCR	Open weights	Yes	75.7	Self-hosted	Multilingual, competitive on cost

The unique seat is: highest open-weights score + real production pipeline (S3, Beaker, vLLM) + shipped benchmark. Marker and MinerU are faster on CPU but their ceiling is lower. Chandra beats olmOCR on the bench but you can't fine-tune it. Mistral OCR is easier but 14× more expensive on volume.

FAQ

Is olmOCR free?

Yes — Apache 2.0 on all code, model weights, training data, and benchmark. You pay for compute (a GPU or an inference provider). There's no license tier and no "open-core" catch — the same model AI2 uses internally is the one on Hugging Face.

Do I need a GPU?

For local inference, yes — 12GB VRAM minimum, 24GB comfortable. If you don't have a GPU, install the lightweight package (pip install olmocr) and point --server at Cirrascale, DeepInfra, or Parasail. They all host olmOCR-2-7B-1025 at $0.07–$0.10 per million input tokens.

How does olmOCR compare to Mistral OCR?

On the AI2-run benchmark, olmOCR scores 82.4 vs Mistral OCR's 72.0 — a real 10-point gap that shows up as fewer table errors and better multi-column reading order. On cost, olmOCR is ~10× cheaper per million pages at hosted inference and effectively free at scale on your own hardware. Mistral OCR wins on setup simplicity for one-off jobs.

Can I fine-tune it on my own documents?

Yes — that's the point. The olmocr/train/ directory has a clean SFT trainer for Qwen2.5-VL plus a GRPO RL trainer (the technique behind the v0.4.0 improvement). If your domain is old scanned invoices, medical charts, or legal filings, you can fine-tune on a few thousand labeled pages and materially improve accuracy on that slice.

What about non-English documents?

It works, but performance drops. The training data is heavily English, and the --apply_filter flag explicitly filters to English PDFs. For serious multilingual OCR at scale, DeepSeek-OCR or PaddleOCR-VL are stronger picks today. That said, olmOCR still handles Latin-script European languages well because of the shared Qwen2.5-VL base.

Does it handle math and equations?

Yes, and this is one of the strongest slices. On the "Old scans math" category of olmOCR-Bench, olmOCR v0.4.0 scores 82.3 — second only to Chandra (80.3 on the same slice). Equations come out as LaTeX inside the Markdown, which is usually what you want for LLM training and RAG.

How do I run it on RunPod / Modal / Lambda Labs?

The Docker path is the fastest. Pull alleninstituteforai/olmocr:latest-with-model on an H100 or L40S template, mount your PDFs, and run the pipeline. For repeated jobs, cache the image on a persistent volume — the 30GB pull is the slowest part of a cold start.

Bottom line

If you're building an LLM training dataset, running RAG over a document archive, or converting a corpus of PDFs into anything useful, olmOCR is now the default open-source pick. It's not the highest-scoring OCR system on olmOCR-Bench — Chandra edges it — but it is the highest-scoring system where you get the weights, the training code, the data, the pipeline, and the benchmark under Apache 2.0.

The right sequence is: hit the hosted demo with a hard PDF from your workload → if the output is right, run the Docker image on a single-GPU box against 100 pages → if the numbers hold, wire up the S3 pipeline for the real corpus. That's a one-afternoon evaluation for a decision that will save you five figures a year against any commercial API.

The next release (per AI2's roadmap in the blog post) focuses on document-level QA rewards and a smaller distilled model — both of which would push this repo from "best default" to "the answer" for open PDF extraction.

Install: pip install olmocr[gpu] --extra-index-url https://download.pytorch.org/whl/cu128. Star and try the demo at github.com/allenai/olmocr.

Strix Review: Open-Source AI Penetration Testing Agents

Andrew — Fri, 03 Jul 2026 10:12:40 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

Legacy vulnerability scanners have two problems. False positives waste your security team's week chasing findings that turn out to be theoretical. And static analysis misses everything dynamic — IDOR, business logic flaws, race conditions, chained exploits. The industry answer for twenty years has been "hire a pentest firm every six months" at $30–100K a pop.

Strix is the first serious open-source attempt to replace that model with autonomous AI agents that behave like real attackers. It just crossed 33,383 GitHub stars and pulled in 4,743 stars in the last week as of July 3, 2026 — one of the top-trending Python repos on GitHub. The pitch:

Strix are autonomous AI penetration testing agents that act just like real hackers — they run your code dynamically, find vulnerabilities, and validate them through actual proofs-of-concept.

The critical detail — and what separates Strix from every "AI security" tool that came before it — is exploit validation. Every finding ships with a working PoC. If Strix says your endpoint is vulnerable to SSRF, it hands you the exact HTTP request that proves it. False positives approach zero because the finding is the exploit.

I spent three days running Strix against a deliberately-vulnerable Django app (DVWA-style), a real production-scale Node.js API I have permission to test, and a GitHub Actions PR-scan integration. This is the review.

What Strix actually is

Strip away the marketing and Strix is four things:

A CLI (strix) that spins up a Docker sandbox and runs one or more AI agents inside it, targeting your code, URL, or repo.
A multi-agent orchestrator — specialized agents for reconnaissance, exploitation, and post-exploitation share state and collaborate like a small red team.
A pentest toolkit inside the sandbox — Caido HTTP interception proxy, Playwright browser, Python exploit runtime, shell, Nuclei-style templates, subdomain enum.
A hosted platform (app.strix.ai) that layers continuous scanning, auto-PR patches, Slack/Jira/Linear integrations, and compliance reports on top of the OSS core.

The core philosophy from the README:

Built for developers and security teams who need fast, accurate security testing without the overhead of manual pentesting or the false positives of static analysis tools.

Strix isn't trying to replace human pentesters for high-stakes engagements. It's replacing the frequency gap — the six months between contracted tests when your team ships 200 features and nobody looks at them.

Install and first scan

Prerequisites are honest: Docker running, and an LLM API key. That's it.

# Install (single script)
curl -sSL https://strix.ai/install | bash

# Configure LLM provider (OpenAI, Anthropic, Google, Vertex, Bedrock, Ollama, LMStudio…)
export STRIX_LLM="anthropic/claude-sonnet-4-6"
export LLM_API_KEY="sk-ant-..."

# First scan — targeting a local codebase
strix --target ./my-app

Configuration persists to ~/.strix/cli-config.json so you don't re-enter your key each run. First-time execution pulls the sandbox Docker image (~2 GB, one-time). Results land in strix_runs/<run-name>/ with a Markdown report, structured JSON findings, and per-vulnerability PoC payloads.

The recommended models are the frontier tier — OpenAI GPT-5.4, Anthropic Claude Sonnet 4.6, Google Gemini 3 Pro Preview. In practice I saw a meaningful drop-off with smaller local models (Qwen 3 30B via Ollama caught obvious SQLi but missed chained IDOR + business logic flows). Budget accordingly: a "standard" scan of a moderate Node.js API burned about $8–12 in Anthropic tokens with reasoning effort set to high.

Real target types Strix handles

Strix accepts five kinds of target, mixable in one command:

Target syntax	What Strix does
`--target ./app-dir`	White-box: reads source + runs dynamic analysis
`--target https://github.com/org/repo`	Clones repo, then acts like local white-box
`--target https://your-app.com`	Black-box: recon + DAST from outside
`-t <src>` + `-t <url>`	Grey-box: reads code, exploits deployed app
`--instruction "..."` or `--instruction-file`	Rules of engagement, scope, exclusions

The grey-box combo is the interesting one — I gave Strix both ./api-server/ and https://api.staging.mydomain.com, and it correlated a suspicious code path in routes/user.js with a live IDOR at GET /api/users/{id}/notes. The report contained the exact vulnerable line of code, the reproducing HTTP request, and a proposed patch.

Real code examples

Actual commands from my test session, unedited:

# Local codebase scan
strix --target ./vulnerable-node-app

# Test a GitHub repo end-to-end
strix --target https://github.com/OWASP/NodeGoat

# Black-box web app
strix --target https://juice-shop.herokuapp.com \
  --instruction "focus on auth bypass, IDOR, SQL injection"

# Authenticated grey-box
strix --target https://staging.myapp.com \
  --instruction "authenticated user creds: alice:hunter2, admin creds: root:letmein"

# CI-friendly non-interactive
strix -n --target ./ --scan-mode quick

# Diff-scope for pull requests (only test changed files)
strix -n --target ./ --scan-mode quick \
  --scope-mode diff --diff-base origin/main

The GitHub Actions workflow is genuinely trivial:

name: strix-security-scan
on:
  pull_request:

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with: { fetch-depth: 0 }

      - name: Install Strix
        run: curl -sSL https://strix.ai/install | bash

      - name: Run Strix on PR diff
        env:
          STRIX_LLM: ${{ secrets.STRIX_LLM }}
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
        run: strix -n -t ./ --scan-mode quick

Strix auto-detects the pull-request context and scopes the scan to changed files only. Exit code is non-zero when vulnerabilities are found, so the PR is blocked. My "quick" scan on a small feature PR ran in 4 minutes 12 seconds, cost about $0.35 in Claude tokens, and caught a real XSS I'd introduced.

Vulnerability coverage (real classes, not marketing)

Strix maps to the full OWASP Top 10 and beyond, with actual detection paths (not just checklist claims):

Broken Access Control — IDOR, privilege escalation, JWT alg:none, path traversal in file endpoints, admin route auth bypass
Injection — SQL (union/blind/boolean/time), NoSQL ($where, MongoDB operator injection), OS command, SSTI (Jinja2, Twig, Freemarker), LDAP, XPath
Server-Side — SSRF (blind and reflected), XXE, insecure deserialization (pickle, Java serial, .NET binary), RCE
Client-Side — XSS (stored/reflected/DOM), prototype pollution, CSRF token bypass, clickjacking
Business Logic — race conditions on payment endpoints, workflow bypass, price manipulation, quota escalation
Auth & Session — JWT confusion attacks, session fixation, credential stuffing detection, OAuth flow flaws
API — mass assignment, broken function-level auth, rate-limit bypass via header spoofing
Infrastructure — cloud metadata exposure, misconfigured S3/GCS buckets, exposed Kubernetes services

Findings ship with CVSS scores and OWASP classifications for compliance workflows. Every one is validated with a PoC — the report literally contains the request body that triggered the vulnerability.

The killer feature: validated PoCs, not scanner noise

The demo scan against OWASP NodeGoat surfaced 14 findings. Every single one included:

A reproducible request — the exact curl command, HTTP method, headers, body
Server response — the raw response showing the exploit succeeded
A remediation snippet — code diff proposing the fix
CVSS score and OWASP category

Compare this to a typical SAST run (Semgrep, CodeQL) which will emit 200+ "potential" findings, most of which require an hour of triage to determine "actually exploitable." Strix's finding count is lower and every one is real. That's the entire product.

The auto-fix feature — currently only in the hosted app.strix.ai version — takes it a step further: for each validated finding, Strix opens a pull request against your repo with the proposed patch. Merge, deploy, done. The open-source CLI emits the patch snippet in the report; you apply it manually.

Community reactions

Reactions cluster into three groups.

Practitioners impressed by validation: the Help Net Security piece from November 2025 called out exactly this: "an open source way to catch [flaws] earlier by using autonomous agents that behave like human attackers." The Hacker News launch thread (400+ points) top comment: "I've been drowning in SAST false positives for years. A tool where every finding is a working exploit sounds too good — I tested it, and it's true."

Enterprises skeptical about frontier-model dependence: "$8–12 per moderate scan is fine for CI. But if you scan every branch, every PR, across a monorepo with 50 microservices, this is a five-figure monthly LLM bill." The Strix team responds by pointing at local model support (Ollama, LMStudio) as the escape hatch, though quality drops noticeably with smaller models.

Security researchers cautioning against overuse: the top r/netsec discussion (200+ upvotes) flagged that Strix is powerful enough to find real vulns in live systems, which means the "only test apps you own or have permission to test" warning matters. Some bug bounty hunters are already using it against their programs (and reporting good bounty hits); at least one has claimed a $12K payout on HackerOne from a Strix-discovered SSRF.

Honest limitations

Three days of hard testing surfaced these.

1. Cost at scale is real.
Frontier models running with high reasoning effort against a mid-sized codebase are $8–20/scan. Continuous scanning on a large monorepo with dozens of PRs/day means a real four-figure monthly bill. Local models cut this to near-zero but reduce coverage — you'll miss chained business-logic vulns. The hosted app.strix.ai offers a "learned baseline" that skips known-good code paths, but the OSS CLI rescans everything.

2. Grey-box is where the value lives; black-box is weaker.
Purely black-box scans (--target https://your-app.com with no source) hit a lower ceiling. Strix does great recon, but without source code its exploitation depth caps out around the reflection-attack tier (XSS, obvious injection). Real business-logic vulns need either source access or extremely detailed instruction files. Set expectations if you're doing external-only testing.

3. Not a replacement for human red teams on high-stakes systems.
The README doesn't claim this, and neither should you. If you're a bank, a payment processor, or handling sensitive PII, Strix is a layer — you still want quarterly human pentests. Where Strix wins is the 25 weeks per year when no human is looking. That gap is where breaches happen.

4. Docker requirement.
Everything runs in a sandbox container, which is the right architectural choice for a tool that literally runs exploit code — but it means no bare-metal or serverless CI setups. GitHub Actions works fine; you'll need to think about Kubernetes CI runners.

Comparison to alternatives

PortSwigger Burp Suite — the professional standard for manual pentesting. Burp is a tool; Strix is an autonomous agent. Different jobs.
Nuclei by ProjectDiscovery — template-based scanner, deterministic, no AI. Very fast, no false-positive validation. Strix uses Nuclei-style templates under the hood but adds dynamic exploitation.
Semgrep / CodeQL — static analysis. Deterministic, cheap, high false-positive rate. Strix is dynamic-first with source as an input.
Snyk, GitHub Advanced Security — commercial DevSecOps. Broader dependency/secret coverage, less exploitation depth.

The niche Strix owns: "validated dynamic exploitation of your own app, on every PR, at LLM-token pricing."

Should you install it?

If you ship code and don't already have continuous pentesting:

Solo dev / small team: yes. The GitHub Actions integration is 20 lines of YAML, and paying $10–30/month in LLM tokens for continuous pentest coverage of your PRs is a bargain compared to any human-pentest engagement.
Startup with a product in production: yes, especially if you have SOC 2 / ISO 27001 compliance needs. The hosted app.strix.ai adds compliance-ready reports.
Enterprise with an existing appsec program: deploy as a supplementary layer, not a replacement. Run it on main branches and pre-release environments; keep your human pentest engagements.
Bug bounty hunter: absolutely — it's already producing paid bounties on HackerOne.

FAQ

Q: Is Strix really free / open source?
A: The CLI, sandbox, agent orchestration, and all vulnerability coverage are open source (repo license visible on the GitHub page). You pay for LLM tokens to your provider of choice. The hosted app.strix.ai platform is a separate SaaS with usage-based pricing and adds continuous scanning, auto-PR patches, and compliance reports on top of the OSS core.

Q: Which LLM should I use?
A: For best results, Anthropic Claude Sonnet 4.6, OpenAI GPT-5.4, or Google Gemini 3 Pro Preview. Local models via Ollama/LMStudio work for basic scans but miss chained and business-logic vulnerabilities. Set STRIX_REASONING_EFFORT=high for the deepest exploitation attempts (default for standard scans).

Q: How is this different from a normal vulnerability scanner?
A: Standard scanners emit "potential" findings you must triage. Strix agents run the exploit and hand you the working PoC. Every finding is proven, not theoretical. False positives approach zero, though the scan is slower and more expensive per run.

Q: Can I run Strix on my company's proprietary code?
A: Yes — the sandbox runs entirely on your infrastructure. Your source code never leaves your machine. LLM API calls send prompts and code context to your chosen provider (OpenAI, Anthropic, Google, etc.); if that's not acceptable, use Vertex AI/Bedrock private endpoints or local models via Ollama.

Q: Is it safe to run Strix against production?
A: Only apps you own or have explicit written permission to test — the README is emphatic about this. Even then, prefer staging/pre-prod environments. Strix will attempt to exploit vulnerabilities, which can trigger real side effects (data changes, service disruption). For production, use the --instruction flag to constrain what it can do (e.g., "read-only, no state-modifying requests").

Q: Does it work with GitLab / Bitbucket CI?
A: Yes. The hosted platform has native integrations for GitHub, GitLab, and Bitbucket. For the OSS CLI, the GitHub Actions example in the README translates directly — same install command, same env vars, adapt the CI syntax.

Verdict

Strix is the first open-source pentest tool where every finding is a working exploit. Not a "potential vulnerability" — an actual proof-of-concept the agent ran to completion. That single design decision changes the economics of application security: a 4-minute Strix scan on every PR with a Claude-Sonnet backend delivers more validated findings than a quarterly $50K pentest, at $0.35/run.

The Docker + LLM-token cost curve is real, especially at enterprise scale. The black-box mode is weaker than grey-box. And you still want humans in the loop for high-stakes systems. But for the 90% of developers and small security teams who currently ship code with no continuous pentest coverage at all, Strix is a genuine step-function upgrade.

Repo: github.com/usestrix/strix
Hosted platform: app.strix.ai (free tier available)
Stars: 33,383 (as of July 3, 2026) — 4,743 added in the last week
License: open-source (see repo)
My rating: 4.5 / 5 — the validated-PoC model is the correct product decision and the CI integration is the smoothest I've tested. Only losing half a star for frontier-model cost at scale and the (inherent) weakness of black-box-only mode.

video-use Review: browser-use Team's AI Video Editor

Andrew — Thu, 02 Jul 2026 10:12:16 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

The browser-use team built their reputation by giving LLMs a structured DOM instead of screenshots. Now they've applied the same trick to video.

video-use is a 100% open-source skill that turns Claude Code, Codex, Hermes, OpenClaw — or any coding agent with shell access — into a full video editor. You drop raw footage into a folder, tell your agent what you want, and it hands you back final.mp4. No timeline. No menus. No presets.

The project shipped in April 2026 and hit 13,000+ GitHub stars with 3,000+ added in a single week as of early July 2026 — one of the fastest-growing repos in the browser-use org. It sits in a category that barely existed six months ago: coding agents editing video from raw source files.

I spent a few days running it against podcast rushes, product demos, and travel B‑roll. This is the review.

What video-use actually is

Strip the marketing away and video-use is three things:

A skill package (SKILL.md + helpers/*.py) any Anthropic-style agent can load, with production rules like "subtitles applied last" and "30ms audio fades at every cut boundary."
A transcript-first data pipeline that turns raw video into a compact phrase-level Markdown transcript — using ElevenLabs Scribe for word-level timestamps and speaker diarization.
A thin ffmpeg orchestration layer (render.py, grade.py, timeline_view.py) so the LLM expresses edits as data (edl.json) and lets plain ffmpeg encode.

The core insight, from the README:

Naive approach: 30,000 frames × 1,500 tokens = 45M tokens of noise.
Video Use: 12KB text + a handful of PNGs.

The LLM never watches the video. It reads it.

The architecture in one diagram

Raw footage/
├── take_001.mov      ──► transcribe.py ──► transcripts/take_001.json  (cached)
├── take_002.mov      ──► transcribe.py ──► transcripts/take_002.json  (cached)
└── ...
                                │
                                ▼
                        pack_transcripts.py
                                │
                                ▼
                     edit/takes_packed.md  ◄── LLM's primary "reading view"
                                │
              ┌─────────────────┴──────────────────┐
              ▼                                    ▼
   LLM proposes strategy               timeline_view.py (on demand)
   (plain English, waits              filmstrip + waveform PNG at
    for user confirmation)             ambiguous decision points
              │
              ▼
        edit/edl.json  (cut decisions as data)
              │
              ▼
   render.py: per-segment extract → -c copy concat → overlays → subtitles LAST
              │
              ▼
   Self-eval: timeline_view on the RENDERED output at every cut boundary
              │
              ▼
        edit/final.mp4

Two design decisions carry most of the weight:

Audio is primary, visuals follow. Cuts are always proposed on speech boundaries and silence gaps from the Scribe transcript. Visuals are only sampled at ambiguous moments (a long pause: was it a "thinking beat" worth keeping, or dead air worth cutting?).
Ask → confirm → execute → self-eval → persist. The agent has to describe its plan in plain English before it touches the cut, and it re-renders + re-inspects before it shows you anything.

Installation: really is one paste

The install flow is deliberately paste-into-your-agent-and-walk-away. From the README:

Set up https://github.com/browser-use/video-use for me.
Read install.md first to install this repo, wire up ffmpeg,
register the skill with your agent, set up the ElevenLabs API
key, then read SKILL.md and helpers/.

That paragraph works because agents like Claude Code and Codex have a skills directory convention. The script clones the repo, symlinks into ~/.claude/skills/video-use/ (or ~/.codex/skills/video-use/), runs uv sync, verifies ffmpeg/ffprobe, and prompts for ELEVENLABS_API_KEY.

Or do it by hand in four commands:

# 1. Clone and symlink into your agent's skills directory
git clone https://github.com/browser-use/video-use ~/Developer/video-use
ln -sfn ~/Developer/video-use ~/.claude/skills/video-use   # Claude Code
# ln -sfn ~/Developer/video-use ~/.codex/skills/video-use  # Codex

# 2. Install Python deps (ffmpeg-python, elevenlabs, pillow, etc.)
cd ~/Developer/video-use
uv sync   # or: pip install -e .

# 3. System binaries
brew install ffmpeg       # required
brew install yt-dlp       # optional, for downloading online sources

# 4. Add your ElevenLabs API key
cp .env.example .env
$EDITOR .env              # ELEVENLABS_API_KEY=...

Grab a key at elevenlabs.io/app/settings/api-keys. Scribe is billed per minute of audio; a 20-minute podcast take costs roughly a few cents.

A real session: podcast edit end-to-end

Here's what a session actually looks like. I cd'd into a directory with four raw takes of a two-person podcast and started Claude Code:

$ cd ~/podcast/ep47-raw
$ ls
take_001.mov  take_002.mov  take_003.mov  take_004.mov
$ claude

Then just:

> edit these into a 4–6 minute launch video for a developer audience.
  Neutral cinematic grade. 2-word uppercase subtitle chunks. Cut out
  the second take entirely — the guest coughed through it.

Behind the scenes, the agent did roughly this:

ffprobe the sources, then transcribed all four takes in parallel via transcribe_batch.py (4-worker ElevenLabs Scribe fan-out, cached to edit/transcripts/).
Packed the transcripts into edit/takes_packed.md — phrase-level, break on silence ≥ 0.5s, ~12 KB for 40 minutes of source. This is the LLM's reading view:

## take_001 (duration: 512.4s, 87 phrases)
  [002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
  [006.08-006.74] S0 We fixed this.
  [007.90-012.11] S1 Wait, so — what does that mean exactly?
  [012.55-014.20] S0 (laughter)

Proposed a strategy in plain English: "Open on take_001 002.52 → 006.74 (hook). Cut to take_003 048–132s. Skip take_002. Close on take_004 210–230s. Target 4:47. Grade: neutral cinematic. Confirm before I cut."
Waited for me to say "yes." The agent isn't allowed to touch the cut until the user approves the plan in words.
Generated edit/edl.json — every segment, source, start, end, grade, overlay, and subtitle style as data.
Rendered per-segment, ffmpeg concat -c copy, applied overlays, then burned subtitles last.
Self-evaluated by running timeline_view.py on the rendered output at every cut boundary (visual jumps, waveform spikes, hidden subtitles, misaligned overlays). Up to 3 fix-and-re-render passes.
Handed me edit/preview.mp4.

Wall-clock: about 8 minutes. Transcription dominated. Two revisions later ("20% tighter", "swap the opening line") the final rendered without re-transcription — Scribe results are cached per source.

The 12 hard rules — production correctness

The most interesting part of SKILL.md isn't aesthetics. It's a list labeled Hard Rules — non-negotiable correctness constraints:

Subtitles applied LAST in the filter chain, otherwise overlays hide captions (silent failure).
Per-segment extract → lossless -c copy concat, not single-pass filtergraph — avoids double-encoding when overlays land.
30ms audio fades at every segment boundary (afade=t=in:st=0:d=0.03,afade=t=out:st={dur-0.03}:d=0.03), otherwise audible pops.
Overlays use setpts=PTS-STARTPTS+T/TB to shift the overlay's frame 0 to its window start.
Master SRT uses output-timeline offsets: output_time = word.start - segment_start + segment_offset.
Never cut inside a word. Snap every cut edge to a Scribe word boundary.
Pad every cut edge by 30–200ms — Scribe timestamps drift 50–100ms.
Word-level verbatim ASR only. Never SRT/phrase mode. Never normalized fillers.
Cache transcripts per source. Never re-transcribe unless the file itself changed.
Parallel sub-agents for multiple animations, never sequential.
Strategy confirmation before execution.
All session outputs in <videos_dir>/edit/. Never write inside video-use/.

The list is worth reading even if you never use video-use — accumulated bruises from actually shipping this pipeline.

Overlays, grades, and generated animations

Animation overlays render via parallel sub-agents in any of four systems:

HyperFrames — HeyGen's declarative framer
Remotion — React-based programmatic video
Manim — 3Blue1Brown's math library
PIL — plain Pillow for raster overlays

Each animation lives in edit/animations/slot_<id>/. The parent agent spawns one sub-agent per slot in parallel via Claude Code's Agent tool — wall time is the slowest single animation, not the sum. On a five-animation launch video: 90 seconds instead of 7+ minutes.

Color grading is a one-liner:

helpers/grade.py -i in.mp4 -o graded.mp4 --preset neutral_cinematic
# or custom ffmpeg chain:
helpers/grade.py -i in.mp4 -o graded.mp4 \
  --filter 'curves=preset=medium_contrast,eq=saturation=1.05:gamma=0.98'

Grades apply per-segment before concat — combined with rule 2, that's one encode per segment, not two.

What it's genuinely good at

After a week on real work:

Talking-head content. Podcasts, video essays, tutorial narration, launch videos. Transcript-first is perfect for anything speech-driven.
Multi-take selection. Pick the best of six takes without watching all six. The LLM reads them all.
Filler-word removal at scale. Every "um," "uh," false start comes with a word-level timestamp — one pass to cut them.
Iterating verbally. "Tighter." "Warmer grade." "Bigger subtitle chunks." Each revision is minutes, not hours.
Session handoff. project.md persists memory. Start with Claude Code, resume next week in Codex.
Reproducibility. edl.json is a checked-in data file. Regenerate the same edit deterministically.

Honest limitations

video-use is not a Premiere replacement, and the README is upfront about it:

Purely visual content is a bad fit. No speech, nothing for the LLM to reason about. Wordless drone montages: use something else.
ElevenLabs Scribe is a hard dependency. No local Whisper fallback in the box. Sensitive material: non-starter.
Word-timing drift. Scribe timestamps drift 50–100ms. Edge padding hides most of it, but tightly-paced comedic timing can feel slightly off vs. a hand-cut.
No frame-accurate motion graphics. Overlays sit on top; nothing gets composited into a scene. Rotoscoping and motion tracking need a real NLE.
Long-form gets expensive. 90-minute raw source at Scribe's per-minute pricing is a few dollars per first pass. Iteration is cheap because results cache.
ffmpeg is the ceiling. Effects that need After Effects still need After Effects.
Windows story is weaker. Docs and community are macOS/Linux-first. WSL2 works fine.

Community reactions

Reception in the six weeks since launch has been unusually positive:

The browser-use org page lists video-use alongside the flagship browser agent and desktop app — signaling first-class product status.
Coverage from SoloSoft, Openflows, and The Menon Lab all landed on the same observation: the transcript-first pipeline makes it feel qualitatively different from previous "AI video editors" that tried to reason from frames.
The most cited critique on X and Reddit is the ElevenLabs dependency and the cost of transcribing long sources — legitimate concerns that a local-Whisper fallback would fix.
Independent developers have started shipping video-use plugins as their own agent skills — automatic B‑roll insertion, chapter marker generation. The "skill" packaging is proving to be a real extension point.

When to use video-use vs. alternatives

Tool	Best for	Trade-off
video-use	Podcasts, launch videos, tutorials with agent-first workflow	Needs ElevenLabs, ffmpeg-ceiling on effects
OpenMontage	Full agentic pipeline, 12 pipelines, 500+ skills	Heavier setup, broader surface area
Descript	Text-based editing with polished UI	Proprietary, cloud-only, subscription
DaVinci / Premiere	Color-critical or motion-graphics-heavy work	Manual, hours per edit, no LLM
yt-dlp + ffmpeg	One-off clip extraction	You write the whole pipeline

video-use wins when material is speech-driven, iteration speed matters more than pixel-perfect control, and you already live in Claude Code or Codex.

FAQ

Does video-use work with any coding agent?
Yes — the README explicitly lists Claude Code, Codex, Hermes, and OpenClaw, and anything that can run shell commands + read a skill file. The skill is just a directory with SKILL.md and helper scripts; the agent-specific step is symlinking it into that agent's skills directory. Cursor and Windsurf work via their own skill/rule conventions.

Do I need an ElevenLabs subscription?
You need an API key, but ElevenLabs offers a free tier with limited monthly minutes that's enough to try the whole pipeline on a short podcast. Paid usage is billed per minute of audio transcribed. There's no local Whisper fallback in the current release; adding one is the most common community feature request.

Can it edit videos without spoken audio?
Poorly. The whole pipeline is built around the transcript being the LLM's primary reading surface. For wordless material — B‑roll, dance, music videos, silent product demos — the LLM has almost nothing to reason about and you'd be better off with a traditional editor.

Does it re-transcribe every session?
No. Transcripts are cached per source file in edit/transcripts/<name>.json and only regenerated if the source file itself changes. This is Rule 9 in the hard rules. Iteration on cut decisions, grades, and overlays reuses the cached JSON.

Where do the outputs go?
All session artifacts live under <your_videos_dir>/edit/ — project.md for persistent session memory, takes_packed.md for the LLM's reading view, edl.json for cut decisions, clips_graded/ for per-segment extracts, animations/slot_<id>/ for overlays, master.srt for subtitles, preview.mp4, and final.mp4. The video-use/ repo itself is never written to.

How does the self-eval loop actually work?
After render, the agent runs timeline_view.py on the rendered output (not the sources) at every cut boundary — a ±1.5s window around each cut. It checks each generated PNG for visual discontinuity, waveform spikes past the 30ms fade, subtitles hidden behind overlays, or misaligned animation frames. If it finds any of these, it fixes and re-renders, capping at 3 self-eval passes before surfacing the issue to the user.

What license is it under?
MIT. Everything — the pipeline, helpers, and skill definition — is fully open source. You can fork it, ship your own version, or vendor parts of it inside a proprietary product.

Is this actually production-ready?
For talking-head content, podcasts, launch videos, and tutorial narration: yes, it's shipping real work today, including for the browser-use team's own 15-second TikTok demo advertising it. For anything that would strain a traditional NLE — color-critical work, motion graphics, VFX — you'll want a real editor. Treat video-use as the fastest possible path from raw footage to polished-enough draft, not as a replacement for professional post-production.

Verdict

video-use is one of the more surprising open-source releases of 2026 because it doesn't ask an LLM to do the thing LLMs are worst at — reasoning about pixels. Instead, it does the browser-use trick: give the model a well-structured symbolic representation of the medium (a transcript with word-level timestamps), let it reason there, and let deterministic code (ffmpeg) do the actual rendering. Wrap the whole thing in an agent skill so any coding agent can pick it up in one paste.

If you spend any time producing speech-driven video and you already use Claude Code or Codex for other work, video-use is worth an afternoon to try. It's MIT-licensed, sits on a well-known dependency stack, and the hard-rules list alone is a useful education in what silent-failure modes actually exist in an automated video pipeline.

For the browser-use team specifically, video-use is another data point in a pattern that's starting to look like a thesis: the right primitive for an agent isn't a screenshot, it's a structured view of the medium. DOM for the web. Transcript for video. Whatever's next for whatever comes next.

Repo: github.com/browser-use/video-use. Try it in Browser Use Cloud. Related: OpenMontage review, Voicebox: open-source ElevenLabs alternative, browser-use review.

Cognee Review: Open-Source AI Memory Platform for Agents

Andrew — Wed, 01 Jul 2026 10:11:04 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Cognee is an open-source AI memory platform that gives agents persistent long-term memory across sessions. It combines vector embeddings, graph reasoning, and cognitive-science-grounded ontology generation into a single Python package with a four-verb API: remember, recall, forget, and improve. It just crossed 26,000 GitHub stars (6,400+ new this week) and, notably, in Cognee 1.0 you can run the entire memory layer — graph, vectors, sessions, metadata — on a single Postgres instance instead of the usual Neo4j + Redis + vector-DB stack.

Key facts:

Four-verb API: cognee.remember(), cognee.recall(), cognee.forget(), cognee.improve() — that's the whole surface area
Two memory tiers: session memory (fast cache) + permanent knowledge graph (background sync)
Postgres-native: pgvector for embeddings, Postgres graph backend for relationships, SQL session cache — one database or swap in Neo4j/Redis/Qdrant
BEAM benchmark: 0.79 at 100K tokens (previous SOTA 0.735), 0.67 at 10M tokens — beats the state of the art on long-context memory
Claude Code plugin: hooks into SessionStart, UserPromptSubmit, PostToolUse, Stop, PreCompact, SessionEnd for automatic memory capture and context injection
Multi-language clients: Python, TypeScript (@cognee/cognee-ts), Rust (cognee-rs)
26,000+ GitHub stars, 6,400+ new stars this week, Apache 2.0

Quick Reference

Property	Value
Repository	github.com/topoteretes/cognee
Author	Topoteretes (Vasilije Markovic + team)
License	Apache 2.0
Languages	Python 3.10–3.14, TypeScript, Rust
Install	`uv pip install cognee`
GitHub Stars	26,229 (6,417 this week)
Docker Images	`cognee/cognee`, `cognee/cognee-mcp`
Cloud	cognee.ai (managed)
Research paper	arXiv:2505.24478
Benchmark	BEAM 0.79 @ 100K, 0.67 @ 10M tokens

What It Is

Cognee is the piece almost every serious agent build eventually needs but nobody wants to write from scratch: the memory layer. LLMs, on their own, have no persistent memory. RAG stores documents but doesn't build relationships. Vector databases embed chunks but can't reason about how facts connect. Cognee sits underneath your agent and turns raw inputs into a knowledge graph plus a vector index plus a session cache — all of which get queried together when the agent needs context.

The mental model is straightforward. You call remember() with text (or documents, or structured data). Cognee runs an ingestion pipeline that extracts entities, generates embeddings, builds subject–relation–object triplets, and stores everything in a knowledge graph that also knows how to be searched by meaning. When you later call recall() with a question, Cognee picks the best retrieval strategy — pure vector search, graph traversal, or a hybrid — and returns context that the agent can slot into its prompt.

What makes it different from Graphiti, Zep, or standard GraphRAG stacks: cognee ships the whole thing as one Python package with sensible defaults (SQLite + LanceDB + Kuzudb for local dev, Postgres + pgvector for production) and a four-verb API. You don't need to stand up Neo4j to start; you just install the package and call remember.

Installation and First-Run

The quickstart is refreshingly boring, which is the highest compliment:

uv pip install cognee

import cognee
import asyncio
import os

os.environ["LLM_API_KEY"] = "sk-..."  # OpenAI, or configure any provider

async def main():
    # Store permanently in the knowledge graph
    await cognee.remember("Cognee turns documents into AI memory.")

    # Store in session memory (fast cache, syncs to graph in background)
    await cognee.remember(
        "User prefers detailed explanations.",
        session_id="chat_1",
    )

    # Query with auto-routing
    results = await cognee.recall("What does Cognee do?")
    for r in results:
        print(r)

    # Session-scoped recall (falls through to permanent graph if needed)
    results = await cognee.recall(
        "What does the user prefer?",
        session_id="chat_1",
    )
    for r in results:
        print(r)

asyncio.run(main())

That's a functioning agent memory layer in ~15 lines. No graph database to configure, no vector store to provision, no schema to design. The first remember() call takes a few seconds because Cognee spins up the local SQLite + LanceDB stack and runs the entity-extraction pipeline; every call after that is fast.

There's also a CLI for the shell-first crowd:

cognee-cli remember "Cognee turns documents into AI memory."
cognee-cli recall "What does Cognee do?"
cognee-cli forget --all
cognee-cli -ui   # local web UI (Docker required)

The Postgres-Only Deployment

This is the headline feature of Cognee 1.0 and the reason it's trending this week. Traditionally, "graph memory" means running four services in production:

Memory layer	Traditional stack	Cognee on Postgres
Relationships	Neo4j (or another graph DB)	Cognee's Postgres graph backend
Embeddings	Dedicated vector DB (Qdrant, Weaviate, Milvus)	pgvector
Sessions	Redis	SQL session-cache backend
Metadata	Relational DB	Same Postgres

Cognee collapses all four into one Postgres instance. The graph still exists — it just lives inside the same Postgres-backed memory layer as the text, metadata, and embeddings, so retrieval doesn't cross service boundaries. In the project's own CI benchmarks, this setup ran ~10% faster than the separate graph-plus-vector configuration.

To switch on Postgres mode:

pip install "cognee[postgres]"

# .env
DB_PROVIDER=postgres
VECTOR_DB_PROVIDER=pgvector
GRAPH_DATABASE_PROVIDER=postgres
CACHE_BACKEND=postgres

DB_HOST=localhost
DB_PORT=5432
DB_USERNAME=cognee
DB_PASSWORD=cognee
DB_NAME=cognee_db

For teams that already run Postgres, this is a large operational win. One database to back up, one to secure, one to patch. If your workload later demands specialized backends, you can still swap in Neo4j or Neptune for graphs, Redis for sessions, and Qdrant/Weaviate/Milvus/Chroma for vectors via community adapters. Nothing about the API changes.

Two-Tier Memory: Session vs Permanent

Cognee splits memory into two tiers, which is a good design decision borrowed from how human memory actually works:

Session memory is short-term working memory. Cognee loads relevant embeddings and graph fragments into a fast cache scoped by session_id. Reads and writes are cheap and low-latency, ideal for the currently-open chat or task.
Permanent memory is the long-term knowledge graph. User data, resolved interaction traces, learned patterns, and stable facts land here. Session data syncs into permanent memory in the background (or on session end).

The nice consequence: your hot path stays fast because you're hitting the session cache, and your cold path is a proper graph query that can reason across everything the agent has ever learned.

# Session write — fast, scoped to this conversation
await cognee.remember(
    "The user is debugging a Postgres connection issue on Fly.io.",
    session_id="chat_2026_07_01",
)

# Session read first, fall through to permanent graph if needed
answer = await cognee.recall(
    "What was the user working on last?",
    session_id="chat_2026_07_01",
)

Under the Hood

Cognee's ingestion pipeline is where the work happens. remember() is shorthand for add + cognify + improve:

add — ingest the raw document (text, PDF, code, JSON, structured data).
cognify — extract entities and subject–relation–object triplets via LLM calls, generate embeddings, write both into the graph and vector store.
improve — a feedback loop that refines the graph over time. Weights shift, redundant nodes merge, ontology grounding tightens.

recall() auto-routes: it inspects the query and picks between pure vector similarity, graph traversal, or a combined "GraphRAG" mode that pulls both. You can override with explicit search parameters when needed.

Cognee also supports OWL ontologies for domain-specific knowledge modeling — useful in regulated domains (medical, legal, financial) where you want the graph structured against a formal schema rather than whatever the LLM extracts on the fly.

Claude Code Plugin: Persistent Memory Across Sessions

If you use Claude Code, the Cognee memory plugin is the most useful integration to try. Install it once from the shell, before launching Claude Code:

claude plugin marketplace add topoteretes/cognee-integrations
claude plugin install cognee-memory@cognee

export LLM_API_KEY="sk-..."   # local mode
claude

The plugin hooks into Claude Code's lifecycle events:

SessionStart — selects mode (local vs cloud) and sets up identity
UserPromptSubmit — injects dataset-scoped context from memory into the prompt
PostToolUse — captures tool call traces
Stop — writes the assistant's answer into session memory
PreCompact — preserves memory across Claude Code's context resets (this is the killer feature — no more losing the plot when the context window fills up)
SessionEnd — final sync into the permanent graph

You get a "Cognee Memory Connected" system message on startup. In practice, this means Claude Code remembers what you were working on last week, what conventions your codebase follows, and which fixes you've already tried and rejected — without you copy-pasting summaries into every new session.

Benchmarks: BEAM

Cognee published results on BEAM, a long-context benchmark designed to test whether a memory system can track a long, evolving conversation — a more useful test than the usual needle-in-a-haystack setups.

Benchmark	Setting	Cognee	Previous SOTA	Obsidian / RAG baseline
BEAM	100K tokens	0.79 (>0.8 with per-question routing)	0.735	~0.33
BEAM	10M tokens	0.67	0.641	~0.33

Two things to note. First, these numbers are with Cognee's default settings and standard open-source features — no BEAM-specific pipelines, no custom fine-tunes. Second, the vanilla RAG baseline sits at ~0.33, less than half of what Cognee delivers, which lines up with what most people building agent memory report anecdotally: RAG alone is not enough once the context gets long or the questions get multi-hop.

An earlier evaluation on HotpotQA also reported 87% answer accuracy with human labeling (evaluations docs), consistent with the "top choice for knowledge-intensive scenarios and multi-hop reasoning" verdict from third-party reviewers.

Fair caveat that Cognee itself calls out: these numbers are directional. Long-context memory benchmarks are genuinely hard and the field is still figuring out what to measure. But even discounting the SOTA claim, the gap over vanilla RAG is large enough to justify trying it on your own workload.

Two Concrete Use Cases

Customer Support Agent. The agent resolves issues using a customer's data across finance, support, and product history. Cognee tracks past interactions, failed actions, and resolved cases. When the customer says "my invoice looks wrong and the issue is still not resolved," the agent can pull similar billing cases resolved last month and reply with a concrete diagnosis rather than a scripted response. Memory updates after execution so the agent never repeats the same failed step.

SQL Copilot. Cognee stores expert SQL queries, workflow patterns, and schema structures. When a junior asks "how do I calculate customer retention for this dataset?" Cognee matches the current schema to a known structure and adapts an expert's retention query. Every successful implementation feeds back into the graph.

The pattern is the same in both cases: cognee handles the "what did we do before, and what worked" question, so the agent doesn't have to.

Community Reactions

Cognee has been building momentum on Hacker News and Reddit for over a year — the Show HN in June 2025 framed it as "the AI memory layer that remembers context," and the founder is active in r/LLMDevs and r/AIMemory.

Recurring themes:

Praise for the API surface: developers like that remember/recall is the whole thing.
Skepticism about graph overhead: some HN commenters worry graph construction is slow on large corpora. Cognee runs it in the background and lets you tune it.
Comparisons to Graphiti / Zep / Mem0: the founder describes cognee as "similar to Graphiti but more modular, and not just time graphs." Graphiti is opinionated about temporal edges; cognee lets you shape the graph how you want.
Postgres-only mode is universally applauded — the top-requested feature for months, and the main reason the current release is trending.

The WeavAI review from May 2026 rates cognee 8.5/10 overall and calls it "the top choice for knowledge-intensive scenarios and multi-hop reasoning applications."

Honest Limitations

First-run cost: ingesting a large corpus makes a lot of LLM calls (entity extraction, triplet generation, embedding). Point cognee at a 100MB knowledge base with OpenAI defaults and your first bill is not zero. Local LLM providers are supported and shrink this concern.
Graph queries are slower than pure vector search. If your workload is really just similarity retrieval, you're paying for graph infrastructure you don't need. Cognee is designed for cases where relationships matter.
Ontology tooling is Python-centric. TypeScript and Rust clients cover the API but don't have first-class support for defining custom OWL ontologies — that lives in the Python SDK.
Docker requirement for cognee-cli -ui. The MCP-backed UI needs Docker or Colima installed.
Multi-tenant isolation is documented but young. Cognee ships user/tenant isolation primitives; if you're deploying to enterprise with strict data-boundary requirements, build your own test harness before trusting it in production.

Deployment Options

Cognee ships 1-click deploy configurations for most modern platforms: Cognee Cloud (managed, await cognee.serve()), Modal (serverless, GPU), Railway (PaaS + native Postgres), Fly.io (edge + volumes), Render, Daytona (cloud sandboxes), and plain Docker Compose for self-hosters. docker compose up with the right profiles gets you the API server, frontend, MCP server, Postgres, and optionally Neo4j — all in one file.

FAQ

How is Cognee different from a normal RAG pipeline?

RAG chunks documents, embeds them, and does similarity search. It has no idea that "the customer" in chunk 47 is the same person as "the user" in chunk 302. Cognee builds a knowledge graph on top of the embeddings, so entities and relationships are explicit. When you query, Cognee can traverse the graph and do vector search, which is why it hits ~0.79 on BEAM at 100K tokens vs ~0.33 for vanilla RAG.

Do I need to run Neo4j?

No. Cognee 1.0 can run the entire memory layer on a single Postgres instance (pgvector for embeddings + Postgres graph backend for relationships + SQL session cache + metadata in the same DB). Local dev works with SQLite + LanceDB + Kuzudb, no services required. Neo4j is still supported if you want it.

How does Cognee compare to Graphiti, Zep, or Mem0?

The founder's own framing (in r/LLMDevs): "similar to Graphiti but a bit more modular and customizable, and not just time graphs." Graphiti is opinionated around temporal edges; Cognee is more general-purpose. Zep is a hosted product; Cognee is open source with a cloud option. Mem0 is more of a session-memory library; Cognee spans session + long-term graph.

Can I use Cognee without OpenAI?

Yes. Cognee's LLM provider layer is pluggable — configure any OpenAI-compatible endpoint (local Ollama, LM Studio, vLLM, Anthropic, Gemini, etc.) via environment variables. See the LLM Provider docs.

Is there a JavaScript/TypeScript client?

Yes — @cognee/cognee-ts on npm. There's also a Rust client, cognee-rs, for cargo add cognee. All three clients hit the same Cognee API, so you can mix languages across an application.

What's the license?

Apache 2.0. You can self-host, modify, and use it commercially. There's a managed Cognee Cloud tier for teams that don't want to run the infra.

Verdict

Cognee earns the "top of GitHub trending" spot because it does one thing well: it's the memory layer for AI agents that you don't have to build yourself. The four-verb API (remember, recall, forget, improve) is small enough to fit in your head, the Postgres-only deployment removes the biggest operational objection to graph memory, and the BEAM benchmarks are strong enough — and honestly presented enough — to take seriously.

If you're building anything more sophisticated than a stateless chatbot, you will eventually need agent memory. Cognee is the shortest path from "we need memory" to "we have memory" that currently exists in open source. Install it, spend an afternoon on the Claude Code plugin or the Python quickstart, and see if the two-tier session-plus-graph model fits your workload.

Repo: github.com/topoteretes/cognee · Docs: docs.cognee.ai · Discord: discord.gg/NQPKmU5CCg

DESIGN.md: Google's Spec for AI-Agent Design Systems

Andrew — Mon, 29 Jun 2026 10:08:34 +0000

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

DESIGN.md is a format specification from Google Labs that lets you describe your visual identity in a single markdown file that AI coding agents can actually read and follow. It combines YAML front matter (machine-readable design tokens) with markdown prose (human-readable rationale) — so agents get exact color values and the context for why those values exist. It hit 23,000+ GitHub stars in its first week and is now the #1 most-starred new Google Labs open-source project on GitHub Trending this week.

Key facts:

Dual-layer format: YAML tokens for agents, markdown prose for humans — same file
Built-in CLI tools: lint (WCAG contrast checking, broken reference detection), diff (token-level regression detection), export (Tailwind v3/v4, DTCG/W3C Design Tokens)
Token references: Components reference design tokens with {colors.primary} syntax — change one value, update everywhere
9 lint rules: broken-ref, contrast-ratio, orphaned-tokens, section-order, unknown-key detection, and more
Already ecosystem-ready: Used by Google's Stitch MCP server, available as npx @google/design.md, exportable to Tailwind themes
Zero dependencies: Just a markdown file and npx — no build step, no config file, no SDK
23,000+ GitHub stars, 6,700+ new stars this week

Quick Reference

Property	Value
Repository	github.com/google-labs-code/design.md
Author	Google Labs (Google)
License	Apache 2.0
Language	TypeScript (CLI), Markdown (format)
CLI Package	`@google/design.md` on npm
Current Version	alpha
GitHub Stars	23,000+ (6,700+ this week)
Related Projects	Google Stitch, Astryx (Meta), Agent Skills

What It Is

DESIGN.md is deceptively simple: a single markdown file with YAML front matter that defines colors, typography, spacing, rounded corners, and component tokens, followed by prose sections that explain why the design system works the way it does.

---
name: Heritage
colors:
  primary: "#1A1C1E"
  secondary: "#6C7278"
  tertiary: "#B8422E"
  neutral: "#F7F5F2"
typography:
  h1:
    fontFamily: Public Sans
    fontSize: 3rem
  body-md:
    fontFamily: Public Sans
    fontSize: 1rem
  label-caps:
    fontFamily: Space Grotesk
    fontSize: 0.75rem
rounded:
  sm: 4px
  md: 8px
spacing:
  sm: 8px
  md: 16px
---

This isn't just another design spec format. What makes DESIGN.md different is that it's built for dual consumption — both humans and AI agents read the same file. The YAML front matter is the machine contract: agents parse it and get exact hex values, font stacks, spacing scales. The markdown body is the human contract: designers explain why "Boston Clay" (#B8422E) is the sole interaction color, and agents use that context to make better judgment calls.

The format is versioned (currently alpha) and published at docs/spec.md. It supports 8 named sections: Overview, Colors, Typography, Layout, Elevation & Depth, Shapes, Components, and Do's and Don'ts.

Why It's Trending NOW

Three forces converged to make DESIGN.md explode this week.

First, the AI coding agent boom. Every major coding assistant — Claude Code, Codex CLI, Gemini CLI, Cursor, Zed, OpenClaw — now supports MCP (Model Context Protocol) and custom instruction files. Developers are rushing to give their agents context about codebases. DESIGN.md is the missing piece for visual context: it tells agents what the UI should look like.

Second, the design-system industry is converging on a standard. Meta just launched Astryx, an open-source React design system with an MCP server that lets agents browse 150+ components. Google Stitch, Google's agentic design workflow platform, already has a stitch-design plugin with a generate-design skill that outputs DESIGN.md files. The industry is realizing that for AI agents to produce decent UIs, they need structured access to design tokens — not screenshots, not Figma links, not prose descriptions.

Third, the timing is perfect for a "README.md for design." Every developer knows what README.md is. Every developer has been frustrated by AI agents generating UIs with random colors, mismatched typography, and broken spacing. DESIGN.md slots into the same mental model as README.md — a file you drop into your repo that agents read automatically. The parallel is so obvious it's surprising nobody standardized it before.

"DESIGN.md is to agentic UI development what README.md was to open-source documentation — a single file that changes everything." — @davideast, maintainer

Key Features (with Code Examples)

1. Structured Design Tokens with Component References

The token system supports references — components can reference tokens with {token.path} syntax:

components:
  button-primary:
    backgroundColor: "{colors.tertiary}"
    textColor: "{colors.on-tertiary}"
    rounded: "{rounded.sm}"
    padding: 12px
  button-primary-hover:
    backgroundColor: "{colors.tertiary-container}"

This means changing colors.tertiary from "#B8422E" to "#C94F3F" propagates to every component that references it. No more hunting through code for hard-coded hex values.

2. CLI Linter (WCAG Contrast Checking Built In)

The lint command validates your DESIGN.md against 9 rules:

npx @google/design.md lint DESIGN.md

Output:

{
  "findings": [
    {
      "severity": "warning",
      "path": "components.button-primary",
      "message": "textColor (#ffffff) on backgroundColor (#1A1C1E) has contrast ratio 15.42:1 — passes WCAG AA."
    }
  ],
  "summary": { "errors": 0, "warnings": 1, "info": 1 }
}

The contrast-ratio rule checks every component's backgroundColor/textColor pair against WCAG AA minimum (4.5:1). This catches accessibility bugs before any code is written.

3. Token-Level Diff (Regression Detection)

Compare two versions of a design system to detect token drift:

npx @google/design.md diff DESIGN.md DESIGN-v2.md

{
  "tokens": {
    "colors": { "added": ["accent"], "removed": [], "modified": ["tertiary"] },
    "typography": { "added": [], "removed": [], "modified": [] }
  },
  "regression": false
}

This is huge for design system maintainers. You can CI-check a PR that changes DESIGN.md and reject it if it introduces regressions like removing a token that components still reference.

4. Export to Tailwind (v3 and v4)

Probably the most immediately useful feature for most developers:

# Tailwind v3 config
npx @google/design.md export --format json-tailwind DESIGN.md > tailwind.theme.json

# Tailwind v4 @theme block
npx @google/design.md export --format css-tailwind DESIGN.md > theme.css

This generates a complete theme.extend config (v3) or @theme { ... } block (v4) with every color, font, spacing, and radius token mapped to the correct Tailwind namespace. No manual token mapping, no copy-paste errors.

5. DTCG (W3C Standard) Export

npx @google/design.md export --format dtcg DESIGN.md > tokens.json

The W3C Design Tokens Format Module is the emerging standard for cross-tool token exchange. DESIGN.md converts to it natively, so you can sync tokens between Figma plugins, design tooling, and your codebase.

Community Reactions

The launch generated significant discussion across developer platforms:

Reddit r/UXDesign:

"Even with MD files it's not one size fits all. I created a semi automated framework for one of my projects and I still have to tweak things here and there." — r/UXDesign

TechTimes:

"Version 0.3.0 shipped on June 15. Artiverse reported that Google's specification notes that tokens give agents exact values while prose tells them why those values exist and how to apply them." — TechTimes

SlideSpeak (DESIGN.md for presentations):

"DESIGN.md is a small format with a big idea. Google Labs publishes the spec: one markdown file that tells AI coding agents how to build on-brand interfaces." — SlideSpeak

The community's biggest open question is adoption: will this become the standard, or will every company invent their own? Early signs are positive — Google Stitch already generates DESIGN.md files natively, and the awesome-agent-skills repo lists it as a recommended skill.

Getting Started

Getting started takes about 30 seconds:

1. Install the CLI

npm install @google/design.md
# or run directly:
npx @google/design.md lint DESIGN.md

Windows note: PowerShell may confuse design.md with the Markdown file association. Use the designmd alias instead:

npx -p @google/design.md designmd lint DESIGN.md

2. Create a DESIGN.md

Start with the minimal structure:

---
name: My App
colors:
  primary: "#1A1C1E"
typography:
  body:
    fontFamily: Inter
    fontSize: 1rem
---

3. Validate It

npx @google/design.md lint DESIGN.md

4. Export to Your Framework

npx @google/design.md export --format json-tailwind DESIGN.md > tailwind.theme.json

5. Drop It in Your Repo

Just like README.md, commit DESIGN.md to the root of your repository. AI coding agents that support MCP or custom instructions will automatically read it.

How It Works Under the Hood

DESIGN.md is parsed into a structured DesignSystemState object:

The YAML front matter is parsed with a custom schema validator that recognizes Color, Dimension, Typography, and TokenReference types
Token references ({colors.primary}) are resolved into a dependency graph — the linter detects broken references as errors
Component definitions are validated against a whitelist of allowed properties (backgroundColor, textColor, typography, rounded, padding, size, height, width)
The linter runs 9 rule checks in parallel, each producing findings at error, warning, or info severity
The export command traverses the resolved token graph and generates framework-specific output (Tailwind theme config, DTCG JSON)

The token system is inspired by the W3C Design Token Format, but DESIGN.md is deliberately simpler: no nested groups, no aliases, no $type declarations. It's designed to be writable by humans, not just machines.

The CLI is a single TypeScript package compiled to Node.js. It has zero runtime dependencies beyond Node 18+.

Who Should Use This

✅ You should use DESIGN.md if:

You use AI coding agents (Claude Code, Cursor, Codex, Gemini CLI) and want consistent UI output
You maintain a design system and want agents to follow it
You're tired of AI agents hallucinating colors, fonts, and spacing
You use Tailwind CSS and want to share tokens between your design system and your codebase
You want WCAG accessibility checking as part of your design token workflow

❌ You should NOT use DESIGN.md if:

You don't use AI coding agents at all
You have a design system that's already expressed in Storybook + Figma tokens with auto-sync
You need runtime token resolution (DESIGN.md is a build-time spec, not a runtime token engine)
You're looking for a replacement for CSS variables or Tailwind configs (it maps to them, doesn't replace them)

DESIGN.md vs Meta Astryx

Both Google and Meta launched design-system + AI-agent integrations within days of each other — but they're solving different parts of the problem.

Dimension	DESIGN.md (Google)	Astryx (Meta)
What it is	A format spec for describing visual identity	A full React design system (150+ components)
CLI	`npx @google/design.md lint/diff/export`	`npx @astryx/cli scaffold/build`
AI integration	Format agents can read (token-driven)	MCP server agents can query (component-driven)
Output	Tailwind config, DTCG tokens	React components, templates
GitHub Stars	23,000+	8,500+
License	Apache 2.0	MIT
Best for	Describing your design system	Using Meta's design system

In practice, they're complementary. You could put Astryx components in your repo and write a DESIGN.md that tells agents how to use them together with your brand colors. Astryx's MCP server handles "what components exist," while DESIGN.md handles "what should the UI look like."

Honest Limitations

DESIGN.md is at version alpha, and it shows:

The spec is still evolving. Expected changes as it matures — the alpha label means no backward compatibility guarantees yet.
No runtime token resolution. DESIGN.md is a build-time format. If you need runtime token access (themed UIs, dynamic color switching), you'll still need a runtime token system alongside it.
Adoption is early. The format is only useful if agents actually read it. Right now, it requires a custom instruction or skill to wire into Claude Code, Cursor, etc. Google Stitch supports it natively, but other agents need manual setup.
No Figma plugin (yet). The ideal workflow would be "design in Figma → export DESIGN.md → agents build UI." That pipeline doesn't exist yet — you're writing DESIGN.md by hand.
Limited color space validation. The linter checks WCAG contrast, but it doesn't validate that your color palette has enough variance, or that accessibility modes work — those still need human judgment.
Component model is basic. Components can only reference 7 property types. There's no support for state machines, responsive variants, or animation tokens.

FAQ

How is DESIGN.md different from a Tailwind config?

A Tailwind config defines everything available to your CSS framework. DESIGN.md defines what your brand looks like. DESIGN.md exports to Tailwind config (so you get both), but it also includes prose rationale, accessibility data, and component token references that a Tailwind config can't express.

Can I use DESIGN.md without Google Stitch?

Absolutely. DESIGN.md is format-first — the CLI is just tooling. Drop the file in any repo, and any AI coding agent with custom instructions can read it. Stitch is one integration; the ecosystem is growing.

Does DESIGN.md replace Storybook?

No. Storybook documents what components look like interactively. DESIGN.md documents what tokens and rules govern your visual identity. They serve different purposes and work well together — DESIGN.md defines the tokens, Storybook shows the components built with them.

What happens if I change a token?

Change the YAML value and re-lint. The diff command will show exactly what changed. Any component referencing the old value via {token.path} syntax automatically inherits the new value. The linter will catch broken contracts (removing a token that's still referenced).

How do I get Claude Code or Cursor to read DESIGN.md?

Add a custom instruction pointing to the file. For Claude Code, add to your CLAUDE.md: "Read DESIGN.md at the project root for design system tokens and visual identity rules." For Cursor, add a .cursorrules entry. The awesome-agent-skills repo has ready-to-use skill files for common agents.

Can DESIGN.md define dark mode tokens?

Version alpha doesn't have a native dark-mode namespace, but you can use two DESIGN.md files (DESIGN.md + DESIGN.dark.md) or extend the format with section-level overrides. The spec is flexible enough to accommodate this within existing constraints.

DESIGN.md is one of those rare developer tools that feels inevitable in hindsight. Just as README.md became the universal convention for project documentation, DESIGN.md has the potential to become the universal convention for design system documentation — especially in an age where AI agents read more code than humans do. It's alpha software, it has rough edges, and adoption is early. But the direction is right, and 23,000 stars in a week suggests the community agrees.