DEV Community: frank chu

Qwen3.8-Max says it beats GPT-5.6 and Fable 5 at computer use. Here's my checklist before I believe any open-weights release

frank chu — Wed, 12 Aug 2026 06:14:16 +0000

Alibaba released Qwen3.8-Max on August 3: a 2.4-trillion-parameter MoE with a 1M-token context window, priced at $2/$6 per million tokens. The claim that got everyone's attention: 86.1 on OSWorld-Verified, ahead of GPT-5.6 Sol Max (83.2) and Claude Fable 5 (85.0) at agentic computer use. Open weights are promised "next week," alongside a 27B sibling.

I want this to be true. Frontier-class open weights would be the best thing to happen to indie builders since Llama. But I've been burned by launch-day benchmark euphoria before, so here's the checklist I actually run before moving any workload — using Qwen3.8-Max as the worked example.

1. Read the license before the model card

"Open weights" is doing a lot of work in that headline, because as I write this, the license hasn't been disclosed. Apache 2.0 and "custom license with a commercial-use clause that names your revenue threshold" are both routinely called open. Until there's a license file, there is no release — there's a press release.

If the weights ship under something restrictive, the 2.4T headline model matters less than whether the 27B sibling gets the permissive license. Which brings me to:

2. Ask which model you'd actually run

A 2.4T MoE is not a thing you self-host. Even with MoE sparsity, serving it means a GPU cluster and serious inference engineering — realistically you'll consume it through Alibaba's API, which makes it "open weights" in a mostly ceremonial sense for a solo builder.

The release that changes my life is Qwen3.8-27B. A 27B that inherits even most of the flagship's agentic training fits on hardware normal people rent. When the weights drop, that's the file I'm downloading first.

3. Separate self-reported from independent numbers

Every number above is from Qwen's own release material. That's not an accusation — everyone launches this way — but self-reported benchmarks have a specific failure mode: the vendor picks the benchmarks, the effort settings, and the comparison models' configs.

Things I wait for: independent runs on Terminal-Bench and OSWorld leaderboards, the first "I reproduced X, got X-minus-something" posts, and — most honest signal of all — what the model does on tasks nobody optimized for. Qwen's own reported Terminal-Bench 2.1 score (86.6) already sits below GPT-5.6 Sol's 88.8, which I actually find reassuring: uniformly-winning launch decks are the suspicious ones.

4. Check the cached-input price, not the sticker

$2/$6 is competitive but not disruptive — Sonnet 5 costs the same on input this month. The disruptive number is $0.25 per million cached input tokens. Agent workloads re-read context obsessively; if your cache-hit rate is decent, this prices the flagship near budget-tier territory. When comparing models for agent use, cached-input price is the number I put in the spreadsheet first.

5. Give it a week of other people's traffic

Launch-week models have a way of getting quietly patched, re-quantized, or rate-limited once real traffic arrives. Unless the model solves a problem you have today, the cost of waiting seven days is near zero, and the information you get is enormous.

Where I land

Genuinely excited, provisionally skeptical. If the license is clean and the 27B holds up, this is the most important open release of the year for people like me. If the license is cute, it's an API with extra marketing.

The checklist isn't cynicism — it's the difference between betting your stack on a model and betting an afternoon on it. Bet afternoons.

What's on your open-model checklist that I'm missing?

NVIDIA's NOOA turns an AI agent into one Python class

frank chu — Mon, 10 Aug 2026 21:46:05 +0000

NVIDIA Labs open-sourced NOOA (NVIDIA Object-Oriented Agents) this week, and the pitch is unusually simple: an agent is a Python class. Not a graph, not a chain, not a YAML pipeline. A class.

I cloned it and got it running the same day. Here's what it actually looks like, what broke, and why I think the core idea matters more than the framework itself.

The whole idea in one code block

from nooa import Agent

class InventoryAgent(Agent, llm=llm):
    """You are an agent that checks inventory using deterministic helper methods."""

    # Plain Python — automatically available as a tool for the LLM
    def get_stock(self, item: str) -> int:
        """Get current stock for an item."""
        return self.inventory.get(item, {}).get("stock", 0)

    # `...` body — the LLM implements this at runtime, calling the methods above
    async def can_fulfill_order(self, items: list[str], budget: float) -> Result:
        """Check if order can be fulfilled within budget."""
        ...

That's from the repo's quickstart, lightly trimmed. The mapping is:

Fields are agent state
Methods with real bodies are deterministic tools
Methods with ... bodies are implemented by an LLM loop at runtime
Docstrings are the prompts
Type annotations are contracts the runtime enforces, with auto-retry on mismatch

No separate tool-schema JSON. No registration step. The model acts by writing Python in a REPL with access to self, so your method signatures are the tool definitions.

Two install gotchas before you try it

The README says pip install nooa. Two things I hit on a clean machine:

1. It's not on PyPI yet. As of today, pip install nooa returns No matching distribution found. Install from source instead:

git clone https://github.com/NVIDIA-NeMo/labs-OO-Agents.git
uv venv --python 3.13 && uv pip install ./labs-OO-Agents

2. No Python 3.14 support. The package pins >=3.12,<3.14. My default interpreter is 3.14, and the install fails with a version error. Use 3.12 or 3.13.

After that, everything imported cleanly and defining an Agent subclass with a generation method worked first try (version installed: 0.0.1.dev1 — this is early software, and it behaves like it).

What's genuinely different here

Most agent frameworks make you maintain two parallel worlds: your code, and a shadow copy of your code described in schemas, prompt templates, and callback wiring. Every refactor has to happen twice.

NOOA's bet is that the language already has all the metadata an LLM needs — signatures, types, docstrings — so the shadow world can be deleted. Your agent diffs like code, tests like code, and refactors like code. mypy and your IDE understand it because there's nothing else to understand.

There's also a strategy layer worth knowing about: PredictStrategy (single completion) vs CodeActStrategy (iterative code execution, capped by max_iterations), swappable per method via a decorator. That's a clean answer to "some steps need one LLM call, some need a loop" without restructuring the agent.

NVIDIA's paper claims a 253-line NOOA agent hits 82.2% on SWE-bench Verified and 86.8% on CyberGym L1. I haven't reproduced those numbers, and you shouldn't take vendor benchmarks at face value — but the interesting claim isn't the score, it's the line count.

The part that should make you nervous

A NOOA agent acts by executing LLM-generated Python. With access to self, imports, and whatever your process can reach. NVIDIA's own docs tell you to run agents in a sandbox, and they mean it — this is exec() with extra steps, by design. If you wouldn't run curl | sh from a model, don't run NOOA agents outside a container either.

My own bias here: I build AI products, and the biggest lesson from my last one was that quality came from composing many small, checkable steps — not from trusting one big end-to-end model call. Most agent frameworks fight that instinct: they want the composition described in their vocabulary of chains and graphs instead of the language I already work in. NOOA is the first design I've seen where the composition just is Python. Which is also why the sandbox warning matters double — when wiring agents into real systems gets this frictionless, you'll ship one faster than you audit it.

Should you use it?

Today: probably not in production. It's a 0.0.1.dev1 that isn't on PyPI yet.

But I'd bet on the direction. We spent two years building agent frameworks that look like workflow engines, and the results are brittle in ways every practitioner knows. "The programming language is the agent definition language" is the first framing I've seen that gets simpler as your agent gets bigger. Worst case, NOOA becomes the CoffeeScript of agents: the thing itself fades, but every framework after it steals the idea.

The examples directory is a genuinely good progressive tutorial — 11 numbered files from first generation method to MCP tools. Start with 03_codeact_tools.py; it's the one that made the design click for me.

Have you tried collapsing your agent stack into plain code? I'd like to hear where it broke.

The AI API price war, in actual numbers: what I'd run my agents on this month

frank chu — Sat, 08 Aug 2026 16:30:59 +0000

On July 30, OpenAI cut the price of GPT-5.6 Luna by 80%. Not a typo: $1.00/$6.00 per million tokens became $0.20/$1.20. Terra dropped 20% to $2/$12. The flagship Sol didn't move.

Meanwhile Anthropic's Sonnet 5 is sitting at an introductory $2/$10 that reverts to $3/$15 on September 1 — three weeks from now. And Alibaba priced Qwen3.8-Max at $2/$6 with $0.25 cached input.

Price is now a strategy weapon, and if you run agents, your bill is suddenly negotiable. Let's do the arithmetic nobody's announcement includes.

The board as of this week

Model	Input /1M	Output /1M	Notes
GPT-5.6 Luna	$0.20	$1.20	was $1/$6 until Jul 30
GPT-5.6 Terra	$2.00	$12.00	was $2.50/$15
GPT-5.6 Sol	$5.00	$30.00	unchanged
Claude Sonnet 5	$2.00	$10.00	$3/$15 after Sep 1
Claude Opus 5	$5.00	$25.00
Qwen3.8-Max	$2.00	$6.00	$0.25 cached input

What that means for a real workload

Take a workload I think is representative of a solo builder running a coding/ops agent daily: 3M input + 300K output tokens per day (agents are input-heavy — context re-reads dominate). Over 30 days that's 90M in / 9M out:

Model	Monthly cost
GPT-5.6 Luna	$28.80
Qwen3.8-Max	$234
Sonnet 5 (intro)	$270
Terra	$288
Sonnet 5 (from Sep 1)	$405
Opus 5	$675
Sol	$720

Two things jump out of that table.

First, the spread between cheapest and priciest is now 25x for the same token volume. A year ago the tiers were maybe 5x apart. The vendors are telling you, loudly, that most agent traffic doesn't need a flagship.

Second, caching changes the ranking. Qwen's $0.25 cached input means that same workload at a realistic 70% cache-hit rate lands around $124/month — agents re-read the same context constantly, so cached-input pricing is worth more to agent builders than headline price. Check your provider's caching discount before comparing base rates; it's where the real money is.

How I'd actually route

The mistake is picking one model. The move is routing by task risk:

High-volume, low-stakes (classification, extraction, summarizing logs, first-pass triage): Luna at $0.20/$1.20 is close to free. If you're doing this on a flagship, you're donating money.
The agent's main loop (code edits, multi-step tool use): Sonnet 5 or Terra tier. This is where quality drops actually cost you time.
Escalation only (gnarly debugging, architecture decisions, final review): Sol or Opus 5, invoked by the cheaper model when it's stuck, not by default.

The catch nobody prints

An 80% price cut on last quarter's mid-tier is also a signal about where that model sits in the lineup now. Vendors cut prices on tiers they've already beaten internally. You're not getting a discount on the frontier; you're getting the previous frontier at clearance. That's still a great deal — just don't confuse it with generosity.

And mind the reversion dates. Sonnet 5's intro pricing expiring Sep 1 is a 50% jump on both sides of the meter. If your margins depend on intro pricing, you don't have a margin — you have a countdown.

What's your actual monthly spend and split? Genuinely curious what routing setups people have landed on — drop your numbers in the comments

Making an AI-maintained wiki run on a free local model: what actually worked

frank chu — Sat, 08 Aug 2026 02:52:01 +0000

I've been building MindBase — Karpathy's "LLM-maintained wiki" pattern as an app. You write notes and drop in sources; the AI maintains a persistent wiki (writes and updates actual markdown pages, not RAG).

When I launched in July, you needed an AI editor and an API key to use it. The last three weeks were about removing both. Here's what I learned making the whole thing run on a free local model, with a real UI.

1. Small models can't chain tool calls — so stop asking them to

My first server-side pipeline was agentic: the model would call search, read pages, then write. With qwen3:14b on Ollama, it fell apart constantly — wrong tool, malformed arguments, loops that never terminated.

The fix was to remove the agency. Every wiki operation — ingest a note, rebuild the context, health-check, research — is now one completion against one strict JSON schema, with everything the model needs gathered into the prompt beforehand. Parse failure retries once with the error appended. That's the entire error handling.

Local models turn out to be very reliable at filling one schema. They're shaky at deciding what to do next. Design accordingly.

2. The 88-second blank screen: thinking-mode models

qwen3 and deepseek-r1 stream their reasoning into a hidden thinking field. My chat UI collected content — which stayed empty for 88 seconds while the model happily reasoned into a field nobody was reading.

One line (think: false on the Ollama API) took replies from 88.7s to 1.1s. If your local-model UI ever shows "nothing happening for a minute" — check this first.

3. Trust comes from one checkbox

The biggest complaint about v1 wasn't quality — it was that ingestion felt like a black box. The AI read your note and something changed in your wiki.

Karpathy's original spec actually has a step my v1 skipped: the LLM discusses takeaways with you before writing. So now every ingest returns takeaways + a plan of proposed updates, each with a checkbox. Nothing is written until you hit Apply.

Same model, same output — but showing the plan before writing fixed the trust problem almost entirely.

4. Make the two layers visible

The pattern has two layers: your raw notes (append-only, yours) and the wiki (AI-maintained). In v1 this existed only in the folder structure, and new users never got it.

Now every note carries a status chip: ✨ Add to wiki while the note is newer than the last build, ✓ In wiki · 2 pages after digestion. Edit the note and it lights up again.

The implementation detail I like: the chip's state is just "file mtime newer than context.md" — the exact rule the build pipeline uses to find undigested sources. No new state, so the UI and the pipeline can't disagree.

5. Hardware detection beats a model dropdown

Asking users to pick an Ollama model is asking them to already be an expert. The setup wizard now reads your RAM/CPU and recommends what actually fits: 8GB → llama3.2:3b, 16GB → qwen3:8b, 32GB+ → qwen3:14b. Then it installs, pulls with a progress bar, and verifies with a real one-token generation (checking that the API answers is not the same as checking the model works — I learned that one the embarrassing way).

Where it's at

Everything is markdown on disk, MIT-licensed, works with Claude Code / Cursor / any MCP client — and now standalone in the browser with zero subscriptions.

Repo: https://github.com/frankchu91/mindbase

If you run local models: what's the biggest model you'd realistically dedicate to background synthesis work? Trying to decide how far to push the recommendation table.

I built an implementation of Karpathy's LLM wiki idea (MCP server, open source)

frank chu — Tue, 21 Jul 2026 05:36:00 +0000

I built an implementation of Karpathy's "LLM wiki" idea: instead of
doing RAG over your notes at query time, the model maintains a wiki as you add sources. It's an MCP server plus a small web UI. MIT licensed, everything is markdown on your own disk.

https://github.com/frankchu91/mindbase

How it works

When you add a source (a thought, a PDF, a URL), the model updates the relevant wiki pages and rewrites context.md, a page that summarizes your current position on the topics you're tracking. When you ask a question later, the answer is mostly already written — the model reads context.md and the linked pages instead of re-deriving everything from raw chunks.

The data layout has three parts with different owners:

sources/       you write, the model reads. Append-only.
context.md     the model writes, you read.
README.md      per-project rules. You write, the model follows.

The append-only guarantee is enforced by tooling, not prompts. Each
operation runs as a sub-agent with a restricted tool list — the agent that rebuilds context.md has no general file-write tool, only one MCP call that does an atomic write with a snapshot of the previous version. I started with prompt-based rules and they held most of the time, which isn't good enough for a layer that's supposed to never be rewritten.

Links between pages are typed (mentions, elaborates, supersedes,
contradicts) and stored in a sqlite index derived from the markdown. The lint command uses this to report actual conflicts, e.g.: a post you saved yesterday claims prefix caching helps long conversations most, but your notes from March say the benefit concentrates in short agentic loops. It also reports orphan pages and stale claims.

There are several other implementations of this pattern in the comments of Karpathy's gist. The main things mine does differently: it runs as an MCP server so it works inside Cursor, Windsurf, Cline, Claude Desktop, and Claude Code, and it has the typed link index for contradiction detection.

Install

Any MCP client — one entry in the config:

{ "mcpServers": { "mindbase": { "command": "npx", "args": ["-y", "mindbase-mcp"] } } }

Claude Code has a fuller version with slash commands and the
sub-agents:

/plugin marketplace add frankchu91/mindbase
/plugin install mb@mindbase

Day to day I use three commands: /mb:contribute to add something,
/mb:build to regenerate context.md, /mb:ask to query. The data
folder can be opened directly as an Obsidian vault.

Current limitations

The interactive ingest flow (model summarizes the source and waits for approval before writing) only works in Claude Code. Other editors get a simpler version. The web UI is for browsing and editing, not running operations. No mobile, no sync.

It's early. I'm looking for around 30 initial users and will help with setup personally. If you try it and something breaks, open an issue — I'm fixing things daily right now.

https://github.com/frankchu91/mindbase