DEV Community

Aaron Vaz
Aaron Vaz

Posted on Fully Autonomous

Hybrid AI setups seem to be most practical setup right now compared to fully local (for certain workflows)

I got tired of burning through cloud api quotas every time an agent writes boilerplate code.

So i tested a tri-tier setup: Gemini 3.8 Flash as the high-level architect, and a local Ternary Bonsai 2 9B MTP model as the actual coder.

Turns out it slashed cloud token usage by 92.7%.

architecture

how it works

  1. Gemini 3.8 Flash (planner): breaks down the task, sets up data structures, CSS tokens, and atomic function signatures (~290 tokens total).
  2. ConvAI Laya (gatekeeper): fast local router that decides if a step is safe to delegate.
  3. Bonsai 2 9B MTP (coder): runs locally via prism llama-server on CPU with MTP speculative decoding. it takes the signatures and outputs 100% of the raw code with thinking turned off.
[Task]
   │
   ▼
[Gemini Flash] ──(Interfaces & Constants)──┐
                                           │
                                           ▼
                                [Local Bonsai 2 9B]
                                           │
                                           ▼
                                [Clean Production Code]
Enter fullscreen mode Exit fullscreen mode

the test: building MONSOON

to test this, i had it build a complete website from scratch for a pop-up shop selling Indian coffee and single-estate cacao:

  • index.html: semantic markup, hero with steam particles, terroir origin cards, off-canvas drawer.
  • styles.css: warm earth luxury palette, glassmorphism, responsive grid, keyframe animations.
  • app.js: tasting flight builder state machine, live INR pricing calculator, and a countdown clock to the Bangalore pop-up.

token breakdown

file cloud tokens (planning) local bonsai tokens (code)
styles.css ~95 1,400
app.js ~85 1,200
index.html ~110 1,103
total ~290 3,703
  • cloud tokens spent: ~290
  • tokens offloaded locally: 3,703
  • net savings: ~92.7%

local bonsai clocked ~6 tok/s on CPU with near 100% draft acceptance via MTP.

repo with all the code & prompt wrappers:
https://github.com/MVYNE06/tri-tier-bonsai-orchestrator

Top comments (1)

Collapse
 
aistackguide profile image
AI Stack Guide •

The 92.7% figure measures the right thing but against the wrong denominator: cloud tokens saved is not cost saved unless the local side is free, and on CPU at ~6 tok/s you are paying for the coder in wall-clock time instead of API spend. That trade is often worth it, but it should be called a trade.

Where I have seen this split hold up is when the planner's output is both small and machine-checkable — signatures, types, a failing test to satisfy. If the planner only describes intent in prose, the local coder has to re-infer the same context and you pay twice, once in tokens and once in debugging.

Two numbers I would want before trusting it on a real repo: whether the generated files parse, lint and pass tests in CI (3,700 tokens of code that compiles is a very different result from 3,700 tokens that merely read well), and p50 latency per file. 1,400 tokens at 6 tok/s is about four minutes per file — fine for boilerplate, painful in a debugging loop.