TL;DR — Carmakers are quietly choosing open-weight models over API-based assistants because a car has to work in a signal-dead parking garage and keep working for 15 years after the vendor stops caring. This episode looks at real deployments — from a Raspberry Pi running a 35B model as an in-car chat agent to Phi-2 running diagnostics on ECU-class hardware — and why owning the weights, not renting them, is becoming the default for in-cabin AI.
Your phone gets replaced every two or three years. Your car doesn't. The average vehicle on the road today is over 12 years old, and a new one sold this year is expected to still be running in 2041. Now ask yourself: which AI vendor is confidently offering an API contract that lasts until 2041? None of them. That mismatch — hardware with a multi-decade lifespan, paired with an AI industry that deprecates endpoints every 18 months — is the quiet reason open-weight models are showing up under the hood.
The parking garage problem
Every cloud-based car assistant has the same failure mode: the underground garage, the mountain tunnel, the rural dead zone, the country where the carrier partnership didn't get signed. A voice assistant that goes silent the moment signal drops isn't a degraded experience, it's a broken product. That's the design constraint a hobbyist project called CarWatch takes seriously: a Raspberry Pi 5 riding in the car, running a 35B-parameter Qwen3.6-A3B model fully locally, quantized down to 14.3GB with Unsloth's dynamic UD-Q3_K_S format. Measured on real hardware — a ~€300 Pi 5 with 16GB of RAM — it generates at 3.5 tokens/second with 25+ tok/s on prompt processing, sustaining 65°C with no cloud, no internet, no subscription, as detailed in the project's own benchmarks and independently written up by gokawiil.com.
The architecture is the interesting part: local is the product, online is the enrichment. Voice input, the assistant's answers, and the owner's-manual RAG (built from the actual 745-page manual, shipped on the SD card with page citations) all work with zero connectivity. Room posts and clip uploads queue in a persistent on-disk outbox and deliver late rather than getting lost. Only reachability, self-updates, and escalation to a bigger cloud model are online-only — and even the escalation path checks a local LAN model server first. That's not a hobbyist quirk; it's the same hierarchy any carmaker would need to build for a feature that has to survive fifteen years of connectivity gaps.
Diagnostics on hardware that already exists
Not every in-car AI needs 35B parameters and a dedicated Pi. A separate project, tinyllm-auto, targets the opposite end: can a conversational assistant run on existing ECU-class silicon, under 8GB of RAM, with no GPU at all? The answer, benchmarked on a Raspberry Pi 4 (8GB), was Phi-2 at 2.7B parameters, 4-bit GGML quantized down to 1.6GB on disk and 3.2GB at runtime, delivering 450ms first-token latency and 45 tokens/second generation — beating a target of 30 tok/s and staying under a 4GB memory ceiling. The project's own model bake-off is worth noting: TinyLlama (1.1B) scored only 62% on a 500-question automotive QA benchmark, Mistral-7B scored highest at 84% but needed 4.1GB, and Phi-2 landed at 78% accuracy as the best fit for the size and latency budget. That's a real trade-off table, not a vendor slide: bigger models answer more diagnostic questions correctly, but the ECU doesn't have the RAM to run them.
Use cases: where this actually gets deployed
Strip away the demo projects and three workloads keep showing up in both hobbyist repos and industry roadmaps:
Diagnostic code translation. "What does P0420 mean?" is exactly the kind of narrow, high-value query a small quantized model handles well — tinyllm-auto's benchmark shows 82% accuracy on diagnostic-code questions specifically, the strongest category in its 500-question test. A cloud LLM does this fine too, but only if the car has signal in the exact moment the check-engine light comes on, which is precisely when a driver is often in a garage or on a rural shoulder.
Owner's-manual Q&A with refusal behavior. CarWatch's RAG pipeline is built to refuse questions the manual doesn't answer, rather than hallucinate a torque spec. For a safety-adjacent domain like vehicle maintenance, a model that says "I don't know" is worth more than one that guesses fluently — and that refusal logic is something you control when you own the weights and the retrieval pipeline, not something you can guarantee from a black-box API.
Cabin monitoring and system self-awareness. CarWatch's assistant reads its own temperature, throttling state, fan, memory, disk, and network status live from the machine before answering questions about itself — the system prompt is built so an unmeasured value can never silently present as a known fact. That's a small but telling design pattern: grounding the model in what the hardware can actually sense, rather than letting it narrate confidently about things it has no sensor for.
The honest trade-off across all three: local models on car-grade hardware are slow by desktop standards — 3.5 tok/s on a Pi 5 running a 35B model is fine for a conversational reply, unusable for anything requiring fast back-and-forth. Carmakers accept that latency in exchange for an assistant that never goes dark.
Why carmakers own the weights, not rent the API
This isn't just a hobbyist pattern — it maps onto where the automotive industry is actually spending. According to an analysis from sbdautomotive.com on Li Auto's custom silicon push, computing capability is already the second-largest cost component in a vehicle after the battery, with per-vehicle compute costs projected to climb from around $2,000 today toward $8,000–$10,000 as multi-modal, always-on AI features expand. The piece frames the automaker's own chip investment as a bet that outsourcing the "digital soul" of the car — meaning the models and the silicon that run them — is as risky as outsourcing the engine was in the combustion era. Waymo has taken the parallel path on the perception side: a custom 5nm chip delivering roughly 1,000 TOPS of dedicated edge compute per vehicle, purpose-built to run neural networks on raw sensor data before it ever leaves the car, according to a technical breakdown from ai2.work. Different companies, same conclusion: if the model is core to the product, you want to own the weights and, increasingly, the silicon they run on.
Open weights fit that logic cleanly. A model you can download, quantize, and freeze onto a chip is a component you control for the vehicle's full service life — no API deprecation notice, no pricing change, no dependency on a vendor's roadmap outliving your warranty period. A closed API is a subscription with a kill switch fifteen years out; an open-weight checkpoint on an SD card is a part you can still order.
Credits & sources
Thanks to the maintainer of xen1024/CarWatch for the detailed, benchmarked write-up of a real offline in-car assistant, and to gokawiil.com for the independent Show HN coverage of that project. Thanks to the maintainer of sreekarvamsi/tinyllm-auto for publishing the model comparison and benchmark tables that made the ECU-scale trade-offs concrete. Thanks to sbdautomotive.com for the analysis of Li Auto's in-house chip strategy, and to ai2.work for the breakdown of Waymo's custom edge-compute silicon.
Tomorrow, Day 22 moves from the dashboard to the chassis: open-weight AI in robotics, and what changes when the model has to control a body instead of just talking to one.
Appendix — the field in one chart
Top comments (0)