The pivot
I spent the last months building a local multi-agent CLI code generator - planner → coder → tester → executor → reviewer, running on open models via Ollama. It closed the loop: it wrote code, ran it, tested it, and fixed it.
It worked. And that's exactly the problem: it solved a problem I don't have. Writing code for files on disk is a solved niche - I was competing with every other code-gen tool for a user who already has one.
What I actually do all day is 3D. So the project became something else: an AI assistant that lives inside Blender and holds real tools for working with models, not just a text box that emits code.
What works right now
The add-on is called Game Asset Assistant, targets Blender 4.2+, and ships as a proper extension zip.
Chat where you work
Docked in the 3D Viewport sidebar (N → AI tab)
Also opens as a standalone floating window with Ctrl+Alt+A
Streamed responses, message history, stop/clear buttons
It knows your scene
Every request carries a compact snapshot injected into the system prompt:
Blender 4.2.0
Scene: Scene | fps=24 | units=METRIC | render=CYCLES
Objects: 12
- Cube [MESH] loc=(0.00, 0.00, 0.00) rot_deg=(0.0, 0.0, 0.0) scale=(1.00, 1.00, 1.00) verts=8 edges=12 faces=6 | uv=UVMap | materials=Steel | modifiers=SUBSURF Selected: Cube Active: Cube | mode=OBJECT So when you ask "how many polygons does the active object have?", the model answers about your object - by name.
One click from answer to action
When a reply contains a
python block, it appears in a box with two buttons:
Run code - executes it in the current scene on the main thread, captures print() output and tracebacks, and pushes both back into the chat. That closes the loop: the model writes, Blender executes, the model sees the error and fixes it.
To Text Editor - saves it as a text datablock for manual review.
Backends
Local (Ollama, e.g. qwen2.5-coder:7b) or any OpenAI-compatible API - switchable in preferences, no code changes.
The part that broke my brain: Blender forbids threads
Here's the non-obvious bit worth the whole post.
Blender's API docs say it plainly: Python threads that outlive the script cause hard-to-diagnose crashes in Blender's own drawing code. So the obvious approach - "spawn a thread, call the LLM API, update the UI when it returns" - is off the table.
The add-on therefore never blocks and never spawns a thread. Instead:
Blender (main thread) worker process network
┌────────────────────────┐ ┌──────────────┐
│ operator appends a │commands │ │
│ JSON line ─────────────┼────────→│ worker.py │──HTTP──→ Ollama / API
│ │ │ │
│ bpy.app.timers, │events │ │
│ every 0.1s, reads new │←────────┤ │
│ lines ─────────────────┤ └──────────────┘
└────────────────────────┘
Commands are just JSON objects appended to commands.jsonl.
Events (chunk, done, error) are appended to events.jsonl.
A bpy.app.timers callback pumps events every 0.1 s - and a timer is not a thread, it's a callback Blender invokes on the main thread between frames. That's the only legal way to do background work.
Side effect: the whole add-on ships without any third-party dependency, and the protocol already has room for tool-call requests in a later phase.
What's not there yet
Being honest about the gaps:
No structured tool-calling - the model emits a code block and we exec() it
No confirmation for destructive operations (the prompt asks the model to warn; that's a request, not a guarantee)
No undo grouping - one Ctrl+Z won't roll back a whole script
UV, rigging and texturing exist only as chat advice, not as one-click tools
Scene snapshot is blind to UV islands, texel density, weights and armature pose
Roadmap
Phase What Why first
0 Richer snapshot, undo-push, destructive-op confirmation Cheap, unlocks everything after
1 Tool-calling over the JSONL channel, executed on the main thread with a registry + JSON schemas The reliability jump: from "80 lines of text that may fail" to named calls
2 Texturing: PBR material presets, texture assignment, packing, bake helpers Stable API, high daily value
3 UV: quick unwrap/pack/texel-density, seam-by-angle, and a UV diagnostics report (overlaps, density) Most repetitive work; the report lets the model read the result and propose fixes
4 Auto-rig: metarig → Rigify, auto-weights, weight validation, export checks Hardest - errors are silent, so diagnostics matter more than generation
5 One-click pipelines: GBC Prep, Char Prep, PBR Bake, Game Export Combines 2–4 into buttons
6 Teach the model: tool catalog in the prompt, domain recipes, few-shot examples Ongoing, parallel
Integration with other Blender add-ons (delegating to UV packers, for example) is on the list - but the final roadmap is still open. I'd rather ship phases 0–3 and let real usage decide what phase 4 looks like.
The plan from here
Post progress as it lands. If you're working on AI × 3D pipelines, or you've wrapped a DCC tool for an LLM - I'd love to compare notes.
Repo link in the comments.
#blender #ai #python #gamedev #3d #showdev
Top comments (1)
github.com/SickSect/BlenderAgent