<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Dhotiiiii</title>
    <description>The latest articles on DEV Community by Dhotiiiii (@rustam335).</description>
    <link>https://dev.to/rustam335</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4131798%2F39ef84e6-f668-4d79-ba5e-b6c3a3bf2e23.jpg</url>
      <title>DEV Community: Dhotiiiii</title>
      <link>https://dev.to/rustam335</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rustam335"/>
    <language>en</language>
    <item>
      <title>Ask ten LLMs for a Blender 5.0 script. Then run it in Blender 5.0.</title>
      <dc:creator>Dhotiiiii</dc:creator>
      <pubDate>Tue, 29 Sep 2026 07:28:01 +0000</pubDate>
      <link>https://dev.to/rustam335/ask-ten-llms-for-a-blender-50-script-then-run-it-in-blender-50-1pe3</link>
      <guid>https://dev.to/rustam335/ask-ten-llms-for-a-blender-50-script-then-run-it-in-blender-50-1pe3</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/kaggle-2026-09-23"&gt;Kaggle Benchmarking Challenge&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Benchmarked
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The question
&lt;/h3&gt;

&lt;p&gt;Ask an LLM for a Blender Python script and it will usually give you one that looks right. Whether it runs depends on which Blender you paste it into. The &lt;code&gt;bpy&lt;/code&gt; API changes on every major release: enums get renamed, attributes are removed, node sockets move, whole subsystems (compositor, sequencer, animation) get restructured. Most of the Blender code a model has seen was written for 2.8 to 3.x. Ask for 5.0 and you often get a 3.x script that dies on the first changed line.&lt;/p&gt;

&lt;p&gt;Text similarity cannot see this. &lt;code&gt;mesh.use_auto_smooth = True&lt;/code&gt; reads perfectly and raises &lt;code&gt;AttributeError&lt;/code&gt; in 4.1 and later. The only judge that counts is the Blender version you asked for. So the benchmark runs every answer in that exact version.&lt;/p&gt;

&lt;p&gt;Two questions per answer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;runs&lt;/strong&gt;: does the script, followed by a per-task assert script, exit 0 in a headless Blender of the target version?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;aware&lt;/strong&gt;: does the answer say what changed? Each answer must end with a &lt;code&gt;WATCH OUT&lt;/code&gt; block naming every API that changed for that version and its replacement, or &lt;code&gt;- none&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These come apart in interesting ways, which is most of what follows.&lt;/p&gt;

&lt;h3&gt;
  
  
  The task
&lt;/h3&gt;

&lt;p&gt;30 scripting tasks, each asked for Blender 3.6, 4.2, 4.5 and 5.0 (119 prompts per model; one task has no 3.6 form). The tasks cover 33 verified API changes: EEVEE engine ids, the boolean solver enum, auto-smooth and custom normals, &lt;code&gt;calc_normals&lt;/code&gt;, Principled BSDF socket names, the OBJ exporter, node group sockets, bone collections, slotted actions, the compositor node tree moving off &lt;code&gt;Scene&lt;/code&gt;, the sequencer's &lt;code&gt;sequences&lt;/code&gt; becoming &lt;code&gt;strips&lt;/code&gt;, and so on. Two control tasks use APIs that did not change, to check the grader does not punish ordinary correct code.&lt;/p&gt;

&lt;p&gt;Every model gets the same system prompt, temperature 0, no tools, no retrieval. The Kaggle task ships the four Blender Linux builds in a dataset, extracts them at the start of the run, and grades each answer by launching &lt;code&gt;blender -b --factory-startup --python&lt;/code&gt; on the target build. A hand-written reference answer per task and version passes its assert in all four builds (119/119), which is the evidence that every prompt is answerable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Models Tested
&lt;/h2&gt;

&lt;p&gt;Ten models through the Kaggle Model Proxy, chosen to span vendors, sizes, open and closed weights, and reasoning styles, within a daily proxy budget of a few dollars:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gpt-5.5, claude-sonnet-5, gemini-3.1-pro-preview&lt;/td&gt;
&lt;td&gt;current frontier from three vendors&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.7-flash, gemini-3.8-flash, claude-haiku-4-5&lt;/td&gt;
&lt;td&gt;the small and cheap tier people actually script with&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;grok-4.20-0309-reasoning&lt;/td&gt;
&lt;td&gt;fourth vendor, reasoning model (grok-4.6 is listed by the proxy but returned 404 on every call)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;deepseek-r1-0528&lt;/td&gt;
&lt;td&gt;open-weight reasoning model that returns its thinking inline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3-coder-480b-a35b-instruct&lt;/td&gt;
&lt;td&gt;open-weight model tuned for code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b&lt;/td&gt;
&lt;td&gt;open-weight OpenAI model&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Total proxy spend for the ten leaderboard runs: $11.23. gpt-5.5 alone was $4.44; qwen3-coder was $0.03.&lt;/p&gt;

&lt;h2&gt;
  
  
  Findings
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Results
&lt;/h3&gt;

&lt;p&gt;1190 graded answers. Aware is re-graded offline with the current parser so every model is judged the same way (more on that under methodology).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;runs&lt;/th&gt;
&lt;th&gt;aware&lt;/th&gt;
&lt;th&gt;runs but unaware&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gpt-5.5&lt;/td&gt;
&lt;td&gt;95%&lt;/td&gt;
&lt;td&gt;71%&lt;/td&gt;
&lt;td&gt;26%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;claude-sonnet-5&lt;/td&gt;
&lt;td&gt;94%&lt;/td&gt;
&lt;td&gt;53%&lt;/td&gt;
&lt;td&gt;42%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.8-flash&lt;/td&gt;
&lt;td&gt;92%&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;17%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.1-pro-preview&lt;/td&gt;
&lt;td&gt;92%&lt;/td&gt;
&lt;td&gt;76%&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.7-flash&lt;/td&gt;
&lt;td&gt;89%&lt;/td&gt;
&lt;td&gt;73%&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;grok-4.20-reasoning&lt;/td&gt;
&lt;td&gt;82%&lt;/td&gt;
&lt;td&gt;29%&lt;/td&gt;
&lt;td&gt;57%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;claude-haiku-4-5&lt;/td&gt;
&lt;td&gt;72%&lt;/td&gt;
&lt;td&gt;31%&lt;/td&gt;
&lt;td&gt;48%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3-coder-480b&lt;/td&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;td&gt;29%&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;deepseek-r1&lt;/td&gt;
&lt;td&gt;67%&lt;/td&gt;
&lt;td&gt;39%&lt;/td&gt;
&lt;td&gt;33%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b&lt;/td&gt;
&lt;td&gt;61%&lt;/td&gt;
&lt;td&gt;29%&lt;/td&gt;
&lt;td&gt;37%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F42ynzp29g0sc6sko6nlw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F42ynzp29g0sc6sko6nlw.png" alt="Share of scripts that run on the target version, by Blender version" width="800" height="351"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Runs by target version:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;3.6&lt;/th&gt;
&lt;th&gt;4.2&lt;/th&gt;
&lt;th&gt;4.5&lt;/th&gt;
&lt;th&gt;5.0&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gpt-5.5&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;87%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;claude-sonnet-5&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.8-flash&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;87%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.1-pro-preview&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.7-flash&lt;/td&gt;
&lt;td&gt;93%&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;grok-4.20-reasoning&lt;/td&gt;
&lt;td&gt;86%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;87%&lt;/td&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;claude-haiku-4-5&lt;/td&gt;
&lt;td&gt;79%&lt;/td&gt;
&lt;td&gt;77%&lt;/td&gt;
&lt;td&gt;73%&lt;/td&gt;
&lt;td&gt;60%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3-coder-480b&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;67%&lt;/td&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;td&gt;60%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;deepseek-r1&lt;/td&gt;
&lt;td&gt;86%&lt;/td&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;td&gt;63%&lt;/td&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b&lt;/td&gt;
&lt;td&gt;93%&lt;/td&gt;
&lt;td&gt;53%&lt;/td&gt;
&lt;td&gt;57%&lt;/td&gt;
&lt;td&gt;43%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Pooled over all ten models: 91% of scripts run on 3.6, 81% on 4.2, 83% on 4.5, 70% on 5.0. Awareness falls faster: 81%, 48%, 40%, 35%.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Every model is worst on 5.0, and the 5.0 failures are the same five APIs
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu5wbmwp4vow0n4t7vnli.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu5wbmwp4vow0n4t7vnli.png" alt="Run rate by change category and version, all models" width="711" height="1050"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Three categories go to 0% on 5.0 across all ten models: the compositor (&lt;code&gt;Scene.node_tree&lt;/code&gt; is gone; the compositor is now a node group on &lt;code&gt;scene.compositing_node_group&lt;/code&gt;), the boolean solver (&lt;code&gt;'FAST'&lt;/code&gt; is now &lt;code&gt;'FLOAT'&lt;/code&gt;, and &lt;code&gt;'MANIFOLD'&lt;/code&gt; is new), and the sequencer (&lt;code&gt;SequenceEditor.sequences&lt;/code&gt; is now &lt;code&gt;strips&lt;/code&gt;, and &lt;code&gt;new_effect()&lt;/code&gt; takes &lt;code&gt;length=&lt;/code&gt; instead of &lt;code&gt;frame_end=&lt;/code&gt;). Animation is at 10%: &lt;code&gt;Action.fcurves&lt;/code&gt; moved under slotted actions (&lt;code&gt;action.layers[0].strips[0].channelbag(slot).fcurves&lt;/code&gt;). Not one model knew any of these. The training data has not caught up with 5.0 and no amount of scale fixes that. The best model on 5.0 (gpt-5.5 and gemini-3.8-flash, 87%) is still exactly as lost on these five as gpt-oss-120b.&lt;/p&gt;

&lt;p&gt;The failures on 4.2 and later are different in character. They are changes from 4.0 to 4.2 that the models have partly absorbed (counts pooled over 4.2, 4.5 and 5.0): &lt;code&gt;use_auto_smooth&lt;/code&gt; removed (19 failures), the legacy OBJ exporter &lt;code&gt;bpy.ops.export_scene.obj&lt;/code&gt; removed (15), &lt;code&gt;Mesh.calc_normals&lt;/code&gt; removed (14), &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; renamed (14), Principled BSDF &lt;code&gt;Specular&lt;/code&gt; renamed to &lt;code&gt;Specular IOR Level&lt;/code&gt; (9). The frontier models mostly get these right; the open-weight models mostly do not. gpt-oss-120b is the clearest case: 93% on 3.6, 53% on 4.2. It knows Blender 3.x well and stopped there.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. A rename that was reverted splits the models into two camps
&lt;/h3&gt;

&lt;p&gt;The render engine id was &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; in 3.6, became &lt;code&gt;BLENDER_EEVEE_NEXT&lt;/code&gt; in 4.2 when EEVEE Next shipped, and went back to &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; in 5.0 when the legacy engine was deleted. Every model failed this task somewhere:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;camp&lt;/th&gt;
&lt;th&gt;models&lt;/th&gt;
&lt;th&gt;3.6&lt;/th&gt;
&lt;th&gt;4.2&lt;/th&gt;
&lt;th&gt;4.5&lt;/th&gt;
&lt;th&gt;5.0&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;learned the 4.2 rename&lt;/td&gt;
&lt;td&gt;gpt-5.5, claude-sonnet-5, claude-haiku-4-5&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;fail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;never learned it&lt;/td&gt;
&lt;td&gt;both gemini flash models, gemini-3.1-pro, grok, deepseek, qwen, gpt-oss&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;fail&lt;/td&gt;
&lt;td&gt;fail&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The second camp is right on 5.0 by accident: they write &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; for every version. The first camp is wrong on 5.0 with full confidence. gpt-5.5's WATCH OUT for the 5.0 prompt reads:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;scene.render.engine = 'BLENDER_EEVEE'&lt;/code&gt;: changed in Blender 4.2 when legacy EEVEE was replaced by EEVEE Next; use &lt;code&gt;scene.render.engine = 'BLENDER_EEVEE_NEXT'&lt;/code&gt; in Blender 5.0.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Claude Sonnet 5's answer for 5.0 is the most instructive failure in the whole set. Its WATCH OUT names both identifiers and says the old one "no longer refers to the current EEVEE renderer in 5.0", so the aware check passes. The script then sets &lt;code&gt;BLENDER_EEVEE_NEXT&lt;/code&gt; and fails. It knows there is a story here and tells the wrong half of it.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Running is not knowing: the "runs but unaware" gap
&lt;/h3&gt;

&lt;p&gt;The gap column in the first table is the share of answers where the script ran but the WATCH OUT block missed the change. It is the number I find most useful, because it separates two very different kinds of model.&lt;/p&gt;

&lt;p&gt;gemini-3.8-flash and gpt-5.5 have small gaps (17% and 26%): when their code works, they can usually say why the old code would not. Claude Sonnet 5 runs 94% of the time but names the change in only 53%; grok-4.20 runs 82% and names it in 29%. Both write &lt;code&gt;- none&lt;/code&gt; under WATCH OUT for most prompts: Sonnet in 85 of 119 answers, grok in 110. qwen3-coder and Haiku do the same (115 and 109), and it is why the whole bottom half of the table sits at 29% to 39% aware. For every 2.80-era change (&lt;code&gt;scene.objects.link&lt;/code&gt; becoming &lt;code&gt;collection.objects.link&lt;/code&gt;, lamps becoming lights, &lt;code&gt;dupli&lt;/code&gt; becoming &lt;code&gt;instance&lt;/code&gt;) they silently write the modern form and declare that nothing changed. That is a judgement about what counts as "changed", not a parse failure, and I decided not to special-case it: the prompt asked for changes relative to older Blender, and 3.6 is old enough that 2.80 changes are still the ones people trip over in old tutorials.&lt;/p&gt;

&lt;p&gt;The reverse gap, "aware but breaks", is small everywhere (1% to 7%). 41 answers named the change and still emitted broken code. Sonnet's EEVEE answer above is one. Three of Haiku's eight are the OBJ exporter: it knows &lt;code&gt;export_scene.obj&lt;/code&gt; became &lt;code&gt;wm.obj_export&lt;/code&gt;, then passes the old &lt;code&gt;use_selection=&lt;/code&gt; keyword to the new operator.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. The awareness curve is steeper than the run curve
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1u9jqxxusw33qng0uvm4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1u9jqxxusw33qng0uvm4.png" alt="Share of answers that name the API change, by Blender version" width="800" height="351"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On 3.6, awareness is 69% to 97%: everybody knows the 2.80 changes. By 4.2 the field splits. The three Gemini models and gpt-5.5 stay between 60% and 80% through 5.0, Sonnet sits at 43% to 53%, and everyone else drops below 30% by 4.5 and stays there. The scripts of that last group still run 57% to 87% of the time on 4.5 because most tasks touch APIs that did not change in that version. They are right without knowing why, and that is exactly the situation where a user cannot tell a stale answer from a current one.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Temperature 0 is not a determinism guarantee
&lt;/h3&gt;

&lt;p&gt;The same model, prompt and settings, run four times through the proxy (gemini-3.7-flash across task versions): 107, 107, 103 and 106 of 119 scripts ran. Zero of the 119 answers were byte-identical between two runs. That is 3 to 4 points of noise on top of the sampling interval, so gpt-5.5 at 95%, Sonnet at 94% and the two Gemini models at 92% are a tie. Sonnet and gemini-3.8-flash actually swapped places between two task versions. The differences that survive the noise are the tiers: the five models above 89%, grok alone at 82%, and the four between 61% and 72%.&lt;/p&gt;

&lt;h3&gt;
  
  
  Methodology notes (the parts that would have silently corrupted the numbers)
&lt;/h3&gt;

&lt;p&gt;Three grading problems showed up in real runs. Each one would have produced a plausible-looking leaderboard that was wrong.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reasoning leaks into the answer.&lt;/strong&gt; deepseek-r1 returns its thinking inline as &lt;code&gt;&amp;lt;think&amp;gt;...&amp;lt;/think&amp;gt;&lt;/code&gt;, and in 29 of 119 answers the thinking contained a draft code fence. The first notebook version ran the draft. Fixed by stripping the block before parsing; deepseek's run rate moved from 62% to 67% and its awareness came down, because the rehearsed WATCH OUT inside the thinking had inflated it. The Kaggle leaderboard still shows 62% for deepseek because that is what the notebook version that ran it computed; the tables here are re-graded offline from the stored answers, and the changed scripts were re-run in the same Blender builds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Output caps truncate thinking models.&lt;/strong&gt; Gemini 3.x thinking tokens count against &lt;code&gt;max_tokens&lt;/code&gt; through the proxy. gemini-3.8-flash spends 12 to 16 thousand tokens thinking about auto-smooth, hit an 8192 cap once, and its "SyntaxError: unterminated string literal" was a truncation, not a model error. The task now raises the cap to 16384 for reasoning models and re-asks once at 32768 if an answer lands within 32 tokens of the cap. One answer needed that retry.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proxy errors are not model errors.&lt;/strong&gt; Rate limits (429) and daily-quota rejections (403) come back as exceptions. The task retries with backoff, excludes prompts that are still lost from the denominator, and aborts if more than 10% are lost rather than record a score. An early version recorded qwen3-coder at 0.8% because 118 of 119 calls were rate-limited; that version is gone.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two limits worth stating. The aware check matches identifiers by substring and quoted enum values exactly, so it is a floor on awareness, not a proof of understanding. And 30 tasks is enough to show the shape of the drift, not to rank models that are three points apart.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where this came from
&lt;/h3&gt;

&lt;p&gt;The case bank and the WATCH OUT contract come from &lt;a href="https://github.com/Rustam335/bpy-compass" rel="noopener noreferrer"&gt;bpy-compass&lt;/a&gt;, a version-aware Blender assistant I built for the Sanity challenge. Its evaluation compared one model with and without a retrieval layer over the release notes. The question this benchmark asks is the one I wanted answered before building that: how bad is the drift without retrieval, for which models, and where exactly. The answer is that on 5.0 it is bad for everyone, and that retrieval over release notes is not a nicety for Blender scripting, it is the difference between the compositor working and not.&lt;/p&gt;

&lt;h2&gt;
  
  
  My Benchmark
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Benchmark on Kaggle: &lt;a href="https://www.kaggle.com/benchmarks/tasks/hoholalagaul/bpy-drift-runs/7" rel="noopener noreferrer"&gt;https://www.kaggle.com/benchmarks/tasks/hoholalagaul/bpy-drift-runs/7&lt;/a&gt; (task v7, ten models, public)&lt;/li&gt;
&lt;li&gt;Blender builds and grading wheel: &lt;a href="https://www.kaggle.com/datasets/hoholalagaul/bpy-drift-blender-builds" rel="noopener noreferrer"&gt;https://www.kaggle.com/datasets/hoholalagaul/bpy-drift-blender-builds&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Code, case bank, reference answers and the aggregation script: &lt;a href="https://github.com/Rustam335/bpy-drift-bench" rel="noopener noreferrer"&gt;https://github.com/Rustam335/bpy-drift-bench&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devchallenge</category>
      <category>kagglechallenge</category>
      <category>ai</category>
      <category>blender</category>
    </item>
    <item>
      <title>bpy-compass: Blender Python answers that run on the version you actually have</title>
      <dc:creator>Dhotiiiii</dc:creator>
      <pubDate>Mon, 28 Sep 2026 19:04:02 +0000</pubDate>
      <link>https://dev.to/rustam335/bpy-compass-blender-python-answers-that-run-on-the-version-you-actually-have-5gdi</link>
      <guid>https://dev.to/rustam335/bpy-compass-blender-python-answers-that-run-on-the-version-you-actually-have-5gdi</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/sanity"&gt;Sanity Challenge&lt;/a&gt;, Path One: Ship an Agent That Queries Real Content.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;You copy a render-setup script from a Blender 4.2 tutorial, run it in Blender 5.0, and get:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;TypeError: bpy_struct: item.attr = val: enum "BLENDER_EEVEE_NEXT" not found in ('BLENDER_EEVEE', ...)&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ask a plain LLM to write that script for 5.0 and it can hand you the very same line. That is exactly what happened in my eval (case 10 below).&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;bpy&lt;/code&gt; API breaks in almost every major release. &lt;code&gt;scene.objects.link&lt;/code&gt; died in 2.80, the context-override dict for operators died in 4.0, the EEVEE engine identifier changed in 4.2 and changed back in 5.0, the fast boolean solver was renamed in 5.0. Old tutorials keep ranking, and a plain LLM answers from a memory where every version is blended together.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;bpy-compass&lt;/strong&gt; answers Blender Python scripting questions &lt;strong&gt;for the Blender version you pick&lt;/strong&gt;, and tells you which popular pattern broke, in which version, and what replaced it. It reads a Sanity Context Knowledge Base built from the official release notes, a curated dataset of API changes and, on purpose, four stale tutorials. Every answer has three blocks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ANSWER&lt;/strong&gt;: a script valid for the requested version (3.6, 4.2, 4.5 or 5.0).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;WATCH OUT&lt;/strong&gt;: the popular patterns that stopped working, in which version, and what replaced them. Each bullet cites the Knowledge Base entry it came from.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SOURCES&lt;/strong&gt;: the Knowledge Base entry paths the agent actually read for this answer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the Knowledge Base has no entry for part of a question, the agent marks those lines &lt;code&gt;# NOT in Knowledge Base:&lt;/code&gt; and the UI highlights them, instead of inventing a citation.&lt;/p&gt;

&lt;p&gt;Ask &lt;em&gt;"Set the render engine to EEVEE and the boolean solver to the fast one"&lt;/em&gt; on &lt;strong&gt;4.2&lt;/strong&gt; and on &lt;strong&gt;5.0&lt;/strong&gt; and you get two different scripts (&lt;code&gt;BLENDER_EEVEE_NEXT&lt;/code&gt; / &lt;code&gt;FAST&lt;/code&gt; vs &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; / &lt;code&gt;FLOAT&lt;/code&gt;), each explaining why. A keyword search over the release notes returns both identifiers with no way to tell which one applies to you.&lt;/p&gt;

&lt;p&gt;To keep myself honest, every generated script is executed in headless Blender against an assert, for bpy-compass &lt;strong&gt;and&lt;/strong&gt; for the same model without the Knowledge Base. The results, failures included, are on the live &lt;code&gt;/eval&lt;/code&gt; page and in the table below.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;


&lt;div&gt;
  &lt;iframe src="https://loom.com/embed/8f5408db7eb94aad81d7fc9b1ab06a23" width="100%" height="400"&gt;
  &lt;/iframe&gt;
&lt;/div&gt;


&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3pvbsjkm49dqq6zbihpn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3pvbsjkm49dqq6zbihpn.png" alt="The same question on Blender 4.2 and 5.0: two different scripts, each with its Watch out" width="800" height="853"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Live app, no login: &lt;strong&gt;&lt;a href="https://bpy-compass.vercel.app" rel="noopener noreferrer"&gt;https://bpy-compass.vercel.app&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Try it in a minute
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Pick &lt;strong&gt;4.2&lt;/strong&gt;, click &lt;em&gt;"Set the render engine to EEVEE and the boolean solver to the fast one."&lt;/em&gt; You get &lt;code&gt;BLENDER_EEVEE_NEXT&lt;/code&gt; and &lt;code&gt;FAST&lt;/code&gt;. Switch to &lt;strong&gt;5.0&lt;/strong&gt; and click it again: &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; and &lt;code&gt;FLOAT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;On &lt;strong&gt;4.5&lt;/strong&gt;, click &lt;em&gt;"Join two meshes with a boolean union and apply it."&lt;/em&gt;, then &lt;strong&gt;Show what a stale tutorial would say&lt;/strong&gt; to see the old-tutorial version next to it, with every break named.&lt;/li&gt;
&lt;li&gt;Ask &lt;em&gt;"Add a text object that says Hello and extrude it 0.2 units."&lt;/em&gt; The Knowledge Base has no entry for text objects, so those lines come back highlighted as &lt;code&gt;# NOT in Knowledge Base:&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Open &lt;strong&gt;/eval&lt;/strong&gt; for every run, failures included.&lt;/li&gt;
&lt;/ol&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/&lt;/code&gt; chat with the version rail. Try the same question on 4.2 and 5.0.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/eval&lt;/code&gt; the headless-Blender results table, filled from Sanity &lt;code&gt;evalRun&lt;/code&gt; documents.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/about&lt;/code&gt; how it works, plus the Knowledge Base Issues screenshots before and after resolution.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Rate limit is 10 questions per minute per IP, so the OpenRouter key survives a public demo.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/Rustam335/bpy-compass" rel="noopener noreferrer"&gt;https://github.com/Rustam335/bpy-compass&lt;/a&gt;&lt;/strong&gt; (MIT)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;app/          chat + version rail · /eval · /about · /studio · api/chat/route.ts
lib/          model.ts (pinned model/provider/temperature) · prompt.ts · context-mcp.ts · sanity.ts · rate-limit.ts
sanity/       schemaTypes/ (apiChange, blenderVersion, testCase, evalRun) · seed/*.json
scripts/      eval.ts (baseline vs bpy-compass → blender -b → evalRun docs) · seed-api-changes.ts
kb-sources/   the four stale tutorials uploaded as Knowledge Base file sources + ATTRIBUTION.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Stack: Next.js App Router on Vercel, Vercel AI SDK with the MCP client, OpenRouter (&lt;code&gt;z-ai/glm-5.3-flash&lt;/code&gt;, provider pinned to &lt;code&gt;novita&lt;/code&gt;, temperature 0), Sanity Context MCP in &lt;code&gt;knowledge_base&lt;/code&gt; mode, Sanity Studio embedded at &lt;code&gt;/studio&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Used Sanity
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What the Knowledge Base is built from
&lt;/h3&gt;

&lt;p&gt;Three kinds of sources, all in one Knowledge Base (&lt;code&gt;bpy-compass&lt;/code&gt;):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Dataset source&lt;/strong&gt;: the project's own structured content. 27 &lt;code&gt;apiChange&lt;/code&gt; documents, each with &lt;code&gt;symbol&lt;/code&gt;, &lt;code&gt;kind&lt;/code&gt; (removed / renamed / behavior), &lt;code&gt;replacement&lt;/code&gt;, &lt;code&gt;before&lt;/code&gt; and &lt;code&gt;after&lt;/code&gt; code, a reference to the &lt;code&gt;blenderVersion&lt;/code&gt; document it changed in, and the release-note URL it was verified against. This is the part a keyword search cannot reconstruct: the &lt;em&gt;relationship&lt;/em&gt; between an old symbol, a version and its replacement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Website sources&lt;/strong&gt;: the official Python API release notes for 2.80, 3.6, 4.0, 4.1, 4.2, 4.4, 4.5 and 5.0 on developer.blender.org.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;File sources&lt;/strong&gt;: four tutorials written in the 2.7x style (&lt;code&gt;scene.objects.link&lt;/code&gt;, &lt;code&gt;obj.select = True&lt;/code&gt;, override dicts, &lt;code&gt;calc_normals&lt;/code&gt;). They are there to make the build &lt;em&gt;find&lt;/em&gt; the conflicts a real user would run into.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  What the build found
&lt;/h3&gt;

&lt;p&gt;Adding the stale tutorials next to the release notes made Sanity Context raise &lt;strong&gt;six Critical conflicts&lt;/strong&gt; on the Issues tab:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp9k8z5i6lm1gddegvg92.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp9k8z5i6lm1gddegvg92.jpg" alt="Issues tab after build 3: six pending conflicts" width="800" height="781"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two of them were plain wrong-vs-right and got a pick (for example, &lt;code&gt;Mesh.calc_normals()&lt;/code&gt; was removed in 4.0, not "available before 4.1"). &lt;strong&gt;Four of them were not conflicts at all&lt;/strong&gt;: both sides were true for different Blender versions (&lt;code&gt;BLENDER_EEVEE_NEXT&lt;/code&gt; in 4.2 vs &lt;code&gt;BLENDER_EEVEE&lt;/code&gt; in 5.0; solver &lt;code&gt;FAST&lt;/code&gt; up to 4.4 vs &lt;code&gt;FLOAT&lt;/code&gt; from 5.0). "Pick a winner" is the wrong tool for that, so I resolved those with version-scoped picks and wrote a standing Instruction:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Every bpy API fact in this knowledge base is scoped to a Blender version. Two sources that give different values for different Blender versions are NOT in conflict: record both, each labeled with its version range. When two sources disagree about the SAME version, the official Blender release notes and the apiChange dataset are ground truth over tutorials. Always keep the old form in the entry, labeled with the version it stopped working in and its replacement.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Saving the Instruction flagged a contradiction in the "Modifiers &amp;amp; Boolean Solvers" entry and rebuilt it. The entry now carries a solver table by version range and a version-safe selection snippet:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fphj9l0cmrbbirfbv5gai.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fphj9l0cmrbbirfbv5gai.jpg" alt="Rebuilt Boolean entry with solver identifiers by Blender version" width="800" height="816"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz607ine0fu6tnzc5h5qf.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz607ine0fu6tnzc5h5qf.jpg" alt="Issues tab with all conflicts resolved" width="800" height="781"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Final state: 14 resolved issues and 4 manual Instructions. Two of the Instructions came from the eval harness catching the Knowledge Base being wrong (more on that below).&lt;/p&gt;

&lt;p&gt;One lesson that cost me a rebuild: stale tutorials that carry an "intentionally outdated" banner produce &lt;strong&gt;zero&lt;/strong&gt; conflicts, because the build reads the banner and files them as history. Real stale tutorials have no banner, so mine do not either.&lt;/p&gt;

&lt;h3&gt;
  
  
  What the agent does with it
&lt;/h3&gt;

&lt;p&gt;The Context MCP endpoint &lt;code&gt;bpy-compass&lt;/code&gt; exposes the Knowledge Base only. Per request:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The server fetches the Knowledge Base outline (&lt;code&gt;initial_context&lt;/code&gt;) and inlines it into the system prompt together with the requested Blender version, so the model knows which entry paths exist.&lt;/li&gt;
&lt;li&gt;The model calls &lt;code&gt;knowledge_base_read&lt;/code&gt; with the paths it needs, up to five tool steps. The UI shows the paths as they are read.&lt;/li&gt;
&lt;li&gt;The system prompt grounds every answer in the Knowledge Base first: every WATCH OUT bullet must cite an entry path, and any code the Knowledge Base does not back must start with &lt;code&gt;# NOT in Knowledge Base:&lt;/code&gt; and is never cited as a source.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The baseline used for the eval gets the exact same model, provider, temperature, system prompt, output contract and user prompt. The only difference is the Knowledge Base: the baseline has no tools, no outline and no KB rules, and is told so.&lt;/p&gt;

&lt;h3&gt;
  
  
  Eval: same model, with and without the Knowledge Base
&lt;/h3&gt;

&lt;p&gt;Each script was run in headless Blender, one exact build per target version (4.2.23 LTS, 4.5.14 LTS, 5.0.1) with &lt;code&gt;--factory-startup&lt;/code&gt;, then the test case's assert script. A row passes only if the script ran &lt;strong&gt;and&lt;/strong&gt; the answer kept its contract: WATCH OUT names every API change the test case expects (old symbol and replacement), and SOURCES lists only Knowledge Base entries the agent actually read. Runs are stored as &lt;code&gt;evalRun&lt;/code&gt; documents in Sanity; &lt;code&gt;/eval&lt;/code&gt; shows one complete run (both contenders, every case), never a mix of runs.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;#&lt;/th&gt;
&lt;th&gt;Question (short)&lt;/th&gt;
&lt;th&gt;Target&lt;/th&gt;
&lt;th&gt;Plain LLM&lt;/th&gt;
&lt;th&gt;bpy-compass&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Create a triangle mesh object and link it to the scene&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Deselect all, select 'Cube', make it active&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Print world-space vertex coordinates&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Boolean UNION with a sphere, then apply the modifier&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;❌ script ran, but WATCH OUT never mentions the override dict → &lt;code&gt;temp_override&lt;/code&gt; change&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Material 'Glow', Principled BSDF emitting orange at strength 5&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Geometry Nodes group with Geometry in/out and a Float 'Scale' socket&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;Export selected objects to OBJ&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;Shade smooth by angle (30°)&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;'Mesh' object has no attribute 'use_auto_smooth'&lt;/code&gt; (wrote it anyway, with a comment saying it was removed)&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;EEVEE engine + 640×480 resolution&lt;/td&gt;
&lt;td&gt;4.2&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;EEVEE engine + 640×480 resolution&lt;/td&gt;
&lt;td&gt;5.0&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;enum "BLENDER_EEVEE_NEXT" not found&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;td&gt;Boolean DIFFERENCE with the fast solver&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;Boolean DIFFERENCE with the fast solver&lt;/td&gt;
&lt;td&gt;5.0&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;enum "FAST" not found in ('FLOAT', 'EXACT', 'MANIFOLD')&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ knew &lt;code&gt;FAST&lt;/code&gt; → &lt;code&gt;FLOAT&lt;/code&gt;, but created the cutter with &lt;code&gt;objects.new&lt;/code&gt; without linking it, then &lt;code&gt;select_set&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Pass rate&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;8/12&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;11/12&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;What the numbers say, honestly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GLM 5.3 Flash already knows the 2.80 and 4.0 breaks.&lt;/strong&gt; Link, select, &lt;code&gt;@&lt;/code&gt; matrix multiply, &lt;code&gt;temp_override&lt;/code&gt;, node-group sockets, the new OBJ exporter: the plain model's scripts get them right (its case 4 script runs; it only failed to &lt;em&gt;explain&lt;/em&gt; the override-dict break in WATCH OUT). A "plain LLMs get everything wrong" story would be false for this model, and I am not telling it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The gap is at the newest boundary, 4.5 → 5.0.&lt;/strong&gt; On cases 10 and 12 the baseline writes the 4.2–4.5 names (&lt;code&gt;BLENDER_EEVEE_NEXT&lt;/code&gt;, &lt;code&gt;FAST&lt;/code&gt;) into a 5.0 script and Blender rejects both. bpy-compass states each rename with a citation and gets the value right. Its one failure, case 12, is an ordinary scripting bug (an unlinked object), not a version mistake; it stays in the table.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The stricter harness cut both scores, and that is the point.&lt;/strong&gt; An earlier run scored 11/12 vs 12/12 when passing only meant "the script ran". After an independent review (below) the harness also checks WATCH OUT and SOURCES, and the baseline gets the identical prompt. Case 4 is the clearest example: the baseline's script runs, but its WATCH OUT never mentions that the context-override dict was removed in 4.0.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Knowledge Base is only as good as its sources, and the eval caught it.&lt;/strong&gt; Case 8 first &lt;em&gt;failed&lt;/em&gt; for bpy-compass: one entry offered &lt;code&gt;modifiers.new(..., 'SMOOTH_BY_ANGLE')&lt;/code&gt;, a modifier type that does not exist. The agent's own WATCH OUT had flagged that two entries disagreed. The fix was an Instruction (the real replacement for &lt;code&gt;use_auto_smooth&lt;/code&gt; is &lt;code&gt;bpy.ops.object.shade_smooth_by_angle&lt;/code&gt;), a rebuild of two entries, and a rerun. The failed run is still in Sanity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A reader reviewed the repo before publish and filed 13 issues.&lt;/strong&gt; The first: every non-5.x case used to run in the 4.5 build, so the one 4.2 case had executed in Blender 4.5.14. The harness now maps each target to its own build and verifies &lt;code&gt;blender --version&lt;/code&gt; before the first LLM call. The rest covered the eval (WATCH OUT/SOURCES were never checked, baseline and compass got different prompts, &lt;code&gt;/eval&lt;/code&gt; could pair results from different runs, &lt;code&gt;--only&lt;/code&gt; could silently run everything), the API route (message validation, the model-config guard only ran in eval, a rate limiter that never forgot an IP) and an inconsistent grounding policy. Every issue is fixed and referenced in the commits; the numbers above are from the rerun after those fixes.&lt;/li&gt;
&lt;li&gt;Two harness lessons: reasoning tokens count against the output budget (one baseline attempt returned 16k characters of thinking and no code, so reasoning is now capped equally for both contenders), and Sanity's CDN caches test-case documents (the scripts read with &lt;code&gt;useCdn: false&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Why the structured part matters
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;apiChange&lt;/code&gt; dataset source is what makes "which version" answerable. A release-notes page says a symbol was removed; it does not say what the replacement is for a user on 4.2 versus 5.0, and it does not know that a 2.7x tutorial still ranks on Google. The dataset ties symbol, version and replacement together, the Knowledge Base build merges it with the release notes and detects where the stale tutorials disagree, and the Instructions encode the one rule a human had to decide: &lt;em&gt;different versions are not a conflict, same version is, and release notes win&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Four target versions: 3.6, 4.2, 4.5 and 5.0.&lt;/li&gt;
&lt;li&gt;Twelve eval cases on one model (GLM 5.3 Flash). They show where the Knowledge Base helps this model, not how every model behaves; a stronger model may already know some of the 5.0 renames.&lt;/li&gt;
&lt;li&gt;The Knowledge Base is only as right as its sources. Case 8 needed a human-written Instruction to fix a wrong entry.&lt;/li&gt;
&lt;li&gt;Code outside the Knowledge Base is marked &lt;code&gt;# NOT in Knowledge Base:&lt;/code&gt;, not verified. It can still be wrong.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sanity Project Details
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Project ID:&lt;/strong&gt; &lt;code&gt;uuc8lnyk&lt;/code&gt; · dataset &lt;code&gt;production&lt;/code&gt; (public read)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Knowledge Base:&lt;/strong&gt; &lt;code&gt;bpy-compass&lt;/code&gt; (&lt;code&gt;kbMlSqMSn4Q9&lt;/code&gt;) · &lt;strong&gt;MCP endpoint:&lt;/strong&gt; &lt;code&gt;bpy-compass&lt;/code&gt; (Knowledge Base source only)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Schema:&lt;/strong&gt; &lt;code&gt;apiChange&lt;/code&gt; (27 docs), &lt;code&gt;blenderVersion&lt;/code&gt; (8), &lt;code&gt;testCase&lt;/code&gt; (12), &lt;code&gt;evalRun&lt;/code&gt; (all runs, including failed ones)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Studio:&lt;/strong&gt; &lt;a href="https://bpy-compass.vercel.app/studio" rel="noopener noreferrer"&gt;https://bpy-compass.vercel.app/studio&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Agent Session
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Agent session transcript:&lt;/strong&gt; &lt;a href="https://dev.to/agent_sessions/building-bpy-compass-sanity-context-knowledge-base-conflict-resolution-and-headless-blender-eval-ppi4qe?slice=Sanity%20Context%20and%20Knowledge%20Base%20builds"&gt;bpy-compass build session: Sanity Context and Knowledge Base builds&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The main Claude Code build session, cut to the Sanity part: enabling Context, the three Knowledge Base builds (0 → 2 → 6 conflicts), resolving the Issues, and wiring the Context MCP endpoint into the chat route.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>sanitychallenge</category>
      <category>sanity</category>
      <category>ai</category>
    </item>
    <item>
      <title>Give Your AI Agent WhatsApp Superpowers with MCP (Official Business API)</title>
      <dc:creator>Dhotiiiii</dc:creator>
      <pubDate>Fri, 18 Sep 2026 16:05:26 +0000</pubDate>
      <link>https://dev.to/rustam335/give-your-ai-agent-whatsapp-superpowers-with-mcp-official-business-api-3f9l</link>
      <guid>https://dev.to/rustam335/give-your-ai-agent-whatsapp-superpowers-with-mcp-official-business-api-3f9l</guid>
      <description>&lt;p&gt;What if your AI assistant could check your store's WhatsApp inbox, look up an order, and send the customer a reply — by itself?&lt;/p&gt;

&lt;p&gt;That's what the &lt;a href="https://modelcontextprotocol.io" rel="noopener noreferrer"&gt;Model Context Protocol&lt;/a&gt; (MCP) makes possible. MCP is now supported across the LLM ecosystem — claude.ai, ChatGPT, Gemini, Cursor, Windsurf, VS Code Copilot, and most agent frameworks speak it — and in this post I'll show you how to wire &lt;strong&gt;any&lt;/strong&gt; MCP-capable client to a live WhatsApp Business platform in under two minutes. No SDKs, no servers to host — it's a remote MCP server.&lt;/p&gt;

&lt;h2&gt;
  
  
  The server
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://tokolaku.id/developers" rel="noopener noreferrer"&gt;Tokolaku&lt;/a&gt; is a WhatsApp business platform for Indonesian SMBs built on Meta's &lt;strong&gt;official&lt;/strong&gt; WhatsApp Business API (Cloud API — so no gray-market gateways, no ban roulette). It exposes a hosted MCP server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://api.tokolaku.id/mcp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Transport:&lt;/strong&gt; Streamable HTTP, stateless&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Auth:&lt;/strong&gt; OAuth 2.1 (interactive clients) &lt;em&gt;or&lt;/em&gt; API key (programmatic)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tools:&lt;/strong&gt; send messages, browse inbox conversations, products, orders, manage webhooks&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Option 1: OAuth-capable clients — just paste the URL
&lt;/h2&gt;

&lt;p&gt;Any modern MCP client with OAuth discovery (claude.ai, ChatGPT, Gemini, LibreChat, and most hosted assistants) needs nothing but the URL:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Open your client's &lt;strong&gt;connector / MCP server settings&lt;/strong&gt; (e.g. &lt;em&gt;Settings → Connectors → Add custom connector&lt;/em&gt;)&lt;/li&gt;
&lt;li&gt;Paste &lt;code&gt;https://api.tokolaku.id/mcp&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;You'll be redirected to sign in with your Tokolaku account and approve access&lt;/li&gt;
&lt;li&gt;Done. Your assistant now has 6 read-only tools scoped to &lt;em&gt;your&lt;/em&gt; account: &lt;code&gt;list_channels&lt;/code&gt;, &lt;code&gt;list_conversations&lt;/code&gt;, &lt;code&gt;get_conversation&lt;/code&gt;, &lt;code&gt;list_products&lt;/code&gt;, &lt;code&gt;list_orders&lt;/code&gt;, &lt;code&gt;get_order&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Behind the scenes this uses PKCE, dynamic client registration (RFC 7591), and standard discovery metadata (RFC 8414/9728) — tokens are minted per user, short-lived, and revocable from the dashboard.&lt;/p&gt;

&lt;p&gt;Try prompts like:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Any unpaid orders from this week? Summarize who I should follow up with."&lt;/p&gt;

&lt;p&gt;"Which products got asked about most in yesterday's chats?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Option 2: Coding agents, CLIs &amp;amp; SDK frameworks — API key
&lt;/h2&gt;

&lt;p&gt;For write access (actually sending messages), use a &lt;strong&gt;secret API key&lt;/strong&gt; from the developer dashboard. Most coding agents and frameworks (Claude Code, Cursor, Windsurf, Cline, VS Code, OpenAI Agents SDK, LangChain, and friends) share the same &lt;code&gt;mcpServers&lt;/code&gt; config convention:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"tokolaku"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"http"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://api.tokolaku.id/mcp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"headers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"x-api-key"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"tk_live_sk_xxx"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Building your own agent instead? Point any MCP client SDK at the Streamable HTTP URL and pass the key as an &lt;code&gt;x-api-key&lt;/code&gt; (or &lt;code&gt;Authorization: Bearer&lt;/code&gt;) header — that's the whole integration.&lt;/p&gt;

&lt;p&gt;Tools follow your key's scopes (&lt;code&gt;tools/list&lt;/code&gt; reflects them), and write tools like &lt;code&gt;send_message&lt;/code&gt; go through the exact same billing, quota, and rate-limit path as the REST API — your agent can't do anything your key can't.&lt;/p&gt;

&lt;p&gt;A fun one to try in your coding agent of choice:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"A customer at +62812xxxx asked about their order this morning. Check the conversation, find the order status, and send them a polite update."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why this pattern is interesting
&lt;/h2&gt;

&lt;p&gt;Most "AI + WhatsApp" setups either screen-scrape the desktop app (fragile, ToS-risky) or make you build a bot pipeline first. A remote MCP server flips it: the &lt;em&gt;platform&lt;/em&gt; speaks MCP natively, and any agent — today's or next year's — gets structured, permissioned access to real business state: conversations, catalog, orders.&lt;/p&gt;

&lt;p&gt;The same endpoint also serves classic REST (&lt;code&gt;POST /api/v1/messages&lt;/code&gt;, webhooks with HMAC signatures) and official SDKs on &lt;a href="https://www.npmjs.com/package/tokolaku-sdk" rel="noopener noreferrer"&gt;npm&lt;/a&gt;, &lt;a href="https://pypi.org/project/tokolaku/" rel="noopener noreferrer"&gt;PyPI&lt;/a&gt;, and &lt;a href="https://packagist.org/packages/tokolaku/sdk" rel="noopener noreferrer"&gt;Packagist&lt;/a&gt; — so the MCP layer is a first-class citizen, not a demo.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Server metadata: &lt;a href="https://github.com/Rustam335/tokolaku-mcp" rel="noopener noreferrer"&gt;https://github.com/Rustam335/tokolaku-mcp&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;API reference &amp;amp; MCP docs: &lt;a href="https://tokolaku.id/api-docs" rel="noopener noreferrer"&gt;https://tokolaku.id/api-docs&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The platform (WhatsApp Business API from Rp99k/mo, zero message markup): &lt;a href="https://tokolaku.id/whatsapp-api" rel="noopener noreferrer"&gt;https://tokolaku.id/whatsapp-api&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Questions or weird edge cases? I'd love to hear what your agents do with a WhatsApp inbox. 👇&lt;/p&gt;

</description>
      <category>mcp</category>
      <category>ai</category>
      <category>whatsapp</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
