<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jon Scott</title>
    <description>The latest articles on DEV Community by Jon Scott (@jonscott79).</description>
    <link>https://dev.to/jonscott79</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4165219%2Fbceb93d1-bee7-4604-acb8-77b3b1d1009a.jpg</url>
      <title>DEV Community: Jon Scott</title>
      <link>https://dev.to/jonscott79</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jonscott79"/>
    <language>en</language>
    <item>
      <title>My AI Math Tutor Started Making Up Pictures Instead of Drawing Them</title>
      <dc:creator>Jon Scott</dc:creator>
      <pubDate>Sat, 07 Nov 2026 04:20:26 +0000</pubDate>
      <link>https://dev.to/jonscott79/my-ai-math-tutor-started-making-up-pictures-instead-of-drawing-them-18e0</link>
      <guid>https://dev.to/jonscott79/my-ai-math-tutor-started-making-up-pictures-instead-of-drawing-them-18e0</guid>
      <description>&lt;p&gt;There's a particular kind of bug you don't expect when building an AI-powered math tutor.&lt;/p&gt;

&lt;p&gt;You expect incorrect calculations. You expect malformed LaTeX. You expect the occasional hallucinated explanation.&lt;/p&gt;

&lt;p&gt;You don't necessarily expect your LLM to start &lt;strong&gt;inventing images&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Apparently, when asked to provide visual content, my LLM decided that actually using the application's image-generation or graphing capabilities was optional.&lt;/p&gt;

&lt;p&gt;Instead, it started generating Markdown image references pointing to external image hosts.&lt;/p&gt;

&lt;p&gt;The images weren't real. The links weren't useful. The AI was essentially telling students, "Here's a picture!" and then pointing at something that didn't exist.&lt;/p&gt;

&lt;p&gt;Fantastic. Just fantastic. 😂&lt;/p&gt;

&lt;h2&gt;
  
  
  The AI equivalent of drawing something and pretending it's a photograph
&lt;/h2&gt;

&lt;p&gt;Here's the problem: Pythos already has an actual visualization system.&lt;/p&gt;

&lt;p&gt;It can generate interactive mathematical graphs. It has a dedicated visualization engine. It knows how to turn equations into something students can explore.&lt;/p&gt;

&lt;p&gt;But the LLM occasionally decided to skip all that and fabricate image references.&lt;/p&gt;

&lt;p&gt;This is a good example of why simply telling an LLM what tools are available isn't enough. You need to enforce the boundaries between what the model can describe and what the application can actually render.&lt;/p&gt;

&lt;h2&gt;
  
  
  So we gave it a cleanup crew
&lt;/h2&gt;

&lt;p&gt;I implemented a new Hallucinated Image Scrubber and Auto-Graph Recovery system.&lt;/p&gt;

&lt;p&gt;The changes touch three areas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;server/vizEngine/visualFidelity.js&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;server/server.js&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;app.js&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here's what changed:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Stop the hallucinations at the source&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The system prompt now explicitly prohibits fabricated Markdown image tags and external image-host URLs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Scrub fabricated image references&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Post-processing filters catch and remove hallucinated image links before they become part of the user-facing response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Recover the actual mathematics&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Rather than simply deleting the bad image references, Pythos attempts to extract the equation being described and convert it into a live &lt;code&gt;[GRAPH: ...]&lt;/code&gt; interactive canvas.&lt;/p&gt;

&lt;p&gt;In other words, if the AI tries to pretend it has drawn something, we try to make the application actually draw it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bigger lesson
&lt;/h2&gt;

&lt;p&gt;This is another reminder that building dependable AI applications isn't just about improving prompts.&lt;/p&gt;

&lt;p&gt;LLMs can generate convincing-looking content that has no relationship to anything the application can actually deliver.&lt;/p&gt;

&lt;p&gt;The solution isn't to blindly trust the model. It's to build systems around it that validate, constrain, and recover from its output.&lt;/p&gt;

&lt;p&gt;Pythos already uses deterministic mathematical engines to verify calculations. Now we're applying that same philosophy to visual output.&lt;/p&gt;

&lt;p&gt;Because apparently, I also need to teach my AI the difference between drawing a graph and making one up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The AI is supposed to help students learn mathematics, not become an abstract artist.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You can try Pythos here:&lt;/p&gt;

&lt;p&gt;🌐 &lt;a href="https://pythos.lanzar.me" rel="noopener noreferrer"&gt;https://pythos.lanzar.me&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source code:&lt;/p&gt;

&lt;p&gt;💻 &lt;a href="https://github.com/JonScott79/Pythos" rel="noopener noreferrer"&gt;https://github.com/JonScott79/Pythos&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;And yes, the images were fake. The bug was real.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>showdev</category>
      <category>webdev</category>
    </item>
    <item>
      <title># We Put Our Math AI Through 50,000 Problems. It Passed All of Them in 11.4 Seconds.</title>
      <dc:creator>Jon Scott</dc:creator>
      <pubDate>Fri, 06 Nov 2026 03:48:10 +0000</pubDate>
      <link>https://dev.to/jonscott79/-we-put-our-math-ai-through-50000-problems-it-passed-all-of-them-in-114-seconds-l1f</link>
      <guid>https://dev.to/jonscott79/-we-put-our-math-ai-through-50000-problems-it-passed-all-of-them-in-114-seconds-l1f</guid>
      <description>&lt;p&gt;&lt;strong&gt;50,000 problems. 10 mathematical and scientific categories. 100% correctly solved. Zero incorrect answers delivered.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We just completed a major milestone in the development of &lt;strong&gt;Pythos Oracle&lt;/strong&gt;, and I wanted to share what we learned along the way.&lt;/p&gt;

&lt;p&gt;But the interesting part isn't the final score.&lt;/p&gt;

&lt;p&gt;It's how we got there.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, a little background
&lt;/h2&gt;

&lt;p&gt;Pythos is a mathematics and physics AI system built around an idea that I believe is important:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Language models should explain mathematics, not be trusted as the mathematical authority.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;LLMs are excellent at explaining concepts, interpreting questions, and communicating solutions. But mathematical correctness requires something more dependable than a convincing explanation.&lt;/p&gt;

&lt;p&gt;That's why Pythos combines language models with deterministic mathematical engines and independent verification.&lt;/p&gt;

&lt;p&gt;We wanted a system that could solve problems, check its work, and refuse to confidently deliver an answer when it couldn't establish correctness.&lt;/p&gt;

&lt;p&gt;So we built a testing campaign to put that architecture under pressure.&lt;/p&gt;

&lt;p&gt;We call it the &lt;strong&gt;Pythos 50K Calibration Campaign&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 50,000-problem challenge
&lt;/h2&gt;

&lt;p&gt;The campaign consists of 50,000 independently generated problems distributed across ten categories.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Category&lt;/th&gt;
&lt;th&gt;Problems&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Arithmetic&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fractions / Decimals / Percentages&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Linear Equations&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Systems of Equations&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quadratics / Polynomials&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Functions / Algebra&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Geometry / Trigonometry&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Calculus&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Probability / Statistics&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Physics / Math Crossover&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;50,000&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each problem is evaluated against its expected result, with verification and delivery behavior included in the process.&lt;/p&gt;

&lt;p&gt;We weren't just interested in whether an engine could produce an answer.&lt;/p&gt;

&lt;p&gt;We wanted to know whether the answer was correct and whether the system could establish that correctness.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first run exposed problems we weren't expecting
&lt;/h2&gt;

&lt;p&gt;Our initial run completed the first 15,000 problems almost instantly.&lt;/p&gt;

&lt;p&gt;Then things slowed down dramatically.&lt;/p&gt;

&lt;p&gt;The culprit wasn't the mathematics. It was the verification infrastructure.&lt;/p&gt;

&lt;p&gt;Pythos was mistakenly interpreting intermediate Cramer's Rule determinant expressions as standalone algebraic equations.&lt;/p&gt;

&lt;p&gt;That triggered unnecessary Python subprocess creation, introducing enormous overhead.&lt;/p&gt;

&lt;p&gt;While investigating that issue, we discovered another defect: an instructional prefix parser was stripping the leading variable from expressions such as:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Solve x^2 = 25&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;The resulting malformed expression caused unnecessary fallback verification.&lt;/p&gt;

&lt;p&gt;We corrected the parsing behavior, improved determinant recognition, and strengthened the single-variable equation invariant.&lt;/p&gt;

&lt;p&gt;The next run completed all 50,000 problems in 11.63 seconds.&lt;/p&gt;

&lt;p&gt;But we weren't finished.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then we found 815 problems Pythos couldn't solve
&lt;/h2&gt;

&lt;p&gt;The initial optimized run reported:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;49,185 correctly solved.&lt;/li&gt;
&lt;li&gt;815 safely withheld.&lt;/li&gt;
&lt;li&gt;Zero incorrect answers delivered.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those 815 withheld problems were all in the physics category.&lt;/p&gt;

&lt;p&gt;Rather than accepting the result, we extracted and individually examined every withheld case.&lt;/p&gt;

&lt;p&gt;They turned out to be perfectly ordinary kinematics questions.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;p&gt;"A particle starts from rest with acceleration 19 m/s². What is its velocity after 20 seconds?"&lt;/p&gt;

&lt;p&gt;The answer is straightforward:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
v = at&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
v = 19 \times 20 = 380\text{ m/s}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;The deterministic router simply didn't recognize one particular natural-language sentence structure.&lt;/p&gt;

&lt;p&gt;Pythos did the right thing by withholding the answer rather than guessing. But it also revealed a gap in our language-to-mathematics routing.&lt;/p&gt;

&lt;p&gt;We added the missing recognition pattern and ran the entire campaign again.&lt;/p&gt;
&lt;h2&gt;
  
  
  The final results
&lt;/h2&gt;

&lt;p&gt;The final run produced the following results:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;============================================================
              PYTHOS 50K CALIBRATION
============================================================

TOTAL PROBLEMS:             50,000
CORRECTLY SOLVED:           50,000 (100.00%)
INCORRECT DELIVERIES:            0
UNKNOWN / WITHHELD:              0
VERIFICATION ESCAPES:            0
FALSE POSITIVES:                 0

RUNTIME:                    11.40 seconds
PEAK NODE HEAP:             148.9 MB
ACTIVE CHILD PROCESSES:          0

============================================================
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every category completed with 5,000 out of 5,000 correct.&lt;/p&gt;

&lt;p&gt;The entire campaign completed in approximately eleven seconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this actually demonstrates
&lt;/h2&gt;

&lt;p&gt;I'm careful about making claims with results like these.&lt;/p&gt;

&lt;p&gt;This doesn't mean Pythos can solve every mathematical problem ever created. It doesn't establish a universal 100% accuracy rate, and a deterministic benchmark isn't the same thing as a live evaluation of an LLM handling arbitrary student questions.&lt;/p&gt;

&lt;p&gt;What it does demonstrate is that, across this particular 50,000-problem corpus:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;All generated problems received the expected solutions.&lt;/li&gt;
&lt;li&gt;No incorrect answers were delivered.&lt;/li&gt;
&lt;li&gt;No problems were unnecessarily withheld in the final run.&lt;/li&gt;
&lt;li&gt;The verification pipeline completed without spawning lingering child processes.&lt;/li&gt;
&lt;li&gt;The system maintained its performance while processing all ten categories.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;More importantly, the campaign helped us discover and repair actual defects.&lt;/p&gt;

&lt;p&gt;That's the part I value most.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why we call it calibration
&lt;/h2&gt;

&lt;p&gt;We aren't interested in running a benchmark just to produce an impressive number.&lt;/p&gt;

&lt;p&gt;The purpose of the Pythos 50K Calibration Campaign is to continuously evaluate and improve the system's recognition, routing, solving, verification, and delivery behavior.&lt;/p&gt;

&lt;p&gt;The process is iterative:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Generate a controlled problem corpus.&lt;/li&gt;
&lt;li&gt;Run the blind validation.&lt;/li&gt;
&lt;li&gt;Identify failures and performance bottlenecks.&lt;/li&gt;
&lt;li&gt;Investigate the underlying causes.&lt;/li&gt;
&lt;li&gt;Improve the relevant engines.&lt;/li&gt;
&lt;li&gt;Run the complete campaign again.&lt;/li&gt;
&lt;li&gt;Preserve the results for comparison against future releases.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A successful run establishes a baseline. Future changes have to earn their place.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next?
&lt;/h2&gt;

&lt;p&gt;This is not the finish line.&lt;/p&gt;

&lt;p&gt;Our next challenge is expanding the corpus with more varied natural-language constructions, unfamiliar problem structures, unit conversions, edge cases, and deliberately misleading inputs.&lt;/p&gt;

&lt;p&gt;We also want to expand testing around initial velocity, negative acceleration, and more complicated physics scenarios.&lt;/p&gt;

&lt;p&gt;The goal is to make Pythos increasingly dependable as an educational tool.&lt;/p&gt;

&lt;p&gt;And that's ultimately why we're building it.&lt;/p&gt;

&lt;p&gt;Pythos is part of a larger effort to make mathematics and science more accessible, while helping students understand not just what the answer is, but how to get there.&lt;/p&gt;

&lt;h2&gt;
  
  
  50,000 problems down. Plenty more to go.
&lt;/h2&gt;

&lt;p&gt;If you're interested in mathematics, physics, AI, or just want to see what we've been building, come give Pythos a try.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Try Pythos:&lt;/strong&gt; &lt;a href="https://pythos.lanzar.me" rel="noopener noreferrer"&gt;https://pythos.lanzar.me&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Explore the source code:&lt;/strong&gt; &lt;a href="https://github.com/JonScott79/Pythos" rel="noopener noreferrer"&gt;https://github.com/JonScott79/Pythos&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Built under &lt;strong&gt;LANZAR | Launch Bigger.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;And the best part?&lt;/p&gt;

&lt;h1&gt;
  
  
  It's FREE!
&lt;/h1&gt;

&lt;p&gt;No subscription. No paywall to get started. Just mathematics, science, and a little curiosity.&lt;/p&gt;

&lt;h1&gt;
  
  
  AI #Mathematics #OpenSource #Education #SoftwareEngineering #ArtificialIntelligence
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>showdev</category>
      <category>webdev</category>
    </item>
    <item>
      <title>I Built a Math AI That Verifies Its Answers. Then I Made It Free.</title>
      <dc:creator>Jon Scott</dc:creator>
      <pubDate>Mon, 26 Oct 2026 02:30:53 +0000</pubDate>
      <link>https://dev.to/jonscott79/i-built-a-math-ai-that-verifies-its-answers-then-i-made-it-free-43ec</link>
      <guid>https://dev.to/jonscott79/i-built-a-math-ai-that-verifies-its-answers-then-i-made-it-free-43ec</guid>
      <description>&lt;h1&gt;
  
  
  What if you built a badass math AI, gave it a full simulation lab and graphing tools, and then made it all free?
&lt;/h1&gt;

&lt;p&gt;Well, I did.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Meet &lt;a href="https://pythos.lanzar.me" rel="noopener noreferrer"&gt;Pythos&lt;/a&gt;.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I started building Pythos because I wanted to tackle a problem with AI and mathematics: language models can explain concepts beautifully, but they can also make algebra mistakes while sounding completely confident.&lt;/p&gt;

&lt;p&gt;For students trying to learn, that matters. A convincing explanation isn't worth much if the answer is wrong.&lt;/p&gt;

&lt;p&gt;So I started building something different.&lt;/p&gt;

&lt;h2&gt;
  
  
  More than another math chatbot
&lt;/h2&gt;

&lt;p&gt;Pythos combines an AI math tutor with deterministic verification engines designed to independently check mathematical claims.&lt;/p&gt;

&lt;p&gt;The idea is straightforward: let the AI handle explanations and tutoring, while dedicated mathematical tools verify the claims they can formally prove.&lt;/p&gt;

&lt;p&gt;Depending on the problem, that can involve symbolic computation, algebraic checks, domain restrictions, and other deterministic methods.&lt;/p&gt;

&lt;p&gt;When a claim can't be verified, the system needs to distinguish that from a confirmed result. That's a core engineering challenge I'm continuing to work on.&lt;/p&gt;

&lt;h2&gt;
  
  
  The learning tools matter, too
&lt;/h2&gt;

&lt;p&gt;I didn't want to build a box that spits out answers and calls itself an educational platform.&lt;/p&gt;

&lt;p&gt;Pythos also brings together graphing tools and a simulation lab, giving students ways to explore mathematical ideas and experiment with problems instead of simply copying solutions.&lt;/p&gt;

&lt;p&gt;The bigger goal is to make it useful as a learning environment, not just an answer generator.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why make it free?
&lt;/h2&gt;

&lt;p&gt;Because students already have enough barriers to education.&lt;/p&gt;

&lt;p&gt;I want people to be able to use these tools without another subscription standing between them and learning something new.&lt;/p&gt;

&lt;p&gt;Pythos is part of my work under LANZAR. It's an ongoing project, and I'm continuing to improve the verification pipeline, tutoring behavior, and overall experience.&lt;/p&gt;

&lt;p&gt;I'm not claiming it's finished or infallible. Building a system that can reliably distinguish a correct answer from a plausible-looking mistake is an ongoing engineering problem—and one I think is worth tackling.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Pythos:&lt;/strong&gt; &lt;a href="https://pythos.lanzar.me" rel="noopener noreferrer"&gt;https://pythos.lanzar.me&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I'd genuinely like feedback from developers, math enthusiasts, students, and educators.&lt;/p&gt;

&lt;p&gt;Try a difficult problem. Test the explanations. Look for edge cases. If something breaks, I want to know about it.&lt;/p&gt;

&lt;p&gt;I'm especially interested in feedback on deterministic verification, how AI tutoring should work, and what features would make this more useful in a real classroom.&lt;/p&gt;

&lt;p&gt;I built it because I thought math education tools could be better.&lt;/p&gt;

&lt;p&gt;Now I want to see what people do with it.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>math</category>
      <category>opensource</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Pythos Oracle Is Live: Building AI Math and Physics Tutoring Around Deterministic Verification</title>
      <dc:creator>Jon Scott</dc:creator>
      <pubDate>Sat, 10 Oct 2026 16:56:45 +0000</pubDate>
      <link>https://dev.to/jonscott79/pythos-oracle-is-live-building-ai-math-and-physics-tutoring-around-deterministic-verification-4pha</link>
      <guid>https://dev.to/jonscott79/pythos-oracle-is-live-building-ai-math-and-physics-tutoring-around-deterministic-verification-4pha</guid>
      <description>&lt;h1&gt;
  
  
  Pythos Oracle Is Live: Building AI Math and Physics Tutoring Around Deterministic Verification
&lt;/h1&gt;

&lt;p&gt;I've been building something I want to share with the developer community.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pythos Oracle is live: &lt;a href="https://pythos.lanzar.me/" rel="noopener noreferrer"&gt;pythos.lanzar.me&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The idea behind Pythos comes from a problem I've been thinking about for a while: large language models can produce convincing mathematical explanations, but a convincing explanation doesn't necessarily mean the answer is correct.&lt;/p&gt;

&lt;p&gt;What happens when we stop treating an AI-generated answer as the final authority and start building systems that independently verify the mathematics?&lt;/p&gt;

&lt;p&gt;That's the engineering problem I'm exploring with Pythos.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Core Idea
&lt;/h2&gt;

&lt;p&gt;Rather than relying exclusively on a language model to generate and validate its own mathematical output, Pythos uses a verification-oriented approach that separates language-model capabilities from deterministic mathematical checks.&lt;/p&gt;

&lt;p&gt;The language model provides the conversational interface and helps explain concepts. Deterministic tools can independently evaluate supported calculations and mathematical claims.&lt;/p&gt;

&lt;p&gt;The distinction matters:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Generating an answer and establishing that it's correct are two different operations.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I've written more about the thinking behind this approach in my earlier article:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://dev.to/"&gt;Why We Stopped Letting LLMs Do Raw Math: Building Pythos With Deterministic Verification&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Build This?
&lt;/h2&gt;

&lt;p&gt;I believe we're in the middle of a technological revolution, but public discussion about AI is often stuck between two extremes: treating it as magic or treating it as an existential threat.&lt;/p&gt;

&lt;p&gt;Neither approach helps us build better systems.&lt;/p&gt;

&lt;p&gt;AI has real capabilities, real limitations, and real infrastructure costs. We should understand those limitations and engineer around them wherever possible.&lt;/p&gt;

&lt;p&gt;For mathematics and physics education, reliability matters. Students need more than a confident answer; they need useful explanations and ways to check the underlying work.&lt;/p&gt;

&lt;p&gt;That's the direction I'm pursuing with Pythos Oracle: making STEM learning more accessible while exploring how deterministic verification can improve the reliability of AI-assisted problem solving.&lt;/p&gt;

&lt;h2&gt;
  
  
  This Is an Ongoing Engineering Project
&lt;/h2&gt;

&lt;p&gt;I'm not claiming to have eliminated AI hallucinations or solved mathematical verification for every possible problem.&lt;/p&gt;

&lt;p&gt;Verification has a defined scope, and the system needs continued testing against difficult problems, edge cases, and unsupported inputs.&lt;/p&gt;

&lt;p&gt;That's part of the work.&lt;/p&gt;

&lt;p&gt;I'd rather identify the limits, test them, and improve the implementation than make claims the evidence can't support.&lt;/p&gt;

&lt;h2&gt;
  
  
  I'd Like Your Feedback
&lt;/h2&gt;

&lt;p&gt;If you're interested in AI engineering, mathematics, physics, or educational technology, take a look:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live project: &lt;a href="https://pythos.lanzar.me/" rel="noopener noreferrer"&gt;Pythos Oracle&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I'm particularly interested in hearing from developers who have experimented with deterministic verification, symbolic mathematics, or hybrid AI architectures.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What kinds of mathematical problems would you use to stress-test a system like this?&lt;/li&gt;
&lt;li&gt;Where do you think deterministic verification provides the most value?&lt;/li&gt;
&lt;li&gt;Where does it become difficult to apply?&lt;/li&gt;
&lt;li&gt;What would you want to see in an AI-powered STEM learning tool?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I'm building Pythos to be useful, not just impressive in a demo.&lt;/p&gt;

&lt;p&gt;Honest technical feedback is welcome.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>opensource</category>
      <category>showdev</category>
    </item>
    <item>
      <title>From Dev.to Comment to Production in 24h: Building an Inspectable Math Verification Contract in Pythos (and Fixing the Pearson Trap)</title>
      <dc:creator>Jon Scott</dc:creator>
      <pubDate>Wed, 07 Oct 2026 03:24:55 +0000</pubDate>
      <link>https://dev.to/jonscott79/from-devto-comment-to-production-in-24h-building-an-inspectable-math-verification-contract-in-2lma</link>
      <guid>https://dev.to/jonscott79/from-devto-comment-to-production-in-24h-building-an-inspectable-math-verification-contract-in-2lma</guid>
      <description>&lt;p&gt;Earlier this week, I published an article here on DEV: &lt;a href="https://dev.to/jonscott79"&gt;&lt;strong&gt;"Why We Stopped Letting LLMs Do Raw Math: Building Pythos With Deterministic Verification"&lt;/strong&gt;&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;The response was humbling, but one particular comment from an enterprise AI engineer at &lt;strong&gt;IT Path Solutions&lt;/strong&gt; stood out. They validated our core thesis—&lt;em&gt;treat the LLM as a conversational interface, never the mathematical source of truth&lt;/em&gt;—and left a game-changing suggestion:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;"Make the verification result part of the data contract... each step could carry the expression evaluated, the verification method, the assumptions used, and the exact state that was checked."&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;They were 100% right. &lt;/p&gt;

&lt;p&gt;In Pythos, our dual-engine verifier (in-process &lt;strong&gt;Math.js&lt;/strong&gt; for exact arithmetic and a sandboxed &lt;strong&gt;SymPy CAS&lt;/strong&gt; daemon for symbolic algebra/calculus) was already validating derivations step-by-step behind the scenes. But once the delivery decision gate passed, we were condensing all that rich telemetry into simple binary flags before shipping the text to the client.&lt;/p&gt;

&lt;p&gt;We decided to build what they suggested. &lt;/p&gt;

&lt;p&gt;Here is how we turned mathematical verification into an explicit, wire-level data contract, built a client-side derivation inspector, and—in a hilarious twist of real-world battle testing—fixed a production bug where Pearson homework boilerplate tricked our deterministic router into computing the area of a right triangle.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The Architecture: The &lt;code&gt;VerificationStepAudit&lt;/code&gt; Data Contract
&lt;/h2&gt;

&lt;p&gt;Prior to v1.8.35, Pythos knew &lt;em&gt;internally&lt;/em&gt; whether a claim was true or false, but the frontend only received markdown text with basic metadata.&lt;/p&gt;

&lt;p&gt;We formalized a new data contract: &lt;strong&gt;&lt;code&gt;VerificationStepAudit&lt;/code&gt;&lt;/strong&gt;. Every evaluated claim emitted over our streaming NDJSON and JSON endpoints now carries structured audit telemetry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kr"&gt;interface&lt;/span&gt; &lt;span class="nx"&gt;VerificationStepAudit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;stepIndex&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;expressionEvaluated&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;verificationMethod&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;SYMPY_CAS&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;MATHJS_EXACT&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;DETERMINISTIC_FASTPATH&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;engine&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;sympy&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;mathjs&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;native&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;assumptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;real&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;complex&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;integer&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;tolerance&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;quadrant&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="nl"&gt;checkedState&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;proposed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;discrepancy&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="nl"&gt;certified&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;rejectionRationale&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Why This Matters for AI Tutoring
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Zero Black-Box Math:&lt;/strong&gt; The student (and educator) isn't just told "Trust me, the answer is $x = \frac{1}{2}$." They can inspect the actual symbolic reduction that certified the step.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Targeted Socratic Revision:&lt;/strong&gt; When an LLM generates a multi-step derivation and makes a sign error on Step 4, we don't ask it to "try again" blindly. The verifier catches the exact step failure and feeds the structured &lt;code&gt;VerificationStepAudit&lt;/code&gt; back into the Socratic revision prompt:
&amp;gt; &lt;em&gt;"Step 4 failed: evaluated $\sin^2\theta + \cos^2\theta = 2$, expected $1$. Discrepancy: Pythagorean identity violation."&lt;/em&gt;
The model corrects itself with mathematical precision.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  2. The UX: The Derivation Inspector
&lt;/h2&gt;

&lt;p&gt;Exposing verification metadata without cluttering the learning experience is tricky. If you dump raw JSON or giant debug banners on students, it overwhelms them.&lt;/p&gt;

&lt;p&gt;Instead, we added a subtle, accessible badge on the bottom-right action bar of assistant messages, sitting cleanly alongside the &lt;strong&gt;Report&lt;/strong&gt; and &lt;strong&gt;Copy&lt;/strong&gt; buttons:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[ 🛡️ Verified (3 steps) ]  [ 📋 Copy ]  [ 🐞 Report ]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Clicking &lt;strong&gt;Verified&lt;/strong&gt; expands a slide-out Derivation Inspector panel directly underneath the response:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Highlights each individual derivation step.&lt;/li&gt;
&lt;li&gt;Displays the exact mathematical expression tested.&lt;/li&gt;
&lt;li&gt;Shows the certified CAS engine badge (&lt;code&gt;SymPy CAS&lt;/code&gt; or &lt;code&gt;Math.js&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Confirms the active assumptions (e.g., $x &amp;gt; 0$, real domain).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now, students aren't just reading AI generated text; they can audit the mathematical proof behind it in real time.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. The Multimodal Polish: Magic Bytes &amp;amp; Vision History
&lt;/h2&gt;

&lt;p&gt;Alongside the audit contract, we upgraded our multimodal screenshot ingestion pipeline. Students frequently screenshot math worksheets and paste them directly into Pythos.&lt;/p&gt;

&lt;p&gt;We ran into two edge cases with upstream vision models (Llama 3.2 Vision on Groq):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;MIME-Type Rejections (400 Bad Request):&lt;/strong&gt; Some browsers generate clipboard image blobs with generic or missing MIME headers. We implemented client-side magic-byte sniffing (inspecting PNG &lt;code&gt;\x89PNG&lt;/code&gt;, JPEG &lt;code&gt;\xFF\xD8\xFF&lt;/code&gt;, and WebP headers) to guarantee strict &lt;code&gt;Content-Type&lt;/code&gt; tagging.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vision TPM Throttling (429 Rate Limit):&lt;/strong&gt; In a multi-turn conversation, re-transmitting multi-megabyte base64 images on every turn quickly incinerates Token-Per-Minute (TPM) limits on free-tier inference gateways. We bounded conversational image history so previous turns retain verified OCR transcripts while releasing redundant base64 payloads once grounded.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  4. The "Pearson Trap": When Homework Boilerplate Met Deterministic Routing
&lt;/h2&gt;

&lt;p&gt;No deployment story is complete without a hilarious production edge case. &lt;/p&gt;

&lt;p&gt;Shortly after releasing v1.8.35, I was using Pythos myself to work through college trigonometry homework from Pearson MyLab. I pasted a problem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Use a right triangle to write the following expression as an algebraic expression. 
Assume that x is positive and that the given inverse trigonometric function is defined for the expression in x.
tangent left parenthesis cosine Superscript negative 1 Baseline 4 x right parenthesis
Question content area bottom
Part 1
tangent left parenthesis cosine Superscript negative 1 Baseline 4 x right parenthesisequals
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I expected Pythos to derive:&lt;br&gt;
$$\tan(\cos^{-1}(4x)) = \frac{\sqrt{1 - 16x^2}}{4x}$$&lt;/p&gt;

&lt;p&gt;Instead, Pythos instantly and cheerfully responded:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;📐 &lt;strong&gt;Right-Triangle Area&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
Here is the right triangle with base 3 and height 4:&lt;br&gt;&lt;br&gt;
$\text{Area} = \frac{1}{2} \times \text{base} \times \text{height} = \frac{1}{2}(3)(4) = 6$&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Wait... what? Where did a 3-4-5 triangle and an area of 6 come from?!&lt;/p&gt;
&lt;h3&gt;
  
  
  The Root Cause
&lt;/h3&gt;

&lt;p&gt;To keep latency near zero and save GPU compute costs, Pythos features an in-process &lt;strong&gt;Deterministic Router&lt;/strong&gt; that short-circuits simple calculations (arithmetic, unit conversions, geometry requests) before invoking the LLM.&lt;/p&gt;

&lt;p&gt;Two bugs in the router had collided:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Screen-Reader Web Boilerplate:&lt;/strong&gt; Pearson MyLab outputs accessibility tags including the phrase:
&amp;gt; &lt;em&gt;"Question content **area&lt;/em&gt;* bottom"*&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overly Eager Regex:&lt;/strong&gt; The router's geometry parser checked for right triangle queries, followed by:
&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;isArea&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b&lt;/span&gt;&lt;span class="sr"&gt;area&lt;/span&gt;&lt;span class="se"&gt;\b&lt;/span&gt;&lt;span class="sr"&gt;/i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;It matched the word &lt;strong&gt;"area"&lt;/strong&gt; in &lt;em&gt;"Question content area bottom"&lt;/em&gt;!&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The Dummy Fallback:&lt;/strong&gt; Because the prompt asked to &lt;em&gt;"Use a right triangle..."&lt;/em&gt; to solve an algebraic expression, no numerical leg lengths were given. The legacy geometry routine fell back to a default &lt;code&gt;3, 4, 5&lt;/code&gt; triangle and happily computed $\frac{1}{2} \times 3 \times 4 = 6$.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3&gt;
  
  
  The Hardened Fix
&lt;/h3&gt;

&lt;p&gt;We patched &lt;code&gt;server/deterministicRouter.js&lt;/code&gt; immediately:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Added an &lt;code&gt;isAlgebraicOrInverseTrig&lt;/code&gt; guard that completely blocks prompts containing algebraic keywords (&lt;code&gt;algebraic expression&lt;/code&gt;, &lt;code&gt;simplify&lt;/code&gt;, &lt;code&gt;expand&lt;/code&gt;) or inverse trig operators (&lt;code&gt;cos^-1&lt;/code&gt;, &lt;code&gt;arcsin&lt;/code&gt;, &lt;code&gt;StartFraction&lt;/code&gt;, &lt;code&gt;Superscript negative 1&lt;/code&gt;) from ever being hijacked as static geometry drawings.&lt;/li&gt;
&lt;li&gt;Sanitized web platform boilerplate (&lt;code&gt;question content area&lt;/code&gt;, &lt;code&gt;content area bottom&lt;/code&gt;) before testing for geometric intents.&lt;/li&gt;
&lt;li&gt;Required geometric area calculations to have explicit numerical side lengths and dedicated question phrasing (e.g., &lt;em&gt;"find the area"&lt;/em&gt;).&lt;/li&gt;
&lt;/ol&gt;


&lt;h2&gt;
  
  
  5. Adding It to the Master Regression Suite
&lt;/h2&gt;

&lt;p&gt;A bug once discovered must never happen again. &lt;/p&gt;

&lt;p&gt;We created &lt;strong&gt;Group G&lt;/strong&gt; in our fast-path hardening suite (&lt;code&gt;test-deterministic-fastpath-hardening.js&lt;/code&gt;) with tests specifically checking Pearson screen-reader transcripts, algebraic inverse trig phrases, and fractional boilerplate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;runTest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;G1: Pearson screen-reader inverse trig prompt must not trigger dummy triangle area&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Use a right triangle to write... tangent left parenthesis cosine Superscript negative 1 Baseline 4 x right parenthesis Question content area bottom...&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;intent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;deterministicRouter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;analyzeDeterministicIntent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[]);&lt;/span&gt;
  &lt;span class="nx"&gt;assert&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strictEqual&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;intent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// Must pass through to LLM + CAS verification!&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We then promoted this into &lt;strong&gt;Suite 25&lt;/strong&gt; of our master regression test harness (&lt;code&gt;test-regression-harness.js&lt;/code&gt;). &lt;/p&gt;

&lt;p&gt;All 25 regression suites—covering SymPy CAS, Math.js adversarial arithmetic, student privacy redaction, streaming verification, and fast-path hardening—are passing 100% green.&lt;/p&gt;




&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;In less than 24 hours:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A great comment from the DEV community gave us the exact conceptual blueprint we needed.&lt;/li&gt;
&lt;li&gt;We turned verification into a first-class, inspectable &lt;code&gt;VerificationStepAudit&lt;/code&gt; data contract.&lt;/li&gt;
&lt;li&gt;We added an accessible client-side UI inspector to prove mathematical derivations to students.&lt;/li&gt;
&lt;li&gt;We caught and eliminated a wild edge-case where Pearson homework boilerplate masqueraded as a geometry problem.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Huge thanks to the DEV community and our commenter from IT Path Solutions for pushing us to make Pythos better. &lt;/p&gt;

&lt;p&gt;You can try out Pythos for free anytime at &lt;a href="https://pythos.lanzar.me" rel="noopener noreferrer"&gt;&lt;strong&gt;pythos.lanzar.me&lt;/strong&gt;&lt;/a&gt; or explore the code and verification engine on &lt;a href="https://github.com/JonScott79/Pythos" rel="noopener noreferrer"&gt;&lt;strong&gt;GitHub&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;What are your thoughts on exposing deterministic verifier contracts directly in AI chat interfaces? Would you find step-level mathematical receipts helpful when studying or building math tools? Let's discuss in the comments!&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>architecture</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Why We Stopped Letting LLMs Do Raw Math: Building Pythos With Deterministic Verification</title>
      <dc:creator>Jon Scott</dc:creator>
      <pubDate>Tue, 06 Oct 2026 03:26:17 +0000</pubDate>
      <link>https://dev.to/jonscott79/why-we-stopped-letting-llms-do-raw-math-building-pythos-with-deterministic-verification-18ep</link>
      <guid>https://dev.to/jonscott79/why-we-stopped-letting-llms-do-raw-math-building-pythos-with-deterministic-verification-18ep</guid>
      <description>&lt;p&gt;Large Language Models are incredible at conceptual analogies, Socratic dialogue, and breaking down complex ideas. &lt;/p&gt;

&lt;p&gt;But when it comes to raw mathematics and physics derivations, &lt;strong&gt;they are notoriously unreliable calculators&lt;/strong&gt;. &lt;/p&gt;

&lt;p&gt;They drop negative signs, fabricate intermediate arithmetic steps, and deliver confidently incorrect answers with total poise. In creative writing, a hallucination is a quirk; in mathematics and physics education, &lt;strong&gt;it completely derails a student's confidence and understanding.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;To solve this, I built &lt;strong&gt;&lt;a href="https://pythos.lanzar.me/" rel="noopener noreferrer"&gt;Pythos&lt;/a&gt;&lt;/strong&gt;—a free, open-access AI math and physics tutor engineered with a fundamentally different philosophy: &lt;strong&gt;never let the language model deliver unchecked math.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The Problem: The Confidence Trap in STEM Chatbots
&lt;/h2&gt;

&lt;p&gt;Most AI tutoring tools take a student’s math problem, pass it to an LLM, and stream the generated response directly to the screen. &lt;/p&gt;

&lt;p&gt;When the model makes an algebraic error in Step 3 of a 6-step calculus integration, the final result is wrong. If the student questions it, the model often apologizes, scrambles its numbers, and hallucinates a second, equally flawed derivation.&lt;/p&gt;

&lt;p&gt;Instead of maximizing model "confidence," we asked a different question: &lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;What if an AI tutor was designed with a strict deterministic verification gate that checks derivations before outputting them—and withholds answers if it cannot prove them?&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  The Architecture: The Deterministic Verification Gate
&lt;/h2&gt;

&lt;p&gt;In &lt;a href="https://pythos.lanzar.me/" rel="noopener noreferrer"&gt;Pythos&lt;/a&gt;, the AI does not operate in a vacuum. Instead, we split the reasoning process into pedagogical explanation and deterministic execution:&lt;/p&gt;

&lt;p&gt;In &lt;a href="https://pythos.lanzar.me/" rel="noopener noreferrer"&gt;Pythos&lt;/a&gt;, the AI does not operate in a vacuum. Instead, we split the reasoning process into pedagogical explanation and deterministic execution:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Student Input&lt;/strong&gt; ➔ Sent to LLM for pedagogical reasoning&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LLM Formulates Steps&lt;/strong&gt; ➔ Generates explanation &amp;amp; step derivations&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deterministic CAS Gate&lt;/strong&gt; ➔ Math engine validates each algebraic step&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Automated Audit&lt;/strong&gt; ➔ Checks signs, values, and consistency&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verification Gate&lt;/strong&gt; ➔ If verified, delivers to student; if unverified, withholds and reroutes!&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Socratic Pedagogical Layer:&lt;/strong&gt; The language model breaks the problem into guiding steps and conceptual intuition rather than just spitting out a single answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deterministic Calculation Safeguard:&lt;/strong&gt; Mathematical claims, factorizations, limits, and algebraic operations are verified through deterministic calculation kernels.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The "Refusal" Safeguard:&lt;/strong&gt; If a mathematical step fails verification or cannot be guaranteed with high confidence, Pythos is programmed to &lt;strong&gt;withhold the response&lt;/strong&gt; rather than guessing. &lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Proving It: 110,000+ Problem Validation Record
&lt;/h2&gt;

&lt;p&gt;To measure reliability, we ran multi-tier benchmark campaigns testing tens of thousands of real exam and homework problems across algebra, calculus, and classical mechanics.&lt;/p&gt;

&lt;p&gt;You can inspect our public mathematical validation record directly here:&lt;br&gt;&lt;br&gt;
👉 &lt;strong&gt;&lt;a href="https://pythos.lanzar.me/validation/" rel="noopener noreferrer"&gt;Pythos Validation Benchmark&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;By pairing language model reasoning with strict algorithmic verification, the pipeline eliminates mathematical hallucinations on verified paths, achieving a 0% incorrect delivery rate across our validation sets by safely withholding unverified steps.&lt;/p&gt;




&lt;h2&gt;
  
  
  Interactive Physics &amp;amp; Math Instruments
&lt;/h2&gt;

&lt;p&gt;Math and physics shouldn't just be static text. We also built real-time, interactive visual instruments right into the tutoring canvas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;2D Projectile Motion Simulator:&lt;/strong&gt; Interactive PhET-style ballistics canvas allowing students to adjust launch angle, velocity, and gravity sliders with live trajectory arcs and calculated range readouts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Right Triangle &amp;amp; Trig Inspector:&lt;/strong&gt; Live geometric recalculations showing exact trigonometric ratios and step-by-step Pythagorean derivations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Function Grapher:&lt;/strong&gt; Real-time 2D plotting engine constrained for fast, touch-accessible mobile exploration.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  100% Free for Students and Educators
&lt;/h2&gt;

&lt;p&gt;Pythos is non-profit, ad-free, and requires no paywalls or subscriptions. It was built by a student for students.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🌐 &lt;strong&gt;Try Pythos Live:&lt;/strong&gt; &lt;a href="https://pythos.lanzar.me/" rel="noopener noreferrer"&gt;pythos.lanzar.me&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;📚 &lt;strong&gt;Study Guides:&lt;/strong&gt; &lt;a href="https://pythos.lanzar.me/algebra/" rel="noopener noreferrer"&gt;Algebra&lt;/a&gt; | &lt;a href="https://pythos.lanzar.me/calculus/" rel="noopener noreferrer"&gt;Calculus&lt;/a&gt; | &lt;a href="https://pythos.lanzar.me/physics/" rel="noopener noreferrer"&gt;Physics&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;💻 &lt;strong&gt;Open Source Repository:&lt;/strong&gt; &lt;a href="https://github.com/JonScott79/Pythos" rel="noopener noreferrer"&gt;github.com/JonScott79/Pythos&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I would love to hear feedback from other developers and educators: How are you handling hallucination boundaries and deterministic validation in your AI applications?&lt;/p&gt;

</description>
      <category>showdev</category>
      <category>ai</category>
      <category>webdev</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
