<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: JonathanSolvesProblems</title>
    <description>The latest articles on DEV Community by JonathanSolvesProblems (@jonathansolvesstuff).</description>
    <link>https://dev.to/jonathansolvesstuff</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4025240%2F8b439bff-c8c7-4012-b315-f48e6d2c1d60.png</url>
      <title>DEV Community: JonathanSolvesProblems</title>
      <link>https://dev.to/jonathansolvesstuff</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jonathansolvesstuff"/>
    <language>en</language>
    <item>
      <title>A maintainer, me, and Sentry's own AI all reached for the same wrong fix</title>
      <dc:creator>JonathanSolvesProblems</dc:creator>
      <pubDate>Wed, 19 Aug 2026 23:35:04 +0000</pubDate>
      <link>https://dev.to/jonathansolvesstuff/a-maintainer-me-and-sentrys-own-ai-all-reached-for-the-same-wrong-fix-4ef4</link>
      <guid>https://dev.to/jonathansolvesstuff/a-maintainer-me-and-sentrys-own-ai-all-reached-for-the-same-wrong-fix-4ef4</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Smash Stories&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Three people tried to fix the same memory leak in Sentry's Python SDK. One was a Sentry maintainer. One was me. One was Sentry's own AI debugger.&lt;/p&gt;

&lt;p&gt;All three fixes were wrong, and they were wrong for the same underlying reason. That convergence is the actual story, and it took me a while to see it, because I was too busy being pleased with my own version.&lt;/p&gt;

&lt;h2&gt;
  
  
  The leak, briefly
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;DedupeIntegration&lt;/code&gt; remembers the last exception it saw so it can drop a duplicate report. It stores a weak reference, because holding the exception would hold its traceback, and a traceback holds every frame, and a frame holds every local variable in it.&lt;/p&gt;

&lt;p&gt;Then there is this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# we can only weakref non builtin types
&lt;/span&gt;&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;integration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weakref&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TypeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;integration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You cannot weak-reference a &lt;code&gt;ValueError&lt;/code&gt;. So the fallback runs, and it does the exact thing the weak reference existed to prevent. It lives in a &lt;code&gt;ContextVar&lt;/code&gt;, so under asyncio every task keeps its own copy: one retained exception, one traceback, one full set of frame locals, per live task.&lt;/p&gt;

&lt;p&gt;Measured across 200 long-lived sessions holding a megabyte each: 205 MB retained, all 200 still reachable after &lt;code&gt;gc.collect()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That part I have written up in full &lt;a href="https://jonathanandrei.com/blog/sentry-python-dedupe-memory-leak-weakref-fix/" rel="noopener noreferrer"&gt;elsewhere&lt;/a&gt;. This post is about the three fixes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Attempt one: the maintainer
&lt;/h2&gt;

&lt;p&gt;Digging through the fork's branch list, I found two abandoned branches from September 2025, neither merged. The first replaces the stored exception with a SHA-256 fingerprint over &lt;code&gt;(type_module, type_name, id(exc_value))&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It is a reasonable instinct. Store a cheap value, not the object. The problem is &lt;code&gt;id()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Not retaining the exception is precisely what frees its address, and CPython reuses freed addresses immediately. Over 2000 distinct, sequentially allocated &lt;code&gt;ValueError&lt;/code&gt;s:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;distinct exceptions created : 2000
distinct fingerprints       : 2
address-reuse collisions    : 1998
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two fingerprints for two thousand distinct errors. Every collision is a real error that gets silently dropped as a duplicate. The fix trades a memory leak for losing almost every error you report, which is worse than the leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  Attempt two: me
&lt;/h2&gt;

&lt;p&gt;I did not see that branch until much later, which is lucky, because I made the same class of mistake independently.&lt;/p&gt;

&lt;p&gt;My idea was a value fingerprint with no &lt;code&gt;id()&lt;/code&gt; in it: exception type, message, and the origin frame from the traceback. All immutable primitives. Nothing retained. I was confident enough to post it publicly on the issue.&lt;/p&gt;

&lt;p&gt;The SDK's own test suite killed it in two different ways within about a minute of running.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;test_breadcrumbs&lt;/code&gt; captures two distinct, never-raised &lt;code&gt;ValueError()&lt;/code&gt; instances. No traceback, no message, so both fingerprints are identical and the second event is dropped.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;test_option_before_breadcrumb&lt;/code&gt; is worse. It calls the same function three times, each raising a separate &lt;code&gt;ValueError("aha!")&lt;/code&gt; from the same line. Same type, same message, same origin frame. Three identical fingerprints, two events wrongly deduplicated.&lt;/p&gt;

&lt;p&gt;The flaw is not in my choice of fields. It is structural:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A value fingerprint cannot distinguish &lt;strong&gt;"the same exception object captured twice"&lt;/strong&gt; from &lt;strong&gt;"the same error raised twice from the same line."&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The first must deduplicate. The second must not. By value, they are identical. No amount of extra fields fixes that, because the thing being asked for is not a property of the value.&lt;/p&gt;

&lt;p&gt;I went back to the issue and said I had been wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Attempt three: the AI
&lt;/h2&gt;

&lt;p&gt;Later I pointed Sentry's own AI debugger, Seer, at the issue in my own Sentry project.&lt;/p&gt;

&lt;p&gt;Its diagnosis was genuinely excellent, and better than I expected. It reached past my application code into a third-party SDK, identified the &lt;code&gt;ContextVar&lt;/code&gt; strong reference, identified that &lt;code&gt;weakref.ref&lt;/code&gt; raises &lt;code&gt;TypeError&lt;/code&gt; on builtin exception types, and identified that ContextVars are per-task under asyncio. That last detail had cost me a full debugging round to work out on my own. It cited &lt;code&gt;dedupe.py&lt;/code&gt; lines as evidence, so it had gone and read the source rather than guessing from a stack trace.&lt;/p&gt;

&lt;p&gt;Then it proposed the fix:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;store a hashable identity tuple like &lt;code&gt;(type(exc), id(exc))&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;code&gt;id()&lt;/code&gt; again. I ran its exact tuple:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;distinct errors raised         : 2000
events wrongly dropped as dupe : 1999  (100.0%)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three independent parties. Two humans and a model. Two different wrong answers that are secretly the same wrong answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The thing all three of us did
&lt;/h2&gt;

&lt;p&gt;Every one of us tried to represent &lt;strong&gt;identity&lt;/strong&gt; with a &lt;strong&gt;value&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;An object's identity is not a property you can read off it. It is the fact of the object existing, distinct from every other object, for as long as it lives. A fingerprint made of type and message describes what the exception &lt;em&gt;is like&lt;/em&gt;. An address describes where it &lt;em&gt;currently sits&lt;/em&gt;. Neither survives the thing that makes identity useful, which is that two objects that look the same are still two objects.&lt;/p&gt;

&lt;p&gt;Once I could say that sentence, the fix was obvious, and it is not a fingerprint at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;_DedupeToken&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;__slots__&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__weakref__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Attach one token to the exception and weakly reference the token. There is exactly one per exception object and it lives exactly as long as the exception does, so comparing tokens is comparing identity. The &lt;code&gt;ContextVar&lt;/code&gt; holds nothing that keeps a traceback alive, and deduplication behaviour does not change at all.&lt;/p&gt;

&lt;p&gt;Same 200 sessions, after the fix: 0.7 MB, nothing pinned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I did not expect
&lt;/h2&gt;

&lt;p&gt;There is a coda that I only found while going back through a screen recording, and it is the most interesting thing I learned all week.&lt;/p&gt;

&lt;p&gt;Seer did not only propose a plan. It also opened a pull request with generated code. &lt;strong&gt;The code is not what the plan said.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;integration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weakref&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TypeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No identity tuple anywhere. It simply stops storing for builtin exceptions. That does remove the leak, and it has none of the address-reuse problem, but it silently disables deduplication for the exception types most errors actually are. It is also, precisely, the fix the maintainers had already declined on the issue months earlier.&lt;/p&gt;

&lt;p&gt;So the plan and the patch disagree with each other, and they are wrong in two different ways.&lt;/p&gt;

&lt;p&gt;I want to be fair here, because the diagnosis was the hard part and it got that right, from evidence, without my framing. But if you let an agent open pull requests: &lt;strong&gt;read the diff, not the summary.&lt;/strong&gt; They are not required to match, and here they did not.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;The comment &lt;code&gt;# we can only weakref non builtin types&lt;/code&gt; sat directly above the line that caused the leak, for years. It described the hazard accurately and then the next line walked into it. A fallback is still a code path, and this one ran for the overwhelming majority of real exceptions.&lt;/p&gt;

&lt;p&gt;And the more useful lesson: when several competent people independently produce the same wrong answer, that is information. It usually means the obvious framing of the problem is the thing that is wrong, not the people. Three of us reached for a fingerprint because "store something small instead of the object" is a good habit. The habit was fine. The question was wrong.&lt;/p&gt;

&lt;p&gt;Everything here is reproducible, including all three failed fixes, at &lt;a href="https://github.com/JonathanSolvesProblems/sentry-dedupe-leak-repro" rel="noopener noreferrer"&gt;JonathanSolvesProblems/sentry-dedupe-leak-repro&lt;/a&gt;. &lt;code&gt;id_reuse_hazard.py&lt;/code&gt; runs the maintainer's approach and Seer's, side by side, and prints exactly how many errors each one would lose.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>python</category>
      <category>ai</category>
    </item>
    <item>
      <title>Python can't weakref a ValueError. Sentry's deduplication found out the expensive way.</title>
      <dc:creator>JonathanSolvesProblems</dc:creator>
      <pubDate>Wed, 19 Aug 2026 23:14:14 +0000</pubDate>
      <link>https://dev.to/jonathansolvesstuff/python-cant-weakref-a-valueerror-sentrys-deduplication-found-out-the-expensive-way-4c5i</link>
      <guid>https://dev.to/jonathansolvesstuff/python-cant-weakref-a-valueerror-sentrys-deduplication-found-out-the-expensive-way-4c5i</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Clear the Lineup&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/2B5g7c9AwB0"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Two and a half minutes: the leak, the two fixes that do not work, and what Sentry's Seer made of it. Everything below is the written version, with the parts the video did not have room for.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Project Overview
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/getsentry/sentry-python" rel="noopener noreferrer"&gt;sentry-python&lt;/a&gt; is Sentry's Python SDK. One of its default integrations is &lt;code&gt;DedupeIntegration&lt;/code&gt;, which stops the same error being reported twice. The mechanism is about as simple as it gets: remember the last exception you saw, and if the next one is the same object, drop the event.&lt;/p&gt;

&lt;p&gt;Remembering an exception is where it gets interesting, because remembering it is exactly what you must not do. An exception holds its traceback, a traceback holds its frames, and a frame holds every local variable in it. Keep the exception and you keep all of that.&lt;/p&gt;

&lt;p&gt;The SDK knew this. That is why it stored a weak reference:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# we can only weakref non builtin types
&lt;/span&gt;&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;integration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weakref&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TypeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;integration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The comment tells you the whole story. You cannot take a weak reference to a builtin exception. &lt;code&gt;ValueError&lt;/code&gt;, &lt;code&gt;KeyError&lt;/code&gt;, &lt;code&gt;TypeError&lt;/code&gt;, the ones almost every error actually is. So &lt;code&gt;weakref.ref(exc)&lt;/code&gt; raises &lt;code&gt;TypeError&lt;/code&gt;, and the fallback quietly does the one thing the weak reference existed to prevent.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/getsentry/sentry-python/issues/6094" rel="noopener noreferrer"&gt;Issue #6094&lt;/a&gt; reported it as unexplained memory growth in an asyncio web crawler.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug Fix or Performance Improvement
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why asyncio makes it hurt
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;_last_seen&lt;/code&gt; is a &lt;code&gt;ContextVar&lt;/code&gt;. Under asyncio, every task gets its own copy of context. So this is not one retained exception process-wide. It is &lt;strong&gt;one retained exception per live task&lt;/strong&gt;, each one dragging along its traceback and every frame local reachable from it.&lt;/p&gt;

&lt;p&gt;The reporter's frames held fetched response bodies between 500 KB and 1 MB.&lt;/p&gt;

&lt;h3&gt;
  
  
  Measuring it honestly
&lt;/h3&gt;

&lt;p&gt;My first instinct was to call it an unbounded leak. I built a worker pool, ran it, and it stayed flat. That was worth finding out before I wrote it down anywhere.&lt;/p&gt;

&lt;p&gt;A fixed pool does not grow, because each worker's next error overwrites its previous one. Retention is bounded at (live tasks x payload). The growth story is not errors over time, it is &lt;strong&gt;tasks over time&lt;/strong&gt;: one task per session, per subscription, per connection.&lt;/p&gt;

&lt;p&gt;So I measured against live task count instead. Each task fails once, reports it, then stays alive the way a real session handler does:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;  live tasks |     retained |   per task |  sessions pinned
-------------+--------------+------------+-----------------
          25 |      26.2 MB |    1047 KB |      25 / 25
          50 |      52.1 MB |    1042 KB |      50 / 50
         100 |     103.3 MB |    1033 KB |     100 / 100
         200 |     205.4 MB |    1027 KB |     200 / 200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;1024 KB retained per live task, dead straight, and &lt;code&gt;sessions pinned&lt;/code&gt; counts weak references to session objects that are still reachable after &lt;code&gt;gc.collect()&lt;/code&gt;. Not a sampling artifact. The garbage collector cannot touch them because a live &lt;code&gt;ContextVar&lt;/code&gt; genuinely still points at them.&lt;/p&gt;

&lt;p&gt;At 200 concurrent sessions that is 205 MB that never comes back.&lt;/p&gt;

&lt;h3&gt;
  
  
  The fix I got wrong first
&lt;/h3&gt;

&lt;p&gt;The maintainers had already rejected the obvious fix. The reporter proposed skipping dedupe for builtins, and Sentry declined, saying they wanted "a more robust fingerprinting approach that can also be used for built-in exceptions" instead.&lt;/p&gt;

&lt;p&gt;So I proposed a fingerprint: exception type, message, and the origin frame from the traceback. All immutable primitives, nothing retained. I posted it on the issue.&lt;/p&gt;

&lt;p&gt;Then the existing test suite told me I was wrong, in two different ways.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;test_breadcrumbs&lt;/code&gt; calls &lt;code&gt;capture_exception(ValueError())&lt;/code&gt; twice with two distinct, never-raised exceptions. No traceback, no message, so both fingerprints are identical and the second event gets dropped.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;test_option_before_breadcrumb&lt;/code&gt; was worse. It calls the same function three times, each raising a separate &lt;code&gt;ValueError("aha!")&lt;/code&gt; from the same line. Same type, same message, same origin frame. Three fingerprints, all identical, two events wrongly deduplicated.&lt;/p&gt;

&lt;p&gt;That is the flaw in the whole idea. &lt;strong&gt;A value fingerprint cannot tell "the same exception object twice" apart from "the same error raised repeatedly from the same line."&lt;/strong&gt; The first must deduplicate. The second must not. No fingerprint distinguishes them, because by value they are the same.&lt;/p&gt;

&lt;p&gt;I went back to the issue and said so.&lt;/p&gt;

&lt;h3&gt;
  
  
  The attempt that came before mine
&lt;/h3&gt;

&lt;p&gt;Late on, I went looking through the fork's branch list and found two abandoned maintainer branches from September 2025: &lt;code&gt;antonpirker/dedupe-integration-memory-usage&lt;/code&gt; and &lt;code&gt;antonpirker/make-dedupe-integration-more-memory-efficient&lt;/code&gt;. Neither was merged.&lt;/p&gt;

&lt;p&gt;The first one fingerprints on &lt;code&gt;(type_module, type_name, id(exc_value))&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That &lt;code&gt;id()&lt;/code&gt; is the interesting part. Once you stop retaining the exception, which is the entire point of the change, its address becomes immediately reusable, and CPython reuses addresses aggressively. Modelling that fingerprint over 2000 distinct, sequentially allocated &lt;code&gt;ValueError&lt;/code&gt;s:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight properties"&gt;&lt;code&gt;&lt;span class="err"&gt;distinct&lt;/span&gt; &lt;span class="err"&gt;exceptions&lt;/span&gt; &lt;span class="py"&gt;created&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2000&lt;/span&gt;
&lt;span class="err"&gt;distinct&lt;/span&gt; &lt;span class="py"&gt;fingerprints&lt;/span&gt;       &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2&lt;/span&gt;
&lt;span class="err"&gt;address-reuse&lt;/span&gt; &lt;span class="py"&gt;collisions&lt;/span&gt;    &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1998&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two fingerprints for two thousand distinct errors. Each collision is a real error that would be silently dropped as a duplicate.&lt;/p&gt;

&lt;p&gt;I mention this not to dunk on an abandoned branch, which is a draft nobody shipped, but because it is the third distinct way I watched this problem punish an obvious solution. Fingerprinting on value collapses errors that came from the same line. Fingerprinting on address collapses errors that reused the same memory. The information being fingerprinted just is not sufficient to express object identity.&lt;/p&gt;

&lt;h3&gt;
  
  
  The fix that works
&lt;/h3&gt;

&lt;p&gt;The problem was never that identity was the wrong key. It was that holding the object was the wrong way to hold identity.&lt;/p&gt;

&lt;p&gt;So keep the identity and drop the object. Attach a small weak-referenceable token to the exception, then weakly reference the token:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;_DedupeToken&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;__slots__&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__weakref__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is exactly one token per exception object, and it lives precisely as long as the exception does. &lt;code&gt;last_seen is token&lt;/code&gt; is therefore equivalent to the old &lt;code&gt;last_seen is exc&lt;/code&gt;, while the &lt;code&gt;ContextVar&lt;/code&gt; holds nothing that keeps a traceback alive.&lt;/p&gt;

&lt;p&gt;Deduplication behaviour is &lt;strong&gt;unchanged&lt;/strong&gt;, which is the part that matters given the maintainers' concern.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;The whole change to the hot path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;exc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;exc_info&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;new_last_seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Any&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# We can only weakref non builtin types.
&lt;/span&gt;    &lt;span class="n"&gt;new_last_seen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;weakref&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;is_duplicate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;last_seen&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TypeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Builtin exception. Referencing it strongly here would pin its
&lt;/span&gt;    &lt;span class="c1"&gt;# traceback and every frame local in that traceback for the
&lt;/span&gt;    &lt;span class="c1"&gt;# lifetime of the ContextVar (#6094). Weakly reference an
&lt;/span&gt;    &lt;span class="c1"&gt;# identity token carried by the exception instead, which dies
&lt;/span&gt;    &lt;span class="c1"&gt;# with it and keeps dedupe keyed on exception identity.
&lt;/span&gt;    &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;_identity_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;new_last_seen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;weakref&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;is_duplicate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;last_seen&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;new_last_seen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;
        &lt;span class="n"&gt;is_duplicate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;last_seen&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the token lookup, after hardening (more on that below):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_identity_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;BaseException&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Optional[_DedupeToken]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__traceback__&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;

        &lt;span class="n"&gt;exc_dict&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__dict__&lt;/span&gt;
        &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;exc_dict&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_DEDUPE_TOKEN_ATTR&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_DedupeToken&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;

        &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;_DedupeToken&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;exc_dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;_DEDUPE_TOKEN_ATTR&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# Exceptions can define ``__dict__`` as a property returning anything,
&lt;/span&gt;        &lt;span class="c1"&gt;# or back it with a mapping that refuses mutation. None of that may
&lt;/span&gt;        &lt;span class="c1"&gt;# break event processing, so fall back to the previous behaviour.
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;__traceback__ is None&lt;/code&gt; check is deliberate. An exception that was never raised has no frames to pin, so there is nothing to fix and no reason to touch it. That keeps &lt;code&gt;vars(exc)&lt;/code&gt; clean for the common case.&lt;/p&gt;

&lt;p&gt;Branch: &lt;a href="https://github.com/getsentry/sentry-python/compare/master...JonathanSolvesProblems:sentry-python:fix/dedupe-builtin-exception-retention" rel="noopener noreferrer"&gt;fix/dedupe-builtin-exception-retention&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Everything in this post is reproducible. The harness is at &lt;a href="https://github.com/JonathanSolvesProblems/sentry-dedupe-leak-repro" rel="noopener noreferrer"&gt;JonathanSolvesProblems/sentry-dedupe-leak-repro&lt;/a&gt;: the scaling measurement, the traceback probe, the five adversarial checks against my own fix, and the script that verifies the LLM review claim by claim.&lt;/p&gt;

&lt;h2&gt;
  
  
  My Improvements
&lt;/h2&gt;

&lt;p&gt;Five tests, and the first one fails without the fix, which is the only thing that makes it a regression test:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;test_dedupe_does_not_retain_builtin_exception&lt;/code&gt; puts a sentinel in the raising frame and asserts it is collectable afterwards&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;test_dedupe_still_dedupes_builtin_exception&lt;/code&gt; re-raises the same object and asserts one event&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;test_dedupe_distinguishes_equal_builtin_exceptions&lt;/code&gt; raises two identical-looking errors from the same line and asserts two events. This is the one fingerprinting broke&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;test_dedupe_leaves_unraised_exception_untouched&lt;/code&gt; asserts &lt;code&gt;vars(exc) == {}&lt;/code&gt; for a never-raised exception&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;test_dedupe_survives_exotic_exception_dict&lt;/code&gt; covers the crashes described below&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I also verified what the &lt;code&gt;ContextVar&lt;/code&gt; actually holds, rather than trusting the memory numbers:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;
&lt;code&gt;_last_seen&lt;/code&gt; holds&lt;/th&gt;
&lt;th&gt;1 MB payload after &lt;code&gt;gc.collect()&lt;/code&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;master&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ValueError&lt;/code&gt; (strong ref)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;alive&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fixed&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ReferenceType&lt;/code&gt; (weakref)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;freed&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;One process note, because it cost me an hour and nearly cost me a wrong claim. Once the fix was committed to a branch, &lt;code&gt;git stash push &amp;lt;file&amp;gt;&lt;/code&gt; had nothing to stash, so my "before and after" runs were quietly comparing the fix against itself. Two measurements came back identical and I believed them for longer than I should have. The tell was a probe printing whether the new symbol was importable, which said &lt;code&gt;True&lt;/code&gt; in both columns. Baselines have to be taken with &lt;code&gt;git checkout master -- &amp;lt;file&amp;gt;&lt;/code&gt;, and the baseline needs to prove it is actually the baseline.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best Use of Sentry
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Sentry features used:&lt;/strong&gt; Error Monitoring, Releases, custom Contexts and Tags, Issue Grouping, Seer (Autofix root cause analysis).&lt;/p&gt;

&lt;p&gt;The interesting part of instrumenting this was discovering that Sentry could not see the bug, and why.&lt;/p&gt;

&lt;p&gt;I built a small asyncio service that opens one long-lived task per session, hits one ordinary &lt;code&gt;ValueError&lt;/code&gt;, reports it, and keeps the session open. Ran it against a real Sentry project twice, tagged as two releases, &lt;code&gt;dedupe@unpatched&lt;/code&gt; and &lt;code&gt;dedupe@patched&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Sentry showed a completely healthy application. One issue, &lt;code&gt;ValueError: malformed frame on session N&lt;/code&gt;, 240 events, &lt;code&gt;handled: yes&lt;/code&gt;, &lt;code&gt;mechanism: generic&lt;/code&gt;. Exactly what you would expect from a service that reports its own handled errors. Nothing in it suggests 62 MB is being retained.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;That is the actual lesson.&lt;/strong&gt; This bug produces no error. The &lt;code&gt;ValueError&lt;/code&gt; is deliberate and correctly handled. Error monitoring alone is structurally incapable of surfacing a retention bug, because retention is not an event.&lt;/p&gt;

&lt;p&gt;There is a nice detail buried in that first issue, too. The frame local Sentry captured for the raising frame reads &lt;code&gt;session [Filtered]&lt;/code&gt;, scrubbed by default PII protection. The one object whose retention was the entire problem was the one thing redacted out of the report.&lt;/p&gt;

&lt;p&gt;So I used Sentry two other ways.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First, as a measuring instrument.&lt;/strong&gt; Every error event carries a custom &lt;code&gt;memory&lt;/code&gt; context with the retained figure at the moment it was sent. The errors are identical across both releases; the context is not. Comparing &lt;code&gt;dedupe@unpatched&lt;/code&gt; against &lt;code&gt;dedupe@patched&lt;/code&gt; on the same issue turns an invisible bug into a visible diff, without a single new error being raised.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Second, by making the leak raise something.&lt;/strong&gt; The demo checks its own retention after the sessions are open, and when session state is still reachable it reports a distinct &lt;code&gt;RetentionLeak&lt;/code&gt; issue carrying the diagnosis:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;sessions_pinned         &lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;60/60&lt;/span&gt;
&lt;span class="na"&gt;retained_mb             &lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;62.6&lt;/span&gt;
&lt;span class="na"&gt;referrer_chain          &lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Session &amp;lt;- frame(handle_message) &amp;lt;- traceback &amp;lt;- traceback &amp;lt;- exception(ValueError)&lt;/span&gt;
&lt;span class="na"&gt;dedupe_last_seen_holds  &lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ValueError&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That last line is the whole bug in one field, and getting it was its own small lesson. My first attempt read &lt;code&gt;DedupeIntegration._last_seen&lt;/code&gt; from the parent task and reported &lt;code&gt;nothing&lt;/code&gt;. &lt;code&gt;_last_seen&lt;/code&gt; is a &lt;code&gt;ContextVar&lt;/code&gt;, so every asyncio task has its own copy and it always reads empty from outside. Reading it from inside the capturing task gives &lt;code&gt;ValueError&lt;/code&gt; on the unpatched release and &lt;code&gt;ReferenceType&lt;/code&gt; on the patched one. The thing that made the diagnostic hard to write is precisely the thing that makes the bug scale with task count.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;release&lt;/th&gt;
&lt;th&gt;
&lt;code&gt;_last_seen&lt;/code&gt; holds&lt;/th&gt;
&lt;th&gt;sessions pinned&lt;/th&gt;
&lt;th&gt;retained&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dedupe@unpatched&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ValueError&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;60 / 60&lt;/td&gt;
&lt;td&gt;62.6 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dedupe@patched&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ReferenceType&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0 / 60&lt;/td&gt;
&lt;td&gt;1.3 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  What Seer made of it
&lt;/h3&gt;

&lt;p&gt;I connected the &lt;code&gt;sentry-python&lt;/code&gt; fork to the project and ran Autofix on the &lt;code&gt;RetentionLeak&lt;/code&gt; issue. I expected it to stop at my demo code, because that is where the stack trace points. It did not.&lt;/p&gt;

&lt;p&gt;Its root cause, verbatim:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;DedupeIntegration stores strong exception references in a ContextVar for builtin exceptions, retaining tracebacks and all frame locals (including large session buffers) for the asyncio task's lifetime.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And the supporting chain:&lt;/p&gt;

&lt;blockquote&gt;
&lt;ul&gt;
&lt;li&gt;The ValueError exceptions are kept alive because &lt;code&gt;DedupeIntegration._last_seen&lt;/code&gt; ContextVar holds a strong reference to each one.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;weakref.ref(exc)&lt;/code&gt; raises TypeError for builtin exception types like ValueError, so the fallback path stores the bare exception object instead of a weakref.&lt;/li&gt;
&lt;li&gt;Each asyncio task has its own copy of the ContextVar (per-task context), so the strong reference to the exception persists for the entire lifetime of each long-lived task.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5pgsebefht72oxxc4e2n.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5pgsebefht72oxxc4e2n.jpg" alt="Seer's root cause analysis, identifying the ContextVar strong reference, the weakref TypeError fallback, and the per-task context" width="800" height="723"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That is the bug, exactly, including the per-task ContextVar detail that cost me a debugging round of my own to work out. It cited &lt;code&gt;dedupe.py L1-L62&lt;/code&gt; as evidence, so it went and read the SDK source rather than guessing from the stack trace. Its five reproduction steps are accurate enough to hand to someone else.&lt;/p&gt;

&lt;p&gt;Then it proposed a fix:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;In &lt;code&gt;dedupe.py&lt;/code&gt;, change the &lt;code&gt;except TypeError&lt;/code&gt; fallback from &lt;code&gt;integration._last_seen.set(exc)&lt;/code&gt; to store a hashable identity tuple like &lt;code&gt;(type(exc), id(exc))&lt;/code&gt; instead of the exception object itself.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl7zsg7onfcsaj3st1bll.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl7zsg7onfcsaj3st1bll.jpg" alt="Seer's proposed plan: replace the bare exception fallback with an identity tuple" width="800" height="723"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That fix does not work, and it fails for the same reason the abandoned maintainer branch does. Storing only an identity key is what allows the exception to be freed, and a freed address is immediately reusable. Running Seer's exact tuple over 2000 distinct errors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight properties"&gt;&lt;code&gt;&lt;span class="err"&gt;distinct&lt;/span&gt; &lt;span class="err"&gt;errors&lt;/span&gt; &lt;span class="py"&gt;raised&lt;/span&gt;         &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2000&lt;/span&gt;
&lt;span class="err"&gt;events&lt;/span&gt; &lt;span class="err"&gt;wrongly&lt;/span&gt; &lt;span class="err"&gt;dropped&lt;/span&gt; &lt;span class="err"&gt;as&lt;/span&gt; &lt;span class="py"&gt;dupe&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1999  (100.0%)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It would trade a memory leak for silently discarding almost every error you report.&lt;/p&gt;

&lt;p&gt;There is a wrinkle I only found afterwards, and it is the most interesting thing in this whole section. Seer also opened a pull request against my fork with generated code, and &lt;strong&gt;the code is not what the plan said&lt;/strong&gt;. It does not store an identity tuple at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;integration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weakref&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TypeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That skips storing anything for builtins. It does remove the leak, and it has none of the address-reuse problem, but it silently disables deduplication for exactly the exception types most errors actually are. It is the fix the reporter originally proposed and that the maintainers explicitly declined, for the reason quoted at the top of this post.&lt;/p&gt;

&lt;p&gt;So the plan and the patch disagree with each other, and they are wrong in two different ways. If you let an agent open pull requests, that is worth knowing: the prose it shows you for review is not necessarily the diff it writes.&lt;/p&gt;

&lt;p&gt;I want to be fair about what that means, because it is not a gotcha. &lt;strong&gt;Seer did the hard part correctly.&lt;/strong&gt; Diagnosing this required reading past the stack trace into a third-party SDK, understanding weakref support on builtin types, and knowing that ContextVars are per-task under asyncio. It got all three from one event and the source. The part it got wrong is the part that requires knowing how CPython recycles memory addresses, which is not visible anywhere in the evidence it was given.&lt;/p&gt;

&lt;p&gt;And it is genuinely interesting that three independent attempts, an abandoned branch by a Sentry maintainer, my own first attempt, and Seer, all reached for a fingerprint or an identity key, and all three are wrong for the same underlying reason. The information available to a fingerprint is not sufficient to express object identity. That the AI converged on the same wrong answer as two humans is more a statement about the problem than about the AI.&lt;/p&gt;

&lt;p&gt;What I actually got out of Seer was the thing I would have wanted from a colleague: an independent confirmation of the diagnosis, arrived at from the evidence rather than from my framing of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best Use of Google AI
&lt;/h2&gt;

&lt;p&gt;I had already run my own adversarial pass on the patch: pickling, &lt;code&gt;copy&lt;/code&gt;/&lt;code&gt;deepcopy&lt;/code&gt;, payload leakage, &lt;code&gt;__slots__&lt;/code&gt; exceptions, and retention through &lt;code&gt;__context__&lt;/code&gt; chains. All five passed.&lt;/p&gt;

&lt;p&gt;So I gave the diff to Gemini 3.6 Flash, told it exactly what I had already covered, and asked only for failure modes I had missed, each with a snippet I could run. Then I ran all of them, because a review you do not verify is just a second opinion with extra steps.&lt;/p&gt;

&lt;p&gt;It returned six. The scoreboard:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;#&lt;/th&gt;
&lt;th&gt;Claim&lt;/th&gt;
&lt;th&gt;Verdict&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;__dict__&lt;/code&gt; as a property returning &lt;code&gt;None&lt;/code&gt; crashes event processing&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Confirmed, real crash&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;a &lt;code&gt;__dict__&lt;/code&gt; backed by a mapping that refuses mutation crashes&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Confirmed, real crash&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;a dynamic &lt;code&gt;__dict__&lt;/code&gt; breaks token identity&lt;/td&gt;
&lt;td&gt;Confirmed, fails open&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;the token is visible in &lt;code&gt;vars(exc)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Confirmed, real tradeoff&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;a thread race yields mismatched tokens&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Refuted&lt;/strong&gt;, 0 of 200 attempts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;__dict__.clear()&lt;/code&gt; loses the token&lt;/td&gt;
&lt;td&gt;Confirmed, fails open&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Claims 1 and 2 were genuine bugs that my own pass had missed. My &lt;code&gt;except (AttributeError, TypeError)&lt;/code&gt; was too narrow, so an exception class doing something unusual with &lt;code&gt;__dict__&lt;/code&gt; could crash Sentry's event processing. The SDK's own contributing guide says integrations must not crash applications, so those were not academic.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;NonDictException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nd"&gt;@property&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__dict__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;

&lt;span class="c1"&gt;# before: AttributeError: 'NoneType' object has no attribute 'get'
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Claim 5 was wrong, and I only know that because I ran it 200 times with a &lt;code&gt;threading.Barrier&lt;/code&gt; instead of nodding along.&lt;/p&gt;

&lt;p&gt;Claims 3 and 6 are real but fail &lt;strong&gt;open&lt;/strong&gt;: you get a duplicate event rather than losing one. That is the correct direction to fail.&lt;/p&gt;

&lt;p&gt;Claim 4 is a fair criticism and I have not made it go away. The token lives in &lt;code&gt;exc.__dict__&lt;/code&gt;, so it shows up in &lt;code&gt;vars(exc)&lt;/code&gt; and makes &lt;code&gt;json.dumps(vars(exc))&lt;/code&gt; raise. I narrowed it to raised exceptions only, and the SDK already does the same kind of thing (&lt;code&gt;sentry_sdk/integrations/django/__init__.py&lt;/code&gt; sets &lt;code&gt;_sentry_drf_request_backref&lt;/code&gt; on a user object), but it is a real tradeoff and the PR says so rather than hiding it.&lt;/p&gt;

&lt;p&gt;Two real crashes found, one confidently wrong claim caught, in a review that took about four minutes. That is a good trade, and it only works if you treat the output as a list of hypotheses instead of a list of findings.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away from this bug
&lt;/h2&gt;

&lt;p&gt;The comment &lt;code&gt;# we can only weakref non builtin types&lt;/code&gt; was right there, in the code, for years. It described the exact hazard and then the next line walked straight into it. A fallback path is still a code path, and this one ran for the overwhelming majority of real exceptions.&lt;/p&gt;

&lt;p&gt;The other thing: my measurements lied to me twice, in opposite directions. Once when I assumed unbounded growth and a flat graph corrected me, and once when a stash silently did nothing and two identical columns looked like a result. Both times the fix was the same, which was to make the harness prove its own setup before trusting a single number out of it.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>python</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Everyone Believes Black Dogs Get Left Behind. I Checked 99,916 Shelter Records. They Don't.</title>
      <dc:creator>JonathanSolvesProblems</dc:creator>
      <pubDate>Sun, 16 Aug 2026 12:18:08 +0000</pubDate>
      <link>https://dev.to/jonathansolvesstuff/everyone-believes-black-dogs-get-left-behind-i-checked-99916-shelter-records-they-dont-2ff5</link>
      <guid>https://dev.to/jonathansolvesstuff/everyone-believes-black-dogs-get-left-behind-i-checked-99916-shelter-records-they-dont-2ff5</guid>
      <description>&lt;p&gt;My family's dog died recently. He lived with my mom, I met him the day he was born, and I knew him almost his whole life. It was hard on all of us in the way that is difficult to explain to anyone who has not had it happen.&lt;/p&gt;

&lt;p&gt;A little while later my mom brought home another dog. He does not replace the first one, and nobody in my family pretends otherwise. But a house with a dog in it is a different house, and hers is whole again.&lt;/p&gt;

&lt;p&gt;That exchange happened about &lt;a href="https://www.aspca.org/helping-shelters-people-pets/us-animal-shelter-statistics" rel="noopener noreferrer"&gt;two million times last year in the United States alone&lt;/a&gt;, counting dogs only. What I had not thought about until this weekend is that there is a queue for it, that the queue is very long for some dogs and very short for others, and that the difference is not what almost everyone believes it is.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://jonathansolvesproblems.github.io/still-here/" rel="noopener noreferrer"&gt;Still Here&lt;/a&gt;&lt;/strong&gt; is a live board of every dog Austin Animal Center has an intake record for and no outcome record. As I write this there are &lt;strong&gt;510&lt;/strong&gt; of them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcpypr40qgzfjrc4932l5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcpypr40qgzfjrc4932l5.jpg" alt="The board opens on Pancho, a white Cairn Terrier who has been waiting 449 days and has no photograph anywhere public" width="800" height="438"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The one at the top is called Pancho. He is a white Cairn Terrier, he was picked up as a stray on May 23, 2025, and he has been there &lt;strong&gt;449 days&lt;/strong&gt;. 99.7% of the dogs in his breed group were adopted in less time than he has already been waiting.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every figure in this post is from the board on 16 August 2026. The site recomputes on every load, so by the time you read this the day counts will have moved and some of these dogs will have gone home.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Then I went looking for the reason the dogs at the top of that board are the ones at the top.&lt;/p&gt;

&lt;p&gt;The board moves. Between two runs of the pipeline a day apart, nine dogs came off it: &lt;strong&gt;seven were adopted&lt;/strong&gt;, one was reclaimed by an owner who came looking, and one was transferred to a partner rescue. Four new dogs arrived. Baxter had been waiting 71 days and went home on the fifteenth.&lt;/p&gt;

&lt;h3&gt;
  
  
  The claim I set out to check
&lt;/h3&gt;

&lt;p&gt;If you have spent any time around animal shelters you have heard of &lt;strong&gt;black dog syndrome&lt;/strong&gt;: the belief that black dogs sit unadopted while lighter dogs go home. It is repeated in shelter training material, in local news segments, and in a great many well-meaning adoption posts every October.&lt;/p&gt;

&lt;p&gt;It is a genuinely checkable claim, and Austin has published enough records to check it.&lt;/p&gt;

&lt;p&gt;I pulled every completed dog stay the city has released since October 2013 and reconstructed how long each one lasted: &lt;strong&gt;99,916 stays&lt;/strong&gt;, of which &lt;strong&gt;51,413&lt;/strong&gt; ended in adoption. Then I took the median wait for each coat color.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;coat&lt;/th&gt;
&lt;th&gt;median days to adoption&lt;/th&gt;
&lt;th&gt;n&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Blue&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;td&gt;2,064&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Brown Brindle&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;2,521&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fawn&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;617&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Brown&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;7,085&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;White&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;8,797&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chocolate&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;1,163&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Black&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;9&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;13,922&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tan&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;6,236&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cream&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;770&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Black dogs are in the fast half. Thirteen thousand nine hundred and twenty-two of them, leaving in a median of nine days, quicker than white, brown or chocolate.&lt;/p&gt;

&lt;p&gt;So the myth does not survive its first contact with the data. That much matches the published research: a &lt;a href="https://www.sciencedaily.com/releases/2016/02/160203185534.htm" rel="noopener noreferrer"&gt;2016 study by Christy Hoffman and colleagues&lt;/a&gt;, in &lt;em&gt;Animal Welfare&lt;/em&gt;, looked at nearly 16,700 records across two shelters and found black dogs left slightly faster than average at both.&lt;/p&gt;

&lt;p&gt;That study also reports the other half of what I found, which I did not know when I started looking: bully breeds stayed roughly 2.5 to 3 times longer than average. Two Pacific Northwest shelters, a decade earlier, a completely separate dataset from mine.&lt;/p&gt;

&lt;p&gt;But there is something much more interesting sitting in that same table, and it took me a second pass to see it.&lt;/p&gt;

&lt;h3&gt;
  
  
  The colors that wait are the colors pit bulls come in
&lt;/h3&gt;

&lt;p&gt;Look at the top of that list again. Blue, brown brindle, fawn. Those are not random slow colors. They are the coats you picture when you picture a pit bull.&lt;/p&gt;

&lt;p&gt;So I split every color by whether the dog was a bully-type breed, meaning pit bull, Staffordshire, or American bulldog:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzo0bobz6ggka2rxmnhob.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzo0bobz6ggka2rxmnhob.jpg" alt="Two charts on a shared scale: every bully-type coat lands between 25 and 32.5 days, every other coat between 7 and 11" width="800" height="475"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;coat&lt;/th&gt;
&lt;th&gt;bully-type&lt;/th&gt;
&lt;th&gt;everyone else&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Brown&lt;/td&gt;
&lt;td&gt;32.5&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Black&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;31&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;8&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tan&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;White&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Red&lt;/td&gt;
&lt;td&gt;27&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chocolate&lt;/td&gt;
&lt;td&gt;26.5&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Blue&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Brown Brindle&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The two columns do not overlap anywhere. Every bully-type cell is between 25 and 32.5 days. Every other cell is between 7 and 11. The &lt;em&gt;fastest&lt;/em&gt; bully-type coat is still 14 days slower than the &lt;em&gt;slowest&lt;/em&gt; coat outside the group.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Within a breed group, coat color moves the median wait by at most 7.5 days. Changing the breed group moves it by 20.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Black only looked fast in the first table because black dogs are the &lt;em&gt;least&lt;/em&gt; likely to be bully-type: 12% of them, against 81% of the blue ones. The color signal was a breed signal the whole time. A black dog that is not a bully breed waits 8 days. A black pit bull waits 31.&lt;/p&gt;

&lt;p&gt;Across the whole corpus: &lt;strong&gt;bully-type dogs wait a median of 28 days, and every other dog waits 8.&lt;/strong&gt; That is 3.5 times, on 9,658 bully-type adoptions against 41,755 others.&lt;/p&gt;

&lt;p&gt;And it is not a historical curiosity. Of the 510 dogs sitting in that shelter today, &lt;strong&gt;216 are bully-type&lt;/strong&gt;. Four of the five longest waits on the board right now are pit bulls.&lt;/p&gt;

&lt;h3&gt;
  
  
  The thing the adoption site does not tell you
&lt;/h3&gt;

&lt;p&gt;Austin publishes, in open data, exactly how long every dog has been in its care. Its adoption site shows none of it.&lt;/p&gt;

&lt;p&gt;I checked the listings the city actually serves to adopters. A listing carries breed, sex, size, age, kennel number, and a set of behavior tags. There is no intake date on it, no length of stay, and no way to sort or filter by either. There is a manual "Long-term resident" tag a staff member can tick, but nothing that gives you a number.&lt;/p&gt;

&lt;p&gt;So a person browsing Austin's adoption page can look straight at Fritz and have no idea he arrived on May 28 of last year.&lt;/p&gt;

&lt;p&gt;Both halves of that are public. They are just never printed next to each other. Still Here is the join.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/RNa5V26TXzI"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Under two minutes, narrated over the live site, with the research citations on screen where the claims are made.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live board: &lt;a href="https://jonathansolvesproblems.github.io/still-here/" rel="noopener noreferrer"&gt;jonathansolvesproblems.github.io/still-here&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The board is a photographic contact sheet, one frame per dog, ordered by how long each has waited. The day count and the glow are that wait: the longer a dog has been there, the more it has burned into the plate.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0m6qo969kn95n9slmcl5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0m6qo969kn95n9slmcl5.jpg" alt="The contact sheet, 510 frames of real dogs with real photographs from Austin's adoption listings" width="800" height="475"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every frame links to that dog's real adoption page, so clicking one takes you somewhere you could actually adopt it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ssvl2tdxgszr7pehhr5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ssvl2tdxgszr7pehhr5.jpg" alt="Canela, 323 days, with her photograph, an adopt button, and a play button to hear her record read aloud" width="800" height="269"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Things worth trying:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Search a breed.&lt;/strong&gt; Type &lt;code&gt;labrador&lt;/code&gt;, &lt;code&gt;husky&lt;/code&gt; or &lt;code&gt;brindle&lt;/code&gt;. Multiple words narrow rather than widen, so &lt;code&gt;black lab&lt;/code&gt; finds 19 dogs. Austin's own site lets you filter by breed but never tells you how long any of them has waited; this does both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Press play on any dog.&lt;/strong&gt; Each one reads its own record aloud, and each breed group opens with its own sound: a howl for the hounds, a husky's howl for the spitz breeds, a yip for the toy dogs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Filter to "Over a year"&lt;/strong&gt; for the 18 dogs who have been there more than 365 days.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Filter to "Unnamed"&lt;/strong&gt; for the ones who arrived without a name.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Click any day count&lt;/strong&gt; to pin that dog to the top, then copy the link. &lt;code&gt;?dog=7950&lt;/code&gt; is Pancho.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/JonathanSolvesProblems" rel="noopener noreferrer"&gt;
        JonathanSolvesProblems
      &lt;/a&gt; / &lt;a href="https://github.com/JonathanSolvesProblems/still-here" rel="noopener noreferrer"&gt;
        still-here
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      510 dogs have an intake record at the Austin animal shelter and no outcome record. This is the board.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Still Here&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;510 dogs have an intake record at the Austin animal shelter and no outcome record. This is the board.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Built for the &lt;a href="https://dev.to/challenges/weekend-2026-08-13" rel="nofollow"&gt;DEV Weekend Challenge: Dog Days Edition&lt;/a&gt;, August 2026.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Live board: &lt;a href="https://jonathansolvesproblems.github.io/still-here/" rel="nofollow noopener noreferrer"&gt;https://jonathansolvesproblems.github.io/still-here/&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;▶ Watch the demo (2 min): &lt;a href="https://www.youtube.com/watch?v=RNa5V26TXzI" rel="nofollow noopener noreferrer"&gt;https://www.youtube.com/watch?v=RNa5V26TXzI&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Write-up: &lt;a href="https://dev.to/jonathansolvesstuff/everyone-believes-black-dogs-get-left-behind-i-checked-99916-shelter-records-they-dont-2ff5" rel="nofollow"&gt;on DEV&lt;/a&gt;
and &lt;a href="https://jonathanandrei.com/blog/still-here-austin-shelter-black-dog-syndrome-99916-stays/" rel="nofollow noopener noreferrer"&gt;on my site&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The longest wait on the board belongs to Pancho, a white Cairn Terrier who was
picked up as a stray on May 23, 2025. As of the last data pull he had been there
&lt;strong&gt;449 days&lt;/strong&gt;. 99.7% of dogs in his breed group were adopted in less time than he
has already been waiting.&lt;/p&gt;
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;The finding&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;Black dog syndrome, the belief that a dark coat keeps a dog in the kennel
does not appear in Austin's records. Across &lt;strong&gt;99,916 completed dog stays&lt;/strong&gt; since
2013, black dogs leave in a median of &lt;strong&gt;9 days&lt;/strong&gt;, faster than white, brown…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/JonathanSolvesProblems/still-here" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;Everything is in the open, including the corpus, so every number here is checkable.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The data has two eras, and the famous one is frozen
&lt;/h3&gt;

&lt;p&gt;Austin publishes intakes and outcomes on Socrata with no API key required. The dataset IDs that every tutorial points at are &lt;strong&gt;archives frozen at 2025-05-05&lt;/strong&gt;. Query &lt;code&gt;wter-evkm&lt;/code&gt; today and you get a tidy 173,812 rows, not one of them from the last fifteen months. The live feeds are different IDs &lt;em&gt;and a different schema&lt;/em&gt;, and finding that out was most of an hour.&lt;/p&gt;

&lt;p&gt;This project reads both. The live outcome feed ships a &lt;code&gt;days_in_shelter&lt;/code&gt; column; the archive does not, so each archived outcome is paired with the most recent intake for that animal that &lt;em&gt;precedes&lt;/em&gt; it. Dogs cycle through the shelter more than once, and matching on animal ID alone would charge a repeat stray's whole history to a single stay. 652 outcomes could not be paired and were dropped rather than guessed at.&lt;/p&gt;

&lt;h3&gt;
  
  
  Snowflake does the counting
&lt;/h3&gt;

&lt;p&gt;All 99,916 stays and the 510 current dogs load through an internal stage, and every number quoted above comes out of a view rather than out of application code. The bully-type definition lives in a single SQL function, &lt;code&gt;IS_BULLY()&lt;/code&gt;, so the one judgement call in the whole analysis is auditable in one place instead of scattered through Python.&lt;/p&gt;

&lt;p&gt;The piece that genuinely wants a warehouse is the per-dog percentile: for each of the 510 waiting dogs, rank its current wait against every completed stay in its breed group. That is 510 live values against 51,413 historical ones, and it is one view.&lt;/p&gt;

&lt;p&gt;I also kept the Python implementation and made the two argue. &lt;code&gt;verify.py&lt;/code&gt; re-runs every headline question locally and diffs it against what Snowflake returned, across the medians, the counts, the color table, the controlled split, the spreads, and the per-dog percentile for the top 25 dogs. &lt;strong&gt;102 of 102 checks agree.&lt;/strong&gt; If they ever disagree, one of them is wrong and the number does not go in the writeup.&lt;/p&gt;

&lt;h3&gt;
  
  
  Finding the faces
&lt;/h3&gt;

&lt;p&gt;The open data has no photographs in it. But Austin's adoption listings run on a platform called Adopets, and its &lt;code&gt;code&lt;/code&gt; field is the same animal id the open data uses, so the two join directly.&lt;/p&gt;

&lt;p&gt;I checked the join rather than trusting it: of the matched dogs that have a name on both sides, &lt;strong&gt;97% agree&lt;/strong&gt;, and the handful that disagree are the shelter renaming a dog after intake. &lt;code&gt;Unknown&lt;/code&gt; becomes &lt;code&gt;Tinkerbelle&lt;/code&gt;. &lt;code&gt;Fat Boy&lt;/code&gt; becomes &lt;code&gt;Big Boy&lt;/code&gt;. The importer refuses to write the mapping at all if agreement drops below 90%, because a wrong face on a real animal is worse than no face.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;377 of the 510 have a real photograph&lt;/strong&gt;, embedded from the shelter's own platform rather than copied.&lt;/p&gt;

&lt;p&gt;133 have none. I nearly wrote that up as dogs being overlooked, which would have been wrong. Their median wait is 20 days against 84 for the listed ones, so most of them are simply recent arrivals. Texas shelters hold a stray for a period before it can be offered for adoption, set by local ordinance rather than by state law, and that is the likeliest explanation for the gap. I cannot verify it dog by dog from public data, so I am not going to claim it as more than the obvious reading. But the unlisted share never drops below about a sixth at any length of stay, and &lt;strong&gt;33 dogs have passed 100 days without appearing publicly at all&lt;/strong&gt;. Pancho is one of them, which is why the biggest frame on the page is empty.&lt;/p&gt;

&lt;p&gt;For those 133 I generated one illustration per breed group and used it as a CSS mask, so the alpha channel carries the drawing while the fill comes from the coat the shelter recorded. A black pit bull and a white Great Pyrenees stay visibly different animals. They illustrate a breed &lt;em&gt;type&lt;/em&gt;, never an individual: generating a portrait of a specific dog nobody has photographed would be inventing evidence about a real animal, and the "no photograph on file" caption stays on every one.&lt;/p&gt;

&lt;h3&gt;
  
  
  Giving them a voice
&lt;/h3&gt;

&lt;p&gt;Each of the longest-waiting dogs reads its own record aloud, and every clip opens with a real generated sound for that breed group: a mournful howl for the hounds, a husky's howl for the spitz breeds, a tiny yip for the toy dogs, a single deep woof for the guardians.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Nothing any dog says is invented.&lt;/strong&gt; Every clause maps to a column:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;They called me Pancho. I'm a white Cairn Terrier. I was picked up as a stray on May 23, 2025. That was 448 days ago, more than a year. 99.7 percent of dogs like me were already home by now. The usual wait is 8 days. Nobody has come for me yet.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The clips are recordings made on 2026-08-15, so a day count spoken aloud can sit a day behind the board, which recomputes on every load.&lt;/p&gt;

&lt;p&gt;Name, coat, breed, intake reason, intake date, elapsed days, cohort percentile, cohort median. Eight fields, no biography, no "loves long walks". The phrasing varies between dogs but is selected by a hash of the animal ID rather than randomly, so re-running produces byte-identical scripts.&lt;/p&gt;

&lt;p&gt;The pit bulls deliberately got the warmest, deepest voice available, because 51 of the dogs without photographs are pit bulls and a growling read would reinforce the exact assumption the other 99,916 records spend this whole page dismantling.&lt;/p&gt;

&lt;p&gt;A straight top-16 by wait length turned out to be thirteen pit bulls, which meant no howl or yip would ever have played anywhere, so the generator tops up with the longest waiter from every breed group the top-16 missed. Twenty-three clips, all ten groups.&lt;/p&gt;

&lt;h3&gt;
  
  
  Things I had to get right, and four I got wrong first
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Medians, not averages.&lt;/strong&gt; Length of stay is heavily right-skewed. The mean wait for a bully-type dog is 66 days against a median of 28, because a handful of dogs stay for years. Every headline number here is a median.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The controlled table was wrong the first time.&lt;/strong&gt; It required only 30 adoptions per cell, and reported that coat color moved the wait by 28 days within bully-type dogs, which would have undercut the entire finding. The top and bottom of that table were Yellow Brindle at n=35 and Gray at n=62. Two thin cells were setting the whole spread. At a floor of 250 the same table shows 7.5 days.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;15 of the dogs on my first board had arrived dead.&lt;/strong&gt; Animals recorded dead on intake never get an outcome row, so the anti-join that finds waiting dogs finds them too, and there they were on a page captioned &lt;em&gt;nobody has come for them yet&lt;/em&gt;. Filtering them took the count from 531 to 516.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;And one more after that.&lt;/strong&gt; Princess has two intake rows a day apart with no outcome between them, so she was on the board twice. The anti-join emits one row per intake, not per dog. Deduplicating by animal id took it to &lt;strong&gt;515&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The charts drew nothing at all for a while.&lt;/strong&gt; Both panels rendered a bar of identical length whatever the number said, because the fill was an inline &lt;code&gt;&amp;lt;span&amp;gt;&lt;/code&gt; and inline elements silently ignore &lt;code&gt;width&lt;/code&gt;. The comparison the whole page is built on was showing two identical rows.&lt;/p&gt;

&lt;p&gt;Because the writeup kept drifting from the pipeline, there is now a &lt;code&gt;factcheck.py&lt;/code&gt; that re-reads the data and asserts every number quoted in this post. It caught eight stale figures the first time I ran it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Snowflake&lt;/strong&gt; and &lt;strong&gt;ElevenLabs&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Snowflake holds the corpus and answers every question in the piece, with the one judgement call isolated in a SQL function and a 102-check cross-examination against an independent Python implementation.&lt;/p&gt;

&lt;p&gt;ElevenLabs gives 23 dogs a voice and gives each breed group its own animal, generated rather than sampled.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this is not
&lt;/h2&gt;

&lt;p&gt;This is one municipal shelter, and an atypical one. Austin Animal Center operates under a &lt;a href="https://austintexas.gov/page/ordinance-change-faqs" rel="noopener noreferrer"&gt;City of Austin mandate of a 95% live-release rate&lt;/a&gt;, so these waits are likely shorter than the national picture rather than longer.&lt;/p&gt;

&lt;p&gt;Nothing here is causal. It shows which dogs wait, not why any individual adopter chose as they did.&lt;/p&gt;

&lt;p&gt;Breed labels deserve a warning too. They are assigned by staff from appearance, and &lt;a href="https://nationalcanineresearchcouncil.com/research_library/summary-analysis-inconsistent-identification-of-pit-bull-type-dogs-by-shelter-staff/" rel="noopener noreferrer"&gt;Olson et al. (2015)&lt;/a&gt; found that shelter staff called 52% of a sample of dogs pit bull-type while DNA put the figure at 21% (&lt;em&gt;The Veterinary Journal&lt;/em&gt; 206: 197-202). So "bully-type" here does not mean &lt;em&gt;a dog with pit bull ancestry&lt;/em&gt;. It means &lt;em&gt;a dog Austin wrote down as a pit bull&lt;/em&gt;. For a question about who gets adopted that is arguably the better variable anyway, because the label on the kennel card is what an adopter actually reads.&lt;/p&gt;

&lt;p&gt;"No outcome record" is also not identical to "in the building". Some of these dogs are in foster care, and the outcome feed lags.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I built it
&lt;/h2&gt;

&lt;p&gt;The interesting thing about the black dog myth is not that it is wrong. It is that it is a &lt;em&gt;kinder&lt;/em&gt; thing to believe. If dogs are passed over for the colour of their coat, that is a superstition, and superstitions can be argued away with a good photo and a hashtag.&lt;/p&gt;

&lt;p&gt;The real pattern is not a superstition. People are avoiding a breed. 216 of the dogs in that shelter tonight are that breed, and the ones at the top of the board have been waiting since before last summer.&lt;/p&gt;

&lt;p&gt;Pancho is not one of them. He is a small white terrier and he has been there 449 days, which is the part I cannot explain and did not try to.&lt;/p&gt;

&lt;p&gt;When my mom brought that second dog home, one dog stopped waiting. I did not think about it in those terms at the time and I doubt she did either. This weekend I built the list of everyone still waiting, and it turns out to be 510 names long.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Live board:&lt;/strong&gt; &lt;a href="https://jonathansolvesproblems.github.io/still-here/" rel="noopener noreferrer"&gt;https://jonathansolvesproblems.github.io/still-here/&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/JonathanSolvesProblems/still-here" rel="noopener noreferrer"&gt;https://github.com/JonathanSolvesProblems/still-here&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Blog Post:&lt;/strong&gt; &lt;a href="https://jonathanandrei.com/blog/still-here-austin-shelter-black-dog-syndrome-99916-stays/" rel="noopener noreferrer"&gt;https://jonathanandrei.com/blog/still-here-austin-shelter-black-dog-syndrome-99916-stays/&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Challenge:&lt;/strong&gt; &lt;a href="https://dev.to/challenges/weekend-2026-08-13"&gt;DEV Weekend Challenge: Dog Days Edition&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Data:&lt;/strong&gt; Austin Animal Center open data, &lt;a href="https://data.austintexas.gov/resource/pyqf-r2dc.json" rel="noopener noreferrer"&gt;live intakes&lt;/a&gt; and &lt;a href="https://data.austintexas.gov/resource/gsvs-ypi7.json" rel="noopener noreferrer"&gt;live outcomes&lt;/a&gt;. No API key needed, so every number here is checkable.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;If you are anywhere near Austin, the board is real and so are the dogs on it.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>weekendchallenge</category>
      <category>snowflake</category>
      <category>elevenlabs</category>
      <category>showdev</category>
    </item>
    <item>
      <title>Passion is the one thing you are not supposed to be able to fake. So I built a machine that manufactures it.</title>
      <dc:creator>JonathanSolvesProblems</dc:creator>
      <pubDate>Sun, 12 Jul 2026 01:52:27 +0000</pubDate>
      <link>https://dev.to/jonathansolvesstuff/passion-is-the-one-thing-you-are-not-supposed-to-be-able-to-fake-so-i-built-a-machine-that-5e58</link>
      <guid>https://dev.to/jonathansolvesstuff/passion-is-the-one-thing-you-are-not-supposed-to-be-able-to-fake-so-i-built-a-machine-that-5e58</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/challenges/weekend-2026-07-09"&gt;Weekend Challenge: Passion Edition&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;The prompt for this challenge was passion, so I took the most literal possible&lt;br&gt;
run at it.&lt;/p&gt;

&lt;p&gt;Every World Cup app ever made is built for people who already care. Fixtures,&lt;br&gt;
scores, brackets, league tables, fantasy teams. Every one of them assumes you&lt;br&gt;
turned up with a side. Nobody builds anything for the person who wants to care&lt;br&gt;
and has nobody to care about, which is most of the planet: a few billion people&lt;br&gt;
will watch this tournament, and the overwhelming majority of them have no dog in&lt;br&gt;
any of it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pick Your Side&lt;/strong&gt; is for them. You name any two nations. It goes and reads their&lt;br&gt;
actual history, picks the one you were always meant to love, tells you why, and a&lt;br&gt;
stadium announcer swears you in.&lt;/p&gt;

&lt;p&gt;Two nations in. One side out, and a reason to mean it.&lt;/p&gt;

&lt;p&gt;The interesting problem is that you cannot fake this with a nice adjective. If&lt;br&gt;
the app tells you Ghana and Uruguay have a bitter history and the history is&lt;br&gt;
invented, then the passion it hands you is counterfeit, and the moment you look&lt;br&gt;
it up you will feel stupid and never open it again. Manufactured passion only&lt;br&gt;
works if the thing it is built out of is true.&lt;/p&gt;

&lt;p&gt;So the whole architecture is bent around that one constraint.&lt;/p&gt;


&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;
&lt;h3&gt;
  
  
  &lt;strong&gt;&lt;a href="https://pick-your-side.vercel.app" rel="noopener noreferrer"&gt;pick-your-side.vercel.app&lt;/a&gt;&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Turn your sound on.&lt;/strong&gt; Half of this app is audio.&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/fND6klvdVqY"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fukxq3n6co77f29j5yzow.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fukxq3n6co77f29j5yzow.png" alt="The Pick Your Side landing page. Huge white type on near black reads " width="800" height="498"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Try &lt;strong&gt;Uruguay against Ghana&lt;/strong&gt;. It will side with Ghana, and it will tell you why:&lt;br&gt;
Luis Suarez's handball on the line in 2010, the penalty Asamoah Gyan hit off the&lt;br&gt;
crossbar in the last minute, the first African side ever to stand that close to a&lt;br&gt;
semi-final. It also knows Ghana beat them 2-0 in the group stage a few days ago,&lt;br&gt;
and it still argues the wound has not closed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa5km4r2f4pvpgnzvqoo7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa5km4r2f4pvpgnzvqoo7.png" alt="The reveal. The screen has flooded green and the word GHANA fills it, with the verdict " width="800" height="558"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Or try &lt;strong&gt;England against Germany&lt;/strong&gt;, if you want to watch it side with the team&lt;br&gt;
that has not won anything since 1966.&lt;/p&gt;

&lt;p&gt;Here is the part I am proudest of. Every fact above came from a live search, and&lt;br&gt;
the app will show you exactly which pages it read:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fki4w8t189vlnla3ue85l.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fki4w8t189vlnla3ue85l.png" alt="The Verified panel, expanded. On the left, the six live searches Gemini actually ran. On the right, the six pages it actually read, each one a link you can open and check." width="800" height="574"&gt;&lt;/a&gt;&lt;/p&gt;


&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/JonathanSolvesProblems" rel="noopener noreferrer"&gt;
        JonathanSolvesProblems
      &lt;/a&gt; / &lt;a href="https://github.com/JonathanSolvesProblems/pick-your-side" rel="noopener noreferrer"&gt;
        pick-your-side
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Pick Your Side&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;The World Cup is on and you do not have a team.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Every World Cup app ever built is for the people who already care. This one is
for everybody else. Name any two nations. It reads their real history, picks the
side you were always meant to love, and a stadium announcer swears you in.&lt;/p&gt;
&lt;p&gt;Built for the &lt;a href="https://dev.to/challenges/weekend-2026-07-09" rel="nofollow"&gt;DEV Weekend Challenge: Passion Edition&lt;/a&gt;.&lt;/p&gt;
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;How it works&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;The interesting part is that none of the history is invented.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. Gemini researches, grounded in Google Search.&lt;/strong&gt;
The first call has the Google Search tool switched on. Gemini goes and finds the
actual head to head record, the actual famous match, and where both sides
actually stand in the tournament being played right now. It comes back with the
pages it read and the searches it ran, and the app shows you both. Ask it about
Morocco and…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/JonathanSolvesProblems/pick-your-side" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;





&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Gemini never gets to work from memory
&lt;/h3&gt;

&lt;p&gt;Because of that constraint, it runs twice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Call one goes and finds out.&lt;/strong&gt; Google Search grounding is switched on, and&lt;br&gt;
Gemini is told to report only what it can verify: every World Cup meeting between&lt;br&gt;
the two nations with year and score, the single most famous match, what each side&lt;br&gt;
has actually won, and where both of them stand in the tournament being played&lt;br&gt;
right now.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;research&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;models&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generateContent&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;gemini-2.5-flash&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`Research &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; and &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; ... Be concise and factual.
    If they have rarely or never played each other, say that plainly
    instead of inventing a rivalry.`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt; &lt;span class="na"&gt;googleSearch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="p"&gt;}]&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Call two writes the case&lt;/strong&gt;, and it only ever sees the facts call one came back&lt;br&gt;
with. It is held to a &lt;code&gt;responseSchema&lt;/code&gt;, so the output is always the same shape:&lt;br&gt;
a side, a verdict, the history, the stakes, one legendary moment, your reason, a&lt;br&gt;
line you can share, and a script for the announcer.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;written&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;models&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generateContent&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;gemini-2.5-flash&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`...Here is what was just researched. Every fact you use must
    come from here. Do not add matches, scores or players that are not in it:
    &amp;lt;researched-facts&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;facts&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/researched-facts&amp;gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;responseMimeType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;responseSchema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;revealSchema&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Why two calls, which is the bit that cost me an hour
&lt;/h3&gt;

&lt;p&gt;I did not want two calls. I wanted one: grounded and schema'd together. The&lt;br&gt;
Gemini docs say Gemini 3 supports exactly that, structured output combined with&lt;br&gt;
Google Search.&lt;/p&gt;

&lt;p&gt;On &lt;code&gt;gemini-2.5-flash&lt;/code&gt;, asking for &lt;code&gt;googleSearch&lt;/code&gt; and a &lt;code&gt;responseSchema&lt;/code&gt; in the&lt;br&gt;
same call &lt;strong&gt;does not throw an error&lt;/strong&gt;. It quietly ignores the schema and hands&lt;br&gt;
you back prose.&lt;/p&gt;

&lt;p&gt;That is a genuinely nasty failure. Nothing fails, nothing warns, the call&lt;br&gt;
succeeds, and then your UI does &lt;code&gt;JSON.parse()&lt;/code&gt; on three paragraphs of essay. I&lt;br&gt;
only caught it because I wrote a throwaway script to poke at the API surface&lt;br&gt;
before I wrote any UI, and I printed the raw output instead of trusting the&lt;br&gt;
status code. If I had trusted it, the reveal would have exploded live.&lt;/p&gt;

&lt;p&gt;So: research first, then shape. Which turned out to be the better architecture&lt;br&gt;
anyway, because splitting them meant the facts arrive with sources attached.&lt;/p&gt;
&lt;h3&gt;
  
  
  The model was thinking, and thinking was making it worse
&lt;/h3&gt;

&lt;p&gt;The whole thing was taking twenty six seconds, which is a long time to look at a&lt;br&gt;
loading screen. &lt;code&gt;gemini-2.5-flash&lt;/code&gt; reasons before it answers unless you tell it&lt;br&gt;
not to. I turned it off on both calls. Same model, same prompts, measured:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;grounded&lt;/th&gt;
&lt;th&gt;written&lt;/th&gt;
&lt;th&gt;total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;thinking on&lt;/td&gt;
&lt;td&gt;11.6s&lt;/td&gt;
&lt;td&gt;14.7s&lt;/td&gt;
&lt;td&gt;26.3s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;thinking off&lt;/td&gt;
&lt;td&gt;5.1s&lt;/td&gt;
&lt;td&gt;1.0s&lt;/td&gt;
&lt;td&gt;6.1s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Four times faster. And the answers got &lt;strong&gt;better&lt;/strong&gt;, which I did not expect. Asked&lt;br&gt;
for a verdict, thinking on, it reasoned its way to:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;France ends Morocco's World Cup run&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That is a news headline. It is not a reason to love anybody. Thinking off:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Unfinished business and a deeper wound&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The first call is looking things up and the second is being asked to write.&lt;br&gt;
Neither one is a reasoning problem, and reasoning at them cost me twenty seconds&lt;br&gt;
and the only good line in the output.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;responseMimeType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;responseSchema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;revealSchema&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;temperature&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;thinkingConfig&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;thinkingBudget&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;   &lt;span class="c1"&gt;// &amp;lt;- this&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Showing the receipts
&lt;/h3&gt;

&lt;p&gt;Because the grounded call comes back with grounding metadata, I know exactly&lt;br&gt;
which pages Gemini read and which searches it ran. So the app shows you.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Verified.&lt;/strong&gt; None of this was invented. Gemini ran 6 live searches and read 6&lt;br&gt;
pages before it made the case.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Expand it and you get the real search strings and links to fifa.com, wikipedia,&lt;br&gt;
fotmob, whatever it actually used. You can go check it. It is the difference&lt;br&gt;
between a story and a fact, and it is the thing I would want if I were the one&lt;br&gt;
being told who to love.&lt;/p&gt;

&lt;p&gt;Here is what convinced me it was worth doing. I asked it about Morocco against&lt;br&gt;
France, and it told me about the 2026 quarter final, a match played days ago.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"This was tragically repeated in the 2026 World Cup quarterfinal, with France&lt;br&gt;
again winning 2-0."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;No language model knows that from training. It went and looked.&lt;/p&gt;
&lt;h3&gt;
  
  
  Telling it to pick the loser
&lt;/h3&gt;

&lt;p&gt;The instruction that made the biggest difference to how the thing feels:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Choose the side with the better story for a newcomer: the deeper wound, the&lt;br&gt;
longer wait, the thing still unfinished. &lt;strong&gt;Do not default to the more famous or&lt;br&gt;
more successful nation.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Without that line it picks Brazil, France, Germany, every time, because they are&lt;br&gt;
the most written about. With it, it picks Morocco over France and argues for the&lt;br&gt;
Atlas Lions. It picks the Netherlands over Senegal and leads with three lost&lt;br&gt;
finals. That single sentence is the difference between an app that reports and an&lt;br&gt;
app that has taste.&lt;/p&gt;
&lt;h3&gt;
  
  
  The voice
&lt;/h3&gt;

&lt;p&gt;Gemini writes the swear-in. ElevenLabs performs it.&lt;/p&gt;

&lt;p&gt;The reveal lands, the nation's colour floods the screen, and a voice comes in&lt;br&gt;
over the top of it. It is the moment the whole thing exists for, and it does not&lt;br&gt;
work on the page as text.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stream&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;textToSpeech&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;convert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;VOICE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`[shouting over a stadium crowd] &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;script&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;modelId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;eleven_v3&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;outputFormat&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;mp3_44100_128&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;voiceSettings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;stability&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.35&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;similarityBoost&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.75&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;style&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.65&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;eleven_v3&lt;/code&gt; takes inline audio tags, so &lt;code&gt;[shouting over a stadium crowd]&lt;/code&gt; is&lt;br&gt;
direction for the read, not words that get spoken. Low stability plus high style&lt;br&gt;
is what stops it sounding like an audiobook.&lt;/p&gt;

&lt;p&gt;And then I listened to it, and the problem was obvious. A man was shouting about&lt;br&gt;
a stadium, in total silence. He sounded unhinged.&lt;/p&gt;

&lt;p&gt;So ElevenLabs builds the stadium too. Their sound effects API takes a text&lt;br&gt;
prompt the same way the voice does, and I asked it for two things: a restless&lt;br&gt;
crowd that murmurs under the scouting screen while the app is thinking, and an&lt;br&gt;
eruption for the instant the nation's name lands.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;el&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;textToSoundEffects&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;convert&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;A packed football stadium erupting the instant a goal goes in. &lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;A sudden enormous crowd roar, then it settles into a sustained wall &lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;of cheering that slowly falls away. No music, no commentary.&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;durationSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The crowd sits at a third of the volume and never competes with the announcer.&lt;br&gt;
Both cues are generated once and committed, so they cost nothing and start&lt;br&gt;
instantly. Sound is on by default, because a muted reveal is just a poster, and&lt;br&gt;
the control to turn it off is on screen the entire time.&lt;/p&gt;

&lt;p&gt;There are three layers under the voice, because I did not want a visitor to&lt;br&gt;
arrive at a silent reveal:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;live text to speech of the script Gemini just wrote for that exact matchup&lt;/li&gt;
&lt;li&gt;if that fails or the free quota is spent, a pre-rendered stinger for that
nation, which ships with the app&lt;/li&gt;
&lt;li&gt;if both fail, silence, and the reveal is otherwise unharmed&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Same matchup, same words, same audio, so it is cached and only the first visitor&lt;br&gt;
to a given pairing costs anything.&lt;/p&gt;

&lt;h3&gt;
  
  
  The announcer kept turning up late
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;eleven_v3&lt;/code&gt; is the expressive one. It also takes 4.4 seconds, so the nation's&lt;br&gt;
name would slam onto the screen and then nothing would happen for four seconds&lt;br&gt;
and then a voice would arrive, to an empty room. Worse, I had put the dramatic&lt;br&gt;
pause &lt;strong&gt;after&lt;/strong&gt; the download instead of measuring it from the start of the&lt;br&gt;
reveal, so the network time and the pause stacked and the announcer was&lt;br&gt;
arriving 5.1 seconds late. I only found that because I scripted a browser to&lt;br&gt;
click through the live site and timestamp the moment the audio element actually&lt;br&gt;
started playing, which is not something a screenshot will ever tell you.&lt;/p&gt;

&lt;p&gt;There is also a trap in the audio tags. &lt;code&gt;[shouting over a stadium crowd]&lt;/code&gt; is&lt;br&gt;
direction, not dialogue, but &lt;strong&gt;only on v3&lt;/strong&gt;. Every other model reads it out&lt;br&gt;
loud. I nearly shipped an announcer who introduces himself by saying the words&lt;br&gt;
"shouting over a stadium crowd".&lt;/p&gt;

&lt;p&gt;So the live voice is &lt;code&gt;eleven_flash_v2_5&lt;/code&gt;, which comes back in under a second and&lt;br&gt;
lands on the beat, and the pre-rendered stingers stay on &lt;code&gt;eleven_v3&lt;/code&gt;, where&lt;br&gt;
latency does not exist because they are generated offline.&lt;/p&gt;

&lt;h3&gt;
  
  
  The colours were a lie
&lt;/h3&gt;

&lt;p&gt;The nations carry their real kit colours. Germany's is &lt;code&gt;#111111&lt;/code&gt;. New Zealand's&lt;br&gt;
is &lt;code&gt;#0A0A0A&lt;/code&gt;. The whole design floods a near-black page with the nation's&lt;br&gt;
colour, so picking Germany flooded near-black with near-black and the giant word&lt;br&gt;
GERMANY simply was not there.&lt;/p&gt;

&lt;p&gt;Eight of the forty one nations failed to clear 4.5:1 against the background. The&lt;br&gt;
fix was not to fake the colours: it was to keep the authentic one as the source&lt;br&gt;
of truth and derive the painted one from it, lifting it toward white only as far&lt;br&gt;
as it has to go to be legible. Brazil's yellow comes out untouched. Germany's&lt;br&gt;
black comes out a graphite you can actually read.&lt;/p&gt;

&lt;h3&gt;
  
  
  The design has one idea
&lt;/h3&gt;

&lt;p&gt;The landing page has no colour in it. Near black, white type, a faint green glow&lt;br&gt;
where the pitch would be. The only colour anywhere is a slow ribbon of all 41&lt;br&gt;
nations drifting past the bottom, every side you could end up on, none of them&lt;br&gt;
yours.&lt;/p&gt;

&lt;p&gt;Then you pick, and the reveal floods the entire screen in your nation's colours.&lt;/p&gt;

&lt;p&gt;That is the whole visual argument, and it is the product in one gesture: you have&lt;br&gt;
nothing, and then you have something. Everything on the reveal is driven off the&lt;br&gt;
chosen nation's real kit colours through CSS custom properties, so Morocco is&lt;br&gt;
red, the Netherlands is orange, Argentina is that pale sky blue.&lt;/p&gt;

&lt;p&gt;Flags are images rather than emoji, incidentally, because Windows does not render&lt;br&gt;
flag emoji at all. It falls back to letter pairs. A judge on a Windows laptop&lt;br&gt;
would have seen "AR" where Argentina's flag should be.&lt;/p&gt;




&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Best use of Google AI.&lt;/strong&gt; Gemini is the entire product, not a feature of it. It&lt;br&gt;
runs twice: once grounded in Google Search to establish real, current, checkable&lt;br&gt;
facts, and once under a forced &lt;code&gt;responseSchema&lt;/code&gt; to turn those facts into an&lt;br&gt;
argument. The app shows you the pages it read. It knows results from a tournament&lt;br&gt;
that is still being played.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Best use of ElevenLabs.&lt;/strong&gt; The swear-in is the emotional payoff of the whole&lt;br&gt;
app, and it is built out of two ElevenLabs APIs. Text to speech is the&lt;br&gt;
announcer: Gemini writes the words for that exact matchup, ElevenLabs is the&lt;br&gt;
voice that makes them land, timed to arrive just behind the nation's name.&lt;br&gt;
Sound effects are the stadium he is shouting into: a restless crowd while the&lt;br&gt;
app is thinking, an eruption when the verdict drops. Turn your sound on.&lt;/p&gt;




&lt;h2&gt;
  
  
  Keeping it standing
&lt;/h2&gt;

&lt;p&gt;The Google AI Studio free tier caps a &lt;strong&gt;project&lt;/strong&gt; at a few hundred requests a&lt;br&gt;
day, across every visitor combined. Two calls per reveal means this app could&lt;br&gt;
serve about 125 people a day in total before handing everyone else an error. Fine&lt;br&gt;
for building. Useless for being read.&lt;/p&gt;

&lt;p&gt;So two things. The app runs on Vertex AI, which needs a billing account and&lt;br&gt;
therefore carries real quota. And the well known matchups are generated ahead of&lt;br&gt;
time by the same grounded pipeline, sources and all, and committed to the repo.&lt;br&gt;
Those answer in half a second and cost nothing. They are cached, not faked.&lt;/p&gt;

&lt;p&gt;When the budget does run out, the failure hands you a matchup that always works&lt;br&gt;
instead of a dead end. I would rather you get a team from a slightly stale cache&lt;br&gt;
than a stack trace.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I did not solve
&lt;/h2&gt;

&lt;p&gt;The nations list is curated to 41 sides, so you cannot pick your local club. A&lt;br&gt;
matchup nobody has asked for before takes about twelve seconds, which I dressed&lt;br&gt;
up as a scouting sequence rather than fixed. And the "what is at stake" text goes&lt;br&gt;
stale six hours after it is written, which is a strange thing to have to think&lt;br&gt;
about, and only a problem because the tournament is still being played while you&lt;br&gt;
read this.&lt;/p&gt;

&lt;h2&gt;
  
  
  One last thing
&lt;/h2&gt;

&lt;p&gt;I should probably admit that I am the target user.&lt;/p&gt;

&lt;p&gt;The World Cup is being played in my country. Everyone I know has lost their mind&lt;br&gt;
over it, and for the whole tournament I have been standing in the middle of that&lt;br&gt;
with nothing: no nation, no shirt, no grudge, nobody to shout at. I did not build&lt;br&gt;
this as a demo of an API. I built it because I wanted in and could not find the&lt;br&gt;
door.&lt;/p&gt;

&lt;p&gt;I pointed it at Uruguay and Ghana, because I did not know anything about either&lt;br&gt;
of them.&lt;/p&gt;

&lt;p&gt;It gave me Ghana.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8q2pm76sjig3ua20sb24.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8q2pm76sjig3ua20sb24.png" alt="The share card in Ghana's green and gold. It reads: " width="800" height="521"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I wrote up the longer version of the build, including the two Gemini traps that&lt;br&gt;
nearly shipped, &lt;a href="https://jonathanandrei.com/blog/pick-your-side-manufactured-passion-gemini-elevenlabs/" rel="noopener noreferrer"&gt;on my own site&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Built by &lt;a href="https://jonathanandrei.com" rel="noopener noreferrer"&gt;Jonathan Andrei&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>weekendchallenge</category>
      <category>ai</category>
      <category>showdev</category>
    </item>
    <item>
      <title>I built an AI agent that runs a local business's weekly marketing (Next.js + Gemini)</title>
      <dc:creator>JonathanSolvesProblems</dc:creator>
      <pubDate>Sat, 11 Jul 2026 13:59:24 +0000</pubDate>
      <link>https://dev.to/jonathansolvesstuff/i-built-an-ai-agent-that-runs-a-local-businesss-weekly-marketing-nextjs-gemini-2i91</link>
      <guid>https://dev.to/jonathansolvesstuff/i-built-an-ai-agent-that-runs-a-local-businesss-weekly-marketing-nextjs-gemini-2i91</guid>
      <description>&lt;p&gt;For the last couple of weeks I've been building &lt;strong&gt;Bloom&lt;/strong&gt;, an AI marketing agent for local businesses like cafes, salons, and gyms. You set it up once, and every week an agent decides what to feature, writes the content in your brand voice, checks its own work, and (on the paid tier) emails your newsletter to your subscribers. On its own.&lt;/p&gt;

&lt;p&gt;It's an early work in progress and I'm sharing it to get honest feedback.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;90-second demo:&lt;/strong&gt; &lt;a href="https://www.youtube.com/watch?v=huHCDcUhIGM" rel="noopener noreferrer"&gt;https://www.youtube.com/watch?v=huHCDcUhIGM&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/huHCDcUhIGM"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;Most local owners get under an hour a day for marketing, so it slips, and posting irregularly quietly costs them reach. It's a time-and-consistency problem, not a talent one. I wanted to build the thing that just never skips a week.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;p&gt;You enter your business once (type, city, brand voice). Then every Monday a scheduled agent runs with no human trigger and:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;decides the week's angle and promotion&lt;/li&gt;
&lt;li&gt;writes 3 social posts and an email newsletter in your voice&lt;/li&gt;
&lt;li&gt;scores its own draft 0-100 and rewrites it if it falls below the bar&lt;/li&gt;
&lt;li&gt;on the Pro tier, emails the newsletter to your subscribers from a verified domain&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every decision is logged to a public activity feed, so you can actually watch it work.&lt;/p&gt;

&lt;h2&gt;
  
  
  The parts that were actually interesting to build
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The self-QA gate is real, not decorative.&lt;/strong&gt; After generating, a second Gemini call reviews the draft and scores it. Below the threshold, the agent rewrites once and keeps the better attempt. Making this reliable meant forcing a JSON response schema, because Gemini would occasionally return an array instead of an object and silently break the gate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Structured generation via Vertex AI.&lt;/strong&gt; Content is Gemini 2.5 Flash through Vertex AI with a forced response schema, so the output is always a clean object.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Idempotent weekly delivery.&lt;/strong&gt; The sender claims each send atomically, so a newsletter can never go out twice even if two workers fire the same Monday.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runs on a plain Docker/Traefik box&lt;/strong&gt; as a standalone Next.js server, with a cron sidecar driving the weekly run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Stack: Next.js, TypeScript, Gemini 2.5 Flash (Vertex AI), Prisma + Neon, Stripe, Resend, Docker + Traefik.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it's at (honestly)
&lt;/h2&gt;

&lt;p&gt;It's live. I've run a real payment through it and had it generate and deliver a real newsletter end to end. But it has basically no real users yet, which is exactly why I'm posting. I'd rather hear what's wrong now.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it, and tell me what's missing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Free preview (no card, about a minute):&lt;/strong&gt; &lt;a href="https://bloom.jonathanandrei.com" rel="noopener noreferrer"&gt;https://bloom.jonathanandrei.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I'd genuinely love feedback on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Would you, or a local business you know, actually use this?&lt;/li&gt;
&lt;li&gt;Does the generated content look good enough to post as-is?&lt;/li&gt;
&lt;li&gt;What would need to be true before you'd pay for it?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Built for the Build with Gemini XPRIZE hackathon. Happy to answer anything in the comments.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>showdev</category>
      <category>nextjs</category>
      <category>startup</category>
    </item>
  </channel>
</rss>
