<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Christian Anderson</title>
    <description>The latest articles on DEV Community by Christian Anderson (@c1-anderson).</description>
    <link>https://dev.to/c1-anderson</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4110779%2Faa18d0ea-793c-4b09-86d0-34a917483d1a.jpg</url>
      <title>DEV Community: Christian Anderson</title>
      <link>https://dev.to/c1-anderson</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/c1-anderson"/>
    <language>en</language>
    <item>
      <title>Your Spotify playlist reader isn't broken. It's reading a boolean.</title>
      <dc:creator>Christian Anderson</dc:creator>
      <pubDate>Sat, 05 Sep 2026 07:23:16 +0000</pubDate>
      <link>https://dev.to/c1-anderson/your-spotify-playlist-reader-isnt-broken-its-reading-a-boolean-2lml</link>
      <guid>https://dev.to/c1-anderson/your-spotify-playlist-reader-isnt-broken-its-reading-a-boolean-2lml</guid>
      <description>&lt;p&gt;If you have written anything against the Spotify Web API in the last two years, some of&lt;br&gt;
your code is quietly wrong right now — and the failure mode is not an exception. It is a&lt;br&gt;
function that returns &lt;code&gt;[]&lt;/code&gt; and looks fine.&lt;/p&gt;

&lt;p&gt;I hit this building two headless tools against the API. Everything below was measured&lt;br&gt;
against the live API, not read in a changelog.&lt;/p&gt;
&lt;h2&gt;
  
  
  First: there were two events, and everyone conflates them
&lt;/h2&gt;

&lt;p&gt;Almost every "Spotify API is broken" post on the internet blames one date for both. I did&lt;br&gt;
this myself, in a README, and had to correct it before publishing. They are unrelated:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;27 November 2024 — removals.&lt;/strong&gt; &lt;code&gt;/recommendations&lt;/code&gt;, &lt;code&gt;/audio-features&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;/audio-analysis&lt;/code&gt;, &lt;code&gt;/related-artists&lt;/code&gt;, and the 30-second preview URLs. This is when the&lt;br&gt;
&lt;em&gt;data&lt;/em&gt; died. Every "build a playlist generator with audio features" tutorial dates from&lt;br&gt;
before this and cannot work.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;February 2026 — renames and trims.&lt;/strong&gt; Nothing was deleted for good reasons here; things&lt;br&gt;
&lt;em&gt;moved&lt;/em&gt;, and limits tightened. This is when working code started failing without saying so.&lt;/p&gt;

&lt;p&gt;If you are debugging a missing &lt;code&gt;energy&lt;/code&gt; or &lt;code&gt;tempo&lt;/code&gt; field, that is the 2024 event and it is&lt;br&gt;
never coming back. If you are debugging an endpoint that used to work last year, read on.&lt;/p&gt;
&lt;h2&gt;
  
  
  The rename that costs you an afternoon
&lt;/h2&gt;

&lt;p&gt;The paths moved:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Was&lt;/th&gt;
&lt;th&gt;Now&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET /playlists/{id}/tracks&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /playlists/{id}/items&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET /users/{id}/playlists&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /me/playlists&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;library operations&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/me/library&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That part is annoying but loud — you get an error, you fix the URL, you move on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The part that isn't loud: the row field renamed too.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Inside each row of the response, &lt;code&gt;item["track"]&lt;/code&gt; became &lt;code&gt;item["item"]&lt;/code&gt;. And here is the&lt;br&gt;
bit that makes it expensive:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;"track"&lt;/code&gt; did not disappear. It survives as a &lt;strong&gt;boolean&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So this code does not crash:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# looks fine, has always worked, now returns nothing
&lt;/span&gt;&lt;span class="n"&gt;tracks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;track&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;   &lt;span class="c1"&gt;# &amp;lt;-- t is now `True`
&lt;/span&gt;        &lt;span class="k"&gt;continue&lt;/span&gt;                    &lt;span class="c1"&gt;#     True.get() never runs; `not True` is False...
&lt;/span&gt;    &lt;span class="n"&gt;tracks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;          &lt;span class="c1"&gt;#     ...so this raises, or your lenient
&lt;/span&gt;                                    &lt;span class="c1"&gt;#     version skips every row silently
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depending on how defensive your reader is, you get one of two outcomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a strict reader raises &lt;code&gt;AttributeError: 'bool' object has no attribute 'get'&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;a lenient reader — the common case, because everyone wraps this in a &lt;code&gt;try&lt;/code&gt; — concludes
&lt;strong&gt;the playlist is empty&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An empty playlist is not an error. Your sync job runs, reports success, and writes&lt;br&gt;
nothing. That is the whole problem with this particular change: it converts a rename into&lt;br&gt;
a silent data loss.&lt;/p&gt;

&lt;p&gt;The fix is trivial once you know:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;item&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;track&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;continue&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Artist genres are gone if you are in Development Mode
&lt;/h2&gt;

&lt;p&gt;This one has no announcement I could find, and it is absolute:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;batch &lt;code&gt;GET /artists?ids=...&lt;/code&gt; returns &lt;strong&gt;403&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;single &lt;code&gt;GET /artists/{id}&lt;/code&gt; returns an artist object with &lt;strong&gt;no &lt;code&gt;genres&lt;/code&gt; key at all&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Note that second one carefully. It is not &lt;code&gt;"genres": []&lt;/code&gt;. The field is &lt;em&gt;absent&lt;/em&gt;. So a&lt;br&gt;
&lt;code&gt;.get("genres", [])&lt;/code&gt; gives you an empty list and you tag every artist &lt;code&gt;unknown&lt;/code&gt; without&lt;br&gt;
ever noticing the capability is gone.&lt;/p&gt;

&lt;p&gt;If you were tagging or grouping by genre, that is dead for dev-mode apps. In my case the&lt;br&gt;
only tag still derivable from the remaining metadata was release decade, so decade became&lt;br&gt;
the default and genre tagging now raises an explicit error rather than silently labelling&lt;br&gt;
the whole library &lt;code&gt;unknown&lt;/code&gt;. &lt;strong&gt;Fail loudly on a capability you no longer have.&lt;/strong&gt; Do not&lt;br&gt;
let it degrade into a plausible-looking result.&lt;/p&gt;
&lt;h2&gt;
  
  
  The limits also moved, quietly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;search &lt;code&gt;limit&lt;/code&gt; is capped at &lt;strong&gt;10&lt;/strong&gt;. It was 50. Requests for more do not error — you just
get 10, so pagination logic built on 50 silently under-fetches.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;popularity&lt;/code&gt; is stripped from responses.&lt;/li&gt;
&lt;li&gt;Premium is now required for playback.&lt;/li&gt;
&lt;li&gt;Development Mode user allowlists dropped from &lt;strong&gt;25 to 5&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one is worth dwelling on if you were planning to ship something. Between a&lt;br&gt;
5-user allowlist, Premium being mandatory, and Extended Quota gated behind 250k MAU,&lt;br&gt;
there is no path from "weekend project" to "product" here any more. Self-hosted script is&lt;br&gt;
the realistic ceiling. Better to know that before you build a billing page.&lt;/p&gt;
&lt;h2&gt;
  
  
  403 means "removed", not "you lack a scope"
&lt;/h2&gt;

&lt;p&gt;Removed endpoints answer &lt;strong&gt;403, not 404&lt;/strong&gt;. This one is already documented in several&lt;br&gt;
places, so I will keep it short — but the practical rule saves real time:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Before you go re-reading the scopes documentation, check whether the &lt;strong&gt;path&lt;/strong&gt; still&lt;br&gt;
exists. A 403 on the Spotify API is far more often a dead endpoint than a permissions&lt;br&gt;
problem.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;
  
  
  The one that actually hurt: the quota window re-arms while you are throttled
&lt;/h2&gt;

&lt;p&gt;This is the finding I would most want to have known in advance.&lt;/p&gt;

&lt;p&gt;When you hit the daily quota you get a 429 with a &lt;code&gt;Retry-After&lt;/code&gt;. I measured one at&lt;br&gt;
&lt;strong&gt;82,510 seconds&lt;/strong&gt; — roughly 23 hours. Fine. But then I kept making calls, and watched&lt;br&gt;
&lt;code&gt;Retry-After&lt;/code&gt; grow &lt;strong&gt;back toward 24 hours&lt;/strong&gt; instead of counting down.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A client that retries in a loop can pin your app in a permanent 429 it never escapes.&lt;/strong&gt;&lt;br&gt;
Every retry pushes the window out again. The naive exponential-backoff loop you would&lt;br&gt;
normally reach for is exactly the wrong tool.&lt;/p&gt;

&lt;p&gt;What works is a circuit breaker that &lt;em&gt;stops calling entirely&lt;/em&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# on 429: record an ABSOLUTE wake-up time and make no further calls until then
&lt;/span&gt;&lt;span class="n"&gt;blocked_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Retry-After&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;blocked_until&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# persist it — see below
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two traps in that ten-line fix, both of which I shipped wrong the first time:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Persist it, or a restart undoes it.&lt;/strong&gt; If the process restarts it will happily call
Spotify on boot and re-arm the window.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Persist the absolute timestamp, not the remaining seconds.&lt;/strong&gt; I wrote an absolute
epoch on save and read back &lt;em&gt;remaining seconds&lt;/em&gt; on load. So a restored value of &lt;code&gt;3600&lt;/code&gt;
was interpreted as an epoch in 1970, the breaker computed a negative remaining time,
concluded it was unthrottled, and called Spotify immediately — reproducing the exact
bug it existed to prevent. The round trip has to agree with itself.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Also worth saying plainly: what exhausted my quota was &lt;strong&gt;bulk harvesting&lt;/strong&gt; — paginating&lt;br&gt;
120 deep across 40+ queries — not normal use. The architectural answer was to stop calling&lt;br&gt;
the API on the hot path at all. A background thread tops up a local index inside a&lt;br&gt;
token-bucket budget, and the user-facing action reads only from that index. It now serves&lt;br&gt;
from 6,559 locally indexed tracks and makes &lt;strong&gt;zero&lt;/strong&gt; API calls in the interaction path.&lt;/p&gt;

&lt;p&gt;If your app calls Spotify in response to a user action, that is the design to move away&lt;br&gt;
from.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Two separate events. 2024 removed the audio data; 2026 moved things and tightened limits.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;row["track"]&lt;/code&gt; → &lt;code&gt;row["item"]&lt;/code&gt;, and &lt;code&gt;track&lt;/code&gt; is now a &lt;strong&gt;boolean&lt;/strong&gt; that will convince a
tolerant reader your playlist is empty.&lt;/li&gt;
&lt;li&gt;Artist &lt;code&gt;genres&lt;/code&gt; is &lt;em&gt;absent&lt;/em&gt;, not empty, for dev-mode apps.&lt;/li&gt;
&lt;li&gt;Search &lt;code&gt;limit&lt;/code&gt; silently caps at 10.&lt;/li&gt;
&lt;li&gt;403 usually means the path is gone, not that your scopes are wrong.&lt;/li&gt;
&lt;li&gt;Never retry a 429 in a loop. The window re-arms. Break the circuit and persist an
absolute timestamp.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;The two tools this came out of are MIT and on GitHub:&lt;br&gt;
&lt;a href="https://github.com/casareanderson/setlisted" rel="noopener noreferrer"&gt;setlisted&lt;/a&gt; (headless playlist sequencing —&lt;br&gt;
its test suite is fully offline, because the daily quota is genuinely exhaustible) and&lt;br&gt;
&lt;a href="https://github.com/casareanderson/spin-that-dice" rel="noopener noreferrer"&gt;spin-that-dice&lt;/a&gt; (the local-index&lt;br&gt;
design described above).&lt;/p&gt;

&lt;p&gt;I also wrote the whole thing up properly as a&lt;br&gt;
&lt;a href="https://asareanderson.gumroad.com/l/taeoza" rel="noopener noreferrer"&gt;6-page PDF&lt;/a&gt;, with a&lt;br&gt;
&lt;a href="https://asareanderson.gumroad.com/l/yigsxw" rel="noopener noreferrer"&gt;2-page cheat sheet&lt;/a&gt; of just the renames if&lt;br&gt;
that is all you need. Everything technical above is in the free repos too — the PDF just&lt;br&gt;
saves you assembling it.&lt;/p&gt;

</description>
      <category>api</category>
      <category>programming</category>
      <category>software</category>
    </item>
    <item>
      <title>A dead GPU degrades gracefully. A working one crash-loops.</title>
      <dc:creator>Christian Anderson</dc:creator>
      <pubDate>Sat, 05 Sep 2026 07:17:32 +0000</pubDate>
      <link>https://dev.to/c1-anderson/a-dead-gpu-degrades-gracefully-a-working-one-crash-loops-bdj</link>
      <guid>https://dev.to/c1-anderson/a-dead-gpu-degrades-gracefully-a-working-one-crash-loops-bdj</guid>
      <description>&lt;p&gt;I put a used £30 Quadro P600 on a riser into a ZimaBlade, to accelerate machine&lt;br&gt;
learning in Immich. It took months of intermittent debugging to get the PCIe link to&lt;br&gt;
train.&lt;/p&gt;

&lt;p&gt;Then I found out that fixing it would have broken the thing I was trying to speed up.&lt;/p&gt;

&lt;p&gt;This is the write-up of both halves: the diagnostic method, which is reusable, and the&lt;br&gt;
research I should have done first, which is the actual lesson.&lt;/p&gt;
&lt;h2&gt;
  
  
  The constraints nobody prints on the box
&lt;/h2&gt;

&lt;p&gt;The ZimaBlade has a PCIe slot. What it does not have is a slot you can just use. From&lt;br&gt;
&lt;code&gt;lspci -vv&lt;/code&gt; on the root port:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;LnkCap:  Speed 5GT/s, Width x4        # PCIe 2.0 x4, ~2GB/s ceiling
SltCap:  PowerLimit 25W               # slot budget
         HotPlug-                     # enumeration only at cold boot
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things fall straight out of that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;25W slot budget.&lt;/strong&gt; The P600 is a 40W card with no external power connector. It
physically cannot be slot-powered here. You need a riser with its own 12V feed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No hot-plug.&lt;/strong&gt; &lt;code&gt;echo 1 &amp;gt; /sys/bus/pci/rescan&lt;/code&gt; is theatre. If the card wasn't there at
cold boot, it isn't there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No Above-4G decoding.&lt;/strong&gt; Every PCI window is sub-4GB, the largest about 1.25GB. Fine
for a 2GB card with a ~256MB BAR aperture; a hard blocker for anything with real VRAM.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Also worth stating plainly, because the mining-riser marketing obscures it: a &lt;strong&gt;VER009S&lt;br&gt;
riser is electrically x1&lt;/strong&gt;, despite having an x16 slot on it. The USB3 cable is just the&lt;br&gt;
physical medium, not the USB protocol. So the ceiling was PCIe 2.0 x1, roughly 500MB/s —&lt;br&gt;
before anything went wrong.&lt;/p&gt;

&lt;p&gt;And the stock ZimaBlade PSU is &lt;strong&gt;12V/3A, 36W&lt;/strong&gt; over USB-C. Board, RAM and two SATA disks&lt;br&gt;
already eat 25–30W of that. There was never a version of this where the card ran off the&lt;br&gt;
onboard supply.&lt;/p&gt;
&lt;h2&gt;
  
  
  The diagnostic worth stealing: presence detect
&lt;/h2&gt;

&lt;p&gt;For a long time the link simply never trained — &lt;code&gt;LnkSta: Width x0&lt;/code&gt;. The question that&lt;br&gt;
matters when you have a card, a riser, a cable, a second PSU and a slot all in series is:&lt;br&gt;
&lt;strong&gt;which end is at fault?&lt;/strong&gt; You do not want to be swapping parts at random in a loft.&lt;/p&gt;

&lt;p&gt;PCIe gives you a free answer. On every card, pins &lt;strong&gt;A1 (&lt;code&gt;PRSNT1#&lt;/code&gt;) and B17 (&lt;code&gt;PRSNT2#&lt;/code&gt;)&lt;br&gt;
are shorted together on the card's own PCB&lt;/strong&gt;. The host grounds A1 and pulls B17 high.&lt;br&gt;
Seat a card, B17 gets pulled low through that short, and the slot reports &lt;code&gt;PresDet+&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The useful property: this is pure DC continuity. It does not involve the data lanes, the&lt;br&gt;
cable, the GPU, or the external supply. So:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SltSta: PresDet-   →  nothing is making contact in the slot. Stop.
                      Cable and ground theories are downstream and cannot be the blocker.
SltSta: PresDet+   →  the board is seated. The fault is further out.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mine was flapping &lt;code&gt;+&lt;/code&gt;/&lt;code&gt;-&lt;/code&gt;, then went steady &lt;code&gt;-&lt;/code&gt; — which told me the riser board was not&lt;br&gt;
seated, and saved me from chasing driver and cable fixes for a mechanical problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I then over-read my own test, and it is worth admitting because it is the more useful&lt;br&gt;
half.&lt;/strong&gt; After reseating, &lt;code&gt;PresDet&lt;/code&gt; read a solid &lt;code&gt;+&lt;/code&gt; ten times running — and I briefly took&lt;br&gt;
that as validating the whole chain. It doesn't. That A1↔B17 short is on the x1 board and&lt;br&gt;
shorts two of the &lt;em&gt;host's own pins inside the host's own ground domain&lt;/em&gt;. It reads &lt;code&gt;+&lt;/code&gt; with&lt;br&gt;
no cable and no GPU attached at all. It proves exactly one thing: that small board is in&lt;br&gt;
the slot.&lt;/p&gt;

&lt;p&gt;A test that can only pass tells you nothing. Know the narrowest claim your evidence&lt;br&gt;
supports.&lt;/p&gt;

&lt;p&gt;Two related rules that saved time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Driver ≠ detection.&lt;/strong&gt; &lt;code&gt;lspci&lt;/code&gt; enumerates with no driver loaded; a driver only binds to
an already-enumerated device. No driver install can make an undetected card appear.
Don't debug drivers while the width is 0.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A spinning fan proves 12V and nothing else.&lt;/strong&gt; It says nothing about data lanes.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  It worked. Then it died 8.5 hours later.
&lt;/h2&gt;

&lt;p&gt;The physical work eventually paid off. The card enumerated, the link trained, the&lt;br&gt;
proprietary driver bound cleanly at boot, all four &lt;code&gt;/dev/nvidia*&lt;/code&gt; nodes appeared.&lt;/p&gt;

&lt;p&gt;Then, 30,709 seconds after boot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from
      0x0 (None) to 0x1 (GPU Reset Required)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Xid 79 is textbook power-delivery or PCIe signal integrity.&lt;/strong&gt; The corroborating evidence&lt;br&gt;
was already in &lt;code&gt;lspci&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;LnkSta:  Speed 2.5GT/s (downgraded), Width x1 (downgraded)
DevSta:  CorrErr+ NonFatalErr+ UnsupReq+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Negotiated down to PCIe &lt;strong&gt;1.0&lt;/strong&gt; x1 — below even the riser's own x1-at-5GT/s ceiling — and&lt;br&gt;
the root port actively logging correctable errors. A link that trains but negotiates &lt;em&gt;below&lt;br&gt;
the worst thing in the chain&lt;/em&gt; is telling you the physical layer is marginal, not that it&lt;br&gt;
works.&lt;/p&gt;

&lt;p&gt;The prime suspect was a ground offset: the ZimaBlade on a floating two-pin 12V brick, the&lt;br&gt;
GPU on an earthed ATX supply, the two domains bonded only through the thin ground wires in&lt;br&gt;
a USB3 cable. PCIe receivers have a narrow common-mode range. That single theory predicts&lt;br&gt;
all three observations — presence detect fine (internal domain), fan spinning (ATX 12V&lt;br&gt;
present), lanes marginal (cross-domain signalling). Mining rigs solve this with tied PSU&lt;br&gt;
grounds for exactly this reason.&lt;/p&gt;

&lt;p&gt;One correction I had to publish against my own earlier notes: &lt;strong&gt;the ZimaBlade's power input&lt;br&gt;
is USB-C, not a barrel jack.&lt;/strong&gt; Every "pigtail it off a Molex" instruction I had written was&lt;br&gt;
wrong. And be aware the SATA-to-6-pin adapters that ship in VER009S kits are a genuine fire&lt;br&gt;
hazard — use proper Molex or PCIe leads.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that makes the whole project pointless
&lt;/h2&gt;

&lt;p&gt;Before re-energising the card, I finally went and read what Immich v3 actually supports.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Immich v3 dropped machine-learning support for Maxwell and Pascal.&lt;/strong&gt; ONNX Runtime no&lt;br&gt;
longer ships kernels for compute capability 5.2 / 6.1. The P600 is Pascal GP107, compute&lt;br&gt;
&lt;strong&gt;6.1&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The failure mode is the whole point:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The &lt;code&gt;v3.x-cuda&lt;/code&gt; ML container does not print a helpful message and fall back. It dies on&lt;br&gt;
&lt;strong&gt;exit 132 — SIGILL&lt;/strong&gt;, an illegal instruction, and Docker restarts it forever.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Now compare that to the situation I was trying to escape. With the GPU dead, the container&lt;br&gt;
logs &lt;code&gt;CUDA failure 100: no CUDA-capable device is detected&lt;/code&gt;, falls back to&lt;br&gt;
&lt;code&gt;CPUExecutionProvider&lt;/code&gt;, and smart search and face recognition keep working — just slowly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The broken state degraded gracefully. The fixed state would have crash-looped.&lt;/strong&gt; Every&lt;br&gt;
hour I spent on that riser was moving the system from "unaccelerated but working" to "down".&lt;/p&gt;

&lt;p&gt;The cliff isn't only Immich, either. Driver branch &lt;strong&gt;580 is the last one supporting&lt;br&gt;
Maxwell, Pascal and Volta&lt;/strong&gt;. cuDNN dropped them at &lt;strong&gt;9.11&lt;/strong&gt; — 9.10.2 is the last usable&lt;br&gt;
release. Three separate vendors retired that generation inside about a year.&lt;/p&gt;

&lt;p&gt;That is the real reason a £30 used Quadro is a worse deal than the price suggests. You are&lt;br&gt;
not buying a slow GPU. You are buying a component with an announced end date on every&lt;br&gt;
software layer above it, and you find out by reading release notes, not benchmarks.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell myself at the start
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Check the software cliff before the hardware.&lt;/strong&gt; One afternoon reading ONNX Runtime and
driver release notes would have cancelled the project before I bought the card. Hardware
debugging is absorbing and feels like progress; compatibility research is boring and
decides whether the work is worth doing at all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graceful degradation can invert.&lt;/strong&gt; "It's broken but working around it" is sometimes a
&lt;em&gt;better&lt;/em&gt; state than the fix. Ask what happens after success before you chase it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Know the narrowest claim your evidence supports&lt;/strong&gt; — my presence-detect test was sound
and I still over-read it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A link that negotiates below the worst component in the chain is a physical-layer
warning&lt;/strong&gt;, not a success.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The honest ending: the card is still dead, and I am no longer in a hurry to fix it. If I&lt;br&gt;
do, it will be pinned to the last ML image that supports the architecture, or used purely&lt;br&gt;
for NVENC transcoding — a fixed-function encoder, unaffected by any CUDA kernel cliff.&lt;/p&gt;




&lt;p&gt;The full write-up, with the complete &lt;code&gt;lspci&lt;/code&gt; output, the diagnostic decision tree and a&lt;br&gt;
read-only script that runs every check above in one pass, is free and MIT on GitHub:&lt;br&gt;
&lt;a href="https://github.com/casareanderson/zimablade-gpu-immich" rel="noopener noreferrer"&gt;casareanderson/zimablade-gpu-immich&lt;/a&gt;.&lt;br&gt;
There is a &lt;a href="https://asareanderson.gumroad.com/l/jthell" rel="noopener noreferrer"&gt;packaged copy on Gumroad&lt;/a&gt; if you'd&lt;br&gt;
rather have it as one download — pay what you want, including nothing.&lt;/p&gt;

&lt;p&gt;Every command output quoted here was captured from the running machine. The build is&lt;br&gt;
documented at the point where the fault is understood and the cable has not yet been&lt;br&gt;
replaced.&lt;/p&gt;

</description>
      <category>homelab</category>
      <category>hardware</category>
      <category>selfhosted</category>
      <category>debugging</category>
    </item>
  </channel>
</rss>
