DEV Community: Will Kline

RTX 5090 survival guide: sm_120, CUDA 12 and 13 side by side, and the xformers trap

Will Kline — Tue, 28 Jul 2026 15:30:13 +0000

Everything on this machine works now. Six standalone image and video apps, a LoRA trainer, an ONNX-based transcriber, all sharing one RTX 5090 on Windows 11, all starting and stopping cleanly from bat files. Getting there meant a full uninstall of one CUDA toolkit, a second toolkit installed beside the one I kept, a manual PATH surgery the cuDNN installer should have done itself, and one library that silently downgraded my torch build the moment I installed it.
If you just bought a 5090 for local AI work, this is the writeup I wanted and couldn't find. Nothing here is theory. Every path and version number below is running on my desk right now.

The one fact that drives everything: sm_120

The 5090 is Blackwell, compute capability sm_120. Older PyTorch wheels don't carry kernels for it. Install a wheel built for an earlier CUDA target and you get the classic failure: torch imports fine, CUDA reports available, and then the first real kernel launch dies with a no-kernel-image error. The card is fine. The wheel just doesn't know your architecture exists.
The rule: PyTorch built against cu128 or cu130. Nothing older. On my box the working pairs are torch 2.11.0+cu130 and torch 2.10.0+cu130, depending on the app. When you install, spell the index URL out explicitly and check the suffix on the installed version string afterward. "pip show torch" reporting a version without a +cu128 or +cu130 suffix means you got a CPU wheel or a stale CUDA build, and you will find out the hard way at first inference.
Check this before anything else. Half the mystery crashes people report on new-architecture cards are just this one mismatch wearing different costumes.

The xformers trap

This is the one that costs people an afternoon, because the failure arrives disguised as success.
You stand up ComfyUI or any diffusion stack, torch 2.11.0+cu130 is installed and working, generation runs. Then you install xformers for the memory-efficient attention, the way every tutorial from the last three years tells you to. Pip resolves dependencies, decides your torch is incompatible with the xformers wheel it selected, and quietly replaces your torch with an older build that has no sm_120 kernels. No warning that matters, just a long pip log most people don't read. Next launch, the app that worked an hour ago crashes at the first sampling step.
My fix was to stop using xformers entirely on this card. SageAttention 2.2.0 covers the attention optimization without touching the torch install, and it behaves on Blackwell. The rule I now follow on every app: after installing anything attention-related, run "pip show torch" again and confirm the version string did not move. If a package manager can change your torch build as a side effect, treat every install as a suspect until proven otherwise.

CUDA 12.9 and 13.2, side by side, on purpose

Here is the part nobody tells you: one CUDA toolkit is not enough, because your apps do not agree on a major version.
PyTorch cu130 builds want the CUDA 13 runtime. But ONNX Runtime with the CUDA execution provider (version 1.24 in my case, driving a speech-to-text app) loads CUDA 12 DLLs by name. And the names are version-suffixed: it asks the OS loader for cudart64_12.dll, cublas64_12.dll, cufft64_11.dll. A CUDA 13 install gives you cudart64_13.dll. The loader will not squint and accept a neighbor. The app just reports that CUDA is unavailable and falls back to CPU, or refuses to start.
So this machine runs both toolkits, installed side by side:

CUDA 13.2 at "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.2" is primary. CUDA_PATH points here, and everything torch-based resolves against it.- CUDA 12.9.1 at "...\CUDA\v12.9" exists purely to serve the version-suffixed CUDA 12 DLLs to ONNX Runtime and anything else still living on the 12.x runtime.One sharp edge inside that second install: the 12.9 DLLs sit in "bin" directly, not in a "bin\x64" subfolder, which differs from where some tooling expects them. When an app claims a DLL is missing, do not trust your assumptions about the layout. Open the folder and look. The general lesson: before installing any CUDA-dependent app, find out which runtime major version it actually loads. The DLL names tell you. An app hunting for cudart64_12.dll will never be satisfied by a CUDA 13 toolkit, no matter how new and shiny. And keep the count at exactly the versions you need. I briefly had CUDA 13.1 on this box too. Three toolkits meant three sets of PATH entries and env vars fighting over resolution order, so 13.1 got fully removed: directory, PATH entries, environment variables, all of it. Two toolkits with clear jobs beat three with overlapping ones. ## cuDNN: the installer does not finish the job cuDNN 9.20 installs to "C:\Program Files\NVIDIA\CUDNN\v9.20" and, helpfully, ships per-CUDA-version DLL directories: "bin\13.2\x64" for the CUDA 13 side and "bin\12.9\x64" for the CUDA 12 side. That layout is exactly what a dual-toolkit machine needs. What the installer does not do is add either directory to your system PATH. Nothing warns you. Apps that need cudnn64_9.dll simply fail to find it, and the error messages rarely say so plainly. I added both directories to the system PATH by hand, and a whole category of intermittent startup failures ended that day. One related Windows note: system-level environment changes need an elevated shell. My reliable pattern is a small .ps1 script invoked with "Start-Process -Verb RunAs" rather than trying to elevate inline, which fails in quiet and creative ways. ## Compiling extensions: MSVC meets the CCCL headers Sooner or later on a card this new, you compile something from source, because the prebuilt wheel for your exact torch and CUDA combination does not exist yet. On Windows with CUDA 13.2 and MSVC 2019, the CCCL headers require the standards-conforming preprocessor, and the default MSVC preprocessor is not it. The build fails deep in template code with errors that suggest anything except the actual cause. The fix is two flags: add "/Zc:preprocessor" to the C++ compiler args, and "-Xcompiler=/Zc:preprocessor" to the NVCC args so the host-compiler passes get it too. Miss the second one and you get a build that half succeeds, which is worse than one that fails honestly. ## Per-app isolation, or how six apps share one card without sharing dependencies Every GPU app on this machine gets its own environment. No shared site-packages, no global installs, no exceptions. ComfyUI runs a venv on Python 3.13.3 with torch 2.11.0+cu130. The video generation app runs conda with Python 3.11.9 and torch 2.10.0+cu130. The 3D generation app, the training tools, each one isolated the same way. When one app's dependency resolver has a bad day, the blast radius is one folder. Each app carries four bat files: start, stop, status, update. Boring on purpose. The start script points at the environment's python.exe by full path and binds the app to its own port. Which brings up the conda trap that will bite anyone scripting Windows automation: conda environments created this way do not ship an activate.bat. Every tutorial that says "call activate envname" writes a script that dies on this box. The fix is to never activate at all. Call "C:\AppName\env\python.exe" by absolute path and the whole class of activation failure disappears, in scripts, in schedulers, in anything unattended. Above the bat files sits a small FastAPI dashboard that starts, stops, and health-checks the fleet, around 19 managed apps at last count, with a mutex so two heavy jobs cannot claim the card at once. But the dashboard is a convenience. The bat files are the contract, and each one works standalone. ## Process hygiene: dead processes still hold your VRAM The 5090 has 32 GB of VRAM and it is still not enough to waste. The failure mode that teaches you this: you stop an app, or it crashes, and the process dies badly, leaving allocated GPU memory behind. The next app launches, tries to claim memory the ghost still owns, and falls over with an out-of-memory error on a card that looks idle. Two habits fixed it permanently:
Run "nvidia-smi" before starting any GPU work and after stopping any app. Stale python processes holding VRAM get killed before anything else launches.- Hunt zombies by executable path, not by port. A crashed process is not listening on its port anymore, but it is absolutely still holding memory. In PowerShell: "Get-Process python | Where-Object {$.Path -like 'AppName'} | Stop-Process -Force". Port-based detection misses exactly the processes you most need to kill.Two smaller Windows-flavored footguns in the same family. Stale __pycache_ directories can serve old bytecode after you edit a module, producing phantom 404s and behavior from code you already deleted, so when an edit refuses to take effect, delete the pycache folders before doubting your own diff. And when a port refuses to bind, "netstat -ano" with a filter on the port number finds the owning PID faster than any amount of guessing. ## The short version
PyTorch cu128 or cu130 only. Verify the +cu suffix on the installed version, not just that import succeeds.- Skip xformers, it will downgrade your torch underneath you. SageAttention 2.2.0 does the job on Blackwell.- Read the DLL names your apps load. cudart64_12.dll means you need a CUDA 12 toolkit installed beside your CUDA 13 one. The suffix is the contract.- Add the cuDNN DLL directories to PATH yourself. The installer will not.- Compiling with MSVC 2019 and CUDA 13.2: "/Zc:preprocessor" for the compiler, "-Xcompiler=/Zc:preprocessor" for NVCC.- One environment per app, called by absolute python.exe path. Conda on Windows has no activate.bat, stop writing scripts that assume it does.- Trust "nvidia-smi", not your process list. Dead processes hold VRAM, and killing by executable path is the only reliable sweep.None of this is hard once you know it. All of it is invisible until you do. The card itself has been flawless, the ecosystem just needed a few months to catch up to sm_120, and the machine that came out the other side runs a real workload every day. The basics, dialed.

An AI audit found two invisible outages and a security hole in my own SaaS

Will Kline — Tue, 21 Jul 2026 15:30:16 +0000

The dashboard was green and the product was down.
Not all the way down. Worse: down in the two places I wasn't looking, while every check I had wrote green to the wall. This is the postmortem of the day I pointed an agent swarm at my own SaaS and it came back with 38 findings, 32 of which survived human verification, including two live outages nobody had noticed and a security hole sitting on every scheduled job in production.
Some context so the rest makes sense. I run a small social-posting SaaS solo, with an AI-agent workforce doing most of the labor. The engine learns a customer's voice once, watches their industry overnight, drafts platform-aware posts, a human approves in a tap, and it publishes to Facebook and Instagram today, with more platforms rolling out. Customer #1 is a high-end audio dealer whose feed the engine writes every night. Real money, real posts, real permalinks.
Which is exactly why I audited it.

Why audit a product that looks fine

When you build with agents at speed, drift accumulates in the gaps between what you shipped and what you believe you shipped. My monitoring measured the things I had thought to measure, which is a very different set from the things that break. So I ran a sweep: an agent swarm over the codebase and the production state, eight review lenses, each lens a separate agent with its own brief and no knowledge of the others' conclusions. Security. Correctness. Tenancy. The publish path. Cron hygiene. And so on.
The swarm raised 38 findings. I verified every one by hand against production before believing any of them. 32 confirmed, 6 killed as noise. Hold onto that ratio, because it's the honest part: an audit that confirms everything it raises is a flattery machine, and an unverified finding is a hypothesis, not a bug.
Here are the three that mattered.

Outage one: the 3.26-second OAuth race

The finding read like a support ticket from the future: no customer could connect a new social channel. Every attempt died in a way that looked exactly like a popup blocker. The OAuth window opens, the customer authorizes on the platform's consent screen, the window closes, and then nothing. No new channel in the list. No error. Try again, same thing.
The mechanism was ugly in the way only race conditions are ugly. The connect flow mints a one-time claim so the OAuth callback knows which customer initiated the dance. A background job was consuming that claim 3.26 seconds after creation, while the customer was still reading the consent screen. By the time the callback arrived, the claim was gone, and the bind failed silently. No exception, no error-level log line, no failed job anywhere. Just a window that closed and a channel list that didn't change.
No human authorizes an OAuth grant in under 3.26 seconds. Which means the failure rate was 100 percent, deterministically, and the whole thing was invisible because every individual component reported success.
Think about the cost of misdiagnosing this one. It presents as a client-side problem. Popup blocker, third-party cookies, wrong browser. You could burn a week of support cycles telling customers to try Chrome while the actual bug sits in a background job that is doing precisely what it was told to do, at precisely the wrong moment.
The fix: the claim is now consumed exactly once, at the callback, atomically. And there is a counter where there used to be faith: connects started versus channels bound. When those two numbers diverge, I get an email before a customer notices anything.

Outage two: the silent draft famine

The second outage was quieter, which I did not think was possible.
The nightly research stage reads a curated set of industry publications so the composer has real, current material to draft from. The swarm found that those fetches were being blocked at the network level from our cloud egress. Not erroring loudly. Blocked. So the research stage returned nothing, the composer had nothing worth drafting, and drafts stopped landing for a billable account.
And here is the part that earned this finding its own section: the scheduler kept reporting "cadence satisfied" the entire time. The cadence check was counting the wrong thing. It measured whether the pipeline ran, not whether posts landed. The pipeline ran beautifully. It ran on schedule, completed every stage, and produced zero output, and every status surface stayed green while landed sat at zero.
That is my working definition of an invisible outage now: every component succeeds and the system produces nothing.
The fix had two parts. The mechanical part was rerouting the feed fetches so they actually resolve from our infrastructure. The important part was changing what the check believes. The only number that means the pipeline works is landed: a post that a human approved and a platform confirmed live. Not drafted, not scheduled, not accepted-by-the-API. Landed. The cadence check now alarms when landed is zero for any paying account, because a green dashboard over an empty feed is not a monitoring system, it's a lullaby.

The security hole: 19 cron routes and one spoofable header

The swarm's security lens found the one that made me put the coffee down.
The product runs on scheduled jobs. Research, compose, publish, reconcile, engagement, the lot: 19 cron routes in total. Every one of them authenticated the caller by checking a user-agent string, the one the hosting platform's cron runner sends, something to the effect of "vercel-cron". A user-agent is not authentication. A user-agent is a suggestion. Anyone on the internet can send that header from curl in one line.
There was supposed to be a real check behind it, a shared secret the routes would verify. The environment variable for it, "CRON_SECRET", was unset in production. So the code path that did the actual verifying never engaged, and the spoofable header was the entire perimeter.
Practical exposure: anyone who guessed or read the route names could have triggered publishes on customer accounts, burned my LLM budget, or fired arbitrary pipeline stages at will. The logs show nobody did. That is luck, not security, and the whole point of the fix is never needing to check the logs with my heart rate up.
The fix took an afternoon: set the secret, verify it on all 19 routes, reject the header-only path entirely, and add a test that asserts no cron route ever answers 200 without it. The lesson took longer to absorb: the swarm found this because one lens existed solely to ask "who can call this, really?" about every route. I had asked that question when I wrote the first cron route, months earlier, and never asked it again as the other 18 accumulated.

What the swarm got wrong

Six findings died in verification, and the shapes are instructive. Duplicate-looking code that was intentional. A "missing index" on a table with a few hundred rows where a scan is fine for years. Pattern-matched bug shapes that dissolved on contact with production reality.
So the operating rule I run now: treat swarm output like a static analyzer with opinions. It is tireless, it reads everything, it holds eight perspectives at once, and it does not know your system's history. The human is the confirmation gate, and 38 raised to 32 confirmed is a ratio I trust. If it had been 38 for 38, I would have suspected the swarm of telling me what I wanted to hear.

Three lessons I paid for

Alarm on absence. Both outages were absence failures: the absence of a channel bind, the absence of a landed post. Monitoring that only fires on errors misses the failure mode where everything succeeds and nothing happens, and in my experience that mode is more common and far more expensive than the loud crash. Count the thing that should exist, and alarm when the count is zero.- Verify before assert. A 201 from a publish API is a receipt for the request, not evidence of the post. The engine now reconciles against the platform's actual state before believing its own database, and the same discipline applies one level up: a green status page is an assertion your checks are making, and assertions want verification.- Landed greater than zero is the only metric. Every intermediate number lies eventually. Drafts generated, jobs completed, requests accepted: all of them stayed healthy through a total output failure. The metric that survives is the one measured at the far end of the pipe, in someone else's system, where your bugs can't reach it.## The cost, and the verdict The sweep ran in a day. The fixes landed over the following days as a stack of small pull requests, each scoped to one confirmed finding. Cheap, against the alternative: the OAuth race alone would have gated every new customer at the exact moment I tried to onboard them, live, with the failure cosplaying as their browser's fault. I build this product with agents, so there is a certain symmetry in agents being the ones to catch what I missed. But the symmetry is not the point. The point is that fresh eyes at scale are now cheap, and the only reason not to point them at your own production system is that you would rather not know. The dashboard is still green. The difference is that now I have reasons to believe it.

One human, one GPU, one company: running a software business with an agent workforce

Will Kline — Tue, 14 Jul 2026 15:30:06 +0000

The whole company is one person and one graphics card.
Thirteen projects in Linear. Three live products taking money, a fourth in production and posting daily for a paying customer. Every pull request reviewed, every deploy approved, every invoice sent from the same desk in Oakland. The workforce that makes this possible is software: agents that research, draft, publish, test, and audit while I sleep.
This is the ops reality of that arrangement. Not the vision-deck version. The version with the failure modes in it, because the failure modes are where all the actual lessons live.

Where this came from

I spent twenty-plus years in high-end audio, from a retail floor in Berkeley to running Sonus Faber for North America. The entire discipline of that industry is chasing a clean signal through a noisy chain. Somewhere around year eighteen I realized that was also a fair description of running a company, and that the noisy chain in my life was no longer analog.
So now I build software. Alone, in the org-chart sense. Not alone in any sense that matters at 3am, when the research agents are reading trade publications and the compose pipeline is drafting posts and the test suites are running against last night's merges.

The shape of the operation

The portfolio, plainly: a marketing engine for HiFi retailers, a real-time dashboard for traders and investors, an info hub for AI hardware that ships an automated daily brief, and a social-posting engine that drafts and publishes every day for a high-end audio dealer, customer #1, who pays real money and expects real posts.
All of it runs through one Windows workstation with a single RTX 5090, and one control plane I call the captain dashboard. Nineteen managed apps live under it: image generation, video generation, 3D, training, transcription, local LLM serving, the works. The dashboard knows what is running, what VRAM each process holds, and what is queued.
That last part matters more than it sounds like it should.

One GPU, many agents, one mutex

Here is the problem nobody warns you about: agents do not politely take turns. If three pipelines each decide they need the GPU at the same moment, you do not get three slow jobs. You get out-of-memory crashes, half-loaded models, and a card that reports full allocation with nothing useful running on it.
The fix was embarrassingly old-fashioned. A mutex. Any job tagged gpu-heavy has to acquire the lock before it touches the card, and everything else queues. My agents wait for VRAM the way engineers wait for the coffee machine: grumbling, in an orderly line, one at a time.
The second fix was hygiene. A force-killed process on Windows will happily die while its GPU allocation lives on as a ghost. So every session starts the same way: check the card, find the zombies, kill them by process path rather than by port, because a crashed process holds memory without listening on anything. It is the least glamorous ritual in the entire company and skipping it costs more time than any other single mistake I make.

What the agents actually do

The honest list, not the marketing list:

Research. Every night, agents read the publications that matter to each product's audience and pull out what changed. For the posting engine that means the customer's industry gets watched while he sleeps.- Drafting. Posts, briefs, code, migrations. Drafts, plural, because most of them die in review and that is the system working as designed.- Testing and auditing. Suites run on merge. Beyond that, I have pointed an agent swarm at my own production SaaS as auditors, and they found real problems I could not see. That story deserves its own writeup, and it is getting one.- Publishing with verification. The posting engine does not trust itself. A successful API response is a receipt for the request, not proof the post exists, so a reconcile step checks the platform's actual state afterward.- Remembering. This one took me too long. Agents kept re-learning things other agents had already learned, so I built memsearch, a shared memory layer with semantic and keyword search over roughly 450 memory files. An agent that hits a known trap now finds the note the last agent left about it.## What the human keeps Three things do not get delegated, and the list is deliberate.
Approvals. Nothing customer-facing goes live without my word. The posting engine drafts, but a human taps approve on every single post. This is not a limitation we apologize for, it is a design decision, and it is the one my paying customer cares about most.- Taste. An agent can tell you a draft is grammatical and grounded. It cannot tell you the draft is boring, or that it sounds like everyone else's feed. Taste is the moat, and taste does not batch.- Accountability. When something breaks, the customer emails a person, and the person answers. There is no tier-one support bot between customer #1 and me. If the pipeline drops his posts, he hears from me plainly, with the fix in the same breath.My stated operating rule for the whole company: consult one dashboard daily, handle only the items that genuinely require a human, and when I have an idea, fire it and have it driven to completion. Most days that holds. The days it does not hold are the next section. ## What it costs, honestly The free key incident. For a stretch of production time, the posting engine's LLM calls ran on a free-tier API key rate-limited to 5 requests per minute, while the paid organization account, credits loaded and ready, sat completely idle. The symptom was a storm of 429s that looked exactly like an architecture problem. I went looking for concurrency bugs and backpressure design flaws. The actual fix was checking which key production was holding. Lesson: when an agent workforce misbehaves, check the boring thing first, because the boring thing is not embarrassed to be the answer. The gate that rejected everything. The posting engine has a factual gate: drafts must ground their claims in a source before they earn the feed. At one point that gate was rejecting 100% of drafts. Not most. All. It turned out to be grading full captions against an 84-character headline, so any caption that actually said something failed, and the captions that said nothing scored best. A quality gate that rewards emptiness is worse than no gate. We fixed it the boring way: give the gate the full source text, not the teaser. The metric that survived. After enough incidents you stop trusting intermediate signals. Queues can look healthy while nothing ships. Dashboards can say cadence satisfied while zero posts land. The only number I fully believe anymore, for a billable account, is landed greater than zero. Did the thing actually appear on the platform today. Everything upstream of that answer is a hypothesis. The real bottleneck. People assume the constraint on a one-person software company is generation capacity. It is not, and it has not been for a while. Agents do not get tired. The bottleneck is review discipline: my ability to read what they produced with full attention, catch the subtle wrongness, and keep the approval step from decaying into a rubber stamp. The day approvals become reflexive is the day the whole model quietly fails, and no alarm fires for that one. ## If you are considering this
Build the control plane before you build the workforce. One dashboard that answers what is running and what is stuck is worth more than the next three agents.- Serialize your scarce resource explicitly. One GPU means one mutex. Hoping the schedulers will not collide is not a strategy, it is a countdown.- Give the agents shared memory early. Every lesson learned by exactly one agent is a lesson your company will pay for again.- Instrument absence, not just failure. Errors page you. Silence does not, and silence is the expensive one.- Keep approvals, taste, and accountability human, and be suspicious of anyone who tells you the third one scales.None of this is a pitch. The products named here are just the systems being run, and the arrangement has cost me real production incidents, several days of chasing phantom architecture problems, and one deeply humbling conversation with a rate limiter. But the arrangement also works. One person, one GPU, thirteen projects, and a workforce that never asks for a standup. The fun part starts now.