<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Konrad Wright</title>
    <description>The latest articles on DEV Community by Konrad Wright (@konradwright).</description>
    <link>https://dev.to/konradwright</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3932249%2F9bb866e5-95b4-4882-8859-23526eb51da0.png</url>
      <title>DEV Community: Konrad Wright</title>
      <link>https://dev.to/konradwright</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/konradwright"/>
    <language>en</language>
    <item>
      <title>How Does a Model Actually Learn?</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Mon, 16 Nov 2026 15:17:24 +0000</pubDate>
      <link>https://dev.to/konradwright/how-does-a-model-actually-learn-3948</link>
      <guid>https://dev.to/konradwright/how-does-a-model-actually-learn-3948</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://wrightfunctions.com/articles/how-a-model-learns/" rel="noopener noreferrer"&gt;wrightfunctions.com&lt;/a&gt;, where the demos are live.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In the &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;first post of this series&lt;/a&gt;, I said a model is a function whose inner numbers are learned from examples instead of written by hand. That's true, but it skips the interesting part. &lt;em&gt;How&lt;/em&gt; does a pile of numbers learn anything?&lt;/p&gt;

&lt;p&gt;The answer is a surprisingly simple game of hot and cold, repeated an enormous number of times. You need three ideas: a way to score how wrong you are, a way to know which direction is less wrong, and a way to check you haven't just memorized the answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step one: a number for "how wrong"
&lt;/h2&gt;

&lt;p&gt;Start with a guess. Before training, the model's weights are random, so its predictions are nonsense. Now compare them with the right answers and boil the difference down to one number.&lt;/p&gt;

&lt;p&gt;That number is the loss, which Google's course defines as &lt;a href="https://developers.google.com/machine-learning/crash-course/linear-regression/loss" rel="noopener noreferrer"&gt;"a numerical metric that describes how wrong a model's predictions are."&lt;/a&gt; Zero means perfect. Bigger means worse.&lt;/p&gt;

&lt;p&gt;For a line through some points, the usual loss is the average squared vertical gap between each point and the line. For a large language model, it's &lt;a href="https://d2l.ai/chapter_recurrent-neural-networks/language-model.html" rel="noopener noreferrer"&gt;how surprised the model was&lt;/a&gt; by each real next word. Once "how wrong" is one number, learning becomes a search: find the weights that make that number as small as possible.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step two: roll downhill
&lt;/h2&gt;

&lt;p&gt;Picture every possible setting of the weights as a landscape, with the loss as the height. You're somewhere on a hillside in fog, and you want to reach the lowest valley. You can't see it, but you can feel which way the ground slopes under your feet. So you take a small step downhill, feel again, and repeat.&lt;/p&gt;

&lt;p&gt;That's gradient descent, and it's old: it's usually credited to &lt;a href="https://ems.press/content/book-chapter-files/27368?nt=1" rel="noopener noreferrer"&gt;Augustin-Louis Cauchy in 1847&lt;/a&gt;. The "feel the slope" part, for millions of weights at once, is backpropagation, popularized in &lt;a href="https://www.nature.com/articles/323533a0" rel="noopener noreferrer"&gt;a 1986 Nature paper&lt;/a&gt;. On the original post, you can try it with the simplest model there is, a straight line with two weights.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/how-a-model-learns/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The learning rate is the size of each step. Google's course describes both ways it fails: too low and the model &lt;a href="https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters" rel="noopener noreferrer"&gt;"can take a long time to converge"&lt;/a&gt;, too high and it &lt;a href="https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters" rel="noopener noreferrer"&gt;"bounces around"&lt;/a&gt; the bottom instead of settling. Choosing it is one of the fiddliest parts of training real models.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same game, at scale
&lt;/h2&gt;

&lt;p&gt;A large language model plays exactly this game, with billions of weights instead of two. Real training doesn't compute the loss on all the data for every step, which would be far too slow. It takes &lt;a href="https://cs231n.github.io/optimization-1/" rel="noopener noreferrer"&gt;a small random batch&lt;/a&gt;, steps, and takes another: stochastic gradient descent. One full pass through the data is an epoch.&lt;/p&gt;

&lt;p&gt;The numbers get large, but the ideas don't change. Meta trained Llama 3 405B with a refined version of gradient descent called AdamW, over &lt;a href="https://arxiv.org/abs/2407.21783" rel="noopener noreferrer"&gt;1.2 million steps&lt;/a&gt; and 15.6 trillion tokens, with a carefully scheduled learning rate that warms up and then slowly shrinks. Every one of those steps was: measure the loss, feel the slope, step downhill.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step three: don't memorize the practice test
&lt;/h2&gt;

&lt;p&gt;The trap is that a flexible enough model can drive the loss on its training examples almost to zero by memorizing them, noise and all. That's overfitting: a model that &lt;a href="https://developers.google.com/machine-learning/crash-course/overfitting/overfitting" rel="noopener noreferrer"&gt;"matches (memorizes) the training set so closely that the model fails to make correct predictions on new data"&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The defence is to hold data back. Train on one set, and score the model on a test set it never trained on. The classic handwritten-digit dataset, MNIST, ships that way: &lt;a href="https://keras.io/api/datasets/mnist/" rel="noopener noreferrer"&gt;60,000 training images and 10,000 test images&lt;/a&gt;. The test score is the honest measure of generalization: how the model does on data it has never seen.&lt;/p&gt;

&lt;p&gt;In practice there's often &lt;a href="https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets" rel="noopener noreferrer"&gt;a third slice&lt;/a&gt;, a validation set, for choosing settings like the learning rate along the way, so the test set stays untouched until the very end. On the original post, you can watch the two scores part ways.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/how-a-model-learns/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;One modern wrinkle: very large neural networks don't always follow this textbook curve. Researchers found that past a certain size, test error can &lt;a href="https://arxiv.org/abs/1912.02292" rel="noopener noreferrer"&gt;fall again after rising&lt;/a&gt;, called "double descent". The rule of checking on held-out data holds either way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why models need retraining
&lt;/h2&gt;

&lt;p&gt;All this explains something from the &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;first post&lt;/a&gt;: why models go stale. Training finds weights that work for the data it saw. Google's course notes that good generalization assumes the training examples are &lt;a href="https://developers.google.com/machine-learning/crash-course/overfitting/overfitting" rel="noopener noreferrer"&gt;"statistically similar"&lt;/a&gt; to real-world data. When the world drifts away from the training data, that assumption breaks, and the fix is to play the game again on newer data.&lt;/p&gt;

&lt;h2&gt;
  
  
  So how does a model learn?
&lt;/h2&gt;

&lt;p&gt;By being wrong in a measurable way, and stepping toward less wrong, over and over. Score the guesses with a loss, follow the slope downhill with gradient descent, keep the steps the right size, and check against data the model has never seen so it learns the pattern instead of the answers. It's the same game whether the model has two weights or a trillion.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://developers.google.com/machine-learning/crash-course/linear-regression/loss" rel="noopener noreferrer"&gt;Linear regression: Loss&lt;/a&gt; (Google Machine Learning Crash Course). "A numerical metric that describes how wrong a model's predictions are."&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://d2l.ai/chapter_recurrent-neural-networks/language-model.html" rel="noopener noreferrer"&gt;Language Models&lt;/a&gt; (Dive into Deep Learning · §9.3). Next-token cross-entropy, the average surprise at each real token.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://ems.press/content/book-chapter-files/27368?nt=1" rel="noopener noreferrer"&gt;Cauchy and the Gradient Method&lt;/a&gt; (Claude Lemaréchal · Documenta Mathematica · 2012). How gradient descent traces back to Cauchy's note of October 18, 1847.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.nature.com/articles/323533a0" rel="noopener noreferrer"&gt;Learning representations by back-propagating errors&lt;/a&gt; (Rumelhart, Hinton &amp;amp; Williams · Nature · 1986). Backpropagation, the method for working out every weight's slope at once.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters" rel="noopener noreferrer"&gt;Linear regression: Hyperparameters&lt;/a&gt; (Google Machine Learning Crash Course). Learning rate, batch size and epochs, and what happens when the learning rate is too low or too high.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://cs231n.github.io/optimization-1/" rel="noopener noreferrer"&gt;Optimization: Stochastic Gradient Descent&lt;/a&gt; (Stanford CS231n course notes). Why real training steps on small random batches instead of the whole dataset.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2407.21783" rel="noopener noreferrer"&gt;The Llama 3 Herd of Models&lt;/a&gt; (Llama Team, Meta · 2024). AdamW with a peak learning rate of 8 × 10⁻⁵, 8,000 warm-up steps and a cosine schedule over 1.2M steps.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.google.com/machine-learning/crash-course/overfitting/overfitting" rel="noopener noreferrer"&gt;Overfitting&lt;/a&gt; (Google Machine Learning Crash Course). Memorizing the training set, and why training data must look like real-world data.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://keras.io/api/datasets/mnist/" rel="noopener noreferrer"&gt;MNIST digits classification dataset&lt;/a&gt; (Keras documentation). 60,000 training images and 10,000 test images of handwritten digits.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets" rel="noopener noreferrer"&gt;Datasets: Dividing the original dataset&lt;/a&gt; (Google Machine Learning Crash Course). Training, validation and test sets, and what each one is for.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/1912.02292" rel="noopener noreferrer"&gt;Deep Double Descent: Where Bigger Models and More Data Hurt&lt;/a&gt; (Nakkiran et al. · 2019). Test error that gets worse, then better, as models grow.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;AI-assisted: I write these with help from AI tools.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>training</category>
      <category>fundamentals</category>
    </item>
    <item>
      <title>What Happens When You Hit Enter?</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Fri, 13 Nov 2026 15:17:24 +0000</pubDate>
      <link>https://dev.to/konradwright/what-happens-when-you-hit-enter-227b</link>
      <guid>https://dev.to/konradwright/what-happens-when-you-hit-enter-227b</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://wrightfunctions.com/articles/when-you-hit-enter/" rel="noopener noreferrer"&gt;wrightfunctions.com&lt;/a&gt;, where the demos are live.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You type a question into a chatbot and press Enter. A beat later, words start streaming out, a few at a time, like someone typing fast. It feels like one action, but it's a small pipeline, and each stage explains something you've probably noticed: why the first word takes a moment, why the rest flow quickly, and why the same question gets a different answer twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Text becomes numbers
&lt;/h2&gt;

&lt;p&gt;A model can't read letters. Your message is first chopped into tokens, common chunks of text learned from data with byte-pair encoding. As a rule of thumb for English, one token is &lt;a href="https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them" rel="noopener noreferrer"&gt;about four characters, or three-quarters of a word&lt;/a&gt;. Each token becomes an ID number, and each ID is swapped for an embedding, a long list of numbers standing for its meaning.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/when-you-hit-enter/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Every word looks back at the words before it
&lt;/h2&gt;

&lt;p&gt;Now the transformer does its work. Its main step is attention, which the 2017 paper &lt;a href="https://arxiv.org/abs/1706.03762" rel="noopener noreferrer"&gt;Attention Is All You Need&lt;/a&gt; made the whole design: each token looks back at all the tokens before it and weighs which ones matter. In "we sat on the river bank", &lt;em&gt;bank&lt;/em&gt; pays attention to &lt;em&gt;river&lt;/em&gt; and settles on the right meaning. A model repeats this through dozens of layers, each refining every token's meaning a little further.&lt;/p&gt;

&lt;p&gt;The output of all that is a single thing: a score for every token in the vocabulary, saying how likely each one is to come &lt;em&gt;next&lt;/em&gt;. That's all a language model ever produces: one next token.&lt;/p&gt;

&lt;h2&gt;
  
  
  The next word is a dice roll
&lt;/h2&gt;

&lt;p&gt;The model doesn't simply take the top-scoring token. It rolls weighted dice, which is sampling. Temperature reshapes the dice first: low values make the favourite almost certain, high values give long shots a real chance. OpenAI's API documents it as a number from 0 to 2, where &lt;a href="https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create" rel="noopener noreferrer"&gt;"higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/when-you-hit-enter/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Many systems also let you trim the unlikely tail before rolling, a trick called &lt;a href="https://arxiv.org/abs/1904.09751" rel="noopener noreferrer"&gt;nucleus sampling&lt;/a&gt;. And even at temperature 0, answers can differ run to run. In 2025, researchers at Thinking Machines &lt;a href="https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/" rel="noopener noreferrer"&gt;traced that to the server&lt;/a&gt;: how many other users' requests are batched with yours changes the arithmetic very slightly, and one flipped token sends the rest of the answer somewhere new.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then it does it again, and again
&lt;/h2&gt;

&lt;p&gt;The chosen token is added to the text, and the whole thing runs again for the next one. That loop is why answers stream in word by word, and it splits serving into two very different phases.&lt;/p&gt;

&lt;p&gt;First, prefill: the model reads your entire prompt in one parallel pass. That's heavy math on many tokens at once. It's most of the pause before anything appears, measured as time to first token. It's also why a long pasted document slows the start of a reply, even when the reply itself is short.&lt;/p&gt;

&lt;p&gt;Then decode: one token per step, each step reading the entire model from memory to produce a single word. Researchers describe it as &lt;a href="https://arxiv.org/abs/2311.18677" rel="noopener noreferrer"&gt;"a compute-intensive prompt computation" phase followed by "a memory-intensive token generation" phase&lt;/a&gt;, which is why GPUs &lt;a href="https://wrightfunctions.com/articles/gpus-for-server-people/" rel="noopener noreferrer"&gt;spend so much of decoding waiting&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cache that makes it cheap
&lt;/h2&gt;

&lt;p&gt;What makes decode affordable is the KV cache. It keeps each token's attention data, so the model doesn't reprocess the whole conversation for every new word. It isn't small, either: for one 13-billion-weight model, the vLLM paper works out &lt;a href="https://arxiv.org/abs/2309.06180" rel="noopener noreferrer"&gt;800 KB per token, and up to 1.6 GB for a single long request&lt;/a&gt;. On the original post, you can hit Enter and watch it fill.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/when-you-hit-enter/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That cache is one reason a model's context window has a limit, and one reason long conversations cost more. As of September 2026, Anthropic's larger Claude models take &lt;a href="https://platform.claude.com/docs/en/models/overview" rel="noopener noreferrer"&gt;1 million tokens&lt;/a&gt;, and every one of them needs a slot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making it faster
&lt;/h2&gt;

&lt;p&gt;Almost every serving trick targets one of these phases. Batching lets many users share each read of the model during decode. Quantization makes the model smaller to read.&lt;/p&gt;

&lt;p&gt;Speculative decoding lets a small model guess several tokens ahead and has the big one check them in one pass, for &lt;a href="https://arxiv.org/abs/2211.17192" rel="noopener noreferrer"&gt;2 to 3 times the speed on the paper's test model, with identical output&lt;/a&gt;. Some systems now &lt;a href="https://arxiv.org/abs/2401.09670" rel="noopener noreferrer"&gt;run prefill and decode on different GPUs&lt;/a&gt; so the two stop getting in each other's way.&lt;/p&gt;

&lt;h2&gt;
  
  
  So what happens when you hit Enter?
&lt;/h2&gt;

&lt;p&gt;Your text becomes tokens, tokens become numbers, and the transformer turns them into odds for one next token. A weighted dice roll picks it, and the whole loop runs again, token after token, until the answer is done. The pause before the first word is prefill; the steady stream after it is decode, one memory-bound step per token. Every stage is something someone has to serve, scale and pay for.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them" rel="noopener noreferrer"&gt;What are tokens and how to count them?&lt;/a&gt; (OpenAI Help Center). For English, a token is about 4 characters, or three-quarters of a word.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/1706.03762" rel="noopener noreferrer"&gt;Attention Is All You Need&lt;/a&gt; (Vaswani et al. · Google · 2017). The transformer, "based solely on attention mechanisms". The design behind nearly every modern language model.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create" rel="noopener noreferrer"&gt;Create chat completion (temperature)&lt;/a&gt; (OpenAI API reference). Temperature from 0 to 2; 0.8 is more random, 0.2 more focused and deterministic.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/1904.09751" rel="noopener noreferrer"&gt;The Curious Case of Neural Text Degeneration&lt;/a&gt; (Holtzman et al. · ICLR 2020). Nucleus (top-p) sampling, which cuts off the unreliable tail of the odds before rolling.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/" rel="noopener noreferrer"&gt;Defeating Nondeterminism in LLM Inference&lt;/a&gt; (Horace He · Thinking Machines Lab · September 2025). Why temperature 0 isn't repeatable on real servers. Batch size changes with load, and most kernels aren't batch-invariant.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2311.18677" rel="noopener noreferrer"&gt;Splitwise: Efficient generative LLM inference using phase splitting&lt;/a&gt; (Patel et al. · 2023). A compute-intensive prompt phase and a memory-intensive generation phase, and why they deserve different hardware.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2309.06180" rel="noopener noreferrer"&gt;Efficient Memory Management for Large Language Model Serving with PagedAttention&lt;/a&gt; (Kwon et al. · SOSP 2023). 800 KB of KV cache per token for OPT-13B, up to 1.6 GB per request, and paging that cache like virtual memory.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://platform.claude.com/docs/en/models/overview" rel="noopener noreferrer"&gt;Models overview&lt;/a&gt; (Anthropic · docs · as of September 2026). 1M-token context windows on the larger Claude models.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2211.17192" rel="noopener noreferrer"&gt;Fast Inference from Transformers via Speculative Decoding&lt;/a&gt; (Leviathan, Kalman &amp;amp; Matias · ICML 2023). A small model drafts, the big one verifies. 2–3× faster on T5-XXL with identical outputs.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2401.09670" rel="noopener noreferrer"&gt;DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving&lt;/a&gt; (Zhong et al. · OSDI 2024). Prefill and decode on separate GPUs, tuned for time to first token and time per output token.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;AI-assisted: I write these with help from AI tools.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>llm</category>
      <category>inference</category>
    </item>
    <item>
      <title>What Is ML Infrastructure?</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Mon, 09 Nov 2026 15:17:25 +0000</pubDate>
      <link>https://dev.to/konradwright/what-is-ml-infrastructure-56ad</link>
      <guid>https://dev.to/konradwright/what-is-ml-infrastructure-56ad</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;wrightfunctions.com&lt;/a&gt;, where the demos are live.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;By now, we're all familiar with the basics of chatbots. Large language models are models trained on a huge pile of text to predict what comes next. It's a neat parlor trick, and left at that, it makes a great wiki helper. The interesting part is where we give it some teeth to bite with.&lt;/p&gt;

&lt;p&gt;That part is &lt;strong&gt;ML infrastructure&lt;/strong&gt;: your AI assistant getting access to your files, your emails, and the rest, able to complete real tasks. Order you real coffee. Steal your real job. That's the sophisticated stuff, and it's where almost all of the engineering lives. Once you can see it, you can't unsee it. Let me build it up one piece at a time, with a few toys you can poke at along the way.&lt;/p&gt;

&lt;h2&gt;
  
  
  First: what even is a "model"?
&lt;/h2&gt;

&lt;p&gt;At the end of the day, a model is just a &lt;em&gt;function&lt;/em&gt; whose contents are decided by the machine, through training. The simplest version is linear regression: a pattern emerges from the data points, you draw a line through it, and any new point lands somewhere on that line. It's &lt;em&gt;learning&lt;/em&gt; in the sense that the pattern is emerging.&lt;/p&gt;

&lt;p&gt;Here's a hands-on example: a 28×28 pixel grid. Each pixel has a sort of memory: "if I'm lit up, I'm most likely this number." Some pixels lean toward several numbers at once, so the model reads them in combination ("if I'm lit and my north neighbor is lit too, I'm probably a 4") when it wants more precision. This little neural network is actually running totally locally, in your browser. (Not relevant, but I like to brag.)&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's the whole model. About nineteen thousand weights (squeezed down with a quantization trick we'll get to), and it worked out its own rules instead of being handed any. The model behind a chatbot is millions of times bigger, but it's the exact same shape: input in, guess out. Everything else in this article is the stuff bolted around that box.&lt;/p&gt;

&lt;p&gt;Here's the picture I keep coming back to: the model is a &lt;strong&gt;recipe&lt;/strong&gt;. Everything else is the &lt;strong&gt;restaurant&lt;/strong&gt;: the kitchen, the walk-in fridges, the supply trucks, the waiters, the dish pit. Everything that turns one good recipe into a thousand hot meals a night without poisoning anybody.&lt;/p&gt;

&lt;h2&gt;
  
  
  The model is the easy part
&lt;/h2&gt;

&lt;p&gt;Getting a model to work &lt;em&gt;once&lt;/em&gt;, on your laptop, is the easy 10%. The other 90% is getting fresh data into it, retraining it without breaking anything, and shoving the result in front of real users, again and again, automatically, forever. A famous paper drew this to scale: in a real ML system, the box with the actual model code is a speck, swallowed by the sprawl of data, serving, and monitoring around it (&lt;a href="https://proceedings.neurips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html" rel="noopener noreferrer"&gt;Sculley et al., "Hidden Technical Debt in Machine Learning Systems"&lt;/a&gt;). That loop has a name, MLOps, and what makes it its own discipline is continuous training: a model has to keep relearning just to stay correct.&lt;/p&gt;

&lt;p&gt;On the original post, you can step through the loop below. Every stage is its own category of tooling and engineering, and the whole thing runs on repeat every time new data shows up.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Step four, the eval gate, is the quiet hero here. The rule is blunt: if the new model isn't actually better than the one already running, it doesn't ship. A big chunk of ML infrastructure exists just to enforce that automatically, so nothing bad slips out to your users at 2am while everyone's asleep. Google's own &lt;a href="https://developers.google.com/machine-learning/guides/rules-of-ml" rel="noopener noreferrer"&gt;&lt;em&gt;Rules of Machine Learning&lt;/em&gt;&lt;/a&gt; says it from the other side: nail your metrics and your infrastructure first, &lt;em&gt;then&lt;/em&gt; get clever with the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the model actually eats
&lt;/h2&gt;

&lt;p&gt;Go back to step one of that loop, &lt;em&gt;ingest data&lt;/em&gt;, the tile that slid into the model box in half a second. That half second hides most of the actual job. A model is only ever as good as what it learned from, and that data doesn't fall from the sky: somebody has to gather it, clean it, label it, and park it somewhere the training run can reach. That whole apparatus is the data layer, and on most real teams it's where most of the work goes.&lt;/p&gt;

&lt;p&gt;Start with labels. Our little digit net learned from tens of thousands of images, each stamped with the right answer: this is a 7, this is a 2. Someone, or something, had to make those stamps, and if they're wrong, the model learns your mistakes faithfully. Feed it enough 7s labeled "1" and it'll call sevens ones, confidently, forever. Garbage in, garbage out isn't a slogan here. It's a mechanism.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Then there's the trap of computing the same data twice. A feature you calculate during training (a user's average order size, say) has to be calculated the exact same way when a live request lands, or the model sees a subtly different world in production than the one it trained on. Teams stand up feature stores and pipelines mostly to kill that train/serve skew, and to keep last week's numbers meaning the same thing as this week's. Not hypothetical: the Uber team behind &lt;a href="https://www.uber.com/us/en/blog/michelangelo-machine-learning-platform/" rel="noopener noreferrer"&gt;Michelangelo&lt;/a&gt;, one of the first company-wide ML platforms, ran a feature store precisely so the exact same transformation code runs offline in training and online in serving, closing the gap by construction.&lt;/p&gt;

&lt;p&gt;And since you'll eventually need to reproduce, audit, or roll back a model, the data needs data versioning too, just like code. "Which exact snapshot did this train on?" is a question you have to be able to answer. This is what people mean when they call data the real moat: the architecture is usually public, but nobody else has your data: gathered, cleaned, labeled, and versioned the way you have it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching it rot
&lt;/h2&gt;

&lt;p&gt;Nothing about the model changes on its own. The weights freeze the second you ship. And yet a model that scored 97% on launch day gets quietly worse, week after week, nobody touching a thing. It didn't break. The world moved, and it didn't.&lt;/p&gt;

&lt;p&gt;This is drift. Handwriting changes, fraud patterns evolve, slang turns over, and the frozen model keeps answering like it's still launch day. The gap between the world it learned and the world it's serving widens, and accuracy leaks out through it. And it's quiet: service up, latency fine, nothing red in the logs. The answers just slowly get worse.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So you watch it. Monitoring an ML model isn't your usual CPU-and-error-rate dashboard. It's watching the &lt;em&gt;predictions themselves&lt;/em&gt;, the kind of quiet distribution shift Chip Huyen's &lt;a href="https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/" rel="noopener noreferrer"&gt;&lt;em&gt;Designing Machine Learning Systems&lt;/em&gt;&lt;/a&gt; spends whole chapters on. Are the outputs still distributed like they used to be? Is confidence sliding? When the real answers finally roll in, is it still right as often? Cross a threshold and an alarm fires, and that alarm is what kicks the loop back to step one: ingest, retrain, re-evaluate, ship. The cycle you clicked through earlier finally has a motor. Drift is what turns it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Now do it a million times
&lt;/h2&gt;

&lt;p&gt;A model answering one request is a demo. A model answering a million requests an hour, every hour, without ever falling over, is a product. Everything between those two is infrastructure. This layer has a name: serving, or inference.&lt;/p&gt;

&lt;p&gt;On the original post, you can drag the traffic up and watch one server struggle. Then switch on autoscaling and try again.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;One fixed server holds up fine, until it doesn't. Push enough traffic and answers slow, then requests start dropping: the dreaded endless spinner. Autoscaling notices the crowd and adds servers so latency stays flat. Bolt on health checks, retries, and monitoring, and you've got something that survives 3am on launch day. None of that reliability is the model's job. It's the infrastructure's.&lt;/p&gt;

&lt;h2&gt;
  
  
  And do it cheaply
&lt;/h2&gt;

&lt;p&gt;One last force: money. Every answer a model gives burns a little compute, and "a little" times "a million" is a real invoice. Which is why "make the model cheaper to run" is its own specialty, not an afterthought.&lt;/p&gt;

&lt;p&gt;Play with the dials to size up a workload. Then flip on the optimizations, quantization and batching, and watch the monthly bill fall. This isn't a paper trick: one landmark method, &lt;a href="https://arxiv.org/abs/2208.07339" rel="noopener noreferrer"&gt;LLM.int8()&lt;/a&gt;, ran a 175-billion-parameter model in 8-bit instead of 16, roughly halving the memory, with no measurable drop in accuracy.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Same model, same answers, a fraction of the cost. Multiply that gap across a growing company and it's obvious why inference optimization is one of the most valued skills in the field right now: it converts, almost directly, into margin.&lt;/p&gt;

&lt;h2&gt;
  
  
  When it breaks, it lies
&lt;/h2&gt;

&lt;p&gt;Regular software fails loud. A bug throws an exception, a stack trace hits the logs, a page goes red, somebody gets paged. The failure announces itself.&lt;/p&gt;

&lt;p&gt;A model won't do you that favor. Ask it something it can't handle and it doesn't throw. It hands you a confident, well-formatted, completely wrong answer, in exactly the same shape as a right one. No exception, because from the model's point of view nothing went wrong: input in, output out, same as always. Your instincts are tuned for loud failure. Machine learning fails silent.&lt;/p&gt;

&lt;p&gt;So a whole layer of infrastructure exists to catch the model being confidently wrong. Google even turned it into a scorecard, &lt;a href="https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/" rel="noopener noreferrer"&gt;&lt;em&gt;The ML Test Score&lt;/em&gt;&lt;/a&gt;, a 28-point rubric for the tests and monitoring that keep a silently-broken model out of production. Guardrails check the output before it reaches a user. When something smells off, you fall back to something safer. None of this makes the model smarter. It just refuses to let a quiet mistake walk out the door. And most of it, as we're about to see, lives in the layer wrapped around the model, not the model itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The twist for language models
&lt;/h2&gt;

&lt;p&gt;Everything so far applies to any model, from our little digit net to the giant behind a chatbot. But once the model &lt;em&gt;is&lt;/em&gt; one of those giants, a large language model, a few new pieces of infrastructure show up, and they're where a lot of today's work is happening.&lt;/p&gt;

&lt;p&gt;The shape's the same: text goes in as tokens (that cost-demo dial, remember), text comes out. But the model is frozen at its training cutoff, so you feed the knowledge in at question time. That's retrieval-augmented generation: look up the relevant documents and hand them to the model along with the question. The &lt;a href="https://arxiv.org/abs/2005.11401" rel="noopener noreferrer"&gt;paper that coined the term&lt;/a&gt; bolted a frozen model onto a searchable index of Wikipedia and got more specific, more factual answers than the model gave alone. Doing that lookup fast means storing text as embeddings in a vector database. A surprising amount of LLM infrastructure is just this retrieval plumbing.&lt;/p&gt;

&lt;p&gt;Tokens aren't quite words and aren't quite characters. On the original post, you can type into the box and watch the split happen: the token count, not the word count, is the number you're billed for.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And here's retrieval itself, made concrete. On the original post, you can pick a question about a company whose private docs the model never saw in training, and watch it either guess from memory or answer straight from the retrieved document.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Two more terms you'll keep hearing. Fine-tuning vs. prompting is the build-versus-configure call of the LLM world: retrain the weights on your own data, or write a sharper prompt and retrieve better context? The second is cheap and usually enough. And the KV cache is the quiet reason serving LLMs is its own discipline. Manage it well and you've won half the battle on speed and cost. The &lt;a href="https://arxiv.org/abs/2309.06180" rel="noopener noreferrer"&gt;vLLM project&lt;/a&gt; made the point by stealing a trick from operating systems, paging that cache the way an OS pages virtual memory, and lifting throughput two-to-four-fold at the same latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  The harness on top
&lt;/h2&gt;

&lt;p&gt;Come back to the very first idea in this piece: a model is just a function. Input in, prediction out. No memory, no initiative. On its own it can't open a file, look something up, run a command, or string two steps together. So how does a bare next-token predictor turn into something that writes code, works a support queue, or grinds through a multi-step task?&lt;/p&gt;

&lt;p&gt;You wrap it in a harness: the loop that lets it take one step, see what happened, and pick the next. That reason-act-observe cycle got its name in the &lt;a href="https://arxiv.org/abs/2210.03629" rel="noopener noreferrer"&gt;ReAct&lt;/a&gt; work, which showed a model interleaving thinking with real actions against outside tools instead of guessing in one shot. The model brings the judgment; the harness brings the hands, the memory, and the guardrails. Claude Code is the obvious example: the underlying model can't touch your filesystem, but wrap it in a harness with file access, a shell, and a feedback loop, and it becomes something that reads a codebase, makes edits, runs the tests, and reacts to what broke. Anthropic's own &lt;a href="https://www.anthropic.com/engineering/building-effective-agents" rel="noopener noreferrer"&gt;guide to building effective agents&lt;/a&gt; draws the line here, between fixed workflows that script the model's steps and true agents that let it drive its own tools.&lt;/p&gt;

&lt;p&gt;On the original post, you can watch one run of that loop: think, call a tool, read what comes back, decide the next move, over and over until the job is done.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is where the earlier pieces come home to roost. The retries, validation, and fallbacks from the silent-failure section mostly live here, in the harness, not the model. So do the evals, except now you're not scoring one prediction, you're scoring whether the whole agent actually got the job done. And it's the layer people actually touch: when someone says they "use an AI tool," they almost always mean a harness with a model inside it. Back to the kitchen: if the model is the recipe, the harness is the line cook, the one who reads it, works the stations, tastes as it goes, and plates the dish.&lt;/p&gt;

&lt;h2&gt;
  
  
  So, what is ML infrastructure?
&lt;/h2&gt;

&lt;p&gt;Put the pieces together and it's this: everything that turns a clever model into a product a lot of people can actually use.&lt;/p&gt;

&lt;p&gt;It's the data plumbing that feeds the model: gathered, labeled, versioned. It's the &lt;a href="https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning" rel="noopener noreferrer"&gt;pipeline that retrains and ships it safely&lt;/a&gt;, with an eval gate on the door and monitoring watching for drift to decide when to pull the trigger. It's the serving layer that answers real requests fast, no matter how big the crowd. It's the cost engineering that keeps the whole thing cheap enough to stay in business. It's the guardrails that catch the model being confidently wrong, because it fails quiet, not loud. And more and more, it's the harness around the model that turns a bare function into something that actually does the work.&lt;/p&gt;

&lt;p&gt;The model is the recipe. ML infrastructure is the restaurant. And if you've ever run anything at scale (servers, deploys, uptime, budgets), you already speak most of the language. The models are the new part. The plumbing is the same job it's always been.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://proceedings.neurips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html" rel="noopener noreferrer"&gt;Hidden Technical Debt in Machine Learning Systems&lt;/a&gt; (Sculley et al. · NeurIPS 2015). The paper that put a diagram to "the model is the small part": in a real system the ML code is a tiny box surrounded by data, serving, and monitoring infrastructure that dwarfs it.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.google.com/machine-learning/guides/rules-of-ml" rel="noopener noreferrer"&gt;Rules of Machine Learning: Best Practices for ML Engineering&lt;/a&gt; (Martin Zinkevich · Google). Forty-three hard-won rules from Google's practice; the through-line is that infrastructure and metrics matter more than the model early on.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning" rel="noopener noreferrer"&gt;MLOps: Continuous Delivery and Automation Pipelines in Machine Learning&lt;/a&gt; (Google Cloud). The canonical write-up of the retrain-and-ship loop, including continuous training as the property unique to ML systems.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/" rel="noopener noreferrer"&gt;The ML Test Score: A Rubric for ML Production Readiness&lt;/a&gt; (Breck et al. · Google, 2017). A 28-point checklist for the testing and monitoring that keep a silently-broken model from reaching users; the backbone of the "when it breaks, it lies" section.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.uber.com/us/en/blog/michelangelo-machine-learning-platform/" rel="noopener noreferrer"&gt;Meet Michelangelo: Uber's Machine Learning Platform&lt;/a&gt; (Uber Engineering). A real, company-wide ML platform, and a clear look at feature stores and the train/serve skew problem from the data-layer section.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/" rel="noopener noreferrer"&gt;Designing Machine Learning Systems&lt;/a&gt; (Chip Huyen · O'Reilly, 2022). The best single book on the whole picture; especially good on data distribution shifts, drift, and what to actually monitor.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2005.11401" rel="noopener noreferrer"&gt;Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks&lt;/a&gt; (Lewis et al. · NeurIPS 2020). The paper that coined RAG: pair a frozen model with a searchable index of Wikipedia and it answers more specifically and factually.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2309.06180" rel="noopener noreferrer"&gt;Efficient Memory Management for LLM Serving with PagedAttention&lt;/a&gt; (Kwon et al. · SOSP 2023). The vLLM paper; the clearest explanation of why the KV cache makes LLM serving its own discipline: 2–4× throughput by paging it like OS memory.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2208.07339" rel="noopener noreferrer"&gt;LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale&lt;/a&gt; (Dettmers et al. · NeurIPS 2022). Quantization made concrete: run a 175-billion-parameter model in 8-bit, roughly half the memory, with no measurable loss in accuracy.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2210.03629" rel="noopener noreferrer"&gt;ReAct: Synergizing Reasoning and Acting in Language Models&lt;/a&gt; (Yao et al. · 2022). An early, readable version of the reason-act-observe loop that sits at the heart of every agent harness.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.anthropic.com/engineering/building-effective-agents" rel="noopener noreferrer"&gt;Building Effective Agents&lt;/a&gt; (Anthropic · 2024). Practical patterns for the harness layer, and a useful line between fixed workflows and genuinely agentic systems.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;AI-assisted: I write these with help from AI tools.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>infrastructure</category>
      <category>mlops</category>
    </item>
    <item>
      <title>Why Pay for the Smartest Model Every Time?</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Wed, 04 Nov 2026 15:17:24 +0000</pubDate>
      <link>https://dev.to/konradwright/why-pay-for-the-smartest-model-every-time-28ed</link>
      <guid>https://dev.to/konradwright/why-pay-for-the-smartest-model-every-time-28ed</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://wrightfunctions.com/articles/model-routing/" rel="noopener noreferrer"&gt;wrightfunctions.com&lt;/a&gt;, where the demos are live.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A restaurant doesn't send every order to the head chef. Toast goes to the line cook. The head chef gets the dish that can go wrong. Plenty of AI apps do the opposite: every request, from "what's your refund policy?" to a thorny legal question, goes to the same big, expensive model.&lt;/p&gt;

&lt;p&gt;The price gap is large. As of September 2026, Anthropic's cheapest current model, Claude Haiku 4.5, costs $1 per million input tokens, and its priciest, Claude Fable 5.1, costs $10 (&lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;). That's ten times the price for questions the small one could often answer just as well. This post covers the four common ways to stop overpaying, and the one thing that keeps them honest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Routing: decide before you ask
&lt;/h2&gt;

&lt;p&gt;Model routing puts a quick decision in front of your models. A small, fast classifier reads each request and picks who answers: the cheap model, the expensive one, or a person. That classifier can be a decision model like &lt;a href="https://wrightfunctions.com/articles/jev-decision-models/" rel="noopener noreferrer"&gt;Jev&lt;/a&gt;, which TypeSafe documents &lt;a href="https://docs.typesafe.ai/patterns/intent-routing" rel="noopener noreferrer"&gt;for exactly this&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;It works at scale. When OpenAI launched GPT-5 in August 2025, it described it as fast and deeper-thinking models behind &lt;a href="https://openai.com/index/introducing-gpt-5/" rel="noopener noreferrer"&gt;"a real-time router that quickly decides which to use"&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Researchers at UC Berkeley and Anyscale built routers that learn from human preference data. Their &lt;a href="https://lmsys.org/blog/2024-07-01-routellm/" rel="noopener noreferrer"&gt;RouteLLM project&lt;/a&gt; reported costs cut by more than 85% on one benchmark while keeping 95% of GPT-4's quality. AWS offers a managed router in Bedrock and claims it can cut costs &lt;a href="https://aws.amazon.com/bedrock/intelligent-prompt-routing/" rel="noopener noreferrer"&gt;by up to 30%&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cascades: try cheap, escalate when unsure
&lt;/h2&gt;

&lt;p&gt;A cascade skips the upfront guess. The small model answers every request and says how sure it is. Below a confidence threshold, the request goes up to the big model. The 2023 &lt;a href="https://arxiv.org/abs/2305.05176" rel="noopener noreferrer"&gt;FrugalGPT paper&lt;/a&gt; chained models into a cascade, with a small scoring model judging each answer, and matched the best single model "with up to 98% cost reduction" on its tests.&lt;/p&gt;

&lt;p&gt;The catch is that escalated requests pay twice, and the whole thing rests on calibration: a small model that's confidently wrong never escalates. On the original post, you can drag the threshold.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/model-routing/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Set it too low and wrong answers slip through cheaply. Set it too high and you're paying for both models on most requests. The sweet spot is where quality stops rising, and you find it by measuring.&lt;/p&gt;

&lt;h2&gt;
  
  
  Distillation: teach the small model your job
&lt;/h2&gt;

&lt;p&gt;If the small model is almost good enough, make it better at your specific task. Distillation trains a small model on a big model's answers. The name comes from &lt;a href="https://arxiv.org/abs/1503.02531" rel="noopener noreferrer"&gt;Hinton and colleagues in 2015&lt;/a&gt;, who built on earlier work squeezing big models into small ones.&lt;/p&gt;

&lt;p&gt;A famous early result, &lt;a href="https://arxiv.org/abs/1910.01108" rel="noopener noreferrer"&gt;DistilBERT&lt;/a&gt;, came out 40% smaller and 60% faster while keeping 97% of its teacher's language understanding. In January 2025, DeepSeek released six smaller models, from 1.5 to 70 billion weights, &lt;a href="https://arxiv.org/abs/2501.12948v1" rel="noopener noreferrer"&gt;distilled from its R1 reasoning model&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For your own app, it's the same move: log what the big model answers on your real traffic, then use fine-tuning to train a small model on those answers. Your cascade then escalates less often.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caching: the cheapest answer is one you already gave
&lt;/h2&gt;

&lt;p&gt;Plenty of requests repeat. A semantic cache stores answers and returns one when a new question means the same thing as an old one, matching them by embeddings rather than exact text. Open-source tools like &lt;a href="https://github.com/zilliztech/GPTCache" rel="noopener noreferrer"&gt;GPTCache&lt;/a&gt; do this.&lt;/p&gt;

&lt;p&gt;Providers also offer prompt caching for the part of a prompt that never changes, like a long system prompt or a reference document. On Anthropic's API, a cache hit on most models &lt;a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching" rel="noopener noreferrer"&gt;costs a tenth of the normal input price&lt;/a&gt;. Caching stacks with everything above.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping it honest
&lt;/h2&gt;

&lt;p&gt;Every one of these trades a little quality risk for a lot of money, so the only safe way to run them is with measurement. Keep a test set of real requests with known good answers. Run it against the cheap path and the expensive path before any change to thresholds, routes or distilled models goes out. That's the eval gate from &lt;a href="https://wrightfunctions.com/articles/what-is-ml-infrastructure/" rel="noopener noreferrer"&gt;the first post in this series&lt;/a&gt;, applied to cost: a change ships only if quality holds.&lt;/p&gt;

&lt;p&gt;Then watch it in production, because routing failures are silent: a misrouted hard question gets a confident, wrong, cheap answer, and nothing crashes.&lt;/p&gt;

&lt;p&gt;Traffic also drifts: a new product launch fills your stream with questions the small model has never seen, and quality slides a percent at a time. Sample escalations and non-escalations, grade them, and track the numbers like any other SLO.&lt;/p&gt;

&lt;h2&gt;
  
  
  So why pay for the smartest model every time?
&lt;/h2&gt;

&lt;p&gt;You shouldn't: most of your traffic is toast. Let a line cook handle it, send the hard dishes to the head chef, reuse what you've already cooked, and train the line cook on the chef's recipes. Along with &lt;a href="https://wrightfunctions.com/articles/quantization-living-history/" rel="noopener noreferrer"&gt;quantization&lt;/a&gt; and &lt;a href="https://wrightfunctions.com/articles/jev-decision-models/" rel="noopener noreferrer"&gt;decision models&lt;/a&gt;, it's one of the big levers on the AI bill. The one rule is that you can't cut the cost of quality you aren't measuring.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;Claude pricing&lt;/a&gt; (Anthropic · as of September 2026). Haiku 4.5 at $1 / $5 per million input / output tokens, Fable 5.1 at $10 / $50.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.typesafe.ai/patterns/intent-routing" rel="noopener noreferrer"&gt;Intent Routing&lt;/a&gt; (TypeSafe AI · docs). A decision model classifies each request and sends it to fixed logic, a specialist LLM or a human.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://openai.com/index/introducing-gpt-5/" rel="noopener noreferrer"&gt;Introducing GPT-5&lt;/a&gt; (OpenAI · August 2025). A fast model, a deeper reasoning model, and "a real-time router that quickly decides which to use."&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://lmsys.org/blog/2024-07-01-routellm/" rel="noopener noreferrer"&gt;RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing&lt;/a&gt; (LMSYS · July 2024). Routers trained on preference data cut costs by over 85% on MT Bench while keeping 95% of GPT-4's performance. Paper at arXiv 2406.18665.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.amazon.com/bedrock/intelligent-prompt-routing/" rel="noopener noreferrer"&gt;Amazon Bedrock Intelligent Prompt Routing&lt;/a&gt; (AWS). Routes within one model family and claims up to 30% lower costs without losing accuracy.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2305.05176" rel="noopener noreferrer"&gt;FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance&lt;/a&gt; (Chen, Zaharia &amp;amp; Zou · 2023). Learned cascades, with a small DistilBERT scoring each answer, matched GPT-4 with up to 98% cost reduction, or beat it by 4% at the same cost.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/1503.02531" rel="noopener noreferrer"&gt;Distilling the Knowledge in a Neural Network&lt;/a&gt; (Hinton, Vinyals &amp;amp; Dean · 2015). Names distillation and builds on Caruana's earlier work compressing an ensemble into a single model that's much easier to deploy.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/1910.01108" rel="noopener noreferrer"&gt;DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter&lt;/a&gt; (Sanh et al. · Hugging Face · 2019). 40% smaller, 60% faster, 97% of the language understanding.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2501.12948v1" rel="noopener noreferrer"&gt;DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning&lt;/a&gt; (DeepSeek-AI · January 2025). Six dense models from 1.5B to 70B, distilled from R1 on top of Qwen and Llama.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/zilliztech/GPTCache" rel="noopener noreferrer"&gt;GPTCache&lt;/a&gt; (Zilliz · GitHub). An open-source semantic cache that matches new questions to past ones by embedding similarity.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching" rel="noopener noreferrer"&gt;Prompt caching&lt;/a&gt; (Anthropic · docs). Cache reads cost 10% of the base input price on most models, even less on a few.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;AI-assisted: I write these with help from AI tools.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>inference</category>
      <category>efficiency</category>
      <category>cost</category>
    </item>
    <item>
      <title>Should You Rent AI or Run Your Own?</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Mon, 02 Nov 2026 15:17:23 +0000</pubDate>
      <link>https://dev.to/konradwright/should-you-rent-ai-or-run-your-own-1k2h</link>
      <guid>https://dev.to/konradwright/should-you-rent-ai-or-run-your-own-1k2h</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://wrightfunctions.com/articles/rent-or-run/" rel="noopener noreferrer"&gt;wrightfunctions.com&lt;/a&gt;, where the demos are live.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every team using AI ends up asking this. You can pay per token to use someone else's model through an API. Or you can download an open-weights model and run it yourself, on GPUs you rent or buy.&lt;/p&gt;

&lt;p&gt;Either way, a question goes in and an answer comes out. Which is cheaper, faster or safer?&lt;/p&gt;

&lt;p&gt;The answer is "it depends", and this post is about what it depends on. Prices are as of September 2026 and they move fast. Treat the numbers as a snapshot and the reasoning as the point.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost: a meter or a lease
&lt;/h2&gt;

&lt;p&gt;An API is a meter. Anthropic charges &lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;$2 per million input tokens and $10 per million output&lt;/a&gt; for Claude Sonnet 5. Hosted open models are much cheaper: Together AI serves Meta's Llama 3.3 70B for &lt;a href="https://www.together.ai/pricing" rel="noopener noreferrer"&gt;$1.04 per million&lt;/a&gt;, in or out. You pay for exactly what you use, and nothing when you don't.&lt;/p&gt;

&lt;p&gt;Self-hosting is a lease. Lambda rents H100s from &lt;a href="https://lambda.ai/pricing" rel="noopener noreferrer"&gt;$4.19 an hour each in a two-GPU machine&lt;/a&gt;, and a 70-billion-weight model at 16 bits needs two of them just to hold its 140 GB of weights. Leave them on all month and that's about $6,100, whether they answer ten questions or ten million. So what decides the cost per token is GPU utilization: a GPU earns its keep only while it's busy.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/rent-or-run/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That break-even point is sobering. Hosted open-model prices are so low that two rented GPUs have to stay busy around the clock to compete. Big, steady workloads can get there, especially with heavy batching and quantization. Many teams can't, and they'd be paying for idle GPUs overnight.&lt;/p&gt;

&lt;h2&gt;
  
  
  Buying the box
&lt;/h2&gt;

&lt;p&gt;Owning changes the math: you pay once, plus power. NVIDIA's RTX 5090 launched at &lt;a href="https://nvidianews.nvidia.com/news/nvidia-blackwell-geforce-rtx-50-series-opens-new-world-of-ai-computer-graphics" rel="noopener noreferrer"&gt;$1,999&lt;/a&gt; with &lt;a href="https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/" rel="noopener noreferrer"&gt;32 GB of memory&lt;/a&gt;, enough for a 30-billion-weight model at 4 bits. At its full &lt;a href="https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/" rel="noopener noreferrer"&gt;575 watts&lt;/a&gt; around the clock, and the US average home rate of about 18 cents per kWh in July 2026 (&lt;a href="https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a" rel="noopener noreferrer"&gt;EIA&lt;/a&gt;), the card alone would burn roughly $77 a month in electricity.&lt;/p&gt;

&lt;p&gt;For bigger models, Apple's Mac Studio now takes &lt;a href="https://www.apple.com/mac-studio/specs/" rel="noopener noreferrer"&gt;up to 512 GB of unified memory&lt;/a&gt;, and NVIDIA's DGX Spark packs &lt;a href="https://www.nvidia.com/en-us/products/workstations/dgx-spark/" rel="noopener noreferrer"&gt;128 GB into a desktop box&lt;/a&gt;. None of these match a data-center GPU's throughput, but for one person or a small team, they don't have to.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nobody else's queue
&lt;/h2&gt;

&lt;p&gt;An API's speed is out of your hands. You share the provider's servers, and every account has a rate limit that rises as you use the service more (&lt;a href="https://platform.claude.com/docs/en/api/rate-limits" rel="noopener noreferrer"&gt;Anthropic's tiers&lt;/a&gt; are one example). Your own GPU has no rate limit, only a capacity limit, and at low load it can be very quick: in &lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/1.0.0/performance.html" rel="noopener noreferrer"&gt;NVIDIA's benchmarks&lt;/a&gt;, a small model on one H100 has a time to first token of about 10 milliseconds for a short prompt with a single user. Load it up and each user waits longer, which is the latency-versus-throughput trade that runs through all of serving.&lt;/p&gt;

&lt;h2&gt;
  
  
  It never left the building
&lt;/h2&gt;

&lt;p&gt;This is often the real reason teams self-host. With your own hardware, prompts never leave your network, which may be the only acceptable answer for some data (the &lt;a href="https://wrightfunctions.com/articles/ai-in-government-cloud/" rel="noopener noreferrer"&gt;government cloud post&lt;/a&gt; is all about that).&lt;/p&gt;

&lt;p&gt;APIs have closed some of the gap. OpenAI says &lt;a href="https://developers.openai.com/api/docs/guides/your-data" rel="noopener noreferrer"&gt;API data isn't used to train its models by default&lt;/a&gt;, and Anthropic &lt;a href="https://privacy.claude.com/en/articles/7996868-is-my-data-used-for-model-training" rel="noopener noreferrer"&gt;says the same for its commercial products&lt;/a&gt;. Some providers also offer zero data retention on request. But "they promised" and "it never left the building" aren't the same thing to an auditor.&lt;/p&gt;

&lt;h2&gt;
  
  
  The best models aren't for sale
&lt;/h2&gt;

&lt;p&gt;As of September 2026, the &lt;a href="https://arena.ai/leaderboard/text" rel="noopener noreferrer"&gt;top of the LMArena text leaderboard&lt;/a&gt; is all closed models: their weights aren't published, so you can only reach them through someone else's service. Open models have closed much of the distance. OpenAI's own open-weights gpt-oss-120b &lt;a href="https://openai.com/index/introducing-gpt-oss/" rel="noopener noreferrer"&gt;runs on a single 80 GB GPU&lt;/a&gt; under the Apache 2.0 license.&lt;/p&gt;

&lt;p&gt;For well-defined tasks like classification, extraction and summarizing your own documents, an open model is often enough. For the hardest reasoning, the frontier APIs usually still lead. Either way, test both on a few hundred of your own real questions before deciding. A leaderboard can't tell you that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who gets paged?
&lt;/h2&gt;

&lt;p&gt;Price comparisons leave out one line: with an API, when the model is down, the provider's engineers are on it. With your own GPUs, it's you. Drivers, inference-server upgrades, security patches, new model versions, failed hardware and capacity planning are all yours now. That's a real cost, and it's paid in engineers' time rather than in the hourly GPU price.&lt;/p&gt;

&lt;h2&gt;
  
  
  So, rent or run?
&lt;/h2&gt;

&lt;p&gt;Rent by default. Per-token pricing is cheap, elastic and someone else's pager. Run your own when your data can't leave, when your volume is big and steady enough to keep GPUs busy, or when you need control over the exact model and its latency. Start on the meter; sign the lease when the meter runs all night.&lt;/p&gt;

&lt;p&gt;Many teams end up with both: an API for the hard questions and a small self-hosted model for the steady, sensitive, repetitive work. Whichever you pick, re-run the numbers every few months, because they won't hold still.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;Claude pricing&lt;/a&gt; (Anthropic · as of September 2026). Claude Sonnet 5 at $2 per million input tokens and $10 per million output.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.together.ai/pricing" rel="noopener noreferrer"&gt;Together AI pricing&lt;/a&gt; (Together AI · as of September 2026). Llama 3.3 70B at $1.04 per million tokens, input or output.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://lambda.ai/pricing" rel="noopener noreferrer"&gt;Lambda pricing&lt;/a&gt; (Lambda · as of September 2026). On-demand H100 SXM at $4.19 per GPU-hour in a two-GPU instance ($3.99 in an eight-GPU one), plus tax.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://nvidianews.nvidia.com/news/nvidia-blackwell-geforce-rtx-50-series-opens-new-world-of-ai-computer-graphics" rel="noopener noreferrer"&gt;NVIDIA Blackwell GeForce RTX 50 Series Opens New World of AI Computer Graphics&lt;/a&gt; (NVIDIA · January 2025). The RTX 5090 at $1,999, available January 30, 2025.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/" rel="noopener noreferrer"&gt;GeForce RTX 5090&lt;/a&gt; (NVIDIA · specs). 32 GB of GDDR7 and 575 W total graphics power.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a" rel="noopener noreferrer"&gt;Electric Power Monthly, Table 5.6.A: Average price of electricity to ultimate customers&lt;/a&gt; (US Energy Information Administration · preliminary, released September 2026). US average residential price of 18.31 cents per kWh in July 2026.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.apple.com/mac-studio/specs/" rel="noopener noreferrer"&gt;Mac Studio technical specifications&lt;/a&gt; (Apple · as of September 2026). M5 Ultra with up to 512 GB of unified memory at 1.2 TB/s.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.nvidia.com/en-us/products/workstations/dgx-spark/" rel="noopener noreferrer"&gt;NVIDIA DGX Spark&lt;/a&gt; (NVIDIA). 128 GB of unified memory in a desktop box, for models up to 200 billion parameters.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://platform.claude.com/docs/en/api/rate-limits" rel="noopener noreferrer"&gt;Rate limits&lt;/a&gt; (Anthropic · docs). Usage tiers that rise automatically with usage history and account standing.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/1.0.0/performance.html" rel="noopener noreferrer"&gt;NVIDIA NIM for LLMs performance benchmarks&lt;/a&gt; (NVIDIA · docs). With 200-token prompts, Llama 3.1 8B on one H100 at about 10 ms to first token with one user, and much higher total throughput, with slower answers, at 250.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.openai.com/api/docs/guides/your-data" rel="noopener noreferrer"&gt;Your data&lt;/a&gt; (OpenAI · docs). API data isn't used to train OpenAI's models unless you opt in. Zero data retention needs prior approval.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://privacy.claude.com/en/articles/7996868-is-my-data-used-for-model-training" rel="noopener noreferrer"&gt;Is my data used for model training?&lt;/a&gt; (Anthropic Privacy Center). Inputs and outputs from Anthropic's commercial products aren't used for training by default.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arena.ai/leaderboard/text" rel="noopener noreferrer"&gt;Text Arena leaderboard&lt;/a&gt; (LMArena · as of September 2026). Every model in the top 16 is proprietary; the best open-weights model ranks 17th.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://openai.com/index/introducing-gpt-oss/" rel="noopener noreferrer"&gt;Introducing gpt-oss&lt;/a&gt; (OpenAI · August 5, 2025). Open-weights models under Apache 2.0; the 120B one runs on a single 80 GB GPU.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;AI-assisted: I write these with help from AI tools.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>inference</category>
      <category>cost</category>
    </item>
    <item>
      <title>Can a Government Agency Just Use ChatGPT?</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Sat, 31 Oct 2026 15:17:22 +0000</pubDate>
      <link>https://dev.to/konradwright/can-a-government-agency-just-use-chatgpt-15h0</link>
      <guid>https://dev.to/konradwright/can-a-government-agency-just-use-chatgpt-15h0</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://wrightfunctions.com/articles/ai-in-government-cloud/" rel="noopener noreferrer"&gt;wrightfunctions.com&lt;/a&gt;, where the demos are live.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Somewhere in a federal agency right now, someone has pasted a paragraph into a chatbot to make it sound less like a memo. Then someone from the security office walks by and asks: &lt;em&gt;where did that paragraph just go?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The model is the same model everywhere. What changes in government is the boundary around it: who runs the servers, whose staff can see the prompts, what the data is allowed to touch, and who signed off.&lt;/p&gt;

&lt;p&gt;Everything below is as of September 2026. This field moves fast and the rules move with it, so check the dates before you rely on any of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nothing runs without a signature
&lt;/h2&gt;

&lt;p&gt;Before any system goes live in an agency, an official has to grant an authority to operate, a formal statement that its remaining risk is acceptable. Doing a full security review of every cloud vendor for every agency would take forever, so FedRAMP does it once and agencies &lt;a href="https://www.fedramp.gov/archive/2020-05-07-how-agencies-can-reuse-a-fedramp-authorization/" rel="noopener noreferrer"&gt;reuse the result&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For years, FedRAMP rated services &lt;a href="https://www.fedramp.gov/archive/2017-11-16-understanding-baselines-and-impact-levels/" rel="noopener noreferrer"&gt;Low, Moderate or High&lt;/a&gt;, by how badly a breach would hurt. New rules that &lt;a href="https://www.fedramp.gov/2026/providers/updating/changes/" rel="noopener noreferrer"&gt;took effect on July 4, 2026&lt;/a&gt; renamed FedRAMP "authorization" to "certification", partly so nobody confuses FedRAMP's review with the agency's own ATO, and &lt;a href="https://www.fedramp.gov/notices/0004" rel="noopener noreferrer"&gt;moved Low, Moderate and High into Classes B, C and D&lt;/a&gt;, with a new pilot Class A. You'll still see "FedRAMP High" on many vendor pages for a while. The Defense Department adds its own impact levels: &lt;a href="https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-dod-il5" rel="noopener noreferrer"&gt;IL4 and IL5&lt;/a&gt; for controlled unclassified information, and &lt;a href="https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-dod-il6" rel="noopener noreferrer"&gt;IL6&lt;/a&gt; for classified work up to Secret.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four places a model can live
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A certified AI service: FedRAMP &lt;a href="https://www.fedramp.gov/ai/" rel="noopener noreferrer"&gt;fast-tracked chat assistants&lt;/a&gt; from August 2025 to April 2026. ChatGPT Enterprise, Gemini for Government and Perplexity were certified. One condition: anything a model learned from customer data had to stay inside the customer's environment unless the customer said otherwise.&lt;/li&gt;
&lt;li&gt;A model service in a government region: AWS lists &lt;a href="https://aws.amazon.com/compliance/services-in-scope/FedRAMP/amazon-bedrock-models/" rel="noopener noreferrer"&gt;Claude, Llama, OpenAI and other models on Bedrock&lt;/a&gt; at FedRAMP High (now Class D) and IL4/5 in its GovCloud regions. Microsoft has had &lt;a href="https://devblogs.microsoft.com/azuregov/azure-openai-authorization/" rel="noopener noreferrer"&gt;Azure OpenAI at FedRAMP High since 2024&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;A classified cloud: Bedrock runs in AWS's &lt;a href="https://aws.amazon.com/blogs/publicsector/amazon-bedrock-launches-with-claude-3-5-sonnet-in-the-aws-top-secret-cloud/" rel="noopener noreferrer"&gt;Top Secret&lt;/a&gt; and &lt;a href="https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-bedrock-is-now-available/" rel="noopener noreferrer"&gt;Secret&lt;/a&gt; regions, and Microsoft brought &lt;a href="https://devblogs.microsoft.com/azuregov/advancing-ai-capabilities-in-azure-for-u-s-government-secret-and-top-secret-clouds/" rel="noopener noreferrer"&gt;GPT-5.2 to its Secret and Top Secret clouds&lt;/a&gt; in January 2026.&lt;/li&gt;
&lt;li&gt;Your own hardware: run an open-weights model inside your own boundary, even an air-gapped one. Google sells &lt;a href="https://cloud.google.com/blog/topics/hybrid-cloud/gemini-is-now-available-anywhere" rel="noopener noreferrer"&gt;Gemini on air-gapped hardware&lt;/a&gt;, so "no internet" no longer means "open models only."&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;▶ &lt;strong&gt;Interactive demo:&lt;/strong&gt; &lt;a href="https://wrightfunctions.com/articles/ai-in-government-cloud/" rel="noopener noreferrer"&gt;try it on the original post&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Cost is less of a barrier than it used to be. In August 2025, GSA, the government's central buyer, struck deals putting &lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-announces-new-partnership-with-openai-delivering-deep-discount-to-chatgpt-08062025" rel="noopener noreferrer"&gt;ChatGPT Enterprise&lt;/a&gt; and &lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-strikes-onegov-deal-with-anthropic-08122025" rel="noopener noreferrer"&gt;Claude&lt;/a&gt; in front of agencies for $1 each for a year, and &lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-google-announce-gemini-onegov-agreement-08212025" rel="noopener noreferrer"&gt;Gemini for Government&lt;/a&gt; for 47 cents through 2026. In September 2026, GSA said ChatGPT would move to &lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-expands-onegov-ai-offerings-with-discounted-openais-chatgpt-09102026" rel="noopener noreferrer"&gt;50% off pay-as-you-go pricing&lt;/a&gt; from October 1.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rules on top
&lt;/h2&gt;

&lt;p&gt;Two memos from the White House budget office (OMB) in April 2025 set the ground rules. &lt;a href="https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-21-Accelerating-Federal-Use-of-AI-through-Innovation-Governance-and-Public-Trust.pdf" rel="noopener noreferrer"&gt;M-25-21&lt;/a&gt; requires testing, impact assessments and human oversight for "high-impact" AI, meaning AI whose output is a principal basis for decisions with a legal, material, binding or significant effect on rights or safety. &lt;a href="https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-22-Driving-Efficient-Acquisition-of-Artificial-Intelligence-in-Government.pdf" rel="noopener noreferrer"&gt;M-25-22&lt;/a&gt; has agencies write contracts that "permanently prohibit" vendors from training public or commercial models on non-public agency inputs and outputs without the agency's explicit consent.&lt;/p&gt;

&lt;p&gt;Vendor choice is now a policy question too. On February 27, 2026, agencies were directed to stop using Anthropic's models. After a federal court paused that directive, &lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-issues-statement-on-anthropic-preliminary-injunction-04032026" rel="noopener noreferrer"&gt;GSA restored them&lt;/a&gt; in April 2026, though an appeals court &lt;a href="https://www.defenseone.com/threats/2026/09/anthropic-lawsuit-supply-chain-risk/416252/" rel="noopener noreferrer"&gt;upheld the Pentagon's risk label&lt;/a&gt; on September 25. The ops lesson applies to any vendor: design so you can swap the model behind your app without re-certifying everything around it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Proving what happened
&lt;/h2&gt;

&lt;p&gt;In government, "it worked" isn't enough. You have to show what the model saw and did. In practice that means an audit log of prompts, responses and actions, tied to a person.&lt;/p&gt;

&lt;p&gt;OMB's May 2026 logging memo, &lt;a href="https://www.whitehouse.gov/wp-content/uploads/2026/05/M-26-14-Ensuring-Effective-and-Efficient-Agency-Logging-and-Network-Visibility-to-Defend-Against-Evolving-Cyber-Threats.pdf" rel="noopener noreferrer"&gt;M-26-14&lt;/a&gt;, doesn't mention prompts, but it replaced the old M-21-31 rules and requires logs that show who did what to which data, searchable for at least 6 months and retrievable for 12. Those logs will hold sensitive text, so they need the same protection as the data itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The agent problem
&lt;/h2&gt;

&lt;p&gt;Chat is the easy case: a person reads every answer. Agents, which act on their own, are arriving in government clouds too. AWS brought &lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/05/bedrock-agentcore-launch-aws-govcloud-us/" rel="noopener noreferrer"&gt;Bedrock AgentCore to GovCloud&lt;/a&gt; in May 2026, and Microsoft's &lt;a href="https://devblogs.microsoft.com/azuregov/advancing-ai-innovation-in-azure-government/" rel="noopener noreferrer"&gt;Foundry Agent Service reached Azure Government&lt;/a&gt; in September.&lt;/p&gt;

&lt;p&gt;An agent that can file, send or change records needs least privilege and human approval for anything it can't undo. Least privilege is already &lt;a href="https://csrc.nist.gov/glossary/term/least_privilege" rel="noopener noreferrer"&gt;a baseline federal control&lt;/a&gt;, and both matter more when prompt injection can steer the agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  So can they just use ChatGPT?
&lt;/h2&gt;

&lt;p&gt;Increasingly, yes, just not the one on your phone. The same families of models now run in certified services, government regions, classified clouds and air-gapped racks. The work is the plumbing it always was: pick the environment that matches the data, keep the prompts inside the boundary, log everything, and keep a person in charge of anything irreversible. It's the same job as anywhere else, with more paperwork.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/rev5/agency-authorization/" rel="noopener noreferrer"&gt;Agency Authorization&lt;/a&gt; (FedRAMP). Cloud providers work with an agency to get an Authority to Operate; the agency grants it.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/archive/2020-05-07-how-agencies-can-reuse-a-fedramp-authorization/" rel="noopener noreferrer"&gt;Do Once, Use Many - How Agencies Can Reuse a FedRAMP Authorization&lt;/a&gt; (FedRAMP · May 6, 2020). A provider is authorized once and any federal agency can reuse the security package; each agency still issues its own ATO.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/archive/2017-11-16-understanding-baselines-and-impact-levels/" rel="noopener noreferrer"&gt;Understanding Baselines and Impact Levels in FedRAMP&lt;/a&gt; (FedRAMP · 2017). Low, Moderate and High, by how severe the harm from a breach would be.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/2026/providers/updating/changes/" rel="noopener noreferrer"&gt;What's changing in the 2026 Consolidated Rules&lt;/a&gt; (FedRAMP · effective July 4, 2026). Authorization becomes Certification, Low, Moderate and High become Classes B, C and D (plus a new pilot Class A), and new Rev5 certifications stop on June 11, 2027.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/notices/0004" rel="noopener noreferrer"&gt;Initial Outcome from RFC-0020 FedRAMP Authorization Designations&lt;/a&gt; (FedRAMP · February 25, 2026). Class A is a new pilot baseline; Class B takes Li-SaaS and Low, Class C Moderate, Class D High.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-dod-il5" rel="noopener noreferrer"&gt;DoD Impact Level 5&lt;/a&gt; (Microsoft Learn, quoting DISA's Cloud Computing SRG). CUI that needs more protection than IL4, plus National Security Systems.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-dod-il6" rel="noopener noreferrer"&gt;DoD Impact Level 6&lt;/a&gt; (Microsoft Learn, quoting DISA's Cloud Computing SRG). Reserved for information classified up to Secret.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/ai/" rel="noopener noreferrer"&gt;FedRAMP AI Prioritization&lt;/a&gt; (FedRAMP · August 2025 to April 2026). Fast-tracked chat assistants for federal workers. ChatGPT Enterprise and API, Gemini for Government and Perplexity came out certified.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.amazon.com/compliance/services-in-scope/FedRAMP/amazon-bedrock-models/" rel="noopener noreferrer"&gt;Amazon Bedrock models in scope for FedRAMP&lt;/a&gt; (AWS · updated September 18, 2026). Which models on Bedrock are FedRAMP High and DoD IL4/5 in AWS GovCloud (US).&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://devblogs.microsoft.com/azuregov/azure-openai-authorization/" rel="noopener noreferrer"&gt;Azure OpenAI Service now authorized for all U.S. Government data classification levels&lt;/a&gt; (Microsoft · April 16, 2025). FedRAMP High and IL4/5 in 2024, IL6 in the Secret cloud and Top Secret in early 2025.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.amazon.com/blogs/publicsector/amazon-bedrock-launches-with-claude-3-5-sonnet-in-the-aws-top-secret-cloud/" rel="noopener noreferrer"&gt;Amazon Bedrock launches with Claude 3.5 Sonnet in the AWS Top Secret cloud&lt;/a&gt; (AWS · January 13, 2025). Bedrock generally available in the AWS Top Secret cloud.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-bedrock-is-now-available/" rel="noopener noreferrer"&gt;Amazon Bedrock is now available&lt;/a&gt; (AWS · November 22, 2025). Bedrock at the Secret level.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://devblogs.microsoft.com/azuregov/advancing-ai-capabilities-in-azure-for-u-s-government-secret-and-top-secret-clouds/" rel="noopener noreferrer"&gt;Announcing GPT-5.2 Availability in Azure for U.S. Government Secret and Top Secret Clouds&lt;/a&gt; (Microsoft · January 15, 2026). GPT-5.2 available in Azure's Secret and Top Secret clouds.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://cloud.google.com/blog/topics/hybrid-cloud/gemini-is-now-available-anywhere" rel="noopener noreferrer"&gt;Gemini is now available anywhere&lt;/a&gt; (Google Cloud · August 28, 2025). General availability of Gemini on Google Distributed Cloud air-gapped.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.ecfr.gov/current/title-32/subtitle-B/chapter-XX/part-2002/subpart-B/section-2002.14" rel="noopener noreferrer"&gt;32 CFR 2002.14 Safeguarding&lt;/a&gt; (eCFR · Controlled Unclassified Information rule). CUI Basic is categorized at no less than the moderate confidentiality impact level, the floor the demo uses.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.fedramp.gov/2026/agencies/use/classes/" rel="noopener noreferrer"&gt;Certification Classes - FedRAMP Consolidated Rules for 2026&lt;/a&gt; (FedRAMP · 2026). Classes describe assurance, not impact level; Class C is presumed adequate for most Low or Moderate agency systems.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-announces-new-partnership-with-openai-delivering-deep-discount-to-chatgpt-08062025" rel="noopener noreferrer"&gt;GSA Announces New Partnership with OpenAI, Delivering Deep Discount to ChatGPT Gov-Wide Through MAS&lt;/a&gt; (GSA · August 6, 2025). ChatGPT Enterprise for $1 per agency for a year.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-strikes-onegov-deal-with-anthropic-08122025" rel="noopener noreferrer"&gt;GSA Strikes Another OneGov Deal with Anthropic to Offer Claude AI to all Branches of Gov for Just $1&lt;/a&gt; (GSA · August 12, 2025). Claude for Enterprise and Claude for Government for $1, across all three branches.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-google-announce-gemini-onegov-agreement-08212025" rel="noopener noreferrer"&gt;GSA, Google Announce Transformative 'Gemini for Government' OneGov Agreement&lt;/a&gt; (GSA · August 21, 2025). Gemini for Government at $0.47 per agency.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-expands-onegov-ai-offerings-with-discounted-openais-chatgpt-09102026" rel="noopener noreferrer"&gt;GSA Expands OneGov AI Offerings with Discounted, Consumption-Based Access to OpenAI's ChatGPT&lt;/a&gt; (GSA · September 10, 2026). A 50% discount on token-based ChatGPT usage, no minimums, in a 27-month offer expected to start October 1, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-21-Accelerating-Federal-Use-of-AI-through-Innovation-Governance-and-Public-Trust.pdf" rel="noopener noreferrer"&gt;M-25-21: Accelerating Federal Use of AI through Innovation, Governance, and Public Trust&lt;/a&gt; (OMB · April 3, 2025). Chief AI Officers, and minimum practices (testing, impact assessment, human oversight) for high-impact AI.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-22-Driving-Efficient-Acquisition-of-Artificial-Intelligence-in-Government.pdf" rel="noopener noreferrer"&gt;M-25-22: Driving Efficient Acquisition of Artificial Intelligence in Government&lt;/a&gt; (OMB · April 3, 2025). Contracts must permanently prohibit training public or commercial models on non-public agency inputs and outputs, absent explicit agency consent.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-issues-statement-on-anthropic-preliminary-injunction-04032026" rel="noopener noreferrer"&gt;GSA Issues Statement on Anthropic Preliminary Injunction&lt;/a&gt; (GSA · April 3, 2026). After the February 27 removal and a March 26 court order, GSA restores Anthropic to its prior status.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://qz.com/pentagon-anthropic-supply-chain-risk-designation-090326" rel="noopener noreferrer"&gt;Pentagon is still calling Anthropic a supply chain risk despite court ruling&lt;/a&gt; (Quartz · September 3, 2026). After an August court ruling, a Defense Department official said the Pentagon's supply chain risk designation of Anthropic remained in force.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.defenseone.com/threats/2026/09/anthropic-lawsuit-supply-chain-risk/416252/" rel="noopener noreferrer"&gt;Anthropic loses legal fight to shed DOD's designation as a 'supply-chain risk'&lt;/a&gt; (Defense One · September 25, 2026). The D.C. Circuit upheld the Pentagon's supply-chain-risk designation of Anthropic, 2–1; Anthropic said it was considering further review.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.whitehouse.gov/wp-content/uploads/2026/05/M-26-14-Ensuring-Effective-and-Efficient-Agency-Logging-and-Network-Visibility-to-Defend-Against-Evolving-Cyber-Threats.pdf" rel="noopener noreferrer"&gt;M-26-14: Ensuring Effective and Efficient Agency Logging and Network Visibility to Defend Against Evolving Cyber Threats&lt;/a&gt; (OMB · May 22, 2026). Rescinds M-21-31. Logs searchable for at least 6 months and retrievable for at least 12.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/05/bedrock-agentcore-launch-aws-govcloud-us/" rel="noopener noreferrer"&gt;Amazon Bedrock AgentCore is now available in AWS GovCloud (US-West)&lt;/a&gt; (AWS · May 5, 2026). The agent runtime arrives in a government region.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://devblogs.microsoft.com/azuregov/advancing-ai-innovation-in-azure-government/" rel="noopener noreferrer"&gt;Advancing trusted AI innovation with Agents and new Azure OpenAI models in Azure Government&lt;/a&gt; (Microsoft · September 9, 2026). Microsoft Foundry Agent Service is now available in Azure Government.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://csrc.nist.gov/glossary/term/least_privilege" rel="noopener noreferrer"&gt;Least Privilege (NIST glossary)&lt;/a&gt; (NIST CSRC, citing SP 800-53 Rev. 5). Each entity gets only the resources and authorizations its function needs.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;AI-assisted: I write these with help from AI tools.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>government</category>
      <category>security</category>
      <category>compliance</category>
    </item>
    <item>
      <title>Community as Infrastructure</title>
      <dc:creator>Konrad Wright</dc:creator>
      <pubDate>Fri, 15 May 2026 04:10:08 +0000</pubDate>
      <link>https://dev.to/konradwright/community-as-infrastructure-40k2</link>
      <guid>https://dev.to/konradwright/community-as-infrastructure-40k2</guid>
      <description>&lt;p&gt;If you're &lt;em&gt;alone&lt;/em&gt;, plugging away at a keyboard with &lt;strong&gt;no one&lt;/strong&gt; to share your concepts with, you've got an &lt;em&gt;infrastructure&lt;/em&gt; problem.&lt;br&gt;
Here's the great news, &lt;u&gt;infrastructure problems CAN be solved&lt;/u&gt;!&lt;/p&gt;

&lt;p&gt;Fading are the days where email lists matter, and the growth of a community will far outweigh than another spam message in someone's inbox. I've seen games on Steam that are polished gems, go on the wayside because someone believed &lt;em&gt;"If I build it, they will come"&lt;/em&gt;. You have to rely on more than just wishlists and vibes. There is hope though.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Let's get into it:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Build your landing&lt;/strong&gt;: This could be a simple Discord Server. It's well known (and sometimes well hated) but widely used for now. Your opinions aside, people ARE there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Get your "First 5"&lt;/strong&gt;: This can be your irl friends, if you're blessed to have any. However, assuming you don't, find people in your field. These should be the people on the same path as you. You're reading this, so you're well on your way. Just read someone else's posts, send them a DM if you think you could grab a beer with them. That's it, kids on a playground do this just as easily. Don't overthink it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Co-Improve your landing&lt;/strong&gt;: Have a community discord bot that each member works on, kit that out with some cool features. Make a website with the boys/gals. Make your space not just useful, but &lt;em&gt;interesting&lt;/em&gt;! Add a Minecraft server for easy engagement. (It's silly but works wonders for member stickiness).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Work on your funnel&lt;/strong&gt;: Now that you've got a group of 5-ish people working on things, congratulations you've got 5x the footprint to get some traffic to your chill pad. Market it however you want, there's a thousand "Join me and my group of highly motivated people" groups that charge a &lt;strong&gt;$100+&lt;/strong&gt; monthly fee. &lt;em&gt;And there will be thousands more!&lt;/em&gt; Or if you and your 5 are working on a joint project because you found great chemistry, just build with an audience and funnel people to watch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Engage in the grind-set&lt;/strong&gt;: Now that you have the attention of a group of people you can perform the &lt;em&gt;Neo-Dev-Cycle&lt;/em&gt; of &lt;strong&gt;"MARKET, SELL, BUILD"&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;You don't need a massive audience. You don't need a viral moment. You need &lt;strong&gt;five people&lt;/strong&gt; who give a damn, a shared space to build in, and the discipline to keep showing up. That's the whole plan. Infrastructure first — everything else follows.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If you feel like you need it, here's your formal invitation to DM me. If you're building cool things, and are willing to see my cool things, that's &lt;strong&gt;all&lt;/strong&gt; we need to be friends.&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>webdev</category>
      <category>discord</category>
      <category>productivity</category>
      <category>writing</category>
    </item>
  </channel>
</rss>
