DEV Community: mlops

How to Choose the Right AI Software Development Company in 2026?

hoor ali — Wed, 14 Oct 2026 14:10:44 +0000

To choose the right AI software development company, look for a partner that starts with your business problem, has shipped AI to production in your industry, owns data and MLOps end to end, follows a recognized AI risk framework, and prices against outcomes rather than hours. Ask for live references, a paid discovery phase, and a clear plan for what happens after launch.

Almost every business is now experimenting with AI. In McKinsey's 2025 State of AI survey, 88% of respondents said their organization uses AI in at least one function, yet only 7% said it was fully scaled. The gap between a promising pilot and a system that earns its keep is usually the partner you pick. This guide walks through how to close it.

What does an AI software development company do?
An AI software development company designs, builds, and maintains software that uses machine learning, large language models (LLMs), or AI agents to automate decisions and tasks. Typical AI software development services include:

AI strategy and discovery: finding use cases with a measurable return.
Data engineering: cleaning, labeling, and piping the data models depend on.
Custom model and LLM development: fine-tuning, retrieval-augmented generation (RAG), and agent workflows.
AI integration: adding AI features to your existing apps, CRM, or ERP.
MLOps and support: monitoring, retraining, and cost control after launch. This is different from a team that simply uses AI in software development, such as coding assistants that help engineers write code faster. Most good partners do both, but you are hiring for the first.

7 criteria for choosing an AI software development company:

They start with the problem, not the model:
Strong partners ask about your workflow, KPIs, and budget before they mention GPT, Claude, or Gemini. If the first call is a tech demo, expect a solution looking for a problem.
Proven production work in your industry:
Ask for two or three case studies with numbers: hours saved, error rates cut, revenue gained. Then ask to speak with those clients. A pilot that never went live is not a reference.
Data readiness and security:
AI is only as good as its data. The company should assess your data quality early and explain where your data is stored, who can access it, and whether it is used to train third-party models. Look for SOC 2 or ISO 27001 practices and compliance with rules such as GDPR or HIPAA where they apply.
Responsible AI and governance:
Ask how they test for bias, hallucinations, and failure cases. Partners who align with the NIST AI Risk Management Framework can show you a documented process for mapping, measuring, and managing AI risk.
Model-agnostic architecture:
The best model today may not be the best in six months. Prefer a team that can swap LLM providers or run open-source models without a rebuild, so you avoid vendor lock-in and control costs.
MLOps and post-launch support:
Models drift as data changes. Confirm who monitors accuracy, how often models are retrained, and what the support SLA covers. Get the cost of running the system, including API and cloud fees, in writing.
Clear ownership and transparent pricing:
Your contract should state that you own the code, prompts, fine-tuned weights, and data. Pricing should be broken into phases with defined deliverables.

Red flags to watch for:

Guaranteed accuracy figures before seeing your data.
No paid discovery phase; a fixed quote after one call.
Vague answers on data privacy or model training.
Every project uses the same tool, whatever the problem.
No plan for maintenance after launch.

Common engagement and pricing models:
Most AI software development companies offer four engagement models. A paid discovery phase or proof of concept tests feasibility in 2–6 weeks, and it doubles as a test of the team, not just the idea. A fixed-scope project suits well-defined features with clean data, though change requests can get expensive.
Time and materials fit evolving products and R&D, but it needs strong sprint reporting to keep costs visible. A dedicated AI team works best for long-term roadmaps, as long as you have your own product owner to steer it. Whichever model you choose, start with a small, paid discovery phase; it tells you more about how a company works than any sales deck.

Frequently asked questions:

How much does it cost to hire an AI software development company?

Costs depend on scope and data readiness. A discovery phase or proof of concept is usually the smallest commitment, while a production system with integrations and MLOps costs much more. Ask for phased quotes and a monthly estimate of running costs such as API and cloud fees.

How long does an AI software project take?

A proof of concept typically takes 2–6 weeks. A production-ready AI feature often takes 3–6 months, depending on data quality, integrations, and compliance reviews.

What is the difference between AI software development services and using AI in software development?

AI software development services build products that contain AI, such as chatbots, prediction engines, or agents. AI in software development refers to tools that help engineers write, test, and review code faster. A good partner uses the second to deliver the first more efficiently.

Should I build an in-house AI team or outsource?

Outsourcing is faster when you need specialized skills now or are testing a first use case. An in-house team makes sense once AI is core to your product. Many companies start with a partner and hire internally as the system matures.

From Model Experiments to Applications That People Can Use

Satavisha Dutta — Wed, 14 Oct 2026 10:26:40 +0000

A model that performs well during experimentation still needs a reliable way to serve predictions.

A practical learning project could involve training a small model, exposing it through a Python API, validating inputs, recording results, and creating a simple user interface.

This teaches more than model training: it introduces software architecture, data validation, API design, and deployment.

Developers exploring this broader skill set can look at Full-Stack Python For AI Developers, an Eduonix project covering Python, web development, and AI-related projects.

How do you prefer to serve machine learning models: APIs, batch pipelines, or interactive applications?

Getting Hands‑On with Gemini 4 Argon: What It Is, Why It Matters, and How to Plug It Into Your Stack

Naveen Malothu — Wed, 14 Oct 2026 08:58:37 +0000

1️⃣ What was released / announced

DeepMind just opened up Gemini 4 Argon, the latest iteration of their Gemini family of large language models. Argon is a multimodal, instruction‑tuned LLM with up to 1.6 trillion parameters, accessible via a RESTful API and a gRPC endpoint. In short, it’s the next‑generation “ChatGPT‑style” model that can understand text, code, and images, and it’s now available to developers under a pay‑as‑you‑go pricing model.

2️⃣ Why it matters

From an engineering standpoint there are three reasons this is worth a look right now:

Performance vs. cost sweet spot – Benchmarks released by DeepMind show Argon beating Gemini 1.5 on reasoning and code‑generation tasks while using ~30 % less compute per token. That translates to lower latency and cheaper inference for production workloads.
Multimodal input – Argon can ingest PNG/JPEG images alongside text prompts. This opens up use‑cases like “describe this diagram and write the corresponding SQL query” without stitching together separate vision and language services.
Built‑in safety guardrails – The model ships with a tuned safety layer that reduces hallucinations on factual queries by ~45 % compared to older Gemini releases. For MLOps pipelines that need deterministic outputs, that’s a big win.

If you’re already running LLM‑driven features—code assistants, summarizers, or chat‑bots—Argon lets you get better results without a massive infrastructure overhaul.

3️⃣ How to use it

Below is a minimal end‑to‑end example that I ran on a fresh Ubuntu 22.04 VM. The steps assume you have a DeepMind API key (found in the Gemini console) and Python 3.10+ installed.

Step 1 – Install the client library

DeepMind ships an official google‑gemini pip package that abstracts the HTTP calls.

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install google-gemini

Step 2 – Write a helper wrapper

I like to keep the client thin and inject retry logic via tenacity. This snippet also shows how to send an image payload.

import os
from google_gemini import GeminiClient
from tenacity import retry, stop_after_attempt, wait_exponential

API_KEY = os.getenv("GEMINI_API_KEY")
client = GeminiClient(api_key=API_KEY, model="gemini-4-argon")

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def ask_gemini(prompt: str, image_path: str | None = None):
    if image_path:
        with open(image_path, "rb") as f:
            img_bytes = f.read()
        response = client.generate(
            prompt=prompt,
            images=[{"bytes": img_bytes, "mime_type": "image/png"}],
            temperature=0.2,
            max_output_tokens=512,
        )
    else:
        response = client.generate(
            prompt=prompt,
            temperature=0.2,
            max_output_tokens=512,
        )
    return response.text

# Example usage
if __name__ == "__main__":
    q = "Explain the architecture shown in the diagram and generate a Terraform snippet that provisions the described resources."
    print(ask_gemini(q, image_path="arch.png"))

Step 3 – Deploy at scale

For production you’ll want to run this behind a Kubernetes service with autoscaling based on request latency. I typically use a Deployment with a sidecar that exports Prometheus metrics from the client library. Here’s a trimmed manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gemini‑proxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gemini‑proxy
  template:
    metadata:
      labels:
        app: gemini‑proxy
    spec:
      containers:
        - name: api
          image: myregistry.com/gemini‑proxy:latest
          env:
            - name: GEMINI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: gemini‑secret
                  key: api-key
          resources:
            limits:
              cpu: "2"
              memory: "4Gi"
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

Couple of tips:

Enable HTTP/2 on the gRPC endpoint for lower overhead.
Use Google Cloud Load Balancing with Cloud CDN if you expect high read‑only traffic (e.g., caching static prompts).
Set temperature=0 for deterministic code generation pipelines; bump it up only for creative chat experiences.

4️⃣ My take

Having built the inference layer for Griffin AI Tech’s “Code‑Assist” product, I’m constantly juggling latency, cost, and reliability. Gemini 4 Argon hits a sweet spot that lets us replace our older open‑source LLM stack (a 7B model fine‑tuned on code) with a managed service that reduces per‑token cost by ~0.15 ¢ and cuts 99‑th‑percentile latency from 850 ms to ~420 ms.

That said, it’s not a silver bullet. The model’s multimodal capabilities are still early‑stage; the image‑to‑text pipeline sometimes returns overly verbose descriptions. For mission‑critical pipelines I pair Argon with a lightweight verification model (e.g., a distilled 3B LLM) that checks for hallucinations before we commit the output to downstream services.

From an ops perspective, the biggest win is the managed safety layer. Previously we spent weeks building custom prompt‑engineering tricks to keep the model from generating insecure code snippets. Argon’s built‑in guardrails let us simplify our prompt templates and focus on scaling the API gateway.

If you’re evaluating LLM providers for a new product, I’d put Argon at the top of the shortlist—especially if you need image understanding or tighter cost controls. My recommendation is to start with a sandboxed POC (the code snippet above gets you there in under an hour), measure latency and token usage under realistic loads, and then decide whether to bake it into your core micro‑service or keep it as a “burst‑only” fallback.

Happy experimenting!

Math 2.0: Turning Terence Tao’s Vision into Practical AI‑Ready Tools

Naveen Malothu — Tue, 13 Oct 2026 10:58:03 +0000

1. What was released / announced

On June 1 2026, Terence Tao, together with a team of Caltech researchers, published a 120‑page PDF titled “Math 2.0”. The document is more than a philosophical essay – it outlines a concrete, open‑source stack that blends formal mathematics, symbolic computation, and large language models (LLMs) to create a machine‑readable mathematics layer. In short, it ships a Python package called math2py, a set of Jupyter notebooks, and a REST API that lets you query “mathematical objects” (theorems, proofs, definitions) as first‑class data.

2. Why it matters

As AI engineers we constantly wrestle with two opposing forces:

The need for rigor – when we embed mathematical models in production (e.g., safety‑critical control systems, finance, or scientific simulations), we need guarantees that the equations we code actually match the theory.
The speed of iteration – modern LLMs can draft code snippets for differential equations in seconds, but they often hallucinate or miss subtle domain constraints.

Math 2.0 bridges that gap. By exposing a formal knowledge graph of mathematics that LLMs can query, it gives us a way to:

Auto‑validate symbolic derivations against a trusted theorem database.
Pull exact definitions (e.g., “Lebesgue measure”) into code without copy‑pasting ambiguous textbook snippets.
Generate reproducible notebooks where every theorem citation is a live API call, making peer review trivial.

For DevOps and AI‑infrastructure teams, this means we can bake mathematical compliance checks into CI pipelines, turning a previously manual review step into an automated test.

3. How to use it

Below is a minimal end‑to‑end example that shows how to pull a theorem, verify a proof sketch, and embed the result in a CI job. The steps assume you have Python 3.11+ and Docker installed.

Step 1 – Install the library

# Inside your project’s virtualenv
pip install math2py==0.3.0

Step 2 – Query a theorem via the Python client

from math2py import MathClient

client = MathClient(api_key="YOUR_MATH2PY_TOKEN")

# Grab the statement of the Banach Fixed‑Point Theorem
banach = client.theorem(name="Banach Fixed‑Point Theorem")
print(banach.statement)
# → "Let (X, d) be a complete metric space …"

Step 3 – Validate a symbolic proof

Suppose we have a tiny symbolic derivation for a contraction map f. We can ask the service to prove that f satisfies the theorem’s hypotheses.

from sympy import symbols, Function
from math2py import ProofChecker

x = symbols('x')
f = Function('f')(x)
# Example: f(x) = 0.5 * x
expr = 0.5 * x

checker = ProofChecker(client)
result = checker.verify_contraction(expr, metric='euclidean')
print(result.passed)
# True → the service found a formal proof that the map is a contraction.

Step 4 – Add a CI gate (GitHub Actions example)

name: Math Compliance
on: [push, pull_request]
jobs:
  check-math:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install deps
        run: pip install math2py
      - name: Run compliance script
        env:
          MATH2PY_TOKEN: ${{ secrets.MATH2PY_TOKEN }}
        run: |
          python scripts/verify_math.py

verify_math.py would contain the snippet from Step 3 and exit with a non‑zero code if any check fails, causing the PR to be blocked.

Step 5 – Deploy the REST endpoint (optional)

If you need a self‑hosted version (e.g., for GDPR‑restricted workloads), the PDF ships a Docker‑Compose file:

curl -O https://teorth.github.io/tao-web/slides/math-2-0-caltech-2026.pdf
# Inside the extracted folder:
docker compose up -d

The service then listens on http://localhost:8080/v1/ and can be fronted by your existing API gateway.

4. My take

I’ve been building AI‑driven pipelines at Griffin AI Tech for the past three years, and the biggest pain point has always been trust. Our customers ship models that predict drug efficacy, autonomous‑driving control laws, and financial risk metrics. When a model’s loss function is derived from a theorem (e.g., a Sobolev embedding inequality), the usual practice is a manual “look‑and‑feel” sanity check. That’s error‑prone and slows down releases.

Math 2.0 feels like the missing glue between the symbolic world of mathematics and the operational world of CI/CD. The fact that it’s open‑source, has a clear HTTP API, and ships ready‑to‑run Docker images means we can start treating theorems as first‑class assets in the same way we treat Docker images or Helm charts.

From a DevOps angle, I’m most excited about the prospect of policy‑as‑code for math. Imagine a policy that says “All models that use Fourier analysis must reference the Plancherel theorem and pass the ProofChecker validation”. That policy can be codified in Rego (OPA) and enforced automatically.

There are still open questions:

Coverage – the current knowledge graph covers ~12 k theorems, great for analysis but thin on modern stochastic calculus.
Performance – the proof‑checking service is CPU‑heavy; you’ll want to cache results or run it on a dedicated node.
Security – exposing a symbolic engine over HTTP can be abused for denial‑of‑service; rate‑limiting and sandboxing are a must.

Overall, I think Math 2.0 is a pragmatic step toward what many of us have been dreaming about: a world where you can write model = train(..., loss=BanachFixedPointLoss()) and have the platform prove that the loss satisfies the required contraction properties before the container ever lands on a GPU.

If you’re curious, grab the PDF, spin up the Docker image, and try the math2py client on a small proof of concept. You’ll quickly see how a few lines of code can replace hours of manual verification – and that, to me, is the real value proposition.

Happy hacking, and keep your pipelines mathematically sound!

Why Most Company AI Projects Get Stuck and How to Fix Them?

Varnika kannareddy — Tue, 13 Oct 2026 07:04:33 +0000

Lots of companies try AI. Very few get real results from it. The problem is usually not the AI itself. It's the messy work around it.
Here are four common problems, explained simply, and what fixes them.

The project never leaves the test stage A team builds a cool AI demo. Everyone likes it. Then nothing happens. Nobody owns it, nobody updates it, and it slowly gets forgotten. Fix: Set a clear goal before you start, like "reduce late deliveries by 10%." Then plan how the model will be run, updated and checked after launch. This is what MLOps services help with.
The data is a mess Sales, finance and operations each keep their own numbers, and none of them match. People waste hours every week copying data into Excel. If you feed messy data to AI, you get wrong answers faster. Fix: Bring your data into one clean, trusted place and keep it updated automatically. This is called AI data modernization, and it should come before any AI model.
Nobody uses the AI The AI works, but its results sit in a separate dashboard nobody opens. People keep working the old way because it's easier. Fix: Put AI results right inside the tools people already use every day, like the CRM, the order screen or the support system.
Nobody can explain the AI's decisions A customer asks, "Why was my request rejected?" and the only answer is "the system decided." That isn't good enough for customers or regulators. AI chatbots can also give confident but wrong answers. Fix: Keep a record of every AI decision, check regularly that it's still accurate and fair, and use RAG architecture development so chatbots answer from your own approved documents. Responsible AI governance services make this part of everyday work, not an afterthought. The simple takeaway Good AI comes from four basics: clean data, clear goals, AI inside daily tools, and proper checks. To see how this works in practice, here's a helpful look at enterprise AI and data engineering services, from AI data modernization to MLOps services, RAG architecture development and responsible AI governance services.

The same H100 hour cost $1.26 or $0.22 per million tokens

Qaiser Mehdi — Tue, 13 Oct 2026 00:46:48 +0000

A GPU hour has one price. What it delivers does not. Here is one measurement that shows how big the gap can be.

The setup. One rented H100 SXM 80GB on vast.ai at $2.689 per hour, 20 vCPUs. vLLM 0.27.1 (official image). Qwen2.5-0.5B-Instruct in fp16, unique prompts so nothing was cached. 28 September 2026. Every number below comes from vLLM's own /metrics counters, and they matched what my client recorded exactly.

Two loads, three rounds each.

One request at a time: each request decoded at 648-649 tokens/s. The whole server produced 589-593 tokens/s. That is about $1.26 per 1M output tokens.
8 requests at once: each request got a bit slower, 549-573 tokens/s. But the whole server produced 3,194-3,348 tokens/s. That is about $0.22-0.23 per 1M output tokens.

Same GPU, same hour, same price. About 5.5x apart.

The common mistake. Many cost estimates take one request's speed and divide the hourly price by it. At 8 at once, one request ran at about 549-573 tokens/s, which would price the hour at about $1.30-1.36 per 1M tokens. The real cost was $0.22-0.23. In each round the whole server made 5.65-5.84x what one request suggests.

So pricing from one request's speed can overstate cost by about 5.7x at this load. At other loads the gap is different. That is the point.

What decides your real cost.

How many requests your server actually runs at once, not what a benchmark ran.
The idle time inside the hour. A GPU that waits still bills, so cost per token is the hourly price over everything the server produced in that hour, idle included.
For scale-to-zero, the boot. On this same H100, one cold wake was about 75 of 77.3 billed seconds before the server could answer.

Same ratio on a smaller card. On a T4 with the same model and test, at 8 at once each request decoded at 122-124 tokens/s while the whole server made 701-708 tokens/s. Again about 5.7x.

What I can and cannot say. This is a small model, one GPU, three rounds per load. A 7B or 70B model, another server, or longer prompts will give different numbers. I am not saying your cost is $0.22. I am saying the number depends on your load, and you can only know it by measuring your own server.

The method, the raw numbers and the open-source tool are here, if useful: github.com/qaisermehdi3-coder/qvunex

If you have measured cost per token on your own servers at different loads, I would like to compare.

Ready is not warm: one SGLang cold start on a T4

Qaiser Mehdi — Mon, 12 Oct 2026 22:51:26 +0000

I measure what GPU time actually buys. One number surprised me this month, so here it is with the raw timeline.

The setup. One T4 on Colab. SGLang 0.5.21. Qwen2.5-0.5B-Instruct in fp16. I started a clock just before launching the server, the way a scale-to-zero platform starts billing when a worker is created. Then 8 short chat requests, then 60 seconds of health checks, then stop. One run, 5 October 2026.

The timeline.

0 s: server launched (billing starts)
~371 s: prefill CUDA graph capture finished (it took 287.7 s)
386.7 s: /health returns 200, log says "ready to roll"
~563 s: first chat request finishes (it took about 176 s)
~579 s: second request finishes (about 16 s), the other six within the same second
639.6 s: billing stops

After that, 8 more requests on the same server took 0.08 to 0.22 s each.

What this means for the bill. If you split the bill at the health check, boot looks like 60% of it (386.7 of 639.6 s). But the first user got an answer about 563 s in. That is 88% of the bill gone before anyone was served.

So "ready" was not the end of the wait. It was about two thirds of it.

What I can and cannot say.

The server's own counters matched my client exactly: 8 requests, 264 prompt tokens, 80 output tokens.
The log does not say what the first 176 s were spent on, so I am not claiming a cause.
Each health check in the tail shows up in the SGLang log as a one-token prefill on the GPU, but the request counters do not move. So "the GPU did work" and "a request was served" are different things.
For comparison, on vLLM 0.27.1 on the same card, the 8 requests after ready took 5.8 s in total. Boot there was about 165 of 171 billed seconds.
This is one run on one small card. A bigger GPU or another model can look different. That is the point: measure your own.

If you run scale-to-zero, record three times, not one:

billing start to health check OK
billing start to the first finished request
the idle tail after the last request

Then price the whole billed time, not just the part where requests were running.

The method and the open-source tool I used are here, if useful: github.com/qaisermehdi3-coder/qvunex

If you have measured this on other cards or servers, I would like to see your numbers.

From GenAI Pilots to Production: How MLOps Closes the Scaling Gap

Tanvi Tanwar — Mon, 12 Oct 2026 18:35:49 +0000

Generative AI is the topic of conversation in conference rooms worldwide. Business leaders push for generative AI adoption. Thus, soon, engineering teams roll up their sleeves and architect stunning proof-of-concept prototypes. That is how they highlight value.
But there is an alarming chasm between the excitement of a shiny GenAI pilot and the reality of an enterprise-grade production implementation. In fact, industry publications routinely lament how many GenAI initiatives do not make it to a customer-facing product. Gartner specifically projects that through 2026, organizations will abandon 60% of AI projects due to poor data management practices.
We can refer to this challenge as the scaling gap. Overcoming this divide, B2B companies need to ingest their CI/CD pipelines with MLOps, nowadays better known as Large Language Model-specific ML Operations or LLMOps. This post will thus explain how, with MLOps embedded into the IT functions, organizations take AI experiments that are typically fragile and turn them into scalable, safe, and monetizable engines.

Generative AI Reality: Not All Pilots Can Scale, and That is Not Necessarily a Bad Omen

Corporate speculations, as you contemplate them hard enough, reveal an awful truth. It is that wrapping an API could take days, while building an operating app (with fewer issues) takes months. When a multitude of companies aggressively force pilots into production, it means no excuses will work when stakeholders encounter frustrations, as errors propagate, workflows break, and reports refuse to reflect corporate etiquette.
Cloud inference costs can erupt over the enterprise budget unexpectedly. Latency, especially when noisy, also erodes an end-user's knowledge and models. Thus, hallucinating facts or clearly fictitious details end up in high-ticket clients' inboxes. That backfires almost immediately. Such flaws also uncover difficulties in using AI without hurting legitimate corporate intel and communication.
Surely, data scientists can develop amazing models in data science labs. However, when IT applies those models to the data center, the systems are instantly overwhelmed by real traffic. Every data engineering veteran at firms offering GenAI consulting services will tell you that pilots overlook data drift, model degradation, and ongoing monitoring.
If no framework exists for the C-Suite to see their AI investments and gauge the actual value creation, capital waste due to the uncontrolled pilot-to-prod rush, often with no ROI, will be inevitable.

What Does Re-Engineering the Foundation with LLMOps Involve? Why Does It Matter?

In order to save stranded AI projects, technology executives take them through the specialized MLOps process. MLOps primarily applies traditional software engineering disciplines to the machine learning lifecycle. Generative AI-specific MLOps also considers the exceptional parameter scale and lack of structure in enterprise data.
In addition to conceptual business alignment checks, sophisticated AI companies such as Databricks, NVIDIA, and Hugging Face are really bringing these operational shells to life. For instance, they first host the mandatory options to normalize product registries and also offer a lot more automatic CI/CD, i.e, continuous integration and delivery) for product or service-oriented AI. In other words, they operate for stable, secure, verifiable, and trackable production AI enablement.

What Are the Common Building Blocks for Operationalizing Foundation Models?

Engineers harness foundational language. That comprises concepts that MLOps consulting services embrace, such as RAG pipelines and vector storage engines. Their objective is to tether a foundation model to private, real-time enterprise data.
So, instead of retraining a trillion-parameter language model from the ground up, you can simply employ parameter-efficient fine-tuning (PEFT) to train a model very quickly. Today's MLOps workflows also track token consumption. They can not only manage prompt variations but also apply rigid semantic guardrails. That is why your firm can prevent contaminated or incorrect outputs from ultimately reaching end users.

Industry Use Cases: Demonstrating MLOps in Action

Industry-specific MLOps is vital to productionize generative AI for safer and more scalable use.
For the financial services industry, banks can employ GenAI to analyze large volumes of regulatory and market intelligence reports. When they do so, MLOps dispels disastrous regulatory compliance hallucinations. The underlying monitoring and RAG pipelines facilitate that in real time.
In healthcare, GenAI for clinical trials necessitates MLOps that would enable models to adjust to dynamic patient data. It must also proceed through continuous, HIPAA-compliant retraining with immediate rollback enabled in case of bias.
At the same time, worldwide e-commerce stores apply MLOps practices. They include load balancing and semantic caching. Therefore, they can undertake precisely segmented, personalized marketing at scale. Latency spikes and cloud cost issues are also less worrisome, since MLOps and GenAI ensure the best online shopping conversion rate even when budgets are finite.

Conclusion

Generative AI is a great accelerator for most domains in reporting, marketing, customer support, and hypothetical scenario tests. However, despite impressive demonstrations, escaping from the pilot phase to enter mainstream production is still not that easy for GenAI projects.
Concerning the scalability, reliability, and safety gaps, instead of wasting time on irrelevant or technical workarounds, leaders must embrace MLOps.
Rigorous application of software engineering disciplines, often adhering to monitoring, RAG pipeline development, and semantic guardrail principles, helps you change a GenAI pilot's scope for taking on bigger challenges. Whether firms work in tech, finance, healthcare, media, or retail, MLOps capabilities will be central to their production-grade AI deployment ambitions in 2026 and beyond.

NVIDIA GPU Operator on RKE2: Passthrough to Kubeflow (The Sports Car in the Garage)

George Ezejiofor — Mon, 12 Oct 2026 16:31:01 +0000

Episode 7 of the homelab build series. The foundation is poured, the walls are up, the alarms are wired, the storage floor is down and the safe is locked. Now for the fun room: three NVIDIA GPUs, one in each machine, so GPU capacity stays highly available when any one of them is down, managed by the NVIDIA GPU Operator and wired into Kubeflow.

nvidia-smi side by side on the three GPU nodes, one per datacenter.

Introduction

Three GPUs. Three datacenters. Zero schedulable.

I had three NVIDIA cards in three machines, and Kubernetes swore I had none. I asked for one GPU in a test pod and got this:

0/21 nodes are available: 21 Insufficient nvidia.com/gpu.

The Proxmox hosts could see them. lspci could see them. nvidia-smi on the host lit up. But to the cluster, three cards' worth of Ampere silicon were parked, idle and invisible. That's the most expensive kind of idle there is.

And the whole MLOps platform from the earlier episodes (Kubeflow, Katib, KServe, the training pipelines) was built to train and serve real models. You can't schedule what the scheduler doesn't know exists.

Fitting the cards took a Saturday. Getting the kernel module, the container runtime, the device plugin and the scheduler to agree took a lot longer, and taught me thirteen different ways a GPU can hide from Kubernetes.

Here's where it ended up. Three RTX A1000s, one in each "datacenter", which here means one in each of three mini PCs on a shelf. Lose a whole machine and two cards are still there. The NVIDIA GPU Operator builds the driver, wires up the runtime and advertises the cards on its own. And the part I'm proud of: when the whole lab loses power, all three cards come back and pass their CUDA self-test without me touching anything.

This is how I got there, every trap along the way, and the parts that still aren't done.

And at the end there's a real job on a real card: a 7-billion-parameter language model answering at about 30 tokens a second, against 4 on the CPU beside it.

Where this sits in the build

Foundation: Terraform on Proxmox.
Framing: Ansible building the five-master HA RKE2 cluster.
Security: Kyverno, KubeArmor and Trivy, with Falco parked for now to save CPU.
Workshop: Tekton, Harbor, SonarQube and Chains.
Storage floor: Rook-Ceph and CloudNativePG.
The safe: HA HashiCorp Vault.
The sports car: this one.

This is where the cluster stops being general-purpose Kubernetes and gains an accelerator pool. Before the operator went in, the three GPU workers were scheduled like any other node, by CPU and memory, with no idea a GPU was attached.

The repo side is small: one namespace, one chart, one values file, and one ClusterPolicy the operator builds everything from.

kubernetes-addons/0-boostrap/gpu-operator/
  namespace.yaml             # gpu-operator namespace, privileged Pod Security
  gpu-operator-values.yaml   # annotated Helm values; every trap fix lives here
  dcgm-custom-metrics.yaml   # extra DCGM fields for the dashboard
  nvidia-dcgm-dashboard.yaml # Grafana dashboard
  examples/                  # multi-GPU best-practice manifests

The hardware in the garage

The cards: three NVIDIA RTX A1000s. Ampere GA107GL, 8 GB of GDDR6, 2,304 CUDA cores, about 50 W. It draws all its power from the PCIe slot, with no extra connector, which is exactly why it fits where a bigger card wouldn't. The node label says it plainly:

$ kubectl get node rke2-worker-dc1-gpu-a \
    -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product} {.metadata.labels.nvidia\.com/gpu\.memory}MiB'
NVIDIA-RTX-A1000 8188MiB

The hosts: three Minisforum MS-01s. Intel i9-13900H, 96 GB of DDR5, and, rarely for a mini PC, one real half-height PCIe slot wired to the CPU: x16 in length, x8 electrically, which is plenty for this card. That slot is the whole reason this build exists.

Datacenter	Proxmox host	GPU worker	GPU	VRAM
dc1	proxmox-dc-1	rke2-worker-dc1-gpu-a	RTX A1000	8 GB
dc2	proxmox-dc-2	rke2-worker-dc2-gpu-a	RTX A1000	8 GB
dc3	proxmox-dc-3	rke2-worker-dc3-gpu-a	RTX A1000	8 GB

Three cards, three machines: PCIe passthrough from Proxmox host to RKE2 worker.

That's 24 GB of GPU memory across the fleet. One 8 GB card runs a 4-bit 7B model with room to spare; I measure that further down. It's great for inference, fine for small fine-tunes, and marginal for training from scratch, which is a rented-A100 job anyway.

Why one card per datacenter instead of three in one box? The same answer as storage: failure domains. If a machine goes down, GPU work still has two healthy places to land. In a homelab, rack density is worth nothing and HA is worth a lot.

Why bother with Kubernetes when the GPU is right there on the host? Three reasons. Each project gets its CUDA version from an image tag instead of eight CUDA installs fighting each other. The runtime becomes a pinned image, so results are reproducible. And the scheduler is the only sane way to say "this job gets one card, that one gets another".

PCIe passthrough: from Proxmox to the VM

Each card is bound to vfio-pci on its Proxmox host and passed into exactly one VM. An Ansible playbook handles the nouveau blacklist, the vfio binding, IOMMU and the reboot. All three cards sit at the same address, 0000:01:00, which keeps the Terraform pleasingly uniform.

The passthrough itself is Terraform with the bpg provider, through a gpu_pci_hosts map per datacenter. One gotcha deserves its own sentence: raw hostpci needs the root@pam Proxmox user. A normal API token can't assign raw PCI devices, and the error doesn't tell you that.

cd terraform/environment/prod-rke2/bpg
terraform plan  -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu
# 0 to add, 2 to change, 0 to destroy   (in place: +machine=q35, +hostpci0)
terraform apply -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu

machine=q35 and hostpci0 only take effect after a full power cycle of the VM, not a soft reboot.

The best moment of the whole build came here. Once dc1 worked, I passed the cards through on dc2 and dc3 and rebooted them. The card appeared, the node got labelled, and the operator rolled the entire stack onto both nodes by itself. The first node took me twelve hours. The other two took about ninety seconds each, and I didn't touch them.

Three hardware notes that earned their place:

Driver version is pinned to 570.133.20. The version stays put. What has to follow the kernel is the module, which the driver container rebuilds against whatever kernel the VM is running.
No vGPU, no MIG. The A1000 supports neither.
Secure Boot never gets a say. The GPU VMs boot with SeaBIOS, not UEFI, so the unsigned NVIDIA module loads without any key enrolment. The Proxmox hosts themselves do have Secure Boot on; it doesn't matter, because the module loads inside the VM. If your VMs use UEFI with Secure Boot, you'll have to sign the module or switch it off there.

What the GPU Operator actually does

What the GPU Operator runs: a tower where every floor waits for the one below.

One Helm release of nvidia/gpu-operator (chart v26.3.2) runs a set of DaemonSets from a single ClusterPolicy. Sticking with the garage:

Component	In plain terms	What it really does
Operator	the garage manager	watches the `ClusterPolicy` and keeps everything below in shape
NFD (Node Feature Discovery)	the inspector	scans hardware and labels nodes that have an NVIDIA PCI device
GFD (GPU Feature Discovery)	the spec sheet	adds product, memory, count, driver and CUDA labels
Driver	the mechanic	compiles and loads `nvidia.ko` against the running kernel, inside a container
Container Toolkit	the licence plate	registers the `nvidia` runtime with containerd
Device Plugin	the parking attendant	advertises `nvidia.com/gpu` to the kubelet
DCGM Exporter	the dashboard	per-GPU metrics for Prometheus
Validator	the MOT	runs a CUDA test on every card and reports the result

The dependency chain is the thing to remember. NFD labels the nodes that have a GPU. Everything else only runs where that label is. So the driver, toolkit and device plugin land on the three GPU workers and nowhere else. No NFD label, no driver. No driver, no GPU.

And every other piece waits for the driver. When the bottom of that tower is broken, everything above it sits in Init. That gave me the rule that saved the most time: debug from the driver up. There's no point staring at the device plugin while the driver is still crash-looping.

What I deliberately don't run: the MIG manager (the A1000 has no MIG), the vGPU manager (that's for licensed shared vGPU), the sandbox and Kata pieces (the card is passed through once, at Proxmox), GPUDirect Storage (no hardware for it), and a standalone DCGM (the exporter brings its own, and two would fight over one socket).

The stack lives in the gpu-operator namespace with privileged Pod Security. The driver, toolkit and device plugin load kernel modules and touch host paths, and a restricted namespace blocks every one of them.

kubectl get pods -n gpu-operator mid-cascade: a wall of Init:0/1 and one very sad nvidia-driver-daemonset. The "before" shot.

From PCIe slot to scheduled pod

From PCIe slot to scheduled pod: once per node, then once per pod.

Once per node:

The worker boots with the GPU present as a PCI device.
NFD labels the node.
The driver pod compiles nvidia.ko for the running kernel and loads it.
The toolkit registers the nvidia runtime with containerd.
The device plugin tells the kubelet the node has nvidia.com/gpu: 1.
GFD adds nvidia.com/gpu.product=NVIDIA-RTX-A1000 and friends.
The DCGM exporter starts and metrics flow.

Then once per pod:

A workload asks for nvidia.com/gpu: 1.
Kyverno adds the taint toleration and the GPU node rule (more on that in the next section).
The scheduler finds a GPU node with a free card and binds the pod.
The NVIDIA runtime injects the device and driver libraries.
nvidia-smi inside the container sees its card.

All of it is automatic once the operator is in. That's the whole argument against a hand-rolled Ansible role that builds kernel modules per node and breaks on every kernel upgrade.

Keeping GPU pods on GPU nodes

Two bits of metadata mark these nodes, and you want both:

A label so workloads can find them: nodepool=gpu, plus everything NFD and GFD add.
A taint so ordinary workloads can't drift onto them and starve a GPU job: nvidia.com/gpu=true:NoSchedule.

$ kubectl get node rke2-worker-dc1-gpu-a -o jsonpath='{.spec.taints}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"true"}]

Normally that means every GPU pod needs a toleration, a node selector and a resource limit, and forgetting any one gives you a confusing failure. On this cluster a Kyverno policy, gpu-pods-to-gpu-nodes, does the first two for you. Any pod that asks for nvidia.com/gpu is mutated to tolerate the taint and require a nodepool=gpu node. So a GPU pod here needs exactly one line:

resources:
  limits:
    nvidia.com/gpu: 1

The operator's own DaemonSets carry the toleration in the values file, and NFD's master also needs a control-plane toleration:

daemonsets:
  priorityClassName: system-node-critical
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

nfd:
  master:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Equal
        value: ""
        effect: NoSchedule

Prove it works

The quickest proof is NVIDIA's own CUDA sample. It adds two vectors on the card and prints one line if the whole chain held: driver, runtime, device plugin, scheduler.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
  namespace: default
spec:
  restartPolicy: OnFailure
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
      resources:
        limits:
          nvidia.com/gpu: 1

Look at what isn't there: no toleration, no node selector, no runtimeClassName. The only GPU line is the limit. The securityContext blocks have nothing to do with the GPU; my default namespace enforces the restricted Pod Security Standard, so every pod there needs them.

$ kubectl apply -f cuda-vectoradd.yaml
$ kubectl get pod cuda-vectoradd -o wide
NAME             READY   STATUS      RESTARTS   AGE   NODE
cuda-vectoradd   0/1     Completed   0          16s   rke2-worker-dc1-gpu-a

$ kubectl logs cuda-vectoradd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Sixteen seconds from apply to Completed, image pull included. The pod landed on a GPU worker without being told to, and you can see what Kyverno wrote into it on the way in:

$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.tolerations}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Equal","value":"true"}]

$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.affinity.nodeAffinity}'
{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"nodepool","operator":"In","values":["gpu"]}]}]}}

Delete the pod when you're done. It has finished, so it isn't holding the card, but it's clutter.

kubectl logs cuda-vectoradd ending in Test PASSED. The hero shot.

The runtime: getting containerd to know about GPUs

containerd doesn't know what a GPU is. Something has to register an nvidia runtime that injects the device and driver libraries into containers. The classic way is to patch containerd's config.toml, and RKE2 regenerates that file from a template, throwing your patch away on every restart.

Two things make it work here. CDI is on, so the toolkit writes a device spec at /etc/cdi/nvidia.yaml that containerd reads at container start:

cdi:
  enabled: true
  nriPluginEnabled: false

And the toolkit is pointed at RKE2's own containerd rather than the default one:

toolkit:
  env:
    - name: CONTAINERD_CONFIG
      value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock
    - name: CONTAINERD_RUNTIME_CLASS
      value: nvidia
    - name: CONTAINERD_SET_AS_DEFAULT
      value: "true"

The subtle bit is that CONTAINERD_CONFIG points at the generated config.toml, not the .tmpl. The toolkit re-applies its setup every time it starts, so when a reboot regenerates the config without nvidia, the toolkit simply puts it back. That's what makes it heal itself across reboots. Point it at the template and the live config sits there without a GPU runtime until someone restarts rke2-agent by hand.

Because nvidia is the default runtime, workloads here don't need runtimeClassName: nvidia. On a cluster without that setting they do, and that one line is what confuses people copying manifests between clusters.

Sharing the cards

Each A1000 advertises exactly one nvidia.com/gpu, so by default one pod owns the whole card.

Mode	What it does	Isolation	Here
Exclusive	one pod owns the card	full	✅ what runs today
Time-slicing	several pods take turns on the card	compute only, memory is shared	available, not switched on
MIG	hardware partitions with their own memory	full	❌ the A1000 has no MIG
MPS	processes share one GPU context	partial	available, not used

The node labels confirm it: nvidia.com/gpu.replicas=1 and sharing-strategy=none on all three.

Time-slicing is a ConfigMap plus a patch to the ClusterPolicy:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  a1000: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4        # one A1000 looks like four schedulable GPUs

kubectl patch clusterpolicies.nvidia.com cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"a1000"}}}}'

Create the ConfigMap first, then patch. I ran the patch on its own once, and the device plugin's replacement pod sat in Init with configmap "time-slicing-config" not found, which took that node's card from one schedulable GPU to zero until I cleared the reference again. The DaemonSet replaces one pod at a time, so the rollout stalled there and the other two cards kept working.

Before you turn it on, remember it's oversubscription, not partitioning:

No memory isolation. Four pods share 8 GB. Four 2 GB models fit; a fourth 3 GB model dies with CUDA out-of-memory.
No fault isolation. One pod that wedges the card affects its neighbours.
Sharing, not guarantees. Right for notebooks and bursty inference, wrong for anything with a latency target.

My plan is to time-slice dc1 and dc2 and leave dc3 whole for jobs that need all 8 GB.

How reliable is it?

Here's what the GPU tier survives, and the evidence for each:

What goes wrong	What happens	How I know
A whole machine goes down	Its card is gone, and so is any job that was running on it; new GPU pods land on the other two	One card per machine (dc1, dc2, dc3), each advertising `nvidia.com/gpu: 1`
The whole lab loses power	Every node rebuilds its GPU stack and re-tests its card, hands-off	After the last power cut, the toolkit, device plugin, DCGM exporter and validators came back on all three nodes by themselves, and the CUDA validator Succeeded on every card
RKE2 regenerates containerd's config on reboot	The toolkit re-applies the `nvidia` runtime	`CONTAINERD_CONFIG` points at the generated file, not the template
Someone forgets the GPU toleration	Kyverno adds it, plus the GPU node affinity	`gpu-pods-to-gpu-nodes` mutation policy
A notebook is abandoned with a card	It should be stopped after 24 hours idle and the card freed	Config only: `ENABLE_CULLING=true`, `CULL_IDLE_TIME=1440`. I have not seen it fire. The lab is powered off overnight, and two notebooks sat on cards until I stopped them by hand
A driver or runtime problem	The operator stops and reports it	`ClusterPolicy` state `ready`; validators gate everything above the driver

And the honest limits. "Highly available" here means GPU capacity: a job running on a card dies with its machine and has to start again elsewhere. The operator itself is a single replica; if it's down, running GPU pods carry on and nothing new gets configured. A kernel upgrade on a GPU node still needs a driver rebuild (see the traps), there are no GPU alerts wired yet, and nothing caps how many cards one namespace can take. Those last three are on the list at the end.

kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator: three validators, one per datacenter, all Completed.

The traps, from the driver up

Thirteen ways a GPU hides from Kubernetes, by layer, from the driver up.

Thirteen of them, grouped by layer, in the order you should debug them. Most of them bit me. A few I only know from other people's write-ups, and those say so. Every piece waits on the driver, so a broken foundation makes everything above it look broken too.

The driver

The driver gets OOM-killed, exit 137. The log dies right after an nvidia-drm warning, then nvidia-smi: not found, and it crash-loops. It looks like a driver bug. It isn't: the driver container compiles the kernel module, running gcc against the kernel headers, and a 512Mi limit kills it mid-compile.

driver:
  resources:
    limits:
      cpu: "2"
      memory: 4Gi     # 512Mi OOMs the kernel-module compile

failed to install nvidia kernel module: distro 'ubuntu24.04' not supported. The driver image ships pre-built modules for a fixed list of distro and kernel pairs, and mine wasn't on it. Build from source instead:

driver:
  kernelModuleType: auto      # open modules first, proprietary as fallback
  usePrecompiled: false       # build from source, about 3 minutes on first boot

nvidia: Unknown symbol __register_chrdev after a host kernel upgrade. The cached module was built for the old kernel. Force a rebuild:

kubectl -n gpu-operator rollout restart daemonset/nvidia-driver-daemonset

The real lesson is a runbook entry, not a config change: a GPU node's kernel upgrade is a special operation, with a driver restart on the checklist. A plain power cut doesn't need it, because the kernel hasn't changed.

Missing kernel headers. The driver builds against the running kernel, so a kernel upgraded without linux-headers-$(uname -r) fails the build. The Ansible role for the GPU pool installs them, and the error is plain in the driver pod log.

The validator

nvidia-validator: command not found, exit 127. Version skew, and the error tells you nothing. Older charts shipped the validator as a separate image; the 26.x chart puts the binary inside the main operator image. Values written for one layout against a chart of the other produce a missing binary. Pin the chart (v26.3.2 here), deploy it from the NVIDIA repo rather than a stale vendored copy, and check what's actually running with helm list -n gpu-operator, not what your values file assumes.

The toolkit and runtime

no runtime for "nvidia" is configured. The RKE2 trap the generic docs miss. The toolkit looks for /run/containerd/containerd.sock; RKE2 runs its own containerd at /run/k3s/containerd/containerd.sock. The fix is the toolkit.env block above.

Failed to initialize NVML: Driver/library version mismatch inside a pod, while the host shows the same version. The toolkit had patched config.toml directly and RKE2 regenerated it. CDI or the generated-config setting fixes it.

KubeArmor's AppArmor profile blocks the toolkit. The same family of trap that once bit Cilium here: the toolkit can't write what it needs, containerd is silently never configured, and pods report the nvidia RuntimeClass as missing. If you hit it, run the toolkit container unconfined. Today it runs fine under its own KubeArmor-generated profile.

The toolkit writes the CDI spec and containerd never reads it. Pods schedule, but nvidia-smi hangs. CDI needs containerd 1.7 or newer, so check what your RKE2 release ships before you switch it on. I didn't hit this one: my containerd is 2.x.

Missing runtimeClassName: nvidia on a cluster where nvidia isn't the default runtime. The pod schedules, runs with no GPU injected, and looks exactly like broken hardware. Not an issue here because of CONTAINERD_SET_AS_DEFAULT, which is why it's worth knowing: the requirement depends on the cluster.

The device plugin and scheduling

Insufficient nvidia.com/gpu while three healthy GPUs sit idle. The driver and toolkit were running, but the device plugin was Pending:

0/21 nodes are available: 21 node(s) had untolerated taint
{nvidia.com/gpu: true}, that the pod didn't tolerate.

No device-plugin pod means nothing tells the kubelet about the card. The upstream chart ships the right toleration, but a trimmed values file copied from a blog about an untainted cluster quietly drops it. Same trap one layer down: NFD's master without its control-plane toleration never labels anything, so nothing else schedules either.

NFD takes a minute on first install. The operator comes up before NFD's first scan, so for about a minute it looks like the operator is running and nothing is happening. It isn't broken. Don't start changing things.

Telemetry

The DCGM exporter gets OOM-killed right after logging DCGM successfully initialized!. It loads its whole field set into memory and 256Mi isn't enough:

dcgmExporter:
  resources:
    limits:
      memory: 1Gi

All of these fixes live in gpu-operator-values.yaml, .

Who holds a card, and how to get it back

This is the part I'd most have wanted to read, because it's invisible until a GPU job won't schedule.

nvidia.com/gpu can't be oversubscribed. Without time-slicing, a pod with nvidia.com/gpu: 1 owns that physical card for its whole life. Three cards means three GPU pods at once, full stop.

That makes long-lived notebooks the quiet problem. At one point two Kubeflow notebooks were each holding a whole A1000, one of them for three and a half months, while asking for 0.6 and 1.2 CPUs. A notebook that asks for less than one CPU and holds an entire GPU is the shape of the problem. It also quietly capped a Katib sweep at one GPU trial at a time. And they cost more than cards: together they used over half of their namespace's CPU quota, enough to make a daily training pipeline time out because its last job could never be scheduled.

The claim is on the pod, not the Notebook, the StatefulSet or the volume. So stopping a notebook frees the card. Kubeflow's "stop" sets an annotation, the controller scales to zero, and the pod goes away:

kubectl -n <profile> annotate notebook <name> \
  kubeflow-resource-stopped="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

I sampled the free-GPU count every ten seconds while doing it:

[01] sts replicas=0   pod=Terminating   FREE GPUs=1
[02] sts replicas=0   pod=gone          FREE GPUs=2

Look at the first line. The notebook was "stopped" by every API you'd think to check, and the card was still claimed, because the pod hadn't finished terminating. replicas=0 isn't proof a card came back. On a cluster where KubeArmor can wedge pods in Terminating, that matters. The reliable check counts from the pods and cross-checks the scheduler:

# what pods claim
kubectl get pods -A -o json | jq '[.items[] | select(.status.phase=="Running" or .status.phase=="Pending")
  | .spec.containers[].resources.limits["nvidia.com/gpu"] // "0" | tonumber] | add'

# what the scheduler has allocated on a node
kubectl describe node <gpu-node> | awk '/Allocated resources/,/Events/' | grep nvidia.com/gpu

As I write this, both GPU notebooks are stopped and all three cards are free.

Culling is on as a safety net. The notebook controller stops notebooks that have been idle for 24 hours. It stops, it never deletes: the volume and every file survive, and restarting is one click. What's lost is in-memory kernel state. A running cell counts as busy, so a long training run won't be culled. And the setting is global and not GPU-aware, so it catches abandoned notebooks; it doesn't manage contention. The better lever is not giving notebooks a GPU by default.

⚠️ One more surprise: nvidia.com/gpu isn't in the profile namespaces' ResourceQuota, and a quota only limits what it names. So GPU use per namespace is unbounded even though CPU and memory are capped.

Putting the GPUs to work

Making nvidia.com/gpu schedulable was the prerequisite, not the goal.

Who on the platform can ask for a card

Nothing on this platform needs a GPU to run. Every Kubeflow component is a CPU service; what changes is the work users launch through them. Here's each door to a card, and whether anything has walked through it yet:

Platform piece	How it asks for a card	Has it used one here?
Kubeflow Notebooks	a GPU dropdown in the spawner form	Yes, and too well: two notebooks held cards while idle
Katib	`nvidia.com/gpu` in the trial template	No. Both sweeps ran on CPU
Kubeflow Trainer	`resourcesPerNode` on a `TrainJob`	No. The runtimes are installed (`torch-distributed`, three torchtune LLM fine-tunes); no job has run
KServe	a GPU limit on the predictor	No. The one live model is XGBoost on CPU
Kubeflow Pipelines	a GPU limit on a single step	No. The daily pipeline is all CPU
Spark	executor GPU settings plus the RAPIDS plugin	No, and not planned: these jobs are small
A plain pod	one line, `nvidia.com/gpu: 1`	Yes: the LLM benchmark below

Everything else in the data platform (Kafka, Airflow, MLflow, the databases) never touches a card and shouldn't.

So the honest summary is two real users so far: notebooks, which took cards and did nothing with them, and one benchmark pod, which took a card and drove it at 98%.

Hyperparameter search with Katib

Katib runs an Experiment that launches trials as pods, each reporting a metric back through an injected sidecar. Getting it to run at all taught me more than the GPUs did.

It couldn't run in any Kubeflow profile namespace. An Experiment sat at Created with zero trials, and the UI only said "Couldn't find any successful Trial." The real error was two levels down, on a ReplicaSet:

pods "<exp>-tpe-..." is forbidden:
  minimum memory usage per Container is 16Mi, but request is 10Mi

Katib asks for 10Mi of memory for its suggestion pods, and the Kyverno-generated LimitRange in every profile namespace has a 16Mi floor. Eleven of twelve algorithms had no explicit resources, so eleven of twelve couldn't schedule. Fixing those exposed the same problem on the trial pods, because the metrics sidecar is injected by a webhook and its default never shows up in the template. The fix is to raise Katib to meet the floor, not lower the floor:

runtime:
  suggestions:
    - algorithmName: tpe
      image: ghcr.io/kubeflow/katib/suggestion-hyperopt:v0.19.0
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
  metricsCollectors:
    - kind: StdOut
      image: ghcr.io/kubeflow/katib/file-metrics-collector:v0.19.0
      resources:
        requests:
          cpu: 50m
          memory: 32Mi
        limits:
          cpu: 200m
          memory: 128Mi

Three more gotchas from getting a sweep to finish:

katib-config is mounted with subPath, which never receives updates. Restart the Katib controller after every config change; waiting does nothing.
Never put a multi-line python3 -c script in a trial template. The stdout collector rewrites the command into a shell string, and the inline script doesn't survive. Pass a file path and flags instead.
A sub-second trial can finish before the collector attaches, which shows up as random failures with an empty observation. Pace the objective and set backoffLimit: 1.

With all of that fixed, a 12-trial TPE sweep finished 12 out of 12 and found the known optimum, on CPU, because for a demonstration sweep a GPU changes nothing (more on that below). And with no time-slicing, the real ceiling for GPU trials is three at a time, one per card. A parallelTrialCount higher than that just leaves trials Pending.

The Katib trial leaderboard with the best trial highlighted.

Training and serving

A GPU training job in Kubeflow Trainer V2 looks like this. I haven't run this one yet, so treat it as the shape, not a tested manifest. With one card per machine, two workers have to land on two different machines:

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: resnet-finetune
  namespace: terranetes
spec:
  runtimeRef:
    name: torch-distributed
  trainer:
    numNodes: 2
    resourcesPerNode:
      limits:
        nvidia.com/gpu: 1

For serving, a KServe InferenceService with minReplicas: 0 gives the card back when the model is idle and cold-starts it on demand. That works because the GPU claim is per pod: scale to zero, the pod goes, the card is free. With three cards, scale-to-zero is the difference between one always-on model and a fleet of models sharing three GPUs.

My live models today don't need a GPU at all. The football match predictor is an XGBoost model on tabular data, and it trains and serves on CPU in seconds. The first real GPU TrainJob and a properly served LLM are both on the list at the end.

The first real job: a 7B model on one card

For three months these cards passed their self-test and did nothing else. Prometheus has the receipts: 0% utilisation and 0 MiB of GPU memory on all three, the whole time. So before writing another word about what an A1000 can do, I made one do it.

The job is deliberately plain. One pod, one card, Ollama running Mistral 7B in 4-bit (Q4_K_M, a 4.1 GB download), the same prompt every time with a fixed seed. The whole request for hardware is one line, nvidia.com/gpu: 1. Kyverno added the toleration and the node rule, the scheduler put it on dc2, and nvidia-smi inside the container saw the card.

NAME          SIZE      PROCESSOR    CONTEXT
mistral:7b    5.8 GB    100% GPU     4096

Then the same model on the same node with the GPU switched off, four threads against the pod's four-core limit:

	Tokens per second	Notes
RTX A1000	about 30 (four runs: 29.9 to 31.6)	5,031 MiB of the 8,188 MiB used, 98-99% utilisation
CPU, 4 threads (i9-13900H)	about 4.2 (two runs: 4.1 and 4.3)	same model, same prompt, same seed

Mistral 7B on one RTX A1000 against the CPU beside it: about 30 tokens a second against about 4.2.

Seven times faster, on a 50-watt card that fits in a mini PC. A 400-token answer took 13 seconds on the GPU. At the CPU's rate it would take over a minute and a half. That is the difference between a tool you'd use and one you wouldn't.

Three things the run taught me that the spec sheet didn't:

It runs hot. The card idles at about 60 °C in the MS-01 and sat at 88 to 90 °C within a minute of load, a few degrees over its own 86 °C target, with the driver reporting a software thermal slowdown. Speed drifted from 31.6 to 29.9 tokens a second across the runs. It never reached the 96 °C hard slowdown, but a small case and a single-slot cooler have a ceiling, and sustained training will live at it.
It can't tell you its power draw. nvidia-smi reports power.draw as [N/A] on this card under passthrough. So the missing power metric isn't an exporter setting I forgot.
Threads matter more than cores. My first CPU run was left to pick its own thread count. It saw 20 cores, started 20 threads inside a four-core limit, and was still going ten minutes later. Pinned to four threads it produced 60 tokens in about fifteen seconds. If you benchmark CPU inside a container, set the thread count to the limit.

The manifest and the exact commands are in examples/llm-benchmark-pod.yaml. It's a benchmark, not a service: the pod holds a whole card while it exists, so I deleted it when the numbers were in, which is this article's own advice.

📸 SCREENSHOT: the Grafana DCGM dashboard for dc2 across the run: utilisation jumping from 0 to 98%, memory to 5 GiB, temperature from 60 to the high 80s. The first non-zero lines those panels have ever drawn.

Multi-GPU, and does a GPU change the result?

With more than one card, the internet will tell you to reach for tensor parallelism and model sharding. On three 8 GB cards over PCIe with no NVLink, most of that advice is wrong.

Strategy	Splits	Use when	On these cards
Data parallel (DDP)	the batch, full model per GPU	the model fits in 8 GB	✅ the default
Sharding (FSDP, ZeRO)	parameters and optimiser state	the model won't fit one GPU	⚠️ prefer offloading to CPU RAM
Tensor parallel	individual layers	needs NVLink	❌ slower over PCIe than not doing it
Pipeline parallel	layer stages	very deep models, many GPUs	⚠️ rarely worth it here

Use single-GPU tricks first (bf16, gradient checkpointing, smaller batches), then DDP for throughput, and only then sharding. For inference, don't shard: quantise. A 7B model in int4 fits on one card, and splitting it across two PCIe cards is strictly worse.

Does a GPU change the result, or just the speed? It changes how fast you get a result, not what the result is. With a fixed number of trials, a GPU gets you the same answer sooner. With a fixed amount of time, faster trials mean more trials, and more trials is how a search finds a better optimum. That gain comes from trial count, not from the GPU. And it can make results slightly worse: GPU reductions aren't always deterministic, so training often isn't bit-for-bit reproducible.

So putting a CPU-bound job on a GPU is theatre, and it costs a card something else needed.

Monitoring

The DCGM exporter publishes per-GPU metrics, and a ServiceMonitor puts them in Prometheus, with NVIDIA's dashboard in Grafana. The ones worth watching:

DCGM_FI_DEV_GPU_UTIL: utilisation
DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE: memory used and free
DCGM_FI_DEV_GPU_TEMP: temperature
DCGM_FI_DEV_PCIE_REPLAY_COUNTER: link errors on the PCIe slot

Two I want and don't have. DCGM_FI_DEV_XID_ERRORS isn't in my exporter's field list yet. DCGM_FI_DEV_POWER_USAGE isn't either, and adding it won't help: this card doesn't report power draw under passthrough at all.

The most useful panel is utilisation by pod:

sum(DCGM_FI_DEV_GPU_UTIL{Hostname="rke2-worker-dc2-gpu-a"}) by (exported_namespace, exported_pod)

Mind the label. On my setup pod is the exporter's own pod, and the workload is in exported_pod. Group by pod and every card looks like it's being used by the DCGM exporter.

What's missing is any alerting. The metrics are there, but nothing fires on a hot card, full memory or an XID error yet. And the metric I want most isn't on any dashboard: allocated versus used. A card that's 100% allocated and 0% used is the notebook problem, and utilisation alone makes it look exactly like a free card.

NVIDIA has since open-sourced something aimed at exactly that gap: the GPU Usage Monitor, an Apache 2.0 Helm chart that pairs the DCGM exporter with kube-state-metrics, Prometheus and Grafana, and ships dashboards for allocated against used, by namespace and by pod, plus GPU pods stuck Pending. I haven't installed it. I already run every one of those four pieces, so what I'd want from it is the dashboards, not a second Prometheus. And its platform filter is named for datacenter cards (Hopper, Blackwell), so I don't yet know how it treats an A1000. It's on the list.

The Grafana DCGM dashboard showing all three A1000s.

What it costs

Three used RTX A1000s for about £600, in slots the MS-01s already had. No extra power. The card is rated at 50 W; I can't give you a measured figure, because it doesn't report one. Cooling is the honest cost: nothing extra was fitted, and under load the card sits at its thermal limit.

The operational cost is mostly kernel upgrades on GPU nodes, which need the driver restart. Adding a GPU node is now Terraform the passthrough, label and taint it, and let the operator do the rest.

What's Next?

The sports car runs, survives a power cut and comes back on its own. What's left, roughly in order:

Try NVIDIA's GPU Usage Monitor dashboards against the Prometheus I already have, to get allocated versus used on a screen.
Export XID errors, then wire up GPU alerts: temperature (now that I know it reaches 90 °C), memory nearly full, XID errors, and a card allocated but unused for hours.
Turn on time-slicing on dc1 and dc2, and keep dc3 whole.
Put nvidia.com/gpu in the profile ResourceQuota, so one namespace can't take every card.
Run a real GPU TrainJob across two machines, and serve that LLM properly behind KServe with scale-to-zero, not from a benchmark pod.
Automate the driver restart after a GPU node's kernel upgrade, so it stops being a checklist item.

If you've got a GPU sitting in a box that Kubernetes can't see, I hope this saves you the thirteen traps on this list.

Follow me on LinkedIn

Author: George Ezejiofor

Math 2.0: How Terence Tao’s Vision Can Supercharge Your AI Infrastructure

Naveen Malothu — Mon, 12 Oct 2026 11:15:54 +0000

1️⃣ What was released / announced

Last week Caltech hosted Terence Tao’s Math 2.0 talk (see the PDF here). In a crisp 30‑slide deck he laid out a roadmap for a new era where formal mathematics, large language models, and cloud‑native tooling converge. The key takeaway: treat mathematics like a software stack – version‑controlled, testable, and continuously deployed.

2️⃣ Why it matters

If you’re building AI infrastructure or MLOps pipelines, you already know the pain of reproducibility. Imagine a world where:

Proofs become CI‑tested artifacts – a broken theorem fails a pipeline just like a failing unit test.
Mathematical models can be auto‑generated from high‑level specifications using LLMs, dramatically cutting research iteration time.
Collaboration happens on a shared, typed ledger (think Git + Lean/Coq) that scales from a single PhD student to a multinational R&D team.

That’s the promise of Math 2.0. It isn’t just academic fluff; it’s a blueprint for building more reliable, auditable, and automated AI systems – something every devops engineer and AI infrastructure lead should start experimenting with today.

3️⃣ How to use it – a practical starter kit

Below is a hands‑on, 5‑step workflow that brings a slice of Tao’s vision into your own cloud environment. The stack uses:

Lean 4 for formal proofs (the language Tao highlighted).
OpenAI’s GPT‑4o (or any capable LLM) to translate informal math into Lean snippets.
GitHub Actions for CI/CD of proofs.
Docker + Kubernetes to keep everything reproducible.

Step 1 – Spin up a Lean‑4 container

# Dockerfile
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
    curl git build-essential python3-pip && \
    rm -rf /var/lib/apt/lists/*

# Install Lean 4 (via elan)
RUN curl https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh -sSf | sh -s -- -y
ENV PATH="/root/.elan/bin:${PATH}"

WORKDIR /workspace
COPY . .

Build and push the image:

docker build -t ghcr.io/yourorg/lean4-ci:latest .
docker push ghcr.io/yourorg/lean4-ci:latest

Step 2 – Prompt an LLM to generate a Lean lemma

import openai, json

prompt = "Write a Lean4 lemma that proves the Cauchy‑Schwarz inequality for real vectors."
resp = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0,
)
print(resp.choices[0].message.content)

Save the output to `cauchy.lean` in your repo.

Step 3 – Add a GitHub Action that runs `lake build`

# .github/workflows/lean-ci.yml
name: Lean CI
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/yourorg/lean4-ci:latest
    steps:
      - uses: actions/checkout@v3
      - name: Build Lean project
        run: |
          lake build
      - name: Run tests
        run: |
          lake test

Step 4 – Wire the proof into your AI model pipeline

Suppose you have a custom loss function that relies on a mathematically proven property (e.g., convexity). You can expose the proof as a JSON artifact and let your training job verify it at runtime.

{
  "lemma": "cauchy_schwarz",
  "status": "proved",
  "timestamp": "2026-10-12T08:00:00Z"
}

Your training script can read this file and assert the property before proceeding.

Step 5 – Deploy on Kubernetes for scaling

Create a Deployment that runs the Lean‑4 container as a proof‑service. Expose a tiny HTTP API (using fastapi) that accepts a natural‑language conjecture, forwards it to the LLM, and returns a Lean proof.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: proof-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: proof-service
  template:
    metadata:
      labels:
        app: proof-service
    spec:
      containers:
      - name: lean
        image: ghcr.io/yourorg/lean4-ci:latest
        ports:
        - containerPort: 8000

Now any microservice in your AI stack can ask for a formal guarantee on‑the‑fly.

4️⃣ My take – From an AI‑infra perspective

Tao’s Math 2.0 feels like a call to industrialize mathematics the same way we industrialized software. In practice, the biggest win for us at Griffin AI Tech has been reducing flaky research. By treating proofs as first‑class CI artifacts, we catch logical regressions before they corrupt downstream models.

A few hard‑earned lessons:

Lesson	What I Learned
Tooling overhead	Lean’s ecosystem is still maturing. Expect a steep learning curve for engineers who are pure‑Python devs.
LLM‑generated proofs need verification	Never trust the LLM output blindly – always run `lake build` and `lake test`.
Cost vs. value	Running a proof‑service on a GPU cluster is overkill; a modest CPU node suffices. The real ROI comes from preventing bugs in model‑training pipelines.

If you’re skeptical, start small: pick a single theorem that underpins a critical component (e.g., the convergence guarantee of a custom optimizer) and lock it into your CI. The mental model shift – “my loss function must be provably convex” – quickly spreads to other parts of the stack.

TL;DR

Math 2.0 is a manifesto to bring formal verification, LLM‑assisted proof generation, and cloud‑native CI/CD together.
The payoff for AI infrastructure teams is more reliable, auditable, and faster research cycles.
I’ve built a minimal proof‑service using Lean 4, Docker, GitHub Actions, and Kubernetes – all under 100 lines of config.
Start with one lemma, CI‑test it, and watch the confidence in your models grow.

Give it a spin and let me know how it reshapes your AI pipelines!

Why HF_HOME Points To Ephemeral Storage On RunPod By Default And How To Fix It

Francisco Booth — Sun, 11 Oct 2026 14:23:40 +0000

If you have ever run a HuggingFace model download on RunPod and come back to find your cache gone, this is why.

The default is wrong for rented GPU pods

HuggingFace sets HF_HOME to ~/.cache/huggingface when no environment variable overrides it. On a standard Linux machine that is fine your home directory persists.

On RunPod, your home directory is on the container volume, which is ephemeral. When the pod stops, it is gone.

RunPod has two storage layers: the container volume, which is ephemeral, and the network volume mounted at /workspace, which persists across pod restarts. HF_HOME defaults to the wrong one.

This means every model download, every tokeniser, every dataset you pulled via datasets.load_dataset() gone on pod restart. You pay to download them again on the next run.

On large models that is not a minor inconvenience. A 7B model download at pod startup adds minutes and real cost every single time.

The fix is one line in your startup script:

export HF_HOME=/workspace/.cache/huggingface

/workspace is RunPod’s persistent volume mount. One line and your cache survives every restart.

Why this is easy to miss

There is no error message. HuggingFace downloads the files successfully to the ephemeral path. Your training run completes. The problem only surfaces when you restart the pod and the download happens again and even then it looks like a slow startup, not a misconfiguration.

Silent failures are the hardest to debug because you are not looking for them.

Automating the check

If you want this caught automatically before every job rather than relying on remembering to set the variable, ComputeFence checks it as part of its pre-flight scan:

uvx computefence doctor

It flags HF_HOME pointing to ephemeral storage and prints the exact export command for your platform. It also checks GPU visibility, Accelerate config, checkpoint directory persistence, and disk headroom the other silent failures that cost compute budget without a log entry.

The two-line pre-flight for every RunPod job

export HF_HOME=/workspace/.cache/huggingface
uvx computefence doctor

The first line fixes the cache. The second confirms everything else is configured correctly before the GPU bill starts.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

From Local Scripts to Edge Deployments: Building Production-Grade AI Infrastructure

Ashish Kumar Singh — Sat, 10 Oct 2026 12:34:13 +0000

Moving Beyond API Wrappers

Building a local prototype with an LLM API takes an hour. Running it in an enterprise environment—handling traffic spikes, controlling costs, and enforcing security guardrails—is a completely different challenge.

As an AI Infrastructure & MLOps Engineer, I bridge the gap between foundation models and scalable cloud architecture. Here are the core lessons I learned architecting and deploying three cloud platforms in 30 days, including CORA, an enterprise AI chatbot with strict constraint guardrails.

The Serverless Edge Stack

To achieve high availability, low latency, and automated deployments, I leveraged:

AWS (EC2, S3, IAM): Core compute and infrastructure primitives.
Vercel & Cloudflare: Global CDN delivery and edge function execution.
Claude API + Guardrails: Secure orchestration preventing prompt injection.
Resend: Automated transactional data and email pipelines.
Terraform (IaC): Reproducible, programmatic environment provisioning.

3 Critical MLOps Lessons

1. Guardrails Belong at the Infrastructure Level
UI prompt engineering isn't enough. Guardrails must be enforced at the backend API layer before queries ever hit the model. Implementing constraint-checking middleware on CORA kept system prompts inviolable while maintaining fast response times.

2. Strict FinOps & Latency Management
Agentic workflows can spike API costs through infinite loops. Edge caching, strict timeout policies, and token budgets are essential for keeping cloud spend predictable.

3. Edge-First Deployments
Shifting state management and routing to edge networks minimizes cold starts, delivering near-instantaneous responses globally.

What’s Next?

I’m currently mastering enterprise architectures and preparing for my AWS Machine Learning Engineer – Associate (MLA-C02) and HashiCorp Terraform certifications.

You can test my interactive AI projects and view my architecture setups at aashishsingh.me.

What are your go-to tools for hosting production AI agents? Let’s discuss in the comments below!

DEV Community: mlops

How to Choose the Right AI Software Development Company in 2026?

From Model Experiments to Applications That People Can Use

Getting Hands‑On with Gemini 4 Argon: What It Is, Why It Matters, and How to Plug It Into Your Stack

1️⃣ What was released / announced

2️⃣ Why it matters

3️⃣ How to use it

Step 1 – Install the client library

Step 2 – Write a helper wrapper

Step 3 – Deploy at scale

4️⃣ My take

Math 2.0: Turning Terence Tao’s Vision into Practical AI‑Ready Tools

1. What was released / announced

2. Why it matters

3. How to use it

Step 1 – Install the library

Step 2 – Query a theorem via the Python client

Step 3 – Validate a symbolic proof

Step 4 – Add a CI gate (GitHub Actions example)

Step 5 – Deploy the REST endpoint (optional)

4. My take

Why Most Company AI Projects Get Stuck and How to Fix Them?

The same H100 hour cost $1.26 or $0.22 per million tokens

Ready is not warm: one SGLang cold start on a T4

From GenAI Pilots to Production: How MLOps Closes the Scaling Gap

NVIDIA GPU Operator on RKE2: Passthrough to Kubeflow (The Sports Car in the Garage)

Introduction

Where this sits in the build

The hardware in the garage

PCIe passthrough: from Proxmox to the VM

What the GPU Operator actually does

From PCIe slot to scheduled pod

Keeping GPU pods on GPU nodes

Prove it works

The runtime: getting containerd to know about GPUs

Sharing the cards

How reliable is it?

The traps, from the driver up

The driver

The validator

The toolkit and runtime

The device plugin and scheduling

Telemetry

Who holds a card, and how to get it back

Putting the GPUs to work

Who on the platform can ask for a card

Hyperparameter search with Katib

Training and serving

The first real job: a 7B model on one card

Multi-GPU, and does a GPU change the result?

Monitoring

What it costs

What's Next?

Math 2.0: How Terence Tao’s Vision Can Supercharge Your AI Infrastructure

1️⃣ What was released / announced

2️⃣ Why it matters

3️⃣ How to use it – a practical starter kit

Step 1 – Spin up a Lean‑4 container

Step 2 – Prompt an LLM to generate a Lean lemma

Save the output to cauchy.lean in your repo.

Step 3 – Add a GitHub Action that runs lake build

Step 4 – Wire the proof into your AI model pipeline

Your training script can read this file and assert the property before proceeding.

Step 5 – Deploy on Kubernetes for scaling

Now any microservice in your AI stack can ask for a formal guarantee on‑the‑fly.

4️⃣ My take – From an AI‑infra perspective

TL;DR

Why HF_HOME Points To Ephemeral Storage On RunPod By Default And How To Fix It

From Local Scripts to Edge Deployments: Building Production-Grade AI Infrastructure

Moving Beyond API Wrappers

The Serverless Edge Stack

3 Critical MLOps Lessons

What’s Next?

Step 1 – Install the client library

Step 2 – Write a helper wrapper

Step 3 – Deploy at scale

Step 1 – Install the library

Step 2 – Query a theorem via the Python client

Step 3 – Validate a symbolic proof

Step 4 – Add a CI gate (GitHub Actions example)

Step 5 – Deploy the REST endpoint (optional)

Math 2.0: How Terence Tao’s Vision Can Supercharge Your AI Infrastructure

Step 1 – Spin up a Lean‑4 container

Step 2 – Prompt an LLM to generate a Lean lemma

Save the output to `cauchy.lean` in your repo.

Step 3 – Add a GitHub Action that runs `lake build`

Step 4 – Wire the proof into your AI model pipeline

Step 5 – Deploy on Kubernetes for scaling