DEV Community: pytorch

PyTorch vs. TensorFlow for Diffusion Models: A Dev Leader's Guide to Tooling and Productivity

Oleg — Wed, 14 Oct 2026 13:00:29 +0000

The Great Debate: PyTorch vs. TensorFlow for Diffusion Models

In the rapidly evolving landscape of Artificial Intelligence, choosing the right tools is paramount for any development team. A recent GitHub Community discussion (Discussion #208944) brought to light a critical decision point for AI developers: selecting between PyTorch and TensorFlow for diffusion models. Making an informed choice here can significantly impact project velocity and output quality, factors often highlighted by effective development measurement tools.

The consensus from the community is clear: for diffusion models, PyTorch is emphatically recommended. Here’s why:

Diagram illustrating the PyTorch ecosystem for diffusion models, including Hugging Face diffusers and optimization tools.### Ecosystem & Library Dominance

The industry standard for diffusion models, Hugging Face diffusers, is built natively on PyTorch. This means that pretrained weights, pipeline schedulers (like Euler, DPM-Solver, UniPC), and fine-tuning scripts (LoRA, DreamBooth, ControlNet) are primarily optimized for PyTorch. This deep integration simplifies development and accelerates time-to-market for new features. Furthermore, PyTorch 2.x's native SDPA (Scaled Dot-Product Attention) and torch.compile offer significant speedups (2x–3x) without external dependencies, directly contributing to a more efficient development lifecycle.

Dynamic Computation Graphs for Agile Development

Diffusion sampling involves iterative denoising loops with dynamic timestep embeddings and conditioning tensors. PyTorch’s imperative design and dynamic computation graphs simplify the implementation and debugging of these multi-step reverse processes. This offers a more straightforward, intuitive experience compared to the often more rigid graph compilation workflows in TensorFlow, fostering agility and reducing debugging overhead for your engineering teams.

Research & Community Leadership

Over 90% of generative AI papers, checkpoints, and open weights released on platforms like arXiv and Hugging Face prioritize PyTorch implementations. This makes PyTorch the go-to framework for staying current with cutting-edge research and leveraging the latest advancements, ensuring your team remains at the forefront of AI innovation.

Visual representation of a classical state-space search problem, showing states, actions, and a goal path.While TensorFlow remains a viable option if your team has existing TensorFlow serving/training infrastructure, it is not the path with the broadest examples or the most current ecosystem support for diffusion models. For new diffusion projects, starting with torch + diffusers + accelerate for multi-GPU training and inference is the recommended approach.

Beyond Frameworks: Understanding State-Space Concepts in AI

The discussion also touched upon "state-space searches," a term that can have different meanings in classical AI versus modern machine learning. Clarifying this distinction is crucial for technical leaders guiding their teams.

Classical State-Space Search: The Foundation of Problem Solving

A State-Space Search is a foundational AI problem-solving paradigm where a problem is formulated as a set of discrete states, transitions (operators), and a target objective. Imagine a maze: each square is a state, moving between squares is an action, and reaching the exit is the goal.

State Space: The graph or tree of all possible configurations the system can be in.
Initial State: Where the search begins.
Actions/Transitions: Valid moves taking state S to successor state S'.
Goal Test: A function verifying if the current state satisfies the objective.
Path Cost: The cumulative cost incurred to reach a given state.

Common search strategies include:

Uninformed (Blind) Search:Breadth-First Search (BFS): Explores level-by-level; optimal if all step costs are equal.
Depth-First Search (DFS): Explores deep paths first; memory efficient but not guaranteed optimal.
Informed (Heuristic) Search:A* Search: Uses an evaluation function f(n) = g(n) + h(n), where g(n) is the actual cost so far and h(n) is an admissible heuristic estimate to the goal. A* is both complete and optimal.

Illustration comparing the linear processing of State-Space Models (Mamba) with the quadratic complexity of traditional Transformer attention.### Modern State-Space Models (SSMs): A New Paradigm for Sequence Processing

In modern generative AI and sequence modeling, the term "State Space Models" (SSMs) refers to architectures like Mamba and S4. These models are distinct from classical search algorithms. They model continuous sequence state transitions in O(L) linear time, offering significant computational advantages over Transformer-based models with their O(L2) attention mechanisms, especially for very long sequences. Understanding this distinction is vital for teams exploring cutting-edge sequence processing techniques.

Strategic Impact: Tooling Choices, Productivity, and Delivery

For dev team members, product/project managers, delivery managers, and CTOs, these technical decisions directly influence your team's productivity and project delivery metrics. Leveraging the right tools can drastically improve outcomes, making it easier to track progress with robust development measurement tools and achieve your targets on a performance KPI dashboard.

An optimal framework choice, like PyTorch for diffusion models, translates into faster iteration cycles, reduced debugging time, and quicker adoption of new research. This directly impacts your team's efficiency and the quality of your output. Furthermore, streamlined workflows, enabled by optimal framework choices, positively impact aspects like pull request analytics for GitHub, offering clearer insights into team efficiency and code velocity. By making informed decisions based on community insights and technical merits, leaders can empower their teams to build more effectively and deliver innovative AI solutions with greater confidence and speed.

Top 5 Dev Tools & Tutorials of the Week (October 14, 2026)

karleeov — Wed, 14 Oct 2026 00:47:13 +0000

1. TensorFlow Quantum 3.0

What it is: TensorFlow Quantum 3.0 is a framework designed for building and experimenting with hybrid quantum-classical machine learning models.

Why now: Released on October 10, 2026, this version brings enhanced support for hybrid quantum-classical models and improved performance for large-scale quantum simulations.

Tutorial angle: Learn how to set up and train a hybrid quantum-classical model using TensorFlow Quantum 3.0.

Difficulty: Intermediate

Start today:

pip install tensorflow-quantum==3.0

2. PyTorch Lightning CLI 2.0

What it is: The PyTorch Lightning CLI 2.0 is a command-line interface for managing and scaling PyTorch models with ease.

Why now: Launched on October 12, 2026, it features a more intuitive interface, multi-GPU training support, and enhanced logging capabilities.

Tutorial angle: Discover how to streamline multi-GPU training and logging using the new CLI features.

Difficulty: Intermediate

Start today:

pip install pytorch-lightning==2.0
lightning --help

3. OpenAI Codex SDK for JavaScript

What it is: The OpenAI Codex SDK for JavaScript allows developers to integrate Codex's code generation capabilities into their applications.

Why now: Released on October 8, 2026, this SDK enables the automation of code generation and the creation of intelligent coding assistants.

Tutorial angle: Learn how to set up and use the Codex SDK to generate code snippets within a JavaScript application.

Difficulty: Beginner

Start today:

npm install openai-codex

4. Django AI Framework 1.0

What it is: Django AI Framework 1.0 integrates AI capabilities into the Django web framework, allowing developers to add machine learning features with minimal configuration.

Why now: Released on October 9, 2026, this framework makes it easier to incorporate AI-driven features like personalized recommendations into Django applications.

Tutorial angle: Explore how to add AI-driven features to your Django web application using the Django AI Framework.

Difficulty: Intermediate

Start today:

pip install django-ai-framework==1.0

5. LangChain SDK for Python

What it is: The LangChain SDK for Python is a set of tools for building language model-driven applications, offering support for chain-of-thought reasoning and multi-hop questions.

Why now: Released on October 13, 2026, it enhances the capabilities of language model applications by supporting more complex reasoning and query processes.

Tutorial angle: Build a language model application using chain-of-thought reasoning with the LangChain SDK.

Difficulty: Intermediate

Start today:

pip install langchain==1.0.0

Building in public: my lead-gen engine

This report was generated using a custom lead-generation engine that aggregates data from various sources, including web searches, GitHub releases, and community platforms like Hacker News. The engine filters out recent topics to focus on the most relevant and impactful tools for developers.

Data Sources: MiniMax-Text-01, web_search, Hacker News, GitHub, sources.py crawl.
Exclusions: Recent 27 topics filtered out to ensure freshness.
Focus: High-impact tools and tutorials for immediate use.

Start with this today

If you're looking to dive into something immediately, try setting up the OpenAI Codex SDK for JavaScript with this one-liner:

npm install openai-codex

Picks of the week

TensorFlow Quantum 3.0: Perfect for those interested in the intersection of quantum computing and machine learning.
PyTorch Lightning CLI 2.0: Essential for anyone scaling PyTorch models across multiple GPUs.
OpenAI Codex SDK for JavaScript: Ideal for developers looking to automate code generation in their applications.

Question for readers

Which of these tools are you most excited to try out? Share your thoughts and projects in the comments below!

Connect

If this kind of post is useful, the easiest way to support the work is to:

Star / follow on dev.to (you're already here 🙂)
Follow on X: @devkingov
Reach out for HK-based dev work — .NET / Azure / system integration / IT security: studio.resurrects.co or email devkingov@gmail.com
Subscribe to weekly HK tech posts → studio.resurrects.co/blog (one email a week, no spam)

MPS backend out of memory on a Mac: what the numbers mean and how to set the watermark

Rodion Kazennov — Wed, 14 Oct 2026 00:01:59 +0000

When a PyTorch job runs out of memory on a Mac, the error ends with a piece of advice:

RuntimeError: MPS backend out of memory (MPS allocated: 18.50 GiB, other allocations: 448.00 KiB, max allowed: 18.72 GiB). Tried to allocate 512.00 MiB on shared pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

I wanted to know what following that advice does, so I filled MPS memory in 512 MiB steps on my 48 GB M5 Pro (macOS 26.5.2, PyTorch 2.14.0), once per setting, under a guard that stops the process when swap grows by 2 GB. Two things came out of it. On this Mac the default limit is higher than the RAM, so a job that outgrows memory slides into swap and never gets to this error. And lowering the limit takes two variables, not one.

Where "max allowed" comes from

max allowed is the limit: PYTORCH_MPS_HIGH_WATERMARK_RATIO times what torch.mps.recommended_max_memory() returns. On my Mac that is 37.44 GiB, and 0.5 × 37.44 = 18.72 GiB, the number in the error above. MPS allocated matched torch.mps.current_allocated_memory() in my log, the memory held by tensors. To see your own base number:

import torch
print(torch.mps.recommended_max_memory() / 2**30, "GiB")

Why a 48 GB Mac swaps first

PyTorch's documentation gives the default high ratio as 1.7. On my Mac that puts the limit at about 63.6 GiB, more memory than the machine has. This is what each setting did:

setting	what stopped the fill	tensors held	swap during the run
default (1.7)	the guard	38.5 GiB	6.1 → 12.0 GB
0.0	the guard	36.0 GiB	5.1 → 9.8 GB
1.0, low 1.0	the guard	34.5 GiB	4.6 → 8.7 GB
0.8, low 0.8	PyTorch, max allowed 29.95 GiB	29.5 GiB	no change
0.5, low 0.5	PyTorch, max allowed 18.72 GiB	18.5 GiB	no change

With the default, with 0.0 and with 1.0, swap started to grow once the process passed 33.5 to 36.2 GB, and PyTorch never raised: its limit was further away than the end of the RAM. So 0.0, the advice inside the error, changed nothing I could see here. The limit was already out of reach, and nothing in PyTorch stopped the run.

Setting it: two variables

Setting only the high ratio fails at the first MPS call:

$ PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.8 python fill.py
RuntimeError: invalid low watermark ratio 1.4

The low ratio defaults to 1.4 on Apple Silicon and may not be above the high one (unless the high one is 0.0, which turns the limit off). Set both:

PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.8 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8 python run.py

From Python, set them before the first tensor goes to mps. PyTorch reads them when MPS starts, not at import torch, and ignores changes made after that:

import os
os.environ["PYTORCH_MPS_HIGH_WATERMARK_RATIO"] = "0.8"
os.environ["PYTORCH_MPS_LOW_WATERMARK_RATIO"] = "0.8"
import torch

The documentation calls the low ratio a soft limit and says that "setting 0.0 disables adaptive commit and garbage collection". I set it equal to the high one and did not measure it on its own.

Picking a value

The limit should stay below the memory your Mac really has free, with your usual apps open. On mine 1.0 (37.44 GiB) was already too much, and 0.8 (29.95 GiB) gave a clean error with no swap. After a clean error you can lower the batch size or the resolution and try again; a run sliding into swap slows down the whole Mac first.

A trap when you test this

My first run filled the tensors with torch.ones. With the default setting it reached the 46 GiB cap I had set, with no error and no swap at all: macOS compresses pages of identical values almost to nothing. Model weights don't compress like that, so a memory test on a Mac needs random data.

When the model really doesn't fit

A clean error only tells you the model doesn't fit. For a pipeline of several models, loading only the models of the current stage does more than any ratio, which is why I wrote stageload. The fill script and the traces of every run above are in its repository, in bench/results/2026-10-13-mps-watermark.

PyTorch on a Mac: why memory looks freed after .to("cpu")

Rodion Kazennov — Tue, 13 Oct 2026 08:43:56 +0000

On a Mac, a tensor moved from mps to the CPU seems to disappear from memory. The process footprint, the number Activity Monitor shows in its Memory column, drops to almost nothing, and so does the MPS driver count. The copy is still there and still takes the same memory. This is the minimal example I put together for a diffusers discussion about offloading, with a 4 GiB tensor standing in for a model's weights (torch 2.14.0, macOS 26.5.2, M5 Pro with 48 GB):

import ctypes, os, time
import torch

# rusage_info_v2 from <libproc.h>; phys_footprint is the Memory column of Activity Monitor
FIELDS = ("user_time", "system_time", "pkg_idle_wkups", "interrupt_wkups", "pageins",
          "wired_size", "resident_size", "phys_footprint")
class RUsageInfoV2(ctypes.Structure):
    _fields_ = [("uuid", ctypes.c_uint8 * 16)] + [(f, ctypes.c_uint64) for f in FIELDS] \
        + [(f"rest{i}", ctypes.c_uint64) for i in range(10)]

libproc = ctypes.CDLL("/usr/lib/libproc.dylib")

def show(step):
    torch.mps.synchronize()
    info = RUsageInfoV2()
    libproc.proc_pid_rusage(os.getpid(), 2, ctypes.byref(info))
    print(f"{step:<14} footprint {info.phys_footprint / 2**30:4.1f} GiB, "
          f"mps driver {torch.mps.driver_allocated_memory() / 2**30:4.1f} GiB")

show("start")
w = torch.randn(2**30, device="mps")      # 4 GiB of float32, standing in for a model's weights
show("on mps")
w = w.to("cpu")                           # what model CPU offload does when a model's turn ends
torch.mps.empty_cache(); time.sleep(2)
show("moved to cpu")
float(w.sum())                            # read every page of the cpu copy
show("cpu copy read")

start          footprint  0.1 GiB, mps driver  0.0 GiB
on mps         footprint  4.2 GiB, mps driver  4.0 GiB
moved to cpu   footprint  0.1 GiB, mps driver  0.0 GiB
cpu copy read  footprint  4.2 GiB, mps driver  0.0 GiB

I ran it three times and got the same output each time.

What the numbers say

Right after the move, both counters are back near zero, but w holds all 4 GiB of its values. Once the CPU reads every page of the copy, the footprint is 4.2 GiB again. So a footprint taken between .to("cpu") and the next time the CPU touches those weights does not show them.

The reason the move frees nothing in the first place is the hardware. On Apple Silicon the CPU and the GPU use one pool of physical memory. On a PC, moving weights from the GPU to the CPU frees VRAM for the next model. On a Mac, the weights "on the CPU" sit in the same RAM the GPU needs.

Why it matters for model CPU offload

diffusers' enable_model_cpu_offload() keeps each model on the CPU and moves it to the GPU only while it runs. I ran Qwen-Image-2.1 at 1024 × 1024 that way on the same Mac (trace and write-up). The process footprint never went above 18.5 GB, which would fit on a 48 GB Mac with room to spare. Swap grew by 3.5 GB during denoising and had grown by 9.6 GB by the VAE decode, and my memory guard stopped the run before it wrote an image.

A chart of the footprint alone would have called that run a fit. The example above shows one way the footprint can stay low while the weights are still in memory. I didn't measure which pages went to swap in the offload run.

How to check your own run

Look at swap, not only at the footprint:

sysctl vm.swapusage             # compare "used" before and after the run
sysctl kern.memorystatus_level  # system-wide free memory, in percent

If swap grows while the job runs, it doesn't fit, whatever the footprint says. stageload (pip install stageload), a small library I wrote for this, records both in a trace every 0.5 s with MemoryMeter, and stageload guard stops a command once swap grows past a budget:

stageload guard --wait-free 40 --swap-budget 8G -- python generate.py

What does free the memory

For a pipeline that runs its models one after another, the fix is to load a model when its stage starts and release it when the stage ends, instead of parking it on the CPU. stageload's release(module) moves every parameter and buffer to PyTorch's meta device, collects garbage and empties the MPS cache; a tensor on meta has no storage, so the memory goes back even while other code still holds a reference to the module. With the text encoder (16.3 GB) and the transformer (13.3 GB) loaded one at a time this way, Qwen-Image-2.1 peaked at 19.0 GB on the same Mac, and swap did not grow in either of two runs. The full setup is in How to run Qwen-Image-2.1 on a 48 GB Mac without swap.

What I did not measure

The free memory of the whole system during the example: other jobs on this Mac move it by several GiB between runs, so I couldn't get a clean number.
Whether del plus torch.mps.empty_cache() gives the memory back as fully as moving to meta, and how quickly.
Other macOS and torch versions, and Macs with more memory.

I Trained a Language Model From Scratch on a 4 GB GPU

Sandra Bamboo — Mon, 12 Oct 2026 08:07:10 +0000

A 4 GB GPU, a small corpus and a lot of things that went wrong in instructive ways.

I trained a 22-million-parameter language model from scratch on a laptop with 4 GB of VRAM. It learned to imitate technical documentation, got stuck repeating fragments of language, and never produced a working Fibonacci function. Near the end, CUDA crashed. The result wasn't a useful chatbot. The failures were the interesting part.

Pulling a pretrained model through Ollama makes modern AI feel deceptively simple. Training even a tiny one from random weights removes that illusion very quickly.

That’s why I started AI Lab, a separate repository for research code, datasets, experiment definitions, models and results. AI.001 was my first experiment: training a small language model from random weights on my laptop. This is the story behind that run, including what the model actually generated. The complete methodology, corpus breakdown and training curves live in the AI.001 experiment write-up.

I wanted to understand what happens before inference

APIs and pretrained local models are wonderful, and they hide almost everything I wanted to see. What does the dataset actually look like? How much does cleaning matter? What happens between a document and a token? What is a checkpoint, really? What does loss tell me, and what does it quietly not tell me? What does generation look like before a model is useful at all?

I couldn’t answer those by calling something that already worked. The only way I know to find out is to build the unglamorous version and watch it misbehave.

The constraint: 4 GB of VRAM

The machine is a Lenovo IdeaPad Gaming laptop: an Intel i7-10750H, 16 GB of RAM and a GTX 1650 with 4 GB of VRAM, running PyTorch with CUDA under WSL2. It is not a good machine to train on, and I’m not presenting it as one.

That’s the point. With a big GPU you can hide a lot of sloppy decisions behind headroom. With 4 GB you have to know why the context length is what it is, why the batch is split into microbatches, and what the optimizer state costs. I settled on a 256-token context and microbatches of 16 with gradient accumulation, so each update still sees a reasonably large batch. Peak memory for the main run was 1.54 GiB, which is comfortably under the card’s limit. I don’t read that as the card being generous. I read it as the result of designing the pipeline around the limit from the start rather than discovering it halfway through.

Building the corpus

Spanish and English Wikipedia, Python documentation and a handful of other technical documentation sets: collecting text is the easy part. Anyone can download a dump. The interesting part is deciding what is allowed into the training set and what that decision does later.

I chose a controlled corpus over a large unfiltered web dataset, and judged sources on quality, licensing, language balance, duplication risk and how painful they were to preprocess. Cleaning and filtering is where the corpus stops being “everything I could download” and becomes something I made on purpose. I’m not going to dump corpus statistics here; the experiment page has them. What I’d rather say is that those choices came back later, in the evaluation, as behaviour I could point at.

The workflow behind AI.001. Checkpoints make recovery possible; held-out loss and generated samples answer different questions.

A falling loss, a CUDA crash, and a checkpoint

Validation loss started at 9.48. With a vocabulary of 12,000 tokens, uniformly guessing the next token would give a loss of about 9.39. After roughly one pass over the corpus, the model reached 4.27. That was progress, and it was satisfying to see. But it answered one narrow question — whether predictions on held-out text had improved — not whether the model could generate anything useful.

Checkpoints stopped being an abstract idea at around the same time. I had tested resuming from a checkpoint on a short run before committing to the long one, and it printed exactly what I hoped to see:

Resumed from: results/EXP-001/resume-test/latest.pt
Resume step: 10

That turned out to be sensible. Shortly after step 3,540 of the main run, CUDA reported an unknown sticky error and training stopped. It wasn’t an out-of-memory failure and the loss had been behaving. I restarted from the step-3,500 checkpoint, with the optimizer, mixed-precision scaler and random state restored, and it ran to the end.

The more instructive surprise was generation. With sampling, later checkpoints produced fragments that looked like technical documentation; an early one mixed WordPress, MDN and Docker material incoherently. With greedy decoding — always taking the most likely next token — the final model often fell into loops. Validation loss did not reveal that failure mode. During generation the model has to continue from its own output, not from the real text used for evaluation. Here is what a few prompts produced:

Prompt: “A WordPress plugin can” → Sampled outputs (temperature 0.8, final checkpoint): “A WordPress plugin can be used when the plugin is loaded” and “A plugin contains a PHP file”
Prompt: “Barcelona es una ciudad” → Greedy decoding: repeated variations of “de la ciudad”
Prompt: “A language model learns” → Greedy decoding: “the language is the language language…”
Prompt: “def fibonacci(n):” → None of the checkpoints I tested produced a valid implementation

Evaluation was more interesting than training

Those examples were more revealing than the loss curve on its own. So I evaluated the final checkpoint by corpus component instead of relying only on an aggregate score. Technical documentation had lower loss than general prose, but there was a wide spread between sources: MDN and WordPress text was more predictable to this model than Docker, Python or Git documentation. The source-by-source results in AI.001 show the actual measurements.

I want to be careful with that ordering. It is an observation from one checkpoint of one run, and it is not evidence that the model is good. What it does is raise better questions than the aggregate loss did. Is a source easy because it is well-written, or because it is repetitive? How much of a low loss is the model knowing something and how much is a predictable format? Did I give some sources more room than they deserved? Is there overlap between domains that I haven’t noticed? Loss can’t separate those on its own.

Tiny models are brutally honest

A 22-million-parameter model, trained for about one pass, has nowhere to hide. It learned how technical documentation looks far better than what it means. It writes convincing headings and parameter lists for APIs that don’t exist, drifts between domains within a paragraph, and never produced a working def fibonacci(n): at any checkpoint I tried.

That is useful for learning precisely because it is bad. A strong model smooths over its mechanics. A weak one shows them: capacity, data mix, repetition, the gap between predicting text and knowing things. This is research, not product development, and I try to keep that straight in my own head.

What I actually got from it

Not a competitive model, and not something I would use to answer questions. What I got was a research workflow: repeatable corpus tooling, training scripts that can pause and resume, and evaluations broken down by source. More importantly, I now know to look at both held-out loss and actual generations before deciding what a model has learned. That feels more useful than a demo that only looks convincing.

Where it goes next

I’m not promising a bigger model. The questions the first run left open are about corpus selection, better evaluation, and changing one decision at a time so I can tell what each change did. The experiment page lists a more focused follow-up, with more source code in the mix and a fixed evaluation I can compare against this one. I’d rather make a small, controlled step that I understand than a large one that I can only describe.

Originally published on sandrabamboo.com: https://sandrabamboo.com/training-my-first-language-model-on-a-gtx-1650/ — for model architecture, corpus sources, training curves and per-source evaluation, see the complete AI.001 experiment: https://sandrabamboo.com/experiments/training-a-language-model-from-scratch-on-4gb-of-vram/

Model security on Hugging Face: what executes when you load

Nikoloz Turazashvili (@axrisi) — Mon, 12 Oct 2026 02:30:00 +0000

Local AI gives you control over where your model runs. Model security also depends on what your loader executes along the way. Researchers have documented Hugging Face model packages containing reverse-shell payloads, and an unsafe loading path can give that code the same access as your Python process. Your machine gets the model and, potentially, an ambitious new colleague.

TL;DR

Model weights are learned numerical data. A model repository can also carry configuration, executable serialization and Python files.
JFrog's Goober Two example and ReversingLabs' nullifAI samples contained reverse-shell payloads. The selected reports do not establish credential theft from actual victims.
A separate Checkmarx research demonstration showed how a missing local model path could resolve to a public repository. Remote-code trust was an important execution prerequisite.
Safetensors, current patched loaders, reviewed revisions and isolation address different parts of the loading path.
Readable secrets and available network access determine how much damage code running in the loader can do.

Model security begins before the first answer

Hugging Face is a platform for sharing machine learning models, datasets and applications. A repository is a useful distribution unit: it can contain the weights, the configuration needed to interpret them and, for custom architectures, implementation code. The contents deserve separate questions because their loading behavior differs.

The weights are the learned numbers. A tensor-data reader interprets their shapes and values. Python object serialization can describe how to rebuild objects, including calls to functions. Repository Python code introduces another execution path when your application authorizes it. Putting all these files under the heading “model” makes installation convenient and the trust boundary easy to miss.

Hugging Face's Pickle Scanning documentation states the central warning plainly: “There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file.” The significant verb is load. Saving bytes to disk and handing those bytes to an unsafe deserializer are different actions.

For the incidents discussed here, the file's instructions become dangerous when a loading path follows them. That can happen before inference begins. A malicious package does not need to persuade the model to reveal anything in its answer if the surrounding software already lets its code run. The benchmark can wait; the process has started accepting guests.

Pickle security: opening the parcel

Pickle stores Python objects with instructions for reconstructing them. Deserialization reads those instructions in sequence. An unsafe loader may resolve and call functions during reconstruction, which gives an untrusted file influence over the host process.

Think of the package as a parcel and the loader as the person opening it. Downloading places the parcel on the shelf. Loading follows the enclosed assembly instructions. If those instructions can invoke arbitrary functions, the activity has crossed from reading data into executing the publisher's choices.

The exact loading API, library version and settings matter. A security statement about one unsafe path cannot be extended to every model loader. Modern restricted loading modes deliberately reduce what object reconstruction is allowed to do. Conversely, overriding a rejection because an unfamiliar checkpoint “needs it” can reopen the boundary you were relying on.

This is also why “poisoned weights” needs explanation. The phrase can refer to model behavior changed by training or editing numerical parameters. The cases in this episode concern executable packaging and repository code around model distribution. Those mechanisms have their own entry points, permissions and defenses.

Three concrete examples, with their limits

Goober Two: a documented reverse shell

In February 2024, JFrog analyzed baller423/goober2. Its Pickle payload initiated a reverse shell during unsafe loading. A reverse shell connects outward and offers command access to the other end of the connection.

The repository was deleted. JFrog also discussed clues suggesting possible researcher involvement, so attribution remains uncertain. The report supplies evidence for the payload's behavior. It does not establish that criminals stole a named victim's cloud keys, and it does not justify inventing a victim count.

That distinction leaves the mechanism plenty worrying. Command access under the loader's account is a meaningful foothold. A development environment often gives that account more access than a model experiment needs.

nullifAI: unusual packaging and an error that arrived late

In February 2025, ReversingLabs described two model samples, including glockr1/ballr7, under the name nullifAI. Both contained platform-aware reverse-shell payloads.

An important detail is the archive format. These samples used seven-zip packaging where ordinary PyTorch loading expected a different structure. Default torch.load would reject that packaging. The dangerous path required extraction followed by unsafe Pickle deserialization. A diagram showing an ordinary model download immediately opening these samples would skip a real prerequisite.

The report also examined broken Pickle streams and a historical scanner blind spot. Its harmless file-writing research test showed that an earlier instruction could execute before a later error stopped loading. The file already existed when the loader complained. Imagine releasing a raccoon from a parcel and then discovering that the shipping label is invalid. The complaint may be accurate; the timing is deeply unhelpful.

ReversingLabs disclosed the finding on January 20, 2025. Hugging Face removed the reported samples within twenty-four hours and changed the scanner. This example describes a historical finding and its response. It supplies a reason to investigate scanner failures, rather than a claim that the same bypass remains unpatched.

Model Confusion: a missing folder resolves somewhere else

Checkmarx's January 2026 Model Confusion article describes a research demonstration involving an unnamed company's sample. A relative identifier such as checkpoints/some-model could refer to a local path or a public model namespace. If the local path was missing, resolution could select a matching remote repository.

Combine that ambiguity with insecure loading and remote-code trust, and the selected repository's code can execute. The company's sample was corrected within a day, according to Checkmarx. The report explicitly describes the demonstration and says it had not found malicious activity in the sensitive-name examples it highlighted.

Two conditions are doing work here: selecting the unexpected remote package, and authorizing an execution-capable loading path. Checking only whether a string looks like a local folder misses the resolution step. A missing directory should not get to improvise its own landlord.

Where readable credentials enter the flow

Once code runs in a process, its access depends on the account, filesystem mounts and other controls around that process. An accessible cloud key, a token available in the environment or a readable private-key file can become a target. Secrets kept outside that access boundary remain outside the straightforward read path.

Sending a secret elsewhere also requires an available network route. Read access and outbound connectivity are independent conditions. Blocking egress can constrain a transfer; keeping the secret out of the process can prevent the read in the first place. Numerical weights alone supply neither capability.

The selected reverse-shell reports document command-access payloads. Credential collection and exfiltration are conditional consequences of that access, rather than proven victim outcomes in these reports. The episode's key movement is therefore labeled “possible consequence,” and the original key remains in place while a copy follows the possible route.

This is where a familiar development setup gets awkward. A container with your home directory or credentials mounted inside exposes those mounted resources to code in the container. The namespace boundary can be useful, but the key you deliberately put inside it is still inside it. A container full of secrets is a gift basket with resource limits.

Safetensors, patched loaders and a useful isolation boundary

Safetensors provides a tensor-data format designed for safe storage without Pickle's executable object reconstruction. Choosing it addresses that serialization route. A repository can still contain Python code alongside the tensor files, so review the code-loading path as a separate decision.

Hugging Face's custom model loading documentation warns about inadvertently executing malicious code and describes trust_remote_code=True. It also recommends pinning a specific revision. Review the exact revision you intend to run, then keep that decision stable while you evaluate the package.

PyTorch's serialization documentation explains that torch.load defaults to weights_only=True starting with version 2.6 when no pickle_module is passed. The restricted unpickler narrows the remote-code-execution surface. The documentation also retains limitations involving denial of service and possible memory corruption, so the setting deserves its actual scope.

For unfamiliar packages, I would separate acquisition and inspection from execution. Prepare a clean environment with only the files the experiment needs. Leave personal keys outside it. Restrict network access according to the task, and make repository-code trust an explicit reviewed choice. The useful question is what a process could reach if its loading behavior surprised you.

Verdict: NEEDS REVIEW

I'd keep open models and put the permission boundary around the loader. The documented examples show why model distribution needs the same attention we give other executable inputs. Free access to a model is valuable; deciding who gets access to the development environment is also valuable.

FAQ

Can downloading a Hugging Face model steal my credentials?

The mechanisms discussed here execute during unsafe loading or authorized repository-code execution. Merely saving the model bytes is a different step. Review the path that will open them.

Does safetensors make a whole repository safe?

It addresses tensor serialization. Review custom Python and dependencies separately, and keep the loader's access appropriately narrow.

Does a failed load mean nothing happened?

Earlier instructions may have executed before a later error. ReversingLabs' harmless test demonstrated exactly that ordering.

Were credentials proven stolen in these examples?

The selected reports establish reverse-shell payloads and a code-execution research demonstration. They do not establish credential theft from actual victims. Readable credentials are a conditional exposure worth preventing.

Sources

This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.

Video transcript and primary sources

I Set Every Seed and Trained Twice. The Models Diverged at Step 1

b1ackjack12 — Mon, 12 Oct 2026 01:16:31 +0000

There's a ritual at the top of every training script, mine included: torch.manual_seed(0), np.random.seed(0), and the comfortable belief that the run is now reproducible. I've typed it for years. Last week, while chasing a one-point accuracy difference I couldn't explain, I finally tested the belief directly: same seed, same data order, same model init, same machine, same everything — train ResNet18 for 200 steps, twice, and compare.

The two loss curves matched at step 0 and diverged at step 1. After 200 steps, the largest difference between corresponding weights was 0.237 — for parameters that mostly live between −0.1 and 0.1, that's not drift, that's a different model. Both runs were "seeded." Neither was reproducible.

Where the randomness sneaks in

The seeds did their job: initialization, data order, and every explicit random draw were identical, which is why step 0 agreed perfectly. The divergence enters in the backward pass, and it isn't randomness in the usual sense at all — it's floating-point arithmetic meeting parallel hardware.

Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit. GPU kernels sum gradient contributions across thousands of threads, many using atomic operations whose completion order depends on the scheduler — which threads won the race on this particular run, on this particular clock. Same numbers, different addition order, last-bit differences in the gradients. Then SGD amplifies them: a 1e-8 disagreement in a gradient becomes a slightly different weight, which becomes a different activation, and the gap compounds every step. By my run's end, noise that began around the last decimal place a float can represent had grown into 0.237. It's the butterfly effect with a commit hash.

None of this is a PyTorch bug; it's documented behavior, and cudnn.benchmark (which I've previously measured and left enabled) adds a second layer by autotuning to potentially different — nondeterministic — conv algorithms per run.

Does 0.237 matter? Only when you're asking small questions

For final accuracy, this nondeterminism behaves like one more seed you didn't choose — my augmentation experiment put ordinary seed-to-seed spread at ±0.3–0.5 points on a small CIFAR setup, and same-seed GPU noise lands in the same statistical bucket. If your effect size dwarfs that, you can ignore this whole post.

The cases where it bites are the small questions, which are most of debugging. "Did my refactor change the numerics?" — you can't diff against the pre-refactor run, because nothing diffs clean, ever. "Is this 0.4-point improvement my new loss term or luck?" — under nondeterminism you can't even hold the luck constant while you toggle the loss term. A/B comparisons where the honest answer requires the same noise on both sides are exactly where "seeded but nondeterministic" quietly wastes days. I know because the one-point mystery that started this post turned out to be two runs of the same code.

The fix, and its real price

PyTorch ships the switch; it just isn't the seed:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
# and, for CUDA >= 10.2, before the process starts:
# CUBLAS_WORKSPACE_CONFIG=:4096:8

With those set, my two 200-step runs matched bit-for-bit: max weight difference exactly 0, every loss value identical to the last digit. Two footnotes from the setup: the CUBLAS_WORKSPACE_CONFIG environment variable is mandatory (you get a clear RuntimeError without it), and use_deterministic_algorithms will refuse to run ops that simply have no deterministic GPU implementation — ResNet18 sailed through, but if your model contains one of the holdouts, the flag fails loudly and your options are CPU fallback or living without determinism for that op. Loud failure, for once, is the feature.

The folklore price for all this is "2–3x slower," usually delivered as a reason not to bother. Measured, on the ResNet18 training step:

precision	default	deterministic	cost
fp32	17.94 ms	20.90 ms	+16.5%
bf16 autocast	12.42 ms	14.37 ms	+15.6%

Sixteen percent. Some of that isn't even determinism proper — it includes giving up cudnn.benchmark, which I measured at ~3% on this model. The 2–3x stories presumably come from specific unlucky ops or older stacks, and maybe your model contains one; that's what the two-minute measurement is for. On mine, bit-exact reproducibility costs about as much as a modestly larger batch.

Where the switch sits now

My default stays fast and nondeterministic — for training throughput, 16% is real money, and ordinary training doesn't need bit-exactness. But the flag has earned a permanent spot in three situations: any debugging session that compares two runs (which is to say, any debugging session), numerics-sensitive refactors where I want git diff-grade certainty that outputs didn't change, and small-effect experiments where the noise floor would otherwise eat the result. The deciding question is one sentence: am I about to compare two runs and interpret the difference? If yes, 16% is the cheapest insurance in this entire optimization series — and unlike the seeds ritual, it actually buys what it claims to.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

Model CPU offload on a 48 GB Mac: 18.5 GB footprint and 9.6 GB of swap

Rodion Kazennov — Sun, 11 Oct 2026 05:44:25 +0000

The second stageload note ran Qwen-Image-2.1 on a 48 GB Mac two ways. The memory guard stopped eager loading in the VAE decode, and staged loading finished. diffusers has a third way built in, enable_model_cpu_offload, which keeps each model on the CPU and moves it to the GPU only while it runs. On 8 October I ran it once, with the same prompt, size, steps, seed and guard limits as the runs of 6 October.

By the process footprint, offload looked like staged: it peaked at 18.5 GB, staged at 19.0 GB. Swap tells them apart. Encoding took 8 s and added no swap. During the 87 s of denoising, swap grew by 3.5 GB and available memory fell to 27 %. The VAE decode took swap growth to 9.6 GB and available memory down to 22 %, and the guard stopped the run 105 s in, before it wrote an image. Staged added no swap in either of its two runs and finished in 94 and 124 s.

The footprint cannot show this, because it does not count pages that are already in swap. On a Mac the CPU and the GPU share one pool of memory, so a model kept "on the CPU" sits in the same RAM as the model that is running. Judged by footprint alone, offload would sit next to staged. Judged by swap, it sits next to eager, which the guard stopped at the same point after 8 GB of new swap.

What I could not show

This is one run. A second would have pushed the machine into swap again for an outcome the guard had already decided. I did not measure which pages went to swap, and I did not try offload on a Mac with more memory.

The script is bench/qwen_image_offload.py, and the trace is in the repository. stageload installs with pip install stageload.

Building a Transformer from Scratch: Part 1 — The Embedding Layer

Dhruv — Sat, 10 Oct 2026 11:04:44 +0000

In an autoregressive Transformer such as GPT, the work begins before the attention mechanism processes a single tensor. A model cannot operate on raw text, and integer token IDs carry no semantic structure on their own. The embedding layer bridges this gap by converting discrete token IDs into continuous, high-dimensional vectors that encode both the meaning of each token and its position in the sequence.

This article explains how the embedding layer works and presents a clean PyTorch implementation.

The Problem Embeddings Solve

A tokenizer splits raw text into segments and maps each one to an integer index. For example, the word "transformer" might map to 41551 (an illustrative value).

Passing these integers directly into a neural network is problematic, because numerical values imply an ordinal relationship that does not exist. Token 41552 is not "greater than" token 41551 in any meaningful sense.

The solution uses two learned lookup tables:

Token embeddings (W_e) map each token ID to a continuous vector of dimension d_model. Through backpropagation, tokens that appear in similar linguistic contexts converge toward similar regions of the vector space.
Positional embeddings (W_p) encode word order. Transformers process all tokens in parallel rather than sequentially, so without explicit position information, "dog bites man" and "man bites dog" would be indistinguishable to self-attention. Each position index 0, 1, ..., T-1 is therefore mapped to a learned vector of dimension d_model.

PyTorch Implementation

The module below combines token embeddings with learned positional embeddings, following the architecture used in GPT-2.

import torch
import torch.nn as nn

class TransformerEmbeddings(nn.Module):
    def __init__(self, vocab_size: int, d_model: int, max_seq_len: int):
        super().__init__()
        # Token lookup matrix: shape (vocab_size, d_model)
        self.token_embeddings = nn.Embedding(vocab_size, d_model)

        # Position lookup matrix: shape (max_seq_len, d_model)
        self.position_embeddings = nn.Embedding(max_seq_len, d_model)

    def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
        # Expected input shape: (batch_size, seq_len)
        batch_size, seq_len = input_ids.shape

        # Guard clause: protect against context window overflow
        if seq_len > self.position_embeddings.num_embeddings:
            raise ValueError(
                f"Sequence length ({seq_len}) exceeds maximum context window "
                f"({self.position_embeddings.num_embeddings})"
            )

        # Generate positional indices [0, 1, ..., seq_len - 1]
        # Shape: (1, seq_len), placed on the same device as the input
        positions = torch.arange(0, seq_len, device=input_ids.device).unsqueeze(0)

        # Retrieve representations
        tok_emb = self.token_embeddings(input_ids)      # (batch_size, seq_len, d_model)
        pos_emb = self.position_embeddings(positions)   # (1, seq_len, d_model)

        # Element-wise addition with broadcasting across the batch dimension
        return tok_emb + pos_emb

Key Implementation Details

1. How `nn.Embedding` Works

nn.Embedding is a trainable weight matrix:

W ∈ R^(vocab_size × d_model)

It is mathematically equivalent to multiplying a one-hot vector by this matrix (x_one_hot @ W), but it is implemented as a direct row lookup. This avoids constructing large, sparse one-hot tensors and removes the associated memory and compute overhead.

2. Device Placement and Broadcasting

Device matching. Passing device=input_ids.device ensures that the position indices are created on the same device (CPU, CUDA, or Apple Silicon MPS) as the input. Omitting it defaults to the CPU and raises a device mismatch error when training on a GPU.
Broadcasting. The positional embedding tensor has shape (1, seq_len, d_model), while tok_emb has shape (batch_size, seq_len, d_model). When the two are added, PyTorch broadcasts the position vectors across every sequence in the batch.

3. Why Addition Rather Than Concatenation

Concatenating the two vectors would widen each token representation to 2 × d_model, increasing the parameter count and memory usage of every subsequent linear projection.

Element-wise addition preserves the channel dimension at d_model. Conceptually, the positional embedding acts as an offset applied to the token vector, allowing the model to distinguish the same token at different positions.

Next Steps

The combined embeddings leave this layer with shape (batch_size, seq_len, d_model) and feed directly into the first Transformer block. The next article in this series covers the attention mechanism.

model.train() vs model.eval() vs torch.no_grad() in PyTorch: What Each One Changes

Wesam Khallaf — Author of PyTorch From Ground Up — Sat, 10 Oct 2026 07:43:54 +0000

PyTorch has three calls that are used around evaluation: model.train(), model.eval() and torch.no_grad(). They control two separate things. train() and eval() set a mode flag that changes how some layers compute their output. torch.no_grad() controls whether autograd records the computation. This article explains each one, shows which layers are affected, and gives the standard train-and-evaluate loop.

The training flag

Every nn.Module has a boolean attribute training. model.train() sets it to True on the model and on every sub-module. model.eval() sets it to False. A new model starts in training mode.

import torch
import torch.nn as nn

torch.manual_seed(0)
model = nn.Sequential(
    nn.Linear(10, 32),
    nn.BatchNorm1d(32),
    nn.ReLU(),
    nn.Dropout(0.5),
    nn.Linear(32, 3),
)

print(model.training)
model.eval()
print([m.training for m in model])
model.train()
print([m.training for m in model])

Output:

True
[False, False, False, False, False]
[True, True, True, True, True]

model.eval() is the same call as model.train(False). Both return the model itself, so model = model.eval() and model.eval() do the same thing.

Setting the flag does nothing by itself. It only matters for layers that read it in their forward(). In the standard layer set, these are mainly the dropout layers and the normalization layers that keep running statistics (BatchNorm1d, BatchNorm2d, BatchNorm3d). nn.Linear, nn.Conv2d, nn.ReLU, nn.LayerNorm and most other layers compute the same output in both modes.

A model made only of Linear and ReLU layers gives identical outputs in both modes:

plain = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 3))
x = torch.randn(4, 10)

plain.train(); a = plain(x)
plain.eval();  b = plain(x)
print(torch.equal(a, b))

True

Dropout in each mode

In training mode, nn.Dropout(p) sets each element of its input to zero with probability p, and multiplies the remaining elements by 1 / (1 - p). In evaluation mode it returns its input unchanged.

drop = nn.Dropout(p=0.5)
ones = torch.ones(10)

drop.train(); print(drop(ones))
drop.eval();  print(drop(ones))

Output:

tensor([0., 0., 2., 0., 0., 0., 2., 2., 0., 2.])
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

With p = 0.5, the surviving elements are multiplied by 2. This keeps the expected value of each element the same in both modes. Over one million ones, the mean of the training-mode output is 1.0004, and the mean of the evaluation-mode output is exactly 1.0. Because of this scaling, the layers after dropout receive inputs of the same average size in training and in evaluation, so no extra correction is needed at test time.

Since the dropped elements are chosen at random on every call, the same input gives a different output on every forward pass in training mode. For a small model with one Dropout(0.5) layer, ten forward passes of one input give:

train mode: 10 different outputs in 10 passes
eval mode:   1 different output  in 10 passes

BatchNorm in each mode

nn.BatchNorm1d normalizes each feature to mean 0 and variance 1, then applies a learned scale and shift. The difference between the modes is where the mean and variance come from.

Training mode: the mean and variance are computed from the current batch. The layer also updates two stored buffers, running_mean and running_var, as a moving average of the batch statistics.
Evaluation mode: the layer uses the stored running_mean and running_var, and does not change them.

The buffers start at mean 0 and variance 1. One forward pass in training mode moves them 10% of the way towards the batch statistics (the default momentum=0.1):

torch.manual_seed(0)
bn = nn.BatchNorm1d(3)
data = torch.randn(64, 3) * 2 + 5          # features with mean about 5, std about 2

print(data.mean(0))
bn.train(); _ = bn(data)
print(bn.running_mean)
print(bn.running_var)

tensor([5.0390, 4.4964, 5.5362])
tensor([0.5039, 0.4496, 0.5536])
tensor([1.2227, 1.2167, 1.3412])

0.5039 is 0.9 * 0 + 0.1 * 5.0390. The running statistics are updated by the forward pass. No backward() call or optimizer step is involved.

In training mode, one sample's output depends on the rest of the batch

Because training mode normalizes with the batch's own statistics, the output for a sample changes when the other samples in the batch change. Below, the same sample s is passed in two different batches, after the layer has seen 200 training batches:

train mode, s in batch A:  [ 0.4099, -0.3283,  0.1971]
train mode, s in batch B:  [-1.6630, -1.8497, -1.7948]
eval mode,  s in batch A:  [-0.1599, -0.2482, -0.1628]
eval mode,  s in batch B:  [-0.1599, -0.2482, -0.1628]
eval mode,  s alone:       [-0.1599, -0.2482, -0.1628]

In evaluation mode the output for s is the same in any batch and on its own, because the statistics are fixed.

A batch of one sample in training mode

A batch of one sample has no variance to compute, so BatchNorm1d refuses it in training mode:

bn.train()
bn(torch.tensor([[5.0, 5.0, 5.0]]))

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 3])

This error usually appears when a single prediction is made on a model that is still in training mode, or when the last batch of an epoch has one sample. In the first case, the fix is model.eval(). In the second, DataLoader(..., drop_last=True) drops the incomplete final batch.

What happens when evaluation runs in training mode

The effect of a missing model.eval() can be measured directly.

A model with Dropout. This is a binary classifier with one Dropout(0.3) layer, trained for 20 epochs on 480 samples and evaluated on 120:

torch.manual_seed(0)
x = torch.randn(600, 10)
y = (x[:, 0] + x[:, 1] > 0).long()

model = nn.Sequential(
    nn.Linear(10, 32), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(32, 2))

After training, the same validation set was evaluated once in evaluation mode and five times in training mode:

eval mode:   val loss 0.0462   accuracy 0.992
train mode:  val loss 0.0520   accuracy 0.967
train mode:  val loss 0.0772   accuracy 0.967
train mode:  val loss 0.0728   accuracy 0.967
train mode:  val loss 0.0831   accuracy 0.967
train mode:  val loss 0.0520   accuracy 0.983

In training mode, the measured loss is higher and changes from run to run, because dropout removes a different 30% of the hidden units each time. The weights are the same in all six rows.

A model with BatchNorm. This is a 3-class classifier with BatchNorm1d and Dropout(0.3), trained for 30 epochs. Its validation accuracy was measured with different validation batch sizes:

validation batch size     train-mode accuracy    eval-mode accuracy
--------------------------------------------------------------------
600                              0.913                 0.932
64                               0.903                 0.932
8                                0.795                 0.932
2                                0.622                 0.932

In evaluation mode the accuracy does not depend on the batch size. In training mode it falls as the batch gets smaller, because the mean and variance of 2 or 8 samples are poor estimates of the statistics the model was trained with.

Validating in training mode also changes the model. After one pass over the validation set in training mode, num_batches_tracked went from 1140 to 1150 and the largest change in running_mean was 0.1975. The validation data has been mixed into the stored statistics, and these are the statistics the model will use later for real predictions.

What torch.no_grad() does

model.eval() changes how layers compute their output. It does not stop autograd. In evaluation mode, the forward pass still builds a computation graph:

model.eval()
out = model(x)
print(out.requires_grad, out.grad_fn)

with torch.no_grad():
    out = model(x)
print(out.requires_grad, out.grad_fn)

True <AddmmBackward0 object at 0x...>
False None

To compute gradients later, autograd stores some intermediate results of the forward pass, such as the input to each linear layer and the output of each ReLU. These stored tensors stay in memory until backward() runs or the output is deleted. Inside torch.no_grad(), nothing is recorded and nothing is stored.

For a network with layers 1024 -> 4096 -> 4096 -> 10, the activations stored for backward are:

batch size     stored with graph     stored under no_grad
---------------------------------------------------------
256                  17.0 MiB              0.0 MiB
4096                272.0 MiB              0.0 MiB

The stored memory grows with the batch size. On a GPU, this is the memory that a validation loop without no_grad() uses for no purpose, and it limits how large the validation batch can be.

A tensor produced under no_grad() has no graph, so calling backward() on it raises:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Why both are needed during evaluation

The two calls do not overlap. no_grad() does not switch layers to evaluation mode. A model in training mode, run under no_grad(), still applies dropout:

model.train()
with torch.no_grad():
    o1, o2 = model(x), model(x)
print(torch.allclose(o1, o2))

False

no_grad() also does not stop BatchNorm from updating its running statistics. Those updates happen in the forward pass and depend only on the training flag:

bn.train()
with torch.no_grad():
    _ = bn(data)          # running_mean still changes

And eval() does not stop the graph from being built, as shown above. The two calls are separate because there are cases where only one is wanted. For example, some interpretability methods compute gradients with respect to the input of a model in evaluation mode, which needs eval() without no_grad().

Call	What it changes	What it does not change
`model.train()`	sets `training=True`: dropout active, BatchNorm uses batch statistics and updates running statistics	gradient recording
`model.eval()`	sets `training=False`: dropout off, BatchNorm uses stored running statistics	gradient recording
`torch.no_grad()`	no graph is built, no activations are stored	layer behaviour, BatchNorm running statistics

torch.inference_mode()

torch.inference_mode() is a stricter version of no_grad(). It also skips some internal bookkeeping (version counters and view tracking), so it can be slightly faster. The cost is that tensors created inside it can never be used in a computation that autograd records:

model.eval()
with torch.inference_mode():
    out = model(x)                       # shape (4, 3)

w = torch.randn(3, requires_grad=True)
(out * w).sum().backward()

RuntimeError: Inference tensors cannot be saved for backward. Please do not use Tensors created in inference mode in computation tracked by autograd. ...

Use inference_mode() for prediction code whose outputs are not used for training. Use no_grad() in a validation loop or anywhere the results may be used in a later computation that needs gradients. Like no_grad(), inference_mode() does not change the training flag, so model.eval() is still required.

The train-and-evaluate loop

The training phase starts with model.train(). The validation phase starts with model.eval() and runs inside torch.no_grad().

for epoch in range(num_epochs):

    # training phase
    model.train()
    train_loss = 0.0
    for bx, by in train_loader:
        pred = model(bx)
        loss = loss_fn(pred, by)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        train_loss += loss.item() * bx.size(0)

    # validation phase
    model.eval()
    val_loss = 0.0
    with torch.no_grad():
        for bx, by in val_loader:
            pred = model(bx)
            val_loss += loss_fn(pred, by).item() * bx.size(0)

    t = train_loss / len(train_loader.dataset)
    v = val_loss / len(val_loader.dataset)
    print(f"Epoch {epoch:2d}  Train: {t:.4f}  Val: {v:.4f}")

model.train() is at the top of every epoch, not once before the loop. The validation phase leaves the model in evaluation mode, so without this line every epoch after the first would train with dropout off and with BatchNorm frozen at its running statistics.

On the Dropout example above, this loop gives:

Epoch  0  Train: 0.6110  Val: 0.4610
Epoch  5  Train: 0.1028  Val: 0.0940
Epoch 10  Train: 0.0489  Val: 0.0609
Epoch 15  Train: 0.0390  Val: 0.0574
Epoch 19  Train: 0.0367  Val: 0.0462

The training loss is computed in training mode, with dropout active, while the validation loss is computed in evaluation mode. The training loss is also averaged over the whole epoch, while the weights are still improving, and the validation loss is measured at the end of the epoch. Both reasons explain why the validation loss can be lower than the training loss.

For predictions after training, the pattern is the same as the validation phase:

model.eval()
with torch.no_grad():
    logits = model(new_x)
    preds = logits.argmax(dim=1)

Summary

model.train() and model.eval() set the training flag on the model and all its sub-modules. The flag only affects layers that read it, mainly Dropout and BatchNorm.
Dropout in training mode zeroes elements with probability p and scales the rest by 1 / (1 - p). In evaluation mode it does nothing.
BatchNorm in training mode uses the current batch's statistics and updates running_mean and running_var. In evaluation mode it uses the stored values and does not change them.
torch.no_grad() stops autograd from building a graph and storing activations. It does not change layer behaviour and does not stop BatchNorm's running statistics from updating.
Evaluation needs both: model.eval() and torch.no_grad(). Call model.train() at the start of every training epoch.
torch.inference_mode() is a stricter no_grad() for prediction code whose outputs will not be used with autograd.

This is one chapter's worth of an idea from my book, PyTorch From Ground Up, which builds everything from tensors upward so nothing stays vague. If it helped: 8 chapters are free, no email required, there's a free one-page tensor cheat-sheet here, every example runs in the companion notebooks on GitHub, and the full book is on Leanpub or in paperback and Kindle on Amazon.

More in this series

How Training Actually Works:

Shape mechanics:

Next in this series: saving and loading a model with state_dict() and checkpoints.

How to make GPU training survive spot preemption (without babysitting it)

Viswaretas Kotra — Fri, 09 Oct 2026 01:59:34 +0000

Short answer: save your model, optimizer and step counter to one directory, write those files atomically, load them on startup, and run on a platform that copies that directory before the machine disappears and restores it on the next machine. Do that and a preemption costs you minutes of work, not the whole run.

Spot and interruptible GPUs are often a fraction of on-demand prices. The catch is that the provider can take the machine back. Here is the pattern we use at Nodus, and it works anywhere.

1. Put all recovery state in one directory

Everything you need to continue goes in one place: weights, optimizer state, LR scheduler, RNG state, data loader position, current step. Results you want to download (final model, eval reports) go somewhere else. Mixing the two makes checkpoints big and slow.

On Nodus that directory is /nodus/state (also in $NODUS_STATE_DIR), and outputs go to /nodus/outputs.

2. Write atomically

A checkpoint taken while you are halfway through writing a file restores a broken state. Write to a temp file, then rename it over the old one. Rename is atomic on POSIX filesystems.

import json, os
from pathlib import Path

STATE_DIR = Path(os.environ.get("NODUS_STATE_DIR", "/nodus/state"))
STATE = STATE_DIR / "progress.json"

def save(step, model, opt):
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    tmp = STATE_DIR / "ckpt.pt.tmp"
    torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step}, tmp)
    os.replace(tmp, STATE_DIR / "ckpt.pt")

3. Resume on startup

Checkpoints restore files, not process memory. Your program starts from the top, so it has to check for saved state:

start = 0
ckpt = STATE_DIR / "ckpt.pt"
if ckpt.exists():
    s = torch.load(ckpt)
    model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"]); start = s["step"]
    print(f"resumed from step {start}")
for step in range(start + 1, total_steps + 1):
    ...

4. Pick a checkpoint cadence

Too often and you waste GPU time writing files. Too rarely and each preemption throws away a lot of work. A practical rule: aim for at least four checkpoints per expected run, and keep checkpointing under about 10% of runtime. Capacity that is rarely interrupted can be checkpointed less often.

Nodus does this math for you with interval: auto, using how often that capacity actually gets interrupted and how long your saves take.

5. Save when the machine is about to go away

Most providers give a short reclaim notice. Use it. On Nodus your program can subscribe to checkpoint requests over a local socket and acknowledge when files are consistent:

import nodus  # pip install nodus-compute; no-ops outside Nodus

nodus.checkpoint.on_request(lambda: save(step, model, opt))

When the capacity gives notice, Nodus sends an urgent request, takes the checkpoint after your ack, and prepares a replacement machine at the same time. The replacement only starts once the old one is provably gone, so two machines never write the same state.

6. Hugging Face Trainer users

Trainer already saves and resumes. Point output_dir at the state directory, set save_steps and save_total_limit=2, and call trainer.train(resume_from_checkpoint=True) when NODUS_RESTORED=1. Or set integration: HFTrainer and Nodus registers the save-on-request callback for you.

Running it

pip install nodus-compute
nodus login
nodus run --gpu H100 --interruptible --checkpoint /nodus/state -d -- python train.py

nodus describe job/<name> shows each attempt, why it ended (for example Preempted) and the latest checkpoint. New accounts get a $30 starter grant, so you can test a preemption-safe run without a card.

FAQ

Does checkpointing restore GPU memory? No. It restores files. Your code reloads its own state.

What if a checkpoint is empty? An empty state directory never replaces an earlier useful checkpoint, so a crash before the first save cannot erase progress.

How many times will it retry? By default up to 8 recoveries (3 for distributed jobs). Two attempts in a row with no progress stops the run with NoProgress.

Does this work for multi-node? Yes, in beta. Each rank writes its shard with torch.distributed.checkpoint and rank 0 commits it.

Full guide: nodus-compute.ai/docs/guides/checkpoints

Loading a PyTorch pipeline one stage at a time: Pixal3D on a 48 GB Mac

Rodion Kazennov — Mon, 05 Oct 2026 13:00:36 +0000

On 4 October a Pixal3D run on my Mac went into its texture stage, and within ten seconds the process grew from 27 to 44 GB. Available memory fell to 6 % and swap grew by 12 GB, so the guard I run heavy jobs under stopped it. The Mac has 48 GB. Before I had that guard, two Pixal3D runs, one of them next to other heavy work, ended with the Mac rebooting.

Pixal3D turns one image into a textured 3D mesh, and Pixal3D-mac is its port to Apple Silicon. The 1024_cascade pipeline runs seven models in turn: a flow model and a decoder for the coarse structure, two flow models for the shape at 512 and 1024, a shape decoder, and a flow model and a decoder for the texture. Around them work four image feature extractors, a background remover and a camera estimator. Each stage needs one or two of the seven models, and the port keeps all of them in memory from the first second to the last.

Why the memory does not come back

On Apple Silicon the CPU and the GPU share one pool of memory. Pixal3D's low-VRAM mode, which is on by default, keeps idle models on the CPU and copies the active one to the GPU. On a PC with a separate graphics card, that saves video memory. On a Mac the CPU copy and the GPU copy sit in the same RAM, so the mode keeps every model resident and adds a second copy of the active one. The four feature extractors also load the same DINOv3 backbone, four times.

Loading one stage at a time

stageload is a small library I wrote for this. It swaps the pipeline's dictionary of models for one that loads a model the first time a stage asks for it, and that releases every model the next stage does not list. Releasing moves each tensor of the module to PyTorch's meta device. That frees the storage even while other code still holds a reference to the module, and calling the module afterwards raises an error that names it and the stage that released it, instead of failing somewhere inside PyTorch.

The port's own run() executes unchanged. stageload wraps four of the pipeline's methods to learn where each stage begins: background removal, the first image conditioning, the conditioning of each shape and texture pass, and the final decoding. The four feature extractors are built so that they share one backbone.

One part of this raises no error when it goes wrong. Pixal3D seeds once at the start of a run and then draws the noise of every stage from the CPU random generator. Building a model runs its random initialisation, which draws from that same generator. A model built lazily in the middle of the run would therefore change the noise of every stage after it, and with it the mesh. stageload saves the generators' state before each load and restores it afterwards: torch's CPU, MPS and CUDA generators, Python's random and NumPy's global one. On a stand-in pipeline that seeds and draws noise the way the port does, a test checks that staged and eager runs produce the same latents bit for bit, and a control that turns the restore off sees them drift apart.

Not only Pixal3D

Nothing in the core of stageload knows about Pixal3D. A pipeline needs three things from it: a function that builds each model on its own, a list of the models each stage uses, and a call where each stage begins. In code you own, that call is one line, models.enter("decode"). In someone else's code, like this port, on_call attaches it to a method that already runs at that point, so the pipeline itself stays as it is. A text-to-image pipeline has the same shape as Pixal3D: a text encoder, a denoiser and an image decoder run one after another. The memory meter and the guard work for any program on a Mac. Pixal3D is the only pipeline I have measured so far; the README shows the core on a small two-model pipeline, and that example runs as a test.

What it changed

On this M5 Pro with 48 GB, I ran the port's example image through 1024_cascade with seed 7 and 2048-pixel textures, two runs per mode, each in its own process under the guard. "Eager" is the port's own loading and "staged" is stageload. Eager runs do not get through the texture stage on this machine within the guard's limits, so the two eager runs stop where it begins. The texture number for eager comes from the run on 4 October that went into it.

stage	eager, peak footprint	staged, peak footprint
setup	20.4 GB	2.9 GB
camera	23.2 GB	6.7 GB
structure	22.2 GB	10.9 GB
shape_512	26.6 GB	11.4 to 11.5 GB
shape_1024	30.5 GB	13.1 GB
texture	stopped by the guard at 44.2 GB	30.1 GB
decode	not reached	17.9 to 18.4 GB
export	not reached	69.6 to 70.2 GB

In every stage both modes ran, staged held 11.3 to 17.5 GB less. The eight loads, the seven models plus the shape decoder a second time for decoding, read 13.2 GB in 24 s out of about ten minutes per run.

Most of the texture stage's 30 GB is working memory: the texture model itself is 2.6 GB. When decoding began, stageload released it together with the feature extractors and emptied the MPS allocator's cache, and within two seconds the footprint was down to 4.5 GB in one run and 8.0 GB in the other.

The highest staged number, about 70 GB, comes in the export, after every model weight has been released. It is the port's mesh processing (remeshing, UV unwrapping, baking), which stageload does not touch. The footprint is the number Activity Monitor shows, and it counts memory macOS has compressed, so it can be larger than the 48 GB of RAM.

What I could not show

The first is whether staged loading changes the result. Every run saved a hash and a copy of each sampler's output. The first latent, the coarse structure, is the same bit for bit in all four runs. From the 512 shape stage on, the port does not repeat itself on MPS even with the same seed: the two eager runs differ by up to 1.77 in the shape latent, and a staged run differs from an eager one by 1.24 to 1.86. So the comparison says any change from staged loading is no larger than the port's own variation, measured on that single pair of eager runs. The texture latent and the final mesh were not compared at all, because eager runs never reach them here.

Building on the CPU did not help. stageload builds each model directly on the GPU, and that changes one tensor the checkpoint does not cover: the rotary frequencies of the sparse attention, computed when a model is built, come out up to 6e-8 apart. To see whether that explained the differences, I ran a staged pass that built every model on the CPU and moved it to the GPU, the way the port does. Its latents differed from the eager runs by 1.84 and 1.12, the same order as every other pair. Loading took 82 s instead of 24 s, and three stages peaked 2.6 to 3.8 GB higher while two came out lower. I went back to building on the GPU.

My own summary had a bug that inflated one number. It took each stage's peak from the moment the stage was recorded as started, but releasing the previous stage's models takes up to 1.7 s. One memory sample in that gap put the decode peak of the CPU-built run at 29.5 GB; counted from the end of the releases, it is 16.7. A code review caught it. A stage's window now begins when the releases its start set off have finished.

The guard

The runs went through a second part of stageload, stageload guard, because other heavy jobs run on this Mac too. It starts one command only when enough memory is available and no process matching a list of heavy jobs has been seen for a while. While the command runs, it stops the command's whole process group if swap grows past a budget or available memory stays under 10 %. It needs no root, and the only processes it ever stops are the ones it started.

The wait for quiet came from a failure. A chain of video jobs leaves short gaps between them, a run started in one of those gaps, and the next video job took swap past the budget within 20 s. In the bench every run started only after three minutes with no heavy job in sight.

On another image

The same staged run on a character from a game I am making took 416 s and loaded the same 13.2 GB in 37 s. Its texture stage peaked at 30.1 GB, as on the example image. Here is the input and the mesh it produced, rendered in Blender:

The code, the traces of every run and the scripts that drew these charts are in the repository. The Pixal3D adapter is tested against one commit of Pixal3D-mac.

DEV Community: pytorch

PyTorch vs. TensorFlow for Diffusion Models: A Dev Leader's Guide to Tooling and Productivity

The Great Debate: PyTorch vs. TensorFlow for Diffusion Models

Dynamic Computation Graphs for Agile Development

Research & Community Leadership

Beyond Frameworks: Understanding State-Space Concepts in AI

Classical State-Space Search: The Foundation of Problem Solving

Strategic Impact: Tooling Choices, Productivity, and Delivery

Top 5 Dev Tools & Tutorials of the Week (October 14, 2026)

1. TensorFlow Quantum 3.0

2. PyTorch Lightning CLI 2.0

3. OpenAI Codex SDK for JavaScript

4. Django AI Framework 1.0

5. LangChain SDK for Python

Building in public: my lead-gen engine

Start with this today

Picks of the week

Question for readers

Connect

MPS backend out of memory on a Mac: what the numbers mean and how to set the watermark

Where "max allowed" comes from

Why a 48 GB Mac swaps first

Setting it: two variables

Picking a value

A trap when you test this

When the model really doesn't fit

PyTorch on a Mac: why memory looks freed after .to("cpu")

What the numbers say

Why it matters for model CPU offload

How to check your own run

What does free the memory

What I did not measure

I Trained a Language Model From Scratch on a 4 GB GPU

I wanted to understand what happens before inference

The constraint: 4 GB of VRAM

Building the corpus

A falling loss, a CUDA crash, and a checkpoint

Evaluation was more interesting than training

Tiny models are brutally honest

What I actually got from it

Where it goes next

Model security on Hugging Face: what executes when you load

TL;DR

Model security begins before the first answer

Pickle security: opening the parcel

Three concrete examples, with their limits

Goober Two: a documented reverse shell

nullifAI: unusual packaging and an error that arrived late

Model Confusion: a missing folder resolves somewhere else

Where readable credentials enter the flow

Safetensors, patched loaders and a useful isolation boundary

Verdict: NEEDS REVIEW

FAQ

Sources

I Set Every Seed and Trained Twice. The Models Diverged at Step 1

Where the randomness sneaks in

Does 0.237 matter? Only when you're asking small questions

The fix, and its real price

Where the switch sits now

Model CPU offload on a 48 GB Mac: 18.5 GB footprint and 9.6 GB of swap

What I could not show

Building a Transformer from Scratch: Part 1 — The Embedding Layer

PyTorch Implementation

Key Implementation Details

1. How nn.Embedding Works

2. Device Placement and Broadcasting

3. Why Addition Rather Than Concatenation

Next Steps

model.train() vs model.eval() vs torch.no_grad() in PyTorch: What Each One Changes

The training flag

Dropout in each mode

BatchNorm in each mode

In training mode, one sample's output depends on the rest of the batch

A batch of one sample in training mode

What happens when evaluation runs in training mode

What torch.no_grad() does

Why both are needed during evaluation

torch.inference_mode()

The train-and-evaluate loop

Summary

1. How `nn.Embedding` Works