TorchTitan Update, October 21, 2026. During PyTorch Conference North America in San Jose, the PyTorch team published TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL. It walks through new support for DeepSeek-V4, Kimi K3 and Qwen3.5, introduces TitanRL for reinforcement learning, and reports DeepSeek-V3 671B training at 2,694,493 tokens per second on 512 GB300 GPUs. The post ends by telling you to clone the repo and follow the quick start.
Step 3 of that quick start downloads a tokenizer with scripts/download_hf_assets.py. I found two bugs in that script, and a test that passed the whole time had been hiding one of them. I opened PR #5621 on October 25, and it merged on October 27.
The update covers the scale end of torchtitan. My side quest happened at the other end, in the script a new user runs before training a real model.
The short version
The download function was handed a list of what to fetch and wrote into that list instead of its own copy, so the caller's list came back one entry longer. Call it twice with the same list and the second call rejects the entry the first call added. The same function also raised an error built from three separate strings, so its message printed as a tuple with literal \n in it, and the test that checked the message could not tell. The fix copies the list and builds one message.
The concept, animated
Figure 1. Trays are list objects and chips are their entries; the wire shows which object the parameter name asset_types refers to. The three states: the first call before the fix, the second call with the same list, and a call after the fix.
download_hf_assets.py is the torchtitan script that downloads a model's tokenizer, config, safetensors index or weights from the Hugging Face Hub before training. You tell it which assets you want as a list, and when you also pass extra file patterns it adds a placeholder entry, additional_patterns, to that list so the download loop picks them up.
The catch is in the word "that". If you passed a string, the function wrapped it in a fresh list and the placeholder went into the fresh list. If you passed a list, the placeholder went into your list. Your list is now ['tokenizer', 'additional_patterns']. The first call still works, because the table of known assets is rebuilt every call and the placeholder is matched during that same call. The second call, with the same list, meets an entry nobody asked for, finds no pattern for it, and stops: Unknown asset type additional_patterns.
Anyone looping over several repos with one shared list of asset types hit this on the second repo.
The second bug, and why a passing test hid it
I found the function through a typo. I typed --assets tokenizers, with a stray "s", and instead of a readable message I got this:
ValueError: ('Unknown asset type tokenizers. Available uses: --assets tokenizer safetensors index config \n', "Or specify exact patterns to download. Example: --additional_patterns '*.safetensors' README.md '*.json' \n", 'Or use --all to download all files')
That is a tuple, with literal \n escapes inside it. The error was raised with three comma-separated strings, and when an exception has more than one argument Python prints the whole tuple as its message.
The existing test for this case used assertIn on a piece of the message. A substring of a tuple's printed form is still a substring, so the test stayed green for as long as the bug existed. A short repro with the Hub calls mocked confirmed both bugs: it printed args: 3, and after one call my list read ['tokenizer', 'additional_patterns']. Before writing any code, I searched torchtitan's issues and PRs for the function and both messages. Neither bug had been reported.
The fix, in one idea
Figure 2. After #5621 the function copies the list first. The copy gets the placeholder entry and the download loop reads the copy; the caller's list is exactly as it was passed in.
Work on a copy. The function now copies the list it was given before adding anything to it, so the caller's list is untouched: list(asset_types) instead of using the caller's object. And the error is now one message built from one string, so it prints as three readable lines. Two edits in one file, 35 lines added and 8 removed across the script and its tests.
The tests changed with it. The existing one now asserts that the exception has exactly one argument before it checks the text, which is what would have caught the tuple. A new one calls the function twice with one list and additional_patterns=["*.txt"], then checks that the list is still ["tokenizer"] and that both calls downloaded the same files.
What changed for torchtitan users
A mistyped asset type now prints three readable lines: Unknown asset type tokenizers. Available uses: --assets tokenizer safetensors index config, then the --additional_patterns example, then the --all option. Scripts that loop over several repos with one shared asset_types list now work past the first repo. Which files get downloaded does not change.
The one behaviour change is for a caller that relied on the mutation. I searched the repo for callers of the function and found none that did. test_tokenizer.py passes a string, which still gets wrapped in a list.
I ran everything on Windows 11 with Python 3.11 and torch 2.16.0.dev20261024+cpu. pre-commit passed flake8, ufmt, pydoclint, codespell and Pyrefly on both files. The lychee link checker needs /bin/bash, so it could not start on Windows. I did not run anything on Linux or GPU, and the PR description says so.
Thank you, tianyu-l
Thank you to @tianyu-l for approving PR #5621 and merging it on October 27, two days after I opened it. It was my first PR to the repo, so CI also needed a maintainer to approve the workflow runs before it could start.
Thanks as well to the PyTorch distributed team at Meta who maintain torchtitan. The repo merged 392 PRs in the 30 days before I opened mine, and outside fixes still got a first review within 0 to 2 days.
I modeled my PR description on #5040, an earlier outside fix to the same function by u7k4rs6. Because #5040 had just touched that function, I picked the smallest change over a rewrite.
What I took away
I don't count a test as protection until I have watched it fail on the bug.
The old test stayed green for as long as the tuple message existed. So after writing the new assertions, I reverted only the script fix and kept the tests: 2 failed and 11 passed. With the fix back, all 13 passed, along with the 40 tests in tests/unit_tests/cpu/scripts.
That revert is the cheapest check I know, and it changes what I can put in a PR. "Tests pass" becomes "these 2 tests fail on main and pass with the change," which a reviewer can verify without taking my word for it. Listing what I did not run does the same job for the parts I could not test.
I'm an ML engineer with a DevOps background, and I'm open to ML engineering roles. My code is on GitHub as Sonica-B, my projects are at shreyaboyane.dev, and I'm on LinkedIn.
Top comments (0)