huggingface_hub v2.3.0 week: how I fixed a Windows crash in hf cache verify

huggingface_hub v2.3.0 came out on October 22, 2026, and its headline number is a good one: a recursive HfFileSystem.find() over allenai/objaverse, about 800k files, went from 776 s to 19.7 s. Lower in the notes is a smaller fix, #5198, for a UnicodeDecodeError under the C locale that broke checkpoint saving in TRL.

That same week I hit the mirror image of that bug. In #5198, text coming into the library failed to decode. In my case, text going out of the hf CLI failed to encode, and it took down hf cache verify on my Windows 11 machine.

The short version

On Windows, Python writes to a terminal in UTF-8 but to a pipe, a file or a CI log in the old ANSI code page, cp1252. hf cache verify began its failure report with a ❌, cp1252 has no ❌, and the command crashed at the exact moment it had bad news to report. The fix prints [FAIL] when the stream cannot encode the emoji. It was merged on October 27.

The concept, animated

Figure 1. The blocks are the real call chain in cli/cache.py and cli/_output.py. The token is the first character the failure report prints. The three states: a console, a pipe on Windows before the fix, and a pipe after it.

Python decides sys.stdout.encoding once, when the program starts, and on Windows it decides differently depending on where the output goes. Into a console: UTF-8, which can encode every character. Into a pipe, a redirected file or a CI step: the ANSI code page, which on my machine is cp1252, a 1990s table of about 250 characters with no ❌ in it. The engraved .encoding on the sys.stdout block in figure 1 flips between the two.

hf cache verify checks that the files in your local Hugging Face cache match the checksums on the Hub, and the first thing it printed when a file was bad was ❌ Checksum verification failed for the following file(s):. In a console, the token passes. In a pipe, sys.stdout cannot encode it: UnicodeEncodeError, position 0. The list of bad files, the one thing you asked for, never comes out, and the command exits with Error: Invalid value. That label is the CLI's name for any ValueError, and UnicodeEncodeError happens to be one, so an output bug looked like bad input.

One line reproduces it with no Hugging Face code at all: PYTHONIOENCODING=cp1252 python -c "print('❌')".

How I ran into it

I use the hf CLI every day on Windows 11, and that week I was exploring it from source (main, which already reported 2.4.0.dev0, on Python 3.11). I wanted the output of hf cache verify saved to a file, as a record of which cached files had gone bad. To get a bad file, I made one: I downloaded config.json from julien-c/dummy-unknown, edited the cached copy so its checksum would no longer match, and ran the check through a Git Bash pipe.

$ hf cache verify julien-c/dummy-unknown --cache-dir ./e2e_cache | cat
Error: Invalid value. 'charmap' codec can't encode character '❌' in position 0: character maps to <undefined>

charmap is how Python names the codec behind cp1252, the character was ❌ at position 0, and the pipe was the last clue. A terminal and a pipe get different encodings, so output that looks fine while you develop can crash in a log file.

Then I searched the repo and found _ascii_safe in cli/_output.py. It came from #4610, which swapped ✓ for [OK] when stdout cannot encode it. That PR also listed the places it had not fixed yet, and ❌ in cache verify was on the list. Before writing code, I checked that no open or closed issue or PR covered it.

The fix, in one idea

Figure 2. After #5255 the command asks _ascii_safe whether stdout can encode ❌ before printing. When it cannot, the plain-text [FAIL] goes out instead and the report prints.

Ask before you print. _ascii_safe(char, fallback) tries to encode the character with the current stdout encoding and returns the fallback if that fails. The answer is cached, because the encoding does not change during a run. The fix calls it once, _ascii_safe("❌", "[FAIL]"), and uses the result in all three failure messages: checksum mismatches, missing files (--fail-on-missing-files) and extra files (--fail-on-extra-files). I picked [FAIL] to match the existing [OK].

The same pipe now prints the report, and the exit code is still 1:

$ hf cache verify julien-c/dummy-unknown --cache-dir ./e2e_cache | cat
[FAIL] Checksum verification failed for the following file(s):
  - config.json: expected d9e9f15bc825e4b2c9249e9578f884bbcb5e3684 (git-sha1), got 8d7314a1cf98df9734b0ab8aa85ffda51006c369

I wrote a regression test before touching cache.py. Click's CliRunner(charset="cp1252") gives the command a cp1252 stdout on any OS, so the bug shows up without a Windows box. On main the test failed with the real UnicodeEncodeError. After the change, pytest tests/test_cli.py -k verify gave 4 passed, and the five CLI test files gave 476 passed and 5 skipped.

What this means if you run hf on Windows

If you send hf cache verify output anywhere other than a console on Windows (a file, a pipe, a CI log), you now get the list of bad files instead of Error: Invalid value. That is the case where the report matters most, because nobody is watching the terminal. A failed check still exits with 1, so scripts behave the same, and a UTF-8 stdout still shows ❌.

The fix landed on main after v2.3.0 was released, so v2.3.0 does not have it. Until it reaches a release, setting PYTHONUTF8=1 makes Python use UTF-8 for stdout and avoids the crash.

I kept the PR to cache verify on purpose. The other gaps from #4610 are untouched, and strings that come from the Hub, such as an emoji in a repo description printed in a table, still fail on a cp1252 stdout. That is the one I want to look at next, and the open question is whether to fall back per glyph or once, centrally.

Who got it over the line

CONTRIBUTING.md asks for an issue before code changes, so on October 25 I opened #5254 and then #5255 with Fixes #5254. The PR body led with the real Windows repro and the before and after output, because a maintainer had asked on another issue whether a bug had been seen on a real cache.

Less than four hours later, community reviewer Ayushdevo ran the cp1252 regression on Python 3.12 and checked that all three failure paths used the same marker and still exited with a failure. I had only run 3.11, so that was a check I could not give myself. Thank you, Ayushdevo.

Maintainer hanouticelina approved it with "Thanks for fixing!" and merged it on October 27, a little over two days after I opened it. Before merging, hanouticelina applied review suggestions that took my regression test out, so what landed is one file: 5 lines added and 4 removed in cli/cache.py. The test had already done its job by then, failing on main and passing after the fix. Thank you for the quick review.

Thanks also to Wauplin, who wrote #4610 and left that list of known gaps behind. It predicted that each remaining gap would be a one-line _ascii_safe() call, and mine was.

What I took away

Before I write a fix in someone else's codebase, I now read the last PR that touched the same problem. Here that was #4610. It gave me the helper, the fallback pattern (✓ became [OK], so ❌ became [FAIL]), and my exact bug on a list of known gaps. A vague "Windows encoding problem" became a small change that followed the maintainers' own plan, with a real repro at the top of the PR.

I'm Shreya Boyane, an ML engineer with an MS in AI from WPI and a background as a Lead DevOps Engineer, so a crash that only shows up in a pipe or a CI log felt like familiar ground. I'm open to ML engineering roles. You can find me on GitHub as Sonica-B, at shreyaboyane.dev, and on LinkedIn.