DEV Community: Shreya Boyane

huggingface_hub v2.3.0 week: how I fixed a Windows crash in hf cache verify

Shreya Boyane — Thu, 29 Oct 2026 01:16:21 +0000

huggingface_hub v2.3.0 came out on October 22, 2026, and its headline number is a good one: a recursive HfFileSystem.find() over allenai/objaverse, about 800k files, went from 776 s to 19.7 s. Lower in the notes is a smaller fix, #5198, for a UnicodeDecodeError under the C locale that broke checkpoint saving in TRL.

That same week I hit the mirror image of that bug. In #5198, text coming into the library failed to decode. In my case, text going out of the hf CLI failed to encode, and it took down hf cache verify on my Windows 11 machine.

The short version

On Windows, Python writes to a terminal in UTF-8 but to a pipe, a file or a CI log in the old ANSI code page, cp1252. hf cache verify began its failure report with a ❌, cp1252 has no ❌, and the command crashed at the exact moment it had bad news to report. The fix prints [FAIL] when the stream cannot encode the emoji. It was merged on October 27.

The concept, animated

Figure 1. The blocks are the real call chain in cli/cache.py and cli/_output.py. The token is the first character the failure report prints. The three states: a console, a pipe on Windows before the fix, and a pipe after it.

Python decides sys.stdout.encoding once, when the program starts, and on Windows it decides differently depending on where the output goes. Into a console: UTF-8, which can encode every character. Into a pipe, a redirected file or a CI step: the ANSI code page, which on my machine is cp1252, a 1990s table of about 250 characters with no ❌ in it. The engraved .encoding on the sys.stdout block in figure 1 flips between the two.

hf cache verify checks that the files in your local Hugging Face cache match the checksums on the Hub, and the first thing it printed when a file was bad was ❌ Checksum verification failed for the following file(s):. In a console, the token passes. In a pipe, sys.stdout cannot encode it: UnicodeEncodeError, position 0. The list of bad files, the one thing you asked for, never comes out, and the command exits with Error: Invalid value. That label is the CLI's name for any ValueError, and UnicodeEncodeError happens to be one, so an output bug looked like bad input.

One line reproduces it with no Hugging Face code at all: PYTHONIOENCODING=cp1252 python -c "print('❌')".

How I ran into it

I use the hf CLI every day on Windows 11, and that week I was exploring it from source (main, which already reported 2.4.0.dev0, on Python 3.11). I wanted the output of hf cache verify saved to a file, as a record of which cached files had gone bad. To get a bad file, I made one: I downloaded config.json from julien-c/dummy-unknown, edited the cached copy so its checksum would no longer match, and ran the check through a Git Bash pipe.

$ hf cache verify julien-c/dummy-unknown --cache-dir ./e2e_cache | cat
Error: Invalid value. 'charmap' codec can't encode character '❌' in position 0: character maps to <undefined>

charmap is how Python names the codec behind cp1252, the character was ❌ at position 0, and the pipe was the last clue. A terminal and a pipe get different encodings, so output that looks fine while you develop can crash in a log file.

Then I searched the repo and found _ascii_safe in cli/_output.py. It came from #4610, which swapped ✓ for [OK] when stdout cannot encode it. That PR also listed the places it had not fixed yet, and ❌ in cache verify was on the list. Before writing code, I checked that no open or closed issue or PR covered it.

The fix, in one idea

Figure 2. After #5255 the command asks _ascii_safe whether stdout can encode ❌ before printing. When it cannot, the plain-text [FAIL] goes out instead and the report prints.

Ask before you print. _ascii_safe(char, fallback) tries to encode the character with the current stdout encoding and returns the fallback if that fails. The answer is cached, because the encoding does not change during a run. The fix calls it once, _ascii_safe("❌", "[FAIL]"), and uses the result in all three failure messages: checksum mismatches, missing files (--fail-on-missing-files) and extra files (--fail-on-extra-files). I picked [FAIL] to match the existing [OK].

The same pipe now prints the report, and the exit code is still 1:

$ hf cache verify julien-c/dummy-unknown --cache-dir ./e2e_cache | cat
[FAIL] Checksum verification failed for the following file(s):
  - config.json: expected d9e9f15bc825e4b2c9249e9578f884bbcb5e3684 (git-sha1), got 8d7314a1cf98df9734b0ab8aa85ffda51006c369

I wrote a regression test before touching cache.py. Click's CliRunner(charset="cp1252") gives the command a cp1252 stdout on any OS, so the bug shows up without a Windows box. On main the test failed with the real UnicodeEncodeError. After the change, pytest tests/test_cli.py -k verify gave 4 passed, and the five CLI test files gave 476 passed and 5 skipped.

What this means if you run hf on Windows

If you send hf cache verify output anywhere other than a console on Windows (a file, a pipe, a CI log), you now get the list of bad files instead of Error: Invalid value. That is the case where the report matters most, because nobody is watching the terminal. A failed check still exits with 1, so scripts behave the same, and a UTF-8 stdout still shows ❌.

The fix landed on main after v2.3.0 was released, so v2.3.0 does not have it. Until it reaches a release, setting PYTHONUTF8=1 makes Python use UTF-8 for stdout and avoids the crash.

I kept the PR to cache verify on purpose. The other gaps from #4610 are untouched, and strings that come from the Hub, such as an emoji in a repo description printed in a table, still fail on a cp1252 stdout. That is the one I want to look at next, and the open question is whether to fall back per glyph or once, centrally.

Who got it over the line

CONTRIBUTING.md asks for an issue before code changes, so on October 25 I opened #5254 and then #5255 with Fixes #5254. The PR body led with the real Windows repro and the before and after output, because a maintainer had asked on another issue whether a bug had been seen on a real cache.

Less than four hours later, community reviewer Ayushdevo ran the cp1252 regression on Python 3.12 and checked that all three failure paths used the same marker and still exited with a failure. I had only run 3.11, so that was a check I could not give myself. Thank you, Ayushdevo.

Maintainer hanouticelina approved it with "Thanks for fixing!" and merged it on October 27, a little over two days after I opened it. Before merging, hanouticelina applied review suggestions that took my regression test out, so what landed is one file: 5 lines added and 4 removed in cli/cache.py. The test had already done its job by then, failing on main and passing after the fix. Thank you for the quick review.

Thanks also to Wauplin, who wrote #4610 and left that list of known gaps behind. It predicted that each remaining gap would be a one-line _ascii_safe() call, and mine was.

What I took away

Before I write a fix in someone else's codebase, I now read the last PR that touched the same problem. Here that was #4610. It gave me the helper, the fallback pattern (✓ became [OK], so ❌ became [FAIL]), and my exact bug on a list of known gaps. A vague "Windows encoding problem" became a small change that followed the maintainers' own plan, with a real repro at the top of the PR.

I'm Shreya Boyane, an ML engineer with an MS in AI from WPI and a background as a Lead DevOps Engineer, so a crash that only shows up in a pipe or a CI log felt like familiar ground. I'm open to ML engineering roles. You can find me on GitHub as Sonica-B, at shreyaboyane.dev, and on LinkedIn.

supervision 0.31 sees depth. ComparisonAnnotator now sees grayscale

Shreya Boyane — Thu, 29 Oct 2026 01:16:10 +0000

supervision 0.31.0 shipped on October 16, and its release notes open with "supervision now sees depth." A depth map is a single-channel image. So is a thermal frame, and so is an MRI slice like the ones in my TumorLytix project. The next two patch releases, 0.31.1 on October 20 and 0.31.2 on October 24, taught the mask, halo, crop and icon annotators to draw on grayscale scenes.

On October 25 I opened a pull request for one annotator that still crashed on those frames. It was merged on October 26. This post is the short version of that side quest, from the first ValueError to a test pattern I plan to reuse.

The short version

A colour is three numbers. A pixel in a grayscale image holds one number. ComparisonAnnotator tried to put the three numbers into every pixel under a mask, and NumPy refused. The rest of the library already had a rule for this case, keep the blue value, and a helper that applies it. The fix routes the annotator's three colours through that helper and adds eight tests that check grayscale output against the BGR output, channel for channel.

The concept, animated

Figure 1. The grids are the colour layer NumPy allocates with the scene's shape, the outline is the boolean mask the annotator writes under, and the chips are the three values of one colour in OpenCV's B, G, R order. The three states: a BGR scene, a grayscale scene before the fix, and after it.

ComparisonAnnotator highlights where two sets of detections on the same frame differ: one colour for regions only the first set covers, one for regions only the second covers, and one for the overlap. Its docstring suggests it for comparing two models, or a ground truth with a prediction.

Under the hood it builds an empty colour layer shaped like the scene and writes each colour into the pixels under its mask. On a BGR frame every pixel has three slots, and a colour's three values fit. On a one-channel frame every pixel has one slot. NumPy cannot place three values into it, and the first colour written is the overlap, so that was the line that failed. In my repro two 41 by 41 boxes overlapped in a 21 by 21 square, which is where the 441 in the message comes from.

The same thing happened for (H, W) arrays, (H, W, 1) arrays and a Pillow "L" image, with opencv-python installed and with the NumPy fallback that supervision uses when OpenCV is absent.

How I got there

I was exploring supervision's annotators with one question: would they draw on the grayscale frames that medical and thermal pipelines produce? I wrote a probe script that runs every concrete annotator on four kinds of scene: (H, W) arrays, (H, W, 1) arrays, BGRA images and float images.

On develop at commit b651cca, most of the grayscale work had already landed. MaskAnnotator, HaloAnnotator, PolygonAnnotator, ColorAnnotator, CropAnnotator and IconAnnotator all handled single-channel scenes. ComparisonAnnotator raised a ValueError on all three single-channel inputs.

Before writing any code, I checked that nobody else was on it. A search for "ComparisonAnnotator" found only merged PRs, none about grayscale, and no open PR or issue. Mohammad Hijjawi, who wrote the grayscale fixes in #2755 and #2788, had listed the annotators that still failed and left them for separate PRs: BackgroundOverlayAnnotator, HeatMapAnnotator and RichLabelAnnotator. ComparisonAnnotator was on neither list, so I took it and left those three alone.

The fix, in one idea

Figure 2. After #2803 every colour passes through _color_for_canvas. On a three-channel scene the tuple goes through unchanged. On a single-channel scene only the blue value reaches the pixel.

The repo already had the answer in a helper. _color_for_canvas, added in #2755, keeps only the first value of the BGR tuple when the canvas is grayscale. That value is the blue component, and it is what cv2.rectangle and cv2.fillPoly write on single-channel images. The fix routes all three of the annotator's colours through it. On a three-channel scene the helper returns the tuple unchanged, so BGR frames take exactly the path they took before. The change to core.py is 8 lines added and 4 removed, plus a docstring update and a changelog line.

Why blue instead of a luminance mix, which would look more natural? Because the rest of the repo already uses blue, so a frame annotated with BoxAnnotator, MaskAnnotator and ComparisonAnnotator gets one gray value per colour. The convention has a cost, and I put it in the PR description: the default red and green both have a blue component of 0, so on a grayscale frame both "only" regions come out the same gray. If you need to tell them apart, pass colours with different blue components. Changing the convention itself would be a decision for the whole library.

Testing layouts against each other

Hard-coded pixel values would tie the test to today's colours and blend math. So the new test class annotates a BGR scene and its grayscale copy with the same detections, then asserts that the grayscale result equals the first channel of the BGR result. That one assertion runs for box, mask and oriented-box detections on (H, W) and (H, W, 1) scenes, which makes six cases. A seventh checks the legend labels the same way. An eighth checks that a Pillow "L" image comes back as an "L" image with the expected pixels.

All 8 fail on develop and pass with the change, on the NumPy fallback and on opencv-python 5.0.0. I ran both backends because OpenCV is optional now and CI tests both. On the fallback, the full suite went from 5,931 passed to 5,939 passed. One unrelated test timed out on my laptop, on both branches.

What the fix changed for users

If your frames come from an MRI scanner, a thermal camera or a depth sensor, you can now compare two sets of detections on them without converting to BGR first. ComparisonAnnotator accepts (H, W) arrays, (H, W, 1) arrays and Pillow "L" images, and it hands back the same layout it received. Its gray values match the boxes and masks that other annotators draw on the same frame. BGR users see no change. The fix is on develop, with a changelog entry under Unreleased.

BGRA frames are still an open question. They need a maintainer decision: strip the alpha channel and reattach it in the input decorator, or reject BGRA outright. I plan to raise it as an issue.

Thank you, Borda and the supervision maintainers

Borda (Jirka Borovec) approved and merged #2803 on October 26, the day after I opened it. Thank you. For a first-time contributor the path was clear: a CLA bot, a maintainer approving the first CI runs, and an AGENTS.md that asks for a changelog line linking the PR number. That last rule meant one small follow-up commit once #2803 existed.

Thanks also to the maintainers who shipped six supervision releases between October 4 and October 24, and to Mohammad Hijjawi, whose #2755 gave me the helper and whose list of remaining failures told me which fixes were already spoken for.

What I took away

When a library accepts several image layouts, test the layouts against each other. The grayscale result should equal one channel of the BGR result, and that single assertion stays true for any colour, opacity or geometry. A hard-coded pixel test covers one case and breaks the day a default colour changes.

The parity test also makes review cheap. A reviewer can read one assertion and know what all eight cases prove, which matters when a maintainer is reading dozens of outside PRs a month. It is the first test I will write the next time I touch an image pipeline, including my own.

I'm open to ML engineering roles, computer vision included. You can find my work at shreyaboyane.dev, on GitHub as Sonica-B, and on LinkedIn.

One extra field turned an A2A error into a TypeError. Fixed in a2a-sdk 1.3.0

Shreya Boyane — Thu, 29 Oct 2026 01:15:53 +0000

On October 27, 2026, SAP announced general availability of inbound Agent2Agent (A2A) interoperability, so third-party agents can now invoke SAP Agents through its Agent Gateway. Every launch like this means more A2A clients calling agents their authors do not control. Two days before SAP's announcement, I had opened a pull request for that exact situation in the A2A Python SDK. An agent could send a valid A2A error, and the calling code would receive a TypeError. My fix shipped the same day as SAP's news, in a2a-sdk 1.3.0.

A2A is an open protocol that lets AI agents discover each other and exchange messages and tasks over JSON-RPC, REST or gRPC. It joined the Agentic AI Foundation in August 2026, and a2a-sdk is its official Python SDK.

The short version

When an agent sent back an error, the SDK handed the reply to a third-party library before looking at it itself. That library is strict about which fields an error may carry, so one extra field, the kind the A2A spec says to ignore, made it throw a raw TypeError. The SDK's own translation of error codes into named exceptions never got a turn. Version 1.3.0 reads the reply itself first, at one place, and the library only builds requests now.

The concept, animated

Figure 1. The blocks are the real call chain in client/transports/jsonrpc.py, in order. The token is the reply body. The three states: an error object with one extra field before 1.3.0, the same reply in 1.3.0, and a body that is not an object in 1.3.0.

A client talks to agents it does not control, so odd replies are normal. The SDK's promise about them is clear: a failure reaches your code either as A2AClientError or as a typed A2A error such as TaskNotFoundError. The A2A spec (section 5.7) adds that implementations should ignore fields they do not recognise.

Now follow the token in the first state. It carries the error code -32001, which means TaskNotFoundError, plus one extra field, reason, that the spec says to ignore. The first block it reaches is JSONRPC20Response(**body) from the json-rpc package. It validates the envelope before the SDK sees anything, and it builds the error with a constructor that accepts only code, message and data. Any other member is, to that library, a programming mistake. So the token drops into the tray labelled raw TypeError: JSONRPCError.__init__() got an unexpected keyword argument 'reason'. The SDK's _create_jsonrpc_error() block never receives it, and your except TaskNotFoundError: branch never runs.

The library was correct by its own rules. It simply ran first, so its rules won and the SDK's contract lost.

How I found seven of these

I was exploring the client with one question: what reaches my code when an agent answers wrong? No issue existed, so I played the misbehaving agent myself, wrote seven bad response bodies by hand, and fed them to the client through a mock transport. All seven came back as raw Python exceptions.

What the agent sent	Before	After
a list, or a string (say, from a proxy)	`TypeError`	`A2AClientError`
neither `result` nor `error`	`ValueError`	`A2AClientError`
`"error": "boom"`	`TypeError`	`A2AClientError`
error object with an extra member	`TypeError`	`TaskNotFoundError`
error object without `message`	`ValueError`	`TaskNotFoundError`
SSE `data:` that is not JSON	`JSONDecodeError`	`A2AClientError`
`event: error` with an extra member	`TypeError`	`TaskNotFoundError`

Following one reply through the code explained the whole table. Nine request methods handed the body straight to the json-rpc validator, and streaming used the same validator on each event. The SDK's own mapping from error codes to exceptions even had a fallback for a missing message, but it could never execute, because the library failed first. Streaming had a second twist: the JSON parse for each event ran in a place where its exception escaped the handler that was supposed to catch it, so a broken data: line leaked a JSONDecodeError.

The fix, in one idea

Figure 2. In 1.3.0 the json-rpc validator is out of the chain. The SDK reads the envelope itself at one choke point, so an error object becomes the typed A2A error whatever extra fields it carries, and a body that is not even an object becomes A2AClientError.

Check the wire envelope yourself, once, and hand libraries only what they can handle. Every reply already passes through one sending function or the streaming loop, so I put one check there. It asks three plain questions in order: is the body an object; if it has an error, is that an object, and if so, map it to the A2A exception by its code; otherwise, is there a result. Anything that fails those questions becomes A2AClientError with a message that says what was wrong. The streaming loop turns a JSONDecodeError into A2AClientError the same way. A falsy error next to a result still counts as success, as before. The client no longer builds JSONRPC20Response objects at all; requests still use JSONRPC20Request.

Wrapping each of the nine calls in its own try/except would have turned TaskNotFoundError into a generic error and copied the same code nine times. One choke point keeps the typed exceptions.

The tests live in a new class, TestMalformedJsonRpcResponses, with 11 cases. Ten fail on main with the exceptions in the table and pass with the change. The eleventh guards that delete_task_push_notification_config still raises on an error response. The existing tests pass unchanged. I ran everything on Windows 11 with Python 3.10.19: lint.sh passed, the transport tests gave 51 passed, and the wider run gave 2213 passed at 93% total coverage. Three failures and four setup errors fail the same way on main, and two integration suites do not run on Windows, so I left those to Linux CI. All 17 checks passed there, and the coverage bot showed client/transports/jsonrpc.py going from 92.66% to 95.51%.

What changed for people using a2a-sdk

If your client calls agents you do not control, version 1.3.0 behaves differently in three ways:

An agent that adds its own field to an error object, or leaves out message, now raises the mapped A2A error, so your except TaskNotFoundError: branch runs.
A malformed body, such as a list, a string, a missing result or a non-object error, raises A2AClientError, so one except clause covers all of them.
A stream with a broken data: line raises A2AClientError where it used to leak a JSONDecodeError.

One behaviour change is worth knowing. A body with "error": null and no result used to parse as an empty object and now raises A2AClientError. That body breaks JSON-RPC 2.0 anyway. Upgrading is pip install -U a2a-sdk.

The v0.3 compatibility transport still uses the same validator-first pattern on main. I kept this PR to the 1.0 client and offered a fix(compat) follow-up in the PR body.

A thank-you to Iwaniukooo11 and the A2A maintainers

I opened PR #1375 on October 25 with no linked issue, just the table and the tests. On October 26, Mateusz Iwaniuk (@Iwaniukooo11) brought the branch up to date with main, approved it and merged it. Thank you. Getting a first PR to a repo reviewed and merged the day after it opens is a generous welcome.

Thanks also to the A2A maintainers for keeping the door open to outside contributors. Of the 40 merges before mine, 29 came from non-members, with a median of 3.4 days from open to merge. Release 1.3.0 carries fixes from at least three outside contributors, mine among them.

What I took away

Where a third-party validator sits in your call chain decides which exceptions your callers see. The stack trace ended inside json-rpc, but the decision that mattered was where the SDK called it.

When you own an error contract, check the wire envelope yourself at one choke point and hand libraries only what they can handle. As more vendors open their agents to outside callers, the exception a client can catch by name is part of the interface, and that boundary is the kind of problem I like to work on.

I'm Shreya Boyane, an ML engineer who builds agent systems, and I'm open to ML and AI engineering roles. You can find my work on GitHub, more about me at shreyaboyane.dev, and I'm happy to connect on LinkedIn.

Isaac Lab 3.0 is GA. One missing close() locked log files on Windows

Shreya Boyane — Thu, 29 Oct 2026 01:15:37 +0000

GTC Berlin, October 21, 2026. NVIDIA announced that Isaac Lab 3.0 is generally available, describing it as a framework for training robot policies "from desktop to data center." Three days later I was at the desktop end of that phrase. My desktop was a Windows 11 laptop, and two of Isaac Lab's logging tests were failing on it with an error that had nothing to do with robots.

Isaac Lab is NVIDIA's open-source framework for training robot policies in GPU-accelerated simulation. Version 3.0 supports the Newton and PhysX physics engines, and the v3.0.0 release notes make Isaac Sim optional for many workflows. This post is about a side quest on that laptop and the one-line fix it produced, now merged as PR #8809.

The short version

A logging handler is the thing that keeps a log file open and writes lines into it. When Isaac Lab reconfigured its logging, it unhooked the old handlers from the logger but never closed them, so the old log file stayed open in the background. Linux lets you delete a file that is still open, so nobody noticed. Windows refuses, so two tests failed on my machine. The fix is one line: close what you unhook.

The concept, animated

Figure 1. Every part carries its real name. The board is the root logger's handlers list. The wire from a FileHandler to its .log file is the handler's .stream, and the lamp reads "stream open" for as long as that wire exists. The three states are the three calls, in order: configure_logging(file=...), removeHandler(old), and the fix, handler.close().

Watch the second state. removeHandler() does exactly what its name says: the old handler leaves the list, so the logger will never call it again. Nothing about that call touches the wire. The old FileHandler still owns an open stream to isaaclab_1.log, and that is what Windows checks when os.remove() arrives: something holds the file, so the delete fails with PermissionError: [WinError 32]. Linux lets a delete go through on an open file, which is why the leak was invisible everywhere except Windows.

Figure 2. The third state: handler.close() ends the stream, the lamp goes out, and the same os.remove() succeeds. The new handler and its file are untouched.

How I got there

I came to Isaac Lab for a different bug. Issue #8534 reported that a contrib humanoid task passed wxyz quaternions into xyzw APIs, so the humanoid trained upside down. I wanted that one. Another contributor had already claimed it and opened a PR.

So I took the fallback on my list. Isaac Lab's docs list Windows 11 as a supported platform, but when I checked in mid-October, no CI job ran its tests on Windows and the windows_ci test marker was applied to zero test files. I work on Windows, so that gap was mine to poke at.

My setup was plain: Windows 11, Python 3.12.12 and CPU-only PyTorch 2.11.0, the same Python and PyTorch versions 3.0 targets, minus CUDA. I did not install Isaac Sim, so the real work was sorting failures by cause.

I ran every unit test in four test folders on main and got 913 passed, 72 failed, 125 skipped and 5 collection errors. Most of the failures were my environment talking: a missing omni module (that is Isaac Sim), a CPU-only torch build, no torchvision. Others were Windows symlink permission errors, because Developer Mode was off, and a few tests with POSIX assumptions about Isaac Sim build trees. I set those aside, since judging them fairly needs Developer Mode or a simulator build.

Two failures in source/isaaclab/test/utils/test_logger.py did not fit any of those buckets.

Why only Windows complained

Both tests failed with PermissionError: [WinError 32]. One failed while its temporary directory was being cleaned up, because the log file inside could not be removed. The other failed on its own os.remove call. These tests are plain Python and run without a simulator or a GPU, so my setup could not explain them.

WinError 32 is Windows saying that some process still has the file open. The only thing that opens that log file is the file handler installed by configure_logging, so I read what that function does with the handlers it replaces. Then I wrote a short probe: configure logging with a file, reconfigure without one, and look at the old handler. Its stream was still an open file. The log file had outlived the configuration that created it.

That is the whole mechanism in figure 1. removeHandler() detaches a handler from a logger and leaves it open. A detached file handler keeps its file until someone calls close(), and nobody did. On Linux the leak is invisible, because an open file can still be deleted. On Windows it is a locked file.

The fix, in one idea

Close what you remove. The loop that unhooks each old handler now also closes it:

for handler in root_logger.handlers[:]:
    root_logger.removeHandler(handler)
    handler.close()

That is the entire code change. Python's own logging.basicConfig(force=True) already removes and closes existing handlers the same way, so the fix brings Isaac Lab in line with the standard library, and the docstring now says so.

The tests changed too. Both now reconfigure logging before cleanup, which releases the file, and one of them asserts that the replaced handler's stream is gone. That assertion checks the handle, not the Windows error, so a Linux CI run can catch the regression too. I only ran it on Windows, where it failed against the old logger.py.

What changed for Isaac Lab users

Any process that calls configure_logging more than once, such as a test session, now releases the previous log file right away instead of holding it until exit. On Windows, those files can be deleted again.

Against main at commit 4a281f3, with the PR's tests, test_logger.py had 2 failures and 12 passes. With the fix, all 14 pass. Across the source/isaaclab/test/utils unit sweep, the same 36 environment failures appeared with and without the change, so the fix moved nothing else.

I flagged one trade-off in the PR. The function now closes every handler it removes, including ones it did not create, such as Kit log bridges or pytest's --log-file handler. The old code had already detached them. Closing a plain stream handler leaves its underlying stream open, so only file-backed handlers behave differently. A maintainer could reasonably prefer closing only the handlers the function made, and I wanted that choice in front of them.

How a first PR got merged in 28 hours

The contributing guide asks for an issue first, so I filed #8808 and then opened #8809 with "Fixes #8808". The repo wanted a changelog fragment instead of a CHANGELOG.rst edit, plus a line in CONTRIBUTORS.md. Isaac Lab's main moves fast, with 587 PRs merged in the 30 days before I started, so I rebased from 4a281f3 onto 259e1a8 before pushing. Both files I touched were identical at the two commits, so my Windows baseline still held.

Three minutes after I opened the PR, the repo's review bot asked for two changes. First, make the cleanup in the file-handler test exception-safe: if an assertion failed halfway, the file would stay open and the cleanup would raise its own PermissionError, hiding the real failure. I moved the reconfiguring call into a finally block. Second, name the tested baseline revision and give explicit failure counts. I rewrote the Testing section with the numbers above.

Then the maintainers took over. AntoineRichard commented run-ci to start the Docker and GPU test jobs, approved the PR, and merged it on October 26, about 28 hours after I opened it. maxkra15 gave the second approval.

Thank you

Thank you to AntoineRichard for the review, the CI run and the merge, and to maxkra15 for the second review. Thanks also to the Isaac Lab maintainers for a contribution path I could follow end to end from the docs, with an issue template, a PR checklist, changelog fragments, and a review bot that caught my gaps before a human had to.

What I took away

Windows is where this bug showed up, but the open file exists on every OS. So the regression test checks the handle instead of the Windows error, and a failure that only Windows could see became a check any CI can run. When a bug appears on one platform, I look for the platform-independent cause and pin the test to that.

The second lesson is about triage. Two of the 72 failures turned into a fix. Getting there meant sorting the other 70 by cause first and writing down why I left each group alone.

Code that removes something holding a file should close it too, because Linux hides the leak that Windows reports. Isaac Lab 3.0 promises desktop to data center, and running its tests on a platform its CI did not cover was a cheap way to find a real bug.

I'm open to roles in robot learning, ML systems and MLOps. You can find my work at shreyaboyane.dev, my code on GitHub as Sonica-B, and me on LinkedIn.

TorchTitan's October update is out. I fixed 2 bugs in its quick-start script

Shreya Boyane — Thu, 29 Oct 2026 01:15:20 +0000

TorchTitan Update, October 21, 2026. During PyTorch Conference North America in San Jose, the PyTorch team published TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL. It walks through new support for DeepSeek-V4, Kimi K3 and Qwen3.5, introduces TitanRL for reinforcement learning, and reports DeepSeek-V3 671B training at 2,694,493 tokens per second on 512 GB300 GPUs. The post ends by telling you to clone the repo and follow the quick start.

Step 3 of that quick start downloads a tokenizer with scripts/download_hf_assets.py. I found two bugs in that script, and a test that passed the whole time had been hiding one of them. I opened PR #5621 on October 25, and it merged on October 27.

The update covers the scale end of torchtitan. My side quest happened at the other end, in the script a new user runs before training a real model.

The short version

The download function was handed a list of what to fetch and wrote into that list instead of its own copy, so the caller's list came back one entry longer. Call it twice with the same list and the second call rejects the entry the first call added. The same function also raised an error built from three separate strings, so its message printed as a tuple with literal \n in it, and the test that checked the message could not tell. The fix copies the list and builds one message.

The concept, animated

Figure 1. Trays are list objects and chips are their entries; the wire shows which object the parameter name asset_types refers to. The three states: the first call before the fix, the second call with the same list, and a call after the fix.

download_hf_assets.py is the torchtitan script that downloads a model's tokenizer, config, safetensors index or weights from the Hugging Face Hub before training. You tell it which assets you want as a list, and when you also pass extra file patterns it adds a placeholder entry, additional_patterns, to that list so the download loop picks them up.

The catch is in the word "that". If you passed a string, the function wrapped it in a fresh list and the placeholder went into the fresh list. If you passed a list, the placeholder went into your list. Your list is now ['tokenizer', 'additional_patterns']. The first call still works, because the table of known assets is rebuilt every call and the placeholder is matched during that same call. The second call, with the same list, meets an entry nobody asked for, finds no pattern for it, and stops: Unknown asset type additional_patterns.

Anyone looping over several repos with one shared list of asset types hit this on the second repo.

The second bug, and why a passing test hid it

I found the function through a typo. I typed --assets tokenizers, with a stray "s", and instead of a readable message I got this:

ValueError: ('Unknown asset type tokenizers. Available uses: --assets tokenizer safetensors index config \n', "Or specify exact patterns to download. Example: --additional_patterns '*.safetensors' README.md '*.json' \n", 'Or use --all to download all files')

That is a tuple, with literal \n escapes inside it. The error was raised with three comma-separated strings, and when an exception has more than one argument Python prints the whole tuple as its message.

The existing test for this case used assertIn on a piece of the message. A substring of a tuple's printed form is still a substring, so the test stayed green for as long as the bug existed. A short repro with the Hub calls mocked confirmed both bugs: it printed args: 3, and after one call my list read ['tokenizer', 'additional_patterns']. Before writing any code, I searched torchtitan's issues and PRs for the function and both messages. Neither bug had been reported.

The fix, in one idea

Figure 2. After #5621 the function copies the list first. The copy gets the placeholder entry and the download loop reads the copy; the caller's list is exactly as it was passed in.

Work on a copy. The function now copies the list it was given before adding anything to it, so the caller's list is untouched: list(asset_types) instead of using the caller's object. And the error is now one message built from one string, so it prints as three readable lines. Two edits in one file, 35 lines added and 8 removed across the script and its tests.

The tests changed with it. The existing one now asserts that the exception has exactly one argument before it checks the text, which is what would have caught the tuple. A new one calls the function twice with one list and additional_patterns=["*.txt"], then checks that the list is still ["tokenizer"] and that both calls downloaded the same files.

What changed for torchtitan users

A mistyped asset type now prints three readable lines: Unknown asset type tokenizers. Available uses: --assets tokenizer safetensors index config, then the --additional_patterns example, then the --all option. Scripts that loop over several repos with one shared asset_types list now work past the first repo. Which files get downloaded does not change.

The one behaviour change is for a caller that relied on the mutation. I searched the repo for callers of the function and found none that did. test_tokenizer.py passes a string, which still gets wrapped in a list.

I ran everything on Windows 11 with Python 3.11 and torch 2.16.0.dev20261024+cpu. pre-commit passed flake8, ufmt, pydoclint, codespell and Pyrefly on both files. The lychee link checker needs /bin/bash, so it could not start on Windows. I did not run anything on Linux or GPU, and the PR description says so.

Thank you, tianyu-l

Thank you to @tianyu-l for approving PR #5621 and merging it on October 27, two days after I opened it. It was my first PR to the repo, so CI also needed a maintainer to approve the workflow runs before it could start.

Thanks as well to the PyTorch distributed team at Meta who maintain torchtitan. The repo merged 392 PRs in the 30 days before I opened mine, and outside fixes still got a first review within 0 to 2 days.

I modeled my PR description on #5040, an earlier outside fix to the same function by u7k4rs6. Because #5040 had just touched that function, I picked the smallest change over a rewrite.

What I took away

I don't count a test as protection until I have watched it fail on the bug.

The old test stayed green for as long as the tuple message existed. So after writing the new assertions, I reverted only the script fix and kept the tests: 2 failed and 11 passed. With the fix back, all 13 passed, along with the 40 tests in tests/unit_tests/cpu/scripts.

That revert is the cheapest check I know, and it changes what I can put in a PR. "Tests pass" becomes "these 2 tests fail on main and pass with the change," which a reviewer can verify without taking my word for it. Listing what I did not run does the same job for the parts I could not test.

I'm an ML engineer with a DevOps background, and I'm open to ML engineering roles. My code is on GitHub as Sonica-B, my projects are at shreyaboyane.dev, and I'm on LinkedIn.

GTC Berlin side quest: one import hid 7 NVIDIA agent tools, so I made it lazy

Shreya Boyane — Wed, 28 Oct 2026 08:46:17 +0000

At GTC Berlin on October 21, NVIDIA expanded its Agent Toolkit: a new Agent Knowledge Foundation for grounding agents in company data, the new Kumo models, the OpenShell secure runtime, and tools like NeMo Relay. Two days later, NeMo Relay turned up somewhere else. The README of NVIDIA's open-source agent library, NeMo Agent Toolkit, gained a deprecation notice that points users to NeMo Relay, NeMo Helix and NeMo Fabric, with end of life set for the end of 2026.

That made me curious about the piece people still install today. So I built a fresh environment and looked at what a user actually gets. Seven of the library's 12 built-in tools never registered, and the error message pointed somewhere else entirely.

The side quest became PR #2474: +10/-3 lines in one source file plus a regression test, approved by an NVIDIA maintainer and merged about 19 hours after I opened it.

The short version

The toolkit registers its built-in tools by importing a list of modules, one after another. One of those modules asked for a LangChain class the moment it was imported. LangChain is not part of the core install, so on a plain install that import failed, the list stopped there, and the seven tools after it were never registered. The only trace was one warning in the log. The fix moves that import to the one method that actually uses it.

The concept, animated

Figure 1. The rack is nat/tool/register.py, one card per built-in tool, named as in the registry. The pointer is the import pass the nat_tools entry point runs top to bottom; a lit lamp is a registered tool. The socket on nvidia_rag.py marks where it needs langchain_core: at import time before the fix, inside to_document() after. The three states are a dev install, a plain install before #2474, and a plain install after it.

Plugins reach nat, the toolkit's package, through Python entry points. The nat_tools entry point loads one file, nat/tool/register.py, which is a list of imports, and every import registers the tool it names. That list is the rack in figure 1, and the import pass is the pointer.

Line 21 of nvidia_rag.py imported LangChain's Document class at module level, which means it needed langchain-core just to be imported, before anyone used the tool. That is the socket at the top of the nvidia_rag card, and the block at the front is the package it asks for.

On every contributor's machine that block reads "installed", because the test extras pull in langchain-core through langchain-community. On a plain install, a virtual environment holding only nvidia-nat-core and nvidia-nat-atif and their declared dependencies, it reads "not installed". The import raises ModuleNotFoundError, the loader catches it for that entry point, writes one warning, and carries on with nothing. Five lamps lit, seven dark: nvidia_rag, nat_retriever, current_request_attributes, code_execution, add_memory, delete_memory and get_memory.

What a fresh install actually gives you

I wrote a minimal workflow whose only setting was _type: code_execution, one of the built-in tools, and ran nat validate on it. It failed with this:

Input tag 'code_execution' found using discriminator() does not match any of the expected tags

So the validator claimed a built-in tool did not exist, and said nothing about imports or missing packages. If the registry did not know code_execution, I wanted to see what it did know, so I listed every registered function after plugin discovery. Five of twelve. The log held the only real clue, a warning that is easy to scroll past: Failed to import plugin 'nat_tools', followed by ModuleNotFoundError: No module named 'langchain_core'.

The same masking had hidden an earlier bug, #2386, where packaging was missing on a plain install (#2392 fixed it). The import was also in the released 1.9.0.

The fix, in one idea

Figure 2. After the fix the socket sits inside to_document(): the package is needed when the RAG tool runs, not when its module is imported. A plain install registers all twelve tools with langchain_core still absent.

Move the import to the moment it is needed. Document is only used inside one method, the one that turns a RAG result into a LangChain document, so the import now lives inside that method:

def to_document(self) -> "Document":
    from langchain_core.documents import Document
    return Document(page_content=self.content, ...)

The type hints stay under TYPE_CHECKING, so editors and type checkers see the same types as before. The tool's build function already imported langchain_core.prompts this way. Using the nvidia_rag tool still needs langchain-core, exactly as before. Importing the module no longer does.

I weighed two other fixes. Declaring langchain-core a core dependency would pull LangChain into the core package for everyone. Wrapping each import in register.py in try/except would hide real errors.

The regression test plays the plain install on any machine: it blocks every langchain_core entry in sys.modules, imports nat.tool.nvidia_rag fresh against a mocked registry, and checks that the tool registers. On the old code it fails with the same ModuleNotFoundError. A second test checks that both RAG response shapes, results and chunks, still produce the same LangChain Document objects. Before the fix, 1 test failed and 2 passed. After it, all 3 pass. Across the core suite, 2,981 tests passed, 310 were skipped and 8 failed; none of the 8 import nat.tool, and the five I reran on develop fail there the same way. I ran everything on Windows 11 with Python 3.11.9, not on Linux.

What changed for users

Built-in tool on a plain install	Before	After
`chat_completion`, `current_datetime`, `current_timezone`, `milvus_document_search`, `github_files_tool`	registered	registered
`nvidia_rag`, `nat_retriever`, `current_request_attributes`, `code_execution`, `add_memory`, `delete_memory`, `get_memory`	missing	registered

nat validate on the code_execution workflow went from exit code 1 and the misleading tag error to Configuration file is valid!
nat info components -t function printed the Failed to import plugin 'nat_tools' warning twice before the fix and zero times after.
A workflow on a plain install can now configure code execution, the retriever, request attributes and the three memory tools without LangChain installed.

Why fix a library that is being deprecated?

The deprecation notice gives users until the end of 2026 to migrate. Until then, anyone who installs only the core packages, in CI, in a container or on a new laptop, gets the environment I tested. The maintainers merged this fix three days after the deprecation notice landed.

Thank you, willkill07

Thank you to willkill07 (Will Killian) at NVIDIA. He started CI on NVIDIA's runners with /ok to test, approved the PR about 12 hours after I opened it, and sent it to merge with /merge about seven hours after that.

Thanks as well to the NeMo Agent Toolkit maintainers, who keep this repo open to outside contributors. When I checked on October 26, 39 of the last 40 merged PRs came from people outside the NVIDIA GitHub organization, with a median of 13.9 hours from open to merge.

The PR template asks every PR to close an issue, so I filed #2473 first. DCO and branch checks passed, CodeRabbit left one minor note on test cleanup, and a maintainer added the bug and non-breaking labels.

What I took away

Your dev environment is a superset of every user's environment, which makes it the one place a missing dependency can never show up.

A plugin system that imports a list of modules stops at the first missing optional dependency. Here that cost seven tools and produced one log warning. Unit tests could not catch it, because they run where every extra is installed.

Before my MS in AI at WPI, I was a Lead DevOps Engineer, and this bug sits between packaging and code, where neither side's tests look. I now build the exact install a user gets and import every plugin before I trust a green test run. I want to propose a CI job that does the same for this repo. The bug I'm looking at now is the same class: on a plain install, nat serve fails with No module named 'sqlalchemy'.

I'm open to ML engineering and agent infrastructure roles. If your team builds AI agents or the tooling under them, I'd be glad to talk. You can find my work at shreyaboyane.dev, my code on GitHub as Sonica-B, and me on LinkedIn.