Python can't weakref a ValueError. Sentry's deduplication found out the expensive way.

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Two and a half minutes: the leak, the two fixes that do not work, and what Sentry's Seer made of it. Everything below is the written version, with the parts the video did not have room for.

Project Overview

sentry-python is Sentry's Python SDK. One of its default integrations is DedupeIntegration, which stops the same error being reported twice. The mechanism is about as simple as it gets: remember the last exception you saw, and if the next one is the same object, drop the event.

Remembering an exception is where it gets interesting, because remembering it is exactly what you must not do. An exception holds its traceback, a traceback holds its frames, and a frame holds every local variable in it. Keep the exception and you keep all of that.

The SDK knew this. That is why it stored a weak reference:

# we can only weakref non builtin types
try:
    integration._last_seen.set(weakref.ref(exc))
except TypeError:
    integration._last_seen.set(exc)

The comment tells you the whole story. You cannot take a weak reference to a builtin exception. ValueError, KeyError, TypeError, the ones almost every error actually is. So weakref.ref(exc) raises TypeError, and the fallback quietly does the one thing the weak reference existed to prevent.

Issue #6094 reported it as unexplained memory growth in an asyncio web crawler.

Bug Fix or Performance Improvement

Why asyncio makes it hurt

_last_seen is a ContextVar. Under asyncio, every task gets its own copy of context. So this is not one retained exception process-wide. It is one retained exception per live task, each one dragging along its traceback and every frame local reachable from it.

The reporter's frames held fetched response bodies between 500 KB and 1 MB.

Measuring it honestly

My first instinct was to call it an unbounded leak. I built a worker pool, ran it, and it stayed flat. That was worth finding out before I wrote it down anywhere.

A fixed pool does not grow, because each worker's next error overwrites its previous one. Retention is bounded at (live tasks x payload). The growth story is not errors over time, it is tasks over time: one task per session, per subscription, per connection.

So I measured against live task count instead. Each task fails once, reports it, then stays alive the way a real session handler does:

  live tasks |     retained |   per task |  sessions pinned
-------------+--------------+------------+-----------------
          25 |      26.2 MB |    1047 KB |      25 / 25
          50 |      52.1 MB |    1042 KB |      50 / 50
         100 |     103.3 MB |    1033 KB |     100 / 100
         200 |     205.4 MB |    1027 KB |     200 / 200

1024 KB retained per live task, dead straight, and sessions pinned counts weak references to session objects that are still reachable after gc.collect(). Not a sampling artifact. The garbage collector cannot touch them because a live ContextVar genuinely still points at them.

At 200 concurrent sessions that is 205 MB that never comes back.

The fix I got wrong first

The maintainers had already rejected the obvious fix. The reporter proposed skipping dedupe for builtins, and Sentry declined, saying they wanted "a more robust fingerprinting approach that can also be used for built-in exceptions" instead.

So I proposed a fingerprint: exception type, message, and the origin frame from the traceback. All immutable primitives, nothing retained. I posted it on the issue.

Then the existing test suite told me I was wrong, in two different ways.

test_breadcrumbs calls capture_exception(ValueError()) twice with two distinct, never-raised exceptions. No traceback, no message, so both fingerprints are identical and the second event gets dropped.

test_option_before_breadcrumb was worse. It calls the same function three times, each raising a separate ValueError("aha!") from the same line. Same type, same message, same origin frame. Three fingerprints, all identical, two events wrongly deduplicated.

That is the flaw in the whole idea. A value fingerprint cannot tell "the same exception object twice" apart from "the same error raised repeatedly from the same line." The first must deduplicate. The second must not. No fingerprint distinguishes them, because by value they are the same.

I went back to the issue and said so.

The attempt that came before mine

Late on, I went looking through the fork's branch list and found two abandoned maintainer branches from September 2025: antonpirker/dedupe-integration-memory-usage and antonpirker/make-dedupe-integration-more-memory-efficient. Neither was merged.

The first one fingerprints on (type_module, type_name, id(exc_value)).

That id() is the interesting part. Once you stop retaining the exception, which is the entire point of the change, its address becomes immediately reusable, and CPython reuses addresses aggressively. Modelling that fingerprint over 2000 distinct, sequentially allocated ValueErrors:

distinct exceptions created : 2000
distinct fingerprints       : 2
address-reuse collisions    : 1998

Two fingerprints for two thousand distinct errors. Each collision is a real error that would be silently dropped as a duplicate.

I mention this not to dunk on an abandoned branch, which is a draft nobody shipped, but because it is the third distinct way I watched this problem punish an obvious solution. Fingerprinting on value collapses errors that came from the same line. Fingerprinting on address collapses errors that reused the same memory. The information being fingerprinted just is not sufficient to express object identity.

The fix that works

The problem was never that identity was the wrong key. It was that holding the object was the wrong way to hold identity.

So keep the identity and drop the object. Attach a small weak-referenceable token to the exception, then weakly reference the token:

class _DedupeToken:
    __slots__ = ("__weakref__",)

There is exactly one token per exception object, and it lives precisely as long as the exception does. last_seen is token is therefore equivalent to the old last_seen is exc, while the ContextVar holds nothing that keeps a traceback alive.

Deduplication behaviour is unchanged, which is the part that matters given the maintainers' concern.

Code

The whole change to the hot path:

exc = exc_info[1]

new_last_seen: "Any"
try:
    # We can only weakref non builtin types.
    new_last_seen = weakref.ref(exc)
    is_duplicate = last_seen is exc
except TypeError:
    # Builtin exception. Referencing it strongly here would pin its
    # traceback and every frame local in that traceback for the
    # lifetime of the ContextVar (#6094). Weakly reference an
    # identity token carried by the exception instead, which dies
    # with it and keeps dedupe keyed on exception identity.
    token = _identity_token(exc)
    if token is not None:
        new_last_seen = weakref.ref(token)
        is_duplicate = last_seen is token
    else:
        new_last_seen = exc
        is_duplicate = last_seen is exc

And the token lookup, after hardening (more on that below):

def _identity_token(exc: BaseException) -> "Optional[_DedupeToken]":
    try:
        if exc.__traceback__ is None:
            return None

        exc_dict = exc.__dict__
        token = exc_dict.get(_DEDUPE_TOKEN_ATTR)
        if isinstance(token, _DedupeToken):
            return token

        token = _DedupeToken()
        exc_dict[_DEDUPE_TOKEN_ATTR] = token
        return token
    except Exception:
        # Exceptions can define ``__dict__`` as a property returning anything,
        # or back it with a mapping that refuses mutation. None of that may
        # break event processing, so fall back to the previous behaviour.
        return None

The __traceback__ is None check is deliberate. An exception that was never raised has no frames to pin, so there is nothing to fix and no reason to touch it. That keeps vars(exc) clean for the common case.

Branch: fix/dedupe-builtin-exception-retention

Everything in this post is reproducible. The harness is at JonathanSolvesProblems/sentry-dedupe-leak-repro: the scaling measurement, the traceback probe, the five adversarial checks against my own fix, and the script that verifies the LLM review claim by claim.

My Improvements

Five tests, and the first one fails without the fix, which is the only thing that makes it a regression test:

• test_dedupe_does_not_retain_builtin_exception puts a sentinel in the raising frame and asserts it is collectable afterwards
• test_dedupe_still_dedupes_builtin_exception re-raises the same object and asserts one event
• test_dedupe_distinguishes_equal_builtin_exceptions raises two identical-looking errors from the same line and asserts two events. This is the one fingerprinting broke
• test_dedupe_leaves_unraised_exception_untouched asserts vars(exc) == {} for a never-raised exception
• test_dedupe_survives_exotic_exception_dict covers the crashes described below

I also verified what the ContextVar actually holds, rather than trusting the memory numbers:

	_last_seen holds	1 MB payload after gc.collect()
master	ValueError (strong ref)	alive
fixed	ReferenceType (weakref)	freed

One process note, because it cost me an hour and nearly cost me a wrong claim. Once the fix was committed to a branch, git stash push <file> had nothing to stash, so my "before and after" runs were quietly comparing the fix against itself. Two measurements came back identical and I believed them for longer than I should have. The tell was a probe printing whether the new symbol was importable, which said True in both columns. Baselines have to be taken with git checkout master -- <file>, and the baseline needs to prove it is actually the baseline.

Best Use of Sentry

Sentry features used: Error Monitoring, Releases, custom Contexts and Tags, Issue Grouping, Seer (Autofix root cause analysis).

The interesting part of instrumenting this was discovering that Sentry could not see the bug, and why.

I built a small asyncio service that opens one long-lived task per session, hits one ordinary ValueError, reports it, and keeps the session open. Ran it against a real Sentry project twice, tagged as two releases, dedupe@unpatched and dedupe@patched.

Sentry showed a completely healthy application. One issue, ValueError: malformed frame on session N, 240 events, handled: yes, mechanism: generic. Exactly what you would expect from a service that reports its own handled errors. Nothing in it suggests 62 MB is being retained.

That is the actual lesson. This bug produces no error. The ValueError is deliberate and correctly handled. Error monitoring alone is structurally incapable of surfacing a retention bug, because retention is not an event.

There is a nice detail buried in that first issue, too. The frame local Sentry captured for the raising frame reads session [Filtered], scrubbed by default PII protection. The one object whose retention was the entire problem was the one thing redacted out of the report.

So I used Sentry two other ways.

First, as a measuring instrument. Every error event carries a custom memory context with the retained figure at the moment it was sent. The errors are identical across both releases; the context is not. Comparing dedupe@unpatched against dedupe@patched on the same issue turns an invisible bug into a visible diff, without a single new error being raised.

Second, by making the leak raise something. The demo checks its own retention after the sessions are open, and when session state is still reachable it reports a distinct RetentionLeak issue carrying the diagnosis:

sessions_pinned         : 60/60
retained_mb             : 62.6
referrer_chain          : Session <- frame(handle_message) <- traceback <- traceback <- exception(ValueError)
dedupe_last_seen_holds  : ValueError

That last line is the whole bug in one field, and getting it was its own small lesson. My first attempt read DedupeIntegration._last_seen from the parent task and reported nothing. _last_seen is a ContextVar, so every asyncio task has its own copy and it always reads empty from outside. Reading it from inside the capturing task gives ValueError on the unpatched release and ReferenceType on the patched one. The thing that made the diagnostic hard to write is precisely the thing that makes the bug scale with task count.

release	_last_seen holds	sessions pinned	retained
dedupe@unpatched	ValueError	60 / 60	62.6 MB
dedupe@patched	ReferenceType	0 / 60	1.3 MB

What Seer made of it

I connected the sentry-python fork to the project and ran Autofix on the RetentionLeak issue. I expected it to stop at my demo code, because that is where the stack trace points. It did not.

Its root cause, verbatim:

DedupeIntegration stores strong exception references in a ContextVar for builtin exceptions, retaining tracebacks and all frame locals (including large session buffers) for the asyncio task's lifetime.

And the supporting chain:

• The ValueError exceptions are kept alive because DedupeIntegration._last_seen ContextVar holds a strong reference to each one.
• weakref.ref(exc) raises TypeError for builtin exception types like ValueError, so the fallback path stores the bare exception object instead of a weakref.
• Each asyncio task has its own copy of the ContextVar (per-task context), so the strong reference to the exception persists for the entire lifetime of each long-lived task.

That is the bug, exactly, including the per-task ContextVar detail that cost me a debugging round of my own to work out. It cited dedupe.py L1-L62 as evidence, so it went and read the SDK source rather than guessing from the stack trace. Its five reproduction steps are accurate enough to hand to someone else.

Then it proposed a fix:

In dedupe.py, change the except TypeError fallback from integration._last_seen.set(exc) to store a hashable identity tuple like (type(exc), id(exc)) instead of the exception object itself.

That fix does not work, and it fails for the same reason the abandoned maintainer branch does. Storing only an identity key is what allows the exception to be freed, and a freed address is immediately reusable. Running Seer's exact tuple over 2000 distinct errors:

distinct errors raised         : 2000
events wrongly dropped as dupe : 1999  (100.0%)

It would trade a memory leak for silently discarding almost every error you report.

There is a wrinkle I only found afterwards, and it is the most interesting thing in this whole section. Seer also opened a pull request against my fork with generated code, and the code is not what the plan said. It does not store an identity tuple at all:

try:
    integration._last_seen.set(weakref.ref(exc))
except TypeError:
    pass

That skips storing anything for builtins. It does remove the leak, and it has none of the address-reuse problem, but it silently disables deduplication for exactly the exception types most errors actually are. It is the fix the reporter originally proposed and that the maintainers explicitly declined, for the reason quoted at the top of this post.

So the plan and the patch disagree with each other, and they are wrong in two different ways. If you let an agent open pull requests, that is worth knowing: the prose it shows you for review is not necessarily the diff it writes.

I want to be fair about what that means, because it is not a gotcha. Seer did the hard part correctly. Diagnosing this required reading past the stack trace into a third-party SDK, understanding weakref support on builtin types, and knowing that ContextVars are per-task under asyncio. It got all three from one event and the source. The part it got wrong is the part that requires knowing how CPython recycles memory addresses, which is not visible anywhere in the evidence it was given.

And it is genuinely interesting that three independent attempts, an abandoned branch by a Sentry maintainer, my own first attempt, and Seer, all reached for a fingerprint or an identity key, and all three are wrong for the same underlying reason. The information available to a fingerprint is not sufficient to express object identity. That the AI converged on the same wrong answer as two humans is more a statement about the problem than about the AI.

What I actually got out of Seer was the thing I would have wanted from a colleague: an independent confirmation of the diagnosis, arrived at from the evidence rather than from my framing of it.

Best Use of Google AI

I had already run my own adversarial pass on the patch: pickling, copy/deepcopy, payload leakage, __slots__ exceptions, and retention through __context__ chains. All five passed.

So I gave the diff to Gemini 3.6 Flash, told it exactly what I had already covered, and asked only for failure modes I had missed, each with a snippet I could run. Then I ran all of them, because a review you do not verify is just a second opinion with extra steps.

It returned six. The scoreboard:

#	Claim	Verdict
1	__dict__ as a property returning None crashes event processing	Confirmed, real crash
2	a __dict__ backed by a mapping that refuses mutation crashes	Confirmed, real crash
3	a dynamic __dict__ breaks token identity	Confirmed, fails open
4	the token is visible in vars(exc)	Confirmed, real tradeoff
5	a thread race yields mismatched tokens	Refuted, 0 of 200 attempts
6	__dict__.clear() loses the token	Confirmed, fails open

Claims 1 and 2 were genuine bugs that my own pass had missed. My except (AttributeError, TypeError) was too narrow, so an exception class doing something unusual with __dict__ could crash Sentry's event processing. The SDK's own contributing guide says integrations must not crash applications, so those were not academic.

class NonDictException(Exception):
    @property
    def __dict__(self):
        return None

# before: AttributeError: 'NoneType' object has no attribute 'get'

Claim 5 was wrong, and I only know that because I ran it 200 times with a threading.Barrier instead of nodding along.

Claims 3 and 6 are real but fail open: you get a duplicate event rather than losing one. That is the correct direction to fail.

Claim 4 is a fair criticism and I have not made it go away. The token lives in exc.__dict__, so it shows up in vars(exc) and makes json.dumps(vars(exc)) raise. I narrowed it to raised exceptions only, and the SDK already does the same kind of thing (sentry_sdk/integrations/django/__init__.py sets _sentry_drf_request_backref on a user object), but it is a real tradeoff and the PR says so rather than hiding it.

Two real crashes found, one confidently wrong claim caught, in a review that took about four minutes. That is a good trade, and it only works if you treat the output as a list of hypotheses instead of a list of findings.

What I took away from this bug

The comment # we can only weakref non builtin types was right there, in the code, for years. It described the exact hazard and then the next line walked straight into it. A fallback path is still a code path, and this one ran for the overwhelming majority of real exceptions.

The other thing: my measurements lied to me twice, in opposite directions. Once when I assumed unbounded growth and a flat graph corrected me, and once when a stash silently did nothing and two identical columns looked like a result. Both times the fix was the same, which was to make the harness prove its own setup before trusting a single number out of it.