A single failed flush was silently killing telemetry in the Sentry Python SDK

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

This entry is a real, tested bug fix in the Sentry Python SDK for issue #7138. The SDK batches logs, metrics and spans and ships them from a background daemon thread. If one flush ever raised, that thread died. From then on the process kept buffering telemetry that was never sent and eventually dropped. No error reached the application. The fix makes the flush loop tolerate a failed batch and keep running.

Bug Fix or Performance Improvement

Sentry's SDK does not send every log or span the moment you create it. It buffers them in a Batcher and a background daemon thread drains the buffer on a timer. That thread runs a simple loop: wait, then flush.

def _flush_loop(self):
    self._active.flag = True
    while self._running:
        self._flush_event.wait(self.FLUSH_WAIT_TIME + random.random())
        self._flush_event.clear()
        self._flush()   # nothing catches an exception here

_flush() serializes the buffered items and hands them to the transport. If any part of that raised, the exception propagated straight out of _flush_loop. Because this is the thread's entry point, the thread ended. Python does not restart it.

The quiet part is what happens next. add() decides whether to start the flusher by checking _flusher_pid == os.getpid(). After the thread has died that check still passes, because the pid never changed, so add() happily keeps appending to a buffer that nothing drains. Items pile up until the buffer hits its hard cap (MAX_BEFORE_DROP, 1000 for logs). Every item after that is dropped through _record_lost. So a single transient error inside one flush turns into permanent, silent telemetry loss for the rest of the process lifetime. On a long-lived web worker the whole point of running the SDK is gone, with nothing in the logs to say why.

The issue was filed with a concrete trigger: a RuntimeError: _core::PyFeatureContext is unsendable ... raised while serializing a value during a flush. But the class of the bug is broader than any one trigger. The batcher sits on the SDK's own hot path and must not let one bad item take down the sender.

Code

The fix follows the issue title exactly: tolerate exceptions in the flush loop. Each loop wraps its flush in capture_internal_exceptions(), the SDK's own context manager for errors that are the SDK's own fault and should be logged rather than thrown at the user.

from sentry_sdk.utils import capture_internal_exceptions, format_timestamp

def _flush_loop(self):
    self._active.flag = True
    while self._running:
        self._flush_event.wait(self.FLUSH_WAIT_TIME + random.random())
        self._flush_event.clear()
        with capture_internal_exceptions():
            self._flush()

The same guard goes on SpanBatcher._flush_loop, which has two flush calls (the pending-buckets flush and the periodic full flush), so all three batchers are covered: logs and metrics through the base Batcher, spans through the SpanBatcher override. A failed batch is now swallowed and logged. The loop stays alive and the next flush delivers everything that queued up in the meantime.

Branch: fix/tolerate-exceptions-in-flush-loop on my fork, commit 85d96cc. sentry-python's bot auto-closes PRs opened before a maintainer has agreed on the issue, so per the repo's own contributing guide the change is staked on the issue thread and pushed as a branch rather than a cold PR that would just be closed.

My Improvements

The diff is small and on point: four files, +94 / -15, with the real source change being the two with capture_internal_exceptions(): guards. The rest is comments and tests.

I added one regression test per loop, test_flush_loop_swallows_flush_exception, in tests/test_logs.py (base Batcher) and tests/tracing/test_span_batcher.py (SpanBatcher). Each builds a batcher whose _flush raises once and then stops the loop, calls _flush_loop() directly on the main thread and asserts it returns instead of raising. Run against the unfixed source both fail with RuntimeError: boom in flush, which is exactly the propagation that killed the real thread. With the fix both pass. They carry the repo's own @pytest.mark.tests_internal_exceptions marker, the suite's opt-in for tests that exercise capture_internal_exceptions, which is a nice confirmation that the fix routes through the SDK's standard internal-error handling rather than a bespoke try/except.

Verification, all green: 73 passed across the logs, metrics and span-batcher suites, ruff check and ruff format --check clean on the changed files, mypy sentry_sdk shows no new errors versus the base (its 28 pre-existing errors are all in an unrelated integration, none in the batcher files).

Best Use of Sentry

This is a fix to Sentry's own delivery pipeline, the part every log, metric and span passes through on its way out. The failure mode it removes is the worst kind for an observability tool: the SDK stops reporting and never says so, so the first you know is a gap in your dashboards during an incident. Making the flush loop survive a single bad batch means the SDK degrades to "log the internal error and keep going" instead of "go silent forever," which is the behavior you want from the thing you rely on to tell you when everything else is broken.

AI disclosure

AI assistance (Claude, Anthropic) was used in developing this change. The design, review and verification were done by the author. Verified locally before submitting: the two new regression tests fail on the unpatched source and pass with the fix, the logs, metrics and span-batcher suites pass (73 tests), ruff check and ruff format --check are clean on the changed files, mypy sentry_sdk shows no new errors versus the base.