Every on-call rotation eventually produces the same scene: a Python worker pinned at 100% CPU, a queue piling up behind it, and someone suggesting a restart under cProfile, which would throw away the in-flight work along with the exact conditions you are trying to observe. Python 3.15, released on October 9, 2026, removes that dilemma. Its new profiling.sampling module, which goes by the name Tachyon, attaches to a process that is already running, reads its call stacks directly from memory and detaches again, with no code changes and no restart.
That job used to belong to third-party tools such as py-spy, which have to learn the memory layout of every new interpreter release and are not always welcome in locked-down production images. Tachyon ships with the interpreter. It also fixes what made cProfile a poor fit for production in the first place: as PEP 799, the proposal behind the new profiling package, points out, cProfile only observes the main thread, adds overhead to every call and can switch off some of the interpreter's own optimizations. Below is a hands-on walkthrough on a small, deliberately slow worker. Every command and number in it comes from a real run of Python 3.15.0 on Linux, so you can reproduce the whole thing in about ten minutes.
Meet the Patient
Here is a 29-line worker that fetches a batch of orders, removes duplicates and serializes the result. The time.sleep stands in for a slow API call. The bug is easy to spot at this size, so pretend it is buried somewhere inside a 40,000-line service.
import json
import random
import time
def fetch_batch(size=8000):
time.sleep(0.25) # stands in for a slow API call
return [{"id": random.randrange(40_000), "qty": 1} for _ in range(size)]
def dedupe(orders):
seen = [] # the bug: a list where a set belongs
unique = []
for order in orders:
if order["id"] not in seen:
seen.append(order["id"])
unique.append(order)
return unique
def main():
while True:
batch = fetch_batch()
payload = json.dumps(dedupe(batch))
print(f"synced {len(payload)} bytes", flush=True)
if __name__ == "__main__":
main()
Start it in the background so it behaves like a long-running service:
python3.15 worker.py > /dev/null &
PID=$!
Question One: Where Is It Right Now?
The cheapest first question is what the process is doing at this exact moment. The dump subcommand answers it with a single snapshot, formatted like a traceback:
python3.15 -m profiling.sampling dump $PID
Stack dump for PID 340, thread 340 (main thread, has GIL, on CPU; most recent call last):
File "worker.py", line 29, in <module>
main()
File "worker.py", line 24, in main
payload = json.dumps(dedupe(batch))
File "worker.py", line 15, in dedupe
if order["id"] not in seen:
Read the header as carefully as the frames. The thread holds the GIL and is running on a CPU, which rules out a stuck socket read or a lock wait before you have opened a single dashboard. For a hung process, this is the traceback that never got printed. By default you only see the main thread: add -a to dump every thread, and --async-aware to rebuild stacks across await boundaries in asyncio code.
Ten Seconds of Samples, Two Different Answers
One snapshot can mislead, because you might have caught a rare code path. Sampling fixes that by taking a snapshot a thousand times per second, the default rate, and counting where they land. Here is a ten-second window, trimmed to the top rows:
python3.15 -m profiling.sampling attach -d 10 $PID
Captured 10000 samples in 10.00 seconds
nsamples sample% tottime (s) cumul% cumtime (s) filename:lineno(function)
5505/5505 55.1 5.505 55.1 5.505 worker.py:7(fetch_batch)
4393/4393 44.0 4.393 44.0 4.393 worker.py:15(dedupe)
Notice the line numbers. Line 7 is the sleep playing the API call, and line 15 is the membership test. Tachyon attributes time to individual lines, which is the difference between knowing that dedupe is slow and knowing which expression inside it is.
Now repeat the run with --mode=cpu, which keeps only the samples taken while the thread was actually executing on a core:
python3.15 -m profiling.sampling attach -d 10 --mode=cpu $PID
nsamples sample% tottime (s) cumul% cumtime (s) filename:lineno(function)
4639/4639 97.9 4.639 97.9 4.639 worker.py:15(dedupe)
32/32 0.7 0.032 0.7 0.032 encoder.py:263(JSONEncoder.iterencode)
Same process, same ten seconds, two different truths. The default wall mode says that more than half of the elapsed time goes to waiting on the API, so no amount of code tuning will push a batch below a quarter of a second. CPU mode says that almost every cycle the worker burns goes to a single line. Pick the mode that matches the complaint: when users say the service is slow, start with wall time, and when the CPU graph is the thing on fire, start with CPU mode.
Two more modes deserve a spot in your memory. --mode=gil counts only the samples in which a thread holds the GIL, which is how you find the code starving every other thread in a threaded server. --mode=exception counts samples taken while an exception is in flight, exposing libraries that use exceptions as control flow at a cost nobody budgeted for. And if you would rather watch than read a report, attach --live $PID opens a top-style terminal view on Unix-like systems.
Prove the Fix With a Diff Flame Graph
The fix is two edits: seen = set() and seen.add(order["id"]). The more valuable skill is proving that it worked. Record a compact binary baseline before you deploy:
python3.15 -m profiling.sampling attach --binary -d 10 -o before.bin $PID
Once the fixed version is running, sample the new process and compare it against that baseline:
python3.15 -m profiling.sampling attach --diff-flamegraph before.bin -d 10 -o diff.html $NEW_PID
The output is one self-contained HTML file. Frames that grew are drawn in shades of red, frames that shrank in shades of blue, and anything that moved by less than 15% stays gray. In this run dedupe came out 99% smaller, and throughput confirmed it: the worker went from 22 batches every ten seconds to 39.
Then comes the trap. The untouched time.sleep on line 7 turned red, flagged as roughly 75% worse, and so did the JSON encoder. Neither got slower. Over a fixed ten-second window, a faster loop simply reaches them more often, so they claim the share of samples that dedupe gave back. In a wall-clock diff, red means "more of the window", not "slower per call", so always read the colors next to a throughput number. A fresh wall profile of the fixed worker puts 98.1% of its samples on line 7. That is the honest next bottleneck, and it is a design question about batching, concurrency or caching rather than a profiling one.
The Fine Print Before You Try This in Production
Tachyon is designed for production use, but a few rules decide whether attach works at all and whether its numbers mean what you think they mean:
-
Permissions. On Linux the profiler needs root, the
CAP_SYS_PTRACEcapability or a relaxed Yamaptrace_scope. Without them it stops with an error that, for containers, points you to--cap-add=SYS_PTRACE. macOS requires root or a debugger entitlement, and Windows needs administrator rights orSeDebugPrivilege. - Matching versions. The profiler and the target must run the same Python minor version, pre-releases must match exactly, and free-threaded and standard builds cannot attach to each other. Pointing 3.15 at a 3.13 process fails with "Can't attach from a Python 3.15 process to a Python 3.13 process".
-
Threads. Only the main thread is sampled by default. In a threaded web server the main thread may sit in an accept loop while the real work happens elsewhere, so add
-a. -
Worker pools.
--subprocessesprofiles Python child processes as well. Attached to amultiprocessingparent whose workers were already running, it wrote a separate flame graph for every child. -
The profiler's own bill. The target does not slow down, but the sampler is not free. At the default 1 kHz it used about 0.6 seconds of CPU over a ten-second window, and at
-r 10khzit consumed a full core, while the worker held steady at 22 batches either way. On a host that is already short of CPU, that core has to come from somewhere. -
Statistics. Times are estimates built from sample counts. The documentation puts the margin at roughly ±0.5% for a function at 5% of 100,000 samples, widening to anywhere between 3% and 7% with only 1,000 samples. For scripts that finish in under a second, exact call counts or differences of 1 or 2%, reach for
profiling.tracingortimeitinstead.
Kubernetes: Borrow the App's Own Image
Containers are where the version rule bites hardest, because the profiler has to run next to the target with the same interpreter. In Kubernetes, the cleanest route is an ephemeral debug container started from the application's own image, which guarantees an identical Python, and joined to the app container's process namespace with --target. The general debugging profile adds the SYS_PTRACE capability that Tachyon needs, and the Kubernetes guide to debugging running pods explains how --target and the profiles work.
kubectl debug -it my-pod --image=registry.example.com/orders-worker:1.42 \
--target=worker --profile=general \
-- python -m profiling.sampling attach -d 30 1
Inside a shared process namespace the application is usually PID 1, and the report prints straight to your terminal. One caveat from the kubectl reference: if the pod forces a non-root user, some capabilities granted by a debug profile may not take effect, so rehearse this in staging long before the night you need it.
Write the Runbook Entry Now
Incidents are a bad time to discover that your container lacks CAP_SYS_PTRACE or that your debug image runs a different Python. Spend fifteen minutes this week attaching Tachyon to a staging copy of your busiest service, write down the exact command that worked, and put it in the runbook right next to the restart instructions. When something pins a CPU at 2 a.m., dump tells you where the process is, a ten-second attach tells you why, and a diff flame graph tells you whether the fix landed. Three commands, zero restarts.
While you are updating the runbook, note that cProfile now lives on as an alias of profiling.tracing, and the old pure-Python profile module warns on import and is scheduled for removal in Python 3.17.
Which service would you point Tachyon at first, and what do you expect the first profile to show? Drop it in the comments.
Top comments (0)