The latest articles on DEV Community by Taylor Parker (@parker_58). https://dev.to/parker_58 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4211029%2Fd7c1eb65-e011-4ca5-a863-f83c859724c8.jpg https://dev.to/parker_58 en Taylor Parker Tue, 27 Oct 2026 15:26:44 +0000 https://dev.to/parker_58/batching-is-the-feature-serving-pytorch-at-low-traffic-3f58 https://dev.to/parker_58/batching-is-the-feature-serving-pytorch-at-low-traffic-3f58 <p>At low traffic, a model server is mostly idle. The obvious optimization — batching — is also the one people skip because it feels like a high-traffic problem.</p> <p>But even at a few requests per second, dynamic batching smoothed out our p99 dramatically. The trick was making it not add latency: a 10ms window, max batch size 8.</p> <p>We ended up writing a tiny batching layer in front of the model, then moved it into the service once the pattern was clear.</p> <p>I'm learning Rust to see what a zero-GC serving layer looks like. If you've built something similar, I'd love to compare notes.</p> machinelearning pytorch performance