Batching & Serving
A GPU pass has a large fixed cost — loading the model's weights from memory — whether it processes one request or thirty-two. Serving one conversation at a time leaves the hardware mostly idle. Batching stacks many requests into a single pass so the fixed cost is paid once and shared: throughput multiplies, at the price of a small wait while the batch fills. Production servers go further with continuous batching, slotting new requests into the batch the moment another finishes, instead of waiting for a whole round.
Slide the batch size. Watch queue time (grey) trade against total time, GPU utilisation climb, and throughput jump as more requests share each pass.
Check yourself
Batch size 8 serves everyone fastest here — so why do latency-sensitive services still cap their batch size?
Go deeper (free): vLLM — Easy, fast LLM serving (blog) ↗