Throughput
Throughput is the volume of work a model-serving system completes per unit of time, commonly measured in tokens per second or requests per second across all concurrent users. It reflects overall serving capacity and cost efficiency, in contrast to latency, which measures the delay of a single request.
Definition
ThroughputWhere latency is about how fast one request feels, throughput is about how much total work the system does. Batching many requests together raises throughput by keeping hardware busy, but larger batches can increase the latency of individual requests, creating a fundamental trade-off in inference serving.
Maximizing throughput lowers the effective cost per token, which matters for high-volume production workloads. Serving systems tune batch size, use paged attention and key-value caching, and choose model precision to push throughput as high as the latency budget allows.
Frequently asked questions
What is Throughput?
Throughput is the volume of work a model-serving system completes per unit of time, commonly measured in tokens per second or requests per second across all concurrent users. It reflects overall serving capacity and cost efficiency, in contrast to latency, which measures the delay of a single request.
Sources
Related terms
← Full AI & prompt engineering glossary
Put Throughput to work with Prompeteer, the Agentic Contextual AI Platform →