---
title: Throughput — AI Glossary
description: Throughput is the volume of work a model-serving system completes per unit of time, commonly measured in tokens per second or requests per second across all concurrent users. It reflects overall serving capacity and cost efficiency, in contrast to latency, which measures the delay of a single request.
canonical: https://prompeteer.ai/glossary/throughput
---

# Throughput

Throughput is the volume of work a model-serving system completes per unit of time, commonly measured in tokens per second or requests per second across all concurrent users. It reflects overall serving capacity and cost efficiency, in contrast to latency, which measures the delay of a single request.

## Definition

Where latency is about how fast one request feels, throughput is about how much total work the system does. Batching many requests together raises throughput by keeping hardware busy, but larger batches can increase the latency of individual requests, creating a fundamental trade-off in inference serving.

Maximizing throughput lowers the effective cost per token, which matters for high-volume production workloads. Serving systems tune batch size, use paged attention and key-value caching, and choose model precision to push throughput as high as the latency budget allows.

## Sources

- [Kwon et al. (2023), Efficient Memory Management for LLM Serving with PagedAttention (vLLM)](https://arxiv.org/abs/2309.06180)
- [NVIDIA — Mastering LLM inference optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/)

## Related terms

- [latency](https://prompeteer.ai/glossary/latency)
- [inference](https://prompeteer.ai/glossary/inference)
- [quantization](https://prompeteer.ai/glossary/quantization)
- [mixture-of-experts](https://prompeteer.ai/glossary/mixture-of-experts)