Latency
Latency is the time delay between sending a request to a model and receiving output. For streaming language models it is often split into time-to-first-token — how long until output begins — and per-token latency, which together determine how responsive an application feels to users.
Definition
LatencyLatency captures responsiveness rather than volume. In a chat interface, time-to-first-token governs how quickly the user sees something happen, while the rate of subsequent tokens governs how fast the full answer streams in. Long prompts, large models, and cold starts all increase latency.
Latency and throughput are related but distinct: a system can process many requests per second (high throughput) while any single request still takes noticeable time (latency). Techniques like caching, smaller or quantized models, speculative decoding, and streaming help reduce perceived and actual latency.
Frequently asked questions
What is Latency?
Latency is the time delay between sending a request to a model and receiving output. For streaming language models it is often split into time-to-first-token — how long until output begins — and per-token latency, which together determine how responsive an application feels to users.
Sources
Related terms
← Full AI & prompt engineering glossary
Put Latency to work with Prompeteer, the Agentic Contextual AI Platform →