---
title: Latency — AI Glossary
description: Latency is the time delay between sending a request to a model and receiving output. For streaming language models it is often split into time-to-first-token — how long until output begins — and per-token latency, which together determine how responsive an application feels to users.
canonical: https://prompeteer.ai/glossary/latency
---

# Latency

Latency is the time delay between sending a request to a model and receiving output. For streaming language models it is often split into time-to-first-token — how long until output begins — and per-token latency, which together determine how responsive an application feels to users.

## Definition

Latency captures responsiveness rather than volume. In a chat interface, time-to-first-token governs how quickly the user sees something happen, while the rate of subsequent tokens governs how fast the full answer streams in. Long prompts, large models, and cold starts all increase latency.

Latency and throughput are related but distinct: a system can process many requests per second (high throughput) while any single request still takes noticeable time (latency). Techniques like caching, smaller or quantized models, speculative decoding, and streaming help reduce perceived and actual latency.

## Sources

- [Google Cloud — Understanding latency and throughput for LLM serving](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/overview)
- [NVIDIA — Mastering LLM inference latency and throughput](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/)

## Related terms

- [throughput](https://prompeteer.ai/glossary/throughput)
- [inference](https://prompeteer.ai/glossary/inference)
- [quantization](https://prompeteer.ai/glossary/quantization)
- [token](https://prompeteer.ai/glossary/token)