Inference
Inference is the process of running a trained model to produce output from new input — as opposed to training, which learns the model's weights. For language models, inference means generating tokens one step at a time, and its speed and cost dominate the economics of deploying models in production.
Definition
InferenceTraining and inference are the two phases of a model's life. Training adjusts weights over large datasets; inference uses the finished weights to answer requests. For autoregressive language models, inference is sequential: each new token depends on all previous ones, which is why generation latency scales with output length.
Inference cost is driven by model size, precision, batch size, and hardware. Optimizations such as quantization, key-value caching, batching, and sparse architectures like mixture of experts all aim to raise throughput and lower per-token cost without materially hurting quality.
Frequently asked questions
What is Inference?
Inference is the process of running a trained model to produce output from new input — as opposed to training, which learns the model's weights. For language models, inference means generating tokens one step at a time, and its speed and cost dominate the economics of deploying models in production.
Sources
Related terms
← Full AI & prompt engineering glossary
Put Inference to work with Prompeteer, the Agentic Contextual AI Platform →