Temperature & Sampling Control

What it is

Sampling control governs how the model turns its next-token probability distribution into actual text. Temperature rescales the distribution — low temperature sharpens it toward the most likely tokens (focused, repeatable output), high temperature flattens it (more varied, exploratory output). Top-p (nucleus) sampling instead restricts choices to the smallest set of tokens whose cumulative probability exceeds a threshold.

These are not prompt wording but decoding settings, yet they materially change results. Deterministic tasks (extraction, classification, code that must be exact) call for low randomness; creative tasks (brainstorming, varied copy) benefit from higher randomness — and techniques like self-consistency deliberately raise temperature to get diverse samples.

When to use

• Extraction, classification, or factual tasks that need consistent, repeatable output → low temperature.
• Brainstorming, creative writing, or idea variety → higher temperature.
• Sampling several diverse candidates (e.g. for self-consistency or ranking) → non-zero temperature.
• Debugging prompt behavior where reproducibility helps → lowest randomness.

How it works

1. Identify whether the task wants determinism or diversity.
2. Set temperature low for focused output, higher for creative variety.
3. Adjust top-p to cap the token pool rather than (or alongside) temperature.
4. Change one parameter at a time and evaluate; avoid stacking extreme settings.

Illustrative structure

The structure is a settings profile, not prompt text: for a deterministic extractor, temperature ≈ 0; for varied brainstorming, temperature raised and/or top-p relaxed; for self-consistency, temperature > 0 to produce diverse chains to vote over.

Pitfalls

• High temperature raises the risk of incoherent or off-task output.
• Very low temperature can cause repetitive, bland, or looping text.
• Tuning temperature and top-p together to extremes interacts unpredictably; change one at a time.
• These settings do not fix a bad prompt — wording and settings are complementary, not substitutes.

Frequently asked questions

Sources

• OpenAI — API reference: temperature and top_p sampling parameters
• Holtzman et al. (2019), The Curious Case of Neural Text Degeneration (nucleus sampling)

Related techniques

• Self-Consistency
• Zero-Shot Prompting
• Structured / JSON Output