---
title: Token — AI Glossary
description: A token is the unit of text a language model reads and generates. Tokenizers split text into tokens — often subword fragments rather than whole words — and each is mapped to an integer ID. Model context limits, pricing, and throughput are all counted in tokens, not characters or words.
canonical: https://prompeteer.ai/glossary/token
---

# Token

A token is the unit of text a language model reads and generates. Tokenizers split text into tokens — often subword fragments rather than whole words — and each is mapped to an integer ID. Model context limits, pricing, and throughput are all counted in tokens, not characters or words.

## Definition

Modern language models do not operate on raw characters or words directly. A tokenizer first breaks input text into a sequence of tokens using an algorithm such as byte-pair encoding, then maps each token to an integer that indexes an embedding table. Common English words are frequently a single token, while rare words, code, and non-Latin scripts often split into several.

Because everything a model processes is measured in tokens, the token count of a prompt and its response determines both cost and how much fits inside the context window. A rough rule of thumb for English is that one token corresponds to about four characters, but this varies widely by content and tokenizer.

## Sources

- [OpenAI — What are tokens and how to count them](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)
- [Sennrich et al. (2016), Neural Machine Translation of Rare Words with Subword Units (BPE)](https://arxiv.org/abs/1508.07909)

## Related terms

- [context-window](https://prompeteer.ai/glossary/context-window)
- [embedding](https://prompeteer.ai/glossary/embedding)
- [inference](https://prompeteer.ai/glossary/inference)