Token
A token is the unit of text a language model reads and generates. Tokenizers split text into tokens — often subword fragments rather than whole words — and each is mapped to an integer ID. Model context limits, pricing, and throughput are all counted in tokens, not characters or words.
Definition
TokenModern language models do not operate on raw characters or words directly. A tokenizer first breaks input text into a sequence of tokens using an algorithm such as byte-pair encoding, then maps each token to an integer that indexes an embedding table. Common English words are frequently a single token, while rare words, code, and non-Latin scripts often split into several.
Because everything a model processes is measured in tokens, the token count of a prompt and its response determines both cost and how much fits inside the context window. A rough rule of thumb for English is that one token corresponds to about four characters, but this varies widely by content and tokenizer.
Frequently asked questions
What is Token?
A token is the unit of text a language model reads and generates. Tokenizers split text into tokens — often subword fragments rather than whole words — and each is mapped to an integer ID. Model context limits, pricing, and throughput are all counted in tokens, not characters or words.
Sources
Related terms
← Full AI & prompt engineering glossary
Put Token to work with Prompeteer, the Agentic Contextual AI Platform →