How Language Models Generate Text: Tokens, Context and Probabilities

From text to units the model processes

A language model does not process a page exactly as a person reads it word by word. Inputs are converted into units commonly called tokens. A token may be a short word, part of a word, punctuation, a number, or another unit defined by the tokenizer. “100 words” therefore does not map to one fixed token count, and segmentation can differ across models and languages. This matters because the amount of material a model can consider is typically measured in tokens, and everything needed for a task must fit within the context the model can process at that point.

Context is the material the model can use during a particular generation request: user instructions, earlier conversation content that the application passes forward, examples, retrieved text, or tool results. A context window limits how much information can be processed in one request, with details varying by product and model. A fact sitting in an external file is not automatically visible to the model; an application must pass it, retrieve the relevant part, or call a tool that can reach it. Late or conflicting information inside context can also affect the result, so organizing context is part of task quality.

Probabilistic generation: selecting a plausible continuation, not retrieving one fixed answer

In a common generative language-model setup, a response is built progressively by choosing a next token based on previous tokens and context. Internally, the model estimates a probability distribution over possible continuations, and the generation process uses those estimates to select what comes next. This helps explain why the same request can have more than one reasonable wording and why literal identity across runs should not be assumed. The probability distribution is not the model’s calibrated probability that each statement is true; it represents the fit of a continuation to what came before. A highly fluent continuation can still be factually weak.

Non-determinism means repeated runs can differ, with the amount of variation depending on the model, decoding process, and application settings. Important workflows should therefore not rely on “the answer will always use the same wording.” If a system needs a stable structure, specify the output format, test it, and use constraints or machine validation when the environment supports them. If the task depends on one factual answer, connect it to evidence and verify content rather than comparing the response character-for-character with an earlier output.

Why the Transformer architecture matters

The 2017 paper Attention Is All You Need introduced the Transformer architecture for sequence modeling using attention mechanisms as the central design rather than recurrent or convolutional networks as the sequence backbone common in earlier approaches. Attention enables layers to calculate relationships among different positions in a sequence. For this course, the key point is not reproducing the equations. It is understanding that modern language models use representations built from relationships within context rather than behaving like a database that looks up a stored sentence for every question. Modern architectures vary, so the 2017 paper is historical foundation, not a complete description of every current system.