Inference & sampling — how AI speaks | How AI Works

01 · One token at a time

Generation is a loop you can pause

Every word you see streamed from a model was a full forward pass ending in a dice roll over the . Hit generate, then pause mid-sentence to see the dice — and click a long-shot token to fork reality.

The best thing about living by the sea is▌

Next-token probabilities — click any to force it (fork the timeline)

Precomputed branching tree (3 levels + completions) with hand-set probabilities — the forking logic is real, the model is canned.

02 · Loading the dice

Temperature, top-p, top-k — one distribution, three knobs

These three settings exist in every AI API. They don't change what the model thinks — they reshape the dice before the roll. All three sliders below act on one real distribution.

Temperature0.70

Top-p0.90

Top-k10

Next token after "My favourite season is…" — real softmax over fixed logits · 4/10 tokens survive the cutoffs

autumn

43%

summer

32%

winter

14%

spring

12%

definitely

2.0%

probably

1.1%

the

0.7%

fall

0.6%

always

0.2%

mango

0.0%

Example completion in this regime

"My favourite season is summer, mostly for the long evenings."Balanced: variety between runs, nonsense still suppressed.

03 · The working-memory budget

Context windows, to scale

The is everything the model can "see" at once. Compare what fits:

200,000 tokens ≈ ~300 pages — an entire novel, or a mid-sized repo.

Knowledge check

Did it stick?

1/3At temperature 0, what does the model do?

2/3Top-p = 0.9 means:

3/3Why does a 200-token answer take ~200× the compute of scoring 1 token?

Loading module…

Common questions this module answers

What does temperature do in an LLM?

Temperature rescales the model's next-token probabilities before sampling. Low temperature (→0) makes it always pick the most likely token (deterministic); high temperature flattens the distribution so unlikely tokens get picked more — more creative, more error-prone.

What is top-p (nucleus) sampling?

Top-p keeps the smallest set of tokens whose probabilities sum to p (say 0.9) and samples only from those, cutting off the long tail of implausible tokens while keeping flexibility among likely ones.

What is a context window?

The maximum number of tokens the model can consider at once — prompt plus generated output. Everything outside the window simply does not exist for the model.