Inference and Decoding

A support bot answering “what’s your return policy?” needs the same correct answer every time. A creative-writing assistant generating story ideas needs a different answer every time. Both are the same model, the same prompt format, the same underlying probability distribution over the next token — the only thing that differs is how a token gets chosen from that distribution.

Greedy Decoding Is Deterministic

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
model.eval()

prompt = "The best way to learn programming is"
ids = tok(prompt, return_tensors="pt")

# Greedy: always take the single highest-probability token, deterministic
for run in range(2):
    out = model.generate(**ids, max_new_tokens=12, do_sample=False)
    text = tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
    print(f"Greedy run {run + 1}: {text!r}")

$ python greedy.py
Greedy run 1: ' to learn from the experience of the programmer.\n\n\n'
Greedy run 2: ' to learn from the experience of the programmer.\n\n\n'

Two runs, identical output. do_sample=False (equivalently, temperature 0 in most hosted APIs) always takes the single token with the highest probability at every step — there is no randomness to produce a different answer.

Temperature Reshapes the Distribution Before Sampling

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
model.eval()

prompt = "The best way to learn programming is"
ids = tok(prompt, return_tensors="pt")

for temp in [0.1, 1.0, 2.0]:
    torch.manual_seed(0)
    out = model.generate(
        **ids, max_new_tokens=12, do_sample=True, temperature=temp, top_k=0, top_p=1.0,
    )
    text = tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
    print(f"temperature={temp}: {text!r}")

$ python temperature.py
temperature=0.1: ' to learn from the experience of programming.\n\n\n\n'
temperature=1.0: ' to study seven different basic languages spoken in "tree" ('
temperature=2.0: ' along gritty pilot Definitionave Girls PROGRAM is "tree shaping bipartisan'

Temperature divides the logits by itself before softmax. Below 1.0, it sharpens the distribution toward the already-most-likely tokens (temperature=0.1 reads almost like greedy decoding). Above 1.0, it flattens the distribution — low-probability tokens become much more likely to get picked, and at 2.0 the output stops being coherent English at all. There is no “correct” temperature; it’s a dial between reliability and variety.

Top-k and Top-p Change Which Tokens Are Even Eligible

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
model.eval()

prompt = "The best way to learn programming is"
ids = tok(prompt, return_tensors="pt")
with torch.no_grad():
    logits = model(**ids).logits[0, -1]
probs = torch.softmax(logits, dim=-1)
sorted_probs, sorted_idx = torch.sort(probs, descending=True)

cumulative = torch.cumsum(sorted_probs, dim=0)
for p in [0.5, 0.9, 0.95]:
    n = (cumulative < p).sum().item() + 1
    print(f"top-p={p}: {n} tokens needed to reach {p*100:.0f}% cumulative probability")

print()
for label, kwargs in [
    ("top_k=5",   dict(do_sample=True, top_k=5, top_p=1.0)),
    ("top_k=50",  dict(do_sample=True, top_k=50, top_p=1.0)),
    ("top_p=0.9", dict(do_sample=True, top_k=0, top_p=0.9)),
]:
    torch.manual_seed(0)
    out = model.generate(**ids, max_new_tokens=12, temperature=1.0, **kwargs)
    text = tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
    print(f"{label:10}: {text!r}")

$ python top_k_top_p.py
top-p=0.5: 1 tokens needed to reach 50% cumulative probability
top-p=0.9: 15 tokens needed to reach 90% cumulative probability
top-p=0.95: 82 tokens needed to reach 95% cumulative probability

top_k=5   : ' to read the code of the program you are programming with.'
top_k=50  : ' to study it with our students and teach them the basics,'
top_p=0.9 : ' to study it with basic materials and is "why" ('

Only 1 token carries 50% of the probability mass for this prompt — the model is quite confident about the next word. But reaching 95% takes 82 tokens — the tail is long. top_k=5 restricts sampling to a fixed 5 candidates no matter what; top_p=0.9 (nucleus sampling) adapts: it would have picked from just 1-2 tokens here if the distribution were as peaked as the 50%-mass number suggests, and from many more on an uncertain prompt. That adaptiveness is the whole argument for preferring top-p over top-k in production.

Repetition Penalty and Stop Sequences

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
model.eval()

prompt = "The history of computers is"
ids = tok(prompt, return_tensors="pt")

print("=== Greedy decoding loops ===")
out = model.generate(**ids, max_new_tokens=40, do_sample=False)
print(repr(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)))

print()
print("=== Same prompt with a repetition penalty ===")
out2 = model.generate(**ids, max_new_tokens=40, do_sample=False, repetition_penalty=1.3)
print(repr(tok.decode(out2[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)))

print()
print("=== Stop sequence cuts generation short ===")
out3 = model.generate(**ids, max_new_tokens=40, do_sample=False, repetition_penalty=1.3,
                       eos_token_id=tok.encode(".")[0])
print(repr(tok.decode(out3[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)))

$ python repetition.py
=== Greedy decoding loops ===
' a fascinating one. It is a fascinating one. It is a fascinating one. It is a fascinating one. It is a fascinating one. It is a fascinating one. It is a fascinating one.'

=== Same prompt with a repetition penalty ===
' a fascinating one. It\'s not just about the speed and complexity, it\'s also how much time we spend on our computer every day in order to make sure that everything works out for us all."'

=== Stop sequence cuts generation short ===
' a fascinating one.'

Greedy decoding walked straight into a loop: once “It is a fascinating one.” became the highest-probability continuation once, it stayed the highest-probability continuation forever — the model has no memory of “I already said this.” A repetition penalty divides the logits of already-generated tokens, which is enough to knock the model out of the loop and onto a different, non-repeating continuation. The stop-sequence run shows the other common fix for a different problem: even without a loop, a model will keep generating past the point you wanted an answer to end, unless something tells it to stop.

Practice

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
ids = tok("The best way to learn programming is", return_tensors="pt")

for seed in [0, 1, 2]:
    torch.manual_seed(seed)
    out = model.generate(**ids, max_new_tokens=12, do_sample=True, temperature=0.1, top_k=0, top_p=1.0)
    print(f"seed={seed}: {tok.decode(out[0][ids['input_ids'].shape[1]:], skip_special_tokens=True)!r}")

seed=0: ' to learn from the experience of programming.\n\n\n\n'
seed=1: ' to learn from the experience of the programmer.\n\n\n'
seed=2: ' to learn from the people who teach you.\n\n\n'

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
ids = tok("The best way to learn programming is", return_tensors="pt")

out_greedy = model.generate(**ids, max_new_tokens=12, do_sample=False)
out_topk1  = model.generate(**ids, max_new_tokens=12, do_sample=True, top_k=1)
print("greedy:", repr(tok.decode(out_greedy[0][ids['input_ids'].shape[1]:], skip_special_tokens=True)))
print("top_k=1:", repr(tok.decode(out_topk1[0][ids['input_ids'].shape[1]:], skip_special_tokens=True)))

greedy: ' to learn from the experience of the programmer.\n\n\n'
top_k=1: ' to learn from the experience of the programmer.\n\n\n'

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")
ids = tok("The history of computers is", return_tensors="pt")

for penalty in [1.3, 1.8]:
    out = model.generate(**ids, max_new_tokens=30, do_sample=False, repetition_penalty=penalty)
    print(f"penalty={penalty}: {tok.decode(out[0][ids['input_ids'].shape[1]:], skip_special_tokens=True)!r}")

penalty=1.3: " a fascinating one. It's not just about the speed and complexity, it's also how much time we spend on our computer every day in order to"
penalty=1.8: " a fascinating one. It's not just about the speed and complexity, it also has its own unique characteristics that make computing interesting to people who are interested"