Top-p (Nucleus) Sampling

Top-p (nucleus) sampling selects the smallest set of tokens whose cumulative probability exceeds threshold p, then samples from them. Unlike top-k (fixed count), top-p adapts: it picks fewer tokens when the model is confident and more when uncertain.

Steps:

Apply softmax to logits
Sort probabilities descending
Compute cumulative sum
Keep tokens up to and including the one that crosses threshold p
Renormalize and return in original token order

Signature: def top_p_sampling(logits, p)

logits: (vocab_size,) — unnormalized scores
p: float — cumulative probability threshold (0 < p ≤ 1)
Returns: (vocab_size,) — renormalized probability distribution

Math

p_{i} = \frac{exp ( z _{i} )}{\sum _{j} exp ( z _{j} )} N_{p} = min {S \subseteq V : i \in S \sum p_{i} \geq p} \overset{p}{^}_{i} = {p_{i} / \sum_{j \in N_{p}} p_{j} 0 i \in N_{p} otherwise

Asked at

Steps:

Apply softmax to logits
Sort probabilities descending
Compute cumulative sum
Keep tokens up to and including the one that crosses threshold p
Renormalize and return in original token order

Signature: def top_p_sampling(logits, p)

logits: (vocab_size,) — unnormalized scores
p: float — cumulative probability threshold (0 < p ≤ 1)
Returns: (vocab_size,) — renormalized probability distribution

Math

p_{i} = \frac{exp ( z _{i} )}{\sum _{j} exp ( z _{j} )} N_{p} = min {S \subseteq V : i \in S \sum p_{i} \geq p} \overset{p}{^}_{i} = {p_{i} / \sum_{j \in N_{p}} p_{j} 0 i \in N_{p} otherwise

Asked at

33. Top-p (Nucleus) Sampling

33. Top-p (Nucleus) Sampling