LLM Fundamentals
Temperature, Top-k, Top-p Sampling
Sampling techniques are the probabilistic mechanisms that determine how an LLM selects the next token from its output distribution. By adjusting these hyperparameters, developers can shift a model's behavior from precise, deterministic logic to creative, unpredictable generation. Understanding these parameters is essential for balancing factual accuracy against the need for linguistic variety and originality.
Understanding the Raw Output Distribution
Before applying any sampling strategy, an LLM processes the input prompt and computes a probability distribution over its entire vocabulary for the next token. This raw output comes in the form of 'logits'—unnormalized scores representing the model's confidence in each candidate token. To make these scores usable, we apply the Softmax function, which transforms them into a probability distribution where all values sum to exactly one. In a deterministic setting, one might simply choose the token with the highest probability, known as greedy decoding. However, this often leads to repetitive, loops-prone, or lackluster text. Understanding that the model essentially treats the next token prediction as a game of chance allows us to manipulate the selection process, enabling us to introduce controlled randomness that makes machine-generated text feel more natural, varied, and human-like during complex creative tasks.
import torch
# Simulated logits for a vocabulary of 5 words
logits = torch.tensor([2.0, 1.0, 0.1, 0.5, 0.8])
# Softmax converts raw logits to probabilities summing to 1.0
probabilities = torch.softmax(logits, dim=0)
print(f"Probabilities: {probabilities.numpy()}")Temperature: Scaling Confidence
Temperature is a hyperparameter that directly scales the logits before the Softmax function is applied. By dividing logits by the temperature value (T), we alter the 'sharpness' of the resulting distribution. When T < 1, the high-probability tokens become even more dominant, while low-probability tokens are suppressed, leading to more conservative and deterministic outputs. Conversely, when T > 1, the probability distribution flattens, making tokens that were previously unlikely now seem more plausible to the sampler. This is effective because it forces the model to explore less common linguistic paths, which is vital for brainstorming or creative writing. It is important to realize that temperature does not change the model's underlying knowledge; it only changes the model's willingness to gamble on tokens that are not the statistically most likely choice during the next-token generation process.
def apply_temperature(logits, t):
# Higher T flattens the distribution, lower T sharpens it
scaled_logits = logits / t
return torch.softmax(scaled_logits, dim=0)
print(f"T=0.5: {apply_temperature(logits, 0.5)}")
print(f"T=1.5: {apply_temperature(logits, 1.5)}")Top-K Sampling: Truncating the Long Tail
Even with temperature adjustments, the long tail of the vocabulary distribution—containing thousands of extremely unlikely tokens—can occasionally be selected, resulting in nonsensical or incoherent text. Top-K sampling addresses this by restricting the model's choice to only the 'K' most probable next tokens. By setting a hard limit, we effectively chop off the tail of the distribution, ensuring the model never picks a completely irrelevant or 'crazy' word, regardless of how much we might have flattened the distribution using temperature. This technique provides a safety net, forcing the model to select only from a set of high-confidence candidates. It is particularly useful in structured tasks where accuracy is paramount, as it prevents the model from wandering too far into linguistic territory that might break the logic or grammatical structure of the generated sentence.
def top_k_sampling(logits, k=3):
# Set all tokens outside the top-k to negative infinity
top_k_values, _ = torch.topk(logits, k)
min_val = top_k_values[:, -1] if logits.dim() > 1 else top_k_values[-1]
logits[logits < min_val] = float('-inf')
return torch.softmax(logits, dim=0)
print(f"Top-3 distribution: {top_k_sampling(logits.clone(), 3)}")Top-P (Nucleus) Sampling: Adaptive Truncation
Top-P, also known as nucleus sampling, offers a more dynamic approach than Top-K. Instead of picking a fixed number of tokens, Top-P selects the smallest subset of tokens whose cumulative probability exceeds a threshold 'p' (e.g., 0.9). This is statistically superior to Top-K because the size of the 'nucleus' changes based on how confident the model is. If the model is very sure of the next token, the set will be small; if the model is uncertain, the set will expand to include more candidates. This adapts perfectly to the context, allowing for narrow selections when precision is needed and wider selections when the model is presented with ambiguous prompts. By using Top-P, you ensure that the model always maintains a consistent level of quality, as it effectively filters out low-probability noise while allowing for creative breadth.
def top_p_sampling(logits, p=0.9):
sorted_logits, indices = torch.sort(logits, descending=True)
cumulative_probs = torch.cumsum(torch.softmax(sorted_logits, dim=0), dim=0)
# Remove tokens where cumulative probability exceeds p
to_keep = cumulative_probs <= p
# Ensure at least one token is kept
to_keep[0] = True
return to_keep, indices
print(f"Keep mask for P=0.9: {top_p_sampling(logits, 0.9)[0]}")Combining Techniques for Robust Generation
In production environments, these sampling strategies are rarely used in isolation; they are often combined to layer their benefits. A common configuration is to use a moderate temperature to encourage diversity, followed by Top-P sampling to prune the long tail of unlikely, nonsensical tokens. By carefully tuning these parameters, you create a generation pipeline that is both creative enough to sound natural and constrained enough to maintain relevance to the user's prompt. The goal is to avoid 'over-sampling,' which introduces hallucinations, while also avoiding 'under-sampling,' which leads to repetitive, robotic output. Monitoring the output and adjusting these values iteratively based on the specific use case—such as factual summarization versus narrative generation—is the hallmark of effective model deployment. Always validate these settings with representative test datasets to ensure the generated quality aligns with your specific product requirements.
def generate_token(logits, t=1.0, p=0.9):
# Apply temperature then truncate via Top-P
probs = apply_temperature(logits, t)
# Implementation-specific: sample from the filtered distribution
return torch.multinomial(probs, num_samples=1)
print(f"Sampled token index: {generate_token(logits, 0.7, 0.9).item()}")Key points
- Temperature scales logits to modify the probability distribution of tokens before selection.
- Lower temperatures make models more deterministic, while higher temperatures encourage increased linguistic variance.
- Top-K sampling limits the model's choices to a fixed number of the most probable next tokens.
- Top-P sampling dynamically adjusts the selection pool based on the cumulative probability threshold.
- Nucleus sampling is generally preferred over Top-K because it adapts to the model's confidence level.
- Combining temperature and Top-P allows for a balance between creative exploration and semantic coherence.
- Excessive temperature can lead to irrelevant tokens, while overly restrictive sampling causes repetitive output.
- Choosing optimal sampling parameters requires iterative testing against the intended application use case.
Common mistakes
- Mistake: Thinking Temperature > 1 makes the model smarter. Why it's wrong: Temperature simply flattens the probability distribution, increasing randomness and the risk of hallucinations. Fix: Use Temperature < 1 for accuracy and > 1 only for creative exploration.
- Mistake: Assuming Top-k and Top-p are mutually exclusive. Why it's wrong: They can be applied simultaneously to prune the tail of the distribution, with Top-k limiting the set size and Top-p setting the cumulative probability threshold. Fix: Use them together to strictly control the vocabulary subset before sampling.
- Mistake: Believing low Temperature removes all randomness. Why it's wrong: Even with low Temperature, the model samples from the remaining probability distribution; it just favors the most likely tokens more heavily. Fix: Use Temperature near 0 for near-deterministic output.
- Mistake: Setting Top-p to 1.0. Why it's wrong: A value of 1.0 means no pruning occurs, potentially allowing the model to choose extremely low-probability, incoherent tokens. Fix: Use values like 0.9 or 0.95 to truncate the 'long tail' of unlikely tokens.
- Mistake: Confusing Top-k with Top-p in terms of reliability. Why it's wrong: Top-k is static and may include very unlikely tokens if the distribution is flat, whereas Top-p is dynamic based on the actual probability mass. Fix: Prefer Top-p for adaptive, context-aware token filtering.
Interview questions
What is the role of Temperature in the sampling process of a Generative AI model?
Temperature is a hyperparameter that controls the randomness of the model's output by scaling the logits before they are passed through a softmax function. When Temperature is set to a low value, such as 0.1, the probability distribution becomes highly peaked, making the model deterministic and prone to choosing the most likely next token. Conversely, a high Temperature, like 1.0 or above, flattens the distribution, giving lower-probability tokens a greater chance of being selected. This makes the generated text more diverse and creative, though it increases the risk of incoherence. Mathematically, it adjusts the softmax calculation: probability = exp(logit / T) / sum(exp(logit / T)), directly impacting how 'conservative' or 'adventurous' the model behaves during sequence generation.
What is Top-k sampling, and how does it limit the vocabulary size during generation?
Top-k sampling is a technique used to constrain the model's output by only considering the top 'k' most probable next tokens. During the inference step, the model calculates the probability distribution for the entire vocabulary, but it then truncates this list, setting the probability of any token outside the top 'k' to zero. The remaining tokens are then re-normalized to sum to one. This approach is highly effective at preventing the model from picking extremely unlikely or nonsensical tokens that reside in the 'long tail' of the distribution. By choosing a specific integer for 'k', practitioners can ensure that the model remains focused on relevant, high-probability candidates while still maintaining a controlled level of stochasticity in the generated text.
Can you explain the mechanism behind Top-p, or Nucleus sampling?
Top-p sampling, often called Nucleus sampling, is a dynamic approach that selects the smallest set of tokens whose cumulative probability exceeds a threshold value 'p'. Instead of picking a fixed number of tokens like in Top-k, the model sorts the entire vocabulary by probability and adds tokens to the candidate pool until the sum of their probabilities reaches 'p'—for example, 0.9. This method is considered superior to Top-k in many scenarios because it allows the model to be more flexible: if the model is confident in its next word, the candidate pool will be small; if it is uncertain, the pool will automatically expand. This ensures quality by cutting off the unreliable long-tail tokens regardless of the distribution's shape.
Compare and contrast Top-k sampling and Top-p sampling. Which would you prefer to use and why?
The primary difference lies in how they manage the vocabulary pool. Top-k is rigid; it always selects from the same number of candidates, regardless of whether the model is highly certain or completely confused. This can lead to 'bad' selections if the top-k tokens are all low-probability, or it can be too restrictive if the top tokens are all equally high-probability. Top-p is more fluid because it adapts the candidate pool size based on the model's confidence. In practice, I generally prefer Top-p because it is more robust to varying levels of model certainty. Using a dynamic threshold allows the model to produce high-quality text while gracefully handling situations where multiple words are equally valid, providing a more balanced output quality across diverse prompts.
How does the interaction between Temperature and Top-p affect the final token selection?
These parameters function in a specific sequence during the generation pipeline. First, the Temperature is applied to the raw logits to 'sharpen' or 'soften' the probability distribution. This scaling shifts the probabilities, making some tokens significantly more likely than others before any filtering occurs. Following this, the Top-p filter is applied to the resulting distribution to truncate the long tail. By combining them, you gain two layers of control: Temperature modulates the 'personality' and overall confidence level of the model's prediction distribution, while Top-p provides a final safety net to discard unlikely tokens that might have been amplified or sustained by the Temperature setting. Adjusting both allows fine-tuned control over both the local coherence and the global creativity of the generated output.
Explain the concept of 'Logit Bias' and how it functions as a constraint on top-k or top-p sampling.
Logit bias is an additive parameter that allows users to manually influence the likelihood of specific tokens before the sampling phase. By providing a dictionary of token IDs and their corresponding bias values, you can force the model to favor or avoid certain words entirely. For example, adding a large positive bias to a specific token will ensure it appears in the top-k or top-p candidates, effectively overriding the model's learned probabilities. This is technically executed by adding the bias vector to the logits: adjusted_logits = logits + bias. This technique is often used in production systems to enforce content safety filters or specific naming conventions, acting as a final heuristic constraint that interacts with the statistical sampling strategies to shape the model's behavior according to business logic or strict task requirements.
Check yourself
1. What occurs when the Temperature parameter is set to a very high value like 2.0?
- A.The model becomes more focused on logical consistency.
- B.The probability distribution becomes flatter, increasing diversity and potential randomness.
- C.The model ignores the Top-p constraint entirely.
- D.The model output becomes identical every time it is run.
Show answer
B. The probability distribution becomes flatter, increasing diversity and potential randomness.
High temperature flattens the distribution by making lower-probability tokens more likely, increasing diversity. It does not ensure logical consistency (option 0), it does not bypass sampling constraints (option 2), and it creates more randomness, not determinism (option 3).
2. If you set Top-k to 1, what is the impact on the model's output generation?
- A.It generates a completely random sequence.
- B.It acts like a greedy search, always picking the single most likely next token.
- C.It selects from the top 10% of probability mass.
- D.It allows the model to choose any token in the vocabulary.
Show answer
B. It acts like a greedy search, always picking the single most likely next token.
Setting Top-k to 1 forces the model to choose the highest probability token at every step, effectively performing greedy search. It does not add randomness (option 0), does not relate to percentage mass (option 2), and restricts the choice to one token, not the whole vocabulary (option 3).
3. How does Top-p (Nucleus Sampling) behave differently than Top-k?
- A.Top-p uses a fixed number of tokens regardless of the distribution shape.
- B.Top-p dynamically adjusts the number of tokens to include based on their cumulative probability mass.
- C.Top-p is only effective when Temperature is set to zero.
- D.Top-p requires significantly more computational power than Top-k.
Show answer
B. Top-p dynamically adjusts the number of tokens to include based on their cumulative probability mass.
Top-p is dynamic because it cuts off the token list once the sum of probabilities reaches a threshold (p). Top-k is fixed (option 0), Top-p works with any temperature (option 2), and the computational difference is negligible (option 3).
4. In a scenario where the model produces highly repetitive text, which adjustment is most likely to help?
- A.Decrease the Temperature parameter to 0.
- B.Increase the Temperature parameter to make the model more creative.
- C.Reduce the Top-p value significantly.
- D.Set Top-k to 1 to force specific word choices.
Show answer
B. Increase the Temperature parameter to make the model more creative.
Increasing Temperature adds randomness, which breaks patterns. Decreasing it (option 0) or setting Top-k to 1 (option 3) makes the model more deterministic and likely to repeat. Reducing Top-p (option 2) further restricts the vocabulary, worsening repetition.
5. Why would a developer choose to use both Top-k and Top-p together?
- A.To ensure the model never uses rare vocabulary.
- B.To provide a hard cutoff on the number of candidates (k) while maintaining adaptive filtering (p).
- C.To increase the speed of token generation exponentially.
- D.To force the model to pick only the single most likely token.
Show answer
B. To provide a hard cutoff on the number of candidates (k) while maintaining adaptive filtering (p).
Using both allows for safety (K limits the absolute number of risky tokens) and quality (P ensures we don't pick from the long tail). It doesn't guarantee vocabulary exclusion (option 0), doesn't significantly change speed (option 2), and wouldn't force a single choice unless K and P were both set to extremely restrictive values (option 3).