Interview Prep
GenAI Interview Questions — Fundamentals
This lesson establishes the foundational principles of generative modeling, covering the transition from discriminative to probabilistic generation. It is essential for understanding how models learn data distributions to create novel, coherent outputs rather than just classifying inputs. Use these fundamentals to architect robust systems that go beyond basic prompt engineering.
Understanding Probabilistic Generation
At its core, generative AI aims to model the probability distribution of a dataset, p(x), rather than just p(y|x) as discriminative models do. In a classification task, a model learns a boundary; in generative AI, the model learns the density of the feature space itself. This is critical because it allows the model to sample from that learned density to generate entirely new data points that share the statistical properties of the training set. If a model only understands boundaries, it cannot synthesize new artifacts. By learning the latent structure of the input space—such as the relationships between pixels in an image or tokens in a sentence—the model can interpolate between known data points. This mastery of the underlying manifold is why generative systems can produce coherent content rather than merely mimicking memorized patterns. Understanding this shift from classification to density estimation is the key to reasoning about how models hallucinate or fail to generalize.
# Simple Gaussian sampling to visualize probability density
import numpy as np
# Define a mean and variance representing our 'learned' distribution
mu, sigma = 0, 1
# Sample new data points from the learned latent distribution
samples = np.random.normal(mu, sigma, 1000)
print(f"Generated {len(samples)} unique data points from the distribution.")The Transformer Architecture: Self-Attention
The Transformer replaced sequential processing with self-attention, a mechanism that allows every token in a sequence to interact with every other token regardless of their relative distance. This is mathematically significant because it solves the vanishing gradient problem encountered by older recurrent architectures when handling long sequences. Through the dot-product attention mechanism, the model computes relevance weights, effectively creating a dynamic pathway for information to flow. This mechanism works by projecting inputs into Query, Key, and Value spaces. The query of one token matches against the keys of others to determine how much 'attention' or focus should be allocated to those segments. By processing the entire sequence in parallel, the model learns complex dependencies and contextual nuances that are critical for coherent text generation. When you ask why a model understands a pronoun reference across paragraphs, the answer lies in the global connectivity provided by these attention heads.
# Simplified self-attention mechanism implementation
import numpy as np
def softmax(x):
return np.exp(x) / np.sum(np.exp(x), axis=0)
# Hypothetical Query and Key vectors for two tokens
query = np.array([0.5, 0.2])
keys = np.array([[0.1, 0.4], [0.3, 0.9]])
# Compute attention scores via dot product
scores = np.dot(keys, query)
weights = softmax(scores)
print(f"Attention weights for sequence: {weights}")The Role of Latent Space Representation
Latent spaces are compressed, abstract representations of high-dimensional data that map meaningful features into a lower-dimensional coordinate system. In generative models, the goal is to organize this space such that semantically similar items are grouped together. This works because, through the training process, the model learns to discard noise and retain structural information. When we manipulate vectors in this space—for example, by adding a 'sentiment' vector to a sentence embedding—we can perform controlled generation. This is why models can maintain consistent styles or topics; the model is navigating a structured internal map rather than outputting random noise. If the latent space is well-regularized, movements along specific axes correspond to meaningful changes in the output, such as shifting the intensity of an image or the tone of a paragraph. Recognizing this geometric interpretation allows engineers to perform steerable generation by biasing the latent sampling process.
# Vector interpolation in latent space
import numpy as np
# Represents two points in a compressed latent space
vector_a = np.array([1.0, 0.5])
vector_b = np.array([0.0, 1.5])
# Linearly interpolate to generate a transition between styles
alpha = 0.5
interpolated = (1 - alpha) * vector_a + alpha * vector_b
print(f"Synthesized latent state: {interpolated}")Temperature and Stochasticity
Temperature is a hyperparameter that scales the logits before the final probability distribution is calculated during generation. Mathematically, it divides the logit values by a constant before applying a softmax function. A low temperature makes the model 'confident' and deterministic, causing it to aggressively pick the most likely tokens, while a high temperature flattens the distribution, increasing the probability of choosing less likely tokens. This matters because it allows for a trade-off between coherence and creativity. If a model lacks stochasticity, it may fall into repetitive loops, as it will always choose the same high-probability paths. Conversely, too much randomness destroys structure. By adjusting this parameter, we control the exploration of the probability space. Understanding this is vital for fine-tuning applications where you need to balance the need for factual accuracy with the requirement for varied, engaging output generation across different user prompts.
# Applying temperature to adjust randomness in selection
import numpy as np
def adjusted_softmax(logits, temp=1.0):
# Higher temperature flattens distribution
return np.exp(logits / temp) / np.sum(np.exp(logits / temp))
logits = np.array([2.0, 1.0, 0.1])
print(f"Low temp: {adjusted_softmax(logits, 0.1)}")
print(f"High temp: {adjusted_softmax(logits, 2.0)}")Context Windows and Memory Constraints
The context window represents the fixed number of tokens that a model can consider at once, dictated by the quadratic complexity of the self-attention mechanism. Because the computation for attention scales with the square of the sequence length, there is a hard limit on how much information a model can hold in its active memory. When a prompt exceeds this limit, the model effectively 'forgets' the earliest parts of the conversation unless specific memory management techniques are applied. This is why long-form document analysis requires techniques like sliding windows or external retrieval systems. The model cannot reason about what it cannot see. Therefore, efficient prompt management and summarization are not just optimization techniques; they are architectural necessities. To design successful systems, one must architect data flow to fit within these finite constraints, ensuring that the most relevant information is always present in the current attention window.
# Simulating a fixed context window limit
context_window_size = 512
input_tokens = ["token"] * 600 # Exceeds capacity
# Truncate to the most recent tokens to maintain coherence
active_context = input_tokens[-context_window_size:]
print(f"Retained {len(active_context)} tokens within the window.")Key points
- Generative models learn the underlying probability distribution of data rather than just mapping inputs to outputs.
- The Transformer architecture uses self-attention to identify long-range dependencies by weighting the importance of different tokens.
- A latent space acts as a compressed, structural map where semantic meaning is represented by geometric position.
- Temperature controls the trade-off between deterministic precision and probabilistic creativity during token sampling.
- The context window is a hard architectural constraint on the amount of information the model can process simultaneously.
- Discriminative models focus on classification boundaries, while generative models focus on the total data manifold.
- Manipulating vectors within a learned latent space enables precise control over the properties of generated output.
- Efficient system design requires managing memory constraints by curating inputs to fit within the active context window.
Common mistakes
- Mistake: Confusing Generative AI with Discriminative AI. Why it's wrong: They serve opposite functions; one creates new data, the other classifies existing data. Fix: Focus on whether the model is trained to identify patterns for categorization or to learn the probability distribution of data to produce new samples.
- Mistake: Assuming more parameters always leads to better performance. Why it's wrong: Larger models are prone to overfitting and excessive computational costs without necessarily improving reasoning. Fix: Evaluate performance based on architectural efficiency, training data quality, and context window utility rather than just parameter count.
- Mistake: Neglecting the importance of the training data distribution in model output. Why it's wrong: Models can only generate content based on the patterns they observed during training. Fix: Understand that data bias and data quality are the primary constraints on a model's 'hallucinations' and general output accuracy.
- Mistake: Thinking that prompting is equivalent to model retraining. Why it's wrong: Prompting optimizes the input to retrieve existing weights, while retraining modifies the weights themselves. Fix: Use Retrieval Augmented Generation (RAG) or fine-tuning for specific knowledge gaps rather than expecting long prompts to fundamentally change model logic.
- Mistake: Viewing 'Temperature' as a setting for accuracy. Why it's wrong: It controls randomness in token selection, not factuality. Fix: Describe temperature as a parameter for controlling output variance, where lower values yield deterministic, focused results and higher values increase diversity.
Interview questions
What is the fundamental difference between discriminative and generative AI models?
The fundamental difference lies in their objective function. A discriminative model focuses on learning the boundary between classes to predict a label for a given input, essentially modeling the conditional probability P(Y|X). In contrast, a generative model learns the actual distribution of the data itself, modeling the joint probability P(X, Y) or simply P(X). By understanding how the data is generated, a generative model can create new, synthetic instances that follow the same statistical properties as the training set. For example, while a discriminative model might classify an image as 'cat', a generative model can be prompted to synthesize an entirely new image of a cat from scratch.
How does the self-attention mechanism in Transformers enable the processing of long-range dependencies?
Self-attention allows a model to weigh the relevance of different words in a sequence regardless of their distance from each other. In older architectures like Recurrent Neural Networks, information had to pass through a hidden state step-by-step, leading to gradient vanishing. With self-attention, the model computes a similarity score between every word pair using Query, Key, and Value vectors. If we have a sentence like 'The animal didn't cross the street because it was too tired,' the model uses self-attention to link 'it' directly to 'animal' by calculating a high dot-product score between their respective representations. This parallel processing captures context globally, allowing the model to understand complex relationships in long text sequences efficiently.
Explain the concept of temperature in text generation and how it affects the output.
Temperature is a hyperparameter used during the sampling stage to control the randomness and creativity of the model's output. Mathematically, it scales the logits before applying the softmax function to the probability distribution. If the temperature is low, say 0.2, the probability distribution becomes 'sharper,' making the model highly confident and deterministic, often leading to repetitive text. If the temperature is high, say 0.8 or above, the distribution flattens, increasing the probability of selecting less likely tokens. This results in more diverse and creative outputs but also raises the risk of producing incoherent or hallucinatory content. Adjusting this allows users to balance precision versus novelty in their specific application.
Compare and contrast Fine-Tuning versus Retrieval-Augmented Generation (RAG) for improving model performance.
Fine-Tuning involves updating the internal weights of a pre-trained model on a specialized dataset to adapt its style or domain knowledge. It is excellent for changing the model's behavior but is computationally expensive and suffers from 'knowledge cutoff' issues. RAG, conversely, keeps the model weights frozen and instead retrieves relevant external documents to inject as context into the prompt at runtime. RAG is better for tasks requiring up-to-date factual accuracy and citations because the model acts as a reasoning engine over provided evidence. While Fine-Tuning is for long-term behavioral changes, RAG is the superior choice for dynamic, data-intensive applications where hallucinations must be minimized through grounded retrieval.
What are the core components and the training objective of a Variational Autoencoder (VAE)?
A VAE consists of an encoder, a decoder, and a latent space. The encoder maps input data to a probability distribution in the latent space (a mean and variance), while the decoder samples from this space to reconstruct the input. The training objective, or ELBO (Evidence Lower Bound), combines two terms: a reconstruction loss, which ensures the output matches the input, and the Kullback-Leibler (KL) divergence, which forces the latent space to follow a standard normal distribution. This regularization is vital because it prevents the model from simply memorizing inputs, ensuring the latent space is continuous and structured, which allows for smooth interpolation between generated samples.
How do Diffusion Models generate high-quality images, and how does the reverse diffusion process work?
Diffusion models generate data by learning to reverse a gradual noise-injection process. During forward diffusion, Gaussian noise is added to an image over several time steps until it becomes pure white noise. The model is trained to predict the noise added at each step, essentially learning the gradient of the data distribution. During inference, we start with random noise and iteratively 'denoise' it using the trained model to recover the structure. The reverse process is defined by $x_{t-1} = 1/sqrt(alpha_t) * (x_t - ((1-alpha_t)/sqrt(1-alpha_t_bar)) * noise_pred)$. By slowly removing noise guided by the model's predictions, we arrive at a coherent, high-fidelity image that adheres to the underlying patterns learned during training.
Check yourself
1. When a generative model produces highly coherent text that is factually incorrect, which fundamental limitation is likely being demonstrated?
- A.A lack of sufficient computational resources during the initial training phase.
- B.The model's tendency to prioritize statistical likelihood over factual verification.
- C.An error in the tokenization process causing semantic drift.
- D.The failure of the user to provide a negative constraint in the system prompt.
Show answer
B. The model's tendency to prioritize statistical likelihood over factual verification.
Generative models predict the most probable next token based on training patterns, not truth. Option 1 is incorrect because scale does not solve hallucination; Option 3 is a technical implementation detail; Option 4 is a band-aid, not a solution to the model's architecture.
2. What is the primary architectural difference between a standard encoder-decoder model and a decoder-only model in generative text tasks?
- A.Encoder-decoder models use significantly fewer parameters during inference.
- B.Decoder-only models are incapable of handling sequence-to-sequence translations.
- C.Decoder-only models process inputs and outputs through a unified self-attention mechanism without an explicit separate encoder block.
- D.Encoder-decoder models always utilize a higher temperature setting by default.
Show answer
C. Decoder-only models process inputs and outputs through a unified self-attention mechanism without an explicit separate encoder block.
Decoder-only architectures treat inputs and outputs as a single sequence, relying on autoregressive self-attention. Option 1 is generally false; Option 2 is incorrect as they excel at translation; Option 4 is unrelated to architectural design.
3. In the context of RAG, why is the retrieval of relevant context prioritized over simply fine-tuning the model on private data?
- A.Fine-tuning is mathematically impossible for models with more than 10 billion parameters.
- B.Retrieval allows the model to ground responses in verifiable sources and reduces the need for frequent retraining.
- C.Fine-tuning always results in catastrophic forgetting of all pre-trained language capabilities.
- D.Retrieval is the only method that enables the model to understand domain-specific terminology.
Show answer
B. Retrieval allows the model to ground responses in verifiable sources and reduces the need for frequent retraining.
RAG provides dynamic access to external data, mitigating hallucinations. Option 1 and 4 are false, and while fine-tuning can cause 'catastrophic forgetting,' it is not an absolute rule as suggested in option 3.
4. How does increasing the 'Temperature' parameter affect the probability distribution of generated tokens?
- A.It flattens the distribution, making less likely tokens more probable to be chosen.
- B.It narrows the distribution, forcing the model to select only the most probable token.
- C.It forces the model to ignore the initial input prompt to increase output diversity.
- D.It increases the model's internal memory capacity during inference.
Show answer
A. It flattens the distribution, making less likely tokens more probable to be chosen.
Higher temperature increases the entropy of the softmax function, making the selection process more random. Option 1 is correct. Option 2 describes lowering temperature, while 3 and 4 are false.
5. Why is the concept of 'alignment' critical in modern generative AI development?
- A.Alignment is used to increase the model's processing speed by pruning unnecessary weights.
- B.Alignment ensures that the model's objective function is calibrated to human intent and safety standards.
- C.Alignment is a technique specifically for converting text generation into image generation.
- D.Alignment replaces the need for a training dataset by using reinforcement learning alone.
Show answer
B. Alignment ensures that the model's objective function is calibrated to human intent and safety standards.
Alignment techniques (like RLHF) bridge the gap between pure statistical prediction and human utility. Option 1 is pruning, 3 is multimodal conversion, and 4 is impossible as models need initial training data to have a foundation.