Core Concepts of Large Language Models
Common LLM Architectures: GPT, BERT, T5, and Their Variants
This lesson explores the foundational architectures behind modern large language models (LLMs), explaining how their design choices enable different capabilities. Understanding these architectures is crucial because it allows you to select or adapt the right model for your specific task, whether it's text generation, comprehension, or translation. You’ll reach for this knowledge when designing, fine-tuning, or evaluating LLMs for real-world applications.
The Transformer: The Backbone of Modern LLMs
At the heart of every major LLM architecture lies the Transformer, introduced in the 2017 paper 'Attention Is All You Need.' The Transformer’s key innovation is its reliance on self-attention mechanisms, which allow the model to weigh the importance of different words in a sequence relative to each other, regardless of their position. This is a departure from earlier architectures like RNNs or LSTMs, which processed sequences step-by-step and struggled with long-range dependencies. Self-attention enables parallelization during training, making it feasible to train on massive datasets efficiently. The Transformer consists of an encoder and a decoder, though many modern architectures use only one of these components. The encoder processes input sequences into contextual representations, while the decoder generates output sequences. This modularity is why the Transformer can be adapted for tasks like text generation (decoder-only), understanding (encoder-only), or translation (encoder-decoder). The self-attention mechanism works by computing three vectors for each word: a query, a key, and a value. The query of one word interacts with the keys of all other words to produce attention scores, which are then used to weight the values. This process is repeated in multiple 'heads' (multi-head attention), allowing the model to focus on different aspects of the input simultaneously.
# Example of scaled dot-product attention, the core of self-attention
import torch
import torch.nn.functional as F
def scaled_dot_product_attention(query, key, value, mask=None):
# query, key, value shapes: (batch_size, num_heads, seq_len, head_dim)
matmul_qk = torch.matmul(query, key.transpose(-2, -1)) # (..., seq_len, seq_len)
# Scale the attention scores to prevent large values from dominating softmax
dk = key.size(-1) # Dimension of key
scaled_attention_logits = matmul_qk / torch.sqrt(torch.tensor(dk, dtype=torch.float32))
# Apply mask (if provided) to prevent attention to future tokens in decoder
if mask is not None:
scaled_attention_logits += (mask * -1e9) # Large negative values become ~0 after softmax
# Softmax to get attention weights
attention_weights = F.softmax(scaled_attention_logits, dim=-1)
# Multiply weights by values to get output
output = torch.matmul(attention_weights, value) # (..., seq_len, head_dim)
return output, attention_weights
# Example usage
batch_size, num_heads, seq_len, head_dim = 2, 3, 5, 64
query = torch.rand(batch_size, num_heads, seq_len, head_dim)
key = torch.rand(batch_size, num_heads, seq_len, head_dim)
value = torch.rand(batch_size, num_heads, seq_len, head_dim)
output, weights = scaled_dot_product_attention(query, key, value)
print("Output shape:", output.shape) # Should be (2, 3, 5, 64)GPT: The Decoder-Only Architecture for Text Generation
GPT (Generative Pre-trained Transformer) is a decoder-only architecture, meaning it uses only the decoder part of the original Transformer. This design choice makes GPT particularly well-suited for text generation tasks, where the model predicts the next token in a sequence based on the previous tokens. The decoder-only approach works because it is inherently autoregressive: each token is generated one at a time, conditioned on the tokens that came before it. This aligns perfectly with the nature of language, where the meaning of a word often depends on the context provided by preceding words. GPT’s architecture stacks multiple decoder layers, each consisting of masked self-attention and feed-forward neural networks. The masking in self-attention ensures that the model cannot 'peek' at future tokens during training, which is critical for maintaining the autoregressive property. During inference, the model generates text by sampling one token at a time, feeding it back into the model as input for the next step. This process continues until a stopping condition is met, such as reaching a maximum length or generating an end-of-sequence token. GPT’s strength lies in its simplicity and scalability; by pre-training on vast amounts of text data, it learns rich linguistic patterns that can be fine-tuned for specific tasks. However, its unidirectional nature (processing text left-to-right) limits its ability to understand context bidirectionally, which is where architectures like BERT excel.
# Simplified GPT-style decoder layer with masked self-attention
import torch.nn as nn
class GPTDecoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout)
self.linear1 = nn.Linear(embed_dim, ff_dim)
self.linear2 = nn.Linear(ff_dim, embed_dim)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.dropout = nn.Dropout(dropout)
self.activation = nn.GELU()
def forward(self, x, mask=None):
# Self-attention with causal masking (no peeking at future tokens)
attn_output, _ = self.self_attn(x, x, x, attn_mask=mask, need_weights=False)
x = x + self.dropout(attn_output)
x = self.norm1(x)
# Feed-forward network
ff_output = self.linear2(self.dropout(self.activation(self.linear1(x))))
x = x + self.dropout(ff_output)
x = self.norm2(x)
return x
# Example usage
embed_dim, num_heads, ff_dim = 512, 8, 2048
decoder_layer = GPTDecoderLayer(embed_dim, num_heads, ff_dim)
input_tensor = torch.rand(10, 32, embed_dim) # (seq_len, batch_size, embed_dim)
# Create a causal mask to prevent attention to future tokens
seq_len = input_tensor.size(0)
causal_mask = torch.triu(torch.ones(seq_len, seq_len) * float('-inf'), diagonal=1)
output = decoder_layer(input_tensor, mask=causal_mask)
print("Output shape:", output.shape) # Should be (10, 32, 512)BERT: The Encoder-Only Architecture for Bidirectional Understanding
BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only architecture, meaning it uses only the encoder part of the Transformer. Unlike GPT, which processes text unidirectionally, BERT reads the entire input sequence at once, allowing it to capture context from both left and right of each word. This bidirectional understanding makes BERT exceptionally powerful for tasks that require deep comprehension of text, such as question answering, sentiment analysis, or named entity recognition. BERT’s key innovation is its pre-training objective: masked language modeling (MLM). During pre-training, random tokens in the input sequence are masked, and the model is trained to predict the original tokens based on the surrounding context. This forces the model to learn rich, bidirectional representations of language. Additionally, BERT uses a next-sentence prediction (NSP) task during pre-training, where it learns to determine whether two sentences follow each other in a document. While NSP has been somewhat deprecated in later variants, MLM remains central to BERT’s success. The encoder-only design means BERT is not inherently generative; it excels at understanding and classifying text rather than generating it. This makes it ideal for tasks where the output is a label, span, or classification rather than free-form text. BERT’s architecture stacks multiple encoder layers, each with self-attention and feed-forward networks, allowing it to build hierarchical representations of the input. Fine-tuning BERT for specific tasks typically involves adding a simple classification head on top of the pre-trained encoder, making it highly adaptable.
# Simplified BERT-style encoder layer with masked language modeling (MLM)
import torch
import torch.nn as nn
class BERTEncoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout)
self.linear1 = nn.Linear(embed_dim, ff_dim)
self.linear2 = nn.Linear(ff_dim, embed_dim)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.dropout = nn.Dropout(dropout)
self.activation = nn.GELU()
def forward(self, x):
# Self-attention (bidirectional, no masking)
attn_output, _ = self.self_attn(x, x, x, need_weights=False)
x = x + self.dropout(attn_output)
x = self.norm1(x)
# Feed-forward network
ff_output = self.linear2(self.dropout(self.activation(self.linear1(x))))
x = x + self.dropout(ff_output)
x = self.norm2(x)
return x
# Example of masked language modeling (MLM) pre-training
class BERTMLMHead(nn.Module):
def __init__(self, embed_dim, vocab_size):
super().__init__()
self.linear = nn.Linear(embed_dim, vocab_size)
def forward(self, x):
return self.linear(x)
# Example usage
embed_dim, num_heads, ff_dim, vocab_size = 512, 8, 2048, 30000
encoder_layer = BERTEncoderLayer(embed_dim, num_heads, ff_dim)
mlm_head = BERTMLMHead(embed_dim, vocab_size)
input_tensor = torch.rand(10, 32, embed_dim) # (seq_len, batch_size, embed_dim)
encoder_output = encoder_layer(input_tensor)
# Simulate masking: replace some tokens with [MASK] and predict them
masked_positions = torch.tensor([2, 5, 7]) # Positions to predict
mlm_logits = mlm_head(encoder_output[masked_positions])
print("MLM logits shape:", mlm_logits.shape) # Should be (3, 30000)T5: The Encoder-Decoder Architecture for Unified Text-to-Text Tasks
T5 (Text-to-Text Transfer Transformer) is an encoder-decoder architecture that frames all NLP tasks as text-to-text problems. This unified approach means that whether the task is translation, summarization, or question answering, the model receives text as input and generates text as output. The encoder processes the input sequence into a contextual representation, which the decoder then uses to generate the output sequence autoregressively. This design allows T5 to handle a wide variety of tasks without architectural changes, simply by prepending a task-specific prefix to the input (e.g., 'translate English to French:'). T5’s strength lies in its flexibility and scalability; by pre-training on a massive corpus using a denoising objective (similar to BERT’s MLM but applied to the entire sequence), it learns robust representations that transfer well to downstream tasks. The encoder-decoder architecture is particularly well-suited for tasks that require mapping one sequence to another, such as translation or summarization, where the input and output lengths may differ. During pre-training, T5 uses a span corruption objective, where random spans of text are replaced with sentinel tokens, and the model is trained to reconstruct the original text. This encourages the model to learn both local and global dependencies in the data. T5’s architecture stacks identical layers in both the encoder and decoder, with cross-attention in the decoder allowing it to attend to the encoder’s output. This makes T5 highly effective for tasks that require understanding and generation in tandem, though it may be overkill for simpler classification tasks where encoder-only models like BERT suffice.
# Simplified T5-style encoder-decoder architecture
import torch.nn as nn
class T5EncoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout)
self.linear1 = nn.Linear(embed_dim, ff_dim)
self.linear2 = nn.Linear(ff_dim, embed_dim)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.dropout = nn.Dropout(dropout)
self.activation = nn.GELU()
def forward(self, x):
attn_output, _ = self.self_attn(x, x, x, need_weights=False)
x = x + self.dropout(attn_output)
x = self.norm1(x)
ff_output = self.linear2(self.dropout(self.activation(self.linear1(x))))
x = x + self.dropout(ff_output)
x = self.norm2(x)
return x
class T5DecoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout)
self.cross_attn = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout)
self.linear1 = nn.Linear(embed_dim, ff_dim)
self.linear2 = nn.Linear(ff_dim, embed_dim)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.norm3 = nn.LayerNorm(embed_dim)
self.dropout = nn.Dropout(dropout)
self.activation = nn.GELU()
def forward(self, x, enc_output, self_mask=None, cross_mask=None):
# Self-attention with causal masking
attn_output, _ = self.self_attn(x, x, x, attn_mask=self_mask, need_weights=False)
x = x + self.dropout(attn_output)
x = self.norm1(x)
# Cross-attention to encoder output
cross_attn_output, _ = self.cross_attn(x, enc_output, enc_output, attn_mask=cross_mask, need_weights=False)
x = x + self.dropout(cross_attn_output)
x = self.norm2(x)
# Feed-forward network
ff_output = self.linear2(self.dropout(self.activation(self.linear1(x))))
x = x + self.dropout(ff_output)
x = self.norm3(x)
return x
# Example usage
embed_dim, num_heads, ff_dim = 512, 8, 2048
encoder_layer = T5EncoderLayer(embed_dim, num_heads, ff_dim)
decoder_layer = T5DecoderLayer(embed_dim, num_heads, ff_dim)
enc_input = torch.rand(10, 32, embed_dim) # (seq_len, batch_size, embed_dim)
dec_input = torch.rand(8, 32, embed_dim) # Shorter sequence for decoder
enc_output = encoder_layer(enc_input)
# Create causal mask for decoder self-attention
seq_len = dec_input.size(0)
causal_mask = torch.triu(torch.ones(seq_len, seq_len) * float('-inf'), diagonal=1)
dec_output = decoder_layer(dec_input, enc_output, self_mask=causal_mask)
print("Decoder output shape:", dec_output.shape) # Should be (8, 32, 512)Variants and Trade-offs: Choosing the Right Architecture for Your Task
When selecting an LLM architecture, the choice between decoder-only (GPT), encoder-only (BERT), or encoder-decoder (T5) depends on the nature of your task and the trade-offs you’re willing to make. Decoder-only models like GPT excel at generative tasks where the output is a continuation of the input, such as text completion or creative writing. Their unidirectional nature makes them fast and efficient for inference, but they struggle with tasks requiring deep bidirectional understanding, like sentiment analysis or named entity recognition. Encoder-only models like BERT are ideal for tasks where the input needs to be deeply understood before producing a fixed output, such as classification or span prediction. Their bidirectional attention allows them to capture context from both directions, but they cannot generate free-form text. Encoder-decoder models like T5 offer the most flexibility, handling tasks that require both understanding and generation, such as translation or summarization. However, they are more complex and computationally expensive to train and deploy. Variants of these architectures often address specific limitations: for example, RoBERTa improves upon BERT by removing the next-sentence prediction task and training on more data, while GPT-3 scales up GPT-2 with more parameters and data. T5’s successors, like FLAN-T5, focus on instruction tuning to improve zero-shot performance. The key to choosing the right architecture lies in understanding the task’s requirements: if it’s generative, lean toward decoder-only; if it’s understanding-heavy, encoder-only; and if it’s a mix, encoder-decoder. Additionally, consider the computational resources available, as larger models require more memory and processing power. Fine-tuning a pre-trained model is often more practical than training from scratch, so leverage existing architectures and adapt them to your needs.
# Example of task-specific fine-tuning heads for different architectures
import torch.nn as nn
# GPT-style fine-tuning for text generation (add a language modeling head)
class GPTLMHead(nn.Module):
def __init__(self, embed_dim, vocab_size):
super().__init__()
self.linear = nn.Linear(embed_dim, vocab_size)
def forward(self, x):
return self.linear(x)
# BERT-style fine-tuning for classification (add a classification head)
class BERTClassificationHead(nn.Module):
def __init__(self, embed_dim, num_classes):
super().__init__()
self.linear = nn.Linear(embed_dim, num_classes)
def forward(self, x):
# Use the [CLS] token representation for classification
cls_token = x[0, :, :] # First token in sequence
return self.linear(cls_token)
# T5-style fine-tuning for translation (no additional head needed)
class T5TranslationModel:
def __init__(self, encoder, decoder, embed_dim, vocab_size):
self.encoder = encoder
self.decoder = decoder
self.lm_head = nn.Linear(embed_dim, vocab_size)
def forward(self, enc_input, dec_input):
enc_output = self.encoder(enc_input)
dec_output = self.decoder(dec_input, enc_output)
return self.lm_head(dec_output)
# Example usage
embed_dim, vocab_size, num_classes = 512, 30000, 2
gpt_lm_head = GPTLMHead(embed_dim, vocab_size)
bert_cls_head = BERTClassificationHead(embed_dim, num_classes)
# Simulate outputs from pre-trained models
gpt_output = torch.rand(10, 32, embed_dim) # GPT output
bert_output = torch.rand(10, 32, embed_dim) # BERT output
lm_logits = gpt_lm_head(gpt_output)
cls_logits = bert_cls_head(bert_output)
print("GPT LM logits shape:", lm_logits.shape) # (10, 32, 30000)
print("BERT classification logits shape:", cls_logits.shape) # (32, 2)Key points
- The Transformer architecture relies on self-attention to weigh the importance of words in a sequence relative to each other, enabling parallel processing and long-range dependency modeling.
- GPT uses a decoder-only architecture with masked self-attention to generate text autoregressively, making it ideal for tasks like text completion but limiting its bidirectional understanding.
- BERT employs an encoder-only architecture with bidirectional self-attention, excelling at tasks requiring deep comprehension of text but lacking generative capabilities.
- T5 frames all tasks as text-to-text problems using an encoder-decoder architecture, offering flexibility for tasks like translation or summarization but at higher computational cost.
- Decoder-only models are best suited for generative tasks, encoder-only models for understanding tasks, and encoder-decoder models for tasks requiring both understanding and generation.
- Variants like RoBERTa and GPT-3 improve upon their base architectures by scaling data, removing less effective pre-training tasks, or increasing model size for better performance.
- Fine-tuning pre-trained models is often more practical than training from scratch, as it leverages existing knowledge and reduces the need for large datasets.
- Choosing the right architecture depends on the task requirements, computational resources, and whether the focus is on generation, understanding, or a combination of both.
Common mistakes
- Mistake: Assuming GPT, BERT, and T5 can be used interchangeably for any NLP task. Why it's wrong: Each architecture is optimized for different tasks—GPT for generative tasks, BERT for bidirectional understanding, and T5 for text-to-text transformations. Fix: Match the architecture to the task (e.g., use BERT for classification, GPT for text generation).
- Mistake: Ignoring tokenization differences between architectures. Why it's wrong: GPT uses byte-pair encoding (BPE), BERT uses WordPiece, and T5 uses SentencePiece. Mixing them can cause input/output mismatches. Fix: Preprocess data with the correct tokenizer for your chosen architecture.
- Mistake: Overlooking positional embeddings in transformer-based models. Why it's wrong: Positional embeddings are critical for transformers to understand sequence order. Skipping them or using incorrect implementations breaks model performance. Fix: Ensure positional embeddings are correctly added to input embeddings.
- Mistake: Fine-tuning all layers of a pre-trained model without justification. Why it's wrong: Full fine-tuning can lead to catastrophic forgetting or overfitting, especially with small datasets. Fix: Use layer-wise learning rate decay or freeze early layers during fine-tuning.
- Mistake: Treating T5’s text-to-text framework as a drop-in replacement for GPT or BERT without adjusting prompts. Why it's wrong: T5 requires task-specific prefixes (e.g., 'translate English to French:') to function correctly. Fix: Design prompts carefully to align with T5’s input-output format.
Interview questions
What is the main architectural difference between GPT and BERT, and why does this difference matter when building your own LLM?
The key difference is that GPT uses a decoder-only transformer architecture, while BERT uses an encoder-only design. GPT is autoregressive—it generates text one token at a time, predicting the next word based on previous ones, which makes it great for tasks like text generation or chatbots. BERT, on the other hand, is bidirectional; it reads the entire input sequence at once to understand context from both left and right, which is ideal for tasks like classification or question answering where full context matters. When building your own LLM, choosing between these depends on your goal: if you need generation, go with a decoder like GPT; if you need understanding, an encoder like BERT is better. For example, in code, a GPT-style model uses masked self-attention to prevent future tokens from being seen, while BERT uses full self-attention across the whole input.
Why is T5 often called a 'text-to-text' model, and how does this simplify training your own LLM?
T5 treats every NLP task as a text-to-text problem, meaning it converts inputs and outputs into plain text strings. For example, instead of having separate heads for translation, summarization, or classification, T5 frames everything as 'input text → output text.' This simplifies training because you only need one architecture and one loss function (cross-entropy on token predictions). When building your own LLM, this uniformity reduces complexity—you don’t need task-specific layers or fine-tuning pipelines. For instance, to train a summarizer, you’d prepend 'summarize: ' to the input and let the model generate the summary. This approach also makes it easier to mix tasks during training, improving generalization.
When building your own LLM, why might you choose a smaller variant like DistilGPT or TinyBERT instead of the full model?
Smaller variants like DistilGPT or TinyBERT are designed to retain most of the performance of their larger counterparts while being faster and more efficient. DistilGPT, for example, uses knowledge distillation to train a smaller model to mimic the behavior of a larger GPT, reducing parameters by 50% with minimal accuracy loss. TinyBERT goes further by distilling both the attention and embedding layers. When building your own LLM, these variants are ideal for edge devices or low-latency applications where computational resources are limited. For instance, if you’re deploying a chatbot on a mobile app, TinyBERT could run locally without needing cloud inference, saving costs and improving responsiveness.
How does the attention mechanism in transformers enable LLMs to handle long-range dependencies, and why is this important for your own LLM?
The attention mechanism in transformers allows the model to weigh the importance of every token in the input sequence when generating each output token. Unlike RNNs, which process tokens sequentially and struggle with long-range dependencies, transformers use self-attention to directly connect distant tokens. For example, in the sentence 'The cat, which was black, sat on the mat,' the model can link 'cat' and 'sat' even though they’re far apart. This is critical for your own LLM because it enables coherent, context-aware outputs. In code, the attention scores are computed as softmax(QK^T / sqrt(d_k)), where Q and K are query and key matrices. Without this, your model would lose track of earlier context, leading to nonsensical or repetitive generations.
Compare the fine-tuning approaches for GPT and BERT when adapting them for a custom task like sentiment analysis. Which would you choose and why?
For GPT, fine-tuning involves continuing the autoregressive training on your task-specific data, often by adding a classification head or prompting the model with task-specific prefixes. For example, you might prepend 'Sentiment: ' to input text and train the model to generate 'positive' or 'negative.' BERT, however, is typically fine-tuned by adding a task-specific layer (like a linear classifier) on top of the [CLS] token’s output, using the entire input sequence’s embeddings. I’d choose BERT for sentiment analysis because its bidirectional context captures nuanced sentiment signals better than GPT’s left-to-right generation. For instance, BERT can detect sarcasm in phrases like 'Great, another delay' by seeing the full context, while GPT might misclassify it due to its sequential nature. However, if you need generation alongside classification, GPT’s flexibility might be preferable.
Explain how you would design a custom LLM architecture that combines the strengths of GPT and BERT for a task like document summarization with question answering. Include key components and training strategies.
To combine GPT and BERT for document summarization with question answering, I’d design a hybrid encoder-decoder architecture inspired by T5 but with task-specific adaptations. The encoder would use BERT-style bidirectional attention to fully understand the input document and question, while the decoder would use GPT-style autoregressive attention to generate summaries or answers. Key components would include: 1) A shared embedding layer to reduce parameters, 2) Cross-attention in the decoder to attend to encoder outputs, and 3) A task prefix like 'summarize: ' or 'answer: ' to guide generation. For training, I’d use a multi-task objective: pre-train the encoder on masked language modeling (like BERT) and the decoder on next-token prediction (like GPT), then fine-tune on a mix of summarization and QA datasets. For example, the loss function would combine cross-entropy for both tasks. This approach leverages BERT’s understanding for context and GPT’s generation for fluent outputs, making it ideal for complex tasks requiring both comprehension and production.
Check yourself
1. When building a custom LLM for summarization, which architecture is most suitable and why?
- A.GPT, because it excels at generating coherent long-form text from prompts.
- B.BERT, because its bidirectional context helps understand the entire input before summarizing.
- C.T5, because it treats summarization as a text-to-text task with a clear input-output format.
- D.A hybrid of GPT and BERT, because combining their strengths improves performance.
Show answer
C. T5, because it treats summarization as a text-to-text task with a clear input-output format.
The correct answer is T5 because it is explicitly designed for text-to-text tasks like summarization, where the input (a long document) and output (a summary) are both treated as text. GPT is better for open-ended generation, BERT lacks generative capabilities, and hybrids are unnecessarily complex for this task.
2. You are fine-tuning a pre-trained BERT model for a classification task with a small dataset. What is the best practice to avoid overfitting?
- A.Train all layers with a high learning rate to adapt quickly to the new data.
- B.Freeze the early layers and only fine-tune the later layers with a low learning rate.
- C.Replace BERT’s tokenizer with a custom one to better fit the dataset.
- D.Use a larger batch size to stabilize training, regardless of learning rate.
Show answer
B. Freeze the early layers and only fine-tune the later layers with a low learning rate.
The correct answer is to freeze early layers and fine-tune later layers with a low learning rate. Early layers capture general language features, while later layers specialize in task-specific patterns. Freezing prevents overfitting to small datasets. High learning rates or custom tokenizers are unnecessary, and batch size alone doesn’t address overfitting.
3. Why does GPT use a unidirectional attention mechanism, while BERT uses bidirectional attention?
- A.GPT’s unidirectional attention is simpler and faster to train, while BERT’s bidirectional attention is more accurate but slower.
- B.GPT is designed for generative tasks where future tokens must not influence past tokens, while BERT is optimized for tasks requiring full context understanding.
- C.BERT’s bidirectional attention is a legacy design, and modern architectures like GPT have moved to unidirectional for better performance.
- D.GPT’s unidirectional attention allows it to handle longer sequences, while BERT’s bidirectional attention is limited to shorter inputs.
Show answer
B. GPT is designed for generative tasks where future tokens must not influence past tokens, while BERT is optimized for tasks requiring full context understanding.
The correct answer is that GPT’s unidirectional attention prevents future tokens from influencing past tokens (critical for generation), while BERT’s bidirectional attention captures full context (critical for tasks like classification). The other options misrepresent the trade-offs or are factually incorrect.
4. You are implementing a custom LLM for a translation task. Which architecture and input format should you use?
- A.GPT with a prompt like 'Translate this sentence: [text]'.
- B.BERT with a prompt like '[text] in [target language]'.
- C.T5 with a prefix like 'translate English to French: [text]'.
- D.A transformer with no positional embeddings, as translation doesn’t require sequence order.
Show answer
C. T5 with a prefix like 'translate English to French: [text]'.
The correct answer is T5 with a prefix like 'translate English to French: [text]' because T5 is designed for text-to-text tasks and explicitly uses task prefixes. GPT can generate translations but lacks T5’s structured input-output format. BERT is not generative, and omitting positional embeddings would break the model.
5. During fine-tuning, your custom LLM’s loss is oscillating wildly. What is the most likely cause and fix?
- A.The learning rate is too low; increase it to speed up convergence.
- B.The batch size is too small; increase it to stabilize gradients.
- C.The positional embeddings are missing or incorrectly implemented; verify their addition to input embeddings.
- D.The tokenizer’s vocabulary is mismatched with the pre-trained model; switch to the correct tokenizer.
Show answer
B. The batch size is too small; increase it to stabilize gradients.
The correct answer is that the batch size is too small, leading to noisy gradients. Increasing batch size stabilizes training. A low learning rate would cause slow convergence, not oscillation. Missing positional embeddings or tokenizer mismatches would cause consistent errors, not oscillations.