Foundations of Machine Learning and NLP
Key Papers in NLP: Attention Is All You Need, BERT, and GPT
This lesson covers three foundational papers in modern NLP that revolutionized how models understand and generate language. Understanding these papers is critical because they introduce the core architectures (Transformer, BERT, GPT) that power today's large language models, and grasping their design choices helps you debug, optimize, or even build your own LLM. You should reach for this material when you need to move beyond using pre-trained models and want to design or adapt architectures for specific tasks or constraints.
The Problem: Why We Needed Attention
Before the Transformer, most NLP models relied on recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to process sequences. RNNs, like LSTMs, process words one at a time, passing a hidden state from one step to the next. This sequential nature makes them slow to train, especially for long sequences, because you can't parallelize the computation. More critically, RNNs struggle to capture long-range dependencies—relationships between words far apart in a sentence—because the hidden state degrades as it propagates. CNNs, while faster, require stacking many layers to capture these dependencies, which is inefficient. The core insight of 'Attention Is All You Need' was that we could abandon recurrence entirely and instead use a mechanism that directly models relationships between all words in a sequence, regardless of distance. This not only enables parallelization but also allows the model to dynamically focus on the most relevant parts of the input for each output. The 'why' behind attention's success lies in its ability to create a flexible, context-aware representation of each word by aggregating information from the entire sequence, weighted by relevance.
# Example: How attention weights capture word relationships
import numpy as np
# Simulate attention scores between words in a sentence
sentence = ['The', 'cat', 'sat', 'on', 'the', 'mat']
# Attention scores (simplified): how much each word attends to 'sat'
attention_to_sat = [0.1, 0.3, 1.0, 0.2, 0.1, 0.1] # 'cat' and 'on' are most relevant
# Normalize to probabilities (softmax)
def softmax(x):
e_x = np.exp(x - np.max(x))
return e_x / e_x.sum()
attention_probs = softmax(attention_to_sat)
print("Attention probabilities for 'sat':", dict(zip(sentence, attention_probs)))
# Output shows 'cat' and 'on' have highest weights, capturing subject-verb and prepositional relationshipsAttention Is All You Need: The Transformer Architecture
The Transformer architecture introduced in 'Attention Is All You Need' is built around the concept of self-attention, which allows the model to weigh the importance of each word in a sequence relative to every other word. The key innovation is the scaled dot-product attention mechanism, which computes attention scores by taking the dot product of queries and keys (both derived from the input embeddings), scaling them, and applying a softmax to get probabilities. These probabilities are then used to weight the values, producing a context-aware representation for each word. The 'why' behind this design is twofold: first, dot products measure similarity, so words that are semantically or syntactically related will naturally have higher attention scores; second, scaling prevents the dot products from growing too large, which would push the softmax into regions with tiny gradients. The Transformer also introduces multi-head attention, which allows the model to focus on different types of relationships simultaneously (e.g., subject-verb agreement vs. coreference). This is achieved by splitting the queries, keys, and values into multiple heads, computing attention for each head in parallel, and then concatenating the results. The Transformer's encoder-decoder structure further enables it to handle sequence-to-sequence tasks like translation, where the encoder processes the input and the decoder generates the output step-by-step, attending to the encoder's representations at each step.
# Scaled dot-product attention implementation
import torch
import torch.nn.functional as F
def scaled_dot_product_attention(query, key, value, mask=None):
# query, key, value shapes: (batch_size, seq_len, d_k)
d_k = query.size(-1)
# Compute attention scores
scores = torch.matmul(query, key.transpose(-2, -1)) / np.sqrt(d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
# Softmax to get attention probabilities
attention_probs = F.softmax(scores, dim=-1)
# Weighted sum of values
output = torch.matmul(attention_probs, value)
return output, attention_probs
# Example usage
batch_size, seq_len, d_k = 2, 5, 64
query = torch.randn(batch_size, seq_len, d_k)
key = torch.randn(batch_size, seq_len, d_k)
value = torch.randn(batch_size, seq_len, d_k)
output, attn_probs = scaled_dot_product_attention(query, key, value)
print("Output shape:", output.shape) # (2, 5, 64)
print("Attention probabilities shape:", attn_probs.shape) # (2, 5, 5)BERT: Bidirectional Context with Masked Language Modeling
BERT (Bidirectional Encoder Representations from Transformers) addressed a fundamental limitation of previous models like the Transformer's decoder or early language models: they could only process text in one direction (left-to-right or right-to-left). This unidirectional constraint meant that the representation of a word couldn't incorporate information from words that came after it, which is critical for tasks like question answering or coreference resolution. BERT's key innovation was to use a masked language modeling (MLM) objective, where random words in the input are masked, and the model is trained to predict them based on the surrounding context. This forces the model to learn bidirectional representations, as the prediction for a masked word depends on both left and right context. The 'why' behind BERT's success lies in its ability to create deeply contextualized word embeddings that capture nuanced meanings. For example, the word 'bank' in 'river bank' and 'bank account' will have different representations because the surrounding words provide disambiguating context. BERT also introduced next sentence prediction (NSP) as a secondary objective, which helps the model understand relationships between sentences, though this has since been shown to be less critical. The architecture itself is a stack of Transformer encoders, making it straightforward to pre-train on large corpora and then fine-tune for specific tasks by adding a simple classification head.
# Masked Language Modeling (MLM) example with BERT-style masking
from transformers import BertTokenizer, BertForMaskedLM
import torch
# Load pre-trained BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
# Example sentence with a masked token
text = "The [MASK] sat on the mat."
inputs = tokenizer(text, return_tensors='pt')
# Predict the masked token
with torch.no_grad():
outputs = model(**inputs)
predictions = outputs.logits
# Get the top 5 predicted tokens for the mask
mask_token_index = torch.where(inputs['input_ids'] == tokenizer.mask_token_id)[1]
predicted_token_ids = torch.topk(predictions[0, mask_token_index], 5).indices[0].tolist()
predicted_tokens = [tokenizer.decode([token_id]) for token_id in predicted_token_ids]
print("Top predictions for [MASK]:", predicted_tokens)
# Output: ['cat', 'dog', 'man', 'child', 'animal'] — contextually appropriate wordsGPT: Autoregressive Language Modeling and Scaling
GPT (Generative Pre-trained Transformer) took a different approach from BERT by focusing on autoregressive language modeling, where the model predicts the next word in a sequence given all previous words. This unidirectional approach might seem limiting compared to BERT's bidirectionality, but it has a critical advantage: it enables the model to generate coherent, long-form text, which is essential for tasks like story generation or dialogue systems. The 'why' behind GPT's success lies in its ability to model the joint probability of sequences, allowing it to capture complex dependencies across many words. GPT's architecture is a stack of Transformer decoders, which are similar to encoders but include a masked self-attention mechanism to prevent the model from 'cheating' by looking at future words during training. This masking ensures that the model learns to generate text step-by-step, making it ideal for generative tasks. Another key insight from GPT is the power of scaling: larger models trained on more data consistently perform better, even without task-specific fine-tuning. This is because the model learns a rich, general-purpose representation of language that can be adapted to many tasks with minimal additional training. GPT also popularized the 'pre-train, then fine-tune' paradigm, where a model is first trained on a large corpus with a simple objective (next word prediction) and then fine-tuned on smaller, task-specific datasets.
# Autoregressive text generation with GPT-2
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch
# Load pre-trained GPT-2 tokenizer and model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
# Generate text autoregressively
prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors='pt')
# Generate text with top-k sampling
output = model.generate(
**inputs,
max_length=50,
do_sample=True,
top_k=50,
pad_token_id=tokenizer.eos_token_id
)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print("Generated text:", generated_text)
# Output: Coherent continuation of the prompt, e.g., "is bright, but it also poses significant challenges..."Comparing the Papers: When to Use Each Approach
While all three papers introduced groundbreaking ideas, they serve different purposes and excel in different scenarios. The Transformer's encoder-decoder architecture is ideal for sequence-to-sequence tasks like machine translation, where the input and output are sequences of different lengths. Its self-attention mechanism allows it to capture relationships between all words in both the input and output, making it highly effective for tasks requiring alignment between sequences. BERT, with its bidirectional context, is best suited for tasks that require understanding the full context of a word or sentence, such as text classification, named entity recognition, or question answering. Its masked language modeling objective forces the model to learn rich, contextual representations that can be fine-tuned for a wide range of downstream tasks. GPT, on the other hand, shines in generative tasks where the goal is to produce coherent, long-form text. Its autoregressive nature makes it perfect for applications like chatbots, story generation, or code completion. The 'why' behind choosing one over the others often comes down to the trade-off between bidirectionality and generativity: BERT's bidirectionality is powerful for understanding, but GPT's unidirectionality is necessary for generation. The Transformer sits in the middle, offering flexibility for tasks that require both understanding and generation. When designing your own LLM, consider whether your task requires understanding (BERT), generation (GPT), or both (Transformer), and choose the architecture accordingly. Scaling laws also play a role: GPT-style models benefit more from larger datasets and model sizes, while BERT's performance plateaus earlier.
# Comparing model outputs for a simple classification task
from transformers import BertTokenizer, BertForSequenceClassification, GPT2Tokenizer, GPT2ForSequenceClassification
import torch
# Example: Sentiment analysis with BERT and GPT
text = "I loved the movie, it was fantastic!"
# BERT
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bert_model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
bert_inputs = bert_tokenizer(text, return_tensors='pt', truncation=True, padding=True)
with torch.no_grad():
bert_outputs = bert_model(**bert_inputs)
bert_prediction = torch.argmax(bert_outputs.logits, dim=1).item()
# GPT (fine-tuned for classification)
gpt_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
gpt_tokenizer.pad_token = gpt_tokenizer.eos_token
gpt_model = GPT2ForSequenceClassification.from_pretrained('gpt2')
gpt_inputs = gpt_tokenizer(text, return_tensors='pt', truncation=True, padding=True)
with torch.no_grad():
gpt_outputs = gpt_model(**gpt_inputs)
gpt_prediction = torch.argmax(gpt_outputs.logits, dim=1).item()
print("BERT prediction (0=negative, 1=positive):", bert_prediction)
print("GPT prediction (0=negative, 1=positive):", gpt_prediction)
# Output: Both models likely predict '1' (positive), but BERT is more reliable for classificationKey points
- The Transformer's self-attention mechanism works because it dynamically weights the importance of each word in a sequence relative to every other word, enabling parallelization and capturing long-range dependencies.
- BERT's masked language modeling objective forces the model to learn bidirectional context, making it ideal for tasks that require understanding the full meaning of a word or sentence.
- GPT's autoregressive approach is necessary for generative tasks because it models the joint probability of sequences, allowing it to produce coherent, long-form text step-by-step.
- Multi-head attention in the Transformer allows the model to focus on different types of relationships simultaneously, such as syntactic and semantic dependencies.
- Scaling laws explain why larger models like GPT perform better: they learn richer, more general-purpose representations that can be adapted to many tasks with minimal fine-tuning.
- The choice between BERT, GPT, or the Transformer depends on whether your task requires understanding, generation, or both, as each architecture has trade-offs in bidirectionality and generativity.
- Pre-training objectives like masked language modeling or next word prediction are critical because they enable models to learn general language representations that can be fine-tuned for specific tasks.
- The Transformer's encoder-decoder architecture is particularly effective for sequence-to-sequence tasks because it can align and transform input sequences into output sequences of different lengths.
Common mistakes
- Mistake: Assuming the Transformer architecture from 'Attention Is All You Need' can be implemented without understanding multi-head attention. Why it's wrong: Multi-head attention is the core innovation enabling parallel processing of different representation subspaces, and skipping its nuances leads to poor model performance or instability. Fix: Study the mathematical formulation of scaled dot-product attention and implement it step-by-step before scaling up.
- Mistake: Treating BERT's masked language modeling (MLM) as identical to traditional autoencoders. Why it's wrong: BERT's MLM randomly masks tokens during training, forcing bidirectional context understanding, unlike autoencoders that reconstruct full inputs. Misapplying this leads to models that fail to capture contextual dependencies. Fix: Implement dynamic masking and ensure the model predicts masked tokens using surrounding context, not just local patterns.
- Mistake: Believing GPT's autoregressive training means it can only generate text sequentially without parallelization. Why it's wrong: While generation is sequential, training leverages parallel processing of input sequences (teacher forcing). Confusing these phases results in inefficient training loops. Fix: Use batched training with causal masking to enable parallelization during training while maintaining autoregressive properties during inference.
- Mistake: Ignoring positional encodings in Transformers because 'attention is permutation-invariant.' Why it's wrong: Attention alone cannot capture word order, which is critical for language understanding. Omitting positional encodings makes the model treat 'cat chases dog' and 'dog chases cat' as identical. Fix: Implement sinusoidal or learned positional encodings and verify their impact on model outputs.
- Mistake: Assuming fine-tuning BERT or GPT for a custom LLM task requires only adjusting the final layer. Why it's wrong: Task-specific fine-tuning often requires modifying the architecture (e.g., adding task heads) and careful hyperparameter tuning. Overlooking this leads to catastrophic forgetting or poor generalization. Fix: Use gradual unfreezing, learning rate scheduling, and task-specific layers to adapt pre-trained models effectively.
Interview questions
What problem did the 'Attention Is All You Need' paper solve in the context of building your own LLM?
The 'Attention Is All You Need' paper introduced the Transformer architecture, which eliminated the need for recurrent or convolutional layers in sequence modeling. Before this, models like RNNs or LSTMs struggled with long-range dependencies because they processed sequences step-by-step, leading to vanishing gradients and inefficiency. The Transformer solved this by using self-attention mechanisms, which allow the model to weigh the importance of every token in the input sequence simultaneously. This parallelization not only improved training speed but also enabled the model to capture relationships between distant tokens more effectively. For example, in a sentence like 'The cat sat on the mat because it was tired,' self-attention helps the model understand that 'it' refers to 'cat' by directly attending to it, regardless of distance. This breakthrough made it feasible to train much larger models efficiently, which is critical when building your own LLM.
How does the self-attention mechanism work in the Transformer, and why is it important for LLMs?
Self-attention is the core innovation of the Transformer architecture. It works by computing three vectors for each token in the input: a query, a key, and a value. These vectors are learned during training. For every token, the model calculates attention scores by taking the dot product of its query vector with the key vectors of all other tokens. These scores are then normalized using a softmax function to produce weights that sum to 1. Finally, the output for the token is a weighted sum of the value vectors, where the weights are the attention scores. This process allows the model to dynamically focus on the most relevant parts of the input for each token. For example, in the sentence 'She poured water into the glass and drank it,' the model can learn to attend strongly to 'water' when processing 'it.' Self-attention is crucial for LLMs because it enables the model to handle long-range dependencies and capture contextual relationships across the entire input sequence, which is essential for tasks like text generation, translation, or summarization. Here’s a simplified code snippet of self-attention: ```python def self_attention(query, key, value): scores = torch.matmul(query, key.transpose(-2, -1)) / torch.sqrt(torch.tensor(key.size(-1), dtype=torch.float32)) weights = torch.softmax(scores, dim=-1) return torch.matmul(weights, value) ```
What is BERT, and how does its pre-training approach differ from traditional language models when building your own LLM?
BERT, or Bidirectional Encoder Representations from Transformers, is a pre-trained language model that introduced a novel approach to understanding context in text. Unlike traditional language models, which process text in a left-to-right or right-to-left manner, BERT uses a bidirectional approach, meaning it considers the entire context of a word by looking at all surrounding words simultaneously. This is achieved through two key pre-training tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). In MLM, random tokens in the input are masked, and the model is trained to predict them based on the surrounding context. For example, in the sentence 'The [MASK] sat on the mat,' the model might predict 'cat.' NSP, on the other hand, trains the model to understand relationships between sentences by predicting whether one sentence logically follows another. This bidirectional training allows BERT to capture deeper contextual relationships, making it highly effective for tasks like question answering, sentiment analysis, or text classification. When building your own LLM, adopting BERT’s pre-training approach can significantly improve performance on downstream tasks, as it provides a richer understanding of language context.
How does GPT differ from BERT in terms of architecture and training objectives, and why might you choose one over the other for your LLM?
GPT and BERT differ fundamentally in their architecture and training objectives, which influences their suitability for different tasks when building your own LLM. GPT, or Generative Pre-trained Transformer, is a unidirectional model that processes text in a left-to-right manner, making it ideal for generative tasks like text completion, summarization, or dialogue generation. It is trained using a causal language modeling objective, where the model predicts the next token in a sequence based on the preceding tokens. For example, given 'The cat sat on the,' GPT might predict 'mat.' This autoregressive approach makes GPT highly effective for tasks requiring coherent and contextually relevant text generation. BERT, on the other hand, is a bidirectional model that processes text in both directions simultaneously, making it better suited for understanding tasks like question answering, sentiment analysis, or named entity recognition. Its Masked Language Modeling (MLM) objective allows it to capture context from both sides of a token, which is why it excels in tasks requiring deep contextual understanding. If your LLM project focuses on generating text, GPT’s architecture is likely the better choice. However, if your goal is to build a model for understanding or classifying text, BERT’s bidirectional approach may be more appropriate. For example, GPT would be ideal for a chatbot, while BERT would be better for a system that extracts key information from documents.
Explain how you would implement a simplified version of the Transformer’s multi-head attention mechanism in your own LLM. Include a code example.
Implementing multi-head attention in your own LLM involves splitting the self-attention mechanism into multiple 'heads,' allowing the model to focus on different parts of the input simultaneously. Each head learns its own set of query, key, and value projections, enabling the model to capture diverse relationships in the data. Here’s how you can implement a simplified version: First, you define linear layers to project the input into multiple query, key, and value matrices. Then, you compute the scaled dot-product attention for each head independently. Finally, you concatenate the outputs of all heads and pass them through a final linear layer to combine the information. This approach improves the model’s ability to capture complex patterns, such as syntactic and semantic relationships in text. Here’s a code example: ```python import torch import torch.nn as nn class MultiHeadAttention(nn.Module): def __init__(self, embed_size, num_heads): super().__init__() self.embed_size = embed_size self.num_heads = num_heads self.head_dim = embed_size // num_heads self.query = nn.Linear(embed_size, embed_size) self.key = nn.Linear(embed_size, embed_size) self.value = nn.Linear(embed_size, embed_size) self.fc_out = nn.Linear(embed_size, embed_size) def forward(self, query, key, value): # Project inputs into multiple heads Q = self.query(query) K = self.key(key) V = self.value(value) # Split into multiple heads Q = Q.view(Q.shape[0], -1, self.num_heads, self.head_dim).transpose(1, 2) K = K.view(K.shape[0], -1, self.num_heads, self.head_dim).transpose(1, 2) V = V.view(V.shape[0], -1, self.num_heads, self.head_dim).transpose(1, 2) # Compute attention scores scores = torch.matmul(Q, K.transpose(-2, -1)) / torch.sqrt(torch.tensor(self.head_dim, dtype=torch.float32)) weights = torch.softmax(scores, dim=-1) # Apply attention to values output = torch.matmul(weights, V) # Concatenate heads and pass through final linear layer output = output.transpose(1, 2).contiguous().view(output.shape[0], -1, self.embed_size) return self.fc_out(output) ``` This implementation allows your LLM to attend to multiple aspects of the input simultaneously, improving its ability to generate or understand text.
When building your own LLM, how would you decide whether to use a BERT-style or GPT-style architecture, and what trade-offs would you consider?
Deciding between a BERT-style or GPT-style architecture for your LLM depends on the specific goals of your project and the trade-offs you’re willing to make. If your primary objective is text generation, such as building a chatbot, writing assistant, or creative content generator, a GPT-style architecture is the clear choice. GPT’s unidirectional, autoregressive design excels at predicting the next token in a sequence, making it ideal for tasks requiring coherent and contextually relevant output. For example, if you’re building a model to generate product descriptions or dialogue, GPT’s ability to maintain context over long sequences is invaluable. However, GPT has limitations in tasks requiring deep contextual understanding, such as question answering or sentiment analysis, where bidirectional context is crucial. On the other hand, if your LLM is intended for understanding tasks, like extracting information from documents, classifying text, or answering questions, a BERT-style architecture is more suitable. BERT’s bidirectional approach allows it to capture context from both sides of a token, enabling it to perform exceptionally well on tasks that require fine-grained understanding of language. For instance, BERT can accurately determine whether 'bank' refers to a financial institution or a riverbank based on surrounding words. The trade-offs to consider include computational resources, as BERT’s bidirectional training can be more demanding, and the nature of the output: GPT generates text, while BERT produces embeddings or classifications. Additionally, you might consider hybrid approaches, such as T5, which combine elements of both architectures to leverage their strengths. Ultimately, the decision hinges on whether your LLM’s primary function is generation or understanding, and how you balance performance, efficiency, and task-specific requirements.
Check yourself
1. When implementing a Transformer decoder for your own LLM, why is causal masking essential during training?
- A.It prevents the model from attending to future tokens, ensuring autoregressive generation mimics inference conditions.
- B.It reduces computational cost by limiting attention to a fixed window of past tokens.
- C.It enforces bidirectional context, similar to BERT's masked language modeling.
- D.It replaces positional encodings by encoding token order in the attention weights.
Show answer
A. It prevents the model from attending to future tokens, ensuring autoregressive generation mimics inference conditions.
The correct answer is that causal masking prevents the model from attending to future tokens, ensuring the training process mirrors inference (where future tokens are unknown). The other options are incorrect: (1) While causal masking does limit attention, its primary purpose is not computational efficiency. (2) Causal masking enforces unidirectional context, unlike BERT's bidirectional approach. (3) Causal masking does not replace positional encodings; they serve orthogonal purposes (order vs. context).
2. You are fine-tuning BERT for a custom LLM task. Why might freezing the lower layers and only training the top layers lead to poor performance?
- A.Lower layers capture general language features (e.g., syntax), while higher layers specialize in task-specific patterns; freezing them prevents adaptation.
- B.Frozen layers cannot propagate gradients, causing the model to fail during backpropagation.
- C.BERT's architecture requires all layers to be trained simultaneously to maintain attention mechanisms.
- D.Lower layers are task-specific, and freezing them forces the model to rely on outdated features.
Show answer
A. Lower layers capture general language features (e.g., syntax), while higher layers specialize in task-specific patterns; freezing them prevents adaptation.
The correct answer is that lower layers capture general language features, while higher layers specialize in task-specific patterns. Freezing lower layers prevents the model from adapting these foundational features to the new task. The other options are incorrect: (1) Frozen layers do propagate gradients (they just don't update their weights). (2) BERT can be fine-tuned with partial layer training. (3) Lower layers are not task-specific; they learn general representations.
3. In 'Attention Is All You Need,' why is the scaled dot-product attention mechanism preferred over additive attention for Transformers?
- A.Scaled dot-product attention is more computationally efficient and parallelizable on GPUs, despite similar theoretical expressiveness.
- B.Additive attention cannot handle variable-length sequences, making it incompatible with Transformer architectures.
- C.Scaled dot-product attention inherently includes positional encodings, eliminating the need for additional components.
- D.Additive attention requires quadratic memory, while scaled dot-product attention uses linear memory.
Show answer
A. Scaled dot-product attention is more computationally efficient and parallelizable on GPUs, despite similar theoretical expressiveness.
The correct answer is that scaled dot-product attention is more computationally efficient and parallelizable, which is critical for training large models. The other options are incorrect: (1) Additive attention can handle variable-length sequences. (2) Scaled dot-product attention does not include positional encodings. (3) Both attention mechanisms use quadratic memory for the attention matrix.
4. When designing your own LLM, you notice that GPT's autoregressive generation produces repetitive text. Which modification would most directly address this issue?
- A.Introduce a penalty for repeating n-grams during decoding, such as nucleus sampling with repetition constraints.
- B.Replace the autoregressive head with a bidirectional attention mechanism to capture broader context.
- C.Increase the model's depth to improve its ability to memorize training data patterns.
- D.Remove positional encodings to allow the model to generate tokens in any order.
Show answer
A. Introduce a penalty for repeating n-grams during decoding, such as nucleus sampling with repetition constraints.
The correct answer is to introduce a penalty for repeating n-grams, such as nucleus sampling with repetition constraints. This directly targets the issue of repetition. The other options are incorrect: (1) Bidirectional attention would break autoregressive generation. (2) Increasing depth may worsen repetition by overfitting. (3) Removing positional encodings would harm coherence, not fix repetition.
5. Why does BERT's masked language modeling (MLM) objective make it unsuitable for direct autoregressive text generation?
- A.MLM trains the model to predict masked tokens using bidirectional context, while autoregressive generation requires unidirectional context.
- B.MLM requires a separate decoder, which BERT lacks, making it impossible to generate text sequentially.
- C.MLM's masking strategy is incompatible with the attention mechanisms used in autoregressive models.
- D.BERT's architecture is designed for classification tasks, not sequence generation.
Show answer
A. MLM trains the model to predict masked tokens using bidirectional context, while autoregressive generation requires unidirectional context.
The correct answer is that MLM trains the model to predict masked tokens using bidirectional context, while autoregressive generation requires unidirectional context (predicting the next token based only on past tokens). The other options are incorrect: (1) BERT can generate text with modifications (e.g., BART), but its training objective is not autoregressive. (2) MLM does not inherently conflict with attention mechanisms. (3) BERT can be adapted for generation tasks, though it is not its primary design.