Core Concepts of Large Language Models
Architecture of Transformer Models: Encoder-Decoder vs. Decoder-Only
Transformer models come in two primary architectures: encoder-decoder and decoder-only, each designed for distinct tasks. Understanding these architectures is crucial because they determine how the model processes input data and generates output, directly impacting performance in tasks like translation or text generation. You’ll reach for these architectures when building systems that require contextual understanding or sequential output generation, such as chatbots, summarization tools, or machine translation systems.
The Core Idea: Self-Attention and Contextual Understanding
At the heart of transformer models lies the self-attention mechanism, which allows the model to weigh the importance of different parts of the input sequence when producing an output. Unlike traditional models that process input sequentially, self-attention enables parallel computation, making transformers highly efficient. The key insight is that words in a sentence don’t exist in isolation; their meaning depends on the surrounding context. For example, the word 'bank' could refer to a financial institution or the side of a river, and self-attention helps the model disambiguate this by attending to other words in the sentence. This mechanism is why transformers excel at capturing long-range dependencies, a critical feature for tasks like translation or summarization. The architecture’s ability to process entire sequences at once, rather than word-by-word, is what gives it a significant speed advantage over older models like RNNs or LSTMs.
# Self-attention calculation in a simplified form
import torch
import torch.nn.functional as F
# Example: Embeddings for 3 tokens in a sentence (batch_size=1, seq_len=3, embed_dim=4)
embeddings = torch.tensor([[[0.1, 0.2, 0.3, 0.4],
[0.5, 0.6, 0.7, 0.8],
[0.9, 1.0, 1.1, 1.2]]])
# Learnable weight matrices for queries, keys, and values
W_q = torch.randn(4, 4) # Query weights
W_k = torch.randn(4, 4) # Key weights
W_v = torch.randn(4, 4) # Value weights
# Compute queries, keys, and values
queries = torch.matmul(embeddings, W_q)
keys = torch.matmul(embeddings, W_k)
values = torch.matmul(embeddings, W_v)
# Scaled dot-product attention
scores = torch.matmul(queries, keys.transpose(-2, -1)) / (4 ** 0.5) # Scale by sqrt(embed_dim)
attention_weights = F.softmax(scores, dim=-1)
output = torch.matmul(attention_weights, values)
print("Attention Weights:\n", attention_weights)
print("Output after Self-Attention:\n", output)Encoder: Capturing Input Context
The encoder in a transformer model is responsible for processing the input sequence and generating a contextual representation of it. It consists of multiple layers, each containing two sub-layers: a self-attention mechanism and a position-wise feed-forward network. The self-attention layer allows the encoder to weigh the importance of each word in the input relative to every other word, creating a rich, context-aware representation. The feed-forward network then applies non-linear transformations to this representation, enabling the model to capture complex patterns. The encoder’s output is a set of vectors, one for each input token, that encode the entire context of the input sequence. This is particularly useful for tasks like machine translation, where the encoder’s output serves as the input to the decoder. The encoder’s strength lies in its ability to compress the input into a fixed-size representation that retains the most relevant information for the task at hand.
# Simplified encoder layer implementation
import torch.nn as nn
class EncoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim):
super().__init__()
self.self_attention = nn.MultiheadAttention(embed_dim, num_heads)
self.feed_forward = nn.Sequential(
nn.Linear(embed_dim, ff_dim),
nn.ReLU(),
nn.Linear(ff_dim, embed_dim)
)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
def forward(self, x):
# Self-attention sub-layer
attn_output, _ = self.self_attention(x, x, x)
x = self.norm1(x + attn_output) # Residual connection
# Feed-forward sub-layer
ff_output = self.feed_forward(x)
x = self.norm2(x + ff_output) # Residual connection
return x
# Example usage
embed_dim = 4
num_heads = 2
ff_dim = 8
encoder_layer = EncoderLayer(embed_dim, num_heads, ff_dim)
input_embeddings = torch.randn(3, 1, embed_dim) # (seq_len, batch_size, embed_dim)
output = encoder_layer(input_embeddings)
print("Encoder Output Shape:", output.shape)Decoder: Generating Sequential Output
The decoder in a transformer model is designed to generate output sequences, such as translations or summaries, one token at a time. It mirrors the encoder’s structure but includes an additional sub-layer: the encoder-decoder attention mechanism. This sub-layer allows the decoder to attend to the encoder’s output, ensuring that the generated tokens are conditioned on the input sequence. The decoder also uses masked self-attention to prevent it from attending to future tokens during training, which is critical for autoregressive generation. Each layer in the decoder refines the output by combining the encoder’s contextual information with its own predictions. This architecture is ideal for tasks where the output length is not fixed, such as text generation or machine translation. The decoder’s ability to dynamically attend to both the input and its own previous outputs is what makes it so powerful for sequential tasks.
# Simplified decoder layer implementation
class DecoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim):
super().__init__()
self.self_attention = nn.MultiheadAttention(embed_dim, num_heads)
self.encoder_decoder_attention = nn.MultiheadAttention(embed_dim, num_heads)
self.feed_forward = nn.Sequential(
nn.Linear(embed_dim, ff_dim),
nn.ReLU(),
nn.Linear(ff_dim, embed_dim)
)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.norm3 = nn.LayerNorm(embed_dim)
def forward(self, x, encoder_output):
# Masked self-attention sub-layer
attn_output, _ = self.self_attention(x, x, x, attn_mask=self._generate_square_subsequent_mask(x.size(0)))
x = self.norm1(x + attn_output)
# Encoder-decoder attention sub-layer
enc_dec_attn_output, _ = self.encoder_decoder_attention(x, encoder_output, encoder_output)
x = self.norm2(x + enc_dec_attn_output)
# Feed-forward sub-layer
ff_output = self.feed_forward(x)
x = self.norm3(x + ff_output)
return x
def _generate_square_subsequent_mask(self, sz):
# Prevents attending to future tokens
mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
return mask
# Example usage
decoder_layer = DecoderLayer(embed_dim, num_heads, ff_dim)
target_embeddings = torch.randn(3, 1, embed_dim) # (seq_len, batch_size, embed_dim)
encoder_output = torch.randn(3, 1, embed_dim) # From encoder
output = decoder_layer(target_embeddings, encoder_output)
print("Decoder Output Shape:", output.shape)Encoder-Decoder Architecture: When to Use It
The encoder-decoder architecture is the go-to choice for tasks that require transforming an input sequence into a different output sequence, such as machine translation or text summarization. The encoder processes the input and creates a fixed-size representation, while the decoder uses this representation to generate the output step-by-step. This separation of concerns is what makes the architecture so effective for tasks where the input and output lengths differ or where the output depends heavily on the input’s context. For example, in machine translation, the encoder captures the meaning of the source sentence, and the decoder generates the target sentence word by word. The encoder-decoder architecture’s strength lies in its ability to handle variable-length inputs and outputs while maintaining a clear flow of information from input to output. However, it can be overkill for tasks where the input and output are closely aligned, such as text generation from a prompt.
# Example of encoder-decoder for a simple translation task
class EncoderDecoder(nn.Module):
def __init__(self, vocab_size, embed_dim, num_heads, ff_dim, num_layers):
super().__init__()
self.encoder = nn.TransformerEncoder(
nn.TransformerEncoderLayer(embed_dim, num_heads, ff_dim),
num_layers=num_layers
)
self.decoder = nn.TransformerDecoder(
nn.TransformerDecoderLayer(embed_dim, num_heads, ff_dim),
num_layers=num_layers
)
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.fc_out = nn.Linear(embed_dim, vocab_size)
def forward(self, src, tgt):
# Embed and encode the source sequence
src_emb = self.embedding(src)
encoder_output = self.encoder(src_emb)
# Embed and decode the target sequence
tgt_emb = self.embedding(tgt)
decoder_output = self.decoder(tgt_emb, encoder_output)
# Project to vocabulary size
output = self.fc_out(decoder_output)
return output
# Example usage
vocab_size = 1000
model = EncoderDecoder(vocab_size, embed_dim, num_heads, ff_dim, num_layers=2)
src = torch.randint(0, vocab_size, (5, 1)) # Source sequence (seq_len, batch_size)
tgt = torch.randint(0, vocab_size, (4, 1)) # Target sequence (seq_len, batch_size)
output = model(src, tgt)
print("Encoder-Decoder Output Shape:", output.shape)Decoder-Only Architecture: Simplicity and Scalability
The decoder-only architecture, used in models like GPT, simplifies the transformer by removing the encoder entirely and relying solely on the decoder. This architecture is particularly effective for tasks where the input and output are part of the same sequence, such as text generation or completion. The decoder-only model processes the input and generates the output autoregressively, attending only to the tokens it has already generated. This makes it highly scalable, as it can handle arbitrarily long sequences without the need for a separate encoder. The decoder-only architecture’s strength lies in its simplicity and efficiency, as it avoids the overhead of managing two separate components. It’s ideal for tasks like chatbots or code generation, where the model needs to generate coherent and contextually relevant text based on a prompt. However, it may struggle with tasks requiring deep bidirectional understanding, such as machine translation, where the encoder’s contextual representation is critical.
# Simplified decoder-only model for text generation
class DecoderOnlyModel(nn.Module):
def __init__(self, vocab_size, embed_dim, num_heads, ff_dim, num_layers):
super().__init__()
self.decoder = nn.TransformerDecoder(
nn.TransformerDecoderLayer(embed_dim, num_heads, ff_dim),
num_layers=num_layers
)
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.fc_out = nn.Linear(embed_dim, vocab_size)
def forward(self, tgt):
# Embed the target sequence
tgt_emb = self.embedding(tgt)
# Generate a mask to prevent attending to future tokens
tgt_mask = self._generate_square_subsequent_mask(tgt.size(0))
# Decode the sequence
decoder_output = self.decoder(tgt_emb, tgt_emb, tgt_mask=tgt_mask)
# Project to vocabulary size
output = self.fc_out(decoder_output)
return output
def _generate_square_subsequent_mask(self, sz):
mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
return mask
# Example usage
model = DecoderOnlyModel(vocab_size, embed_dim, num_heads, ff_dim, num_layers=2)
tgt = torch.randint(0, vocab_size, (5, 1)) # Input sequence (seq_len, batch_size)
output = model(tgt)
print("Decoder-Only Output Shape:", output.shape)Key points
- Self-attention is the foundation of transformer models, allowing them to weigh the importance of each word in a sequence relative to all others.
- The encoder processes input sequences into contextual representations, making it ideal for tasks requiring deep understanding of the input.
- The decoder generates output sequences autoregressively, attending to both the input and its own previous outputs for coherence.
- Encoder-decoder architectures excel at tasks like machine translation, where input and output sequences differ in length or structure.
- Decoder-only architectures simplify the model by removing the encoder, making them highly scalable for tasks like text generation.
- Masked self-attention in decoders prevents the model from attending to future tokens, ensuring autoregressive generation works correctly.
- The choice between encoder-decoder and decoder-only architectures depends on whether the task requires bidirectional input understanding or unidirectional output generation.
- Residual connections and layer normalization in transformer layers help stabilize training and improve model performance.
Common mistakes
- Mistake: Assuming encoder-decoder and decoder-only architectures are interchangeable for all tasks. Why it's wrong: Encoder-decoder models excel at sequence-to-sequence tasks (e.g., translation) where input and output lengths differ, while decoder-only models are optimized for autoregressive generation (e.g., text completion). Fix: Match the architecture to the task—use encoder-decoder for tasks requiring input-output alignment and decoder-only for open-ended generation.
- Mistake: Ignoring the role of attention masks in decoder-only models. Why it's wrong: Decoder-only models rely on causal masks to prevent future token leakage during training, ensuring predictions depend only on prior tokens. Without proper masking, the model may 'cheat' by peeking ahead. Fix: Implement triangular attention masks to enforce autoregressive behavior during both training and inference.
- Mistake: Overlooking positional encodings in decoder-only architectures. Why it's wrong: Decoder-only models lack an encoder to provide input order context, so positional encodings are critical for capturing sequence structure. Fix: Always include positional encodings (e.g., learned or sinusoidal) to enable the model to distinguish token order.
- Mistake: Treating the encoder’s output as a fixed representation in encoder-decoder models. Why it's wrong: The encoder’s output is a dynamic context vector that the decoder attends to, but it’s not static—it’s refined via cross-attention. Fix: Design the decoder to actively query the encoder’s output using cross-attention mechanisms, not just passively consume it.
- Mistake: Assuming decoder-only models can’t handle bidirectional context. Why it's wrong: Decoder-only models are inherently unidirectional (left-to-right), which limits their ability to process bidirectional context like encoder-decoder models. Fix: For tasks requiring bidirectional context (e.g., summarization), use encoder-decoder architectures or prepend bidirectional pre-training (e.g., BERT-style) before fine-tuning with a decoder.
Interview questions
What is the basic difference between an encoder-decoder and a decoder-only transformer architecture?
The encoder-decoder architecture consists of two main components: the encoder, which processes the input sequence and creates a contextual representation, and the decoder, which generates the output sequence based on that representation. This design is ideal for tasks like translation, where the input and output sequences differ. In contrast, a decoder-only architecture skips the encoder entirely and relies solely on the decoder to process and generate sequences autoregressively. This is efficient for tasks like text generation, where the model predicts the next token based on previous ones. For example, in a decoder-only model, you might see a loop like `for token in input_tokens: output = model.generate(token)`, whereas an encoder-decoder would first encode the entire input before decoding.
Why would you choose an encoder-decoder architecture when building your own LLM?
You’d choose an encoder-decoder architecture when your task involves transforming one sequence into another, such as translation, summarization, or question answering. The encoder processes the input into a fixed-length representation, capturing its meaning, while the decoder uses that representation to generate the output step-by-step. This separation is powerful because it allows the model to handle inputs and outputs of different lengths or structures. For instance, in a translation task, the input might be a long sentence in one language, and the output a shorter or longer sentence in another. The encoder-decoder’s explicit bottleneck forces the model to learn meaningful representations, which can improve performance on complex tasks.
When is a decoder-only architecture more suitable for building an LLM, and why?
A decoder-only architecture is more suitable for tasks where the input and output are part of the same sequence, such as text generation, autocomplete, or dialogue systems. This architecture processes the input autoregressively, meaning it generates one token at a time while conditioning on the previously generated tokens. It’s simpler to implement and train because it avoids the complexity of aligning encoder and decoder representations. For example, in a chatbot, the model can generate responses by predicting the next token in a continuous stream. Decoder-only models also scale well with large datasets, as they can leverage vast amounts of unstructured text for pretraining. The training loop often looks like `loss = model(input_ids, labels=input_ids)`, where the model learns to predict the next token in the sequence.
How does the attention mechanism differ between encoder-decoder and decoder-only transformers?
In an encoder-decoder transformer, the attention mechanism operates in two distinct ways. The encoder uses self-attention to process the input sequence, allowing each token to attend to every other token in the input. The decoder, however, uses two types of attention: self-attention over the previously generated tokens and cross-attention over the encoder’s output. This cross-attention allows the decoder to focus on relevant parts of the input sequence while generating the output. In a decoder-only transformer, there’s only self-attention, where each token attends to all previous tokens in the sequence. This simplifies the architecture but limits the model’s ability to explicitly separate input and output processing. For example, in code, the decoder’s attention might look like `attention_scores = softmax(Q @ K.T / sqrt(d_k))`, where `Q` comes from the decoder and `K` from the encoder in an encoder-decoder setup.
Compare the training process of encoder-decoder and decoder-only architectures for building an LLM. Which one is more efficient and why?
The training process for encoder-decoder architectures is more complex because it involves two separate components that must be trained jointly. The encoder processes the input, and the decoder generates the output while attending to the encoder’s representations. This requires careful alignment of the two components, often using teacher forcing, where the decoder is fed the correct previous tokens during training. Decoder-only architectures, on the other hand, are simpler to train because they treat the entire task as a single sequence prediction problem. The model learns to predict the next token in the sequence, which can be done efficiently with a single forward pass. Decoder-only training is generally more efficient because it avoids the overhead of managing two separate models and can leverage large-scale pretraining on unstructured text. For example, the loss function for a decoder-only model is straightforward: `loss = cross_entropy(logits, labels)`, where `logits` are the model’s predictions and `labels` are the next tokens in the sequence.
Imagine you’re building a custom LLM for a summarization task. Would you use an encoder-decoder or decoder-only architecture, and how would you implement the key components?
For a summarization task, I would choose an encoder-decoder architecture because it’s specifically designed to handle input-output transformations where the input and output sequences differ in length or structure. The encoder would process the long input document into a contextual representation, while the decoder would generate the concise summary by attending to that representation. To implement this, I’d start with a pretrained encoder-decoder model like T5 or BART, then fine-tune it on a summarization dataset. The key components would include: 1) an encoder with self-attention layers to process the input, 2) a decoder with self-attention and cross-attention to generate the summary, and 3) a final linear layer to predict the output tokens. During training, I’d use teacher forcing to stabilize the decoder’s learning. For example, the decoder’s cross-attention would compute `attention_weights = softmax(Q_decoder @ K_encoder.T)`, allowing it to focus on the most relevant parts of the input document while generating the summary.
Check yourself
1. When building a custom LLM for a translation task, why is an encoder-decoder architecture typically preferred over a decoder-only model?
- A.Encoder-decoder models are simpler to implement and require fewer parameters.
- B.Decoder-only models cannot generate sequences longer than the input, making them unsuitable for translation.
- C.Encoder-decoder models explicitly separate input processing (encoder) from output generation (decoder), better handling variable-length input-output mappings.
- D.Decoder-only models lack attention mechanisms, which are critical for translation tasks.
Show answer
C. Encoder-decoder models explicitly separate input processing (encoder) from output generation (decoder), better handling variable-length input-output mappings.
The correct answer is that encoder-decoder models explicitly separate input processing from output generation, which is ideal for tasks like translation where input and output lengths differ. The encoder compresses the input into a context vector, while the decoder generates the output autoregressively. Option 1 is wrong because encoder-decoder models are often more complex. Option 2 is incorrect because decoder-only models can generate sequences of arbitrary length. Option 4 is false because decoder-only models do use attention (e.g., causal self-attention).
2. In a decoder-only LLM, what is the primary purpose of the causal attention mask?
- A.To reduce computational overhead by limiting the number of attention heads.
- B.To ensure the model only attends to tokens before the current position, preventing future token leakage.
- C.To enforce bidirectional context, similar to how encoders process input.
- D.To replace positional encodings, as masks can encode token order.
Show answer
B. To ensure the model only attends to tokens before the current position, preventing future token leakage.
The correct answer is that the causal attention mask ensures the model only attends to prior tokens, preventing it from 'cheating' by looking ahead during training. This enforces autoregressive behavior. Option 1 is incorrect because masks don’t reduce computational overhead. Option 3 is wrong because masks enforce unidirectional context, not bidirectional. Option 4 is false because masks and positional encodings serve different purposes.
3. You are designing a custom LLM for a chatbot that must generate responses based on user input. Why might a decoder-only architecture be a better choice than an encoder-decoder model?
- A.Decoder-only models are inherently better at understanding bidirectional context, which is critical for chatbots.
- B.Encoder-decoder models require paired input-output data, while decoder-only models can be fine-tuned on unstructured text.
- C.Decoder-only models can process input and generate output in a single pass, making them faster for real-time applications.
- D.Encoder-decoder models cannot generate coherent responses longer than a few sentences.
Show answer
B. Encoder-decoder models require paired input-output data, while decoder-only models can be fine-tuned on unstructured text.
The correct answer is that decoder-only models can be fine-tuned on unstructured text (e.g., next-token prediction), while encoder-decoder models often require paired input-output data (e.g., translation pairs). This makes decoder-only models more flexible for chatbots. Option 1 is wrong because decoder-only models are unidirectional. Option 3 is incorrect because both architectures can generate output in a single pass. Option 4 is false because encoder-decoder models can generate long sequences.
4. What is a key limitation of decoder-only models when compared to encoder-decoder models for tasks like summarization?
- A.Decoder-only models cannot generate summaries longer than the input document.
- B.Decoder-only models lack the ability to process the entire input document bidirectionally before generating output.
- C.Encoder-decoder models are always faster during inference for summarization tasks.
- D.Decoder-only models require explicit cross-attention mechanisms to process input, which encoder-decoder models lack.
Show answer
B. Decoder-only models lack the ability to process the entire input document bidirectionally before generating output.
The correct answer is that decoder-only models lack bidirectional context, which is critical for summarization tasks where the model must understand the entire input before generating a concise output. Encoder-decoder models process the input bidirectionally in the encoder. Option 1 is incorrect because decoder-only models can generate outputs of arbitrary length. Option 3 is false because inference speed depends on implementation, not architecture. Option 4 is wrong because cross-attention is a feature of encoder-decoder models, not a limitation of decoder-only models.
5. When implementing a custom LLM, you notice that your decoder-only model’s outputs are repetitive and lack coherence. Which of the following is the most likely cause?
- A.The model is using an encoder-decoder architecture instead of a decoder-only one.
- B.The causal attention mask is incorrectly implemented, allowing the model to attend to future tokens.
- C.The positional encodings are missing or improperly applied, causing the model to lose track of token order.
- D.The model’s vocabulary size is too small, limiting its expressive power.
Show answer
C. The positional encodings are missing or improperly applied, causing the model to lose track of token order.
The correct answer is that missing or improper positional encodings can cause the model to lose track of token order, leading to repetitive or incoherent outputs. Positional encodings are critical for decoder-only models to understand sequence structure. Option 1 is irrelevant because the model is confirmed to be decoder-only. Option 2 is less likely because incorrect masks would cause training instability, not necessarily repetitive outputs. Option 4 could cause issues but is not the most likely cause of repetition.