Foundations of Machine Learning and NLP
Word Embeddings: Word2Vec, GloVe, and FastText
Word embeddings are dense vector representations of words that capture semantic and syntactic relationships, enabling machines to understand language contextually. They matter because they transform raw text into numerical form, which is essential for training large language models and improving performance in tasks like translation, sentiment analysis, and question answering. You reach for word embeddings when preprocessing text data for any NLP task where meaning and relationships between words are critical.
Why Words Need Numerical Representations
Before word embeddings, words were often represented using one-hot encoding, where each word is a sparse vector with a single '1' and the rest '0's. This approach is inefficient because it treats words as isolated entities, ignoring their relationships and context. For example, 'king' and 'queen' would be as dissimilar as 'king' and 'apple' in one-hot encoding, despite their semantic connection. Word embeddings solve this by mapping words to dense vectors in a continuous space, where similar words are closer together. This transformation is crucial because machine learning models, including neural networks, operate on numerical data. By converting words into vectors, we enable models to process text mathematically, capturing nuances like analogies (e.g., 'king' - 'man' + 'woman' ≈ 'queen'). The shift from sparse to dense representations also reduces computational overhead, making it feasible to train models on large datasets.
# One-hot encoding example: inefficient and ignores word relationships
vocab = ['king', 'queen', 'man', 'woman', 'apple']
vocab_size = len(vocab)
# Create a dictionary to map words to indices
word_to_index = {word: idx for idx, word in enumerate(vocab)}
# One-hot encode a word
def one_hot_encode(word):
vector = [0] * vocab_size
vector[word_to_index[word]] = 1
return vector
print("One-hot encoding for 'king':", one_hot_encode('king'))
print("One-hot encoding for 'queen':", one_hot_encode('queen'))
# Output shows no relationship between 'king' and 'queen' despite their semantic similarityThe Distributional Hypothesis: Foundation of Word Embeddings
The distributional hypothesis states that words appearing in similar contexts tend to have similar meanings. This idea is the backbone of word embeddings, as it allows us to infer semantic relationships from large corpora of text. For instance, if 'coffee' and 'tea' frequently appear in sentences with words like 'drink', 'morning', or 'cup', the hypothesis suggests they are semantically related. Word embeddings leverage this by training models to predict words based on their context or vice versa. The key insight is that the meaning of a word is defined by the company it keeps, not by its dictionary definition. This approach enables embeddings to capture subtle relationships, such as synonyms, antonyms, or even analogies, without explicit supervision. For example, the model learns that 'Paris' is to 'France' as 'Berlin' is to 'Germany' because these pairs appear in similar contexts across the training data.
# Simple co-occurrence matrix to illustrate the distributional hypothesis
from collections import defaultdict
import numpy as np
# Sample corpus: sentences where words appear in similar contexts
corpus = [
"i drink coffee in the morning",
"i drink tea in the morning",
"she enjoys coffee with breakfast",
"he enjoys tea with breakfast",
"paris is the capital of france",
"berlin is the capital of germany"
]
# Build a vocabulary and co-occurrence matrix
vocab = set()
for sentence in corpus:
vocab.update(sentence.split())
vocab = list(vocab)
vocab_size = len(vocab)
word_to_index = {word: idx for idx, word in enumerate(vocab)}
# Initialize co-occurrence matrix
co_occurrence = np.zeros((vocab_size, vocab_size))
# Populate the matrix: count how often words appear together in a window
window_size = 2
for sentence in corpus:
words = sentence.split()
for i, word in enumerate(words):
for j in range(max(0, i - window_size), min(len(words), i + window_size + 1)):
if i != j:
co_occurrence[word_to_index[word], word_to_index[words[j]]] += 1
print("Co-occurrence matrix (rows/columns: vocab words):")
print(co_occurrence)
# Notice how 'coffee' and 'tea' have similar co-occurrence patternsWord2Vec: Learning Embeddings with Shallow Neural Networks
Word2Vec is a framework for learning word embeddings using shallow neural networks. It comes in two flavors: Continuous Bag of Words (CBOW) and Skip-gram. CBOW predicts a target word from its context words, while Skip-gram does the opposite, predicting context words from a target word. Both methods rely on the idea that words with similar meanings will have similar contexts, and thus, their embeddings will be close in the vector space. The training process involves adjusting the embeddings to maximize the probability of observing the actual context words given a target word (or vice versa). For example, if the target word is 'king', the model might adjust the embeddings so that words like 'queen', 'monarch', or 'palace' are more likely to appear in its context. The beauty of Word2Vec lies in its simplicity and efficiency, as it can be trained on large corpora using stochastic gradient descent. The resulting embeddings capture linear relationships, enabling analogies like 'king' - 'man' + 'woman' ≈ 'queen' to emerge naturally from the data.
# Word2Vec Skip-gram example using gensim (a popular NLP library)
from gensim.models import Word2Vec
# Sample corpus: sentences split into lists of words
sentences = [
["i", "drink", "coffee", "in", "the", "morning"],
["i", "drink", "tea", "in", "the", "morning"],
["she", "enjoys", "coffee", "with", "breakfast"],
["he", "enjoys", "tea", "with", "breakfast"],
["paris", "is", "the", "capital", "of", "france"],
["berlin", "is", "the", "capital", "of", "germany"]
]
# Train a Word2Vec model
model = Word2Vec(sentences, vector_size=10, window=2, min_count=1, sg=1) # sg=1 for Skip-gram
# Get the embedding for 'coffee'
coffee_embedding = model.wv['coffee']
print("Embedding for 'coffee':", coffee_embedding)
# Find similar words based on cosine similarity
similar_words = model.wv.most_similar('coffee', topn=3)
print("Words similar to 'coffee':", similar_words)
# Demonstrate the analogy: king - man + woman ≈ queen
try:
analogy_result = model.wv.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
print("Analogy result (king - man + woman ≈):", analogy_result)
except:
print("Analogy not possible with this small corpus, but the concept is illustrated")GloVe: Global Vectors for Word Representation
GloVe (Global Vectors for Word Representation) takes a different approach to learning word embeddings by leveraging global co-occurrence statistics from a corpus. Unlike Word2Vec, which relies on local context windows, GloVe uses the entire co-occurrence matrix to capture relationships between words. The key idea is that the ratio of co-occurrence probabilities between words can encode meaningful semantic information. For example, the ratio of the probability that 'ice' co-occurs with 'solid' versus 'steam' co-occurs with 'solid' is higher than the ratio for 'gas', reflecting the physical states of water. GloVe formulates the learning problem as a weighted least squares regression, where the objective is to minimize the difference between the dot product of word vectors and the logarithm of their co-occurrence probability. This approach ensures that the embeddings capture both local and global patterns in the data. GloVe embeddings often perform better on tasks requiring broader context, such as named entity recognition or part-of-speech tagging, because they incorporate more information about word relationships across the entire corpus.
# GloVe-like co-occurrence matrix factorization (simplified)
import numpy as np
from scipy.sparse.linalg import svds
# Reuse the co-occurrence matrix from the earlier example
co_occurrence_matrix = co_occurrence # From the distributional hypothesis section
# Apply logarithmic scaling to the co-occurrence counts
log_co_occurrence = np.log(co_occurrence_matrix + 1) # +1 to avoid log(0)
# Factorize the matrix using SVD (Singular Value Decomposition)
# This is a simplified version of what GloVe does
U, S, Vt = svds(log_co_occurrence, k=5) # k is the embedding dimension
# The word embeddings are the rows of U (or columns of Vt, depending on implementation)
word_embeddings = U
# Get the embedding for 'coffee' (assuming it's the first word in vocab)
coffee_index = word_to_index['coffee']
print("GloVe-like embedding for 'coffee':", word_embeddings[coffee_index])
# Note: This is a simplified demonstration. Actual GloVe uses weighted least squares.FastText: Handling Subword Information and Rare Words
FastText extends the Word2Vec framework by incorporating subword information, which allows it to handle rare words and morphological variations more effectively. Instead of treating each word as an atomic unit, FastText represents words as bags of character n-grams. For example, the word 'unhappiness' might be broken down into n-grams like 'un', 'nh', 'ha', 'pp', 'pi', 'in', 'ne', 'es', 'ss', and the full word itself. This approach enables the model to share information across words with common roots or affixes, improving generalization. For instance, the embeddings for 'happy' and 'unhappy' will be influenced by their shared n-grams, capturing their semantic relationship. FastText is particularly useful for languages with rich morphology or when dealing with out-of-vocabulary words, as it can generate embeddings for unseen words by combining the embeddings of their n-grams. This makes it robust in real-world applications where rare or misspelled words are common. Additionally, FastText can be trained on smaller datasets while still producing high-quality embeddings, thanks to its ability to leverage subword information.
# FastText example using gensim
from gensim.models import FastText
# Sample corpus with morphological variations
sentences = [
["happy", "people", "smile", "often"],
["unhappy", "people", "frown", "often"],
["happiness", "is", "contagious"],
["unhappiness", "is", "not"],
["i", "am", "happy", "today"],
["she", "was", "unhappy", "yesterday"]
]
# Train a FastText model
model = FastText(sentences, vector_size=10, window=2, min_count=1, sg=1, min_n=3, max_n=6) # min_n/max_n for n-gram range
# Get the embedding for 'unhappiness' (a rare word)
unhappiness_embedding = model.wv['unhappiness']
print("Embedding for 'unhappiness':", unhappiness_embedding)
# Find similar words based on subword information
similar_words = model.wv.most_similar('unhappiness', topn=3)
print("Words similar to 'unhappiness':", similar_words)
# Demonstrate handling of out-of-vocabulary (OOV) words
oov_word = "happify"
oov_embedding = model.wv[oov_word] # FastText can generate embeddings for OOV words
print(f"Embedding for OOV word '{oov_word}':", oov_embedding)Key points
- Word embeddings convert words into dense numerical vectors, enabling machines to process and understand language mathematically.
- The distributional hypothesis underpins word embeddings by asserting that words with similar contexts share similar meanings.
- Word2Vec uses shallow neural networks to learn embeddings by predicting words from their context or vice versa, capturing semantic relationships efficiently.
- GloVe leverages global co-occurrence statistics to create embeddings that reflect broader patterns in the corpus, often outperforming Word2Vec in tasks requiring extensive context.
- FastText improves upon Word2Vec by incorporating subword information, allowing it to handle rare words, morphological variations, and out-of-vocabulary terms effectively.
- Embeddings like Word2Vec, GloVe, and FastText enable analogical reasoning, such as solving 'king' - 'man' + 'woman' ≈ 'queen', by capturing linear relationships in vector space.
- The choice between Word2Vec, GloVe, and FastText depends on the specific NLP task, dataset size, and language characteristics, such as morphological complexity.
- Word embeddings are foundational for large language models, as they provide the initial numerical representations that models use to learn higher-level linguistic patterns.
Common mistakes
- Mistake: Assuming Word2Vec's skip-gram and CBOW are interchangeable for all tasks. Why it's wrong: Skip-gram works better for rare words and large datasets, while CBOW is faster and better for frequent words. Fix: Choose the architecture based on your dataset size, vocabulary frequency, and computational resources when building your LLM's embedding layer.
- Mistake: Ignoring context window size when training word embeddings. Why it's wrong: A small window (e.g., 2-5) captures syntactic relationships, while a large window (e.g., 10+) captures semantic relationships. Fix: Experiment with window sizes to align with your LLM's downstream tasks (e.g., smaller for grammar-heavy tasks, larger for meaning-based tasks).
- Mistake: Using pre-trained GloVe embeddings without fine-tuning for domain-specific LLMs. Why it's wrong: GloVe embeddings are trained on general corpora (e.g., Wikipedia) and may not capture domain-specific nuances. Fix: Fine-tune GloVe embeddings on your LLM's target corpus or train from scratch if the domain is highly specialized.
- Mistake: Treating FastText as identical to Word2Vec for subword information. Why it's wrong: FastText includes subword (n-gram) embeddings, which help with rare/misspelled words, while Word2Vec does not. Fix: Use FastText when your LLM needs to handle morphologically rich languages or noisy text (e.g., social media data).
- Mistake: Evaluating embeddings solely on intrinsic tasks (e.g., word similarity) without extrinsic validation. Why it's wrong: Intrinsic tasks don't guarantee performance in downstream LLM tasks (e.g., text generation, classification). Fix: Always validate embeddings on your LLM's target task (e.g., perplexity in language modeling) before finalizing the embedding layer.
Interview questions
What are word embeddings, and why are they important when creating your own large language model?
Word embeddings are dense vector representations of words that capture semantic and syntactic relationships. They are crucial when building a large language model because they allow the model to understand context and meaning beyond simple word frequencies. Instead of treating words as discrete symbols, embeddings map them to continuous vector spaces where similar words are closer together. This improves the model's ability to generalize, handle synonyms, and perform tasks like analogy solving. For example, in the vector space, 'king' - 'man' + 'woman' might approximate 'queen,' showing how embeddings encode relationships. Without embeddings, LLMs would struggle with understanding nuances in language, leading to poorer performance on downstream tasks.
How does Word2Vec work, and what are its two main architectures?
Word2Vec is a shallow neural network model that learns word embeddings by predicting words based on their context. It has two main architectures: Continuous Bag of Words (CBOW) and Skip-gram. In CBOW, the model predicts a target word from its surrounding context words, while Skip-gram does the opposite—it predicts context words from a target word. Both architectures use a sliding window to define context. For example, in Skip-gram, given the sentence 'the cat sat on the mat,' if the target word is 'sat,' the context words might be 'cat,' 'on,' and 'the.' The model adjusts the word vectors to maximize the probability of predicting these context words. Word2Vec is efficient and scalable, making it popular for pre-training embeddings in LLMs.
Explain how GloVe differs from Word2Vec in terms of training objectives.
GloVe, or Global Vectors for Word Representation, differs from Word2Vec by focusing on global co-occurrence statistics rather than local context windows. While Word2Vec uses a predictive approach (e.g., Skip-gram or CBOW), GloVe leverages a count-based method. It constructs a co-occurrence matrix from the entire corpus, where each entry represents how often two words appear together. The model then learns embeddings by minimizing the difference between the dot product of word vectors and the logarithm of their co-occurrence probability. This approach captures global patterns, such as how often 'ice' and 'steam' appear together, which Word2Vec might miss if they rarely appear in the same local context. GloVe often produces embeddings that better reflect semantic relationships because it considers the entire corpus.
Why would you choose FastText over Word2Vec or GloVe when building your own LLM?
FastText improves upon Word2Vec and GloVe by incorporating subword information, which is especially useful for handling rare words, misspellings, or morphologically rich languages. Unlike Word2Vec and GloVe, which treat words as atomic units, FastText breaks words into character n-grams (e.g., 'where' might be split into 'wh,' 'her,' 'ere'). This allows the model to share representations across words with similar subword structures, like 'running' and 'runner.' For example, if the corpus contains 'unhappiness' but not 'happiness,' FastText can still generate a meaningful embedding for 'happiness' by leveraging the subword 'happi.' This makes FastText more robust for LLMs, particularly in languages with complex word forms or limited training data. It also helps with out-of-vocabulary words, a common challenge in real-world applications.
Compare the strengths and weaknesses of Word2Vec and GloVe for pre-training embeddings in an LLM.
Word2Vec and GloVe both produce high-quality word embeddings, but they have distinct strengths and weaknesses for LLM pre-training. Word2Vec excels in capturing local context relationships because it uses a sliding window to predict words. This makes it efficient for large corpora and good at learning syntactic similarities, like pluralization or verb tenses. However, it ignores global co-occurrence statistics, which can limit its ability to capture broader semantic patterns. GloVe, on the other hand, leverages global statistics, making it better at capturing semantic relationships like analogies (e.g., 'king' to 'queen' as 'man' to 'woman'). However, GloVe requires constructing a co-occurrence matrix, which can be memory-intensive for very large corpora. For LLMs, Word2Vec might be preferable if training speed and scalability are priorities, while GloVe could be better if semantic understanding is critical. FastText could be a compromise, offering subword benefits while maintaining efficiency.
How would you integrate pre-trained word embeddings like Word2Vec or GloVe into your own LLM, and what challenges might you face?
To integrate pre-trained embeddings like Word2Vec or GloVe into an LLM, you would typically initialize the embedding layer of your model with the pre-trained vectors. For example, in PyTorch, you might load the embeddings using `torch.nn.Embedding.from_pretrained()` and set `freeze=False` to allow fine-tuning during training. This gives your LLM a strong starting point, reducing the need to learn embeddings from scratch. However, challenges include vocabulary mismatch—if the pre-trained embeddings don’t cover all words in your corpus, you’ll need to handle out-of-vocabulary words, perhaps by averaging subword embeddings or using a fallback strategy. Another challenge is domain adaptation; pre-trained embeddings might not align with your LLM’s specific domain (e.g., medical or legal text). You may need to fine-tune the embeddings on your corpus to improve performance. Finally, memory constraints can arise if the embedding matrix is very large, requiring techniques like dimensionality reduction or quantization.
Check yourself
1. When building an LLM for a domain with many rare words (e.g., medical texts), which Word2Vec architecture is most suitable and why?
- A.CBOW, because it averages context words and is faster for frequent terms
- B.Skip-gram, because it predicts context from a target word and handles rare words better
- C.Either, since both architectures perform identically for rare words
- D.Neither, because Word2Vec cannot handle rare words at all
Show answer
B. Skip-gram, because it predicts context from a target word and handles rare words better
The correct answer is Skip-gram. Skip-gram trains on individual word-context pairs, giving rare words more exposure during training, which is critical for domains with specialized vocabulary. CBOW averages context vectors, which dilutes the signal for rare words. The other options are incorrect because Word2Vec *can* handle rare words (especially Skip-gram), and the architectures are not identical in performance.
2. Your LLM needs to generate text in a morphologically rich language (e.g., Finnish). Why might FastText be a better choice than GloVe for the embedding layer?
- A.FastText includes subword (n-gram) embeddings, which capture morphological variations and improve handling of rare words
- B.GloVe cannot be trained on non-English languages, making FastText the only viable option
- C.FastText is always faster to train than GloVe, regardless of the language
- D.GloVe embeddings are limited to 50 dimensions, while FastText supports higher dimensions
Show answer
A. FastText includes subword (n-gram) embeddings, which capture morphological variations and improve handling of rare words
The correct answer is that FastText includes subword embeddings. This allows it to represent rare or morphologically complex words (e.g., conjugations, compounds) by breaking them into n-grams, which is critical for morphologically rich languages. The other options are wrong: GloVe works for any language, training speed depends on implementation, and GloVe supports arbitrary dimensions.
3. You observe that your LLM's embeddings perform well on word similarity tasks but poorly on a downstream classification task. What is the most likely explanation?
- A.The embeddings were trained with a window size too large for the classification task's syntactic needs
- B.Intrinsic evaluation (e.g., word similarity) does not correlate with extrinsic performance (e.g., classification)
- C.The classification task requires a different vocabulary than the one used to train the embeddings
- D.The embeddings were not fine-tuned on the classification task's domain-specific corpus
Show answer
D. The embeddings were not fine-tuned on the classification task's domain-specific corpus
The correct answer is that the embeddings were not fine-tuned on the domain-specific corpus. Intrinsic tasks (e.g., word similarity) often don't reflect performance on downstream tasks, especially if the embeddings lack domain-specific knowledge. While the other options may contribute, fine-tuning is the most direct fix for this mismatch. Window size and vocabulary are secondary concerns.
4. During LLM training, you notice that words with similar meanings (e.g., 'happy' and 'joyful') are close in the embedding space, but words with opposite meanings (e.g., 'happy' and 'sad') are also close. What is the most likely cause?
- A.The embedding layer was trained with a very small context window, capturing only syntactic relationships
- B.The embedding layer was trained with a very large context window, causing antonyms to appear in similar contexts
- C.The LLM's loss function penalizes dissimilarity too aggressively, forcing all words to cluster together
- D.The vocabulary size is too small, limiting the embedding space's ability to separate meanings
Show answer
B. The embedding layer was trained with a very large context window, causing antonyms to appear in similar contexts
The correct answer is a very large context window. Large windows capture broad semantic relationships, which can cause antonyms (e.g., 'happy' and 'sad') to appear in similar contexts (e.g., 'I feel ___ today'). Small windows focus on syntax, and the other options don't directly explain the clustering of opposites. The loss function and vocabulary size would affect overall performance, not specifically antonyms.
5. You are designing an LLM for a social media platform where text contains many misspellings (e.g., 'goood' instead of 'good'). Which embedding method is most robust to such noise?
- A.Word2Vec, because it uses negative sampling to handle noisy data
- B.GloVe, because it relies on global co-occurrence statistics, which are less sensitive to typos
- C.FastText, because it uses subword embeddings to represent misspelled words as combinations of n-grams
- D.All methods perform equally poorly on misspelled words, so preprocessing is the only solution
Show answer
C. FastText, because it uses subword embeddings to represent misspelled words as combinations of n-grams
The correct answer is FastText. Its subword embeddings allow it to represent misspelled words (e.g., 'goood') as combinations of n-grams (e.g., 'goo', 'ood'), making it robust to noise. Word2Vec and GloVe treat words as atomic units and fail on misspellings. While preprocessing helps, FastText is specifically designed to handle such cases without it.