Foundations of Machine Learning and NLP
Tokenization and Text Preprocessing Techniques
Tokenization and text preprocessing are the foundational steps that convert raw text into numerical representations a language model can process. Without proper tokenization, models struggle with vocabulary mismatches, rare words, and structural inconsistencies, leading to poor performance. You reach for these techniques whenever you handle raw text data, whether for training, fine-tuning, or inference in your own LLM.
Why Tokenization Exists: From Characters to Meaningful Units
Tokenization is the process of breaking text into smaller units called tokens, which serve as the basic building blocks for language models. The core challenge is balancing granularity and efficiency: splitting text into individual characters preserves all information but loses semantic meaning, while splitting into full words captures meaning but explodes vocabulary size. Tokens act as a middle ground, enabling models to learn patterns at a manageable scale. For example, the word 'unhappiness' might be split into 'un', 'happi', 'ness' to share knowledge across related words. This trade-off is critical because language models operate on fixed-size vocabularies, and rare or unseen words must be represented using subword units. The choice of tokenization directly impacts the model's ability to generalize, handle rare words, and process text efficiently during training and inference.
# Example: Character-level tokenization (simplest form)
text = "Tokenization matters."
char_tokens = list(text) # Split into individual characters
print("Character tokens:", char_tokens)
# Output: ['T', 'o', 'k', 'e', 'n', 'i', 'z', 'a', 't', 'i', 'o', 'n', ' ', 'm', 'a', 't', 't', 'e', 'r', 's', '.']
# Why this works: Preserves all input information but loses word-level semantics.
# Use case: Rare for LLMs, but useful for understanding the extremes of tokenization.Word-Level Tokenization: The Naive Approach
Word-level tokenization splits text into whole words, treating each word as a distinct token. This approach is intuitive because it aligns with how humans read and write, and it preserves semantic meaning by keeping words intact. However, it suffers from two major limitations: vocabulary explosion and out-of-vocabulary (OOV) words. As the training corpus grows, the vocabulary size can balloon into millions of entries, making the model's embedding layer computationally expensive. Additionally, any word not seen during training becomes an OOV token, forcing the model to treat it as a generic 'unknown' symbol. This is problematic for languages with rich morphology or domains with specialized terminology. Despite these drawbacks, word-level tokenization is still used in simpler models or as a baseline for comparison.
# Example: Word-level tokenization with basic cleaning
import re
def word_tokenize(text):
# Lowercase and split on whitespace, then remove punctuation
text = text.lower()
words = re.findall(r'\b\w+\b', text) # Matches word boundaries
return words
text = "Tokenization, like preprocessing, isn't trivial!"
word_tokens = word_tokenize(text)
print("Word tokens:", word_tokens)
# Output: ['tokenization', 'like', 'preprocessing', 'isnt', 'trivial']
# Why this works: Simple and interpretable, but fails on rare/hyphenated words.
# Note: 'isn\'t' becomes 'isnt' due to punctuation handling.Subword Tokenization: The Best of Both Worlds
Subword tokenization addresses the limitations of word-level tokenization by breaking words into smaller, meaningful units. This approach allows the model to handle rare words by composing them from frequent subwords, while also keeping the vocabulary size manageable. For example, the word 'tokenization' might be split into 'token', 'ization', enabling the model to share knowledge between 'token' and 'tokens'. The most common subword algorithms are Byte Pair Encoding (BPE), WordPiece, and Unigram. These methods start with a base vocabulary (e.g., characters) and iteratively merge the most frequent pairs of tokens to form subwords. The key insight is that subword tokenization leverages the statistical properties of language: frequent words remain intact, while rare words are split into subwords that appear elsewhere in the corpus. This balance improves generalization and reduces OOV issues.
# Example: Byte Pair Encoding (BPE) tokenization (simplified)
from collections import defaultdict
def get_stats(vocab):
pairs = defaultdict(int)
for word, freq in vocab.items():
symbols = word.split()
for i in range(len(symbols)-1):
pairs[symbols[i], symbols[i+1]] += freq
return pairs
def merge_vocab(pair, v_in):
v_out = {}
bigram = ' '.join(pair)
replacement = ''.join(pair)
for word in v_in:
w_out = word.replace(bigram, replacement)
v_out[w_out] = v_in[word]
return v_out
# Base vocabulary: characters with frequencies
vocab = {
't o k e n i z a t i o n': 10,
't o k e n': 5,
'p r e p r o c e s s i n g': 8
}
# Perform one merge step
pairs = get_stats(vocab)
best_pair = max(pairs, key=pairs.get)
vocab = merge_vocab(best_pair, vocab)
print("Vocabulary after one merge:", vocab)
# Output: Merges the most frequent pair (e.g., 't o' -> 'to')
# Why this works: Iteratively builds subwords from frequent character pairs.Handling Special Cases: Numbers, Punctuation, and Whitespace
Text preprocessing must handle edge cases like numbers, punctuation, and whitespace consistently to avoid introducing noise into the model. Numbers are particularly tricky because they can appear in many forms (e.g., '100', '100.5', '100%'), and treating them as distinct tokens may not generalize well. A common approach is to normalize numbers by replacing digits with a placeholder (e.g., '100' → '<NUM>') or splitting them into individual digits. Punctuation poses similar challenges: while some punctuation marks (like periods) may indicate sentence boundaries, others (like apostrophes) are part of contractions. The solution is to either split punctuation into separate tokens or attach them to adjacent words, depending on the use case. Whitespace normalization is equally important, as inconsistent spacing (e.g., multiple spaces or tabs) can lead to spurious tokens. The goal is to ensure that semantically equivalent inputs produce identical token sequences, reducing noise during training.
# Example: Normalizing numbers, punctuation, and whitespace
import re
def preprocess_text(text):
# Normalize whitespace: replace multiple spaces/tabs with single space
text = re.sub(r'\s+', ' ', text).strip()
# Replace numbers with <NUM> placeholder
text = re.sub(r'\d+(\.\d+)?', '<NUM>', text)
# Split punctuation (except apostrophes in contractions)
text = re.sub(r'([^\w\s\'])', r' \1 ', text)
# Clean up extra spaces from punctuation splitting
text = re.sub(r'\s+', ' ', text).strip()
return text
text = "The price is $100.50, but 50% off!"
processed_text = preprocess_text(text)
print("Processed text:", processed_text)
# Output: "The price is $ <NUM> , but <NUM> % off !"
# Why this works: Ensures consistent handling of numbers/punctuation across inputs.
# Note: '$' is treated as punctuation and split from the number.Building a Tokenizer Pipeline: Putting It All Together
A complete tokenizer pipeline combines multiple preprocessing steps into a cohesive workflow, ensuring that raw text is transformed into a sequence of tokens ready for the model. The pipeline typically includes: (1) text cleaning (removing unwanted characters), (2) normalization (lowercasing, number handling), (3) tokenization (splitting into subwords/words), and (4) post-processing (adding special tokens like [CLS] or [SEP]). The order of these steps matters: for example, lowercasing should happen before tokenization to avoid treating 'Word' and 'word' as distinct tokens. Special tokens are added to mark sentence boundaries, padding, or other structural elements. The pipeline must also handle edge cases like very long sequences by truncating or splitting them. A well-designed pipeline ensures that the model receives consistent, high-quality input, which is critical for training stability and inference accuracy. The final output is a sequence of token IDs, which are mapped to embeddings in the model's vocabulary.
# Example: Full tokenizer pipeline (simplified BPE + preprocessing)
import re
from collections import defaultdict
class SimpleBPETokenizer:
def __init__(self, vocab_size=100):
self.vocab_size = vocab_size
self.vocab = {}
self.merges = {}
self.special_tokens = {"[PAD]": 0, "[UNK]": 1}
def train(self, corpus):
# Step 1: Initialize vocabulary with characters
vocab = defaultdict(int)
for text in corpus:
for word in text.split():
vocab[' '.join(list(word))] += 1
# Step 2: Iteratively merge pairs
for i in range(self.vocab_size - len(self.special_tokens)):
pairs = defaultdict(int)
for word, freq in vocab.items():
symbols = word.split()
for j in range(len(symbols)-1):
pairs[symbols[j], symbols[j+1]] += freq
if not pairs:
break
best_pair = max(pairs, key=pairs.get)
vocab = self._merge_pair(best_pair, vocab)
self.merges[best_pair] = i
# Step 3: Build final vocabulary
self.vocab = {word: idx + len(self.special_tokens) for idx, word in enumerate(vocab)}
self.vocab.update(self.special_tokens)
def _merge_pair(self, pair, vocab):
new_vocab = {}
bigram = ' '.join(pair)
replacement = ''.join(pair)
for word in vocab:
new_word = word.replace(bigram, replacement)
new_vocab[new_word] = vocab[word]
return new_vocab
def tokenize(self, text):
# Step 1: Preprocess text
text = re.sub(r'\s+', ' ', text.lower()).strip()
text = re.sub(r'\d+', '<NUM>', text)
# Step 2: Split into words
words = re.findall(r'\b\w+\b', text)
# Step 3: Apply BPE merges
tokens = []
for word in words:
word = ' '.join(list(word))
for pair, _ in sorted(self.merges.items(), key=lambda x: x[1]):
word = word.replace(' '.join(pair), ''.join(pair))
tokens.extend(word.split())
# Step 4: Map to token IDs
token_ids = []
for token in tokens:
token_ids.append(self.vocab.get(token, self.special_tokens["[UNK]"]))
return token_ids
# Example usage
corpus = ["tokenization is important", "tokenize this text"]
tokenizer = SimpleBPETokenizer(vocab_size=50)
tokenizer.train(corpus)
tokens = tokenizer.tokenize("Tokenizing 123 numbers!")
print("Token IDs:", tokens)
# Output: Token IDs for the input text, e.g., [3, 5, 1, 2] (depends on merges)
# Why this works: Combines preprocessing, BPE, and special tokens into a pipeline.Key points
- Tokenization bridges the gap between raw text and numerical representations by breaking text into meaningful units called tokens.
- Word-level tokenization is intuitive but suffers from vocabulary explosion and out-of-vocabulary issues, limiting its scalability.
- Subword tokenization balances granularity and efficiency by splitting rare words into frequent subwords, improving generalization.
- Byte Pair Encoding (BPE) and similar algorithms iteratively merge frequent character pairs to build a subword vocabulary.
- Numbers, punctuation, and whitespace must be normalized consistently to avoid introducing noise into the model's input.
- A tokenizer pipeline combines preprocessing, tokenization, and post-processing steps to ensure high-quality input for the model.
- Special tokens like [PAD] or [UNK] are added during post-processing to handle padding and unknown words in the vocabulary.
- The choice of tokenization strategy directly impacts the model's ability to handle rare words, generalize, and process text efficiently.
Common mistakes
- Mistake: Skipping text normalization (lowercasing, removing accents) before tokenization. Why it's wrong: Inconsistent casing or diacritics can create duplicate tokens (e.g., 'Hello' vs 'hello'), bloating the vocabulary and reducing model efficiency. Fix: Apply normalization as a preprocessing step before tokenization.
- Mistake: Using whitespace tokenization for languages without clear word boundaries (e.g., Chinese, Japanese). Why it's wrong: Whitespace tokenization fails to segment characters into meaningful units, leading to poor model performance. Fix: Use language-specific tokenizers like Jieba for Chinese or MeCab for Japanese.
- Mistake: Ignoring subword tokenization (e.g., Byte-Pair Encoding) for rare or unseen words. Why it's wrong: Out-of-vocabulary (OOV) words break the model's ability to process text. Fix: Use subword tokenization to handle rare words by breaking them into known subword units.
- Mistake: Hardcoding a fixed vocabulary size without experimentation. Why it's wrong: Too small a vocabulary misses important tokens, while too large increases computational cost. Fix: Experiment with vocabulary sizes based on the dataset and task requirements.
- Mistake: Not handling special tokens (e.g., [PAD], [UNK], [CLS]) during tokenization. Why it's wrong: Missing special tokens can cause errors during training (e.g., padding mismatches) or inference (e.g., unknown token handling). Fix: Explicitly define and include special tokens in the tokenizer configuration.
Interview questions
What is tokenization, and why is it important when building your own large language model?
Tokenization is the process of breaking down raw text into smaller units called tokens, which can be words, subwords, or even individual characters. It's a critical first step in building a large language model because models don't understand raw text—they process numerical representations. Tokenization enables the model to handle vocabulary efficiently, manage out-of-vocabulary words through subword techniques, and maintain consistent input sizes. Without proper tokenization, the model would struggle with text normalization, leading to poor performance and inefficiency in training and inference. For example, in Python-like pseudocode, you might use a tokenizer like this: `tokens = tokenizer.encode('Hello world!')`, which converts text into token IDs the model can process.
Explain the difference between word-level and subword tokenization. Which one would you prefer for your LLM and why?
Word-level tokenization splits text into whole words, while subword tokenization breaks words into smaller units, like prefixes or suffixes. Word-level tokenization is simple and works well for languages with clear word boundaries, but it struggles with rare or out-of-vocabulary words, leading to a large vocabulary size. Subword tokenization, such as Byte Pair Encoding (BPE) or WordPiece, handles rare words better by breaking them into known subwords, reducing vocabulary size and improving generalization. For building an LLM, I'd prefer subword tokenization because it balances vocabulary efficiency and flexibility. For instance, BPE can tokenize 'unhappiness' into ['un', 'happiness'], allowing the model to reuse subwords and handle unseen words more effectively.
What is Byte Pair Encoding (BPE), and how does it work in the context of tokenization for LLMs?
Byte Pair Encoding (BPE) is a subword tokenization algorithm that iteratively merges the most frequent pairs of characters or subwords to build a vocabulary. It starts with a base vocabulary of individual characters and gradually combines them into larger subwords based on their frequency in the training data. For example, if 'th' appears frequently, BPE merges it into a single token. This process continues until the desired vocabulary size is reached. BPE is widely used in LLMs because it efficiently handles rare words and reduces vocabulary size while maintaining meaningful subword units. Here’s a simplified example: if the corpus contains 'low', 'lower', and 'newest', BPE might merge 'low' and 'er' into 'lower', allowing the model to generalize better to unseen words like 'newest'.
Why is text normalization important before tokenization, and what are some common normalization techniques?
Text normalization is crucial before tokenization because raw text often contains inconsistencies like different cases, punctuation, or special characters, which can confuse the model. Normalization ensures uniformity, reducing noise and improving the model's ability to learn patterns. Common techniques include converting text to lowercase to avoid case sensitivity, removing punctuation or special characters, expanding contractions (e.g., 'don't' to 'do not'), and handling Unicode characters. For example, normalizing 'Hello!' and 'hello?' to 'hello' ensures the model treats them as the same token. Without normalization, the model might treat 'Hello' and 'hello' as distinct tokens, wasting vocabulary space and reducing generalization. In code, you might normalize text like this: `text = text.lower().strip()` before tokenization.
Compare the trade-offs between using a fixed vocabulary size and a dynamic vocabulary size in your LLM's tokenizer.
A fixed vocabulary size, like in BPE or WordPiece, sets a maximum number of tokens during training, which simplifies model architecture and ensures consistent input dimensions. However, it may struggle with rare or unseen words, leading to out-of-vocabulary (OOV) tokens. A dynamic vocabulary size, such as in Unigram tokenization, adjusts the vocabulary based on the data, which can improve handling of rare words but complicates training and inference due to variable input sizes. For an LLM, a fixed vocabulary size is often preferred because it ensures stability during training and deployment, even if it means some OOV tokens. For example, BPE with a fixed vocabulary size of 50,000 tokens is common in models like GPT, as it balances coverage and efficiency. Dynamic vocabularies are riskier because they can lead to unpredictable behavior during inference.
How would you handle rare or out-of-vocabulary (OOV) words in your LLM's tokenizer, and why is this critical for model performance?
Handling rare or OOV words is critical because they can degrade model performance by introducing noise or forcing the model to rely on unknown tokens. Subword tokenization techniques like BPE or WordPiece are effective because they break rare words into known subwords, allowing the model to generalize. For example, the word 'unhappiness' might be split into ['un', 'happiness'], enabling the model to understand it even if it hasn't seen the full word before. Another approach is to use a special 'UNK' token for truly unknown words, but this should be a last resort. Additionally, pretraining on diverse datasets ensures the tokenizer learns a broad vocabulary. Without proper OOV handling, the model might generate incoherent outputs or fail to understand user inputs. For instance, if the tokenizer encounters 'quokka' and splits it into ['qu', 'okka'], the model can still process it meaningfully, whereas a word-level tokenizer might mark it as 'UNK' and lose context.
Check yourself
1. Why is subword tokenization (e.g., BPE) preferred over word-level tokenization for building an LLM?
- A.It reduces the vocabulary size to a fixed number of words, improving efficiency.
- B.It handles rare and unseen words by breaking them into known subword units, preserving semantic meaning.
- C.It eliminates the need for text normalization, simplifying preprocessing.
- D.It guarantees perfect tokenization for all languages, including those without whitespace.
Show answer
B. It handles rare and unseen words by breaking them into known subword units, preserving semantic meaning.
The correct answer is that subword tokenization handles rare and unseen words by breaking them into known subword units. This preserves semantic meaning and avoids out-of-vocabulary issues. The other options are incorrect: subword tokenization does not reduce vocabulary size to words (it often increases it for subwords), it does not eliminate normalization, and it does not guarantee perfect tokenization for all languages.
2. A student trains an LLM on a dataset where all text is lowercase but forgets to normalize new input text during inference. What is the most likely consequence?
- A.The model will fail to process the input entirely due to tokenization errors.
- B.The model will treat uppercase and lowercase versions of the same word as identical tokens.
- C.The model will create separate tokens for uppercase and lowercase versions, potentially reducing performance.
- D.The model will automatically normalize the input text during inference, avoiding any issues.
Show answer
C. The model will create separate tokens for uppercase and lowercase versions, potentially reducing performance.
The correct answer is that the model will create separate tokens for uppercase and lowercase versions, potentially reducing performance. Since the training data was lowercase, the tokenizer may not recognize uppercase variants, leading to out-of-vocabulary tokens or redundant entries. The other options are incorrect: the model won't fail entirely, it won't treat them as identical, and it won't normalize input automatically unless explicitly programmed to do so.
3. During preprocessing, a student removes all punctuation from the text. How might this affect the LLM's performance?
- A.It will improve performance by reducing vocabulary size and simplifying tokenization.
- B.It will have no effect, as punctuation is irrelevant for language modeling.
- C.It may harm performance by losing semantic or syntactic cues (e.g., question marks, commas).
- D.It will cause the tokenizer to fail, as punctuation is required for subword tokenization.
Show answer
C. It may harm performance by losing semantic or syntactic cues (e.g., question marks, commas).
The correct answer is that removing punctuation may harm performance by losing semantic or syntactic cues. Punctuation often carries meaning (e.g., questions vs statements) or affects sentence structure. The other options are incorrect: removing punctuation doesn't always improve performance, it is not irrelevant, and tokenizers can function without it.
4. A student uses a tokenizer with a vocabulary size of 50,000 for an LLM trained on a small dataset of 10,000 words. What is the most likely issue?
- A.The vocabulary size is too small, causing frequent out-of-vocabulary errors.
- B.The vocabulary size is too large, leading to inefficient memory usage and slower training.
- C.The vocabulary size is optimal, as it matches the dataset size exactly.
- D.The vocabulary size is irrelevant, as the tokenizer will dynamically adjust it.
Show answer
B. The vocabulary size is too large, leading to inefficient memory usage and slower training.
The correct answer is that the vocabulary size is too large, leading to inefficient memory usage and slower training. A large vocabulary increases embedding matrix size and computational cost without adding value for a small dataset. The other options are incorrect: 50,000 is not too small for 10,000 words, it doesn't match the dataset size, and tokenizers don't dynamically adjust vocabulary size.
5. Why is it important to include special tokens like [PAD] and [UNK] in an LLM's tokenizer?
- A.They are required for the model to generate text during inference.
- B.They handle padding for variable-length sequences and unknown tokens during training/inference.
- C.They replace all punctuation and whitespace in the input text.
- D.They ensure the tokenizer works for all languages, including those without subwords.
Show answer
B. They handle padding for variable-length sequences and unknown tokens during training/inference.
The correct answer is that special tokens handle padding for variable-length sequences ([PAD]) and unknown tokens ([UNK]). This ensures the model can process batches of text and handle out-of-vocabulary words. The other options are incorrect: special tokens aren't required for generation, they don't replace punctuation/whitespace, and they don't guarantee multilingual support.