Caching Strategies — CDN, Redis, Memcached

The Core Concept: Spatial and Temporal Locality

At the heart of caching lies the principle of locality: if you access a piece of data once, you are statistically likely to access it again soon (temporal locality), and if you access one piece of data, you are likely to access nearby data (spatial locality). A cache works by intercepting a request before it reaches the origin server. When a request arrives, the system checks the cache; if the key exists, it returns the value (a cache hit), which is orders of magnitude faster than a database query. If the key is missing (a cache miss), the system fetches data from the persistent store and populates the cache for future requests. Understanding this flow is vital because every cache adds complexity regarding data consistency; you must decide how long a piece of data should 'live' in memory before it is considered stale, a concept known as Time-to-Live (TTL).

# Simple dictionary-based cache simulation
cache = {}

def get_data(key, fetch_func):
    if key in cache:
        return cache[key] # Return hit immediately
    value = fetch_func(key) # Fetch from origin
    cache[key] = value # Populate for next time
    return value

CDN: Edge Caching for Static Assets

Content Delivery Networks (CDNs) act as the first line of defense, sitting at the 'edge' of the internet, geographically closer to the end-user than your central server. CDNs are optimized for static assets like images, stylesheets, and scripts. By caching these assets globally, a user in Tokyo fetches their content from a nearby Japanese edge node rather than crossing the ocean to your primary server in California. This drastically reduces round-trip time (RTT). The logic here is that static content rarely changes; thus, we can set long expiration times. CDNs use a distributed architecture to minimize network latency. When considering a CDN, think about your traffic distribution: if your users are global, a CDN is mandatory to ensure uniform performance. You must manage cache invalidation carefully; if you update a global file, you need a mechanism to purge the edge nodes so users don't see outdated versions.

# Simulate a CDN cache hit vs miss scenario
edge_nodes = {'us-east': {}, 'eu-west': {}}

def serve_asset(node, asset_url):
    if asset_url in edge_nodes[node]:
        return "Serving from Edge: " + edge_nodes[node][asset_url]
    # Simulate fetching from origin
    data = "Global File Content"
    edge_nodes[node][asset_url] = data
    return "Serving from Origin: " + data

Memcached: Distributed Memory Object Caching

Memcached is a high-performance, distributed memory object caching system designed for simplicity. It is an in-memory key-value store that excels in scenarios where you need to cache the results of expensive database queries or API calls. Unlike a database, Memcached does not provide persistent storage; if the server restarts, the data is gone. Its strength lies in its multi-threaded architecture, allowing it to scale horizontally by adding more nodes. Because it is distributed, your client application must be 'cache-aware'—typically using consistent hashing to determine which specific Memcached node should store a given key. Use Memcached when you have a simple key-value workload that requires extremely low latency and high concurrency, but do not need advanced data structures or data persistence features beyond simple expiration logic.

import hashlib

class MemcachedCluster:
    def __init__(self, nodes):
        self.nodes = nodes # List of server addresses

    def get_node(self, key):
        # Consistent hashing logic to route key to node
        h = int(hashlib.md5(key.encode()).hexdigest(), 16)
        return self.nodes[h % len(self.nodes)]

# Node lookup for distributed caching

Redis: Advanced Data Structures and Persistence

Redis, or Remote Dictionary Server, is significantly more powerful than Memcached. While both act as in-memory stores, Redis supports complex data structures such as lists, sets, sorted sets, and hashes, allowing you to perform operations directly in memory that would otherwise require pulling the whole object back to the application. For example, using a sorted set in Redis, you can implement a leaderboard feature by storing user scores and ranking them entirely within the cache layer. Redis also offers persistence, allowing it to take snapshots of the data or write updates to an append-only file, ensuring that your cache does not lose its state during a reboot. Because of this persistence and the ability to perform atomic operations on data structures, Redis is often used not just as a cache, but as a secondary primary data store for transient data like session management or real-time counters.

# Redis-style sorted set operation
redis_cache = {'leaderboard': [('user_a', 50), ('user_b', 120)]}

def update_score(user, score):
    # Sort the list after adding an element
    redis_cache['leaderboard'].append((user, score))
    redis_cache['leaderboard'].sort(key=lambda x: x[1], reverse=True)
    return redis_cache['leaderboard']

Cache Eviction and Consistency Policies

Since cache memory is finite, you must decide what happens when the cache fills up. This is the 'eviction policy.' The most common strategy is Least Recently Used (LRU), which discards the item that has not been accessed for the longest time, assuming that older items are less relevant. Other strategies include First-In-First-Out (FIFO) or Least Frequently Used (LFU). Beyond eviction, you must handle the 'write-through' versus 'write-back' dilemma. In a write-through cache, the application writes to the cache and the database simultaneously, ensuring consistency but adding latency. In write-back, the application writes only to the cache, and the cache updates the database asynchronously. Write-back is faster but carries the risk of data loss if the cache server fails before syncing. Choose these policies based on your tolerance for stale data versus your requirement for write performance.

from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity):
        self.cache = OrderedDict()
        self.capacity = capacity

    def put(self, key, value):
        if key in self.cache: self.cache.move_to_end(key)
        self.cache[key] = value
        if len(self.cache) > self.capacity:
            self.cache.popitem(last=False) # Evict oldest

Interview questions

All System Design interview questions →

Check yourself

Take the full System Design quiz →