Database Replication

Single-Leader Replication

Single-leader replication is the foundational architecture where all write operations must pass through a single primary node, while reads can be distributed across various follower nodes. The primary node is responsible for processing incoming write requests and propagating these changes to its replicas. Because only one node handles writes, the system avoids complex conflict resolution scenarios that arise in multi-master setups, making it the most predictable model for most applications. When a write occurs on the primary, it logs the change and ships it to followers, which then update their own local storage. The delay between the primary update and the follower update is known as replication lag. This lag is a critical design constraint, as it means followers may briefly show stale data. Understanding this trade-off is fundamental to designing distributed systems that balance consistency and performance.

# Simple simulation of primary write and log propagation
class DatabaseNode:
    def __init__(self, name): self.name = name; self.data = {}
    def write(self, key, value):
        self.data[key] = value
        print(f"{self.name} updated: {key}={value}")

primary = DatabaseNode("Primary")
replica = DatabaseNode("Replica-1")

# Synchronous write propagation
primary.write("user_id", 101)
replica.write("user_id", 101) # Simulated replication log applying change

Handling Replication Lag

Replication lag is the unavoidable temporal gap between the primary database committing a transaction and a follower applying the same change. In high-traffic systems, this window can grow, causing users to see stale data. A common symptom is 'read-your-own-writes' inconsistency, where a user updates their profile but sees the old version upon refreshing because their read request was routed to a lagging replica. To mitigate this, developers can implement session-based routing, ensuring a user's reads are directed to the primary for a short period after they perform a write. Alternatively, one can check the replica's replication status against the primary's last applied log index before serving a request. By understanding that replicas are eventually consistent, you can design application logic that handles these minor delays gracefully rather than assuming real-time global consistency across all database nodes.

# Check if replica is caught up before allowing read
def get_user_data(replica, replica_lsn, primary_lsn):
    # Compare Log Sequence Numbers (LSN) to ensure freshness
    if replica_lsn < primary_lsn:
        return "Error: Stale data, retrying on primary..."
    return replica.data.get("user_id")

# Simulating a scenario where replica is behind
print(get_user_data(replica, 10, 15)) # Returns stale error status

Multi-Leader Architectures

Multi-leader replication allows writes to occur on multiple nodes simultaneously, which is highly beneficial for geo-distributed systems where a user in Tokyo and a user in New York shouldn't have to wait for cross-continental round trips to perform a write. Each leader acts as a client to the other, asynchronously propagating write logs to keep nodes synchronized. The core complexity here is conflict resolution; if the same record is updated on two different leaders at roughly the same time, the system must reconcile the divergent states. Common strategies include 'last-write-wins' (using timestamps), or application-level conflict resolution where the system prompts the user to resolve the clash. While this architecture significantly improves write throughput and fault tolerance, it introduces significant complexity that must be managed by the application layer to maintain data integrity.

# Simulating conflict detection in multi-master
def update_record(node_id, key, value, timestamp):
    # Store value with timestamp for conflict resolution
    return {"val": value, "ts": timestamp}

node_a = update_record("A", "key1", "v1", 100)
node_b = update_record("B", "key1", "v2", 105) # Node B has a later timestamp

# Conflict resolution: Keep latest timestamp
final_state = node_b if node_b['ts'] > node_a['ts'] else node_a
print(f"Resolved state: {final_state['val']}")

Leaderless Replication

In leaderless replication, there is no designated primary node; instead, clients send write requests to multiple nodes in parallel. This decentralized approach removes the single point of failure inherent in single-leader models but complicates the process of reading data, as different nodes may contain conflicting versions of the same record. To handle this, reading is typically performed by querying a quorum of replicas. If you have five replicas and set your read quorum to three (R=3), you are statistically likely to receive the most recent version of the data. This requires nodes to store version vectors or timestamps to track the history of an object. Leaderless systems are exceptionally resilient and excel in environments where downtime is unacceptable, though they place the burden of reconciliation on the read-path logic and potentially return inconsistent results without proper quorum settings.

# Quorum read logic
def read_with_quorum(replicas, key):
    responses = [r.data.get(key) for r in replicas]
    # In a real system, return the value with the highest version/timestamp
    return max(responses, key=lambda x: x['version']) 

# Setup replicas with differing data
rep1 = {'data': {'val': 10, 'version': 1}}; rep2 = {'data': {'val': 10, 'version': 2}}
print(read_with_quorum([rep1, rep2], 'val'))

Failover and Consistency

Failover is the process of promoting a follower node to primary status when the current primary fails. This is a critical operation that must be handled with care to prevent 'split-brain' scenarios, where two nodes mistakenly believe they are the primary, leading to data corruption as they diverge. High-availability systems use consensus algorithms to manage the election of a new leader. Beyond failover, consistency models like 'strong consistency' ensure that once a write is acknowledged, all subsequent reads reflect that update, usually at the cost of latency. Systems often offer configurable consistency levels, allowing developers to choose between performance and strict accuracy. Designing a robust database layer requires mapping these replication strategies to your specific business requirements, ensuring that the cost of maintaining consistent data does not outweigh the benefits provided by the chosen replication architecture.

# Simple failover logic for leader election
class ClusterManager:
    def __init__(self, nodes):
        self.nodes = nodes
        self.primary = nodes[0]

    def failover(self):
        print(f"Primary {self.primary} down. Promoting new leader...")
        self.primary = self.nodes[1] # Simple promotion logic
        print(f"New primary is: {self.primary}")

manager = ClusterManager(["Node1", "Node2"])
manager.failover()

Interview questions

All System Design interview questions →

Check yourself

Take the full System Design quiz →