Load Balancing

The Core Concept: Traffic Distribution

At its simplest, a load balancer acts as an intelligent intermediary that sits in front of a group of servers. The primary goal is to ensure that compute resources are utilized efficiently by preventing any single server from being overwhelmed by requests while others sit idle. By acting as a reverse proxy, the load balancer receives the initial request from the client and forwards it to one of the available backend nodes based on a predefined algorithm. This separation of the public-facing entry point from the internal processing logic is crucial for reliability. If a backend server is overloaded or performing poorly, the load balancer can redirect traffic to healthier nodes. This mechanism enables the system to handle traffic fluctuations without requiring the client to know the internal topology of the server cluster, effectively decoupling the service consumption from the underlying infrastructure complexity.

# Simple Round Robin distribution logic
servers = ['server1', 'server2', 'server3']
request_count = 0

def get_next_server():
    global request_count
    # Pick server based on index modulo array size
    server = servers[request_count % len(servers)]
    request_count += 1
    return server

# Simulating distribution
for _ in range(5): print(get_next_server())

Algorithm Selection: Static vs Dynamic

The intelligence of a load balancer lies in its selection strategy. Static algorithms like Round Robin or Weighted Round Robin make decisions based on fixed configurations without considering the current state of the backend servers. While computationally efficient, these approaches often fail in heterogeneous environments where some servers might have more CPU or RAM than others. Dynamic algorithms, conversely, account for real-time metrics. Least Connections, for example, tracks the number of active sessions on each node, sending new traffic to the server with the lowest current load. This is superior for long-lived connections, such as web sockets or database streaming, where a simple round-robin might inadvertently send a new, intensive request to a server already struggling with complex background tasks. Understanding the relationship between request duration and server state allows architects to choose an algorithm that maximizes resource utilization and minimizes latency for the end user.

# Dynamic Least Connections tracking
server_connections = {'s1': 0, 's2': 5, 's3': 2}

def get_least_loaded():
    # Find the key with the minimum value
    return min(server_connections, key=server_connections.get)

# Select the optimal node based on real-time load
target = get_least_loaded()
print(f"Routing to: {target}")

Ensuring Health and Reliability

A load balancer is only useful if it knows which servers are actually capable of processing requests. If a load balancer continues to send traffic to a dead server, it creates a black hole where requests simply time out, degrading the user experience. To mitigate this, load balancers implement health checks—frequent probes that verify the operational status of backend services. These checks might include sending a simple ping, verifying that a specific heartbeat endpoint returns a 200 OK status, or checking the server's local resource utilization metrics. When a server fails these checks consecutively, it is marked as unhealthy and removed from the active rotation list. Once the server passes enough consecutive probes, it is automatically re-introduced. This automated lifecycle management is the backbone of high-availability systems, allowing for self-healing infrastructure that can tolerate individual component failures without manual intervention.

# Simple health check probe logic
server_status = {'s1': True, 's2': False}

def filter_healthy(pool):
    # Return only nodes that are marked as active
    return [s for s in pool if server_status.get(s, False)]

# Route only to functional nodes
print(f"Available: {filter_healthy(['s1', 's2'])}")

Session Persistence and State

In many web applications, the server needs to remember the state of a user across multiple requests. This is often handled through session tokens or cookies. If a load balancer sends a user's first request to Server A and their second request to Server B, the user might lose their session information unless the state is synchronized across all nodes. Session persistence, or 'sticky sessions,' solves this by ensuring that all requests from a specific client are routed to the same backend server for the duration of a session. While convenient for legacy stateful architectures, it introduces a significant drawback: it can lead to uneven load distribution if certain clients generate significantly more traffic than others. Modern, scalable system design typically favors stateless backend services where state is stored in an external distributed cache rather than on the server, allowing the load balancer full freedom in distribution.

# Sticky session using a client identifier hash
def get_persistent_server(client_id, server_pool):
    # Hash the client_id to ensure the same server is always picked
    index = hash(client_id) % len(server_pool)
    return server_pool[index]

print(get_persistent_server("user_123", ['s1', 's2']))

Scaling Through Layers

As traffic volumes grow, even the load balancer itself can become a bottleneck. This is why architects often implement multi-layer load balancing. Layer 4 balancing works at the transport layer, focusing on IP addresses and ports to distribute traffic quickly with minimal overhead. Layer 7 balancing, however, inspects the actual content of the request, such as HTTP headers or URL paths, to make sophisticated routing decisions. By combining these, one might use a hardware load balancer or a high-performance router at Layer 4 to distribute initial traffic across a cluster of software-based Layer 7 proxies. This tiered approach allows the system to handle millions of concurrent connections by offloading TLS termination and path-based routing to specialized software instances while keeping the entry point lightweight and resilient. This architectural design pattern ensures that no single layer of the stack is forced to perform more processing than its capabilities allow.

# L7 Routing based on URL path
routes = {'/api': 'api_server', '/web': 'web_server'}

def route_request(path):
    # Inspect path to direct traffic
    return routes.get(path, 'default_server')

print(f"Directing to: {route_request('/api')}")

Interview questions

All System Design interview questions →

Check yourself

Take the full System Design quiz →