Load Testing APIs

Understanding Baseline Performance

Before attempting to stress your system, you must establish a baseline. Baseline performance refers to how your API behaves under a single, isolated request. Without knowing the 'perfect' response time—when there is no competition for CPU, memory, or network bandwidth—you cannot possibly measure the impact of concurrency. When you send one request, you are testing the efficiency of your code path, database indexing, and network overhead in a vacuum. If your baseline is already slow, adding load will only exacerbate the issue exponentially. We analyze the baseline by running a single request repeatedly to calculate the average response time. This establishes the theoretical limit of your system. Understanding this helps you distinguish between inherent latency (code overhead) and contention-induced latency (waiting for resources). Without a solid baseline, you are essentially guessing at the source of performance degradation when traffic inevitably surges in your production environment.

# Measure baseline latency for a single request
import time
import requests

def get_baseline(url, iterations=10):
    latencies = []
    for _ in range(iterations):
        start = time.time()
        requests.get(url) # Fetching resource
        latencies.append(time.time() - start)
    return sum(latencies) / iterations # Average latency

Simulating Concurrency with Threading

Once you have a baseline, the next step is to introduce concurrency. Concurrency simulates multiple users hitting your API endpoints at the exact same time. This is vastly different from sequential requests because it forces the application server and database to manage multiple execution contexts simultaneously. In a single-threaded environment, requests are queued, meaning latency is purely additive. However, in modern web servers, requests are handled in parallel, sharing finite hardware resources like file descriptors, database connections, and memory buffers. When you run concurrent tasks, you start to see performance degradation caused by context switching, lock contention on shared data, and thread starvation. By increasing the number of active threads, you move from testing logic to testing infrastructure limits. This reveals the 'saturation point' of your server, where adding one more user results in a non-linear increase in response time, often indicating that the system is exhausted and can no longer process requests efficiently.

import threading

def fire_request(url):
    # Concurrent worker to simulate a user session
    requests.get(url)

threads = [threading.Thread(target=fire_request, args=('http://api.local/data',)) for _ in range(50)]
for t in threads: t.start() # Dispatch 50 simultaneous requests
for t in threads: t.join()

Identifying Bottlenecks through Throughput

Throughput, typically measured in requests per second (RPS), is the ultimate metric for assessing how much traffic your API can handle before it fails to meet your service level objectives. As you increase the number of concurrent users, you will find that throughput eventually plateaus. This plateau is rarely arbitrary; it is usually dictated by the most constrained resource in your stack. It could be the disk I/O speed, CPU cycle availability, or a hard limit on the number of open database connections. When you observe throughput flattening despite increasing load, you have hit a bottleneck. To reason about this, consider the 'Little’s Law': the number of requests in a system is equal to the arrival rate multiplied by the time spent in the system. If the time spent increases due to resource waiting, throughput must decrease unless concurrency is increased. Finding this limit allows you to provision infrastructure precisely.

import time

# Measuring requests per second (RPS)
start_time = time.time()
for i in range(1000):
    requests.get('http://api.local/data')
total_time = time.time() - start_time
print(f"Throughput: {1000 / total_time} RPS") # Calculated capacity

Stress Testing for Failure Modes

Stress testing goes beyond normal traffic patterns to discover the point of complete system collapse. While load testing focuses on normal usage, stress testing asks, 'What happens when everything breaks?' This is vital because production environments suffer from unexpected spikes, DDoS attacks, or cascading failures where one failing service blocks others. By pushing the system until it returns status codes like 503 (Service Unavailable) or 504 (Gateway Timeout), you identify your failure modes. You want to ensure that your API fails gracefully, providing meaningful error messages rather than simply hanging or leaking sensitive stack traces. Stress testing also reveals if your circuit breakers are configured correctly—do they open when the latency exceeds the threshold, or do they remain closed and allow the system to get crushed? Understanding these failure modes ensures that your system remains resilient and recovers quickly once the underlying load is normalized or managed.

def stress_test(url, limit):
    # Keep hitting until we get a non-200 code
    for _ in range(limit):
        response = requests.get(url)
        if response.status_code >= 500:
            print(f"Failing at load: {response.status_code}")
            break

Analyzing Latency Percentiles

Relying solely on average latency is a dangerous trap because averages hide the experience of users at the extremes. If 95% of your users experience 100ms latency but 5% experience 5 seconds, your average will look acceptable, but your outlier performance is abysmal. This is why you must calculate percentiles, specifically the 95th (P95) and 99th (P99) percentiles. P99 latency tells you exactly how slow your slowest users are. When analyzing these during a load test, you will often find that the P99 latency grows much faster than the average. This disparity is usually caused by garbage collection pauses, lock contention, or slow database queries triggered by specific, non-cached requests. By monitoring these percentiles, you gain visibility into the tail latency of your system. A robust API design must aim to keep P99 latency within acceptable bounds, ensuring that even under heavy load, the experience remains consistent for all consumers.

import numpy as np

latencies = [0.1, 0.12, 0.11, 5.0] # Simulated high-load outcomes
p95 = np.percentile(latencies, 95)
p99 = np.percentile(latencies, 99)
print(f"P95: {p95}s, P99: {p99}s") # Identifying the outlier impact

Interview questions

All REST API Design interview questions →

Check yourself

Take the full REST API Design quiz →