Multiprocessing Basics

The Global Interpreter Lock and Why We Need Multiprocessing

Python's implementation relies on a mechanism called the Global Interpreter Lock (GIL), which ensures that only one thread executes bytecode at any given moment, even on multi-core systems. This design choice simplifies memory management and garbage collection but fundamentally limits the capability of multi-threaded programs to achieve true parallelism for CPU-intensive tasks. When you attempt to perform complex mathematical computations or heavy data processing using multiple threads, you will find that the application fails to utilize the actual processing power of your hardware effectively. Multiprocessing bypasses this restriction by spawning entirely separate instances of the Python interpreter, each with its own memory space and execution environment. Because these processes do not share the same GIL, they can execute simultaneously on distinct CPU cores. This architecture effectively transforms your program into a distributed system on a single machine, allowing for massive performance gains in compute-heavy workflows while keeping the core logic maintainable and structured.

import multiprocessing
import os

def show_identity():
    # Each process has a unique ID, proving they are distinct instances
    print(f"Process ID: {os.getpid()}")

if __name__ == '__main__':
    # Spawn two separate processes to demonstrate execution
    p1 = multiprocessing.Process(target=show_identity)
    p2 = multiprocessing.Process(target=show_identity)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Spawning Processes with the Process Class

The fundamental building block of the multiprocessing module is the Process class, which provides an object-oriented interface for managing concurrent execution. When you instantiate a Process object, you specify the target function and any arguments that should be passed to it when execution begins. Crucially, the start method must be invoked to initiate the child process, triggering the creation of a new system process that inherits the environment of the parent but runs independently. The join method acts as a synchronization barrier, causing the parent thread to pause until the child process completes its task. This pattern is vital for ensuring that your application does not terminate prematurely before background work finishes. By managing these lifecycle events explicitly, you retain complete control over task execution order and termination. Understanding this pattern is essential, as it allows you to orchestrate complex task dependencies where specific operations must conclude before the main application proceeds to subsequent stages of computation.

import multiprocessing
import time

def task(name, delay):
    print(f"Task {name} starting...")
    time.sleep(delay) # Simulate work
    print(f"Task {name} finished.")

if __name__ == '__main__':
    # Create a process that runs a function with arguments
    proc = multiprocessing.Process(target=task, args=('Worker-1', 2))
    proc.start()
    # Ensure the parent process waits for the task to finish
    proc.join()

Handling Return Values with Process Pools

While creating individual Process objects provides granular control, managing dozens of manual lifecycles is error-prone and inefficient. The Pool class simplifies this by providing a worker pool pattern where a set number of processes are pre-spawned and kept ready to execute tasks. This abstraction is highly effective because it minimizes the overhead of creating and destroying process instances repeatedly for short-lived tasks. Using the map method of the Pool object, you can distribute an iterable of data points across your worker processes automatically. The Pool collects the results and returns them to the caller in an ordered list, effectively abstracting away the underlying synchronization and communication complexities. This mechanism is perfect for batch processing, such as applying a computationally expensive filter to a large dataset, where you want to partition the work among available CPU cores efficiently without manually tracking the status of every sub-task in your system.

from multiprocessing import Pool

def compute_square(n):
    return n * n

if __name__ == '__main__':
    # Pool manages a set of worker processes
    with Pool(processes=4) as pool:
        data = [1, 2, 3, 4, 5]
        # Map distributes data across the pool processes
        results = pool.map(compute_square, data)
        print(f"Calculated squares: {results}")

Inter-Process Communication via Queues

Because separate processes operate in isolated memory spaces, you cannot simply share standard Python objects between them; modifying a global variable in one process will not reflect in another. To facilitate communication, the multiprocessing module provides a Queue class that is specifically designed to be safe for concurrent access. A Queue acts as a thread-safe and process-safe conduit where one process can push data, and another can pull it asynchronously. This decoupling is a cornerstone of robust concurrent design, as it allows producers and consumers to work at different speeds without risking data corruption or race conditions. When designing your architecture, view Queues as the primary mechanism for passing state updates or task results back to the main process. This message-passing model ensures that your data remains consistent and predictable, as every component in your system interacts only through well-defined, thread-safe interfaces rather than shared memory which requires complex locking strategies.

import multiprocessing

def producer(q):
    # Send data into the queue
    for i in range(5):
        q.put(f"Data packet {i}")

def consumer(q):
    # Retrieve data from the queue
    while True:
        item = q.get()
        if item is None: break
        print(f"Consumed: {item}")

if __name__ == '__main__':
    queue = multiprocessing.Queue()
    p1 = multiprocessing.Process(target=producer, args=(queue,))
    p2 = multiprocessing.Process(target=consumer, args=(queue,))
    p1.start()
    p2.start()
    p1.join()
    queue.put(None) # Signal termination
    p2.join()

Avoiding Memory Hazards with Synchronization

When multiple processes must interact with shared resources that are not managed by a Queue, such as a file on disk or a hardware device, you encounter the risk of resource contention. Even though processes have separate memory, external resources are global and susceptible to race conditions. To prevent this, the multiprocessing module provides primitives like Locks and Semaphores. A Lock acts as an exclusive gatekeeper; only one process can acquire it at any given time, ensuring that the code inside the protected 'critical section' executes atomically. Using these primitives is a defensive programming practice that prevents corrupted data and undefined behavior in your application. However, be aware that excessive reliance on locks can negate the benefits of parallelism, as processes spend their time waiting for access rather than performing productive work. Always aim to structure your data pipelines to be as independent as possible, using synchronization only when absolutely necessary to safeguard external shared assets.

import multiprocessing

def safe_increment(counter, lock):
    # Use the lock to ensure only one process writes at a time
    with lock:
        temp = counter.value
        counter.value = temp + 1

if __name__ == '__main__':
    lock = multiprocessing.Lock()
    counter = multiprocessing.Value('i', 0)
    procs = [multiprocessing.Process(target=safe_increment, args=(counter, lock)) for _ in range(10)]
    for p in procs: p.start()
    for p in procs: p.join()
    print(f"Final count: {counter.value}")

Interview questions

All Python interview questions →

Check yourself

Take the full Python quiz →