Parallel and concurrent computing are no longer niche topics for high-performance researchers; they are essential for anyone wanting to squeeze real performance out of modern hardware.
1. Motivation: The End of "Free Lunch"
For decades, software got faster simply because hardware engineers increased CPU clock speeds. However, around 2004, we hit a "Power Wall." Increasing clock speeds further generated more heat than could be dissipated.
- CPU Core Stagnation: Instead of making one core faster (increasing GHz), manufacturers began adding more cores to a single chip.
- The Shift: To gain performance now, developers must write code that can run across these multiple cores simultaneously.
2. Serial vs. Parallel Execution
The difference lies in how tasks are queued and processed.
| Feature | Serial Execution | Parallel Execution |
|---|---|---|
| Workflow | One task must finish before the next begins. | Multiple tasks (or parts of a task) run at the same time. |
| Hardware | Uses a single processor core. | Uses multiple cores or multiple processors. |
| Analogy | A single grocery checkout line. | Multiple checkout lanes open at once. |
3. Key Definitions
Concurrency vs. Parallelism
These terms are often used interchangeably, but they describe different concepts:
- Concurrency: The art of dealing with many things at once. It’s about structure. A system is concurrent if it can handle multiple tasks by switching between them (interleaving).
- Parallelism: The act of doing many things at once. It’s about execution. It requires hardware capable of running tasks at the exact same moment.
Deterministic vs. Non-deterministic Execution
- Deterministic: Given the same input, the program always produces the same output and follows the same execution path.
- Non-deterministic: The outcome or the order of execution can change between runs, even with the same input. This is common in parallel systems because the thread scheduler decides when each task runs, often leading to different interleaving.
4. Common Pitfalls
Writing parallel code is notoriously difficult because of the "bugs" that only appear when timing is just right (or wrong).
Race Conditions
A race condition occurs when the output depends on the sequence or timing of uncontrollable events.
- Example: Two threads try to increment a counter simultaneously. If they both read "10," add 1, and write back "11," the counter only increases by 1 instead of 2.
Deadlocks
A deadlock is a "Mexican Standoff" in code. It happens when:
- Thread A holds Resource 1 and waits for Resource 2.
- Thread B holds Resource 2 and waits for Resource 1. Neither can proceed, and the program freezes.
Synchronization Issues
To prevent race conditions, we use "locks" or "mutexes." However, over-synchronizing leads to problems:
- Contention: Too many threads fighting for the same lock, which slows the system down to serial speeds.
- Starvation: A thread is perpetually denied access to resources because other "greedier" threads keep taking them.
Understanding how memory is allocated is the "make or break" moment for designing parallel systems. It dictates how your workers (threads or processes) talk to each other and how much they’ll fight over resources.
2.1 Shared Memory Parallelism (Multithreading)
In this model, multiple threads live within a single process. Imagine a single kitchen (the memory) where multiple chefs (threads) are working at the same counter.
- Shared Space: All threads can see and modify the same variables. This makes communication lightning-fast because you don't have to "send" data; it's already there.
- The Synchronization Tax: Since everyone is touching the same "ingredients," you need strict rules (locks/mutexes) to prevent them from chopping the same carrot at the same time. This adds significant logic complexity.
- The Python Catch (GIL): In standard Python (CPython), the Global Interpreter Lock (GIL) ensures only one thread executes Python bytecode at a time. Even on a 16-core machine, multithreading in Python won't give you a 16x speedup for CPU-heavy math; it’s mostly useful for I/O tasks like downloading files.
2.2 Distributed Memory Parallelism (Multiprocessing)
Here, you have multiple processes, each with its own private "kitchen." No process can peek into another's memory.
- Independence: Since memory isn't shared, you don't have to worry about one process accidentally overwriting another’s variables. This eliminates many race conditions.
- Message Passing: If Process A needs data from Process B, it must be explicitly "sent" over a communication channel (like a Pipe or Queue). This is called Message Passing.
- True Parallelism: Because each process has its own memory and its own instance of the Python interpreter, the GIL is bypassed. This is the go-to method for compute-bound tasks (e.g., heavy data processing, image rendering).
- The Overhead: Creating a new process is "heavier" and slower than creating a thread, and sending large amounts of data between processes can be a performance bottleneck.
Summary Comparison
| Feature | Multithreading (Shared) | Multiprocessing (Distributed) |
|---|---|---|
| Memory | Shared among all threads | Private to each process |
| Communication | Fast (Shared variables) | Slower (Message passing) |
| Complexity | High (Needs locks/semaphores) | Lower (Isolation) |
| Python GIL | Restricted by GIL | Bypasses GIL |
| Best Use Case | I/O-bound (API calls, DB reads) | CPU-bound (Math, Data Science) |
The Global Interpreter Lock (GIL) is perhaps the most famous (and infamous) technical detail of the Python programming language. It is essentially a "safety latch" that has shaped how the entire Python ecosystem handles performance.
3.1 What is the GIL and Why Does It Exist?
The GIL is a mutex (a lock) that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once.
- The Reason: Python uses reference counting for memory management. If two threads increment or decrement the "use count" of an object simultaneously, it could lead to memory leaks or, worse, deleting an object that is still in use.
- The Benefit: It makes the implementation of CPython (the standard Python version) much simpler and faster for single-threaded programs. It also makes integrating C libraries (which might not be thread-safe) much easier.
3.2 Impact on Python Multithreading
Because of the GIL, even if your computer has 32 CPU cores, a standard Python program using threading will only utilize one core at a time for execution.
- The Illusion of Parallelism: To a human, it looks like threads are running in parallel because the GIL switches between them very quickly (every 5ms or so).
- CPU-Bound Bottleneck: If your code is doing heavy math (CPU-bound), multithreading actually makes it slower than a single-threaded program. This is because of the "lock overhead"—the time wasted by threads fighting over who gets to hold the GIL.
3.3 How the GIL is Bypassed
The GIL isn't an impenetrable wall; it’s more like a gate that can be opened under specific conditions.
1. Native Extensions (The "C" Escape)
Libraries like NumPy, SciPy, and Pandas are written in C or Fortran. When you perform a massive matrix multiplication in NumPy, the library "releases" the GIL, does the heavy lifting in C across multiple cores, and "grabs" the GIL back only when it’s done.
Note: This is why Python is a powerhouse for Data Science despite the GIL.
2. I/O Operations
When a thread is waiting for something external—like a website to respond, a file to be read from a disk, or a database query—it voluntarily releases the GIL.
- While Thread A waits for a download, Thread B can take the GIL and start working. This makes Python threads excellent for network-heavy tasks.
3. Multiprocessing
As we discussed earlier, the GIL is per-interpreter. By using the multiprocessing module, you launch entirely separate instances of the Python interpreter.
- Each process has its own GIL.
- Each process can sit on its own CPU core.
- This is the standard way to achieve "True Parallelism" in Python for pure Python code.
Summary: Threading vs. Multiprocessing in Python
| Task Type | Recommended Approach | Why? |
|---|---|---|
| CPU-Bound (Math, Compression) | multiprocessing |
Bypasses GIL, uses all cores. |
| I/O-Bound (Web Scraping, API) | threading |
Efficiently uses "waiting time." |
| Scientific Computing | NumPy / Pandas |
Releases GIL internally in C code. |
In Python, the threading module is the go-to choice for tasks where the bottleneck isn't your CPU's speed, but rather the latency of external systems.
4.1 Threading Use Cases
Threads are ideal for I/O-bound workloads. In these scenarios, the processor spends most of its time idle, waiting for a response from a device or network.
- Network Requests: Fetching data from multiple APIs or web scraping. While Thread A waits for a server in New York to respond, the GIL is released, allowing Thread B to start a request to a server in London.
- Disk Operations: Reading or writing multiple files. Since disk I/O is significantly slower than CPU cache, threads allow you to overlap the "wait time" of different file operations.
- User Interfaces (GUIs): Keeping the interface responsive. One thread handles the "click" events while a background thread does the heavy lifting, preventing the window from freezing.
4.2 ThreadPoolExecutor
Modern Python development favors the concurrent.futures.ThreadPoolExecutor over the older threading.Thread class. It provides a higher-level interface for managing a "pool" of threads.
What is a Thread Pool?
Instead of creating and destroying a thread for every single task (which is expensive), you create a Pool of workers that stay alive and pick up tasks from a queue as they become available.
Key Methods: map vs. submit
The ThreadPoolExecutor offers two primary ways to run tasks:
-
map(func, *iterables):- Works like the built-in
map. - Executes the function across all items in the iterable in parallel.
- Pros: Very simple; returns results in the order they were submitted.
- Works like the built-in
-
submit(func, *args):- Schedules a single callable and returns a Future object.
- Pros: More flexible; allows you to handle individual task completion and different arguments for each task.
Code Example: Efficiently Fetching Data
from concurrent.futures import ThreadPoolExecutor
import requests
urls = ["https://google.com", "https://python.org", "https://github.com"]
def fetch_status(url):
response = requests.get(url)
return f"{url}: {response.status_code}"
# Using a context manager ensures threads are cleaned up automatically
with ThreadPoolExecutor(max_workers=3) as executor:
# 'map' handles the distribution of URLs to the 3 threads
results = list(executor.map(fetch_status, urls))
for r in results:
print(r)
Why use a Pool instead of manual Threads?
- Resource Management: It prevents you from accidentally spawning 10,000 threads and crashing your system.
-
Cleanliness: Using the
withstatement (context manager) ensures that all threads are joined and resources are released even if an error occurs. - Future Objects: It provides "Futures," which are placeholders for results that haven't happened yet, allowing you to check if a task is "done" or if it "cancelled."
Would you like to move on to Section 5: Multiprocessing, to see how we handle those heavy CPU-bound tasks instead?
http://googleusercontent.com/youtube_content/0
To master parallel computing, you must be able to diagnose the bottleneck. Is your code waiting for the "brain" (CPU) or the "delivery truck" (I/O)? Choosing the wrong tool for the workload can actually make your code slower.
6.1 Identifying Workload Characteristics
CPU-Bound (Compute-Heavy)
The speed is limited by the CPU's clock speed and core count.
- Examples: Matrix multiplication, image processing, data compression, searching for prime numbers.
- Significance: These tasks keep the processor usage at 100%.
I/O-Bound (Wait-Heavy)
The speed is limited by Input/Output operations. The CPU often sits idle, waiting for data.
- Examples: Web scraping (Network), reading thousands of small CSVs (Disk), waiting for a database query to return.
- Significance: Processor usage is usually low; the system is waiting on external latency.
6.2 Performance Comparison Table
Here is how each execution style behaves under different pressures:
| Workload Type | Serial Execution | Multithreading | Multiprocessing |
|---|---|---|---|
| I/O-Bound | Very Slow (Total wait time) | Fastest (Overlaps wait time) | Fast (But uses more memory) |
| CPU-Bound | Slow | Slowest (GIL overhead + context switching) | Fastest (Uses all cores) |
6.3 Demonstrations (Mental Model)
I/O-Bound: The sleep() Test
Imagine a task that does nothing but time.sleep(1). This simulates waiting for a network response.
- Serial: To do this 10 times, it takes 10 seconds.
- Multithreading: You spawn 10 threads. They all start "sleeping" at the same time. The total time is roughly 1 second.
-
Why? The GIL is released during
sleep(), letting threads wait in parallel.
CPU-Bound: The Mathematical Loop
Imagine calculating the sum of squares for 50 million numbers.
- Serial: Takes X seconds.
- Multithreading: Takes X + overhead seconds. Because of the GIL, only one thread can do math at a time. The CPU is essentially "juggling" threads, which wastes time.
- Multiprocessing: If you have 4 cores, it takes roughly X / 4 seconds. Each core handles a chunk of the numbers independently.
Summary: The Decision Tree
- Is the CPU usage low while the program is running? $\rightarrow$ It's I/O-bound. Use
threadingorasyncio. - Is one core pegged at 100%? $\rightarrow$ It's CPU-bound. Use
multiprocessingor a library likeNumPy. - Are you limited by memory? $\rightarrow$ Be careful with
multiprocessing, as each process copies the memory space.
While Multithreading is like having one chef with multiple hands, Multiprocessing is like hiring four chefs in four separate kitchens. This is the only way to achieve "true" parallelism for Python-native code.
7.1 The multiprocessing Module
This module bypasses the GIL by creating entirely new instances of the Python interpreter for each task.
- Process-based parallelism: Each process has its own memory space and its own GIL.
- Safety: Since memory isn't shared by default, one process can't accidentally corrupt another's data.
7.2 Pool, Map, and Starmap
The multiprocessing.Pool class is the workhorse for data-parallel tasks.
-
map(func, iterable): The simplest way to parallelize. It chops the iterable into chunks and sends them to the worker processes. -
starmap(func, iterable_of_tuples): Used when your function requires multiple arguments.-
Example: If
func(x, y)is your function,starmaptakes[(1, 2), (3, 4)].
-
Example: If
ProcessPoolExecutor
Found in concurrent.futures, this provides an identical interface to the ThreadPoolExecutor we saw earlier. It is generally preferred in modern code for its consistency and better error handling.
7.3 Communication & Shared Memory
Sometimes processes do need to talk to each other. Since they don't share memory, we use special constructs:
| Tool | Description | Best For... |
|---|---|---|
Value / Array |
Allocates a small piece of shared memory (C-style) that all processes can see. | Simple counters or flags. |
Queue |
A thread- and process-safe FIFO (First-In-First-Out) pipe. | Passing complex objects or results back to the main process. |
Pipe |
A direct connection between two processes. | Fast, two-way communication between exactly two workers. |
7.4 Limitations in Interactive Environments (Jupyter)
A common "gotcha" for data scientists is that the multiprocessing module often fails or behaves unpredictably in Jupyter Notebooks or the IPython console.
- Serialization (Pickling): Python must "pickle" (serialize) your function and data to send it to the other process. If you define a function inside a notebook cell, the worker process might not be able to find its definition.
- The
if __name__ == "__main__":block: On Windows and macOS, you must wrap your multiprocessing code in this block to prevent a recursive loop of process creation.- Jupyter doesn't always handle this entry point correctly.
Workaround: If you run into issues in Jupyter, move your functions into a separate .py file and import them into your notebook.
7.5 Summary: When to use Multiprocessing
- YES: For "number crunching" (e.g., calculating $\pi$ to a billion digits).
- YES: For heavy image/video processing.
- NO: For simple I/O (it uses way more RAM than threads).
- NO: When you need to share massive amounts of data (the "pickling" overhead will kill your performance).
When multiple threads or processes try to change the same piece of data at the same time, you enter the world of Race Conditions. This is the most common source of "heisenbugs"—bugs that seem to disappear when you try to look for them.
8.1 Shared State Modification Problems
A race condition occurs when the final outcome of a program depends on the timing or scheduling of the execution.
If two threads are incrementing a shared variable, the operation looks like one step in Python (x += 1), but the CPU sees three distinct steps:
- Read the current value of $x$.
- Add 1 to that value.
- Write the new value back to memory.
If Thread A is interrupted after step 1, and Thread B finishes all three steps, Thread A will eventually overwrite Thread B's work with an outdated value.
8.2 Demonstration of Incorrect Results
In a perfectly synchronized world, if you have 10 threads each adding 1 to a counter 100,000 times, the result should be 1,000,000.
In a race condition scenario, the result might be 742,384. This happens because thousands of "updates" were lost when threads stomped on each other’s data.
8.3 Threading vs. Multiprocessing Behavior
The way these two handle "shared state" is fundamentally different, which changes how they fail.
- In Multithreading: Race conditions are common and dangerous. Because all threads share the same memory, they can all "see" and "touch" the same variables globally.
-
In Multiprocessing: Race conditions are rare by default. Since each process has its own memory, incrementing
xin Process A does nothing toxin Process B.-
Exception: You only face race conditions in multiprocessing if you explicitly use Shared Memory constructs (like
ValueorArray) or shared external resources (like a database or a file on disk).
-
Exception: You only face race conditions in multiprocessing if you explicitly use Shared Memory constructs (like
8.4 Synchronization Primitives
To fix these issues, we use tools that force threads to "wait their turn."
1. The Lock (Mutex)
A Lock is the simplest tool. It has two states: locked and unlocked.
- A thread must "acquire" the lock before touching the shared data.
- If another thread holds the lock, everyone else must wait.
- Analogy: The "talking stick" in a meeting. You can't speak unless you hold the stick.
2. The Semaphore
A Semaphore is like a Lock, but it allows a specific number of threads to enter.
- Analogy: A restaurant with 10 tables. The first 10 groups get in; the 11th must wait until someone leaves.
3. The RLock (Re-entrant Lock)
A standard Lock can cause a thread to "deadlock itself" if it tries to acquire the same lock twice. An RLock allows the same thread to acquire the lock multiple times without freezing.
Summary: The Cost of Safety
While synchronization prevents data corruption, it comes with a performance price:
- Overhead: Managing locks takes CPU time.
- Serial Bottlenecks: If every thread is waiting for the same lock, your "parallel" program is actually running one-by-one (serial).
Numerical integration is a "perfect" parallel problem. It follows the embarrassingly parallel pattern, where a large task can be easily divided into independent sub-tasks that don't need to communicate with each other.
9.1 The Grid-Based Technique (Rectangle Rule)
To find the area under a curve f(x) between a and b, we divide the interval into $N$ small rectangles. The total area is the sum of the areas of these rectangles.
$$Area \approx \sum_{i=0}^{N-1} f(x_i) \Delta x$$
In a serial approach, a single CPU core calculates rectangle #1, then #2, then #3, all the way to N. If N is 100 million, this takes a significant amount of time.
9.2 Identifying Parallelizable Regions
The beauty of integration is that the calculation of "Rectangle #500" does not depend on the result of "Rectangle #499."
- The Strategy: Split the total range $[a, b]$ into sub-intervals.
- The Workers: If you have 4 cores, Core 1 handles the first 25%, Core 2 the second 25%, and so on.
- The Reduction: Once all cores finish their local sums, you add those 4 sums together to get the final answer.
9.3 Implementation Strategies
Multithreading Approach
- Performance: Low. Because integration is CPU-bound (pure math), the Python GIL will prevent the threads from running the math in parallel.
- Use Case: Only beneficial if the function $f(x)$ involved an I/O wait (e.g., fetching a coordinate from a remote database), which is rare in pure math.
Multiprocessing Approach
-
Performance: High. This is the correct tool. By using a
ProcessPoolExecutor, each core gets a chunk of the grid. - Efficiency: You get nearly "linear scaling." If 1 core takes 10 seconds, 4 cores should take roughly 2.5 seconds.
9.4 Performance Measurement
To prove the speedup, we use the time module. It is vital to measure only the calculation, excluding the time it takes to set up the data.
import time
start_time = time.time()
# ... Parallel Integration Logic ...
end_time = time.time()
print(f"Execution Time: {end_time - start_time:.4f} seconds")
Critical Metrics:
- Speedup ($S$): $S = \frac{T_{serial}}{T_{parallel}}$
- Efficiency ($E$): $E = \frac{S}{Number\ of\ Cores}$ (Ideally, this is close to 1.0 or 100%).
9.5 Summary Table: Integration Performance
| Method | Execution | Expected Speedup |
|---|---|---|
| Serial | One core, one by one. | 1x (Baseline) |
| Multithreading | Context switching on one core. | ~0.9x (Slower due to overhead) |
| Multiprocessing | Multiple cores simultaneously. | ~3.8x (on a 4-core machine) |
| NumPy (Vectorized) | Optimized C-backend/SIMD. | Fastest (often 50x - 100x) |
To wrap up our foundations, we look at the "low-hanging fruit" of the computing world. An Embarrassingly Parallel problem is one where little to no effort is needed to separate the problem into a number of parallel tasks.
10.1 Definition and Characteristics
A problem is embarrassingly parallel if there is no dependency (or very little) between the sub-tasks.
- No Communication: Task A doesn't need to know what Task B is doing to finish its job.
- No Shared State: Workers don't need to update a global variable constantly (which avoids those pesky race conditions).
- High Scalability: These problems scale almost perfectly; doubling your CPU cores usually halves the execution time.
10.2 Core Examples
Monte Carlo Simulations
These simulations use repeated random sampling to obtain numerical results (like predicting stock market trends or calculating $\pi$). Since every "random trial" is independent, you can run a million trials on one core or divide them across a thousand cores with zero logic changes.
Weather Ensemble Models
Meteorologists don't just run one weather forecast; they run dozens of "ensembles" with slightly different starting conditions. Since Forecast A doesn't affect Forecast B, they are computed in parallel across massive supercomputers.
Batch Data Processing
Imagine you have 10,000 high-resolution photos to resize. Resizing photo #1 has nothing to do with photo #100. This is a classic "Map" operation where a worker pool can chew through the pile of files as fast as the disk can provide them.
CNN (Convolutional Neural Network) Workloads
In Deep Learning, a Convolutional layer applies filters to an image. Each "pixel" calculation or each "filter" application can be done independently. This is why GPUs—which have thousands of tiny cores—are so much faster than CPUs for AI tasks.
FFT (Fast Fourier Transform)
While the classic DFT is $O(N^2)$, the FFT reduces complexity to $O(N \log N)$. In many implementations, the data is split into "even" and "odd" parts that can be processed recursively in parallel, making it a staple of digital signal processing.
Summary of the "Parallel Spectrum"
| Type | Communication Needs | Difficulty to Parallelize |
|---|---|---|
| Embarrassingly Parallel | None | Very Easy |
| Coarse-Grained | Occasional | Moderate |
| Fine-Grained | Constant/Frequent | Hard (High risk of overhead) |
While Python’s multiprocessing is great for a single machine, MPI (Message Passing Interface) is the gold standard for high-performance computing (HPC) across clusters of multiple computers. It is the language of supercomputers.
11.1 MPI Fundamentals
Unlike the shared-memory models we’ve discussed, MPI is built entirely on the Distributed-Memory Model.
-
Independent Processes: Each process has its own address space. There is no shared "global variable." If Process 0 has a variable
x, Process 1 cannot see it unless Process 0 explicitly sends it. - The "Rank": Every process in an MPI job is assigned a unique ID called a Rank (starting from 0). You use this rank to tell each process what part of the work it should do.
-
The "Communicator": This is a group of processes that can talk to each other. The default group containing all your processes is called
COMM_WORLD.
11.2 mpi4py: MPI for Python
The mpi4py library provides the Python bindings for the MPI standard. It allows Python scripts to communicate across a network.
Key Concepts
-
COMM_WORLD: The primary communicator. -
Get_size(): Tells you the total number of processes running. -
Get_rank(): Tells the current process its unique ID. -
Point-to-Point Communication: Using
send()andrecv()to move data between specific ranks. -
Collective Communication: Using
bcast()(one-to-all) orreduce()(all-to-one) to synchronize data.
11.3 Running MPI Programs
You cannot run an MPI script by simply typing python script.py. You must use a process manager, typically mpirun or mpiexec, which handles the launching of multiple instances across your CPU cores or network nodes.
The Command:
mpirun -n 4 python3 my_script.py
(This launches 4 independent instances of your script.)
Example: The "Who Am I?" Pattern
from mpi4py import MPI
# Get the communicator
comm = MPI.COMM_WORLD
# Get the size (total processes) and rank (my ID)
size = comm.Get_size()
rank = comm.Get_rank()
print(f"Hello! I am process {rank} out of {size} total processes.")
if rank == 0:
data = {'key': 'value'}
comm.send(data, dest=1)
print("Process 0 sent data to Process 1.")
elif rank == 1:
data = comm.recv(source=0)
print(f"Process 1 received: {data}")
Summary: MPI vs. Multiprocessing
| Feature | multiprocessing |
mpi4py (MPI) |
|---|---|---|
| Scope | Single Machine (Multi-core) | Multi-Node (Clusters/Supercomputers) |
| Memory | Shared-memory constructs available | Strictly Distributed (Message Passing) |
| Launch | Standard Python interpreter |
mpirun / mpiexec
|
| Scaling | Limited by one motherboard | Scales to thousands of CPUs |
This completes the technical foundation of Parallel & Concurrent Computing! We've traveled from CPU core stagnation all the way to distributed supercomputing.
Would you like me to create a "Cheat Sheet" summarizing which tool (Threading, Multiprocessing, or MPI) you should use based on the specific type of project you are working on?
In MPI, communication is how independent processes coordinate to solve a single problem. There are two primary ways processes "talk": one-to-one (Point-to-Point) or all-together (Collective).
12.1 Point-to-Point Communication
This is the most basic form of messaging, involving exactly two processes: a sender and a receiver.
-
send(obj, dest): The source process sends a Python object to a specific rank. -
recv(source): The destination process waits to receive an object from a specific rank. -
Blocking Communication: By default, these operations are "blocking." The sender waits until the message is safely in the transmission buffer, and the receiver waits (sleeps) until the message actually arrives. If you
recv()and no one eversend(), your program will hang forever.
12.2 Collective Communication
Collective operations involve all processes in a communicator (e.g., COMM_WORLD). These are highly optimized and usually much faster than writing multiple point-to-point loops.
| Operation | Description | Analogy |
|---|---|---|
Broadcast (bcast) |
One process sends the same data to everyone else. | A teacher giving a handout to the whole class. |
Scatter (scatter) |
One process takes a list and gives one piece to each process. | Dealing a deck of cards to players. |
Gather (gather) |
One process collects a piece of data from everyone else into a list. | A teacher collecting homework from every student. |
Reduce (reduce) |
Everyone sends data to one process, which "crunches" it (e.g., Sum, Max). | Everyone votes, and the teller announces only the total count. |
12.3 Performance: The "Case" Matters
In mpi4py, there is a massive performance difference between lowercase methods (e.g., send) and uppercase methods (e.g., Send).
Lowercase Methods (send, recv, bcast)
- Mechanism: Uses pickle to serialize Python objects.
- Flexibility: Can send almost any Python object (dicts, lists, custom classes).
- Performance: Slower. The overhead of pickling and unpickling large amounts of data can create a bottleneck.
Uppercase Methods (Send, Recv, Bcast)
- Mechanism: Uses Buffer-based communication. It points directly to a contiguous block of memory.
- Flexibility: Requires data to be in a buffer-like format, typically NumPy arrays.
- Performance: Extremely Fast. This is near-C speeds because it avoids the Python overhead and communicates the raw memory directly.
Rule of Thumb: If you are moving NumPy arrays for math, always use the uppercase methods (e.g.,
comm.Send(my_array, dest=1)).
12.4 Summary: When to use what?
- Use Point-to-Point for complex logic where specific workers need unique instructions.
- Use Collective for mathematical synchronization (e.g., summing partial results of an integral).
- Use Uppercase methods whenever you are doing heavy data lifting with NumPy.
Top comments (0)