Why did every rank except rank 0 still show -1 after MPI_Reduce?

MPI_Reduce combines every rank's value using the given operator and writes the result into the receive buffer on exactly one rank — the root, specified by its argument. Every other rank's receive buffer is never written at all, so it still holds whatever it was initialized to (-1 here), not zero and not the answer.

What does MPI_Allreduce actually do differently from MPI_Reduce?

The same combination happens, but the result is written into every rank's receive buffer, not just the root's — it's equivalent to an MPI_Reduce immediately followed by an MPI_Bcast of the result to everyone, except MPI implementations can do it in one coordinated step rather than two separate ones.

What does MPI_Bcast do, and how is it different from MPI_Reduce?

MPI_Bcast takes a value that already exists on one rank (the root) and copies it to every other rank unchanged — no combination happens. MPI_Reduce is the opposite direction: it takes a value from every rank and combines them down to one result. Bcast is one-to-many; Reduce is many-to-one; Allreduce is many-to-many.

What do MPI_Scatter and MPI_Gather do?

MPI_Scatter takes an array that exists on the root and splits it into equal pieces, sending one piece to each rank (including the root itself). MPI_Gather is the reverse: it collects one value from every rank into an array assembled on the root, in rank order. Together they're the standard pattern for "split work across ranks, then collect the results."

Do collective calls need to happen in the same order on every rank?

Yes — every rank participating in a collective operation (Bcast, Reduce, Scatter, Gather, and so on) must call it, and the collectives across all ranks must occur in the same relative order. A rank that skips a collective call, or calls a different one at that point, will cause the others to hang waiting for a participant that never arrives — the same class of problem as the point-to-point deadlock from the previous lesson, just at the collective-operation level.

MPI Collectives: Why MPI_Reduce Leaves Every Rank but One Empty

MPI_Reduce combines every rank’s data correctly — and still leaves most ranks with nothing to show for it, because “combines” and “distributes the result” are two separate promises.

Broadcast: one rank’s value, copied to everyone

// mpi_bcast.c
#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int value = (rank == 0) ? 99 : 0;
    printf("rank %d before bcast: value = %d\n", rank, value);
    MPI_Bcast(&value, 1, MPI_INT, 0, MPI_COMM_WORLD);
    printf("rank %d after bcast: value = %d\n", rank, value);

    MPI_Finalize();
    return 0;
}

mpicc -o mpi_bcast mpi_bcast.c
mpirun -np 4 ./mpi_bcast

rank 2 before bcast: value = 0
rank 3 before bcast: value = 0
rank 0 before bcast: value = 99
rank 1 before bcast: value = 0
rank 2 after bcast: value = 99
rank 0 after bcast: value = 99
rank 1 after bcast: value = 99
rank 3 after bcast: value = 99

Only rank 0 started with 99 — every rank, including rank 0 itself, has it after the broadcast. MPI_Bcast doesn’t compute anything; it just copies rank 0’s value to every participant.

The bug: MPI_Reduce leaves every non-root rank untouched

// mpi_reduce_bug.c
#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int value = rank + 1;
    int total = -1;
    MPI_Reduce(&value, &total, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
    printf("rank %d: total = %d\n", rank, total);

    MPI_Finalize();
    return 0;
}

mpicc -o mpi_reduce_bug mpi_reduce_bug.c
mpirun -np 4 ./mpi_reduce_bug

rank 0: total = 10
rank 1: total = -1
rank 2: total = -1
rank 3: total = -1

1 + 2 + 3 + 4 = 10 — correctly computed, and visible on exactly one rank. Ranks 1 through 3 still show -1, their initial value, completely untouched by the reduction. MPI_Reduce’s contract is “combine everyone’s value and write the result on the root” — nothing more.

The fix: MPI_Allreduce

// mpi_allreduce.c
int value = rank + 1;
int total = -1;
MPI_Allreduce(&value, &total, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
printf("rank %d: total = %d\n", rank, total);

mpicc -o mpi_allreduce mpi_allreduce.c
mpirun -np 4 ./mpi_allreduce

rank 1: total = 10
rank 2: total = 10
rank 3: total = 10
rank 0: total = 10

Same computation, same correct 10 — but now every rank has it, because MPI_Allreduce has no root argument at all; the result goes to everyone by design, not as an extra step.

Scatter and Gather: splitting work, then collecting results

// mpi_scatter_gather.c
#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int data[4] = {10, 20, 30, 40};
    int my_piece;
    MPI_Scatter(data, 1, MPI_INT, &my_piece, 1, MPI_INT, 0, MPI_COMM_WORLD);
    printf("rank %d received piece: %d\n", rank, my_piece);

    int doubled = my_piece * 2;
    int gathered[4];
    MPI_Gather(&doubled, 1, MPI_INT, gathered, 1, MPI_INT, 0, MPI_COMM_WORLD);

    if (rank == 0) {
        printf("rank 0 gathered: %d %d %d %d\n", gathered[0], gathered[1], gathered[2], gathered[3]);
    }

    MPI_Finalize();
    return 0;
}

mpicc -o mpi_scatter_gather mpi_scatter_gather.c
mpirun -np 4 ./mpi_scatter_gather

rank 0 received piece: 10
rank 0 gathered: 20 40 60 80
rank 1 received piece: 20
rank 2 received piece: 30
rank 3 received piece: 40

MPI_Scatter split {10, 20, 30, 40} into one value per rank, in rank order. Each rank doubled its own piece independently — real distributed work, no communication needed for that part. MPI_Gather then collected the four doubled values back into one array on rank 0, in the same rank order the scatter used — {20, 40, 60, 80}, correctly assembled even though the four ranks computed their pieces at different, unsynchronized moments.

Practice

1. Use MPI_Reduce with MPI_MAX instead of MPI_SUM.

int value = (rank + 1) * 7;
int max_val = -1;
MPI_Reduce(&value, &max_val, 1, MPI_INT, MPI_MAX, 0, MPI_COMM_WORLD);
if (rank == 0) printf("rank 0: max = %d\n", max_val);

rank 0: max = 28

Same MPI_Reduce call shape as the sum example, different operator — rank 3’s value (4 * 7 = 28) is the largest across all four ranks, and only rank 0 sees it, exactly like the sum case.

2. Confirm the previous lesson's point-to-point exchange and this lesson's collectives solve genuinely different problems.

Point-to-point (MPI_Send/MPI_Recv/MPI_Sendrecv) moves data between two specific named ranks — rank 0 to rank 1, exactly. Collectives (MPI_Bcast, MPI_Reduce, MPI_Scatter, MPI_Gather) always involve every rank in the communicator at once, with a shape (one-to- many, many-to-one, or many-to-many) built into which collective you call. Reaching for a loop of point-to-point sends to fan a value out to every rank, instead of one MPI_Bcast call, is both more code and typically slower — collectives are usually implemented with communication patterns (like a broadcast tree) more efficient than repeating MPI_Send from one rank to every other.

3. Explain why every rank must call the same collective, in the same order, even ranks that don't care about the result.

Collectives are synchronization points across the whole communicator — MPI_Reduce on rank 0 can’t complete until every other rank has also reached its own matching MPI_Reduce call and contributed its value. A rank that skips the call (say, inside an if branch it doesn’t take) leaves every other rank blocked waiting for a contribution that will never come — the collective equivalent of the point-to-point deadlock from the previous lesson, just involving the whole group instead of one pair.

Next: choosing between OpenMP and MPI — and combining both in one program.

Broadcast: one rank’s value, copied to everyone

The bug: MPI_Reduce leaves every non-root rank untouched

The fix: MPI_Allreduce

Scatter and Gather: splitting work, then collecting results

Practice

Frequently Asked Questions