When should I reach for OpenMP instead of MPI, or vice versa?

OpenMP fits a single machine's shared memory — one process, multiple threads, cheap communication because it's just reading and writing shared variables. MPI fits work that has to span multiple machines (or wants isolated processes on one machine) with no shared memory at all, communicating explicitly through messages. If the problem fits in one machine's RAM and doesn't need process-level isolation, OpenMP alone is usually simpler; MPI earns its complexity once a single machine's cores aren't enough.

Why combine MPI and OpenMP in the same program at all?

A common real shape: MPI ranks are distributed across separate machines (or given separate memory spaces on one machine) to divide a large problem, and each rank then uses OpenMP to parallelize its own local slice across that machine's cores. This is exactly how many real HPC clusters are programmed — MPI across nodes, OpenMP within a node — and neither model has to do both jobs by itself.

What is oversubscription, concretely?

It's requesting more total execution contexts (MPI ranks times OpenMP threads per rank) than the machine has physical cores to run them on. The OS has to time-slice more logical workers across fewer cores than they'd like, adding scheduling overhead that doesn't exist when the total logical thread count matches the real hardware.

Why did 144 logical threads (12 ranks x 12 threads) run measurably slower than 12 (4 ranks x 3 threads) on a 12-core machine, for the same total work?

With 12 physical cores, 144 competing logical threads means each core is time-slicing across roughly 12 of them instead of running exactly one — every context switch is real overhead that pure computation doesn't need. The well-sized case (12 total logical threads for 12 real cores) lets every thread run essentially uninterrupted; the oversubscribed case pays a genuine, measured 1.8x cost for scheduling contention that does zero useful work.

How do I pick how many ranks and threads-per-rank to use for a hybrid program?

Their product should be close to the real core count available, not chosen independently — 4 ranks x 3 threads and 2 ranks x 6 threads both total 12 on a 12-core machine and would perform similarly; 12 ranks x 12 threads on that same machine oversubscribes by 12x. Always check nproc (or the equivalent on the target machine) and size total_ranks * threads_per_rank to match it, rather than picking each number independently.

OpenMP vs MPI, Combined: a Hybrid Program and a Real Oversubscription Penalty

MPI ranks and OpenMP threads compose cleanly in one program — until each rank's thread count multiplies against the rank count and quietly oversubscribes the machine's real core count, measured at 1.8x slower.

By SkillByExample · Published Sep 13, 2026

parallel-programming openmp mpi hybrid performance

MPI ranks and OpenMP threads aren’t mutually exclusive — a program can use both. The two numbers multiply, though, and that multiplication is easy to get wrong.

A working hybrid program

// hybrid.c
#include <stdio.h>
#include <mpi.h>
#include <omp.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    long local_sum = 0;
    int threads_used = 0;
    #pragma omp parallel num_threads(4)
    {
        #pragma omp single
        threads_used = omp_get_num_threads();
        #pragma omp for reduction(+:local_sum)
        for (int i = rank * 1000; i < (rank + 1) * 1000; i++) {
            local_sum += i;
        }
    }
    printf("rank %d (using %d OpenMP threads) local_sum = %ld\n", rank, threads_used, local_sum);

    long global_sum = 0;
    MPI_Reduce(&local_sum, &global_sum, 1, MPI_LONG, MPI_SUM, 0, MPI_COMM_WORLD);
    if (rank == 0) {
        printf("global_sum across %d ranks = %ld\n", size, global_sum);
    }

    MPI_Finalize();
    return 0;
}

mpicc -fopenmp -o hybrid hybrid.c
mpirun -np 4 ./hybrid

rank 3 (using 4 OpenMP threads) local_sum = 3499500
rank 1 (using 4 OpenMP threads) local_sum = 1499500
rank 0 (using 4 OpenMP threads) local_sum = 499500
global_sum across 4 ranks = 7998000
rank 2 (using 4 OpenMP threads) local_sum = 2499500

Four MPI ranks, each dividing its own slice of the range across 4 OpenMP threads internally — 16 execution contexts total, cooperating through two different mechanisms at once: OpenMP’s shared memory within a rank, MPI’s explicit MPI_Reduce message across ranks. Each rank’s partial sum is correct on its own (499500 really is 0+1+...+999), and the final MPI_Reduce combines all four into the correct grand total, 7998000.

The bug: oversubscribing the machine’s real core count

This machine has 12 real cores. A hybrid program’s total execution contexts is ranks × threads_per_rank — and nothing stops that product from wildly exceeding the hardware:

OMP_NUM_THREADS=3 mpirun -np 4 ./hybrid_work

ranks=4 threads_per_rank=3 total_logical=12  sum=3001995.54  slowest_rank_time=0.291s

4 × 3 = 12 — matches the core count exactly. Now the same total workload, same correct answer, wildly different sizing:

OMP_NUM_THREADS=12 mpirun --oversubscribe -np 12 ./hybrid_work

ranks=12 threads_per_rank=12 total_logical=144  sum=3001995.53  slowest_rank_time=0.526s

12 × 12 = 144 logical threads competing for 12 real cores — a measured 1.8x slowdown (0.526s vs 0.291s) for identical total work and an identical correct answer. Nothing is functionally wrong; the program still computes the right sum. The cost is pure scheduling overhead: 144 threads time-slicing across 12 cores do real context-switching work that 12 threads running one-per-core never have to.

The fix: size ranks × threads to the real core count

The 4 × 3 = 12 configuration above already is the fix — choosing rank count and threads-per-rank so their product, not either number alone, matches nproc. Two other sizings that land on the same total:

OMP_NUM_THREADS=6 mpirun -np 2 ./hybrid_work

ranks=2 threads_per_rank=6 total_logical=12  sum=3001995.54  slowest_rank_time=0.378s

Same 12 total, same rough ballpark as the 4×3 case’s 0.291s — noticeably higher here (0.378s), a reminder that “same total logical threads” doesn’t guarantee identical timing; fewer, fatter MPI ranks each running more OpenMP threads has its own overhead characteristics (more work serialized inside each rank’s single setup, for instance). The 12x-oversubscribed case’s 0.526s is still the clear outlier of the three — the core lesson (don’t wildly exceed real core count) holds even though the well-sized configurations aren’t perfectly identical to each other.

Practice

1. Check the real core count available before choosing a hybrid configuration.

nproc

The number every hybrid sizing decision should be checked against — ranks × threads_per_rank should land at or below this, not be picked from habit or a tutorial’s example numbers.

2. Try a moderate oversubscription (2x) and see whether the penalty is as severe as the 12x case.

OMP_NUM_THREADS=4 mpirun --oversubscribe -np 6 ./hybrid_work

ranks=6 threads_per_rank=4 total_logical=24  sum=3001995.53  slowest_rank_time=0.394s

24 logical threads on 12 cores (2x oversubscription) costs 0.394s — worse than the well-sized 4×3 case’s 0.291s, but nowhere near the 0.526s the 12x case measured. Oversubscription penalty scales with how far past the real core count the total goes, not as an all-or-nothing effect.

3. Check whether pure OpenMP (one rank, 12 threads) actually matches the well-sized hybrid's timing.

OMP_NUM_THREADS=12 mpirun -np 1 ./hybrid_work

ranks=1 threads_per_rank=12 total_logical=12  sum=3001995.54  slowest_rank_time=0.432s

Same 12 total logical threads as the 4×3 hybrid case — but 0.432s, slower than that case’s 0.291s, not a match. Staying at-or-below the real core count avoids the severe oversubscription penalty (the 12x case’s 0.526s), but it doesn’t guarantee every equal-total split performs identically — 12 threads all reducing into one shared variable inside a single process has more contention on that reduction than 4 separate processes each reducing only 3 threads’ worth before a single cheap MPI_Reduce combines the four partial sums. Sizing to the core count is necessary to avoid the worst penalty; it isn’t sufficient to guarantee the fastest possible split on its own.

Next: measuring real speedup as thread count scales, and the hard ceiling Amdahl’s Law puts on it.

A working hybrid program

The bug: oversubscribing the machine’s real core count

The fix: size ranks × threads to the real core count

Practice

Frequently Asked Questions