Parallel Programming Tutorial with Examples.
Parallel programming tutorial with examples: task and data parallelism, race conditions, synchronisation and common parallel patterns.
Beginner
Start here — no prior experience needed
OpenMP Basics: Parallel Regions and the Loop That Ran on Every Thread
A bare #pragma omp parallel runs the whole block on every thread — including the entire loop inside it, redundantly, unless #pragma omp for actually divides the iterations.
OpenMP Scheduling: Static's Load Imbalance, Measured
Static scheduling hands out equal-sized, fixed chunks up front — great when every iteration costs the same, a real 30% slower when they don't. Dynamic scheduling fixes it, measured, not asserted.
Intermediate
For developers with core concepts down
Race Conditions: A Sum That Gives Three Different Wrong Answers
Three runs of the same unsynchronized parallel sum produce three different wrong totals — reduction(+:sum) fixes it correctly and is 115x faster than the naive #pragma omp critical alternative, measured.
Atomics vs Locks, and the Race Hiding Inside a Check-Then-Act
#pragma omp atomic is nearly 5x faster than a lock for a plain increment — but it only protects the single write it's attached to, not the if-check before it. Three widened-race runs give three different wrong answers to prove it.
MPI Basics: a Real Deadlock That Only Shows Up at Scale
Two ranks calling MPI_Send before MPI_Recv works fine for a single int — and hangs forever for a half-million-element array. The same bug, passing at small scale and deadlocking in production.
MPI Collectives: Why MPI_Reduce Leaves Every Rank but One Empty
MPI_Reduce computes the right answer and hands it to exactly one rank — every other rank's result buffer is untouched. MPI_Allreduce is the one-line fix when everyone needs the answer.
OpenMP vs MPI, Combined: a Hybrid Program and a Real Oversubscription Penalty
MPI ranks and OpenMP threads compose cleanly in one program — until each rank's thread count multiplies against the rank count and quietly oversubscribes the machine's real core count, measured at 1.8x slower.
Advanced
Production-grade patterns for experienced engineers
Also part of Parallel Programming
Taught in other topics. Each link opens the lesson in its home topic.
Concurrency with pthreads in C
POSIX threads in C: create and join threads, see a real race condition, fix it with mutexes or atomics, and use condition variables.
GPU Fundamentals: SIMT & Memory Hierarchy
Learn SIMT execution, warps, branch divergence and the GPU memory hierarchy (registers, shared, L2, DRAM) from CUDA programs run on a real T4.