Apache Spark Tutorial with Examples.
Apache Spark tutorial covering the core architecture: RDDs, DAG execution, shuffles, Spark SQL and cluster tuning, with real job output.
Beginner
Start here — no prior experience needed
Spark Architecture: Driver, Executors, and Cluster Managers
Submit a job and watch the driver plan it, executors run it, and the cluster manager allocate resources — reading each component's output live.
RDDs: The API Underneath DataFrames
Work directly with RDDs, measure how much slower they are than the equivalent DataFrame, and identify the few cases where you still need them.
Spark SQL and the Catalyst Optimizer
Register views, query them with SQL, then read Catalyst's four plan stages to see exactly which rewrites it applied to your query.
Intermediate
For developers with core concepts down
Deploying Spark Jobs with spark-submit
Package an application, submit it to a cluster, and work through failures that only appear outside the shell — missing jars and wrong Python.
Reading the Spark UI to Diagnose a Slow Job
Work through the Jobs, Stages, SQL, and Executors tabs on a deliberately broken job, and learn which number to look at first for each kind of slowness.
Advanced
Production-grade patterns for experienced engineers
Also part of Apache Spark
Taught in other topics. Each link opens the lesson in its home topic.
Structured Streaming in PySpark
Run the same DataFrame code over an unbounded stream — with windows, watermarks, checkpoints, and output modes that decide what you compute.
MLlib Basics: A Real Pipeline, Fit and Evaluated
Assemble feature vectors, chain them into a Pipeline with a classifier, and evaluate the fitted model with AUC and accuracy on held-out data.
PySpark in Ten Minutes: Your First DataFrame
Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.
Transforming Columns in PySpark
Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.
Grouping and Aggregation in PySpark
Group, aggregate, and pivot — then read the query plan to see the shuffle that groupBy introduces and why it dominates the job's runtime.
Joins and Broadcast Joins in PySpark
Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.
The Spark Execution Model: Jobs, Stages, and Caching
Trace one query from DataFrame to tasks, see where stage boundaries come from, and prove that caching changes the plan rather than just the timing.
Tuning PySpark Jobs
Diagnose and fix the four things that actually make Spark jobs slow — skew, small files, spill, and wasted shuffles — measuring each one.