PySpark Tutorial with Examples.
PySpark tutorial for data engineers: distributed processing with Apache Spark, PySpark DataFrames, Databricks pipelines and Structured Streaming.
Beginner
Start here — no prior experience needed
PySpark in Ten Minutes: Your First DataFrame
Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.
Reading and Writing Data in PySpark
Read CSV, JSON, and Parquet with an explicit schema, handle malformed rows deliberately, and partition output so downstream reads skip more.
Transforming Columns in PySpark
Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.
Grouping and Aggregation in PySpark
Group, aggregate, and pivot — then read the query plan to see the shuffle that groupBy introduces and why it dominates the job's runtime.
Intermediate
For developers with core concepts down
Joins and Broadcast Joins in PySpark
Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.
Window Functions in PySpark
Rank rows within groups, compute running totals, and compare each row to its neighbours — without collapsing the DataFrame the way groupBy does.
UDFs and Pandas UDFs in PySpark
Measure what a Python UDF costs against a built-in function, then rewrite it as a pandas UDF and watch most of that cost disappear.
MLlib Basics: A Real Pipeline, Fit and Evaluated
Assemble feature vectors, chain them into a Pipeline with a classifier, and evaluate the fitted model with AUC and accuracy on held-out data.
Advanced
Production-grade patterns for experienced engineers
The Spark Execution Model: Jobs, Stages, and Caching
Trace one query from DataFrame to tasks, see where stage boundaries come from, and prove that caching changes the plan rather than just the timing.
Tuning PySpark Jobs
Diagnose and fix the four things that actually make Spark jobs slow — skew, small files, spill, and wasted shuffles — measuring each one.
Structured Streaming in PySpark
Run the same DataFrame code over an unbounded stream — with windows, watermarks, checkpoints, and output modes that decide what you compute.
Also part of PySpark
Taught in other topics. Each link opens the lesson in its home topic.
Spark Architecture: Driver, Executors, and Cluster Managers
Submit a job and watch the driver plan it, executors run it, and the cluster manager allocate resources — reading each component's output live.
RDDs: The API Underneath DataFrames
Work directly with RDDs, measure how much slower they are than the equivalent DataFrame, and identify the few cases where you still need them.
Reading the Spark UI to Diagnose a Slow Job
Work through the Jobs, Stages, SQL, and Executors tabs on a deliberately broken job, and learn which number to look at first for each kind of slowness.
Spark SQL and the Catalyst Optimizer
Register views, query them with SQL, then read Catalyst's four plan stages to see exactly which rewrites it applied to your query.
Spark Memory Management and Debugging Failures
Read Spark's memory model, then reproduce and fix the three failures that actually take jobs down — driver OOM, executor OOM, and disk spill.
Deploying Spark Jobs with spark-submit
Package an application, submit it to a cluster, and work through failures that only appear outside the shell — missing jars and wrong Python.