Databricks Tutorial with Examples.
Databricks tutorial: unified data and AI on Databricks with Delta Lake, Unity Catalog, notebooks, jobs and MLflow integration.
Beginner
Start here — no prior experience needed
The Databricks Workspace and Compute
Pick the right compute for the job, run your first notebook against a Delta table, and understand what you are billed for while it runs.
Delta Lake: The Transaction Log
ACID on object storage — read the _delta_log, travel back to a previous version, MERGE upserts, and see schema enforcement reject a bad write.
Unity Catalog: Namespaces, Grants, and Lineage
The three-level namespace, managed versus external tables, volumes for files, and lineage you get without instrumenting anything.
Ingesting Files with Auto Loader and COPY INTO
Incremental file ingestion that remembers what it loaded, infers a schema, evolves when a column appears, and rescues the data that does not fit.
The Medallion Architecture
Bronze, silver and gold as three concrete tables — what belongs in each layer, why bronze keeps the bad rows, and when it's more than you need.
Intermediate
For developers with core concepts down
Declarative Pipelines and Data Expectations
Define tables and quality rules, and let the pipeline derive the DAG — with expectations that drop, quarantine or fail, and CDC handled by AUTO CDC.
Jobs, Workflows, and Asset Bundles
Multi-task jobs with dependencies, parameters passed between tasks, retries and conditional runs — then the same job as reviewable YAML.
Optimizing Delta Tables
Fix the small file problem with OPTIMIZE, replace partitioning and Z-ORDER with liquid clustering, and measure data skipping instead of guessing at it.
Advanced
Production-grade patterns for experienced engineers
Structured Streaming on Databricks
Checkpoints and exactly-once semantics, watermarks that bound state, foreachBatch for streaming upserts, and the numbers that show a stream falling behind.
Cost Control and Cluster Policies
Attribute DBUs to teams with tags, cap what people can launch with cluster policies, and decide when Photon and spot instances pay off.
Also part of Databricks
Taught in other topics. Each link opens the lesson in its home topic.
PySpark in Ten Minutes: Your First DataFrame
Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.
Reading and Writing Data in PySpark
Read CSV, JSON, and Parquet with an explicit schema, handle malformed rows deliberately, and partition output so downstream reads skip more.
Transforming Columns in PySpark
Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.
Joins and Broadcast Joins in PySpark
Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.
Window Functions in PySpark
Rank rows within groups, compute running totals, and compare each row to its neighbours — without collapsing the DataFrame the way groupBy does.
The Spark Execution Model: Jobs, Stages, and Caching
Trace one query from DataFrame to tasks, see where stage boundaries come from, and prove that caching changes the plan rather than just the timing.
Reading the Spark UI to Diagnose a Slow Job
Work through the Jobs, Stages, SQL, and Executors tabs on a deliberately broken job, and learn which number to look at first for each kind of slowness.
Incremental Loads and Change Data Capture
Four ways to extract only what changed, why deletes are the hard part, and building SCD type 2 history from a CDC stream that arrives out of order.
MLOps Experiment Tracking
Track ML experiments with MLflow 3 locally: log params and metrics, compare runs with search_runs, per-epoch metrics, and a registry with aliases.