Databricks.
Unified data and AI on Databricks — Delta Lake, Unity Catalog, notebooks, jobs, and MLflow integration.
Beginner
Start here — no prior experience needed
The Databricks Workspace and Compute
Pick the right compute for the job, run your first notebook against a Delta table, and understand what you are billed for while it runs.
Delta Lake: The Transaction Log
ACID on object storage — read the _delta_log, travel back to a previous version, MERGE upserts, and see schema enforcement reject a bad write.
Unity Catalog: Namespaces, Grants, and Lineage
The three-level namespace, managed versus external tables, volumes for files, grants that new tables inherit, and lineage you get without instrumenting anything.
Ingesting Files with Auto Loader and COPY INTO
Incremental file ingestion that remembers what it loaded, infers a schema, evolves when a column appears, and rescues the data that does not fit.
The Medallion Architecture
Bronze, silver and gold as three concrete tables — what belongs in each layer, why bronze keeps the bad rows, and when the pattern is more structure than you need.
Intermediate
For developers with core concepts down
Declarative Pipelines and Data Expectations
Define tables and quality rules, and let the pipeline derive the DAG — with expectations that drop, quarantine or fail, and CDC handled by AUTO CDC.
Jobs, Workflows, and Asset Bundles
Multi-task jobs with dependencies, parameters passed between tasks, retries and conditional runs — then the same job defined as YAML you can review in a pull request.
Optimizing Delta Tables
Fix the small file problem with OPTIMIZE, replace partitioning and Z-ORDER with liquid clustering, and measure data skipping instead of guessing at it.
Advanced
Production-grade patterns for experienced engineers
Structured Streaming on Databricks
Checkpoints and exactly-once semantics, watermarks that bound state, foreachBatch for streaming upserts, and the monitoring numbers that tell you a stream is falling behind.
Cost Control and Cluster Policies
Attribute DBUs to teams with tags, cap what people can launch with cluster policies, and decide when Photon, spot instances and serverless actually pay for themselves.