Course · 102 lessons

Data Engineering Tools Roadmap.

A beginner-to-advanced course in reliable data pipelines: batch and streaming fundamentals, Spark, Airflow orchestration, Kafka, dbt, Snowflake and Databricks.

Go from data-pipeline fundamentals to a production-grade, multi-engine data platform. This path is structured into three stages, walking the same batch → stream → transform → warehouse shape across every tool a real data engineering team touches.

Beginner — foundations

Data engineering fundamentals: file formats, idempotency, batch pipelines, and physical data layout — the safety contracts every pipeline below depends on. Plus the vocabulary and decisions that come before any of that: batch vs. streaming, data lake vs. warehouse vs. lakehouse, and star-schema data modeling basics.
Spark & PySpark: cluster architecture, RDDs, Catalyst, and your first DataFrame — reading, writing, and transforming data at scale.
Airflow: your first DAG, passing data between tasks, and scheduling/backfill.
Kafka: topics, partitions, offsets, and producing/consuming your first messages.
dbt: models, ref, sources, tests, and the view/table/ephemeral materializations.
Snowflake: warehouses, loading data, time travel, and semi-structured VARIANT data.
Databricks: the workspace, Delta Lake’s transaction log, Unity Catalog, and the medallion architecture.
R: reading/writing data and dplyr-based transformation and reshaping, for pipelines that lean on R instead of (or alongside) Python.

Intermediate — design patterns and production reliability

Data engineering: data quality checks, incremental loads and CDC, orchestration patterns that tie multiple tools together, and testing a pipeline before it ships.
Spark & PySpark: reading the Spark UI to diagnose slow jobs, spark-submit deployment, joins, window functions, and UDFs.
Airflow: dynamic task mapping, sensors, branching, and connections/hooks.
Kafka: partitioning and ordering guarantees, the schema registry, Kafka Connect, SASL/ACL security, and delivery semantics.
dbt: incremental models, Jinja/macros, and snapshots for slowly changing dimensions.
Snowflake: micro-partitions and clustering, streams/tasks, secure data sharing, and Snowpark for building pipelines as Python DataFrames instead of SQL strings.
Databricks: declarative pipelines with expectations, jobs/workflows, and optimizing Delta tables.

Advanced — correctness, performance, and cost at scale

Data engineering: schema evolution/data contracts, exactly-once streaming semantics, and lambda/kappa streaming architecture.
Spark & PySpark: memory management and debugging failures, the execution model, performance tuning, and Structured Streaming.
Scala: packaging and shipping a Spark job as a JAR with sbt.
Airflow: testing DAGs, and running Airflow in production (idempotency, pools, SLAs).
Kafka: replication/retention internals, Kafka Streams, and producer/consumer tuning.
dbt: documentation/contracts/lineage, and deployment with Slim CI.
Snowflake: query profiling/cost attribution, and dynamic tables.
Databricks: Structured Streaming and cost control with cluster policies.
R: querying databases from R with dbplyr, and building reproducible pipelines with purrr.

Staff-level — production concerns no single tool teaches

Data engineering: data lineage, governance, security and PII handling, observability, cost optimization, CI/CD for data pipelines, disaster recovery, and a capstone lesson tying lake/warehouse, orchestration, governance, and observability into one coherent data platform architecture.

The roadmap, step by step

data-engineering Step 1 of 102

Data Engineering Fundamentals

Build a complete ingest-transform-serve pipeline in fifty lines, then break it the three ways real pipelines break.

data-engineering Step 2 of 102

File Formats: CSV, JSON, Parquet

Measure the same dataset as CSV, JSON and Parquet — size, query time, column pruning — and see why a columnar format changes what queries are affordable.

data-engineering Step 3 of 102

Idempotency and Safe Retries

Every pipeline gets run twice. Four patterns that make that harmless — overwrite, delete-insert, merge, and deduplication on a natural key.

data-engineering Step 4 of 102

Batch Pipelines and Time Windows

Logical time versus wall-clock time, high-water marks that lose rows, and backfills that finish — what decides a nightly job is correct.

data-engineering Step 5 of 102

Partitioning and Physical Layout

Partition pruning measured, the cardinality mistake that creates 50,000 tiny files, and why sorting inside a partition often beats adding another one.

data-engineering Step 6 of 102

Batch vs. Streaming: The Real Trade-off

Run the same ten orders through a batch pipeline and a streaming one, and measure the actual latency and pass-count difference instead of guessing.

data-engineering Step 7 of 102

Data Lake vs. Data Warehouse vs. Lakehouse

Load the same two files three ways to see schema-on-read, schema-on-write, and the transactional guarantees a lakehouse adds on top of lake-style storage.

data-engineering Step 8 of 102

Data Modeling Basics: Star Schemas and Denormalization

Load the same order data into a normalized schema and a star schema, then run the same query to see why analytics tables trade storage for join-free reads.

spark Step 9 of 102

Spark Architecture: Driver, Executors, and Cluster Managers

Submit a job and watch the driver plan it, executors run it, and the cluster manager allocate resources — reading each component's output live.

spark Step 10 of 102

RDDs: The API Underneath DataFrames

Work directly with RDDs, measure how much slower they are than the equivalent DataFrame, and identify the few cases where you still need them.

spark Step 11 of 102

Spark SQL and the Catalyst Optimizer

Register views, query them with SQL, then read Catalyst's four plan stages to see exactly which rewrites it applied to your query.

pyspark Step 12 of 102

PySpark in Ten Minutes: Your First DataFrame

Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.

pyspark Step 13 of 102

Reading and Writing Data in PySpark

Read CSV, JSON, and Parquet with an explicit schema, handle malformed rows deliberately, and partition output so downstream reads skip more.

pyspark Step 14 of 102

Transforming Columns in PySpark

Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.

pyspark Step 15 of 102

Grouping and Aggregation in PySpark

Group, aggregate, and pivot — then read the query plan to see the shuffle that groupBy introduces and why it dominates the job's runtime.

airflow Step 16 of 102

Your First Airflow DAG

Install Airflow, write a three-task DAG, trigger it from the CLI, and read the scheduler's output to see what actually ran and in what order.

airflow Step 17 of 102

Passing Data Between Tasks: XComs and the TaskFlow API

Move a value from one task to the next with XComs, rewrite the same DAG with the TaskFlow API, and see why XComs are the wrong place for a DataFrame.

airflow Step 18 of 102

Scheduling, Logical Dates, and Backfill

Work out why a daily DAG for the 14th runs on the 15th, then use that same property to backfill a month of history correctly.

kafka Step 19 of 102

Kafka in Ten Minutes: Your First Topic

Start a Kafka broker, create a topic, produce three messages, and read them back from the console — the whole loop, end to end, in ten minutes.

kafka Step 20 of 102

Kafka Topics, Partitions, and Offsets

Watch a topic split across partitions, see how offsets are assigned per partition, and learn why Kafka only guarantees ordering inside a single partition.

kafka Step 21 of 102

Producing to Kafka from Python

Send records with confluent-kafka, read the broker's acknowledgement, handle delivery failures, and see why the producer is asynchronous by default.

kafka Step 22 of 102

Kafka Consumers and Consumer Groups

Read records from Python, watch two consumers split the partitions of a topic, and see what a rebalance does to in-flight work.

dbt Step 23 of 102

Your First dbt Model

Set up a dbt project on DuckDB, load seed data, build a model with dbt run, and read the SQL dbt actually sent to the warehouse.

dbt Step 24 of 102

Models, ref, and the DAG

How ref builds a dependency graph, why layering into staging and marts pays off, and the selector syntax that runs exactly the models you changed.

dbt Step 25 of 102

Sources and Freshness Checks

Declare raw tables as sources so lineage starts at the loader, then let dbt source freshness fail the build when data stops arriving.

dbt Step 26 of 102

Testing Data with dbt

Generic tests, singular tests, and custom ones — plus how to see the rows that failed and why dbt build stops a bad model reaching your marts.

dbt Step 27 of 102

Materializations: View, Table, Ephemeral

The same SELECT stored four different ways — what dbt runs for each, how config precedence resolves, and how to choose without guessing.

snowflake Step 28 of 102

Snowflake Architecture and Your First Queries

Create a warehouse, a database and a table, then see why separating storage from compute means two teams can query the same data without competing.

snowflake Step 29 of 102

Virtual Warehouses: Sizing, Scaling, and Cost

Scale up for one slow query, scale out for many concurrent ones — with credit maths, auto-suspend, and resource monitors that stop a runaway bill.

snowflake Step 30 of 102

Loading Data: Stages and COPY INTO

Get files into tables with stages, file formats and COPY INTO — including validation, error handling, and why re-running a load does not duplicate rows.

snowflake Step 31 of 102

Time Travel and Zero-Copy Cloning

Query a table as it was before a bad update, UNDROP what you deleted, and clone a whole database in seconds without paying for a second copy.

snowflake Step 32 of 102

Semi-Structured Data: VARIANT, JSON, and FLATTEN

Store JSON in a VARIANT column, read it with path notation, explode arrays with LATERAL FLATTEN, and know when to promote a field to a real column.

databricks Step 33 of 102

The Databricks Workspace and Compute

Pick the right compute for the job, run your first notebook against a Delta table, and understand what you are billed for while it runs.

databricks Step 34 of 102

Delta Lake: The Transaction Log

ACID on object storage — read the _delta_log, travel back to a previous version, MERGE upserts, and see schema enforcement reject a bad write.

databricks Step 35 of 102

Unity Catalog: Namespaces, Grants, and Lineage

The three-level namespace, managed versus external tables, volumes for files, and lineage you get without instrumenting anything.

databricks Step 36 of 102

Ingesting Files with Auto Loader and COPY INTO

Incremental file ingestion that remembers what it loaded, infers a schema, evolves when a column appears, and rescues the data that does not fit.

databricks Step 37 of 102

The Medallion Architecture

Bronze, silver and gold as three concrete tables — what belongs in each layer, why bronze keeps the bad rows, and when it's more than you need.

r Step 38 of 102

Getting Started with R

Run R from a script rather than a console, load a dataset into a tibble, and learn the two vector behaviours that surprise everyone coming from Python.

r Step 39 of 102

Reading and Writing Data

CSV with declared types, Parquet with arrow, partitioned datasets queried lazily, and handling a file larger than memory without leaving R.

r Step 40 of 102

Transforming Data with dplyr

The five verbs that cover most pipeline work, grouped summaries with .by, across() for column sets, and the grouped-tibble trap to watch for.

r Step 41 of 102

Tidying and Reshaping Data

Pivot between wide and long with tidyr, split and combine columns, unnest list-columns from JSON, and fill the gaps a time series leaves behind.

data-engineering Step 42 of 102

Data Quality Checks That Run Every Load

Six checks worth writing, quarantine instead of dropping, thresholds against a trailing average, and a reconciliation test that catches the rest.

data-engineering Step 43 of 102

Incremental Loads and Change Data Capture

Four ways to extract only what changed, why deletes are the hard part, and building SCD type 2 history from a CDC stream that arrives out of order.

data-engineering Step 44 of 102

Orchestration Patterns

Task granularity, dependencies that skip rather than fail, retries with backoff, and the sensor pattern that waits for data instead of hoping it arrived.

data-engineering Step 45 of 102

Testing Data Pipelines

Write tests for a transform function, then let a 'harmless' refactor slip through code review and see the test suite catch it instead of production.

spark Step 46 of 102

Reading the Spark UI to Diagnose a Slow Job

Work through the Jobs, Stages, SQL, and Executors tabs on a deliberately broken job, and learn which number to look at first for each kind of slowness.

spark Step 47 of 102

Deploying Spark Jobs with spark-submit

Package an application, submit it to a cluster, and work through failures that only appear outside the shell — missing jars and wrong Python.

pyspark Step 48 of 102

Joins and Broadcast Joins in PySpark

Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.

pyspark Step 49 of 102

Window Functions in PySpark

Rank rows within groups, compute running totals, and compare each row to its neighbours — without collapsing the DataFrame the way groupBy does.

pyspark Step 50 of 102

UDFs and Pandas UDFs in PySpark

Measure what a Python UDF costs against a built-in function, then rewrite it as a pandas UDF and watch most of that cost disappear.

pyspark Step 51 of 102

MLlib Basics: A Real Pipeline, Fit and Evaluated

Assemble feature vectors, chain them into a Pipeline with a classifier, and evaluate the fitted model with AUC and accuracy on held-out data.

airflow Step 52 of 102

Dynamic Task Mapping in Airflow

Generate a variable number of parallel tasks at runtime with expand(), fan the results back in, and see why a Python loop isn't the same.

airflow Step 53 of 102

Sensors and Deferrable Operators

Wait for a file with a poke sensor, watch it hold a worker slot for an hour, then convert it to a deferrable operator that holds nothing.

airflow Step 54 of 102

Branching and Trigger Rules in Airflow

Pick a path at runtime with a branch operator, watch the skip propagate further than you expected, and use trigger rules to stop it.

airflow Step 55 of 102

Connections, Hooks, and Variables

Store credentials outside your DAG files, use hooks to talk to Postgres and S3, and see why a Variable read at parse time hammers your metadata database.

airflow Step 56 of 102

Assets: Scheduling on Data, Not on Time

Trigger a downstream DAG the moment an upstream one updates a dataset, with no cron expression and no sensor polling a filesystem for changes.

kafka Step 57 of 102

Keys, Partitioning, and Ordering Guarantees

See how a record key decides its partition, why that is the only ordering guarantee Kafka gives you, and what a custom partitioner changes.

kafka Step 58 of 102

Serialization and the Schema Registry

Break a consumer by changing a JSON field, then fix it properly with Avro and a Schema Registry that rejects incompatible changes before they ship.

kafka Step 59 of 102

Kafka Connect: Source and Sink Connectors, and a Live Offset

Move data in and out of Kafka without writing a producer or consumer — a real FileStreamSource connector, a FileStreamSink connector, and the REST API that reports exactly how far each one has read.

kafka Step 60 of 102

Kafka Security: SASL Authentication and a Real ACL Denial

A default Kafka cluster trusts every connection completely. Turn on SASL/PLAIN and an ACL authorizer, and watch an unauthorized client fail with the exact exception Kafka throws — for a topic, then for a consumer group.

kafka Step 61 of 102

At-Most-Once, At-Least-Once, and Exactly-Once

Reproduce message loss and duplication on purpose, then configure the producer and consumer to get each of Kafka's three delivery guarantees.

dbt Step 62 of 102

Incremental Models

Build only the new rows: is_incremental, unique_key, merge strategies, and the late-arriving data problem that silently loses records.

dbt Step 63 of 102

Jinja and Macros

Generate repetitive SQL with loops, factor logic into macros, read what dbt compiled, and query the warehouse at compile time with run_query.

dbt Step 64 of 102

Snapshots and Slowly Changing Dimensions

Source tables overwrite history. Snapshots capture it — the timestamp and check strategies, the dbt_valid_from columns, and how to query a row as it was.

snowflake Step 65 of 102

Micro-Partitions, Pruning, and Clustering

Why one filter scans the whole table and another scans 2% of it — reading partition metadata, measuring pruning, and pricing a clustering key.

snowflake Step 66 of 102

Streams and Tasks

Capture changes with a stream, consume them exactly once, and schedule the work with tasks — including task DAGs and a silent suspend to avoid.

snowflake Step 67 of 102

Roles, Masking, and Secure Data Sharing

Build a role hierarchy that survives a new schema, hide columns with masking policies, filter rows per role, and share data without copying it.

snowflake Step 68 of 102

Snowpark: Python DataFrames on Snowflake

Write Snowflake pipelines as Python DataFrame code instead of SQL strings, and use Snowflake's own local testing framework to run it without a warehouse.

databricks Step 69 of 102

Declarative Pipelines and Data Expectations

Define tables and quality rules, and let the pipeline derive the DAG — with expectations that drop, quarantine or fail, and CDC handled by AUTO CDC.

databricks Step 70 of 102

Jobs, Workflows, and Asset Bundles

Multi-task jobs with dependencies, parameters passed between tasks, retries and conditional runs — then the same job as reviewable YAML.

databricks Step 71 of 102

Optimizing Delta Tables

Fix the small file problem with OPTIMIZE, replace partitioning and Z-ORDER with liquid clustering, and measure data skipping instead of guessing at it.

data-engineering Step 72 of 102

Schema Evolution and Data Contracts

Which schema changes are safe, why backward and forward compatibility differ, and enforcing a contract in CI so a breaking change fails the build.

data-engineering Step 73 of 102

Streaming: Exactly-Once Semantics

Simulate a consumer crash and watch at-least-once produce duplicates, then fix it three ways — atomic commits, idempotent keys, dedup on read.

data-engineering Step 74 of 102

Streaming Architecture: Lambda and Kappa

Build a lambda architecture's batch and speed layers by hand, watch a late event slip past both, then see why kappa makes reconciliation unnecessary.

spark Step 75 of 102

Spark Memory Management and Debugging Failures

Read Spark's memory model, then reproduce and fix the three failures that actually take jobs down — driver OOM, executor OOM, and disk spill.

pyspark Step 76 of 102

The Spark Execution Model: Jobs, Stages, and Caching

Trace one query from DataFrame to tasks, see where stage boundaries come from, and prove that caching changes the plan rather than just the timing.

pyspark Step 77 of 102

Tuning PySpark Jobs

Diagnose and fix the four things that actually make Spark jobs slow — skew, small files, spill, and wasted shuffles — measuring each one.

pyspark Step 78 of 102

Structured Streaming in PySpark

Run the same DataFrame code over an unbounded stream — with windows, watermarks, checkpoints, and output modes that decide what you compute.

scala Step 79 of 102

sbt, Packaging, and Shipping a JAR

Graduate from scala-cli to sbt, manage dependencies and modules, and build an assembly JAR that runs under spark-submit without dragging Spark.

scala Step 80 of 102

Calling Java from Scala, and Scala from Java

Convert java.util.Optional and java.util.List at the boundary, then read the javap output that shows why Java sees two .class files for one Scala object.

airflow Step 81 of 102

Testing Airflow DAGs

Catch import errors and cycles with a validation test, unit-test task logic without a scheduler, and run a whole DAG in-process with dag.test().

airflow Step 82 of 102

Airflow in Production: Idempotency, Pools, and SLAs

Turn a DAG that works into one that survives — idempotent writes, pools that stop a backfill saturating a database, and alerting.

kafka Step 83 of 102

Replication, ISR, Retention, and Compaction

Kill a broker and watch leadership move, then see the difference between deleting old records by time and compacting down to the latest value.

kafka Step 84 of 102

Stream Processing with Kafka Streams

Build a word count, a windowed aggregation, and a stream-table join in Kafka Streams, and see where the state actually lives when your app restarts.

kafka Step 85 of 102

Tuning Kafka Producers and Consumers

Benchmark batching, compression, and fetch sizing against a real broker, and read the metrics that tell you which one is actually your bottleneck.

dbt Step 86 of 102

The Semantic Layer: Defining Metrics Once with MetricFlow

Define revenue as one governed metric instead of retyping SUM(amount) in every dashboard, then query it with mf and hit two real MetricFlow gotchas.

dbt Step 87 of 102

Documentation, Contracts, and Lineage

Doc blocks, exposures and persisted comments, then model contracts and access levels that turn a mart into a stable interface other teams can build on.

dbt Step 88 of 102

Deployment and Slim CI

Run dbt in production: environments and targets, artifacts, state:modified with defer for CI that builds only what changed, and retrying a failed run.

snowflake Step 89 of 102

Query Profiling and Cost Attribution

Read the query profile to find the expensive operator, fix exploding joins and spilling, and attribute credits to the team that spent them with query tags.

snowflake Step 90 of 102

Dynamic Tables

Declare the result you want and a freshness target, and Snowflake keeps it up to date incrementally — replacing most stream-and-task pipelines.

databricks Step 91 of 102

Structured Streaming on Databricks

Checkpoints and exactly-once semantics, watermarks that bound state, foreachBatch for streaming upserts, and the numbers that show a stream falling behind.

databricks Step 92 of 102

Cost Control and Cluster Policies

Attribute DBUs to teams with tags, cap what people can launch with cluster policies, and decide when Photon and spot instances pay off.

r Step 93 of 102

Databases and dbplyr

Write dplyr and have it execute as SQL on the warehouse — lazy tables, show_query, where translation stops, and choosing R, DuckDB or data.table.

r Step 94 of 102

Functions, purrr, and Reproducible Pipelines

Write functions that take column names, iterate with purrr instead of loops, catch failures without stopping, and pin an environment for later.

data-engineering Step 95 of 102

Data Lineage: Tracing a Column Back to Its Source

Build a small lineage graph by hand, trace one output column back to the CSV it came from, and run impact analysis on a source change before it ships.

data-engineering Step 96 of 102

Data Governance: Catalogs, Classification, and Access Policy

Classify every column in a catalog by sensitivity, enforce role-based access against it, and watch three roles get three different answers to one query.

data-engineering Step 97 of 102

Data Security and PII: Hashing, Masking, and Tokenization

Run the same email and SSN through three PII techniques and see what each gives up: hashing for joins, masking for display, tokenization for reversal.

data-engineering Step 98 of 102

Pipeline Observability: Tracing, Anomaly Detection, and Alerting

Trace a pipeline run stage-by-stage, then flag a run 3x longer than usual and an error rate crossing its threshold - operational health, not data quality.

data-engineering Step 99 of 102

Cost Optimization: Measuring What Partition and Column Pruning Actually Save

Write twelve months of Parquet files, then measure the real bytes and time saved by partition pruning and column pruning instead of taking it on faith.

100

data-engineering Step 100 of 102

CI/CD for Data: Gating Promotion with Real Tests

Promote a staging table to production only when it passes real gate checks, and watch a build with a duplicate key get blocked while a clean build ships.

101

data-engineering Step 101 of 102

Disaster Recovery: RPO, RTO, and What a Backup Actually Loses

Back up a database, keep writing to it, lose it entirely, then restore - and measure the real RPO and RTO numbers instead of quoting them from a slide.

102

data-engineering Step 102 of 102

Data Platform Architecture: Wiring the Pieces Together

Run one pipeline through a lake, a quality gate, and a warehouse layer, capturing lineage and a trace - then watch the gate halt a bad batch.