Skip to main content
Course · 55 lessons

Data Engineer Roadmap.

A course from Python fundamentals to a working data engineering toolchain: Spark, Airflow, Kafka, dbt and a cloud warehouse, built as a practical on-ramp.

Become productive as a data engineer: start from Python fundamentals, build one pipeline through every core tool a real team uses, then harden it until it’s the kind of thing you’d actually put on-call for. This is a on-ramp, not full mastery — once you’re through it, the Data Engineering Tools Roadmap covers every tool here in much greater depth, plus Scala, R, and the advanced correctness/cost/governance material this path deliberately leaves for later.

Foundations — Python for data work

  • Python: variables, data types, control flow, strings, data structures, functions, error handling, file I/O, and modules — the language fundamentals every pipeline below is written in.

Core toolchain — one pipeline, every tool

  • Data engineering fundamentals: file formats, your first batch pipeline, the batch-vs-streaming decision, and data lake/warehouse/lakehouse vocabulary.
  • Spark & PySpark: your first DataFrame — reading, writing, transforming, and aggregating data.
  • Airflow: your first DAG, passing data between tasks, and scheduling/backfill.
  • Kafka: topics, partitions, offsets, and producing/consuming your first messages.
  • Data modeling basics: star schemas and fact/dimension tables, right before dbt turns them into real models.
  • dbt: models, ref, sources, and tests.
  • Snowflake: warehouses, compute, and loading data.

On the job — reliability, not just a working demo

  • Data engineering: idempotency and retries, partitioning and physical layout, data quality checks, incremental loads/CDC, and testing a pipeline before it ships.
  • Spark & PySpark: joins and window functions.
  • Airflow: sensors and connections/hooks for talking to the outside world.
  • Kafka: delivery semantics and schema serialization.
  • dbt: incremental models.
  • Snowflake: micro-partitions and clustering.

Growing further — a taste of what’s next

  • Data engineering: schema evolution and contracts.
  • Spark & PySpark: performance tuning.
  • Databricks: the lakehouse platform and the medallion architecture.
  • Airflow: production hardening patterns.
  • dbt: deployment and CI.

Ready for more? The Data Engineering Tools Roadmap picks up from here with full depth on every tool above, plus Scala packaging, R pipelines, and the streaming-exactly-once, cost-governance, and security material that a first 90 days on the job doesn’t usually need yet.

The roadmap, step by step

1
python Step 1 of 55

Introduction to Python

Learn what Python is, its history, and real-world use cases — a practical Python tutorial for beginners starting from scratch.

2
python Step 2 of 55

Setting Up Python

Install Python, create and activate a virtual environment, install and freeze packages with pip, and run a first script, with real command output.

3
python Step 3 of 55

Python Variables and Type System

See how Python variables work by running them: names as references, mutable vs immutable objects, dynamic typing, PEP 8 names and LEGB scope.

4
python Step 4 of 55

Python Data Types

Master Python's built-in data types — int, float, str, bool, None — and how to inspect and convert between them with real type conversion.

5
python Step 5 of 55

Control Flow in Python

Run if/elif/else, match-case, for and while loops, for/else, break/continue and comprehensions in Python, each example shown with its real output.

6
python Step 6 of 55

Python Strings

Slice, search, split, join and format Python strings with f-strings and format specs, plus escapes and raw strings, each example with real output.

7
python Step 7 of 55

Python Data Structures

Use Python lists, tuples, dicts, sets and comprehensions through small runnable programs, plus a real timing of list vs set membership tests.

8
python Step 8 of 55

Python Collections

Work through list, dict, set and tuple, then deque, defaultdict, Counter and namedtuple from the collections module, each with its real printed output.

9
python Step 9 of 55

Python Functions and Parameters

Define Python functions with positional, keyword, default, *args and **kwargs parameters, see the mutable-default trap, lambdas and return values run.

10
python Step 10 of 55

Error Handling in Python

Handle exceptions with try/except/else/finally, chain and re-raise them, define custom exceptions, use except* and logging, all with real tracebacks.

11
python Step 11 of 55

File I/O, pathlib, and JSON in Python

Read and write text files with open() and with, choose the right file mode, navigate paths with pathlib, and round-trip JSON and CSV, all with real output.

12
python Step 12 of 55

Python Modules and Packages

Build a real package step by step: import styles, the __name__ guard, __init__.py and __all__, relative imports, sys.path, venv and pyproject.toml.

13
data-engineering Step 13 of 55

Data Engineering Fundamentals

Build a complete ingest-transform-serve pipeline in fifty lines, then break it the three ways real pipelines break.

14
data-engineering Step 14 of 55

File Formats: CSV, JSON, Parquet

Measure the same dataset as CSV, JSON and Parquet — size, query time, column pruning — and see why a columnar format changes what queries are affordable.

15
data-engineering Step 15 of 55

Batch Pipelines and Time Windows

Logical time versus wall-clock time, high-water marks that lose rows, and backfills that finish — what decides a nightly job is correct.

16
data-engineering Step 16 of 55

Batch vs. Streaming: The Real Trade-off

Run the same ten orders through a batch pipeline and a streaming one, and measure the actual latency and pass-count difference instead of guessing.

17
data-engineering Step 17 of 55

Data Lake vs. Data Warehouse vs. Lakehouse

Load the same two files three ways to see schema-on-read, schema-on-write, and the transactional guarantees a lakehouse adds on top of lake-style storage.

18
pyspark Step 18 of 55

PySpark in Ten Minutes: Your First DataFrame

Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.

19
pyspark Step 19 of 55

Reading and Writing Data in PySpark

Read CSV, JSON, and Parquet with an explicit schema, handle malformed rows deliberately, and partition output so downstream reads skip more.

20
pyspark Step 20 of 55

Transforming Columns in PySpark

Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.

21
pyspark Step 21 of 55

Grouping and Aggregation in PySpark

Group, aggregate, and pivot — then read the query plan to see the shuffle that groupBy introduces and why it dominates the job's runtime.

22
airflow Step 22 of 55

Your First Airflow DAG

Install Airflow, write a three-task DAG, trigger it from the CLI, and read the scheduler's output to see what actually ran and in what order.

23
airflow Step 23 of 55

Passing Data Between Tasks: XComs and the TaskFlow API

Move a value from one task to the next with XComs, rewrite the same DAG with the TaskFlow API, and see why XComs are the wrong place for a DataFrame.

24
airflow Step 24 of 55

Scheduling, Logical Dates, and Backfill

Work out why a daily DAG for the 14th runs on the 15th, then use that same property to backfill a month of history correctly.

25
kafka Step 25 of 55

Kafka in Ten Minutes: Your First Topic

Start a Kafka broker, create a topic, produce three messages, and read them back from the console — the whole loop, end to end, in ten minutes.

26
kafka Step 26 of 55

Kafka Topics, Partitions, and Offsets

Watch a topic split across partitions, see how offsets are assigned per partition, and learn why Kafka only guarantees ordering inside a single partition.

27
kafka Step 27 of 55

Producing to Kafka from Python

Send records with confluent-kafka, read the broker's acknowledgement, handle delivery failures, and see why the producer is asynchronous by default.

28
kafka Step 28 of 55

Kafka Consumers and Consumer Groups

Read records from Python, watch two consumers split the partitions of a topic, and see what a rebalance does to in-flight work.

29
data-engineering Step 29 of 55

Data Modeling Basics: Star Schemas and Denormalization

Load the same order data into a normalized schema and a star schema, then run the same query to see why analytics tables trade storage for join-free reads.

30
dbt Step 30 of 55

Your First dbt Model

Set up a dbt project on DuckDB, load seed data, build a model with dbt run, and read the SQL dbt actually sent to the warehouse.

31
dbt Step 31 of 55

Models, ref, and the DAG

How ref builds a dependency graph, why layering into staging and marts pays off, and the selector syntax that runs exactly the models you changed.

32
dbt Step 32 of 55

Sources and Freshness Checks

Declare raw tables as sources so lineage starts at the loader, then let dbt source freshness fail the build when data stops arriving.

33
dbt Step 33 of 55

Testing Data with dbt

Generic tests, singular tests, and custom ones — plus how to see the rows that failed and why dbt build stops a bad model reaching your marts.

34
snowflake Step 34 of 55

Snowflake Architecture and Your First Queries

Create a warehouse, a database and a table, then see why separating storage from compute means two teams can query the same data without competing.

35
snowflake Step 35 of 55

Virtual Warehouses: Sizing, Scaling, and Cost

Scale up for one slow query, scale out for many concurrent ones — with credit maths, auto-suspend, and resource monitors that stop a runaway bill.

36
snowflake Step 36 of 55

Loading Data: Stages and COPY INTO

Get files into tables with stages, file formats and COPY INTO — including validation, error handling, and why re-running a load does not duplicate rows.

37
data-engineering Step 37 of 55

Idempotency and Safe Retries

Every pipeline gets run twice. Four patterns that make that harmless — overwrite, delete-insert, merge, and deduplication on a natural key.

38
data-engineering Step 38 of 55

Partitioning and Physical Layout

Partition pruning measured, the cardinality mistake that creates 50,000 tiny files, and why sorting inside a partition often beats adding another one.

39
data-engineering Step 39 of 55

Data Quality Checks That Run Every Load

Six checks worth writing, quarantine instead of dropping, thresholds against a trailing average, and a reconciliation test that catches the rest.

40
data-engineering Step 40 of 55

Incremental Loads and Change Data Capture

Four ways to extract only what changed, why deletes are the hard part, and building SCD type 2 history from a CDC stream that arrives out of order.

41
data-engineering Step 41 of 55

Testing Data Pipelines

Write tests for a transform function, then let a 'harmless' refactor slip through code review and see the test suite catch it instead of production.

42
pyspark Step 42 of 55

Joins and Broadcast Joins in PySpark

Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.

43
pyspark Step 43 of 55

Window Functions in PySpark

Rank rows within groups, compute running totals, and compare each row to its neighbours — without collapsing the DataFrame the way groupBy does.

44
airflow Step 44 of 55

Sensors and Deferrable Operators

Wait for a file with a poke sensor, watch it hold a worker slot for an hour, then convert it to a deferrable operator that holds nothing.

45
airflow Step 45 of 55

Connections, Hooks, and Variables

Store credentials outside your DAG files, use hooks to talk to Postgres and S3, and see why a Variable read at parse time hammers your metadata database.

46
kafka Step 46 of 55

At-Most-Once, At-Least-Once, and Exactly-Once

Reproduce message loss and duplication on purpose, then configure the producer and consumer to get each of Kafka's three delivery guarantees.

47
kafka Step 47 of 55

Serialization and the Schema Registry

Break a consumer by changing a JSON field, then fix it properly with Avro and a Schema Registry that rejects incompatible changes before they ship.

48
dbt Step 48 of 55

Incremental Models

Build only the new rows: is_incremental, unique_key, merge strategies, and the late-arriving data problem that silently loses records.

49
snowflake Step 49 of 55

Micro-Partitions, Pruning, and Clustering

Why one filter scans the whole table and another scans 2% of it — reading partition metadata, measuring pruning, and pricing a clustering key.

50
data-engineering Step 50 of 55

Schema Evolution and Data Contracts

Which schema changes are safe, why backward and forward compatibility differ, and enforcing a contract in CI so a breaking change fails the build.

51
pyspark Step 51 of 55

Tuning PySpark Jobs

Diagnose and fix the four things that actually make Spark jobs slow — skew, small files, spill, and wasted shuffles — measuring each one.

52
databricks Step 52 of 55

The Databricks Workspace and Compute

Pick the right compute for the job, run your first notebook against a Delta table, and understand what you are billed for while it runs.

53
databricks Step 53 of 55

The Medallion Architecture

Bronze, silver and gold as three concrete tables — what belongs in each layer, why bronze keeps the bad rows, and when it's more than you need.

54
airflow Step 54 of 55

Airflow in Production: Idempotency, Pools, and SLAs

Turn a DAG that works into one that survives — idempotent writes, pools that stop a backfill saturating a database, and alerting.

55
dbt Step 55 of 55

Deployment and Slim CI

Run dbt in production: environments and targets, artifacts, state:modified with defer for CI that builds only what changed, and retrying a failed run.