Data Engineering Tutorial with Examples.
Data engineering tutorial with runnable pipelines: ingestion, transformation, orchestration and performance tuning across batch and streaming.
Beginner
Start here — no prior experience needed
Data Engineering Fundamentals
Build a complete ingest-transform-serve pipeline in fifty lines, then break it the three ways real pipelines break.
File Formats: CSV, JSON, Parquet
Measure the same dataset as CSV, JSON and Parquet — size, query time, column pruning — and see why a columnar format changes what queries are affordable.
Idempotency and Safe Retries
Every pipeline gets run twice. Four patterns that make that harmless — overwrite, delete-insert, merge, and deduplication on a natural key.
Batch Pipelines and Time Windows
Logical time versus wall-clock time, high-water marks that lose rows, and backfills that finish — what decides a nightly job is correct.
Partitioning and Physical Layout
Partition pruning measured, the cardinality mistake that creates 50,000 tiny files, and why sorting inside a partition often beats adding another one.
Batch vs. Streaming: The Real Trade-off
Run the same ten orders through a batch pipeline and a streaming one, and measure the actual latency and pass-count difference instead of guessing.
Data Lake vs. Data Warehouse vs. Lakehouse
Load the same two files three ways to see schema-on-read, schema-on-write, and the transactional guarantees a lakehouse adds on top of lake-style storage.
Data Modeling Basics: Star Schemas and Denormalization
Load the same order data into a normalized schema and a star schema, then run the same query to see why analytics tables trade storage for join-free reads.
Intermediate
For developers with core concepts down
Data Quality Checks That Run Every Load
Six checks worth writing, quarantine instead of dropping, thresholds against a trailing average, and a reconciliation test that catches the rest.
Incremental Loads and Change Data Capture
Four ways to extract only what changed, why deletes are the hard part, and building SCD type 2 history from a CDC stream that arrives out of order.
Orchestration Patterns
Task granularity, dependencies that skip rather than fail, retries with backoff, and the sensor pattern that waits for data instead of hoping it arrived.
Testing Data Pipelines
Write tests for a transform function, then let a 'harmless' refactor slip through code review and see the test suite catch it instead of production.
Advanced
Production-grade patterns for experienced engineers
Streaming: Exactly-Once Semantics
Simulate a consumer crash and watch at-least-once produce duplicates, then fix it three ways — atomic commits, idempotent keys, dedup on read.
Schema Evolution and Data Contracts
Which schema changes are safe, why backward and forward compatibility differ, and enforcing a contract in CI so a breaking change fails the build.
Streaming Architecture: Lambda and Kappa
Build a lambda architecture's batch and speed layers by hand, watch a late event slip past both, then see why kappa makes reconciliation unnecessary.
Data Lineage: Tracing a Column Back to Its Source
Build a small lineage graph by hand, trace one output column back to the CSV it came from, and run impact analysis on a source change before it ships.
Data Governance: Catalogs, Classification, and Access Policy
Classify every column in a catalog by sensitivity, enforce role-based access against it, and watch three roles get three different answers to one query.
Data Security and PII: Hashing, Masking, and Tokenization
Run the same email and SSN through three PII techniques and see what each gives up: hashing for joins, masking for display, tokenization for reversal.
Pipeline Observability: Tracing, Anomaly Detection, and Alerting
Trace a pipeline run stage-by-stage, then flag a run 3x longer than usual and an error rate crossing its threshold - operational health, not data quality.
Cost Optimization: Measuring What Partition and Column Pruning Actually Save
Write twelve months of Parquet files, then measure the real bytes and time saved by partition pruning and column pruning instead of taking it on faith.
CI/CD for Data: Gating Promotion with Real Tests
Promote a staging table to production only when it passes real gate checks, and watch a build with a duplicate key get blocked while a clean build ships.
Disaster Recovery: RPO, RTO, and What a Backup Actually Loses
Back up a database, keep writing to it, lose it entirely, then restore - and measure the real RPO and RTO numbers instead of quoting them from a slide.
Data Platform Architecture: Wiring the Pieces Together
Run one pipeline through a lake, a quality gate, and a warehouse layer, capturing lineage and a trace - then watch the gate halt a bad batch.
Also part of Data Engineering
Taught in other topics. Each link opens the lesson in its home topic.
Your First Airflow DAG
Install Airflow, write a three-task DAG, trigger it from the CLI, and read the scheduler's output to see what actually ran and in what order.
Kafka Topics, Partitions, and Offsets
Watch a topic split across partitions, see how offsets are assigned per partition, and learn why Kafka only guarantees ordering inside a single partition.
Spark Architecture: Driver, Executors, and Cluster Managers
Submit a job and watch the driver plan it, executors run it, and the cluster manager allocate resources — reading each component's output live.
Models, ref, and the DAG
How ref builds a dependency graph, why layering into staging and marts pays off, and the selector syntax that runs exactly the models you changed.
Delta Lake: The Transaction Log
ACID on object storage — read the _delta_log, travel back to a previous version, MERGE upserts, and see schema enforcement reject a bad write.
The Medallion Architecture
Bronze, silver and gold as three concrete tables — what belongs in each layer, why bronze keeps the bad rows, and when it's more than you need.
File I/O, pathlib, and JSON in Python
Read and write text files with open() and with, choose the right file mode, navigate paths with pathlib, and round-trip JSON and CSV, all with real output.
Error Handling in Python
Handle exceptions with try/except/else/finally, chain and re-raise them, define custom exceptions, use except* and logging, all with real tracebacks.
Terraform Fundamentals (IaC)
Write a first Terraform config, apply it, change it and destroy it, seeing the real plan each time and why Terraform compares config, state and reality.