Airflow Tutorial with Examples.

Your First Airflow DAG

Install Airflow, write a three-task DAG, trigger it from the CLI, and read the scheduler's output to see what actually ran and in what order.

airflow beginner

Passing Data Between Tasks: XComs and the TaskFlow API

Move a value from one task to the next with XComs, rewrite the same DAG with the TaskFlow API, and see why XComs are the wrong place for a DataFrame.

airflow beginner

Scheduling, Logical Dates, and Backfill

Work out why a daily DAG for the 14th runs on the 15th, then use that same property to backfill a month of history correctly.

Intermediate

For developers with core concepts down

5 lessons

Dynamic Task Mapping in Airflow

Generate a variable number of parallel tasks at runtime with expand(), fan the results back in, and see why a Python loop isn't the same.

Sensors and Deferrable Operators

Wait for a file with a poke sensor, watch it hold a worker slot for an hour, then convert it to a deferrable operator that holds nothing.

Branching and Trigger Rules in Airflow

Pick a path at runtime with a branch operator, watch the skip propagate further than you expected, and use trigger rules to stop it.

Connections, Hooks, and Variables

Store credentials outside your DAG files, use hooks to talk to Postgres and S3, and see why a Variable read at parse time hammers your metadata database.

Assets: Scheduling on Data, Not on Time

Trigger a downstream DAG the moment an upstream one updates a dataset, with no cron expression and no sensor polling a filesystem for changes.

Advanced

Production-grade patterns for experienced engineers

2 lessons

airflow advanced

Testing Airflow DAGs

Catch import errors and cycles with a validation test, unit-test task logic without a scheduler, and run a whole DAG in-process with dag.test().

airflow advanced

Airflow in Production: Idempotency, Pools, and SLAs

Turn a DAG that works into one that survives — idempotent writes, pools that stop a backfill saturating a database, and alerting.