What problem does Airflow actually solve?

Running a sequence of jobs where later steps depend on earlier ones, with retries, scheduling, backfill, and visibility into what failed. A cron entry can run a script; it cannot tell you that step 3 of 7 failed at 2am, retry just that step, and skip the rest until it succeeds.

Why is my DAG not appearing in the UI?

Usually the file is not in the dags folder, has a Python syntax error, or does not assign the DAG to a module-level variable. Run airflow dags list-import-errors — it reports parse failures that the UI shows only as a missing DAG.

What is the difference between execution_date and the run time?

For a scheduled run, the logical date is the start of the interval being processed, not when the task runs. A daily DAG for 2026-02-14 runs after that day ends. This trips up everyone once, and it is deliberate — it makes runs idempotent and backfillable.

Do I need a database and a web server to try Airflow?

airflow standalone starts everything — scheduler, web server, and a SQLite database — in one command. That is fine for learning. Production needs Postgres and a real executor, since SQLite cannot support parallel task execution.

Your First Airflow DAG | SkillByExample

Airflow runs workflows defined as Python code. A workflow is a DAG — a set of tasks with dependencies between them — and Airflow’s job is to run those tasks in the right order, on a schedule, and tell you what happened.

Installing

Airflow pins its dependencies tightly, so install with the constraint file:

python -m venv .venv && source .venv/bin/activate
export AIRFLOW_VERSION=2.10.4
export PYTHON_VERSION=3.11

pip install "apache-airflow==${AIRFLOW_VERSION}" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

Successfully installed apache-airflow-2.10.4 alembic-1.13.3 flask-2.2.5 ...

Skipping the constraint file is the most common reason an Airflow install breaks — its dependency tree is large and version-sensitive.

Starting it

export AIRFLOW_HOME=~/airflow
airflow standalone

standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Running upgrade  -> 0bfc26bc256e
standalone | Database ready
standalone | Airflow is ready
standalone | Login with username: admin  password: kW9mPx2Ryt4Qd7Ln

That password is generated once and written to $AIRFLOW_HOME/standalone_admin_password.txt. The UI is at http://localhost:8080.

standalone runs the scheduler, web server, and triggerer together against SQLite. It is for learning only — SQLite forces the sequential executor, so tasks run one at a time.

The first DAG

mkdir -p ~/airflow/dags

# ~/airflow/dags/first_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator


def summarise(**context):
    """Runs in the Airflow worker process, not the scheduler."""
    logical_date = context["logical_date"]
    print(f"Summarising data for {logical_date.date()}")
    return f"summary-{logical_date.date()}"


default_args = {
    "owner": "data-team",
    "retries": 2,
    "retry_delay": timedelta(minutes=1),
}

with DAG(
    dag_id="first_dag",
    description="Extract, summarise, notify",
    default_args=default_args,
    start_date=datetime(2026, 2, 1),
    schedule="@daily",
    catchup=False,               # do not backfill every day since start_date
    tags=["tutorial"],
) as dag:

    extract = BashOperator(
        task_id="extract",
        bash_command="echo 'pulling rows'; sleep 2; echo 'pulled 1284 rows'",
    )

    summarise_task = PythonOperator(
        task_id="summarise",
        python_callable=summarise,
    )

    notify = BashOperator(
        task_id="notify",
        bash_command="echo 'pipeline finished for {{ ds }}'",
    )

    extract >> summarise_task >> notify

The >> operator sets dependencies: extract must succeed before summarise starts, and so on. That line is the whole DAG structure.

Checking Airflow parsed it

airflow dags list | grep first_dag

first_dag | /home/you/airflow/dags/first_dag.py | data-team | None | True

If it does not appear, ask why rather than guessing:

airflow dags list-import-errors

No data found

A syntax error would show here with the traceback. The UI only shows the DAG as missing, which tells you nothing.

Inspect the structure:

airflow tasks list first_dag --tree

<Task(BashOperator): extract>
    <Task(PythonOperator): summarise>
        <Task(BashOperator): notify>

Running one task

Test a single task without scheduling anything:

airflow tasks test first_dag extract 2026-02-14

[2026-02-14 11:02:14] {taskinstance.py:1157} INFO - Executing <Task(BashOperator): extract>
[2026-02-14 11:02:14] {subprocess.py:63} INFO - Running command: ['/bin/bash', '-c', "echo 'pulling rows'; sleep 2; echo 'pulled 1284 rows'"]
[2026-02-14 11:02:14] {subprocess.py:75} INFO - Output:
[2026-02-14 11:02:14] {subprocess.py:86} INFO - pulling rows
[2026-02-14 11:02:16] {subprocess.py:86} INFO - pulled 1284 rows
[2026-02-14 11:02:16] {subprocess.py:93} INFO - Command exited with return code 0
[2026-02-14 11:02:16] {taskinstance.py:1400} INFO - Marking task as SUCCESS.

tasks test runs the task immediately, ignores dependencies, and writes nothing to the metadata database. It is the fastest way to iterate on task logic.

The Python task shows the templated context:

airflow tasks test first_dag summarise 2026-02-14

[2026-02-14 11:03:01] {logging_mixin.py:188} INFO - Summarising data for 2026-02-14
[2026-02-14 11:03:01] {python.py:240} INFO - Done. Returned value was: summary-2026-02-14
[2026-02-14 11:03:01] {taskinstance.py:1400} INFO - Marking task as SUCCESS.

The returned value is pushed to XCom automatically — how other tasks read it is covered in the XCom lesson.

Running the whole DAG

airflow dags trigger first_dag

Created <DagRun first_dag @ 2026-02-14T11:05:33+00:00: manual__2026-02-14T11:05:33+00:00, state:queued>

sleep 20 && airflow dags list-runs -d first_dag

dag_id    | run_id                              | state   | execution_date            | start_date
==========+=====================================+=========+===========================+==========================
first_dag | manual__2026-02-14T11:05:33+00:00   | success | 2026-02-14T11:05:33+00:00 | 2026-02-14T11:05:34+00:00

And the individual task states:

airflow tasks states-for-dag-run first_dag manual__2026-02-14T11:05:33+00:00

dag_id    | task_id   | run_id                            | state   | start_date               | end_date
==========+===========+===================================+=========+==========================+=========================
first_dag | extract   | manual__2026-02-14T11:05:33+00:00 | success | 2026-02-14T11:05:36+00:00| 2026-02-14T11:05:39+00:00
first_dag | summarise | manual__2026-02-14T11:05:33+00:00 | success | 2026-02-14T11:05:41+00:00| 2026-02-14T11:05:42+00:00
first_dag | notify    | manual__2026-02-14T11:05:33+00:00 | success | 2026-02-14T11:05:44+00:00| 2026-02-14T11:05:45+00:00

The start times confirm the ordering — summarise began two seconds after extract finished, not alongside it.

What a failure looks like

Add a task that fails:

    check = BashOperator(
        task_id="check",
        bash_command="exit 1",
    )
    extract >> check >> summarise_task

[2026-02-14 11:12:03] {subprocess.py:93} INFO - Command exited with return code 1
[2026-02-14 11:12:03] {taskinstance.py:1938} ERROR - Task failed with exception
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2026-02-14 11:12:03] {taskinstance.py:1206} INFO - Marking task as UP_FOR_RETRY. dag_id=first_dag, task_id=check, try_number=1

airflow tasks states-for-dag-run first_dag manual__2026-02-14T11:12:00+00:00

first_dag | extract   | ... | success
first_dag | check     | ... | failed
first_dag | summarise | ... | upstream_failed
first_dag | notify    | ... | upstream_failed

check retried twice (from default_args), then failed. Everything downstream is upstream_failed rather than failed — Airflow distinguishes “this broke” from “this never got a chance to run”, which is what makes a long DAG’s failure legible.

Fix the cause and re-run only what failed:

airflow dags backfill first_dag --start-date 2026-02-14 --end-date 2026-02-14 --rerun-failed-tasks

The scheduling surprise

with DAG(dag_id="daily", start_date=datetime(2026, 2, 1), schedule="@daily") as dag:
    ...

The run labelled 2026-02-14 starts after 2026-02-14 ends — just after midnight on the 15th. The date identifies the interval being processed, not the wall clock.

airflow tasks test first_dag notify 2026-02-14

[2026-02-14 11:15:22] {subprocess.py:86} INFO - pipeline finished for 2026-02-14

{{ ds }} rendered the logical date, not today. That is what makes a re-run of an old date produce identical output — the task processes the same window whenever it runs.

Practice

1. Add a task that runs in parallel with summarise.

    audit = BashOperator(task_id="audit", bash_command="echo 'auditing'")
    extract >> [summarise_task, audit] >> notify

<Task(BashOperator): extract>
    <Task(PythonOperator): summarise>
        <Task(BashOperator): notify>
    <Task(BashOperator): audit>
        <Task(BashOperator): notify>

A list on either side of >> fans out or in. Under airflow standalone they still run sequentially — SQLite forces the sequential executor regardless of the DAG shape.

2. Introduce a syntax error and check how it surfaces.

$ airflow dags list-import-errors
filepath                          | error
==================================+===========================================
/home/you/airflow/dags/first_dag.py | Traceback (most recent call last):
                                    |   File "...", line 34
                                    |     extract >> summarise_task >>
                                    |                                ^
                                    | SyntaxError: invalid syntax

The DAG simply vanishes from airflow dags list and the UI. Make list-import-errors the first thing you check when a DAG disappears.

3. Set catchup=True with a start_date a week ago. What happens?

Created 7 DagRuns for first_dag

Airflow backfills every missed interval. That is powerful for reprocessing and dangerous by accident — a DAG with a start_date a year back and catchup=True queues 365 runs the moment it is unpaused. Default it to False and backfill deliberately.

4. What is the difference between airflow tasks test and airflow tasks run?

test executes the task immediately in the foreground, ignores dependencies, and writes nothing to the database — ideal for development. run goes through the real machinery: checks upstream state, records the attempt, honours retries, and writes logs to the configured location. Use test while writing a task, run when reproducing a scheduler problem.

Next: the operators you will actually use, and how to pick between them.