How do I pass a value from one Databricks task to another?

Set it with `dbutils.jobs.taskValues.set` and read it downstream with `taskValues.get`, naming the producing task. Values are small — use them for counts, dates and flags, and pass large results through a table.

Should tasks share a job cluster?

Usually yes. A shared job cluster starts once for the whole run instead of once per task, which removes several minutes of startup from a multi-task job. Give a task its own cluster only when it needs different resources, such as a GPU or far more memory.

How do retries work in Databricks Jobs?

Set `max_retries` and a retry interval per task. Retries re-run only the failed task, not the whole job, and downstream tasks wait. Make tasks idempotent first — a retried non-idempotent append duplicates data.

What are Databricks Asset Bundles?

A YAML definition of jobs, pipelines and their configuration, deployed with the `databricks bundle` CLI. They put orchestration in version control with per-environment targets, so a job change is reviewed in a pull request rather than clicked in a UI.

Jobs, Workflows, and Asset Bundles

A job is a DAG of tasks. Each task runs a notebook, a SQL query, a pipeline, a dbt project or a wheel, and the interesting part is what happens between them — dependencies, parameters, retries, and what runs when something fails.

A multi-task job

{
  "name": "bookshop-nightly",
  "job_clusters": [{
    "job_cluster_key": "shared",
    "new_cluster": {
      "spark_version": "16.4.x-scala2.13",
      "node_type_id": "m6gd.xlarge",
      "autoscale": {"min_workers": 2, "max_workers": 8},
      "data_security_mode": "SINGLE_USER"
    }
  }],
  "tasks": [
    {
      "task_key": "ingest",
      "job_cluster_key": "shared",
      "notebook_task": {
        "notebook_path": "/Repos/data-eng/bookshop/01_ingest",
        "base_parameters": {"run_date": "{{job.parameters.run_date}}"}
      },
      "max_retries": 2,
      "min_retry_interval_millis": 60000
    },
    {
      "task_key": "transform",
      "depends_on": [{"task_key": "ingest"}],
      "pipeline_task": {"pipeline_id": "8f2c1a44-8e21-4b0e-9a3c-1d84f0b27a51"}
    },
    {
      "task_key": "publish",
      "depends_on": [{"task_key": "transform"}],
      "job_cluster_key": "shared",
      "sql_task": {
        "warehouse_id": "a1b2c3d4e5f60718",
        "query": {"query_id": "3d9c4b21-77a1-4e02-b8f1-5c2e91a4d883"}
      }
    }
  ],
  "schedule": {"quartz_cron_expression": "0 0 5 * * ?", "timezone_id": "Europe/London"},
  "parameters": [{"name": "run_date", "default": "{{job.start_time.iso_date}}"}],
  "email_notifications": {"on_failure": ["[email protected]"]}
}

Run 4482 of bookshop-nightly

  ingest      SUCCESS   2m 14s
  transform   SUCCESS   4m 02s
  publish     SUCCESS      41s

Run completed in 7m 12s

The job_clusters block is the cost decision. All three tasks share one cluster, so the 4-minute startup is paid once instead of three times — on a job with ten tasks that is most of the runtime.

{{job.start_time.iso_date}} is a dynamic value reference. Others worth knowing: {{job.id}}, {{job.run_id}}, {{task.name}}, and {{job.parameters.<name>}}.

Passing values between tasks

# in the ingest notebook
row_count = spark.table("bookshop.bronze.orders").count()
dbutils.jobs.taskValues.set(key="rows_ingested", value=row_count)
print(f"ingested {row_count} rows")

ingested 13932 rows

# in a downstream notebook
rows = dbutils.jobs.taskValues.get(taskKey="ingest", key="rows_ingested", default=0)
print(f"upstream ingested {rows} rows")
if rows == 0:
    dbutils.notebook.exit("no data — nothing to publish")

upstream ingested 13932 rows

Task values are for small facts — counts, dates, flags. Anything larger goes through a table; they are not a data channel.

dbutils.notebook.exit ends a task successfully with a message, which is how you signal “nothing to do” without a spurious failure.

Conditional execution

{
  "task_key": "check_freshness",
  "condition_task": {
    "op": "GREATER_THAN",
    "left": "{{tasks.ingest.values.rows_ingested}}",
    "right": "0"
  }
},
{
  "task_key": "publish",
  "depends_on": [{"task_key": "check_freshness", "outcome": "true"}],
  "notebook_task": {"notebook_path": "/Repos/data-eng/bookshop/03_publish"}
},
{
  "task_key": "alert_no_data",
  "depends_on": [{"task_key": "check_freshness", "outcome": "false"}],
  "notebook_task": {"notebook_path": "/Repos/data-eng/bookshop/99_alert"}
}

Run 4483 of bookshop-nightly

  ingest          SUCCESS   1m 44s   (rows_ingested = 0)
  check_freshness SUCCESS      2s    → false
  publish         SKIPPED
  alert_no_data   SUCCESS     18s

Branching without a wrapper notebook full of if statements. run_if on a task gives the other half — running something because an upstream failed:

{
  "task_key": "cleanup",
  "depends_on": [{"task_key": "transform"}],
  "run_if": "AT_LEAST_ONE_FAILED"
}

Values are ALL_SUCCESS (default), AT_LEAST_ONE_SUCCESS, NONE_FAILED, ALL_DONE, AT_LEAST_ONE_FAILED, ALL_FAILED. ALL_DONE is the one for a cleanup step that must run either way.

Looping over a parameter

{
  "task_key": "backfill",
  "for_each_task": {
    "inputs": "{{job.parameters.dates}}",
    "concurrency": 4,
    "task": {
      "task_key": "backfill_iteration",
      "notebook_task": {
        "notebook_path": "/Repos/data-eng/bookshop/01_ingest",
        "base_parameters": {"run_date": "{{input}}"}
      }
    }
  }
}

  backfill  SUCCESS  12m 41s
    ├── backfill_iteration (2026-01-01)  SUCCESS  3m 02s
    ├── backfill_iteration (2026-01-02)  SUCCESS  3m 11s
    ├── backfill_iteration (2026-01-03)  SUCCESS  2m 58s
    └── backfill_iteration (2026-01-04)  SUCCESS  3m 04s

Four dates, four concurrent iterations, one task definition. This replaces the loop-inside-a- notebook pattern, and each iteration retries independently.

Retries and idempotence

"max_retries": 2,
"min_retry_interval_millis": 60000,
"timeout_seconds": 3600

Run 4484

  ingest  FAILED    (attempt 1)  ConnectException: timed out reading from S3
  ingest  FAILED    (attempt 2)  ConnectException: timed out reading from S3
  ingest  SUCCESS   (attempt 3)  2m 18s
  transform SUCCESS 4m 06s

Run completed in 9m 44s

Retries only help if the task is safe to run twice. COPY INTO, MERGE and Auto Loader with a checkpoint all are; a bare INSERT INTO ... SELECT is not, and will duplicate on every retry. Establish idempotence first, then turn on retries — in that order.

Always set timeout_seconds. A task that hangs holds its cluster until someone notices, which is an expensive way to find out.

The job as code

Clicking a job together in the UI leaves no review trail and no way to promote it between environments. Asset bundles fix both:

# databricks.yml
bundle:
  name: bookshop

resources:
  jobs:
    nightly:
      name: bookshop-nightly-${bundle.target}
      schedule:
        quartz_cron_expression: "0 0 5 * * ?"
        timezone_id: Europe/London
      job_clusters:
        - job_cluster_key: shared
          new_cluster:
            spark_version: 16.4.x-scala2.13
            node_type_id: m6gd.xlarge
            autoscale: {min_workers: 2, max_workers: 8}
      tasks:
        - task_key: ingest
          job_cluster_key: shared
          notebook_task:
            notebook_path: ./notebooks/01_ingest.py
          max_retries: 2
        - task_key: transform
          depends_on: [{task_key: ingest}]
          pipeline_task:
            pipeline_id: ${resources.pipelines.medallion.id}

targets:
  dev:
    default: true
    mode: development
    variables: {catalog: bookshop_dev}
  prod:
    mode: production
    variables: {catalog: bookshop_prod}
    run_as: {service_principal_name: etl-svc}

databricks bundle validate -t prod
databricks bundle deploy -t prod

Name: bookshop
Target: prod
Workspace:
  Host: https://dbc-a1b2c3d4-e5f6.cloud.databricks.com
  User: etl-svc

Validation OK!

Uploading bundle files to /Workspace/Users/etl-svc/.bundle/bookshop/prod/files...
Deploying resources...
Updating deployment state...
Deployment complete!

databricks bundle run nightly -t prod

Run URL: https://dbc-a1b2c3d4-e5f6.cloud.databricks.com/#job/882041/run/4485

2026-09-09 05:00:12 "bookshop-nightly-prod" RUNNING
2026-09-09 05:07:24 "bookshop-nightly-prod" TERMINATED SUCCESS

mode: development prefixes resources with your username and pauses schedules, so deploying to dev cannot start a job that competes with production. That single behaviour is the reason to adopt bundles even for one job.

Monitoring

select
    j.name,
    r.result_state,
    count(*)                                                      as runs,
    round(avg(r.period_end_time - r.period_start_time) / 60e3, 1) as avg_minutes
from system.lakeflow.job_run_timeline r
join system.lakeflow.jobs j using (job_id)
where r.period_start_time >= current_timestamp() - interval 7 days
group by all
order by runs desc
limit 5;

name                    result_state  runs  avg_minutes
----------------------  ------------  ----  -----------
bookshop-nightly-prod   SUCCEEDED       26          7.4
bookshop-nightly-prod   FAILED           2         12.1
hourly-refresh          SUCCEEDED      168          2.1

Two failures out of 28, and the failed runs take longer than the successful ones — usually retries burning time before giving up. Charting avg_minutes per job over weeks is how you notice a pipeline that has quietly doubled in cost.

Practice

1. Pass a row count between two tasks.

dbutils.jobs.taskValues.set(key="rows", value=13932)
# downstream
dbutils.jobs.taskValues.get(taskKey="ingest", key="rows", default=0)

Always pass default — without it, a run where the upstream task was skipped raises instead of degrading gracefully, and the failure message points at the wrong task.

2. Add a condition task that skips publishing when there is no data.

  check_freshness SUCCESS  → false
  publish         SKIPPED
  alert_no_data   SUCCESS

SKIPPED, not FAILED — so the run is green and the alert still fires. An empty source is usually a normal condition, and treating it as a failure trains people to ignore alerts.

3. Force a task failure and watch retries.

  ingest  FAILED   (attempt 1)
  ingest  FAILED   (attempt 2)
  ingest  SUCCESS  (attempt 3)

Then make the task a plain INSERT INTO ... SELECT and do the same: the table now holds three copies of the batch. Retries amplify non-idempotence rather than causing it.

4. Deploy the same bundle to dev and prod.

databricks bundle deploy -t dev
databricks bundle deploy -t prod

Deployment complete!   # job "[dev sam] bookshop-nightly-dev", schedule PAUSED
Deployment complete!   # job "bookshop-nightly-prod",          schedule ACTIVE

The dev copy is name-prefixed and paused. One definition, two environments, and the difference is a CLI flag rather than a checklist someone follows by hand.

Next: performance — file sizes, liquid clustering, and what OPTIMIZE actually does.