What is Slim CI in dbt?

Building only the models a pull request changed, plus their descendants, by comparing against the production `manifest.json` with `state:modified`. Combined with `--defer`, unchanged parents are read from production instead of rebuilt, so a one-model change costs one model's build time.

What does dbt --defer do?

It tells dbt to resolve any `ref` to a model that is not being built in this run against the deferred environment — usually production — rather than your development schema. That is what makes it possible to build one model in isolation without first building its parents.

Which dbt artifacts matter in production?

`manifest.json` describes the project as parsed and is the input to state comparison; `run_results.json` records timing and status per node and is what monitoring should read. Both land in `target/` and should be uploaded from every production run.

Should I run dbt from Airflow or a scheduler?

Either works. A single `dbt build` on a schedule is simplest and correct for most teams. Splitting models across Airflow tasks buys per-model retries and finer scheduling at the cost of duplicating dbt's DAG in another system — worth it only when other, non-dbt work must interleave.

Deployment and Slim CI | SkillByExample

Production dbt is the same commands with a different target and a stricter attitude to failure. The interesting part is CI: a project of 400 models takes 20 minutes to build, and a pull request that changes one model should not.

Environments are targets

# profiles.yml
bookshop:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: bookshop.duckdb
      schema: dev_alice
      threads: 4
    ci:
      type: duckdb
      path: ci.duckdb
      schema: "ci_pr_{{ env_var('PR_NUMBER', '0') }}"
      threads: 8
    prod:
      type: duckdb
      path: /warehouse/prod.duckdb
      schema: analytics
      threads: 16

dbt build --target prod

17:30:02  Running with dbt=1.9.1
17:30:02  Found 8 models, 2 seeds, 1 snapshot, 14 data tests, 1 source, 431 macros
17:30:02
17:30:02  Concurrency: 16 threads (target='prod')
17:30:02
17:30:02  1 of 25 START seed file analytics.raw_customers ............... [RUN]
...
17:30:09  Done. PASS=25 WARN=0 ERROR=0 SKIP=0 TOTAL=25

One codebase, three schemas, no branching in the SQL. Anything genuinely environment-specific goes through target:

{% if target.name == 'prod' %}
  where ordered_at >= '2020-01-01'
{% else %}
  where ordered_at >= current_date - 30    -- dev builds a month, not five years
{% endif %}

Limiting development builds to a recent window is the single biggest quality-of-life change on a large project.

The production run

dbt source freshness --target prod
dbt build --target prod

17:35:11  1 of 1 PASS freshness of raw.raw_orders ........................ [PASS in 0.06s]
17:35:11  Done.

17:35:12  Concurrency: 16 threads (target='prod')
17:35:12  1 of 25 START seed file analytics.raw_customers ............... [RUN]
17:35:12  3 of 25 START snapshot analytics_snapshots.customers_snapshot . [RUN]
17:35:13  8 of 25 START sql table model analytics.customer_orders ....... [RUN]
17:35:14  9 of 25 FAIL 1 relationships_stg_orders_customer_id__customer_id__ref_stg_customers_  [FAIL 1 in 0.09s]
17:35:14  14 of 25 SKIP sql table model analytics.daily_revenue ......... [SKIP]
17:35:15
17:35:15  Completed with 1 error and 0 warnings:
17:35:15
17:35:15  Done. PASS=22 WARN=0 ERROR=1 SKIP=2 TOTAL=25

build rather than run then test, so a failing test skips its descendants instead of publishing bad numbers. The exit code is non-zero, which is what the scheduler alerts on.

Pick up where it stopped rather than starting over:

dbt retry --target prod

17:41:30  Running with dbt=1.9.1
17:41:30  Found 8 models, 2 seeds, 1 snapshot, 14 data tests, 1 source, 431 macros
17:41:30
17:41:30  1 of 3 START test relationships_stg_orders_customer_id__customer_id__ref_stg_customers_  [RUN]
17:41:30  1 of 3 PASS relationships_stg_orders_customer_id__customer_id__ref_stg_customers_  [PASS in 0.04s]
17:41:30  2 of 3 START sql table model analytics.daily_revenue ........... [RUN]
17:41:30  2 of 3 OK created sql table model analytics.daily_revenue ...... [OK in 0.09s]
17:41:30
17:41:30  Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3

Three nodes instead of twenty-five. dbt retry reads run_results.json and re-runs only what failed or was skipped.

Artifacts

ls target/*.json

target/catalog.json
target/manifest.json
target/run_results.json
target/semantic_manifest.json

run_results.json is what monitoring should read — status and timing per node:

jq -r '.results[] | [.status, .execution_time, .unique_id] | @tsv' target/run_results.json | sort -k2 -rn | head -5

success	4.812	model.bookshop.customer_orders
success	2.043	model.bookshop.daily_revenue
success	0.518	snapshot.bookshop.customers_snapshot
fail	0.094	test.bookshop.relationships_stg_orders_customer_id
success	0.081	model.bookshop.stg_orders

Load that into a table after every run and you get build-time trends for free — which is how you find the model that quietly went from 40 seconds to 9 minutes.

manifest.json is the input to state comparison, and that is what makes CI fast.

Slim CI

Save production’s manifest somewhere CI can fetch it, then compare against it:

dbt build --select state:modified+ --defer --state ./prod-artifacts --target ci

17:52:08  Running with dbt=1.9.1
17:52:08  Found 8 models, 2 seeds, 1 snapshot, 14 data tests, 1 source, 431 macros
17:52:08
17:52:08  Concurrency: 8 threads (target='ci')
17:52:08
17:52:08  1 of 4 START sql view model ci_pr_412.stg_orders ............... [RUN]
17:52:08  1 of 4 OK created sql view model ci_pr_412.stg_orders .......... [OK in 0.05s]
17:52:08  2 of 4 START test not_null_stg_orders_order_id ................. [RUN]
17:52:08  2 of 4 PASS not_null_stg_orders_order_id ....................... [PASS in 0.03s]
17:52:08  3 of 4 START sql table model ci_pr_412.customer_orders ......... [RUN]
17:52:08  3 of 4 OK created sql table model ci_pr_412.customer_orders .... [OK in 0.09s]
17:52:08  4 of 4 START test unique_customer_orders_customer_id ........... [RUN]
17:52:08  4 of 4 PASS unique_customer_orders_customer_id ................. [PASS in 0.03s]
17:52:08
17:52:08  Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4

Four nodes out of twenty-five, because the PR touched stg_orders. The two pieces:

state:modified+ — the models whose code differs from the stored manifest, plus everything downstream of them.
--defer — stg_customers was not built in this run, so its ref resolved to analytics.stg_customers in production rather than an empty CI schema.

Without --defer, state:modified+ fails on the first unbuilt parent. The two flags are a pair.

Related selectors:

Selector	Selects
`state:modified`	changed code, config, or contract
`state:new`	nodes absent from the comparison manifest
`state:modified+`	those plus descendants — the usual CI selector
`result:error+`	nodes that errored in the previous run
`1+result:failed`	failed tests and their parents, for debugging

A CI workflow

# .github/workflows/dbt-ci.yml
name: dbt CI
on: pull_request

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      PR_NUMBER: ${{ github.event.pull_request.number }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }

      - run: pip install dbt-core dbt-duckdb
      - run: dbt deps

      # The manifest from the last successful production run.
      - name: Fetch production artifacts
        run: |
          mkdir -p prod-artifacts
          aws s3 cp s3://bookshop-dbt/prod/manifest.json prod-artifacts/manifest.json

      - name: Build changed models only
        run: dbt build --select state:modified+ --defer --state ./prod-artifacts --target ci

      - name: Drop the PR schema
        if: always()
        run: dbt run-operation drop_pr_schema --args "{pr: $PR_NUMBER}"

Run dbt build --select state:modified+ --defer --state ./prod-artifacts --target ci
17:58:44  Found 8 models, 2 seeds, 1 snapshot, 14 data tests, 1 source, 431 macros
17:58:44  Concurrency: 8 threads (target='ci')
17:58:45  Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4

if: always() on the cleanup step matters — without it, every failed CI run leaves a schema behind, and a year later someone is deleting four hundred of them by hand.

The run-operation command runs a macro without building anything:

-- macros/drop_pr_schema.sql
{% macro drop_pr_schema(pr) %}
  {% do run_query('drop schema if exists ci_pr_' ~ pr ~ ' cascade') %}
  {% do log('dropped schema ci_pr_' ~ pr, info=True) %}
{% endmacro %}

18:01:20  dropped schema ci_pr_412

Scheduling

The simplest production setup that works:

0 5 * * *  cd /opt/bookshop && dbt source freshness --target prod && dbt build --target prod

From Airflow, one task is usually right:

BashOperator(
    task_id="dbt_build",
    bash_command="cd /opt/bookshop && dbt build --target prod",
)

Splitting dbt across many Airflow tasks gives per-model retries and finer scheduling, at the cost of maintaining the same DAG in two systems. Do it when non-dbt work has to interleave — an export that must run between two marts — and not before.

Practice

1. Run a build, then change one model and use state:modified+.

cp -r target prod-artifacts
# edit models/staging/stg_orders.sql
dbt ls --select state:modified+ --state ./prod-artifacts

bookshop.staging.stg_orders
bookshop.marts.customer_orders
bookshop.marts.daily_revenue
bookshop.marts.country_revenue

Use dbt ls first to see what a selector resolves to. It costs nothing and it is much better than discovering the selector was wrong after CI builds the whole project.

2. Try state:modified+ without --defer on a clean schema.

18:08:33  1 of 2 START sql table model ci_pr_412.customer_orders ......... [RUN]
18:08:33  1 of 2 ERROR creating sql table model ci_pr_412.customer_orders  [ERROR in 0.04s]

18:08:33    Runtime Error in model customer_orders (models/marts/customer_orders.sql)
18:08:33      Catalog Error: Table with name stg_customers does not exist!

The unchanged parent was never built into the CI schema. --defer sends that ref to production instead, which is the whole reason the flag exists.

3. Read the slowest models out of run_results.json.

jq -r '.results[] | select(.status=="success") | [.execution_time, .unique_id] | @tsv' \
  target/run_results.json | sort -rn | head -3

4.812	model.bookshop.customer_orders
2.043	model.bookshop.daily_revenue
0.518	snapshot.bookshop.customers_snapshot

Persist this after every production run. Build-time regressions are gradual and invisible without a trend, and this file is the only place the data exists.

4. Fail a test in production, then use dbt retry.

# first run
18:15:02  Done. PASS=22 WARN=0 ERROR=1 SKIP=2 TOTAL=25
# after fixing the data
18:16:44  Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3

Three nodes instead of twenty-five. On a project where a full build is 40 minutes, dbt retry after a transient failure is the difference between a ten-minute recovery and re-running the night.

That closes the dbt track. The thread through all ten lessons: dbt only ever generates SQL and runs it in order — when something looks like magic, read target/compiled/ and it stops being magic.