MLOps Data Versioning with DVC

Real-World Scenario

A team’s best churn model came from “the March data”, but the CSV has been overwritten twice since and nobody can reproduce the result. The code is in Git; the data isn’t, because committing a few gigabytes of CSV to Git makes every clone slow. DVC fills the gap: Git versions a tiny pointer file with the data’s hash, and DVC stores the data itself in a cache and a remote.

Every example below runs locally with DVC 3, pandas and scikit-learn, and uses a plain directory, /tmp/dvc-storage, as the DVC remote, so no cloud account is needed. Install with pip install dvc pandas scikit-learn pyyaml joblib.

A Git Repo With a Local DVC Remote

A DVC project is a Git repository with a .dvc/ directory added. The remote is where data goes on dvc push, like a Git remote for code.

git init -q -b main ml-project
cd ml-project
dvc init
dvc remote add -d myremote /tmp/dvc-storage
cat .dvc/config

$ dvc init
Initialized DVC repository.

You can now commit the changes to git.

+---------------------------------------------------------------------+
|                                                                     |
|        DVC has enabled anonymous aggregate usage analytics.         |
|     Read the analytics documentation (and how to opt-out) here:     |
|             <https://dvc.org/doc/user-guide/analytics>              |
|                                                                     |
+---------------------------------------------------------------------+

What's next?
------------
- Check out the documentation: <https://dvc.org/doc>
- Get help and share ideas: <https://dvc.org/chat>
- Star us on GitHub: <https://github.com/treeverse/dvc>

$ dvc remote add -d myremote /tmp/dvc-storage
Setting 'myremote' as a default remote.

$ cat .dvc/config
[core]
    remote = myremote
['remote "myremote"']
    url = /tmp/dvc-storage

-d makes myremote the default, so dvc push and dvc pull need no arguments. The remote’s URL lives in .dvc/config, which is committed to Git, so everyone who clones the repo gets it. Commit the setup with git add .dvc .dvcignore && git commit -m "Initialize DVC with a local remote". Swapping to a cloud bucket later is one line, dvc remote add -d myremote s3://bucket/path, plus the matching plugin such as dvc-s3; nothing else in this lesson changes.

Tracking a Dataset With dvc add

This script writes a synthetic churn dataset. The row count is also the random seed, so the same size always produces the same file. Save it as make_data.py.

import sys
from pathlib import Path
import numpy as np
import pandas as pd

rows = int(sys.argv[1]) if len(sys.argv) > 1 else 1_000
rng = np.random.default_rng(rows)                     # same size -> same file
df = pd.DataFrame({
    "tenure": rng.integers(1, 72, rows),
    "monthly": rng.uniform(20, 120, rows).round(2),
    "support_calls": rng.poisson(1.5, rows),
})
score = 0.04 * df.monthly - 0.05 * df.tenure + 0.6 * df.support_calls
df["churned"] = (score + rng.normal(0, 2, rows) > 2.5).astype(int)
Path("data").mkdir(exist_ok=True)
df.to_csv("data/customers.csv", index=False)
print(f"wrote {rows} rows, churn rate {df.churned.mean():.1%}")

$ python make_data.py
wrote 1000 rows, churn rate 38.6%

$ dvc add data/customers.csv

To track the changes with git, run:

	git add data/customers.csv.dvc data/.gitignore

To enable auto staging, run:

	dvc config core.autostage true

$ cat data/customers.csv.dvc
outs:
- md5: efca55b0d4576c2fe23dab9a475ab5d6
  size: 13006
  hash: md5
  path: customers.csv

$ cat data/.gitignore
/customers.csv

dvc add did three things. It copied the CSV into DVC’s cache, .dvc/cache, under its MD5 hash. It wrote customers.csv.dvc, a five-line pointer holding that hash and the size in bytes. And it added the CSV itself to data/.gitignore, so Git can’t commit it by accident. Git versions the pointer; the data never enters Git. Commit and tag this version so you can come back to it:

git add make_data.py data/customers.csv.dvc data/.gitignore
git commit -qm "Dataset v1: 1000 customers"
git tag data-v1

Pushing to the Remote

dvc push copies whatever the committed pointers refer to from the local cache to the remote.

$ dvc push
1 file pushed

$ find /tmp/dvc-storage -type f
/tmp/dvc-storage/files/md5/ef/ca55b0d4576c2fe23dab9a475ab5d6

$ du -sh .git
260K	.git

The remote is content-addressed: the file is stored under its own hash, efca55b0…, split into a two-character directory and the rest, exactly the hash in the .dvc pointer. Two identical files are stored once, and a new version is a new hash beside the old one, so nothing is overwritten. Meanwhile .git is 260 KB. It grows a little with each commit, but not with the size of the dataset, because Git only ever sees the pointer.

The Failure: A Fresh Clone Has No Data

A teammate clones the repository. Git gives them everything it versions, which is the pointer, not the data.

$ git clone -q ml-project teammate

$ ls data
customers.csv.dvc

$ head -3 data/customers.csv
head: cannot open 'data/customers.csv' for reading: No such file or directory

$ dvc pull
A       data/customers.csv
1 file fetched and 1 file added

$ head -3 data/customers.csv
tenure,monthly,support_calls,churned
15,102.7,1,0
38,92.56,2,1

The clone has customers.csv.dvc and nothing else in data/, so any training script fails with a missing file. dvc pull reads the pointer, fetches that hash from the remote in .dvc/config, and places the file in the workspace. The A means added. git clone followed by dvc pull is the whole reproduction recipe: the teammate now has byte-for-byte the dataset the pointer names.

A Pipeline in dvc.yaml

Tracking one file is half of it. The other half is recording how data becomes a model, so DVC knows what to re-run when something changes. Hyperparameters go in params.yaml:

prepare:
  test_size: 0.25
train:
  C: 1.0
  max_iter: 200

prepare.py splits the data using test_size:

import pandas as pd
import yaml
from pathlib import Path
from sklearn.model_selection import train_test_split

params = yaml.safe_load(open("params.yaml"))["prepare"]
df = pd.read_csv("data/customers.csv")
train, test = train_test_split(df, test_size=params["test_size"],
                               random_state=0, stratify=df.churned)
Path("data/prepared").mkdir(exist_ok=True)
train.to_csv("data/prepared/train.csv", index=False)
test.to_csv("data/prepared/test.csv", index=False)
print(f"train {len(train)} rows, test {len(test)} rows")

train.py fits a logistic regression with C and max_iter, and writes the model and a metrics file:

import json
import joblib
import pandas as pd
import yaml
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

params = yaml.safe_load(open("params.yaml"))["train"]
train = pd.read_csv("data/prepared/train.csv")
test = pd.read_csv("data/prepared/test.csv")
X, y = train.drop(columns="churned"), train.churned
model = LogisticRegression(C=params["C"], max_iter=params["max_iter"]).fit(X, y)

auc = roc_auc_score(test.churned, model.predict_proba(test.drop(columns="churned"))[:, 1])
joblib.dump(model, "model.pkl")
json.dump({"auc": round(auc, 4), "test_rows": len(test)}, open("metrics.json", "w"))
print(f"test AUC {auc:.4f}")

dvc.yaml ties them together. Each stage names its command, what it reads (deps), which parameters it uses, and what it writes (outs):

stages:
  prepare:
    cmd: python prepare.py
    deps: [prepare.py, data/customers.csv]
    params: [prepare.test_size]
    outs: [data/prepared]
  train:
    cmd: python train.py
    deps: [train.py, data/prepared]
    params: [train.C, train.max_iter]
    outs: [model.pkl]
    metrics:
      - metrics.json:
          cache: false

dvc repro runs the pipeline. Run it twice:

$ dvc repro
'data/customers.csv.dvc' didn't change, skipping
Running stage 'prepare':
> python prepare.py
train 750 rows, test 250 rows
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'

Running stage 'train':
> python train.py
test AUC 0.8376
Updating lock file 'dvc.lock'

To track the changes with git, run:

	git add .gitignore data/.gitignore dvc.lock

To enable auto staging, run:

	dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.

$ dvc repro
'data/customers.csv.dvc' didn't change, skipping
Stage 'prepare' didn't change, skipping
Stage 'train' didn't change, skipping
Data and pipelines are up to date.

$ cat metrics.json
{"auc": 0.8376, "test_rows": 250}

The first run executes both stages in dependency order, because train reads data/prepared, which prepare writes. It also creates dvc.lock, which records the hash of every dependency and output and the value of every parameter used. The second run finds nothing changed and skips everything. That is the difference from a plain script: DVC compares hashes, not timestamps, so it knows when there’s no work to do. model.pkl and data/prepared are cached and git-ignored like the dataset; metrics.json has cache: false, so Git keeps it and it can be diffed. Commit with git add . && git commit -qm "Pipeline: prepare + train, C=1.0".

Changing One Parameter Re-runs One Stage

Now try stronger regularization: set C: 0.001 in params.yaml. Before running anything, ask DVC what’s stale.

$ dvc status
train:
	changed deps:
		params.yaml:
			modified:           train.C

$ dvc repro
'data/customers.csv.dvc' didn't change, skipping
Stage 'prepare' didn't change, skipping
Running stage 'train':
> python train.py
test AUC 0.8108
Updating lock file 'dvc.lock'

To track the changes with git, run:

	git add dvc.lock

To enable auto staging, run:

	dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.

$ dvc params diff
Path         Param    HEAD    workspace
params.yaml  train.C  1.0     0.001

$ dvc metrics diff
Path          Metric    HEAD    workspace    Change
metrics.json  auc       0.8376  0.8108       -0.0268

DVC tracks parameters individually, not the whole params.yaml file, so a change to train.C makes only train stale; prepare doesn’t use it and is skipped. On a real pipeline, where preparation might take an hour, that skip is the point. dvc params diff and dvc metrics diff compare the workspace with the last commit: the change cost 0.0268 AUC, which is the whole experiment summarized in two tables. Commit it anyway as a recorded negative result: git add . && git commit -qm "Stronger regularization: C=0.001".

A New Dataset Version

Regenerate the data with 1,500 rows, standing in for next month’s export, and re-run.

$ python make_data.py 1500
wrote 1500 rows, churn rate 40.9%

$ dvc status
prepare:
	changed deps:
		modified:           data/customers.csv
data/customers.csv.dvc:
	changed outs:
		modified:           data/customers.csv

$ dvc repro
Verifying data sources in stage: 'data/customers.csv.dvc'

Running stage 'prepare':
> python prepare.py
train 1125 rows, test 375 rows
Updating lock file 'dvc.lock'

Running stage 'train':
> python train.py
test AUC 0.8215
Updating lock file 'dvc.lock'

To track the changes with git, run:

	git add dvc.lock data/customers.csv.dvc

To enable auto staging, run:

	dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.

$ dvc push
5 files pushed

This time the data changed, so both stages are stale and both run. You don’t need to run dvc add again: dvc repro re-hashed the tracked CSV and updated its pointer, which is why the hint lists data/customers.csv.dvc. dvc push sent 5 files: the new CSV, the model, the two split files, and a small .dir manifest DVC keeps for the data/prepared directory output. The 1,000-row version is still in the remote under its own hash. Commit with git add . && git commit -qm "Dataset v2: 1500 customers".

Going Back: git checkout Is Not Enough

Check out the data-v1 tag to get last month’s dataset back.

$ git checkout -q data-v1

$ wc -l data/customers.csv
1501 data/customers.csv

$ dvc status
data/customers.csv.dvc:
	changed outs:
		modified:           data/customers.csv

$ dvc checkout
M       data/customers.csv
D       data/prepared/
D       model.pkl

$ wc -l data/customers.csv
1001 data/customers.csv

$ git checkout -q main

$ dvc checkout
M       data/customers.csv
A       data/prepared/
A       model.pkl

After git checkout, the file still has 1,501 lines (1,500 rows plus a header): Git restored the v1 pointer, but DVC files are not Git files, so the data in the workspace is still v2. dvc status catches the mismatch. dvc checkout makes the workspace match the pointers: it restores the 1,000-row CSV from the cache (M, modified) and removes the model and split files (D), because at the data-v1 commit the pipeline didn’t exist yet. Returning to main and running dvc checkout again restores all three. Always run the two commands together; a Git hook or git checkout ... && dvc checkout saves forgetting.

Practice

1. Compare the current workspace (right after the dataset v2 commit) with the first pipeline commit, two commits back, using both diff commands.

   Solution

   $ git log --oneline
   d953d38 Dataset v2: 1500 customers
   ba01e43 Stronger regularization: C=0.001
   6057dff Pipeline: prepare + train, C=1.0
   9c1b279 Dataset v1: 1000 customers
   f3b22c2 Initialize DVC with a local remote
   
   $ dvc params diff HEAD~2
   Path         Param    HEAD~2    workspace
   params.yaml  train.C  1.0       0.001
   
   $ dvc metrics diff HEAD~2
   Path          Metric     HEAD~2    workspace    Change
   metrics.json  auc        0.8376    0.8215       -0.0161
   metrics.json  test_rows  250       375          125

   Both commands take any Git revision: a relative one like HEAD~2, a hash or a tag. The commit hashes will differ on your machine. Note what the diff can’t tell you on its own: AUC fell 0.0161, but two things changed since HEAD~2, the value of C and the dataset, as test_rows going from 250 to 375 shows. Change one thing per commit if you want each diff to have one cause.

2. Simulate losing your laptop’s cache: delete .dvc/cache and the dataset, then try dvc checkout. What fails, and how do you recover?

   Solution

   $ rm -rf .dvc/cache data/customers.csv
   
   $ dvc checkout
   ERROR: Checkout failed for following targets:
   data/customers.csv
   Is your cache up to date?
   <https://error.dvc.org/missing-files>
   
   $ dvc pull
   A       data/customers.csv
   5 files fetched and 1 file added
   
   $ wc -l data/customers.csv
   1501 data/customers.csv

   dvc checkout only copies from the local cache, so with the cache gone it has nothing to restore from. dvc pull is dvc fetch (remote to cache) plus dvc checkout (cache to workspace), so it recovers everything that was pushed: here all 5 files from the last push. Anything you never pushed, like the C=0.001 model, would be gone for good, which is why dvc push belongs next to every git push.

3. Change prepare.test_size to 0.2. Predict which stages dvc repro will run, then check.

   Solution

   $ dvc status
   prepare:
   	changed deps:
   		params.yaml:
   			modified:           prepare.test_size
   
   $ dvc repro
   'data/customers.csv.dvc' didn't change, skipping
   Running stage 'prepare':
   > python prepare.py
   train 1200 rows, test 300 rows
   Updating lock file 'dvc.lock'
   
   Running stage 'train':
   > python train.py
   test AUC 0.8175
   Updating lock file 'dvc.lock'
   
   To track the changes with git, run:
   
   	git add dvc.lock
   
   To enable auto staging, run:
   
   	dvc config core.autostage true
   Use `dvc push` to send your updates to remote storage.

   Both run. dvc status names only prepare as having a changed parameter, but prepare produces new split files, and those are a dependency of train, so train re-runs too. A change propagates downstream through outputs, never upstream.