MLOps Experiment Tracking

Real-World Scenario

A team tunes a churn model for two weeks. By the end there are fifty notebook cells, three spreadsheets and a Slack thread claiming “the depth-8 one was best”. Nobody can say which data it saw or reproduce its score. An experiment tracker records every run as it happens: parameters, metrics, the model file, and tags. It then lets you query runs like a database and promote a winner by name.

Every example here runs with MLflow 3 (tested with 3.16), scikit-learn and pandas, against MLflow’s default local store in the current directory, with no server and no account: pip install mlflow scikit-learn pandas. Run the scripts in one directory, in order, because later ones read the runs earlier ones logged.

A First Run and Where It Goes

A run is one execution of your training code. Everything logged inside with mlflow.start_run() belongs to that run. Save this as first_run.py.

import mlflow

print("tracking URI:", mlflow.get_tracking_uri())
mlflow.set_experiment("churn")

with mlflow.start_run(run_name="hello") as run:
    mlflow.log_param("model", "logistic")
    mlflow.log_params({"C": 1.0, "max_iter": 200})
    mlflow.log_metric("auc", 0.84)
    mlflow.set_tag("dataset", "customers-v1")

print("run id:", run.info.run_id)
print("status:", mlflow.get_run(run.info.run_id).info.status)
print("params:", mlflow.get_run(run.info.run_id).data.params)

$ python first_run.py
tracking URI: sqlite:////tmp/mlops-demo/mlflow.db
2026/09/29 18:41:11 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/09/29 18:41:11 INFO mlflow.store.db.utils: Updating database tables
2026/09/29 18:41:12 INFO mlflow.tracking.fluent: Experiment with name 'churn' does not exist. Creating a new experiment.
run id: 783722f8e8d84171bbd928f63b201770
status: FINISHED
params: {'model': 'logistic', 'C': '1.0', 'max_iter': '200'}
$ ls
mlflow.db

With no tracking URI set, MLflow 3 creates a SQLite database, mlflow.db, in the current directory. Older tutorials, and older MLflow versions, say ./mlruns; the INFO lines show the database being created on first use. set_experiment created the churn experiment because it didn’t exist. Leaving the with block ended the run as FINISHED. If the script had crashed inside it, the run would be marked FAILED. Note what came back from params: '1.0' and '200' are strings. Parameters are stored as text, so compare them as text when you query later. To browse runs in a browser, run mlflow ui in the same directory.

Logging a Model: The Untrusted-Types Failure

Parameters and metrics are small. The model file is the artifact that matters. The natural first attempt with a random forest fails. Save this as untrusted.py.

import mlflow
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

X, y = make_classification(n_samples=200, random_state=0)
model = RandomForestClassifier(n_estimators=10, random_state=0).fit(X, y)
mlflow.set_experiment("churn")

with mlflow.start_run(run_name="untrusted"):
    try:
        mlflow.sklearn.log_model(model, name="model")
    except mlflow.exceptions.MlflowException as e:
        print(type(e).__name__ + ":", str(e).split(" Root error")[0])
    info = mlflow.sklearn.log_model(model, name="model",
                                    skops_trusted_types=["sklearn.tree._tree.Tree"])
    print("logged:", info.model_uri, "| format:",
          info.flavors["sklearn"]["serialization_format"])

$ python untrusted.py
MlflowException: The saved sklearn model references untrusted types. If you are sure loading these types is safe, set the 'skops_trusted_types' parameter when calling 'log_model' or 'save_model' to the list of trusted types.
logged: models:/m-a06ed35006a347ee8545758c4510500f | format: skops

MLflow now saves scikit-learn models with skops instead of pickle, because unpickling a file can run arbitrary code. skops only loads types it has vetted, and every tree-based model, including random forests and gradient boosting, contains sklearn.tree._tree.Tree, which isn’t on that list. MLflow test-loads what it saved, so log_model fails. The full error names the type. You trained this model yourself, so trusting it is safe: pass the type in skops_trusted_types and the second call succeeds. The URI it returns is also new in MLflow 3: models:/m-… is the model’s own ID, a first-class object rather than a folder inside the run. Logging it also created a mlruns/ directory beside mlflow.db. The database holds the metadata, and mlruns/<experiment id>/models/m-…/artifacts/ holds each model’s files: model.skops, an MLmodel descriptor, requirements.txt and more.

Comparing Three Configurations

Now the real experiment: three random forests of different depths, each in its own run, with test and training AUC. Save this as compare.py.

import mlflow
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=4000, n_features=20, n_informative=6,
                           flip_y=0.1, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
mlflow.set_experiment("churn")

for name, depth, trees in [("shallow", 3, 100), ("medium", 8, 100), ("unlimited", None, 100)]:
    with mlflow.start_run(run_name=name):
        mlflow.log_params({"max_depth": depth, "n_estimators": trees})
        model = RandomForestClassifier(max_depth=depth, n_estimators=trees,
                                       random_state=0).fit(X_train, y_train)
        auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
        mlflow.log_metrics({"auc": auc, "train_auc": roc_auc_score(
            y_train, model.predict_proba(X_train)[:, 1])})
        mlflow.sklearn.log_model(model, name="model", input_example=X_test[:2],
                                 skops_trusted_types=["sklearn.tree._tree.Tree"])
        print(f"{name:10} test AUC {auc:.4f}")

$ python compare.py
shallow    test AUC 0.9002
medium     test AUC 0.9258
unlimited  test AUC 0.9287

Each loop iteration is a separate run, so each configuration keeps its own parameters, metrics and model. input_example stores two sample rows with the model, which MLflow uses to infer the input schema; that becomes useful when the model is served. Each of the three models gets its own m-… directory under mlruns/, with the input example stored beside it.

Querying Runs Like a Table

Printing results as you go doesn’t scale to fifty runs. search_runs returns every run in an experiment as a pandas DataFrame, filtered and sorted. Save this as search.py.

import mlflow

runs = mlflow.search_runs(experiment_names=["churn"],
                          filter_string="metrics.auc > 0.9",
                          order_by=["metrics.auc DESC"])
cols = ["tags.mlflow.runName", "params.max_depth", "params.n_estimators",
        "metrics.train_auc", "metrics.auc"]
table = runs[cols].rename(columns=lambda c: c.split(".")[-1])
print(table.round(4).to_string(index=False))
print(f"\n{len(runs)} of {len(mlflow.search_runs(experiment_names=['churn']))} runs matched")

$ python search.py
  runName max_depth n_estimators  train_auc    auc
unlimited      None          100     1.0000 0.9287
   medium         8          100     0.9868 0.9258
  shallow         3          100     0.9249 0.9002

3 of 5 runs matched

Columns are named params.*, metrics.* and tags.*; the run name is the system tag mlflow.runName. The filter dropped the hello run, with an AUC of 0.84, and the untrusted run, which has no AUC metric at all. That makes 3 of 5. The table shows something the printed AUCs didn’t. unlimited wins on test AUC, but its training AUC is a perfect 1.0000: it has memorized the training set, and the gap to test (0.0713) is the largest of the three. That’s the argument for logging training metrics too. A 0.003 AUC lead may not be worth the most overfit model, and without train_auc you couldn’t have seen it.

Metrics Over Time With step

A metric logged repeatedly with a step becomes a curve, not a single number. This logs validation loss for each of 30 training epochs. Save it as epochs.py.

import mlflow
from mlflow import MlflowClient
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=2000, n_features=20, flip_y=0.1, random_state=1)
X_train, X_val, y_train, y_val = X[:1500], X[1500:], y[:1500], y[1500:]
model = SGDClassifier(loss="log_loss", learning_rate="constant", eta0=0.0005, random_state=0)
mlflow.set_experiment("churn")

with mlflow.start_run(run_name="sgd-epochs") as run:
    for epoch in range(1, 31):
        model.partial_fit(X_train, y_train, classes=[0, 1])
        mlflow.log_metric("val_loss", log_loss(y_val, model.predict_proba(X_val)), step=epoch)

history = MlflowClient().get_metric_history(run.info.run_id, "val_loss")
for m in history[:3] + history[9::10]:
    print(f"epoch {m.step:2}  val_loss {m.value:.4f}")
best = min(history, key=lambda m: m.value)
print(f"best  epoch {best.step:2}  val_loss {best.value:.4f}")

$ python epochs.py
epoch  1  val_loss 0.5683
epoch  2  val_loss 0.4977
epoch  3  val_loss 0.4547
epoch 10  val_loss 0.3632
epoch 20  val_loss 0.3429
epoch 30  val_loss 0.3386
best  epoch 30  val_loss 0.3386

The run’s summary metric, run.data.metrics["val_loss"], holds only the last value. The whole curve is in get_metric_history, one entry per step, and mlflow ui plots it. This curve falls fast for three epochs, then flattens: 0.3429 at epoch 20 and 0.3386 at 30. The best epoch is the last, so training for longer might still help a little. Had the minimum come at epoch 12, the history would tell you to stop early, and to save the model from that epoch rather than the last.

The Failure: Parameters Can’t Be Changed

A tempting shortcut is to reuse a run and “just update” a parameter when you retry. Save this as param_change.py.

import mlflow

mlflow.set_experiment("churn")
with mlflow.start_run(run_name="retry") as run:
    mlflow.log_param("C", 1.0)
    try:
        mlflow.log_param("C", 0.1)                 # "just update it"
    except mlflow.exceptions.MlflowException as e:
        print(type(e).__name__ + ":", str(e).split(" for run ID")[0])
    mlflow.log_param("C", 1.0)                     # same value again: allowed
    print("params:", mlflow.get_run(run.info.run_id).data.params)
print("status:", mlflow.get_run(run.info.run_id).info.status)

$ python param_change.py
MlflowException: Changing param values is not allowed. Param with key='C' was already logged with value='1.0'
params: {'C': '1.0'}
status: FINISHED

Parameters are write-once per run. A run is a record of what produced its metrics, and if C could change after the metrics were logged, the record would lie. Re-logging the same value is fine, which makes retries of the same code safe. For a new value, start a new run. Metrics are different: they can be logged many times, which is what step relies on.

Registering the Best Model Under an Alias

Runs record everything; the registry names the few models that matter. This registers the best run’s model as churn-model and points the alias champion at it. Save it as register.py.

import mlflow
from mlflow import MlflowClient

best = mlflow.search_runs(experiment_names=["churn"], order_by=["metrics.auc DESC"],
                          max_results=1, output_format="list")[0]
model_id = best.outputs.model_outputs[0].model_id          # MLflow 3: models have their own id
version = mlflow.register_model(f"models:/{model_id}", "churn-model")

client = MlflowClient()
client.set_registered_model_alias("churn-model", "champion", version.version)
champion = client.get_model_version_by_alias("churn-model", "champion")
print(f"champion: version {champion.version} from run {best.info.run_name!r}, "
      f"auc {best.data.metrics['auc']:.4f}")

model = mlflow.sklearn.load_model("models:/churn-model@champion")
print("loaded:", type(model).__name__, "max_depth =", model.max_depth)

$ python register.py
Successfully registered model 'churn-model'.
Created version '1' of model 'churn-model'.
champion: version 1 from run 'unlimited', auc 0.9287
loaded: RandomForestClassifier max_depth = None

Registering creates version 1 of churn-model. Every later registration under the same name adds version 2, 3 and so on. The alias champion is a movable pointer to one version, and models:/churn-model@champion loads whatever it currently points to, so serving code never hard-codes a version number. The loaded model has max_depth = None: it’s the unlimited forest. Older tutorials promote models through stages such as Staging and Production with transition_model_version_stage; MLflow has deprecated stages in favour of aliases like this one.

Practice

1. Use search_runs to list only the runs whose training AUC is above 0.98, with the gap between training and test AUC, largest gap first.

   Solution

   import mlflow
   
   runs = mlflow.search_runs(experiment_names=["churn"],
                             filter_string="metrics.train_auc > 0.98")
   runs["gap"] = runs["metrics.train_auc"] - runs["metrics.auc"]
   for _, r in runs.sort_values("gap", ascending=False).iterrows():
       print(f"{r['tags.mlflow.runName']:10} train {r['metrics.train_auc']:.4f}  "
             f"test {r['metrics.auc']:.4f}  gap {r['gap']:.4f}")

   $ python prac_overfit.py
   unlimited  train 1.0000  test 0.9287  gap 0.0713
   medium     train 0.9868  test 0.9258  gap 0.0610

   filter_string runs inside the tracking store, but it can only compare a column with a constant, so the gap is computed afterwards in pandas. shallow is filtered out because its training AUC is 0.9249. Both remaining forests overfit by 6 to 7 AUC points, which would argue for a depth between 3 and 8 in the next round.

2. Register the shallow run’s model as a new version, give it the alias challenger, load both aliases, then promote the challenger by moving champion.

   Solution

   import mlflow
   from mlflow import MlflowClient
   
   client = MlflowClient()
   shallow = mlflow.search_runs(experiment_names=["churn"], output_format="list",
                                filter_string="attributes.run_name = 'shallow'")[0]
   v = mlflow.register_model(f"models:/{shallow.outputs.model_outputs[0].model_id}", "churn-model")
   client.set_registered_model_alias("churn-model", "challenger", v.version)
   
   for alias in ("champion", "challenger"):
       mv = client.get_model_version_by_alias("churn-model", alias)
       model = mlflow.sklearn.load_model(f"models:/churn-model@{alias}")
       print(f"@{alias:10} -> version {mv.version}, max_depth={model.max_depth}")
   
   client.set_registered_model_alias("churn-model", "champion", v.version)   # promote
   print("after promotion, @champion ->",
         client.get_model_version_by_alias("churn-model", "champion").version)

   $ python prac_challenger.py
   Registered model 'churn-model' already exists. Creating a new version of this model...
   Created version '2' of model 'churn-model'.
   @champion   -> version 1, max_depth=None
   @challenger -> version 2, max_depth=3
   after promotion, @champion -> 2

   A model version can carry several aliases and an alias names exactly one version, so setting champion to version 2 moves it off version 1 in a single call. Anything loading models:/churn-model@champion picks up the shallower model on its next load. Rolling back is the same call with the old version number.

3. Read the sgd-epochs run back from the tracking store and find the first epoch where validation loss dropped below 0.35, and how much the last 10 epochs improved it.

   Solution

   import mlflow
   from mlflow import MlflowClient
   
   run = mlflow.search_runs(experiment_names=["churn"], output_format="list",
                            filter_string="attributes.run_name = 'sgd-epochs'")[0]
   history = MlflowClient().get_metric_history(run.info.run_id, "val_loss")
   first = next(m for m in history if m.value < 0.35)
   print(f"val_loss first below 0.35 at epoch {first.step} ({first.value:.4f})")
   print(f"the last 10 epochs improved it by {history[-11].value - history[-1].value:.4f}")

   $ python prac_threshold.py
   val_loss first below 0.35 at epoch 15 (0.3489)
   the last 10 epochs improved it by 0.0043

   The run was found by name, in a new process, long after epochs.py exited, which is the point of a tracking store: the curve outlives the script. attributes.run_name filters on a run attribute rather than a tag. The loss crossed 0.35 halfway through training, and the last third bought only 0.0043 more. On a model that took hours per epoch, that is the evidence for stopping at around epoch 20 next time.