TensorFlow Model Deployment

Real-World Scenario

A payments team has trained a fraud classifier. The checkout service needs predictions over HTTP with the ability to roll out a retrained model without downtime, and a mobile app wants the same model on the device for offline checks. This lesson takes one small Keras model through each of those routes: a SavedModel served by TF Serving, a FastAPI wrapper, and an int8 TFLite file run with LiteRT.

All examples ran with TensorFlow 2.21 and Keras 3.15 on a CPU-only machine, with TF Serving 2.20 in Docker, LiteRT (ai-edge-litert) 2.2, and FastAPI 0.141. TensorFlow’s startup log lines on stderr (no CUDA driver found, oneDNN enabled) are left out of the outputs below.

The Model to Deploy

A synthetic “fraud” label — a fixed rule over three of twenty features — keeps the model small and its predictions checkable. The script saves the model and three sample rows that every later example reuses.

import numpy as np
import keras

rng = np.random.default_rng(0)
X = rng.normal(size=(5000, 20)).astype("float32")
y = ((X[:, 0] + 0.5 * X[:, 1] - X[:, 2]) > 1.0).astype("float32")   # "fraud" rule

keras.utils.set_random_seed(0)
model = keras.Sequential([
    keras.Input(shape=(20,), name="features"),
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(32, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid", name="fraud_probability"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X, y, epochs=10, batch_size=64, validation_split=0.2, verbose=0)
print(f"fraud rate: {y.mean():.3f}")
print(f"val accuracy: {model.evaluate(X[4000:], y[4000:], verbose=0)[1]:.4f}")

model.save("fraud.keras")
np.save("sample.npy", X[:3])
print(model.predict(X[:3], verbose=0).round(4).ravel())

$ python train.py
fraud rate: 0.257
val accuracy: 0.9690
[0.     0.9799 0.    ]

validation_split=0.2 holds out the last 1,000 rows, so X[4000:] is exactly the validation set, and the model gets 96.9% of it right. The second sample row is fraud with probability 0.9799 and the other two are near zero. Every deployment route below should reproduce these three numbers — that is the check that nothing was lost on the way.

Exporting a SavedModel

A .keras file needs Keras and the model’s Python classes to load. TF Serving and the TFLite converter need a SavedModel: a self-contained graph plus weights with named signatures. In Keras 3 that is model.export.

import numpy as np
import tensorflow as tf
import keras

model = keras.models.load_model("fraud.keras")
model.export("export/fraud/1", verbose=False)        # SavedModel, version 1

loaded = tf.saved_model.load("export/fraud/1")
print(list(loaded.signatures.keys()))
serve = loaded.signatures["serving_default"]
print(serve.structured_input_signature)
print(serve.structured_outputs)

x = np.load("sample.npy")
result = serve(tf.constant(x))
print({name: t.numpy().round(4).ravel() for name, t in result.items()})

$ python export.py
['serve', 'serving_default']
((), {'features': TensorSpec(shape=(None, 20), dtype=tf.float32, name='features')})
{'output_0': TensorSpec(shape=(None, 1), dtype=tf.float32, name='output_0')}
{'output_0': array([0.    , 0.9799, 0.    ], dtype=float32)}

The signature takes a batch of any size with 20 float32 features — the input is called features because the keras.Input was given that name — and returns one probability per row under the key output_0. tf.saved_model.load restores it with TensorFlow alone, and the predictions match train.py. The trailing /1 is a version number: TF Serving expects each version in its own numbered subdirectory. Calling model.save("export/fraud/1") instead raises a ValueError in Keras 3.

The saved_model_cli tool that installs with TensorFlow inspects a SavedModel without writing any code:

saved_model_cli show --dir export/fraud/1 --tag_set serve --signature_def serving_default

$ saved_model_cli show --dir export/fraud/1 --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
  inputs['features'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 20)
      name: serving_default_features:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['output_0'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: StatefulPartitionedCall_1:0
Method name is: tensorflow/serving/predict

-1 is the SavedModel spelling of “any batch size”. The input and output names shown here are exactly the keys a TF Serving request and response use.

Serving with TF Serving

The tensorflow/serving image serves every version under /models/<MODEL_NAME>. Mount the export directory and start it:

docker run -d --name tfserving -p 8501:8501 \
  -v "$PWD/export/fraud:/models/fraud" -e MODEL_NAME=fraud tensorflow/serving
curl -s localhost:8501/v1/models/fraud

$ docker run -d --name tfserving -p 8501:8501 -v "$PWD/export/fraud:/models/fraud" -e MODEL_NAME=fraud tensorflow/serving
f11f23cb383b3a387a4d03f7851e6fedac40967209988d7aa0ce3a125edad016
$ curl -s localhost:8501/v1/models/fraud
{
 "model_version_status": [
  {
   "version": "1",
   "state": "AVAILABLE",
   "status": {
    "error_code": "OK",
    "error_message": ""
   }
  }
 ]
}

Port 8501 is the REST API (gRPC is on 8500). The status endpoint confirms version 1 loaded; no Python and no model code ran inside the container. A client sends JSON to :predict:

import json
import urllib.request
import numpy as np

URL = "http://localhost:8501/v1/models/fraud:predict"
rows = np.load("sample.npy").tolist()

def post(payload):
    request = urllib.request.Request(URL, data=json.dumps(payload).encode(),
                                     headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request) as response:
        return json.load(response)

print(post({"instances": rows}))                              # row format
print(post({"signature_name": "serving_default", "inputs": {"features": rows[:1]}}))
try:
    post({"instances": [[1.0, 2.0]]})                        # wrong width
except urllib.error.HTTPError as e:
    print(e.code, json.load(e)["error"].splitlines()[0])

$ python client.py
{'predictions': [[1.29186892e-05], [0.979891896], [1.83526566e-07]]}
{'outputs': [[1.29186892e-05]]}
400 Matrix size-incompatible: In[0]: [1,2], In[1]: [20,64]

The server returns the same three probabilities as train.py, at full precision. The REST API accepts two request shapes: instances, a list of rows, answered with predictions; and inputs, keyed by the signature’s input names, answered with outputs. A request with 2 features instead of 20 comes back as HTTP 400 carrying TensorFlow’s own error — the first matrix multiply cannot combine a 1×2 input with the 20×64 weights. Validate inputs before they reach the server if you want friendlier messages.

Rolling Out a New Version

TF Serving watches the model directory. Adding a new numbered subdirectory is a deployment. Here version 1 is copied to 2 to stand in for a retrained export:

cp -r export/fraud/1 export/fraud/2
sleep 5
curl -s localhost:8501/v1/models/fraud

$ cp -r export/fraud/1 export/fraud/2
$ sleep 5
$ curl -s localhost:8501/v1/models/fraud
{
 "model_version_status": [
  {
   "version": "2",
   "state": "AVAILABLE",
   "status": {
    "error_code": "OK",
    "error_message": ""
   }
  },
  {
   "version": "1",
   "state": "END",
   "status": {
    "error_code": "OK",
    "error_message": ""
   }
  }
 ]
}

Without a restart, the server loaded version 2 and then unloaded version 1 (END). The default policy serves only the highest version, so :predict requests now go to version 2. Rolling back is deleting the 2 directory: the server then reloads version 1. To keep several versions live at once, for A/B tests, TF Serving takes a model config file with a version policy.

Serving with FastAPI

When a prediction needs Python around it — input validation, feature lookups, business rules — a small web app that loads the .keras file is simpler than TF Serving. Install it with pip install fastapi uvicorn and save as app.py:

from contextlib import asynccontextmanager
import numpy as np
import keras
from fastapi import FastAPI
from pydantic import BaseModel, Field

models = {}

@asynccontextmanager
async def lifespan(app):
    models["fraud"] = keras.models.load_model("fraud.keras")   # load once
    yield

app = FastAPI(lifespan=lifespan)

class Transaction(BaseModel):
    features: list[float] = Field(min_length=20, max_length=20)

@app.post("/predict")
def predict(tx: Transaction):
    x = np.array([tx.features], dtype="float32")
    p = float(models["fraud"](x, training=False)[0, 0])
    return {"fraud_probability": round(p, 4), "is_fraud": p >= 0.5}

$ uvicorn app:app --port 8000
INFO:     Started server process [11786]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

The lifespan function loads the model once at startup, not on every request. For a single row, calling the model directly — models["fraud"](x, training=False) — skips the batching machinery predict sets up on every call. The Field constraint makes FastAPI reject a request with the wrong number of features before the model ever sees it. With the server running, a client in a second terminal:

import json
import urllib.request
import numpy as np

def post(features):
    body = json.dumps({"features": features}).encode()
    request = urllib.request.Request("http://localhost:8000/predict", data=body,
                                     headers={"Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(request) as response:
            return response.status, json.load(response)
    except urllib.error.HTTPError as e:
        return e.code, json.load(e)["detail"][0]["msg"]

for row in np.load("sample.npy").tolist():
    print(*post(row))
print(*post([1.0, 2.0]))

$ python api_client.py
200 {'fraud_probability': 0.0, 'is_fraud': False}
200 {'fraud_probability': 0.9799, 'is_fraud': True}
200 {'fraud_probability': 0.0, 'is_fraud': False}
422 List should have at least 20 items after validation, not 2

Same three predictions, now with a business-level is_fraud flag. The short request fails with HTTP 422 and a readable message from Pydantic’s validation, where TF Serving returned a 400 with a matrix-shape error. uvicorn runs one worker process by default; pass --workers to run several, each with its own copy of the model.

Converting to TFLite

The TFLite converter turns a SavedModel into a single .tflite flatbuffer for phones and small devices. Optimize.DEFAULT stores the weights as 8-bit integers.

import os
import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model("export/fraud/1")
with open("fraud_float.tflite", "wb") as f:
    f.write(converter.convert())

converter = tf.lite.TFLiteConverter.from_saved_model("export/fraud/1")
converter.optimizations = [tf.lite.Optimize.DEFAULT]      # int8 weights
with open("fraud_quant.tflite", "wb") as f:
    f.write(converter.convert())

for name in ["fraud.keras", "fraud_float.tflite", "fraud_quant.tflite"]:
    print(f"{name:20s} {os.path.getsize(name):>7,} bytes")

$ python to_tflite.py
fraud.keras           67,144 bytes
fraud_float.tflite    16,012 bytes
fraud_quant.tflite     7,264 bytes

The model has 3,457 parameters — 13.8 KB as float32. The .keras file is several times that: it is a zip whose model.weights.h5 alone is 63,136 bytes, because it also holds the Adam optimizer’s state — two moment estimates per weight — so training can resume after loading. The float TFLite file is the weights plus a small graph, and quantizing the weights to one byte each cuts it to 7.3 KB. On a model this small the fixed overhead dominates; on a large model, weight quantization approaches a 4× reduction.

Running It with LiteRT

tf.lite.Interpreter still works in TF 2.21 but prints a deprecation warning pointing to LiteRT, TFLite’s new name, whose interpreter ships separately: pip install ai-edge-litert. It does not import TensorFlow at all.

import numpy as np
from ai_edge_litert.interpreter import Interpreter

x = np.load("sample.npy")

for path in ["fraud_float.tflite", "fraud_quant.tflite"]:
    interpreter = Interpreter(model_path=path)
    inp = interpreter.get_input_details()[0]
    out = interpreter.get_output_details()[0]
    print(path, "expects", inp["shape"], inp["dtype"].__name__)
    interpreter.resize_tensor_input(inp["index"], x.shape)     # batch of 3
    interpreter.allocate_tensors()
    interpreter.set_tensor(inp["index"], x)
    interpreter.invoke()
    print("  ", interpreter.get_tensor(out["index"]).ravel().round(4))

$ python run_tflite.py
fraud_float.tflite expects [ 1 20] float32
   [0.     0.9799 0.    ]
fraud_quant.tflite expects [ 1 20] float32
   [0.     0.9803 0.    ]

The interpreter works on raw tensor indices: look up the input and output details, set the input, invoke, read the output. The converted model defaults to a batch of 1, so resize_tensor_input followed by allocate_tensors makes room for three rows. The float model matches the Keras model exactly; the weight-quantized one moves the fraud probability from 0.9799 to 0.9803 — the cost of rounding its weights. LiteRT also prints an INFO line on stderr saying it created the XNNPACK delegate, its optimized CPU backend.

Full Integer Quantization

Microcontrollers and many phone accelerators run only integer math. Full int8 quantization also converts activations, inputs, and outputs to 8-bit, and needs sample inputs to measure each tensor’s range. Here those samples are drawn from the same distribution as the training data; in practice use a few hundred real training rows.

import os
import numpy as np
import tensorflow as tf

calibration = np.random.default_rng(1).normal(size=(200, 20)).astype("float32")

def representative_data():
    for row in calibration:
        yield [row[None]]                        # one sample per step

converter = tf.lite.TFLiteConverter.from_saved_model("export/fraud/1")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
with open("fraud_int8.tflite", "wb") as f:
    f.write(converter.convert())
print(f"fraud_int8.tflite {os.path.getsize('fraud_int8.tflite'):,} bytes")

$ python int8.py
fraud_int8.tflite 8,584 bytes

The fully quantized file is slightly larger than the weight-only one, mostly because it also records a scale and zero point for every activation tensor. Running it takes a quantization step on each side, because the model now expects int8 in and gives int8 out. Saved as run_int8.py:

import numpy as np
from ai_edge_litert.interpreter import Interpreter

interpreter = Interpreter(model_path="fraud_int8.tflite")
interpreter.allocate_tensors()
inp = interpreter.get_input_details()[0]
out = interpreter.get_output_details()[0]
print("input:", inp["dtype"].__name__, inp["quantization"])
print("output:", out["dtype"].__name__, out["quantization"])

in_scale, in_zero = inp["quantization"]
out_scale, out_zero = out["quantization"]
for row in np.load("sample.npy"):
    q = np.clip(np.round(row / in_scale + in_zero), -128, 127).astype(np.int8)[None]
    interpreter.set_tensor(inp["index"], q)
    interpreter.invoke()
    raw = int(interpreter.get_tensor(out["index"])[0, 0])    # int8 math would overflow
    print(f"raw {raw:5d} -> probability {(raw - out_zero) * out_scale:.4f}")

$ python run_int8.py
input: int8 (0.028629176318645477, -4)
output: int8 (0.00390625, -128)
raw  -128 -> probability 0.0000
raw   124 -> probability 0.9844
raw  -128 -> probability 0.0000

Each tensor carries a (scale, zero_point) pair: real = (int8 - zero_point) × scale. The input scale of 0.0286 with zero point −4 covers −3.55 to 3.75, matching the smallest and largest values in the calibration data (−3.549 and 3.752); inputs outside that range are clipped. The output scale is exactly 1/256 with zero point −128, so int8 -128 means probability 0 and 127 means 0.996. The fraud row comes out as 0.9844 — coarser than 0.9799, because a probability now has only 256 possible values. The int() matters: subtracting -128 from an int8 value of 124 in NumPy’s int8 arithmetic overflows and wraps to a negative number instead of 252.

Practice

import json
import urllib.request
import numpy as np

row = np.load("sample.npy")[1:2].tolist()
for version in [2, 1]:
    url = f"http://localhost:8501/v1/models/fraud/versions/{version}:predict"
    request = urllib.request.Request(url, data=json.dumps({"instances": row}).encode())
    try:
        with urllib.request.urlopen(request) as response:
            print(version, json.load(response))
    except urllib.error.HTTPError as e:
        print(version, e.code, json.load(e)["error"])

$ python p1_versions.py
2 {'predictions': [[0.979891896]]}
1 404 Servable not found for request: Specific(fraud, 1)

import numpy as np
import tensorflow as tf
import keras

model = keras.models.load_model("fraud.keras")
archive = keras.export.ExportArchive()
archive.track(model)

@tf.function(input_signature=[tf.TensorSpec((None, 20), tf.float32, name="features")])
def score(features):
    p = model(features, training=False)
    return {"probability": p, "is_fraud": p >= 0.5}

archive.add_endpoint("serving_default", score)
archive.write_out("export_labels/fraud/1", verbose=False)

serve = tf.saved_model.load("export_labels/fraud/1").signatures["serving_default"]
print(serve.structured_outputs)
result = serve(features=tf.constant(np.load("sample.npy")))
print({name: t.numpy().ravel() for name, t in result.items()})

$ python p2_archive.py
{'probability': TensorSpec(shape=(None, 1), dtype=tf.float32, name='probability'), 'is_fraud': TensorSpec(shape=(None, 1), dtype=tf.bool, name='is_fraud')}
{'probability': array([1.2918689e-05, 9.7989190e-01, 1.8352657e-07], dtype=float32), 'is_fraud': array([False,  True, False])}

import numpy as np
from ai_edge_litert.interpreter import Interpreter

X = np.random.default_rng(42).normal(size=(1000, 20)).astype("float32")

def run(path, quantize):
    interp = Interpreter(model_path=path)
    inp, out = interp.get_input_details()[0], interp.get_output_details()[0]
    interp.resize_tensor_input(inp["index"], X.shape)
    interp.allocate_tensors()
    s_in, z_in = inp["quantization"]
    s_out, z_out = out["quantization"]
    x = np.clip(np.round(X / s_in + z_in), -128, 127).astype(np.int8) if quantize else X
    interp.set_tensor(inp["index"], x)
    interp.invoke()
    y = interp.get_tensor(out["index"]).astype("float32")
    return ((y - z_out) * s_out if quantize else y).ravel()

float_p = run("fraud_float.tflite", quantize=False)
int8_p = run("fraud_int8.tflite", quantize=True)
print(f"max probability difference: {np.abs(float_p - int8_p).max():.4f}")
print(f"same decision: {np.mean((float_p >= 0.5) == (int8_p >= 0.5)):.3f}")

$ python p3_agreement.py
max probability difference: 0.0745
same decision: 0.995