What does 'data platform architecture' mean beyond a single pipeline?

A single pipeline moves one dataset from source to output. A platform is the set of shared concerns every pipeline on it gets for free: a common storage layout (lake/warehouse/lakehouse), a shared quality-gating pattern, lineage capture, and observability — so the tenth pipeline doesn't reinvent what the first one already solved.

Why does this lesson combine lake, quality gates, and lineage instead of building one clean example for each?

Each concept has its own dedicated lesson already — data-lake-warehouse-lakehouse, data-quality, data-lineage, observability. This one exists specifically to show them composed in one pipeline, because in a real platform they are never separate: every stage of a real pipeline is simultaneously a storage decision, a quality checkpoint, and a lineage edge, and none of the individual lessons shows that overlap.

Why does the gold stage take so much longer than bronze and silver in this lesson's trace?

The group-by aggregate in the gold stage is DuckDB's first use of that particular query operator in this connection, and operator setup carries a real one-time cost independent of the tiny row count here. This is exactly why the observability lesson's stage-by-stage tracing matters — a raw total-duration number would have hidden which stage actually dominates, and 'it's the newest operator type, not more data' is a real, traceable explanation rather than a guess.

Is this a template for a production platform?

No — it is a minimal, honest skeleton meant to show how the pieces fit, at a scale small enough to run in a few milliseconds. A production platform adds orchestration (the orchestration-patterns lesson), real access control (data-governance), PII handling (data-security-pii), CI/CD gating (ci-cd-for-data), and a real backup strategy (disaster-recovery) around this same core shape.

Data Platform Architecture: Wiring the Pieces Together

Run one small pipeline through all three layers — lake, quality-gated transform, warehouse aggregate — capturing lineage and a stage-by-stage trace as it goes.

Three layers, one lineage dict, one trace dict

import duckdb, time, json, os

con = duckdb.connect()
lineage = {}
run_trace = {}

os.makedirs("platform_lake", exist_ok=True)
with open("platform_lake/orders.json", "w") as f:
    json.dump([
        {"order_id": 1, "customer_id": 1, "amount": 25.5, "status": "completed"},
        {"order_id": 2, "customer_id": 2, "amount": 12.0, "status": "completed"},
        {"order_id": 3, "customer_id": 1, "amount": 8.0, "status": "completed"},
    ], f)

def timed_stage(name, fn):
    t0 = time.perf_counter()
    result = fn()
    run_trace[name] = time.perf_counter() - t0
    return result

Bronze: lake, schema-on-read

def bronze_stage():
    con.execute("create schema if not exists bronze")
    con.execute("create or replace table bronze.orders as select * from read_json_auto('platform_lake/orders.json')")
    lineage["bronze.orders"] = {"op": "ingest (lake, schema-on-read)", "inputs": ["platform_lake/orders.json"]}
    return con.execute("select count(*) from bronze.orders").fetchone()[0]

bronze_rows = timed_stage("bronze", bronze_stage)
print(f"bronze: {bronze_rows} rows ingested as-is")

$ python platform_architecture.py
== platform run: bronze (lake) -> silver (quality-gated) -> gold (warehouse) ==
bronze: 3 rows ingested as-is

Silver: a quality gate that can halt the pipeline

def silver_stage():
    checks = con.execute("select count(*) filter (amount < 0) from bronze.orders").fetchone()[0]
    if checks:
        raise ValueError(f"quality gate failed: {checks} row(s) with negative amount")
    con.execute("create schema if not exists silver")
    con.execute("create or replace table silver.orders as select * from bronze.orders where status = 'completed'")
    lineage["silver.orders"] = {"op": "quality-gated filter", "inputs": ["bronze.orders"]}
    return con.execute("select count(*) from silver.orders").fetchone()[0]

silver_rows = timed_stage("silver", silver_stage)
print(f"silver: {silver_rows} rows passed the quality gate")

silver: 3 rows passed the quality gate

Gold: warehouse, schema-on-write

def gold_stage():
    con.execute("create schema if not exists gold")
    con.execute("create or replace table gold.revenue_by_customer as "
                "select customer_id, sum(amount) as revenue from silver.orders group by customer_id")
    lineage["gold.revenue_by_customer"] = {"op": "aggregate (warehouse, schema-on-write)", "inputs": ["silver.orders"]}
    return con.execute("select * from gold.revenue_by_customer order by customer_id").df()

gold_df = timed_stage("gold", gold_stage)
print("gold:")
print(gold_df.to_string(index=False))

gold:
 customer_id  revenue
           1     33.5
           2     12.0

The platform summary: lineage plus trace, together

print("lineage:")
for table, edge in lineage.items():
    print(f"  {table}  <- {edge['op']} <- {', '.join(edge['inputs'])}")
print("run trace:")
for stage, seconds in run_trace.items():
    print(f"  {stage:<8} {seconds*1000:.2f}ms")

== platform summary ==
lineage:
  bronze.orders  <- ingest (lake, schema-on-read) <- platform_lake/orders.json
  silver.orders  <- quality-gated filter <- bronze.orders
  gold.revenue_by_customer  <- aggregate (warehouse, schema-on-write) <- silver.orders
run trace:
  bronze   3.22ms
  silver   1.77ms
  gold     584.34ms

The gold stage’s aggregate dominates the trace — not because it processed more data (all three stages touch the same 3 rows), but because it is the first group-by DuckDB’s query engine runs in this connection, and that operator’s one-time setup cost dwarfs three rows of actual work. Nothing in this lesson asserted that; the trace measured it, exactly the way the observability lesson’s stage breakdown is meant to be read — as a place to look, not a conclusion to assume.

The same platform, a bad batch

with open("platform_lake/orders.json", "w") as f:
    json.dump([
        {"order_id": 1, "customer_id": 1, "amount": 25.5, "status": "completed"},
        {"order_id": 3, "customer_id": 1, "amount": -5.0, "status": "completed"},  # bad row
    ], f)

bronze_rows = timed_stage("bronze", bronze_stage)
print(f"bronze: {bronze_rows} rows ingested as-is (including the one bad row)")
try:
    timed_stage("silver", silver_stage)
except ValueError as e:
    print(f"PIPELINE HALTED at silver: {e}")
    print("gold stage skipped - the gate stopped a bad row from ever reaching an aggregate")

bronze: 2 rows ingested as-is (including the one bad row)
PIPELINE HALTED at silver: quality gate failed: 1 row(s) with negative amount
gold stage skipped - the gate stopped a bad row from ever reaching an aggregate

bronze.orders still has the bad row — the lake layer keeps everything, on purpose, exactly as the data-lake-warehouse-lakehouse lesson’s “raw data is kept” framing describes. The gate is at the silver boundary, not at ingestion, so the platform still has the original data to investigate and reprocess once the source issue is fixed, instead of having discarded it.

Practice

1. Add a fourth, valid order for a third customer and re-run the full pipeline.

gold:
 customer_id  revenue
           1    33.50
           2    12.00
           3    19.99

One more row flows cleanly through all three layers and lineage stages — nothing about the platform’s shape changes as data volume grows, which is the actual point of building it once. Note pandas now prints every value to two decimal places (33.50, not 33.5) once the column contains a value that genuinely needs them (19.99) — a column-wide formatting decision, not a per-row one.

2. Write a `trace_table` function over this lesson's lineage dict and trace `gold.revenue_by_customer`.

def trace_table(table, depth=0):
    pad = "  " * depth
    if table not in lineage:
        print(f"{pad}{table}  <- source")
        return
    edge = lineage[table]
    print(f"{pad}{table}  <- {edge['op']} <- {', '.join(edge['inputs'])}")
    for inp in edge["inputs"]:
        trace_table(inp, depth + 1)

trace_table("gold.revenue_by_customer")

gold.revenue_by_customer  <- aggregate (warehouse, schema-on-write) <- silver.orders
  silver.orders  <- quality-gated filter <- bronze.orders
    bronze.orders  <- ingest (lake, schema-on-read) <- platform_lake/orders.json
      platform_lake/orders.json  <- source

This is table-level lineage — the same structure as the dedicated data-lineage lesson’s tracker, minus its column-level detail (this dict doesn’t record which specific column of silver.orders fed revenue). Table-level is enough to answer “what feeds this table”; the dedicated lesson’s column-level version is what’s needed to answer “what feeds this column.”

3. What role does the quality gate in this lesson play relative to the CI/CD lesson's deploy gate?

Both block bad output from reaching the next stage, but at different points: this lesson’s gate runs mid-pipeline, between silver and the data that would become gold, on every single run. The CI/CD lesson’s gate runs before a new pipeline version is promoted to production — one guards data quality per run, the other guards code changes per deploy. A mature platform needs both; they catch different failure classes.

4. Which of this platform's pieces would need to change to add a second data source, e.g. a customers file?

The lineage dict and the stage functions both need one more entry each (a bronze.customers stage, and a lineage edge for it) — but the pattern (lake ingest, gated transform, warehouse aggregate, traced and recorded) does not change. That reusability across sources, without reinventing quality gating or lineage capture each time, is what “platform” means as opposed to “one pipeline.”

This closes the data-engineering track’s production-concerns arc: batch vs. streaming, storage shape, modeling, testing, streaming architecture, lineage, governance, security, observability, cost, CI/CD, and disaster recovery, composed here into one platform.