What is lambda architecture?

Running two parallel pipelines over the same events: a batch layer that periodically reprocesses everything from scratch (slow, complete, accurate) and a speed layer that processes only recent events in real time (fast, approximate). A query merges both layers' results. The complexity cost is maintaining two separate codepaths that must agree.

What is kappa architecture, and how is it different?

Kappa architecture uses one stream-processing pipeline for everything — no separate batch layer. Corrections happen by replaying the event log from any point through the same code, instead of maintaining a second codebase. It trades the batch layer's simplicity (dumb, periodic recomputation) for a requirement that the stream processor be able to replay historical data efficiently, which needs a log retaining enough history (Kafka with long retention, for instance).

Why would anyone still choose lambda over kappa today?

When the batch computation genuinely needs different tooling than the streaming one — a nightly Spark job doing a complex join a streaming engine cannot efficiently express yet, or a batch layer that already exists and works. Kappa is the simpler mental model when a team can build it from scratch, but 'simpler in theory' does not erase a working batch pipeline someone already operates.

What counts as stateful vs. stateless stream processing?

Stateless processing handles each event independently — parsing a JSON payload, validating a field, filtering by a condition — and needs no memory of prior events. Stateful processing needs to remember something across events — a running total, a deduplication set, a session window — which is what makes exactly-once semantics and watermarks (see the streaming-exactly-once lesson) necessary in the first place.

Streaming Architecture: Lambda and Kappa

Build a lambda architecture’s two layers by hand, feed them a late-arriving event, and watch the merged view be wrong until the next batch run corrects it.

The event log, including one late arrival

from datetime import datetime, timedelta

events = [
    (datetime(2026, 1, 4, 0, 5), 12.0), (datetime(2026, 1, 4, 3, 0), 8.0),
    (datetime(2026, 1, 4, 9, 0), 20.0), (datetime(2026, 1, 4, 14, 0), 15.0),
    (datetime(2026, 1, 4, 22, 0), 9.0),
    # arrives at 23:50 -- after the batch run below already fired
    (datetime(2026, 1, 4, 23, 50), 30.0),
]
batch_cutoff = datetime(2026, 1, 5, 0, 0)

Batch layer: complete, but only as of its last run

events_at_batch_time = events[:5]  # the late event has not arrived yet
batch_total = sum(amount for ts, amount in events_at_batch_time if ts < batch_cutoff)
print(f"batch layer total for 2026-01-04: ${batch_total:.2f} (5 events seen)")

$ python streaming_architecture.py
== lambda architecture ==
-- batch layer: reprocesses everything up to the cutoff, run once a day --
batch layer total for 2026-01-04: $64.00 (5 events seen)

Speed layer: live, but only sees what arrives after the cutoff

speed_layer_events = [(ts, amt) for ts, amt in events if ts >= batch_cutoff]
speed_total = sum(amt for ts, amt in speed_layer_events)
print(f"speed layer total since cutoff: ${speed_total:.2f} ({len(speed_layer_events)} events)")

-- speed layer: only events since the batch cutoff, updated live --
speed layer total since cutoff: $0.00 (0 events)

The merged view a dashboard actually shows

print(f"batch (${batch_total:.2f}) + speed (${speed_total:.2f}) = ${batch_total + speed_total:.2f}")

-- merged view: what a dashboard shows right now --
batch ($64.00) + speed ($0.00) = $64.00
but the late event (arrived 23:50, after the batch already ran) is invisible to both layers

The $30 order that arrived at 23:50, timestamped for the 4th, is invisible to both layers — too late for the batch run that already fired, and its own timestamp is before the speed layer’s cutoff so the speed layer does not count it as “since cutoff” either. This is exactly the late-event problem the streaming-exactly-once lesson’s watermarks section covers, showing up one layer higher, in the architecture itself.

Reconciliation: the next batch run fixes it

corrected_total = sum(amount for ts, amount in events if ts < batch_cutoff)
print(f"corrected batch total for 2026-01-04: ${corrected_total:.2f} (6 events, late one now included)")
print(f"yesterday's merged view was off by ${corrected_total - (batch_total + speed_total):.2f}")

-- next day's batch run reprocesses the FULL log, including the late arrival --
corrected batch total for 2026-01-04: $94.00 (6 events, late one now included)
yesterday's merged view was off by $30.00

The batch layer is self-healing — every run reprocesses everything, so a late event is simply included next time. But there was a real, wrong number shown for up to a full day, and nothing told anyone it would correct itself.

Kappa: replay the same code instead of maintaining two

running_total = 0.0
for ts, amount in sorted(events):
    if ts.date() == batch_cutoff.date() - timedelta(days=1):
        running_total += amount
print(f"kappa: single replay of the log gives ${running_total:.2f} directly - no reconciliation step needed")

== kappa architecture: one stream, no separate batch layer ==
kappa: single replay of the log gives $94.00 directly - no reconciliation step needed

Same correct answer, but there was never a “wrong answer” window — the same stream-processing logic that handles new events also handles a full historical replay. What lambda solved with a second codebase, kappa solves by making the one codebase replayable, at the cost of needing an event log (like Kafka) that retains enough history to replay from.

Practice

1. Add a second late event arriving at 23:55 for $5, and recompute the corrected batch total.

corrected batch total for 2026-01-04: $99.00 (7 events, both late ones now included)

Every late arrival before the next batch run is silently absorbed the same way — the architecture does not distinguish “one late event” from “several.”

2. Change the speed layer to also pick up events from the *previous* calendar day, not just after the cutoff timestamp.

This is the “backfill window” trick real speed layers use — instead of trusting event timestamps blindly, re-scan a small trailing window (say, the last 2 hours before cutoff) on every speed-layer update, accepting some double-counting risk against the batch layer in exchange for catching moderately-late events sooner. It does not fully solve the problem (a sufficiently late event still slips through), just narrows the window where it can happen.

3. Classify each processing step in this lesson as stateful or stateless.

Stateless: filtering events by timestamp (ts < batch_cutoff) — each event is judged independently. Stateful: sum(...) and running_total — both need to remember an accumulated value across multiple events, which is what makes them harder to make exactly-once and harder to scale across parallel workers than a stateless filter.

4. Why can't the speed layer just always recompute the full total instead of only "since cutoff"?

It could, but then it is reimplementing the batch layer’s full reprocessing on every update — exactly the cost kappa architecture accepts deliberately (a fast enough, cheap enough replay) and exactly what lambda’s two-layer split was designed to avoid (the speed layer trades completeness for being cheap enough to run continuously).

Next: data lineage — tracking where every value in a pipeline’s output actually came from.