Kafka Tutorial with Examples.
Kafka tutorial: event streaming with Apache Kafka, from topics, partitions, producers and consumers to Kafka Streams and ksqlDB.
Architecture at a glance
Click any box to open the lesson that teaches it
Kafka architecture at a glance
producer → serializer → partitioner → brokers → consumer group
How it fits together (5 steps)
- A record starts at the producer. Its key and value are run through a serializer — plain bytes, or Avro checked against a Schema Registry — before the partitioner hashes the key to pick a partition.
- The record lands on a broker in a topic that is split into partitions, each its own ordered offset log, with replicas kept in sync and old records dropped by time or compacted.
- A consumer group divides those partitions among its members, each one reading and committing its own offset — never more active consumers than partitions.
- Two clients sit alongside producers and consumers: Kafka Connect moves data in and out of topics through source/sink connectors with no code, and Kafka Streams processes a topic's records in a library, with state backed by a changelog topic.
- What you tune once it's running: at-most-once, at-least-once, or exactly-once delivery, SASL authentication and per-resource ACLs, and batching, compression, and fetch sizing against real metrics.
Beginner
Start here — no prior experience needed
Kafka in Ten Minutes: Your First Topic
Start a Kafka broker, create a topic, produce three messages, and read them back from the console — the whole loop, end to end, in ten minutes.
Kafka Topics, Partitions, and Offsets
Watch a topic split across partitions, see how offsets are assigned per partition, and learn why Kafka only guarantees ordering inside a single partition.
Producing to Kafka from Python
Send records with confluent-kafka, read the broker's acknowledgement, handle delivery failures, and see why the producer is asynchronous by default.
Kafka Consumers and Consumer Groups
Read records from Python, watch two consumers split the partitions of a topic, and see what a rebalance does to in-flight work.
Intermediate
For developers with core concepts down
Keys, Partitioning, and Ordering Guarantees
See how a record key decides its partition, why that is the only ordering guarantee Kafka gives you, and what a custom partitioner changes.
At-Most-Once, At-Least-Once, and Exactly-Once
Reproduce message loss and duplication on purpose, then configure the producer and consumer to get each of Kafka's three delivery guarantees.
Serialization and the Schema Registry
Break a consumer by changing a JSON field, then fix it properly with Avro and a Schema Registry that rejects incompatible changes before they ship.
Kafka Connect: Source and Sink Connectors, and a Live Offset
Move data in and out of Kafka without writing a producer or consumer — a real FileStreamSource connector, a FileStreamSink connector, and the REST API that reports exactly how far each one has read.
Kafka Security: SASL Authentication and a Real ACL Denial
A default Kafka cluster trusts every connection completely. Turn on SASL/PLAIN and an ACL authorizer, and watch an unauthorized client fail with the exact exception Kafka throws — for a topic, then for a consumer group.
Advanced
Production-grade patterns for experienced engineers
Replication, ISR, Retention, and Compaction
Kill a broker and watch leadership move, then see the difference between deleting old records by time and compacting down to the latest value.
Stream Processing with Kafka Streams
Build a word count, a windowed aggregation, and a stream-table join in Kafka Streams, and see where the state actually lives when your app restarts.
Tuning Kafka Producers and Consumers
Benchmark batching, compression, and fetch sizing against a real broker, and read the metrics that tell you which one is actually your bottleneck.
Also part of Kafka
Taught in other topics. Each link opens the lesson in its home topic.
Idempotent Consumers: Surviving a Duplicate Delivery
The outbox pattern can deliver the same event twice; a consumer that tracks processed event IDs stays correct even when it does.
Dead-Letter Queues: What Happens When Retries Keep Failing
A consumer retrying a poison message forever wastes work on something that will never succeed — route it to a dead-letter queue, then replay once fixed.