Home / Data Platform / Streaming & CDC
Kafka, Event Hubs, Debezium CDC and ingestion patterns: throughput, offsets, recovery and design. 36 posts.
The core Kafka concepts in plain terms. Create a topic, produce and consume messages, see how partitions and keys affect ordering, and read consumer group lag, all with the standard Kafka command-line tools.
Move partitions between brokers without hurting producers and consumers. Generate rack-aware plans, throttle replication traffic, monitor progress, handle stuck reassignments, and rebalance leaders after adding or removing brokers.
Producers fail with RecordTooLargeException or MESSAGE_TOO_LARGE. Find which limit is hit, raise the right settings in the right order, or shrink the messages instead.
After a producer release, consumers fail with SerializationException or schema-not-found errors and lag explodes. Identify the bad schema version, stop the producer, get consumers past the bad records, restore compatibility settings and prevent unchecked schema changes.
Someone ran kafka-topics --delete on a live topic. Producers fail or auto-create an empty topic, consumers reset, and data is gone from the brokers. Freeze auto-creation, recreate the topic with the right config, replay from mirrors, sinks or sources, and fix consumer offsets.
The Kafka Connect REST calls on-call engineers use most. Find failed tasks, restart only what failed, pause or stop connectors, change log levels and reset offsets safely.
Consumer groups keep rebalancing, lag grows and messages are processed twice. Find which timeout triggers it, then fix processing time, assignment strategy and pod restarts with static membership or KIP-848.
Exactly-once pipelines fail with fenced producers, or read_committed consumers stop making progress while the topic keeps growing. Diagnose with kafka-transactions.sh, abort hanging transactions safely and fix transactional.id and timeout design.
Kafka 4.0 removed ZooKeeper and old protocol versions, raised Java minimums and removed several configs. Before you upgrade, confirm KRaft mode on 3.3+ metadata, find clients older than 2.1, move brokers to Java 17, and clean configs that no longer exist.
Producers or consumers fail with Kafka authorization exceptions. Find the principal the broker really sees, read the authorizer log, and add the minimal ACLs with kafka-acls.sh.
The Schema Registry REST calls you need on call - list subjects, inspect versions and IDs, test compatibility before deploying, change modes and clean up safely after a 409 rejection.
The Kafka command-line tools on-call engineers use most, grouped by task. Check cluster health, manage topics and configs, inspect and reset consumer groups, and move partitions safely.
Real-time pipelines cost more to build and run, and many requests for "real time" don't need it. Turn freshness requests into explicit latency requirements, map them to batch, micro-batch or streaming, account for the operational cost of each, and use hybrid designs where only part of the data must be fast.
The streaming backbone is hard to change once producers depend on it. Compare self-managed and managed Kafka with cloud-native services on protocol compatibility, operations effort, ecosystem and network fit, and decide by who will run it and how portable it must be.
Each source system has a best way to get data out. Classify sources by type, change rate and the load you may put on them, pick log-based CDC, incremental or full extracts, API pulls, file drops or event streams accordingly, and design for deletes, schema changes, replays and source protection.
Connectors running one task, tiny batches and verbose JSON converters move far less data than the cluster can handle. Measure per-connector throughput and lag, raise tasks to match partitions, tune connector and client batch settings, pick efficient converters, use DLQs instead of stopping on bad records, and size the worker cluster.
Most Kafka performance problems are client settings, not broker size. Measure where time goes, tune producer batching, linger and compression, keep idempotence and acks safe, tune consumer fetch and poll settings, scale consumers to partitions and check the broker side last.
Kafka clusters grow disks because every topic keeps data longer than anyone needs, with more partitions than consumers can use. Measure size per topic, set retention and compaction per use case, move long retention to tiered storage, compress, and right-size partition counts.
Event Hubs namespaces are often sized once for a launch and never revisited, or throttle because partitions were chosen badly. Measure ingress, egress and events per second, choose the tier, set throughput units with auto-inflate and a plan to scale down, pick partition counts for consumer parallelism, and review Capture and retention costs.
Recover a Debezium CDC connector that fails because the source purged the binlog or WAL it needed, or its offsets were lost. Choose between incremental snapshots, a filtered re-snapshot or a new connector, and keep consumers consistent.
Fix Event Hubs consumers that keep disconnecting each other, stall on partitions or restart from the wrong position. Check consumer groups, partition ownership in the checkpoint store, client versions and retention.
Get a Kafka broker or KRaft controller quorum back after a crash. Read the startup errors, handle long log recovery, corrupted index or segment files, wrong cluster IDs and lost controller quorum without making data loss worse.
Recover when a consumer group suddenly starts from the beginning (duplicates) or from the end (missed data). Find why offsets were lost, stop consumers, reset to the right position by time or saved offsets, and protect committed offsets.
Restore a Kafka cluster where partitions are offline or under-replicated after one or more brokers failed. Find affected partitions, bring brokers back, decide on unclean leader election only as a last resort, and rebalance safely.
Decide when streaming is worth it, then move a batch pipeline to Kafka and Structured Streaming into Delta tables. Covers exactly-once sinks, watermarks for late data, stateful aggregations, triggers and operating streams in production.
Replace cron jobs that poll SFTP folders once a night with ingestion that reacts to each file as it lands. Use managed SFTP into object storage, storage events, Auto Loader or Snowpipe, file validation and a quarantine path.
Replace database triggers and timestamp-based incremental extracts with log-based change data capture. Prepare MySQL, PostgreSQL or SQL Server, deploy Debezium on Kafka Connect, land changes in the lakehouse with MERGE and remove the old triggers.
Move a production Kafka cluster off ZooKeeper before ZooKeeper support disappears. Pre-checks, deploying KRaft controllers in migration mode, moving brokers, finalising, and the rollback window you get before finalisation.
Move Kafka off self-run VMs onto a managed service. Compare MSK, Confluent Cloud and the Event Hubs Kafka endpoint, replicate topics and offsets with MirrorMaker 2 or Cluster Linking, migrate consumers before producers and cut over safely.
Diagnose ServerBusy and quota-exceeded errors in Event Hubs, size throughput or processing units, fix partition hot spots and consumer lag.
Check connector status, source-side replication slots and binlogs, and the sink, to find where change-data-capture lag builds up and recover without losing changes.
Find which topics fill the broker disks, free space safely by lowering retention, recover an offline broker, and prevent it with quotas, alerts and tiered storage.
Measure lag per partition, tell slow processing from rebalance storms and hot partitions, and reset offsets safely when you need to.
Replicate Kafka topics and consumer group offsets to a DR cluster with MirrorMaker 2. Covers naming policies, offset translation, monitoring lag, and a consumer failover runbook.
Configure Kafka so that a broker or availability zone can fail without data loss or downtime, with the right replication settings, rack awareness, controller quorum, producer settings and monitoring.
Run a single combined broker/controller Kafka node with KRaft for development and test, set replication factors that work on one node, and list the settings you must change before production.
No posts in this category yet.