Home / Data Platform / Streaming & CDC

Data PlatformRunbookAdvanced

Transformation: self-managed Kafka to a managed service (MSK, Confluent Cloud or Event Hubs)

Move Kafka off self-run VMs onto a managed service. Compare MSK, Confluent Cloud and the Event Hubs Kafka endpoint, replicate topics and offsets with MirrorMaker 2 or Cluster Linking, migrate consumers before producers and cut over safely.

Deepak Bhardwaj3 min read

Applies toApache Kafka 2.8+/3.x sources, Amazon MSK (provisioned/serverless), Confluent Cloud, Azure Event Hubs (Kafka protocol), MirrorMaker 2

Choose the target

Target Strengths Check
Amazon MSK Real Apache Kafka, AWS integration, IAM auth option You still choose broker sizes and versions (except Serverless)
Confluent Cloud Fully managed, Schema Registry, connectors, ksqlDB, Cluster Linking Pricing model, networking options
Azure Event Hubs (Kafka endpoint) No cluster to run, Azure-native Kafka feature compatibility (transactions, compaction, some admin APIs) — check current support

1. Inventory

kafka-topics.sh --bootstrap-server old:9092 --list | wc -l
kafka-topics.sh --bootstrap-server old:9092 --describe --topic orders
kafka-consumer-groups.sh --bootstrap-server old:9092 --list
kafka-configs.sh --bootstrap-server old:9092 --describe --entity-type topics --all | grep -E "retention|cleanup.policy|max.message"

Record: topics, partitions, retention, compaction, message sizes, ACLs, client libraries and versions, auth method (SASL/SCRAM, mTLS), Connect connectors, Schema Registry usage.

2. Build the target

  • Create topics with the same partition counts and configs (partition count matters for key ordering).
  • Recreate ACLs/roles; generate new client credentials.
  • Networking: private connectivity (PrivateLink, VPC peering, Private Endpoint) from producer and consumer networks.

3. Replicate

MirrorMaker 2 (works for any Kafka-compatible target):

# mm2.properties
clusters = old, new
old.bootstrap.servers = old1:9092
new.bootstrap.servers = b-1.msk.example:9096
old->new.enabled = true
old->new.topics = orders.*, payments.*
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
sync.group.offsets.enabled = true
emit.checkpoints.enabled = true

IdentityReplicationPolicy keeps topic names unchanged (no old. prefix). Offset sync translates committed consumer offsets so consumers can resume on the new cluster near where they left off.

Confluent Cluster Linking gives byte-for-byte mirror topics with identical offsets when moving to Confluent Cloud/Platform.

4. Cut over: consumers first, then producers

  1. Stop a consumer group on the old cluster; let MM2 sync its offsets.
  2. Start the same group on the new cluster (translated offsets) — it reads the mirrored data.
  3. When all consumers of a topic have moved, switch the producers to the new cluster.
  4. Stop replication for that topic once the old topic drains.

Warning

Offset translation is approximate with MM2: expect a small number of duplicates at the switch. Consumers must be idempotent. Exact offsets require Cluster Linking or a careful drain-and-switch.

5. Connect and schemas

  • Move Kafka Connect connectors to managed connectors or a Connect cluster pointed at the new bootstrap servers; copy connector offsets where supported.
  • Export/import Schema Registry subjects with the same IDs (or use a schema exporter) so existing messages still deserialise.

6. Decommission

Keep the old cluster read-only for the rollback window, then shut it down and delete its storage.

Checklist

  • Target chosen; feature compatibility verified
  • Topics, configs, ACLs recreated
  • Replication running with offset sync
  • Consumers moved, then producers, per topic group
  • Connectors and schemas migrated; old cluster retired

Related fixes