Applies toApache Kafka (self-managed, Strimzi), Amazon MSK (provisioned, serverless), Confluent Cloud/Platform, Azure Event Hubs (Kafka endpoint), Amazon Kinesis Data Streams, Google Pub/Sub
The options at a glance
| Self-managed Kafka | Amazon MSK | Confluent Cloud | Azure Event Hubs | Kinesis Data Streams | |
|---|---|---|---|---|---|
| Protocol | Kafka | Kafka | Kafka | Native AMQP/HTTPS + Kafka endpoint | Kinesis API |
| Operations | You run brokers, upgrades, storage | AWS runs brokers; you size and tune | Fully managed | Fully managed | Fully managed |
| Ecosystem | Full (Connect, Streams, any tool) | Full Kafka; MSK Connect | Full plus managed connectors, schema registry, stream processing | Kafka clients work; not every Kafka feature/API | AWS integrations (Lambda, Firehose, Flink) |
| Portability | Highest | Kafka-portable | Kafka-portable, multi-cloud | Kafka clients portable; ops tied to Azure | AWS-specific |
| Network fit | Anywhere | VPC-native on AWS | Private networking options per cloud | VNet/Private Link on Azure | VPC endpoints on AWS |
Check each service's current Kafka compatibility (transactions, compacted topics, Kafka Streams, admin APIs) for the features you rely on.
Decision questions
- Do you need the Kafka protocol and ecosystem? If yes, the choice is between self-managed, MSK, Confluent or Event Hubs' Kafka endpoint.
- Who will operate it? Self-managed Kafka needs skills in partitions, ISR, upgrades and capacity. Without them, use a managed service.
- Where are producers and consumers? Keep the platform in the same cloud and region to avoid latency and egress.
- How portable must it be? Multi-cloud or hybrid points to Kafka-compatible options.
- What else do you need? Schema registry, connectors, stream processing, governance. Count them as part of the platform, not add-ons for later.
Typical fits
| Situation | Often a good fit |
|---|---|
| Azure-centric, telemetry and ingestion into Fabric/Databricks | Event Hubs |
| AWS-centric, Kafka skills in the team | MSK |
| AWS-centric, simple streams into Lambda/Firehose | Kinesis |
| Multi-cloud or wanting a full managed Kafka ecosystem | Confluent Cloud |
| On-prem or strict control requirements | Self-managed Kafka (Strimzi on Kubernetes or VMs) |
Design points regardless of choice
| Area | Decide |
|---|---|
| Topic naming and ownership | Domain-based names, an owner per topic |
| Schemas | Registry with compatibility rules (see the schema registry post) |
| Partitioning | Key choice and partition counts sized for consumer parallelism |
| Retention | Per topic by use; tiered storage where available |
| Security | mTLS/OAuth/SASL, ACLs per service identity, private networking |
| DR | Replication across regions (MirrorMaker 2, Cluster Linking, geo-replication) and tested failover |
| Quotas | Per-client quotas to stop one producer starving others |
Tip
Treat the choice as a platform product: count the cost of operations staff, schema governance and connectors, not only broker or throughput-unit pricing.
Checklist
- Kafka protocol need decided
- Operating model and skills confirmed
- Compatibility of required Kafka features verified
- Network placement with producers/consumers planned
- Schemas, naming, security, DR and quotas designed