OCI Streaming production checklist: partition sizing, retention, keys, consumer groups and alarms

On OCI Streaming, the expensive mistakes get made at creation time. Partition count and retention are fixed once the stream exists, so the only fix later is a new stream and a migration of producers and consumers. Work through this checklist before production traffic arrives, and again before any change in volume.

#1. Decisions you can't change later

• Partition count is sized for peak, not average (see section 2). Oracle's documentation says partition count can't be changed after the stream is created.
• Retention is long enough for your worst consumer outage. Retention is set at creation, from 24 hours up to 7 days, and can't be changed afterwards. If a consumer can be down over a long weekend, 24 hours loses data.
• The stream pool is the right one. Private endpoint, encryption key and Kafka settings belong to the pool. Moving a stream between pools isn't an in-place change.
• The stream name is unique in its compartment. Kafka clients resolve the topic by name, so duplicate names across pools cause confusing authentication and metadata errors.

oci streaming admin stream create \
  --name orders-events \
  --partitions 12 \
  --retention-in-hours 168 \
  --stream-pool-id ocid1.streampool.oc1.eu-frankfurt-1.aaaa... \
  --wait-for-state ACTIVE

#2. Partition maths

Documented per-partition limits:

Check the Limits on Streaming Resources page before you finalize the count, because limits can change.

Size from the largest of these:

partitions_write = peak_ingest_MBps / 1        (leave 30-50% headroom)
partitions_read  = peak_read_MBps_per_group / 2
partitions_par   = number of consumer instances you need in parallel
partitions       = max(partitions_write, partitions_read, partitions_par) rounded up

Worked example: a 6 MB/s peak, two consumer groups each reading everything, and 8 parallel consumers needed in the busiest group. Write needs 6 partitions, or 9 with 50% headroom. Read is per group, so each group needs 3. Parallelism needs 8. Choose 12: it covers the headroom and grows into 12 consumers.

• Keys spread evenly. Messages with the same key go to the same partition. A low-cardinality key (country, tenant with one huge tenant) creates a hot partition that throttles while the others sit idle. Check per-partition rates during load tests, not just stream totals.
• Messages are well under 1 MB. Put large payloads in Object Storage and send a pointer.
• Producers batch. PutMessages accepts several messages per call. Batching cuts request overhead, but the 1 MB/s per partition limit still applies to the bytes.

#3. Producers

• Producers handle throttling with backoff and jitter. On the native API, check the per-message results in a PutMessages response. Failures are reported per entry, not only as an HTTP error.
• Producers retry with idempotent consumers in mind. Streaming doesn't offer exactly-once delivery, so a retried put can create a duplicate.
• Kafka producers set enable.idempotence=false and don't use transactions. Both are unsupported, and recent Java clients enable idempotence by default.
• Producers have a bounded local buffer and a defined drop or block policy for when Streaming is unavailable.

#4. Consumer groups

Consumer groups are how Streaming spreads partitions across instances and keeps offsets. The behavior to plan around:

• Each partition is reserved by one instance of a group at a time. More instances than partitions means idle instances.
• An instance that stops reading or heartbeating past its timeout loses its reservations, and its partitions are rebalanced to the remaining instances.
• With commit-on-get, offsets are committed when the next GetMessages call is made. A consumer that crashes mid-batch replays that batch on another instance. A consumer that commits before processing finishes can skip messages. For at-least-once processing, turn commit-on-get off and commit explicitly after work is done.

EP=$(oci streaming admin stream get --stream-id "$STREAM_ID" \
  --query 'data."messages-endpoint"' --raw-output)

oci streaming stream cursor create-group-cursor \
  --stream-id "$STREAM_ID" --endpoint "$EP" \
  --group-name orders-billing \
  --instance-name "$(hostname)" \
  --type TRIM_HORIZON \
  --commit-on-get false \
  --timeout-in-ms 30000

# Inspect the group's reservations and committed offsets
oci streaming stream group get \
  --stream-id "$STREAM_ID" --endpoint "$EP" --group-name orders-billing

• Instance names are stable and unique per running consumer (hostname or pod name). Two instances with the same name share reservations and collide.
• The timeout is longer than your slowest batch. If processing a batch takes longer than the timeout, the partition moves to another instance mid-batch and both process it. Send heartbeats (the ConsumerHeartbeat API) during long work, or process smaller batches.
• Commit after processing, either through explicit commits or by making sure processing finishes before the next get.
• Consumers are idempotent. Rebalances and retries both cause replays.
• Each consumer group respects 5 GETs per second per partition. A consumer that polls an empty partition in a tight loop wastes that budget. Sleep briefly when a get returns no messages.

Warning

Resetting a group's cursor to TRIM_HORIZON replays every message still within retention, up to seven days of traffic. Test downstream idempotency before you run a reset in production, and agree on it with the owning team.

#5. Security and network

• Private stream pool for internal producers and consumers. NSG on the pool endpoint allows only the client subnets on 443 (native API) and 9092 (Kafka).
• Customer-managed key from OCI Vault on the pool if your data classification requires it, with key-deletion protections in place.
• Least-privilege IAM, split by direction:

Allow dynamic-group orders-producers to use stream-push in compartment streaming-prod
  where target.stream.id = 'ocid1.stream.oc1.eu-frankfurt-1.aaaa...'
Allow dynamic-group billing-consumers to use stream-pull in compartment streaming-prod

• Kafka clients use auth tokens stored in Vault, with a rotation plan (see the Kafka connection troubleshooting post).
• Workloads on OCI use instance principals or workload identity on the native API wherever possible, instead of long-lived tokens.

#6. Observability

Streaming metrics are in the oci_streaming namespace. Alarm on at least these:

# 12 partitions x 1 MB/s x 0.7 = 8808038 bytes/s
PutMessagesThroughput.Bytes[1m]{resourceId = "<stream-ocid>"}.rate() > 8808038
GetMessagesLatency.Time[5m]{resourceId = "<stream-ocid>"}.mean() > 1000
PutMessagesThroughput.Count[10m]{resourceId = "<stream-ocid>"}.absent()

The first query alerts at 70% of aggregate write capacity. Compute the threshold from your partition count, because alarm queries take a literal value. Tune the threshold, and remember that one hot partition can throttle well before that. The last query catches producers that have silently stopped. Streaming also publishes throttling and fault metrics. Names vary between releases, so check the Streaming metrics reference for your region and alarm on any throttled-request metric being above zero for several minutes.

• Consumer lag is measured by the consumer. Streaming doesn't give you a ready-made "lag per group" alarm. Publish a custom metric with the age of the newest processed message (now minus message timestamp) per group and partition, and alarm on it.
• Alarm on lag against retention. Page when lag passes a fraction of retention, for example 25%, not when data is already about to expire.

#7. Operational readiness

• Load test at 1.5 times expected peak with production key distribution, and record per-partition rates.
• Rehearse a consumer instance failure and confirm rebalancing works and duplicates are handled.
• Document the "new stream" migration procedure (dual-write, drain, cut consumers over), because it is how partition count or retention gets changed.
• Owners, runbook links and alarm topics are tagged on the stream and pool.

#Prevent it next time

• Treat stream creation as a reviewed change. Partition count, retention and pool all go into the review, with the sizing calculation attached.
• Keep sizing inputs (peak MB/s, groups, consumer parallelism) in the repo next to the Terraform for the stream, and recheck them each quarter.
• Build lag and absence alarms into the stream module, so no stream ships without them.