OCI Queue consumers at scale: visibility timeout, long polling, dead-letter queues and poison messages

#Goal

Consumers that keep up with the backlog, process each message effectively once and isolate bad messages, with nobody deleting the queue at 3 a.m. to unblock it.

OCI Queue is a pull-based, at-least-once queue. A consumer receives a message, which hides it for the visibility timeout, and must delete it with its receipt before the timeout expires. If the consumer doesn't delete it in time, the message becomes visible again and is delivered again. After a configured number of deliveries, it moves to the queue's dead-letter queue (DLQ).

Every design decision below follows from that cycle.

#The settings that matter

Minimum and maximum values for each setting, plus message size and throughput limits, are in the Queue service limits. Check the current values instead of hard-coding them.

#Create a queue with a DLQ

oci queue queue-admin queue create -c $C --display-name orders-q \
  --visibility-in-seconds 60 \
  --timeout-in-seconds 20 \
  --retention-in-seconds 345600 \
  --dlq-delivery-count 5 \
  --wait-for-state SUCCEEDED

Message operations go to the queue's messages endpoint, not the regional admin endpoint:

QID=ocid1.queue.oc1..aaaa
EP=$(oci queue queue-admin queue get --queue-id $QID \
      --query 'data."messages-endpoint"' --raw-output)
echo $EP     # e.g. https://cell-1.queue.messaging.<region>.oci.oraclecloud.com

If message commands fail against the default endpoint, pass the global --endpoint "$EP" option. SDK clients must be built with service_endpoint set to this value.

#Smoke test from the CLI

oci queue messages put-messages --queue-id $QID --endpoint "$EP" \
  --messages '[{"content":"{\"orderId\":42}"}]'

oci queue messages get-messages --queue-id $QID --endpoint "$EP" \
  --limit 10 --visibility-in-seconds 30 --timeout-in-seconds 10

oci queue messages delete-message --queue-id $QID --endpoint "$EP" \
  --message-receipt "$RECEIPT"

oci queue messages get-stats --queue-id $QID --endpoint "$EP"

get-stats reports visible and in-flight message counts for the queue and its DLQ. Those are the two numbers to put on your dashboard.

#A consumer that survives production

import json, time, threading, oci
from oci.queue.models import (DeleteMessagesDetails, DeleteMessagesDetailsEntry,
                              UpdateMessageDetails)

signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()
q = oci.queue.QueueClient(config={}, signer=signer, service_endpoint=EP)

VISIBILITY = 60          # seconds
EXTEND_EVERY = 40        # renew before expiry

def extend_lease(receipt, stop):
    while not stop.wait(EXTEND_EVERY):
        q.update_message(QID, receipt, UpdateMessageDetails(visibility_in_seconds=VISIBILITY))

while True:
    resp = q.get_messages(QID, visibility_in_seconds=VISIBILITY,
                          timeout_in_seconds=20, limit=10)
    done = []
    for m in resp.data.messages:
        stop = threading.Event()
        t = threading.Thread(target=extend_lease, args=(m.receipt, stop), daemon=True)
        t.start()
        try:
            body = json.loads(m.content)
            process(body)                    # must be idempotent
            done.append(DeleteMessagesDetailsEntry(receipt=m.receipt))
        except PermanentError:
            # Bad payload: let it reach the DLQ instead of retrying forever
            log_poison(m)
        except Exception:
            pass                             # transient: visible again after timeout
        finally:
            stop.set()
    if done:
        q.delete_messages(QID, DeleteMessagesDetails(entries=done))

Model and method names follow the OCI Python SDK oci.queue package; check your SDK version if an import fails.

What this does on purpose:

• Long polling (timeout_in_seconds=20) instead of a tight loop, which cuts empty receives and API calls.
• Lease extension with update_message, so a slow message isn't redelivered to a second consumer while the first is still working on it.
• Batch delete after processing, never before. Deleting first turns a crash into data loss.
• No catch-all delete. Transient errors make the message visible again; repeated failures move it to the DLQ.

Warning

Never set a visibility timeout shorter than your processing time "to get faster retries". Every message that takes longer is processed twice, in parallel. With non-idempotent side effects (payments, emails, inventory), that's an incident.

#Idempotency: the part the queue can't do for you

At-least-once delivery means duplicates happen even when everything is configured correctly: lease expiry during GC pauses, a crash between processing and delete, network timeouts on delete.

Put a stable business ID in the message content. Don't rely on the queue's message ID for idempotency if a producer can resend the same business event as a new message.

#Poison messages and the DLQ

A poison message fails every time: bad schema, a missing referenced record, a payload too large for a downstream API. Without a DLQ it cycles until retention expires, consuming capacity and filling logs.

• Set --dlq-delivery-count so poison messages leave the main queue after a few attempts.
• Classify errors in code: permanent (don't retry; log it, let it go to the DLQ or park it yourself) versus transient (let it be redelivered).
• Alarm on DLQ visible messages greater than 0. A DLQ that fills silently is just slower data loss, because DLQ messages also expire.

Messages in the DLQ can be consumed before the service deletes them. Inspect them in the Console or with the SDK, fix the cause, and replay them by putting their content back on the main queue. Purge only once you've captured what you need:

oci queue queue-admin queue purge --queue-id $QID --purge-type DLQ

Caution

--purge-type NORMAL or BOTH deletes every message in the main queue. There is no undo. Use DLQ unless you are certain.

#Scaling consumers

With channels, one noisy tenant can be capped with --channel-consumption-limit so it doesn't starve the others. Consumers can read a specific channel with --channel-filter.

#Prevent it next time

• Treat visibility timeout, DLQ count and retention as reviewed settings in Terraform, and remember that retention is fixed at creation.
• Alarm on main-queue visible messages (backlog age), in-flight messages and DLQ visible messages.
• Load-test with slow and failing messages, not just happy-path throughput.
• Make every consumer idempotent before it goes to production, and test it by delivering the same message twice.