Sharing one LLM tokens-per-minute quota across many app replicas: a Redis token bucket that counts tokens, not requests

#The goal

You have 20 pods sharing one deployment's tokens-per-minute (TPM) quota. Each pod retries 429s on its own, so when traffic rises they all hit the limit together, back off together and come back together. Throughput swings between full and idle, and p95 latency is mostly retry wait time.

A per-pod limiter doesn't help: 20 pods each limited to a twentieth of the quota waste capacity whenever load is uneven. You need one shared budget that every replica checks before it calls the model. This how-to builds one in Redis.

#Design decisions

Warning

This limiter is cooperative. Any client that calls the endpoint without it, such as a batch job, a notebook or another team's app, uses the same provider quota without the bucket knowing. Either put every caller behind it, or set the bucket's capacity below the quota by the amount those other callers use.

#Step 1: the Lua script

The bucket holds two fields: tokens available and last refill time in microseconds. It refills at rate tokens per second up to capacity. A request takes cost tokens if they're available. Otherwise it gets back how long to wait.

-- KEYS[1] bucket key
-- ARGV[1] capacity (tokens)  ARGV[2] refill rate (tokens/sec)
-- ARGV[3] cost (tokens; negative = refund)  ARGV[4] floor: tokens to keep for higher priority
-- ARGV[5] force: '1' = deduct even if it takes the bucket negative (used after a 429)
redis.replicate_commands()
local cap   = tonumber(ARGV[1])
local rate  = tonumber(ARGV[2])
local cost  = tonumber(ARGV[3])
local floor = tonumber(ARGV[4])
local force = ARGV[5] == '1'

local t = redis.call('TIME')
local now = tonumber(t[1]) * 1000000 + tonumber(t[2])

local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now

tokens = math.min(cap, tokens + (now - ts) / 1000000 * rate)

local allowed, wait_ms = 0, 0
if cost <= 0 then
  tokens = math.min(cap, tokens - cost)        -- refund
  allowed = 1
elseif force then
  tokens = math.max(-cap, tokens - cost)       -- debt: everyone waits until it refills
  allowed = 1
elseif tokens - cost >= floor then
  tokens = tokens - cost
  allowed = 1
else
  wait_ms = math.ceil((cost + floor - tokens) / rate * 1000)
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate * 1000) * 2)
return {allowed, wait_ms, math.floor(tokens)}

Notes:

• redis.replicate_commands() is needed on Redis versions before 5 to call TIME before a write. On Redis 5 and later, effects replication is the default and the call does nothing, so it's safe to leave in.
• floor gives you priority classes. Interactive traffic uses floor = 0. Batch uses floor = 20% of capacity, so it can never take the last fifth of the bucket.
• If a single request costs more than capacity, it can never be admitted. Reject those in the client (step 2) instead of letting them wait forever.

#Step 2: the client wrapper

import math, random, time
import redis, tiktoken

r = redis.Redis(host="redis.internal", port=6380, ssl=True, password=REDIS_KEY)
bucket = r.register_script(LUA_SOURCE)
enc = tiktoken.get_encoding("o200k_base")

QUOTA_TPM = 450_000                  # provider quota for this deployment
CAPACITY  = int(QUOTA_TPM * 0.85)    # what we allow ourselves per minute
RATE      = CAPACITY / 60.0
FLOOR     = {"interactive": 0, "batch": int(CAPACITY * 0.2)}
KEY       = "llm:tpm:prod-gpt-chat"  # one key per deployment and region

def estimate(messages, max_tokens):
    text = "".join(m["content"] for m in messages if isinstance(m.get("content"), str))
    return int(len(enc.encode(text)) * 1.1) + 10 * len(messages) + max_tokens

def acquire(cost, priority="interactive", max_wait_s=20):
    if cost > CAPACITY:
        raise ValueError(f"request needs {cost} tokens, bucket capacity is {CAPACITY}")
    deadline = time.monotonic() + max_wait_s
    while True:
        allowed, wait_ms, _ = bucket(keys=[KEY], args=[CAPACITY, RATE, cost, FLOOR[priority], 0])
        if allowed:
            return
        sleep_s = wait_ms / 1000 * random.uniform(1.0, 1.3)   # jitter avoids a thundering herd
        if time.monotonic() + sleep_s > deadline:
            raise TimeoutError("token budget not available")
        time.sleep(sleep_s)

def chat(client, model, messages, max_tokens=800, priority="interactive"):
    reserved = estimate(messages, max_tokens)
    acquire(reserved, priority)
    resp = None
    try:
        resp = client.chat.completions.create(model=model, messages=messages, max_tokens=max_tokens)
        return resp
    finally:
        used = resp.usage.total_tokens if resp is not None and resp.usage else reserved
        refund = reserved - used
        if refund > 0:
            bucket(keys=[KEY], args=[CAPACITY, RATE, -refund, 0, 0])

The estimate uses a local tokenizer with a 10% margin. It won't match the provider's count exactly, and it doesn't need to: the refund corrects the error after every call. Use the encoding that matches your model family, or a character-based estimate for models without a public tokenizer.

Tip

If the call fails with an exception, the finally block keeps the whole reservation (used = reserved). That is the safe choice: a request that timed out on your side may still have been counted by the provider.

#Step 3: keep the provider's 429 handling

The bucket reduces 429s. It doesn't remove them: the provider may also enforce a requests-per-minute limit, count tokens differently from your estimate, or have other callers. Keep a retry with backoff that respects Retry-After, and when a 429 arrives, take tokens out of the bucket so every pod slows down, not only the one that got it:

from openai import RateLimitError

def on_429(err: RateLimitError):
    retry_after = float(err.response.headers.get("retry-after", "5"))
    drain = int(RATE * retry_after)
    bucket(keys=[KEY], args=[CAPACITY, RATE, drain, 0, 1])   # force: may push the bucket negative
    time.sleep(retry_after * random.uniform(1.0, 1.3))

#Step 4: watch it

Log one line per acquire with the wait time, reserved tokens, actual tokens and priority. Then track:

redis-cli -h redis.internal -p 6380 --tls -a "$REDIS_KEY" HGETALL llm:tpm:prod-gpt-chat

Compare with the provider's view. For Azure OpenAI, look at the deployment's token and throttling metrics in Azure Monitor. For Bedrock, check the CloudWatch InvocationThrottles metric for the model.

#Failure modes

Caution

Failing open without a fallback limiter means that a Redis outage lets all 20 pods hit the provider at full speed. The 429 storm then lands on the same moment you are trying to fix Redis. Test that failover path on purpose.

#Prevent it next time

• Put every caller of a deployment behind the same bucket key, including batch jobs and evaluation runs.
• Set max_tokens from measured output lengths per use case, not a global default.
• Review quota, capacity factor and floor settings whenever the provider quota changes, and keep them in config, not code.