Load testing an LLM endpoint before go-live: TTFT, tokens per second, concurrency and where 429s start

#Goal

Before an LLM feature goes live, you need numbers for four questions:

1. How fast does the first token arrive (TTFT) at expected load?
2. How many output tokens per second does each user get, and how many does the endpoint deliver in total?
3. At what concurrency does p95 latency break the SLO?
4. At what load do 429s or 5xx errors start, and how does the client handle them?

A generic HTTP load test won't tell you this. LLM latency depends on input and output length, and the result isn't finished until the stream ends. The test below streams real requests and measures each stage.

#Before you start

• Use production-shaped prompts. Take 50–200 real prompts (with PII removed), or match their token lengths. A test of 50-token prompts says nothing about a RAG app that sends 6,000.
• Fix the output length. Set max_tokens to what production uses. Output length dominates end-to-end time.
• Test the real path. Go through the same gateway, private endpoint and proxy that production uses. Proxies that buffer streams make TTFT look like total latency.
• Agree on the SLO first, for example "p95 TTFT under 2 s and p95 end-to-end under 15 s at 40 concurrent users, error rate under 1%".

Caution

Load tests against a shared Azure OpenAI deployment or a shared quota use up the same TPM as production traffic and can trigger 429s for real users. Use a separate deployment or run outside business hours, and tell the owners of the shared quota first. Tests also cost real money. Estimate tokens × requests before you start.

#Step 1: a streaming load test script

This script runs a fixed number of requests at a given concurrency and records TTFT, end-to-end time, output tokens and errors for each request. It works with any OpenAI-compatible endpoint.

# llm_load.py
import asyncio, json, random, statistics, sys, time
from openai import AsyncOpenAI, APIStatusError

BASE_URL = sys.argv[1]           # e.g. http://vllm:8000/v1
MODEL = sys.argv[2]              # model or deployment name
CONCURRENCY = int(sys.argv[3])
TOTAL = int(sys.argv[4])
PROMPTS = [json.loads(l)["prompt"] for l in open("prompts.jsonl")]

client = AsyncOpenAI(base_url=BASE_URL, api_key="<key>", max_retries=0, timeout=120)

async def one():
    t0 = time.perf_counter()
    ttft, out_tokens = None, 0
    try:
        stream = await client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": random.choice(PROMPTS)}],
            max_tokens=400,
            stream=True,
            stream_options={"include_usage": True},
        )
        async for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content and ttft is None:
                ttft = time.perf_counter() - t0
            if chunk.usage:
                out_tokens = chunk.usage.completion_tokens
        return {"ok": True, "ttft": ttft, "e2e": time.perf_counter() - t0, "out": out_tokens}
    except APIStatusError as e:
        return {"ok": False, "status": e.status_code}
    except Exception as e:
        return {"ok": False, "status": type(e).__name__}

async def main():
    sem = asyncio.Semaphore(CONCURRENCY)
    async def guarded():
        async with sem:
            return await one()
    start = time.perf_counter()
    results = await asyncio.gather(*[guarded() for _ in range(TOTAL)])
    wall = time.perf_counter() - start

    ok = [r for r in results if r["ok"] and r["ttft"]]
    errors = {}
    for r in results:
        if not r["ok"]:
            errors[r["status"]] = errors.get(r["status"], 0) + 1
    def p(vals, q):
        return statistics.quantiles(vals, n=100)[q - 1] if len(vals) > 1 else float("nan")
    ttft = [r["ttft"] for r in ok]
    e2e = [r["e2e"] for r in ok]
    tps_user = [r["out"] / (r["e2e"] - r["ttft"]) for r in ok if r["e2e"] > r["ttft"] and r["out"]]
    print(json.dumps({
        "concurrency": CONCURRENCY, "requests": TOTAL, "ok": len(ok), "errors": errors,
        "ttft_p50": p(ttft, 50), "ttft_p95": p(ttft, 95),
        "e2e_p50": p(e2e, 50), "e2e_p95": p(e2e, 95),
        "tokens_per_s_per_user_p50": p(tps_user, 50),
        "total_output_tokens_per_s": sum(r["out"] for r in ok) / wall,
        "requests_per_s": len(ok) / wall,
    }, indent=2))

asyncio.run(main())

max_retries=0 is deliberate. You want to see every 429 and timeout, not have the SDK hide them by retrying. For Azure OpenAI, use the /openai/v1/ base URL of your resource (https://<resource>.openai.azure.com/openai/v1/) with the deployment name as model, or switch to AsyncAzureOpenAI if your setup still uses dated API versions.

#Step 2: run a concurrency sweep

Run the same test at increasing concurrency, with enough requests per step for stable percentiles (at least 5× the concurrency, and 100 or more):

for c in 1 4 8 16 32 64; do
  python llm_load.py http://vllm:8000/v1 my-model $c $((c * 10 > 100 ? c * 10 : 100)) \
    | tee "result-c$c.json"
done

Put the results in a table:

#Reading the curve

Your usable capacity is the highest concurrency where every SLO still holds. Plan production at a margin below it, not at it.

#Step 3: on self-hosted vLLM, use the built-in benchmark too

Recent vLLM releases include a serving benchmark that reports TTFT, time per output token and inter-token latency. Flags vary by version; check vllm bench serve --help on your build (older releases ship benchmarks/benchmark_serving.py instead).

vllm bench serve \
  --backend vllm \
  --base-url http://vllm:8000 \
  --model <served-model-name> \
  --dataset-name random \
  --random-input-len 2000 --random-output-len 400 \
  --num-prompts 500 \
  --request-rate inf \
  --max-concurrency 32 \
  --save-result

--max-concurrency caps how many requests are in flight at once and --request-rate controls how fast they arrive. Sweeping concurrency with --request-rate inf gives you the same curve as Step 2 with server-side detail.

While the test runs, watch the server's queue and cache metrics:

curl -s http://vllm:8000/metrics | grep -E \
  "vllm:num_requests_running|vllm:num_requests_waiting|vllm:(gpu_)?kv_cache_usage_perc|vllm:num_preemptions"

Metric names vary by vLLM version (gpu_cache_usage_perc was renamed kv_cache_usage_perc in newer releases). Rising num_requests_waiting with KV cache usage near 100% means the KV cache is the limit. A growing preemption count means requests are being evicted and recomputed, which shows up as TTFT spikes.

#Step 4: on hosted endpoints, find the quota edge

On Azure OpenAI, OpenAI and Bedrock, the limit you usually hit first is quota (TPM/RPM), not model speed. Record:

• The concurrency where the first 429 appears, and the retry-after header values.
• Whether 429s come from the model endpoint or from your gateway (API Management returns its own 429 for its rate-limit policies).
• Latency at 70–80% of quota, which is where you should run in production.

Check the deployment's quota while you test:

az cognitiveservices account deployment show \
  -g <rg> -n <aoai-name> --deployment-name <deployment> \
  --query "{sku:sku.name, capacity:sku.capacity, limits:properties.rateLimits}" -o json

Then repeat the sweep with your production client settings (SDK retries on, real backoff) to confirm that retries recover from short 429 bursts without making them worse.

Warning

Clients that retry 429s immediately with no backoff, at high concurrency, turn a brief limit into a sustained outage. If error rates climb sharply with retries on, fix the backoff and jitter before go-live.

#Prevent it next time

• Keep prompts.jsonl and the sweep script in the repo, and rerun them after every model, prompt-length, quota or serving-config change.
• Store each sweep's results with the date, model version and config so you can compare runs.
• Alert in production on the metrics you measured here: p95 TTFT, queue depth, KV cache usage and 429 rate.
• Re-test before seasonal peaks and after moving regions or deployment types. Throughput per unit of quota differs between them, so check the current limits.
• Treat the capacity number as an input to autoscaling and quota requests, not as a one-off report.