Applies toLLM applications on hosted APIs (Azure OpenAI, Bedrock, Anthropic, OpenAI, Gemini) or self-hosted models; RAG and agent workflows
Break it down
Trace each request (see the OpenTelemetry tracing post) and record:
| Segment | Typical causes of delay |
|---|---|
| Pre-processing (auth, guardrails, query rewrite) | Extra LLM calls in sequence |
| Retrieval (embedding + vector search + rerank) | Remote embedding call, large top-k, slow reranker |
| Queue / rate-limit waits | 429 retries, saturated PTU or GPU |
| Time to first token (TTFT) | Long prompts, model size, cold paths |
| Generation | Output length × per-token latency |
| Post-processing (validation, citations) | Extra LLM calls |
Fixes by segment
| Segment | Fix |
|---|---|
| Perceived wait | Stream tokens to the UI (check proxies don't buffer; see the streaming Sev1 post) |
| Sequential steps | Run guardrail checks, retrieval and query rewriting in parallel where independent |
| Retrieval | Embedding model in the same region; smaller top-k with reranking; cache embeddings of frequent queries |
| Rate-limit waits | Spread load across deployments/regions; fail over instead of long retries |
| TTFT | Shorter prompts, prompt caching (see the prompt caching post), smaller model for simple requests |
| Generation | Limit output length; structured outputs; concise instructions |
| Extra LLM steps | Merge steps into one call where quality allows, or use small fast models for them |
Parallel steps example
import asyncio
async def handle(question):
moderation, rewritten = await asyncio.gather(
check_input(question), # guardrail
rewrite_query(question), # small model
)
if not moderation.ok:
return refuse()
docs = await retrieve(rewritten)
async for token in stream_answer(question, docs):
yield token
Region and network
- Keep app, vector store, embedding and LLM endpoints in the same region where possible.
- Private endpoints add little latency; cross-region or cross-cloud hops add more.
- Reuse HTTP connections (keep-alive) in the client.
Choosing models with latency in mind
Compare candidate models on your prompts for: TTFT p95, tokens/sec, and quality. A slightly less capable model that answers in half the time may give a better user experience for chat; keep the larger model for hard requests (see the model routing post).
Set and watch SLOs
| SLI | Example SLO |
|---|---|
| TTFT p95 | < 1.5 s |
| Full response p95 | < 8 s |
| Error/timeout rate | < 1% |
Tip
Streaming plus a fast first sentence changes how users feel about speed more than shaving a second off total time. Fix streaming before anything else.
Checklist
- Latency traced per segment
- Streaming end to end
- Independent steps parallelised
- Prompts trimmed and cached; outputs limited
- Services co-located by region
- Latency SLOs defined and monitored