Sev1: chat answers arrive all at once or get cut off. Fixing LLM streaming behind proxies and gateways

#Symptoms

#Step 1: test each hop

# Direct to the model endpoint (bypass everything)
curl -N -s "https://aoai-prod.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21" \
  -H "api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"stream":true,"messages":[{"role":"user","content":"Count from 1 to 200 slowly"}]}' | head -20

# Through your gateway / app URL
curl -N -s https://chat.example.com/api/chat -H "Content-Type: application/json" -d '{"message":"Count from 1 to 200"}'

-N disables curl's own buffering. Watch whether chunks arrive progressively. Repeat at each hop (backend pod, ingress, gateway, CDN) to find where it breaks.

#Step 2: fix by component

#NGINX / NGINX Ingress

location /api/chat {
    proxy_pass http://chat-backend;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 300s;
    proxy_http_version 1.1;
    chunked_transfer_encoding on;
}

Ingress annotations: nginx.ingress.kubernetes.io/proxy-buffering: "off", nginx.ingress.kubernetes.io/proxy-read-timeout: "300".

The backend can also send X-Accel-Buffering: no to disable buffering per response.

#Azure API Management

• Make sure no policy reads the response body (context.Response.Body.As<string>(), logging of response bodies, set-body on the response). Those force buffering.
• In diagnostics/logging settings, don't log response bodies for streaming APIs.
• Check forward-request timeout covers long answers.

Check the APIM documentation on SSE support for your tier for current guidance.

#Azure Application Gateway / Front Door

• Application Gateway: raise the backend request timeout in backend settings (default is often 20–30 s).
• Front Door: origin response timeout; check that caching and compression aren't applied to the streaming route.

#AWS ALB / CloudFront

• ALB: idle timeout (default 60 s) must exceed the longest gap between tokens and ideally the whole response.
• CloudFront: origin response timeout / keep-alive; disable caching for the route.

#Corporate proxies and antivirus

Some inspect and buffer responses. If only some users are affected, compare network paths. A proxy bypass for the chat domain or a WebSocket alternative may be needed.

#Step 3: backend code

• Flush after each chunk (framework-specific; e.g. FastAPI StreamingResponse with an async generator, Express res.write without compression middleware).
• Disable response compression on the streaming route (gzip waits for data to compress).
• Send heartbeats (an SSE comment line : keep-alive) every 10–15 s during long tool calls, so idle timeouts don't fire.

async def event_stream():
    async for chunk in llm_stream(messages):
        yield f"data: {json.dumps({'delta': chunk})}\n\n"
    yield "data: [DONE]\n\n"

return StreamingResponse(event_stream(), media_type="text/event-stream",
                         headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

Tip

Put a streaming check in your synthetic monitoring: measure time to first byte through the public URL. Buffering regressions after an infrastructure change show up as a jump from 1 s to 20+ s.

#Prevent it next time

• Document every hop and its timeout/buffering setting for the chat route.
• Infrastructure changes to gateways and ingresses tested with a streaming request.
• Heartbeats for long-running agent responses.