LLM streaming responses arrive all at once: SSE buffered by NGINX, Application Gateway, API Management or Front Door

#Symptoms

• curl -N against the model endpoint shows tokens arriving one by one.
• Through your production URL, the browser shows nothing for 10-30 seconds, then the whole answer at once.
• Time-to-first-token looks fine in model metrics, but users complain the chat "hangs".
• Long answers sometimes fail with 504 or a cut-off response.

Some layer between the model and the browser is buffering the response: waiting for the whole body (or a big chunk) before forwarding it.

#How streaming works

Most LLM APIs stream with server-sent events (SSE): one long HTTP response with Content-Type: text/event-stream, sending lines like:

data: {"choices":[{"delta":{"content":"Hel"}}]}

data: {"choices":[{"delta":{"content":"lo"}}]}

data: [DONE]

Every proxy on the path must forward each chunk as it arrives. Buffering and compression both break that.

#Step 1: Find the buffering layer

Test each hop from the outside in, using curl -N (no buffering in curl) and watching timestamps:

time curl -sN https://chat.example.com/api/chat \
  -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" \
  -d '{"messages":[{"role":"user","content":"Count from 1 to 50 slowly"}],"stream":true}' \
  | while IFS= read -r line; do printf '%s %s\n' "$(date +%T.%3N)" "${line:0:60}"; done

Run the same against:

1. The public URL (CDN / Front Door).
2. The gateway directly (APIM / Application Gateway), bypassing the CDN.
3. The ingress / NGINX directly.
4. Your backend service (port-forward in Kubernetes).
5. The model endpoint.

The first hop where timestamps bunch together at the end is the buffering layer.

kubectl port-forward svc/chat-backend 8080:80
curl -sN http://localhost:8080/api/chat -d '...'

#Step 2: Your own backend

Before blaming proxies, make sure your app streams:

# FastAPI example: proxy the model's stream to the client without collecting it
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import OpenAI

app = FastAPI()
client = OpenAI()   # or AzureOpenAI(...)

@app.post("/api/chat")
def chat(body: dict):
    def gen():
        stream = client.chat.completions.create(model=body.get("model", "gpt-4o-mini"),
                                                messages=body["messages"], stream=True)
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                yield f"data: {chunk.choices[0].delta.content}\n\n"
        yield "data: [DONE]\n\n"
    return StreamingResponse(gen(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

Common app mistakes: collecting the whole stream into a string before returning; compression middleware (GZipMiddleware) applied to the streaming route; a framework that buffers by default.

Tip

The X-Accel-Buffering: no response header tells NGINX (including ingress-nginx) not to buffer that response, without changing global config.

#Step 3: NGINX and ingress-nginx

Plain NGINX:

location /api/chat {
    proxy_pass http://chat_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    gzip off;
    proxy_read_timeout 300s;
}

ingress-nginx annotations on the Ingress for the streaming path:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"

Check that gzip is off for text/event-stream in the controller ConfigMap if you enabled use-gzip.

#Step 4: Azure API Management

APIM can stream, but policies that read the response body force buffering. In the policy for the streaming operation:

• Remove or skip <set-body>, <find-and-replace>, or <log-to-eventhub> that reads context.Response.Body on the outbound path.
• For forward-request, set buffer-response="false":

<backend>
    <forward-request timeout="300" buffer-response="false" />
</backend>

• Azure OpenAI token-metric and token-limit policies have documented streaming behaviour; check the current APIM docs for whether your policy version supports streaming responses.
• Diagnostic settings that log response bodies can also cause buffering; set body bytes to 0 for that API.

#Step 5: Application Gateway, Front Door, CDNs

Feature names change; look for "response buffering", "compression" and "timeout" settings at each layer.

#Step 6: Keep connections alive

Long pauses (a model thinking before the first token, or tool calls in an agent) can trip idle timeouts at load balancers. Send an SSE comment line every 10-15 seconds:

yield ": keep-alive\n\n"

Clients ignore comment lines, but proxies see traffic and keep the connection open.

#Symptom → fix table

#Prevent it next time

• Put a streaming check in CI/CD smoke tests: measure time to first byte versus time to last byte through the public URL.
• Give streaming endpoints their own route with buffering, caching and compression off at every layer.
• Document each layer's timeout and set them consistently (longer than your longest expected answer).
• Monitor client-side time-to-first-token, not only server-side.