LLM streaming broken behind a proxy: answers arrive all at once or cut off mid-sentence

#Symptoms

• Calling the model API directly streams tokens as expected. Through your app's gateway, the answer appears all at once after 20-60 seconds.
• Long answers stop mid-sentence, often at a suspiciously round time (30 s, 60 s, 100 s, 120 s, 240 s).
• The browser shows net::ERR_INCOMPLETE_CHUNKED_ENCODING or the client logs "stream ended unexpectedly".
• Streaming works locally and breaks only in the deployed environment.

Most LLM APIs stream with server-sent events (SSE): one long HTTP response with Content-Type: text/event-stream, sending data: {...} chunks as tokens are produced. Anything between client and model that buffers the response or times out idle or long connections breaks it.

#Step 1: find the layer that breaks it

Test from the outside in with curl -N (no buffering in curl itself) and watch whether chunks arrive gradually:

# 1. Model endpoint directly
curl -N https://my-model.internal/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer $KEY" \
  -d '{"model":"my-model","stream":true,"messages":[{"role":"user","content":"Count from 1 to 200 slowly"}]}'

# 2. Your backend service, bypassing ingress (port-forward in Kubernetes)
kubectl port-forward svc/chat-backend 8080:80
curl -N http://localhost:8080/api/chat -d '{"q":"Count from 1 to 200 slowly"}'

# 3. Through the ingress / gateway / public URL
curl -N https://chat.example.com/api/chat -d '{"q":"Count from 1 to 200 slowly"}'

The first hop where output arrives in one block is the culprit. Add -v to see response headers; look for Content-Encoding, Content-Length (should be absent) and Transfer-Encoding: chunked or HTTP/2 framing.

#Step 2: fix by layer

#Your own backend

Before blaming the proxy, check that the app forwards chunks as it gets them:

• Python/FastAPI: return a StreamingResponse with media_type="text/event-stream"; don't collect the generator into a list.
• Disable response compression middleware for the streaming route; gzip buffers until it has enough data.
• Send Cache-Control: no-cache and, for NGINX in front, X-Accel-Buffering: no.

from fastapi.responses import StreamingResponse

@app.post("/api/chat")
async def chat(req: ChatRequest):
    async def events():
        async for chunk in llm_stream(req):          # your model call with stream=True
            yield f"data: {chunk.model_dump_json()}\n\n"
        yield "data: [DONE]\n\n"
    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

#NGINX

location /api/chat {
    proxy_pass http://chat_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;          # pass chunks through immediately
    proxy_cache off;
    gzip off;
    proxy_read_timeout 600s;      # time allowed between two reads from upstream
    proxy_send_timeout 600s;
}

proxy_read_timeout is measured between reads, so a model that "thinks" silently for a long time before the first token can hit it even with streaming on.

#ingress-nginx on Kubernetes

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"

#Azure API Management

APIM buffers responses by default. In the API's policy, forward requests without buffering for streaming operations, and don't add policies that read the response body (logging the body, set-body, content validation), because they force buffering:

<backend>
    <forward-request timeout="600" buffer-response="false" />
</backend>

Check that diagnostics/Application Insights logging for that API doesn't log response bodies. APIM's documentation has a section on streaming responses with current limitations.

#Cloud load balancers and gateways

Check each product's current limits; some timeouts have maximums you can't raise.

#Step 3: keep the connection alive

Even with buffering off, a silent gap (tool calls, long reasoning before the first token) can trip idle timeouts. Send an SSE comment line every 10-15 seconds while waiting:

yield ": keep-alive\n\n"    # SSE comment, ignored by clients

#Symptom map

#Prevent it next time

• Add a synthetic check that streams a long answer through the public URL and measures time to first byte and total duration.
• Keep streaming routes on a separate path with their own proxy settings, so you don't disable buffering site-wide.
• Document every timeout in the request path in one table; the shortest one wins.
• Log time to first token and stream completion at the backend so you can tell model slowness from proxy problems.