Hugging Face Text Embeddings Inference (TEI) cheat sheet: images, launch flags, batching limits, API calls and metrics

Text Embeddings Inference (TEI) is the usual choice when you self-host embedding or cross-encoder reranker models. It is simpler than a general LLM server, but its batching limits and truncation rules decide whether an ingestion job runs at full speed or fails with 413s and 422s. This sheet is for engineers who already run it and want the exact knobs.

#Pick the right image

TEI ships one image per GPU architecture. A wrong tag either fails at startup or falls back to slower kernels.

Volta GPUs are not supported. Check the README for the release you deploy; tags change with each version.

Find the compute capability of a node before choosing:

nvidia-smi --query-gpu=name,compute_cap --format=csv

#Launch

docker run --gpus all -p 8080:80 \
  -v /data/hf-cache:/data \
  -e HF_TOKEN=$HF_TOKEN \
  ghcr.io/huggingface/text-embeddings-inference:89-1.9 \
  --model-id BAAI/bge-m3 \
  --revision <commit-sha> \
  --dtype float16 \
  --max-batch-tokens 32768 \
  --max-client-batch-size 64 \
  --max-concurrent-requests 512 \
  --api-key "$TEI_API_KEY" \
  --json-output

Tip

Pin --revision to a commit SHA. A floating main revision can change the model weights under you, and vectors produced by different revisions should not share one index.

#Flags that matter

Warning

The documented default for --auto-truncate has differed between the help text and actual behaviour across releases. Set it explicitly. If it is off, TEI refuses to start when the model's max input length exceeds --max-batch-tokens; if it is on, the max input length is cut down to --max-batch-tokens.

#Batching: how the limits interact

• One client request with N inputs must have N ≤ --max-client-batch-size, or TEI rejects it.
• The body must fit in --payload-limit. Long chunks times 64 inputs can pass the first limit and fail this one.
• The batcher fills each forward pass up to --max-batch-tokens across all queued requests. Bigger values raise throughput on GPUs with spare memory and raise per-request latency.
• --max-concurrent-requests is your back-pressure point. Beyond it, clients get an overload error and should back off, not retry immediately.

A sizing rule that holds up: set --max-batch-tokens to at least max_seq_len × 4 for the model, then raise it while watching GPU memory and p99 latency. For a query-time service, keep it lower than for an ingestion service. Running two TEI deployments of the same model (one for queries, one for bulk ingestion) is often cleaner than one compromise setting.

#API calls

H="Authorization: Bearer $TEI_API_KEY"

# Health and model info
curl -s localhost:8080/health -H "$H" -o /dev/null -w "%{http_code}\n"
curl -s localhost:8080/info -H "$H" | jq

# Dense embeddings
curl -s localhost:8080/embed -H "$H" -H 'Content-Type: application/json' \
  -d '{"inputs": ["reset the admin password", "rotate the key"], "truncate": true}' | jq '.[0] | length'

# OpenAI-compatible route
curl -s localhost:8080/v1/embeddings -H "$H" -H 'Content-Type: application/json' \
  -d '{"model": "bge-m3", "input": ["reset the admin password"]}' | jq '.data[0].embedding | length'

# Reranker (cross-encoder model loaded)
curl -s localhost:8080/rerank -H "$H" -H 'Content-Type: application/json' \
  -d '{"query": "how do I rotate keys", "texts": ["Key rotation runbook", "Holiday calendar"]}' | jq

# Sparse embeddings (SPLADE pooling)
curl -s localhost:8080/embed_sparse -H "$H" -H 'Content-Type: application/json' \
  -d '{"inputs": "rotate the key"}'

The full request schema, including fields such as normalize and truncation direction, is on the running server's OpenAPI page at /docs. Use it rather than guessing, because fields were added across 1.x releases.

Check that /info reports the model, revision and max input length you expect after every rollout. It is the fastest way to catch a deployment that pulled the wrong model.

#Python client for bulk jobs

import os, requests

URL = "http://tei:8080/embed"
HEAD = {"Authorization": f"Bearer {os.environ['TEI_API_KEY']}"}
BATCH = 32  # must not exceed --max-client-batch-size

def embed_all(texts):
    out = []
    for i in range(0, len(texts), BATCH):
        r = requests.post(URL, headers=HEAD,
                          json={"inputs": texts[i:i + BATCH], "truncate": True},
                          timeout=60)
        if r.status_code in (413, 422, 429):
            raise RuntimeError(f"{r.status_code}: {r.text[:300]}")
        r.raise_for_status()
        out.extend(r.json())
    return out

#Errors and what they mean

#Metrics

Prometheus metrics are on /metrics (on the API port for HTTP servers, or --prometheus-port). Metric names vary by version; list them:

curl -s localhost:8080/metrics | grep -E '^te_' | cut -d'{' -f1 | sort -u

Alert on queue size growing for more than a few minutes, p99 inference time, and the rate of non-2xx responses. Pair them with GPU utilisation and memory from DCGM Exporter so you can tell "batch too small" from "GPU saturated".

histogram_quantile(0.99, sum by (le) (rate(te_request_duration_bucket[5m])))

Confirm the histogram name in your build's /metrics output before using the query.

#Prevent it next time

• Pin image tag and model revision together, and record both in the index metadata.
• Set truncation explicitly in both the server flags and client requests.
• Size client batches from /info at startup instead of hard-coding them.
• Separate query and ingestion deployments so a backfill cannot push query latency up.