Embedding ingestion job fails: input too long, batch token limits, 429s and empty vectors when indexing thousands of documents

#Symptoms

• This model's maximum context length is 8192 tokens, however you requested 9541 tokens (or a provider's equivalent "input too long" / ValidationException).
• Too many inputs / 'input' array must have less than N items errors.
• 429 Too Many Requests after a few minutes, then the job dies.
• The job "succeeds", but search quality is poor: some vectors are all zeros, duplicated, or attached to the wrong chunk.
• Re-running the job duplicates documents in the vector store.

#Step 1: Know your model's limits

Every embedding model has:

Tip

Self-hosted models often truncate silently at their max length (512 tokens is common). Your 2,000-token chunks get embedded as their first 512 tokens, and nobody notices until search misses content at the end of chunks.

#Step 2: Count tokens with the model's tokenizer

Characters aren't tokens. Count with the right tokenizer:

# OpenAI / Azure OpenAI embedding models
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")       # check the encoding for your model
def n_tokens(text: str) -> int:
    return len(enc.encode(text))

# Hugging Face models (self-hosted)
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("BAAI/bge-base-en-v1.5")
def n_tokens_hf(text: str) -> int:
    return len(tok.encode(text, add_special_tokens=True))

Find the offenders in your chunk set:

too_long = [(c["id"], n_tokens(c["text"])) for c in chunks if n_tokens(c["text"]) > MAX_TOKENS]
print(len(too_long), too_long[:5])

Usual causes: tables or code blocks that the splitter didn't break, PDFs extracted without line breaks, base64 images embedded in HTML, or a splitter configured in characters while you think in tokens.

#Step 3: Fix chunking, don't just truncate

def split_by_tokens(text: str, max_tokens: int = 500, overlap: int = 50):
    ids = enc.encode(text)
    step = max_tokens - overlap
    for start in range(0, len(ids), step):
        yield enc.decode(ids[start:start + max_tokens])

Use it as a safety net after your structural splitter (headings, paragraphs). Keep a margin below the model limit (for example 90%) for special tokens and tokenizer differences.

#Step 4: Batch by count and tokens

def batches(items, max_items=64, max_tokens=60000):
    batch, tokens = [], 0
    for it in items:
        t = it["tokens"]
        if batch and (len(batch) >= max_items or tokens + t > max_tokens):
            yield batch
            batch, tokens = [], 0
        batch.append(it)
        tokens += t
    if batch:
        yield batch

Set max_items and max_tokens from your provider's documented limits, below the limit.

#Step 5: Retry 429s and transient errors properly

import time, random

def embed_with_retry(client, model, texts, max_attempts=8):
    for attempt in range(max_attempts):
        try:
            resp = client.embeddings.create(model=model, input=texts)
            return [d.embedding for d in sorted(resp.data, key=lambda d: d.index)]
        except Exception as e:
            status = getattr(e, "status_code", None) or getattr(getattr(e, "response", None), "status_code", None)
            if status in (429, 500, 502, 503, 504) and attempt < max_attempts - 1:
                retry_after = None
                hdrs = getattr(getattr(e, "response", None), "headers", {}) or {}
                if "retry-after" in hdrs:
                    retry_after = float(hdrs["retry-after"])
                time.sleep(retry_after or min(60, 2 ** attempt) + random.random())
                continue
            raise

Note the sort by index: always map embeddings back to inputs by the index the API returns, not by assuming order.

Throughput tips:

• Run a few workers in parallel but stay under the TPM limit; a token-bucket limiter is better than hammering until 429.
• For very large backfills, use the provider's batch API where available (cheaper, higher limits, slower turnaround).
• Spread load across deployments/regions only if your quotas allow it and the model is identical.

#Step 6: Validate before you upsert

Bad vectors are worse than failed ones, because they fail silently at query time.

import math

def validate(vecs, expected_dim):
    for i, v in enumerate(vecs):
        if len(v) != expected_dim:
            raise ValueError(f"vector {i}: dim {len(v)} != {expected_dim}")
        norm = math.sqrt(sum(x * x for x in v))
        if norm == 0 or math.isnan(norm):
            raise ValueError(f"vector {i}: zero or NaN norm")

Common sources of bad vectors: empty strings (some APIs return an error, some return a vector of an empty input), whitespace-only chunks, and mixing two models with the same dimension.

chunks = [c for c in chunks if c["text"].strip()]

#Step 7: Make re-runs idempotent

Use a deterministic ID per chunk and upsert, so re-running replaces rather than duplicates:

import hashlib

def chunk_id(doc_id: str, chunk_index: int, model: str) -> str:
    return hashlib.sha256(f"{doc_id}|{chunk_index}|{model}".encode()).hexdigest()

Store doc_id, chunk_index, source URL, document version/hash and embedding model name as metadata. Then:

• Skip chunks whose content hash hasn't changed since the last run (saves cost).
• Delete chunks for a document before re-inserting a new version that has fewer chunks.
• Never mix vectors from different models in one index.

#Step 8: Checkpoint progress

A 6-hour job that restarts from zero on failure wastes money. Record completed batches (doc IDs or a high-water mark) in a small table or file, and resume from there.

#Symptom → fix table

#Prevent it next time

• Store chunk token counts in your pipeline and alert when any exceed the model limit.
• Validate dimensions and norms on every batch before writing.
• Keep the embedding model name in metadata and in the index name.
• Load-test ingestion against your real quota before the big backfill.