Weaviate batch imports silently lose objects: reading batch errors, vectorizer module rate limits and timeouts

#Symptoms

• The import job ends without an exception, but the object count is lower than the source.
• Logs show warnings about failed objects that nobody read.
• Errors like 429 Too Many Requests, context deadline exceeded, maximum context length coming from the vectorizer.
• The import slows down over time and Weaviate CPU is low (waiting on the embedding API).

#Why objects go missing

A batch request succeeds as a whole even if individual objects in it fail. Each failed object is reported in the response, and the Python client v4 collects them for you, but doesn't raise. If the code doesn't check, failures are silent.

With a vectorizer module, Weaviate calls the embedding provider for every object during import. That call is the most common point of failure.

#Step 1: check the batch errors

import weaviate
client = weaviate.connect_to_custom(...)
kb = client.collections.get("KbChunk")

with kb.batch.fixed_size(batch_size=100, concurrent_requests=2) as batch:
    for c in chunks:
        batch.add_object(properties={"doc_id": c.doc_id, "content": c.text}, uuid=c.uuid)
        if batch.number_errors > 50:
            print("too many errors, stopping")
            break

failed = kb.batch.failed_objects
print(len(failed))
for f in failed[:5]:
    print(f.message, f.object_.uuid)

Group the messages to see the main cause:

from collections import Counter
print(Counter(f.message[:80] for f in failed).most_common(10))

#Step 2: match the error to a cause

#Step 3: rate-limit the batch

The client can pace requests for you:

with kb.batch.rate_limit(requests_per_minute=3000) as batch:
    for c in chunks:
        batch.add_object(properties={...}, uuid=c.uuid)

requests_per_minute here means objects per minute sent for vectorisation. Set it below your provider quota, leaving room for live query traffic (query-time near_text also calls the vectorizer).

Other options:

#Step 4: or bring your own vectors

For large imports, embedding outside Weaviate is often more robust:

with kb.batch.fixed_size(batch_size=200) as batch:
    for c in chunks:
        batch.add_object(properties={...}, vector=c.embedding, uuid=c.uuid)

• You control retries, caching and parallelism against the embedding API.
• Re-imports don't re-embed unchanged chunks if you cache embeddings by content hash.
• Keep the collection's vectorizer configured with the same model, so near_text queries embed with the same model as your documents.

#Step 5: re-import failed objects safely

Use deterministic UUIDs so retrying is an upsert, not a duplicate:

from weaviate.util import generate_uuid5
uuid = generate_uuid5(f"{doc_id}#{chunk_no}")

Retry only failures:

retry_ids = {str(f.object_.uuid) for f in failed}
to_retry = [c for c in chunks if str(c.uuid) in retry_ids]
with kb.batch.rate_limit(requests_per_minute=1000) as batch:
    for c in to_retry:
        batch.add_object(properties={...}, uuid=c.uuid)

#Step 6: verify counts

total = kb.aggregate.over_all(total_count=True).total_count
print(total, len(source_chunks))

For multi-tenant collections, count per tenant with kb.with_tenant(t).aggregate.over_all(...).

Write the count comparison into the job: fail the job if the difference is above a small tolerance.

#Step 7: timeouts on the client side

Large batches with slow vectorizers can exceed client timeouts:

from weaviate.classes.init import AdditionalConfig, Timeout
client = weaviate.connect_to_custom(
    ..., additional_config=AdditionalConfig(timeout=Timeout(init=30, query=60, insert=300)),
)

Raise insert timeout only together with smaller batches; a 5-minute batch is still a batch that will fail sometimes.

#Decision table

#Prevent it next time

• Always check failed_objects and fail the job on errors above a threshold.
• Deterministic UUIDs for every object.
• Enforce maximum chunk tokens before import.
• Compare source and Weaviate counts at the end of every import.
• Consider embedding outside Weaviate for large or frequent imports.