Home / AI & GenAI

AI & GenAIHow-toIntermediate

Optimization: lowering RAG ingestion cost - incremental indexing, deduplication and choosing when to OCR

Re-processing every document on every run wastes document parsing, OCR and embedding spend. Track content hashes to process only changed documents, skip duplicates and boilerplate, choose parsing and OCR per document type, batch embedding calls, and measure cost per document indexed.

Deepak Bhardwaj2 min read

Applies toRAG ingestion pipelines using Azure AI Document Intelligence, Amazon Textract, open-source parsers, Azure AI Search indexers, Bedrock Knowledge Bases, or custom pipelines with any vector store

Where ingestion cost comes from

Step Cost driver
Parsing/OCR Per page for cloud document services; CPU/GPU for self-hosted
Chunking Cheap
Embedding Per token (hosted) or GPU time (self-hosted)
Index writes Vector store compute and storage
Enrichment (summaries, metadata extraction with LLMs) Per token, often the most expensive

Process only what changed

import hashlib

def content_hash(raw_bytes: bytes) -> str:
    return hashlib.sha256(raw_bytes).hexdigest()

for doc in source.list_documents(modified_since=last_run):
    h = content_hash(doc.read())
    if state.get(doc.id) == h:
        continue                     # unchanged content, skip parse/embed
    chunks = parse_and_chunk(doc)
    upsert(doc.id, embed(chunks))    # replace this document's chunks only
    state[doc.id] = h

Also handle deletes: remove chunks for documents deleted at the source (see the resilient RAG pipelines post).

Chunk-level reuse

Hash each chunk's text too. When a document changes slightly, re-embed only new or changed chunks and keep the rest.

Skip waste

Waste Fix
Duplicate files in several folders Deduplicate by content hash
Boilerplate (headers, footers, disclaimers) on every page Strip before chunking
Irrelevant file types (images without text, archives) Filter early
Old versions nobody should see Index only current versions

Choose parsing per document type

Document Approach
Born-digital PDFs, Office files Text extraction (cheap); layout model only if tables matter
Scanned PDFs, images OCR / layout model
Complex tables and forms Layout model with table extraction
HTML, Markdown Direct parsing

Detect whether a PDF has a text layer before sending it to OCR.

Batch and size embedding calls

  • Send many chunks per embedding request (within limits) to reduce overhead.
  • Use batch APIs for big backfills where available.
  • Consider smaller embedding dimensions (see the vector quantization post).

Measure

Track per run: documents seen, skipped, processed; pages OCR'd; embedding tokens; cost per processed document. A full re-index should be a rare, planned event (e.g. embedding model change).

Tip

Add a "skipped unchanged" counter to the ingestion dashboard. If it's near zero on a daily run over a mostly static library, change detection isn't working.

Checklist

  • Document and chunk hashes stored
  • Only changed documents processed; deletes handled
  • Duplicates and boilerplate removed
  • OCR only where there's no text layer
  • Embedding calls batched
  • Cost per document tracked

Related fixes