Bedrock Knowledge Base sync fails or documents go missing: ingestion statistics, failure reasons and file-level logs

#Symptom

One of these:

• The data source sync shows FAILED.
• The sync shows COMPLETE, but a file you uploaded never appears in answers.
• The assistant still quotes a document you replaced or deleted.

A COMPLETE job doesn't mean every file was indexed. Individual files can fail or be ignored while the job itself finishes, so check the job, then the files.

#Check 1: the latest job

export KB=KB12345678 DS=DS12345678

aws bedrock-agent list-ingestion-jobs \
  --knowledge-base-id $KB --data-source-id $DS \
  --sort-by attribute=STARTED_AT,order=DESCENDING --max-items 5 \
  --query 'ingestionJobSummaries[].{id:ingestionJobId,status:status,started:startedAt,stats:statistics}'

aws bedrock-agent get-ingestion-job \
  --knowledge-base-id $KB --data-source-id $DS --ingestion-job-id <job-id> \
  --query 'ingestionJob.{status:status,reasons:failureReasons,stats:statistics}'

Read the statistics like this:

In the console, open the data source, then Sync history and View warnings to see the same reasons.

#Check 2: which files are affected

List everything that isn't fully indexed:

aws bedrock-agent list-knowledge-base-documents \
  --knowledge-base-id $KB --data-source-id $DS \
  --query "documentDetails[?status!='INDEXED'].{uri:identifier.s3.uri,status:status,reason:statusReason}" \
  --output table

Check one specific file:

aws bedrock-agent get-knowledge-base-documents \
  --knowledge-base-id $KB --data-source-id $DS \
  --document-identifiers 'dataSourceType=S3,s3={uri=s3://my-kb-bucket/policies/leave.pdf}' \
  --query 'documentDetails[].{status:status,reason:statusReason,updated:updatedAt}'

Statuses you'll see most often: INDEXED, PARTIALLY_INDEXED, FAILED, IGNORED, METADATA_PARTIALLY_INDEXED, METADATA_UPDATE_FAILED. NOT_FOUND means the knowledge base has no record of that file. statusReason is documented as appearing with IGNORED, so for other failures use the logs in Check 4.

#Check 3: compare S3 with the index

This finds files in the bucket that the knowledge base doesn't know about. It needs boto3, s3:ListBucket and bedrock:ListKnowledgeBaseDocuments.

import boto3

KB, DS = "KB12345678", "DS12345678"
BUCKET, PREFIX = "my-kb-bucket", "policies/"

s3 = boto3.client("s3")
agent = boto3.client("bedrock-agent")

in_s3 = set()
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=PREFIX):
    for obj in page.get("Contents", []):
        key = obj["Key"]
        if not key.endswith((".metadata.json", "/")):
            in_s3.add(f"s3://{BUCKET}/{key}")

status = {}
pages = agent.get_paginator("list_knowledge_base_documents").paginate(
    knowledgeBaseId=KB, dataSourceId=DS)
for page in pages:
    for doc in page["documentDetails"]:
        status[doc["identifier"]["s3"]["uri"]] = doc["status"]

for uri in sorted(in_s3 - status.keys()):
    print("NOT IN KB   ", uri)
for uri, st in sorted(status.items()):
    if st != "INDEXED":
        print(f"{st:<12}", uri)

Then confirm what retrieval returns, independent of the model and your app:

aws bedrock-agent-runtime retrieve --knowledge-base-id $KB \
  --retrieval-query text="annual leave carry-over" \
  --query 'retrievalResults[].{score:score,uri:location.s3Location.uri}'

If the file is INDEXED but never returned, this is a retrieval or chunking problem, not an ingestion one. See the RAG tuning posts instead.

#Check 4: turn on file-level logs

Knowledge base application logs record each file's status and the reason it failed. Set them up before you need them, because they only cover jobs that run after delivery is active.

ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
REGION=us-east-1
LOG_GROUP=/aws/vendedlogs/bedrock/kb-$KB

aws logs create-log-group --log-group-name $LOG_GROUP

aws logs put-delivery-source --name kb-$KB-src \
  --log-type APPLICATION_LOGS \
  --resource-arn arn:aws:bedrock:$REGION:$ACCOUNT:knowledge-base/$KB

DEST_ARN=$(aws logs put-delivery-destination --name kb-$KB-dst \
  --output-format json \
  --delivery-destination-configuration destinationResourceArn=arn:aws:logs:$REGION:$ACCOUNT:log-group:$LOG_GROUP \
  --query 'deliveryDestination.arn' --output text)

aws logs create-delivery --delivery-source-name kb-$KB-src \
  --delivery-destination-arn $DEST_ARN

Your identity needs bedrock:AllowVendedLogDeliveryForResource and the logs: delivery permissions. Re-run the sync, then use this in CloudWatch Logs Insights:

fields @timestamp, event.document_location.s3_location.uri as file,
       event.status as status, event.status_reasons.0 as reason
| filter level in ["ERROR", "WARN"]
    or event.status in ["RESOURCE_IGNORED", "EMBEDDING_FAILED", "INDEXING_FAILED"]
| sort @timestamp desc
| limit 200

#Common causes and fixes

Supported text formats include .txt, .md, .html, .doc/.docx, .csv, .xls/.xlsx and .pdf, and text must be UTF-8 for .txt, .md and .html. The list and size limits change, so check the current Bedrock docs and quotas:

aws service-quotas list-service-quotas --service-code bedrock \
  --query "Quotas[?contains(QuotaName, 'ngestion')].{name:QuotaName,value:Value}" --output table

#Checking the role and source

aws bedrock-agent get-knowledge-base --knowledge-base-id $KB \
  --query 'knowledgeBase.{status:status,role:roleArn,store:storageConfiguration.type,model:knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration.embeddingModelArn}'

aws bedrock-agent get-data-source --knowledge-base-id $KB --data-source-id $DS \
  --query 'dataSource.{status:status,s3:dataSourceConfiguration.s3Configuration}'

aws iam simulate-principal-policy --policy-source-arn <role-arn> \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::my-kb-bucket/policies/leave.pdf \
  --query 'EvaluationResults[].{action:EvalActionName,decision:EvalDecision}'

The simulator only evaluates IAM policies you can pass to it, so an allowed result doesn't rule out a bucket policy or KMS key policy problem.

#Vector store permissions

#Re-run the sync

Check that nothing is already running, then start one job:

aws bedrock-agent list-ingestion-jobs --knowledge-base-id $KB --data-source-id $DS \
  --filters attribute=STATUS,operator=EQ,values=STARTING,IN_PROGRESS,STOPPING \
  --query 'ingestionJobSummaries[].ingestionJobId'

aws bedrock-agent start-ingestion-job --knowledge-base-id $KB --data-source-id $DS \
  --query 'ingestionJob.ingestionJobId' --output text

Caution

A sync removes files from the index when they've left S3. If an upstream copy job emptied or partly wrote the prefix, syncing takes those documents out of production answers. Compare the object count with what you expect before you sync, and stop a bad run with aws bedrock-agent stop-ingestion-job if needed.

Warning

Don't run IngestKnowledgeBaseDocuments and StartIngestionJob at the same time. Documents added directly to an S3 data source also aren't written to S3, so keep S3 as the single way in unless you have a reason not to.

#Prevent it next time

• Enable application logs on every knowledge base when you create it.
• After each scheduled sync, read numberOfDocumentsFailed and alert if it's above zero. Don't alert on job status alone.
• Run the S3-to-index comparison daily and alert on files missing from the index.
• Add a guard to the sync job that skips the run if the object count dropped sharply since the last one.
• Keep one known test document in the source, and run a retrieve for it after each sync.