Milvus deletes don't free space and searches slow down: segments, delete logs, compaction and garbage collection

#Symptoms

• You deleted half a collection, but get_collection_stats row count barely changed.
• MinIO/S3 usage grows every week although data volume is stable.
• Query node memory stays high after large deletes.
• Searches and filtered queries are slower than before a bulk upsert.

#How Milvus stores and deletes data

• Data is written into segments. Growing segments accept inserts; when full or flushed they become sealed and get indexed.
• A delete doesn't rewrite the segment. It writes a delete log (a tombstone). Searches must check every result against these delete logs.
• An upsert is a delete plus an insert, so heavy upserts create tombstones too.
• Compaction rewrites segments without deleted rows and merges small segments. Only then is space and memory freed.
• Garbage collection later removes the old segment files from object storage, after a safety delay.

So: delete → tombstone (instant, logical) → compaction (later, physical) → GC (later still, storage).

#Step 1: confirm what you're looking at

from pymilvus import MilvusClient
client = MilvusClient(uri="http://milvus:19530", token="root:<password>")

client.get_collection_stats(collection_name="kb_chunks")      # row_count may include deleted rows until compaction
client.query(collection_name="kb_chunks", filter="", output_fields=["count(*)"])   # live count, excludes deleted

The count(*) query is the number of visible entities. If it's right and stats are higher, deletes worked and compaction hasn't caught up yet.

#Step 2: look at segments

Attu (Milvus web UI) shows segments per collection with state, row counts and size. With pymilvus' ORM utility:

from pymilvus import connections, utility
connections.connect(uri="http://milvus:19530", token="root:<password>")
for s in utility.get_query_segment_info("kb_chunks"):
    print(s.segmentID, s.num_rows, s.state, s.index_name)

Warning signs:

#Step 3: trigger and watch compaction

Milvus compacts automatically (mix compaction when deleted ratio passes a threshold, merge of small segments, and periodic clustering compaction if configured). After a large planned delete, trigger it:

job_id = client.compact(collection_name="kb_chunks")
print(client.get_compaction_state(job_id=job_id))      # Executing → Completed

Compaction needs data node (or streaming/compaction node, depending on version) CPU and object storage bandwidth. On a busy cluster it can take hours; run it off-peak.

Relevant settings in milvus.yaml (names from the default config; verify in your version):

If auto compaction was disabled during a bulk load and never re-enabled, that's your cause.

#Step 4: wait for garbage collection

After compaction, old segment files stay in object storage until GC removes them (by default after a tolerance period, often around a day). Storage dropping a day or so after compaction is normal. If it never drops:

• Check data coordinator logs for GC errors (permissions to delete objects in the bucket).
• Check that the backup tool or another process isn't holding the same prefix.
• Bucket versioning keeps deleted objects as old versions; add a lifecycle rule to expire non-current versions.

kubectl logs -n milvus <datacoord-or-mixcoord-pod> | grep -iE "gc|garbage|remove" | tail -20

#Step 5: fix the write pattern

For full refreshes, the alias switch is far cheaper than delete + insert: dropping a collection removes its segments without compaction.

client.alter_alias(collection_name="kb_chunks_20261028", alias="kb_chunks_live")
client.drop_collection(collection_name="kb_chunks_20261027")

#Step 6: check search after cleanup

After compaction completes:

• Segment count should drop and segments should be larger.
• Search latency should return to baseline.
• Query node memory should fall once the compacted segments are loaded and old ones released.

If latency stays high, look at filters (missing scalar indexes) and search parameters, not compaction.

#Decision table

#Prevent it next time

• Never call flush() per write; batch inserts.
• Use diff-based ingestion so unchanged chunks aren't upserted.
• Use alias switching for full rebuilds.
• Graph segment count, compaction backlog and object storage size per collection.