MongoDB sharded cluster out of balance: jumbo chunks, a hot shard and a balancer that won't move data

#Symptoms

• One shard's disk and CPU are much higher than the others.
• sh.status() shows chunks or data heavily skewed to one shard.
• The balancer is enabled, but nothing moves; or migrations run constantly and slow down the cluster.
• Inserts are slower than expected even though there are many shards.

#Step 1: how unbalanced is it?

sh.status()
sh.getBalancerState()          // enabled?
sh.isBalancerRunning()         // currently migrating?
sh.balancerCollectionStatus("shop.orders")

balancerCollectionStatus tells you whether the collection is balanced and, if not, why (for example chunksImbalance, defragmentingChunks, zoneViolation).

Data per shard:

db.getSiblingDB("shop").orders.getShardDistribution()

Since MongoDB 6.0 the balancer balances on data size per shard rather than chunk counts, so look at data sizes, not just number of chunks.

#Step 2: is the balancer allowed to run?

use config
db.settings.find({ _id: "balancer" })

sh.startBalancer()
sh.enableBalancing("shop.orders")
// Balancing window, server time
db.settings.updateOne({ _id: "balancer" },
  { $set: { activeWindow: { start: "01:00", stop: "05:00" } } }, { upsert: true })

#Step 3: jumbo chunks

A chunk that exceeds the configured range size and can't be split (because all documents share the same shard key value, or very few values) is marked jumbo. The balancer won't move it.

use config
db.chunks.find({ jumbo: true }, { min: 1, max: 1, shard: 1, uuid: 1 }).limit(20)

Causes and fixes:

Clear the jumbo flag (MongoDB 4.2.3+):

db.adminCommand({ clearJumboFlag: "shop.orders", find: { tenant_id: "acme" } })

For chunks that are big but splittable, split manually:

sh.splitFind("shop.orders", { tenant_id: "acme", order_id: 500000 })

#Step 4: the hot shard (monotonic shard key)

A shard key like { _id: 1 } with ObjectId, { created_at: 1 } or an increasing counter sends all new inserts to the last range, on one shard. The balancer moves old data later, but the insert load stays on one shard.

Options:

• Hashed shard key for insert-heavy collections without range queries on the key:

sh.shardCollection("shop.events", { device_id: "hashed" })

• Compound key with a high-cardinality leading field: { tenant_id: 1, created_at: 1 } spreads tenants, keeps range queries per tenant.
• Refine an existing key by adding suffix fields (MongoDB 4.4+). Requires an index on the new full key:

db.orders.createIndex({ tenant_id: 1, order_id: 1 })
db.adminCommand({ refineCollectionShardKey: "shop.orders", key: { tenant_id: 1, order_id: 1 } })

• Reshard to a completely new key (MongoDB 5.0+). It copies the collection online, then switches with a short write block:

db.adminCommand({ reshardCollection: "shop.orders", key: { customer_id: "hashed" } })

Warning

Resharding needs free disk space for a full extra copy of the collection on the receiving shards, plus oplog capacity, and adds load during the copy. Check the documented requirements and run it in a quiet period. Monitor progress with $currentOp filtered on resharding operations.

#Step 5: migrations hurting performance

Constant migrations compete with user traffic.

• Set a balancing window outside peak hours (Step 2).
• On MongoDB 7.0+, chunk merging is automatic (AutoMerger), which reduces the number of tiny chunks; check it's not disabled.
• Check _secondaryThrottle and _waitForDelete settings only on advice for your version; defaults are usually right.

Check migration history:

use config
db.changelog.find({ what: /moveChunk|moveRange/ }).sort({ time: -1 }).limit(10)

#Decision table

#Prevent it next time

• Choose shard keys with high cardinality, even write distribution and support for common queries; test with production-like data.
• Never shard on a pure timestamp or ObjectId for insert-heavy collections.
• Monitor data size per shard and alert on imbalance.
• Check balancer state after every maintenance.