Sharding: Why 60,000 Fresh Documents Landed on Exactly One Shard

Two shards, a collection sharded from the start, sixty thousand documents inserted through the router — and every single one lands on one shard.

Config servers and shards use different default ports

docker run -d --name mongo-config --network mongo-rs -p 27021:27017 mongo:8 --configsvr --replSet configRS --bind_ip_all
docker run -d --name mongo-shard1 --network mongo-rs -p 27022:27017 mongo:8 --shardsvr --replSet shard1RS --bind_ip_all
docker run -d --name mongo-shard2 --network mongo-rs -p 27023:27017 mongo:8 --shardsvr --replSet shard2RS --bind_ip_all
docker exec mongo-config mongosh --quiet --eval 'db.runCommand({ping:1})'

MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017

mongosh’s default target, 27017, refuses the connection — even though the container is definitely running. Checking the actual startup log:

docker logs mongo-config 2>&1 | grep "Waiting for connections"

{"msg":"Waiting for connections","attr":{"port":27019,"ssl":"off"}}

--configsvr makes this server default to port 27019, not 27017 — and --shardsvr defaults to 27018 — specifically so a config server, a shard, and a router can all run on one host without colliding. Connect on the real port instead:

docker exec mongo-config mongosh --port 27019 --quiet --eval 'db.runCommand({ping:1})'

{ ok: 1 }

Building the cluster

docker exec mongo-config mongosh --port 27019 --quiet --eval '
rs.initiate({_id: "configRS", configsvr: true, members: [{_id: 0, host: "mongo-config:27019"}]})
'
docker exec mongo-shard1 mongosh --port 27018 --quiet --eval '
rs.initiate({_id: "shard1RS", members: [{_id: 0, host: "mongo-shard1:27018"}]})
'
docker exec mongo-shard2 mongosh --port 27018 --quiet --eval '
rs.initiate({_id: "shard2RS", members: [{_id: 0, host: "mongo-shard2:27018"}]})
'
docker run -d --name mongo-router --network mongo-rs -p 27024:27017 mongo:8 mongos --configdb configRS/mongo-config:27019 --bind_ip_all --port 27017
docker exec mongo-router mongosh --quiet --eval 'sh.addShard("shard1RS/mongo-shard1:27018")'
docker exec mongo-router mongosh --quiet --eval 'sh.addShard("shard2RS/mongo-shard2:27018")'

Each piece is its own tiny replica set — a config server, two shards — tied together by a mongos router that’s the only thing client code ever talks to directly.

The bug: sharding doesn’t mean instant distribution

docker exec mongo-router mongosh --quiet --eval 'sh.enableSharding("bookshop")'
docker exec mongo-router mongosh --quiet --eval 'db.getSiblingDB("bookshop").books.createIndex({genre: 1})'
docker exec mongo-router mongosh --quiet --eval 'sh.shardCollection("bookshop.books", {genre: 1})'

docker exec mongo-router mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const genres = ["scifi", "fantasy", "mystery", "romance", "history", "biography"];
const bulk = [];
for (let i = 0; i < 60000; i++) {
  bulk.push({ title: "Book " + i, genre: genres[i % genres.length], price: 5 + (i % 50) });
  if (bulk.length === 10000) { db.books.insertMany(bulk); bulk.length = 0; }
}
if (bulk.length) db.books.insertMany(bulk);
'
docker exec mongo-router mongosh --quiet --eval 'db.getSiblingDB("bookshop").books.getShardDistribution()'

Shard shard2RS at shard2RS/mongo-shard2:27018
{
  data: '4.22MiB',
  docs: 60000,
  chunks: 1,
  'estimated data per chunk': '4.22MiB',
  'estimated docs per chunk': 60000
}
---
Totals
{
  data: '4.22MiB',
  docs: 60000,
  chunks: 1,
  'Shard shard2RS': [ '100 % data', '100 % docs in cluster', '73B avg obj size on shard' ]
}

The collection was sharded on genre before a single document existed — and 100% of all 60,000 documents still landed on one shard. A new collection starts as exactly one chunk, and MongoDB only splits and migrates chunks once they cross a size threshold (128MB by default). 4.22MiB never got close.

The fix: split, then move, explicitly

docker exec mongo-router mongosh --quiet --eval 'sh.splitAt("bookshop.books", { genre: "mystery" })'
docker exec mongo-router mongosh --quiet --eval 'db.getSiblingDB("bookshop").books.getShardDistribution()'

Shard shard2RS at shard2RS/mongo-shard2:27018
{ data: '4.22MiB', docs: 60000, chunks: 2, 'estimated docs per chunk': 30000 }

Two chunks now — but still 100% on shard2RS. splitAt divides the key range; it doesn’t relocate anything. The actual move is a separate step:

docker exec mongo-router mongosh --quiet --eval 'sh.moveChunk("bookshop.books", { genre: "biography" }, "shard1RS")'
docker exec mongo-router mongosh --quiet --eval 'db.getSiblingDB("bookshop").books.getShardDistribution()'

Shard shard2RS at shard2RS/mongo-shard2:27018
{ data: '2.13MiB', docs: 60000, chunks: 1, ... }
---
Shard shard1RS at shard1RS/mongo-shard1:27018
{ data: '2.13MiB', docs: 30000, chunks: 1, ... }
---
Totals
{
  data: '4.26MiB', docs: 90000, chunks: 2,
  'Shard shard2RS': [ '50.04 % data', '66.66 % docs in cluster' ],
  'Shard shard1RS': [ '49.95 % data', '33.33 % docs in cluster' ]
}

Data is now on both shards — moveChunk did what splitAt alone couldn’t. (Also note the Totals.docs: 90000 here — more on that next.)

getShardDistribution() reports estimates, not exact counts

docker exec mongo-router mongosh --quiet --eval 'db.getSiblingDB("bookshop").books.countDocuments()'

60000

The real count is still 60000, not the 60000 + 30000 = 90000 the distribution report’s per-shard numbers summed to. getShardDistribution() is built from chunk metadata that can briefly overstate the real picture — use countDocuments() when the exact number actually matters, not the summary report.

Practice

docker logs mongo-router 2>&1 | grep "Waiting for connections"

{"msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}

docker exec mongo-router mongosh --quiet --eval '
db.getSiblingDB("config").chunks.find({}, { shard: 1, min: 1, max: 1, _id: 0 }).toArray();
'

[
  { min: { genre: MinKey() }, max: { genre: 'mystery' }, shard: 'shard1RS' },
  { min: { genre: 'mystery' }, max: { genre: MaxKey() }, shard: 'shard2RS' }
]

docker exec mongo-router mongosh --quiet --eval 'sh.getBalancerState()'

true

Next: performance and profiling — the slow query log and explain() together, closing the track.