MongoDB · lesson 5 of 10
Intermediate
6- 2 Embedding vs Referencing: A 5.9MB Document and a 50x Fetch Time
- 3 explain() Proves It: 200,000 Docs Scanned vs. a Covered Query That Touches Zero
- 4 The Aggregation Stage-Order Bug: Wrong Results, No Error
- 5 When the Aggregation Optimizer Can (and Can't) Use Your Index
- 6 Change Streams: Watching a Collection in Real Time (Replica Set Required)
- 7 Multi-Document Transactions: When One Update Isn't Enough
Advanced
3Also part of MongoDB
1When the Aggregation Optimizer Can (and Can't) Use Your Index
A $match placed after $group looks like it must scan everything first — but MongoDB's optimizer can rewrite it into a pure index lookup when filtering on the group key itself. Filter on a computed value like a $sum, and that optimization disappears entirely.
By SkillByExample · Published
The literal order of pipeline stages doesn’t always predict what actually runs — MongoDB’s optimizer can rewrite a pipeline before executing it, and knowing when it will (and won’t) is the difference between a real gap and something that only looks like one.
A $match after $group that isn’t slow
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
{ $group: { _id: "$author", count: { $sum: 1 } } },
{ $match: { _id: "Author 4999" } }
]).explain("executionStats");
print("docsExamined:", r.executionStats.totalDocsExamined);
print("executionTimeMillis:", r.executionStats.executionTimeMillis);
printjson(r.queryPlanner.winningPlan.queryPlan);
'
docsExamined: 0
executionTimeMillis: 4
{
isCached: false,
queryPlan: {
stage: 'GROUP',
planNodeId: 3,
inputStage: {
stage: 'PROJECTION_COVERED',
planNodeId: 2,
transformBy: { author: true, _id: false },
inputStage: {
stage: 'IXSCAN',
planNodeId: 1,
indexName: 'author_1',
direction: 'forward',
indexBounds: { author: [ '["Author 4999", "Author 4999"]' ] }
}
}
}
}
This pipeline reads like it must group all 200,000 books by author and then filter down to
one — instead, docsExamined: 0 and a real IXSCAN with bounds ["Author 4999", "Author 4999"]. The optimizer noticed that $match’s field (_id, which came from $group’s
$author key) is the exact field an existing index covers, and rewrote the whole pipeline into
a direct index lookup before it ever ran.
The same shape, on a field the optimizer can’t push down
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
{ $group: { _id: "$author", count: { $sum: 1 } } },
{ $match: { count: { $gt: 39 } } }
]).explain("executionStats");
print("cursor stage docsExamined:", r.stages[0].$cursor.executionStats.totalDocsExamined);
print("cursor executionTimeMillis:", r.stages[0].$cursor.executionStats.executionTimeMillis);
print("match nReturned:", r.stages[1].nReturned);
'
cursor stage docsExamined: 200000
cursor executionTimeMillis: 65
match nReturned: 5000
Same two-stage shape, filtering on count instead of _id — and this time the explain()
output itself changes shape: instead of one fused queryPlan, there’s a stages array with a
separate $cursor (which scanned all 200,000 documents, 65ms) followed by a genuinely separate
$match stage. count is a value $group computes, not a real indexed field on any
document — there’s nothing for the optimizer to push a filter down into, so every document has
to be read and grouped before $match can even run.
Reading which case you’re in
The tell is in queryPlanner.winningPlan: a single queryPlan object (as in the first
example) means the optimizer fused everything into one plan; a stages array with separate
$cursor and $match entries (as in the second) means it didn’t. Guessing from the pipeline’s
written order isn’t reliable — explain() is the only way to know which happened for a
specific field and a specific set of indexes.
Practice
1. Confirm the same group-key pushdown works when the index is a compound one.
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
{ $group: { _id: "$genre", count: { $sum: 1 } } },
{ $match: { _id: "scifi" } }
]).explain("executionStats");
print("docsExamined:", r.executionStats.totalDocsExamined);
print("keysExamined:", r.executionStats.totalKeysExamined);
printjson(r.queryPlanner.winningPlan.queryPlan.inputStage.stage);
'
docsExamined: 0
keysExamined: 33334
PROJECTION_COVERED
Even better than the author example: docsExamined is still 0, and the input stage is
PROJECTION_COVERED, not a plain FETCH. Since count only needs to know how many
documents matched — not any field beyond genre itself — and genre is the leading field of
the genre_1_price_1 index, the whole grouped count is answered from the index alone.
keysExamined: 33334 shows the real work (walking that many index entries), but not one
actual document was read to do it.
2. Move the equivalent filter before $group explicitly, and compare.
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
{ $match: { genre: "scifi" } },
{ $group: { _id: "$genre", count: { $sum: 1 } } }
]).explain("executionStats");
print("docsExamined:", r.executionStats.totalDocsExamined);
'
docsExamined: 0
Identical docsExamined to the previous task — writing the filter explicitly before $group
here doesn’t change anything, because the optimizer had already produced the same plan on its
own. Explicit early filtering is still the right default, though: it’s not always this
equivalent, and it never depends on trusting an optimizer to notice.
3. Confirm a $match on a field with no index at all, after $group, forces the same full scan either way.
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
{ $group: { _id: "$published_year", count: { $sum: 1 } } },
{ $match: { _id: 2000 } }
]).explain("executionStats");
print("cursor docsExamined:", r.stages ? r.stages[0].$cursor.executionStats.totalDocsExamined : r.executionStats.totalDocsExamined);
'
cursor docsExamined: 200000
published_year has no index — filtering on the group key alone doesn’t help if that key was
never indexed in the first place. The pushdown optimization needs both an unindexed-computation-free
field and an actual index on it to have anything to push down into.
Next: change streams — watching a collection for real-time changes.