Skip to main content
MongoDB · lesson 5 of 10
MongoDB intermediate Lesson 5 of 10

When the Aggregation Optimizer Can (and Can't) Use Your Index

A $match placed after $group looks like it must scan everything first — but MongoDB's optimizer can rewrite it into a pure index lookup when filtering on the group key itself. Filter on a computed value like a $sum, and that optimization disappears entirely.

By SkillByExample · Published

mongodb aggregation performance explain query-optimization

The literal order of pipeline stages doesn’t always predict what actually runs — MongoDB’s optimizer can rewrite a pipeline before executing it, and knowing when it will (and won’t) is the difference between a real gap and something that only looks like one.

A $match after $group that isn’t slow

docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
  { $group: { _id: "$author", count: { $sum: 1 } } },
  { $match: { _id: "Author 4999" } }
]).explain("executionStats");
print("docsExamined:", r.executionStats.totalDocsExamined);
print("executionTimeMillis:", r.executionStats.executionTimeMillis);
printjson(r.queryPlanner.winningPlan.queryPlan);
'
docsExamined: 0
executionTimeMillis: 4
{
  isCached: false,
  queryPlan: {
    stage: 'GROUP',
    planNodeId: 3,
    inputStage: {
      stage: 'PROJECTION_COVERED',
      planNodeId: 2,
      transformBy: { author: true, _id: false },
      inputStage: {
        stage: 'IXSCAN',
        planNodeId: 1,
        indexName: 'author_1',
        direction: 'forward',
        indexBounds: { author: [ '["Author 4999", "Author 4999"]' ] }
      }
    }
  }
}

This pipeline reads like it must group all 200,000 books by author and then filter down to one — instead, docsExamined: 0 and a real IXSCAN with bounds ["Author 4999", "Author 4999"]. The optimizer noticed that $match’s field (_id, which came from $group’s $author key) is the exact field an existing index covers, and rewrote the whole pipeline into a direct index lookup before it ever ran.

The same shape, on a field the optimizer can’t push down

docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
  { $group: { _id: "$author", count: { $sum: 1 } } },
  { $match: { count: { $gt: 39 } } }
]).explain("executionStats");
print("cursor stage docsExamined:", r.stages[0].$cursor.executionStats.totalDocsExamined);
print("cursor executionTimeMillis:", r.stages[0].$cursor.executionStats.executionTimeMillis);
print("match nReturned:", r.stages[1].nReturned);
'
cursor stage docsExamined: 200000
cursor executionTimeMillis: 65
match nReturned: 5000

Same two-stage shape, filtering on count instead of _id — and this time the explain() output itself changes shape: instead of one fused queryPlan, there’s a stages array with a separate $cursor (which scanned all 200,000 documents, 65ms) followed by a genuinely separate $match stage. count is a value $group computes, not a real indexed field on any document — there’s nothing for the optimizer to push a filter down into, so every document has to be read and grouped before $match can even run.

Reading which case you’re in

The tell is in queryPlanner.winningPlan: a single queryPlan object (as in the first example) means the optimizer fused everything into one plan; a stages array with separate $cursor and $match entries (as in the second) means it didn’t. Guessing from the pipeline’s written order isn’t reliable — explain() is the only way to know which happened for a specific field and a specific set of indexes.

Practice

1. Confirm the same group-key pushdown works when the index is a compound one.
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
  { $group: { _id: "$genre", count: { $sum: 1 } } },
  { $match: { _id: "scifi" } }
]).explain("executionStats");
print("docsExamined:", r.executionStats.totalDocsExamined);
print("keysExamined:", r.executionStats.totalKeysExamined);
printjson(r.queryPlanner.winningPlan.queryPlan.inputStage.stage);
'
docsExamined: 0
keysExamined: 33334
PROJECTION_COVERED

Even better than the author example: docsExamined is still 0, and the input stage is PROJECTION_COVERED, not a plain FETCH. Since count only needs to know how many documents matched — not any field beyond genre itself — and genre is the leading field of the genre_1_price_1 index, the whole grouped count is answered from the index alone. keysExamined: 33334 shows the real work (walking that many index entries), but not one actual document was read to do it.

2. Move the equivalent filter before $group explicitly, and compare.
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
  { $match: { genre: "scifi" } },
  { $group: { _id: "$genre", count: { $sum: 1 } } }
]).explain("executionStats");
print("docsExamined:", r.executionStats.totalDocsExamined);
'
docsExamined: 0

Identical docsExamined to the previous task — writing the filter explicitly before $group here doesn’t change anything, because the optimizer had already produced the same plan on its own. Explicit early filtering is still the right default, though: it’s not always this equivalent, and it never depends on trusting an optimizer to notice.

3. Confirm a $match on a field with no index at all, after $group, forces the same full scan either way.
docker exec mongo-lesson mongosh --quiet --eval '
db = db.getSiblingDB("bookshop");
const r = db.books.aggregate([
  { $group: { _id: "$published_year", count: { $sum: 1 } } },
  { $match: { _id: 2000 } }
]).explain("executionStats");
print("cursor docsExamined:", r.stages ? r.stages[0].$cursor.executionStats.totalDocsExamined : r.executionStats.totalDocsExamined);
'
cursor docsExamined: 200000

published_year has no index — filtering on the group key alone doesn’t help if that key was never indexed in the first place. The pushdown optimization needs both an unindexed-computation-free field and an actual index on it to have anything to push down into.

Next: change streams — watching a collection for real-time changes.

Frequently Asked Questions

Doesn't a $match after $group always have to process the whole collection first?
Not necessarily — MongoDB's aggregation optimizer can rewrite a pipeline before running it, and a $match filtering on the exact field used as the $group key can be pushed down into the initial query, using an index on that field. This lesson's winningPlan shows exactly that: a GROUP stage sitting on top of an IXSCAN with real index bounds, not a full collection scan.
Why couldn't the same optimization apply to $match on the count field?
count doesn't exist as a real field on any document — it's a value $group computes by summing as it processes input. There's no index on a value that doesn't exist until after the aggregation runs, so a $match filtering on it can only run after $group has produced its output, over every group, which means every input document had to be read first.
How can I tell, for a given pipeline, whether an optimization like this actually happened?
explain("executionStats") is the only reliable way — check queryPlanner.winningPlan for a single fused query plan (the pushed-down case) versus a stages array with separate $cursor and $match entries (the not-pushed-down case). Guessing from how the pipeline reads is exactly how this surprised expectations in this lesson.
Does this mean $match should always go right after $group?
Put $match as early as the actual filter allows — if it can run before $group, do that explicitly rather than relying on the optimizer to notice. This lesson's finding is about what happens when a $match after $group is the only option (filtering on the group key itself, post-hoc), not a reason to skip filtering earlier when that's actually possible.
What's the practical takeaway for writing production aggregation pipelines?
Don't assume a pipeline's performance from its written order — measure it with explain(). The optimizer will sometimes do better than the literal stage order suggests (as with the group-key match here) and sometimes exactly as bad as it looks (as with the computed-field match) — the only way to know which is to check, for the specific fields and indexes actually involved.