How $exists can become cuplrit in MONGODB?

From a long time I had a doubt that $exists makes the query inefficient, but I was not confident enough to conclude any points soon. Today I observed something which made me confident to not use $exists going forward in my queries until it is extremely (I mean extremely) needed.

Here is why?

[
  {
    $match: {
      project_id: ObjectId('XXXXXXXXXXXX'),
      source: "marketing",
      attempt_count: { $exists: true },
    },
  },
  { $project: { _id: 1 } },
]

Motive :- Get all the records from event_logs where attempt_count exists (docs which has field attempt_count: 1/2/3)

Even after giving the hint :- Query was inefficient

db["event_logs"].aggregate([
  {
    $match: {
      project_id: ObjectId('XXXXXXXXXXXX'),
      source: "marketing",
      attempt_count: { $exists: true },
    },
  },
  { $project: { _id: 1 } },
], { hint: "project_id_1_source_1_attempt_count_1" }).explain("executionStats")

The point here is why it is checking 16k totalkeys and totalDocs while the nreturned is 9k, the ideal case should be totalkeys = totalDocs = nreturned (making it a covered query)

Solution for above issue :-

Use range $gte, $gt, $lte, $lt

I used this query:

db["event_logs"].aggregate([
  {
    $match: {
      project_id: ObjectId('XXXXXXXXXXXX'),
      source: "marketing",
      attempt_count: { $gte: 1 },
    },
  },
  { $project: { _id: 1 } },
], { hint: "project_id_1_source_1_attempt_count_1" }).explain("executionStats")

Magic 🎉

CULPRIT:- $exists

Reason by LLM:

1. Point of $exists: It tells us the field presence.

-> Now according to our requirement it seems to be very much perfect to use it. What if I convert $exists into something as a range?

1. In layman terms: What I have understood is that $exists makes your query search the needed as well as not needed docs because $exists tells MongoDB cursor to check all the docs where there is no attempt_count and check all the docs where there is attempt_count exists and then it is only able to segregate between those sets of docs to give you the required nReturned.

Proof:

1. My needed docs were 9k and not 16k, but $exists made the query scan all 16k docs (around 8k extra).