mdb-lucene

The 5+1 punchline · the same five steps, then one MQL pipeline

Each step on its own is a boring everyday query. Watch what happens at the bottom when the same five steps become five stages of one aggregation pipeline.

1"Look up your booking." — operational record · findOne

db.bookings.findOne({
  user_id:  "u_8421",
  depart_at: { $gte: ISODate("2026-05-19"),
               $lt:  ISODate("2026-05-20") }
})
// → { _id: "BK-7724", flight_no: "AA1132", seat: "27A",
//     fare_class: "Q", origin: "JFK", destination: "SFO" }

2"Read what's been said in this chat so far." — session state · scoped find + sort

db.chats.find(
  { session_id: "sess_3f9c", user_id: "u_8421" }
).sort({ ts: 1 })

3"Look up the airline's reschedule policy." — vector recall · $vectorSearch with pre-filter pushdown

db.policies.aggregate([
  { $vectorSearch: {
      index:       "policy_index",
      path:        "embedding",
      queryVector: embed("rescheduling rules"),
      filter:      { airline: "AA" },   // pushed INTO HNSW, not after
      limit:       3
  }},
  { $project: { title: 1, body: 1,
                _score: { $meta: "vectorSearchScore" } }}
])

4"Recall what you usually prefer." — long-term memory · findOne on the user's preferences

db.user_memory.findOne(
  { user_id: "u_8421" },
  { preferences: 1 }
)

5"Check what worked last time." — tool traces · time-series · most-recent successful call

db.tool_traces.find({
  user_id: "u_8421",
  tool:    "airline_reschedule_api",
  ts:      { $gte: ISODate("2026-04-01") }
}).sort({ ts: -1 }).limit(1)

// One agent turn. One aggregation pipeline. One round trip.
db.bookings.aggregate([

  // Step 1 — operational record (the seed for everything else)
  { $match: {
      user_id:  "u_8421",
      depart_at: { $gte: ISODate("2026-05-19"),
                   $lt:  ISODate("2026-05-20") }
  }},

  // Step 4 — long-term memory, joined onto the user
  { $lookup: {
      from: "user_memory", localField: "user_id",
      foreignField: "user_id", as: "memory"
  }},

  // Step 2 — session/chat state, ordered, scoped to this session
  { $lookup: {
      from: "chats",
      let:  { uid: "$user_id" },
      pipeline: [
        { $match: { $expr: { $and: [
            { $eq: ["$user_id",    "$$uid"] },
            { $eq: ["$session_id", "sess_3f9c"] }
        ]}}},
        { $sort: { ts: 1 } }
      ],
      as: "chat"
  }},

  // Step 3 — vector recall over the policy corpus, pre-filter pushed INTO HNSW
  { $lookup: {
      from: "policies",
      pipeline: [
        { $vectorSearch: {
            index:       "policy_index",
            path:        "embedding",
            queryVector: embed("rescheduling rules"),
            filter:      { airline: "AA" },
            limit:       3
        }},
        { $project: { title: 1, body: 1 } }
      ],
      as: "policy"
  }},

  // Step 5 — most recent successful call to this tool for this user (time-series)
  { $lookup: {
      from: "tool_traces",
      let:  { uid: "$user_id" },
      pipeline: [
        { $match: { $expr: { $eq: ["$user_id", "$$uid"] },
                    tool: "airline_reschedule_api",
                    ts:   { $gte: ISODate("2026-04-01") } }},
        { $sort:  { ts: -1 } },
        { $limit: 1 }
      ],
      as: "last_attempt"
  }},

  // Final shape — exactly what the agent's reasoning step needs as input.
  { $project: {
      booking:      { _id: "$_id", flight_no: "$flight_no",
                      seat: "$seat", origin: "$origin",
                      destination: "$destination" },
      memory:       { $first: "$memory" },
      chat:         1,
      policy:       1,
      last_attempt: { $first: "$last_attempt" }
  }}

])
// → ONE document containing operational + memory + chat + policy + traces.
//   ONE round trip. ONE query language. ONE ops surface. ONE auth model.
//   ONE backup. ONE SOC2 audit. ONE on-call rotation.

That's it. $match + four $lookups + a $project — operators that have existed in MongoDB since long before "AI agent" was a job title. The $vectorSearch stage is one of the four lookups. Vector recall is a stage. The aggregation framework already knew how to do four other kinds of stages; it now also knows how to do this one; it composes them all without you needing to write any glue.

The unbroken thread · what MongoDB has been absorbing since 2009

Vector search is one expression of a strategy MongoDB has been running, one capability at a time, since 2009. Every "you'll need a separate database for that" wave got the same answer: extend the query layer; never rebuild the foundation.

Every node above was the same answer to the same question. Same answer. Same framework. Same model. Sixteen years of compounding. That's the macro-strategy: absorb complexity into the platform; build convenience into the query language. Vector search is the latest expression of it.

This is the static landing page · run the live demo for the interactive panels

The full mdb-lucene demo includes two interactive panels:

1. The composable-pipeline panel — one MongoDB Atlas aggregation pipeline ($vectorSearch + $lookup + $addFields + $project + $sort) against movies + reviews data, side-by-side with the equivalent raw Lucene + glue column that quantifies the cost of the alternative.
2. The engine-equivalence panel — Atlas $vectorSearch versus raw KnnFloatVectorQuery against the same MiniLM embeddings on the same 20 docs, agreeing to four decimal places. The proof that the engine is commodity, therefore the contest is at the layers above and around it.

To run the live demo:
git clone https://github.com/fabian-valle-simmons/mdb-lucene
cd mdb-lucene
docker compose up
Then open http://localhost:8088. Five minutes. One Docker host. Real MongoDB Atlas Local + a real Lucene HNSW service. Run your own queries against the live engines.

Why static here? The interactive panels rely on a Python embedding model (MiniLM) that turns free-form prose into 384-d vectors at query time — that lives in the demo's Python process, not in your browser. The static landing page above is the strategic argument; the live demo is the working proof on a laptop.

The honest position · when a vector database is the right answer

This page argues vector databases are the wrong shape for AI agents. That argument is true. But specialized vector databases ship in production today against workloads where they're absolutely the right pick.

Use a vector database when your product is vector search. Use an agentic database when your product is an agent.