docs: add Atlas Vector Search and Atlas Search documentation

[RaschidJFR]

Summary

Adds comprehensive documentation for Atlas Vector Search and Atlas Search in Mongoose. These features are fully implemented in the codebase but were previously undocumented.

This PR focuses on Mongoose-specific usage patterns, with references to MongoDB's official documentation for implementation details.

What's New

Atlas Vector Search (docs/atlas-vector-search.md)

• Creating vector search indexes with schema.searchIndex() and type: 'vectorSearch'
• Querying with $vectorSearch aggregation stage
• Atlas Automated Embeddings
• Third-party embedding providers
• Pre-filtering and search options

Atlas Search (docs/atlas-search.md)

• Creating text search indexes with Lucene analyzers
• Managing indexes (create, list, update, delete)
• Text search queries with $search aggregation stage
• Multi-field and fuzzy search
• Compound queries with scoring
• Hybrid search patterns (sequential re-ranking)
• Production best practices

Changes

New Files (2)

• docs/atlas-vector-search.md - Complete vector search guide (~260 lines)
• docs/atlas-search.md - Complete text search guide (~390 lines)

Modified Files (3)

• docs/guide.md - Enhanced autoSearchIndex section with vector search example
• docs/layout.pug - Added sidebar navigation entries
• docs/source/index.js - Registered new doc routes

Navigation

• Added to sidebar: Atlas Search → Vector Search
• Updated docs/guide.md autoSearchIndex section

[vkarpov15]

I walked through these tutorials and have a lot of comments but mostly minor. Mostly I just want to make sure I can fully test the code samples end to end in both cases, and in both cases I wasn't quite able to make it all the way through. The $rankFusion issue for atlas-search.md and not having a way to calculate auto embeddings for atlas-vector-search.md are the blockers right now.

[vkarpov15]

Mongoose property names are typically camelCase, but this is fine since I presume this guide is using MongoDB's sample_mflix sample data set. It would be good to link out to that so users know they can get this data set.

[RaschidJFR]

added in the prose
UPDATE: Given m_flix's embeddings are stored as binary data rather than an array of numbers, a working example for it would be too complex for the purpose of this guide. There is no suitable sample dataset available from mongodb for a simple, working example.
Will omit the reference/link for now.

[vkarpov15]

I agree, using plot_embedding is fine since we're using sample_mflix. However, in that case we should have a link to documentation or some information about sample_mflix so users know this is a data set they can import into Atlas directly. I think this page: https://www.mongodb.com/docs/manual/sample-data/sample-mflix/#std-label-mflix-embedded_movies would be a good one to link to.

[RaschidJFR]

Added a reference to mflix and updated all examples to be compatible with its Voyage version (using plot_embedding_voyage_3_large instead of plot_embedding)

[vkarpov15]

Some minor comments on the search indexes docs, those look mostly good.

For vector search, I would like that document to be more immediately usable. If we make it clear that document is meant to integrate with the embedded_movies collection in sample_mflix (https://www.mongodb.com/docs/manual/sample-data/sample-mflix/#std-label-mflix-embedded_movies), we can make it so that users can immediately use semantic search on the movies database with only one additional step of setting up a Voyage AI API key.

An alternative path would be to create a new schema and minimal data set from scratch - that approach has the benefit of not requiring working with buffers, which may be a bit confusing (although absolutely makes sense - buffers are much more space efficient than arrays of numbers for storing vectors and vectors are huge relative to your average name, age, etc. properties). However, I think there's a benefit to being immediately able to search a non-trivial data set as a way to demonstrate the power of semantic search.

[vkarpov15]

Given that this document references createSearchIndexes() as the preferred way to create search indexes, I would advise not setting autoSearchIndex: true here - just omit autoSearchIndex or put it in a separate section. People will copy/paste this code snippet along with createSearchIndexes() and that will lead to them double-creating search indexes. createSearchIndexes() doesn't require autoSearchIndex option to be set.

[RaschidJFR]

Removed from code example.
Added a link to https://mongoosejs.com/docs/guide.html#autoSearchIndex for more info.

[vkarpov15]

In my brief experiments 5 is very high. For example, with query 'mongodb' still gives a score of 1.97 for an article with title 'MongoDB Tutorial':

  {
    _id: new ObjectId('000000000000000000000001'),
    content: 'MongoDB is a document database designed for developer productivity.',
    title: 'MongoDB Tutorial',
    score: 1.973141074180603
  }

I'd recommend making this something like $gte: 0.1 or some other low value to make it reasonably likely the user will get a result when executing a query.

It would also be helpful to provide some sample data too

[RaschidJFR]

With this example agains the mflix dataset, the minimum score is 1.27. Nontheless, I lowered to 0.1 to back the idea of starting low.

[vkarpov15]

Minor pedantic point - "Automatically sync indexes" -> "Automatically create indexes from your schema"

In Mongoose, syncing indexes e.g. syncIndexes() drops indexes which are not in the schema. Creating indexes just creates new indexes without dropping existing indexes.

[RaschidJFR]

updated

[vkarpov15]

I don't see anything in the referenced docs about throwing an error if the index already exists. Also in my experiments, I'm calling Article.createSearchIndexes() every time I run a script - no errors.

[vkarpov15]

Given that other examples in this document use "mongodb tutorial" as an example, I'd prefer to use those semantics for this example so the user doesn't need to create a new data set specifically for this example.

[RaschidJFR]

Reworked the guide to use mflix examples

[vkarpov15]

This suggestion is incorrect AFAIK, would throw a $search is only valid as the first stage in a pipeline error. That's why filter is a property in $search. I recommend instead putting that $search should be the first stage in the pipeline so users don't get errors by surprise.

[RaschidJFR]

Thanks for catching this. Fixed

[vkarpov15]

Why is dynamic: true bad in production - slows down writes?

[RaschidJFR]

Because with dynamic:true every field gets indexed, making the search index potentially larger than the raw data if not used carefully.

Added a not about this

Source: https://www.mongodb.com/docs/search/performance/index-performance/?deployment-type=atlas#dynamic-vs-static-mappings

[vkarpov15]

I agree, using plot_embedding is fine since we're using sample_mflix. However, in that case we should have a link to documentation or some information about sample_mflix so users know this is a data set they can import into Atlas directly. I think this page: https://www.mongodb.com/docs/manual/sample-data/sample-mflix/#std-label-mflix-embedded_movies would be a good one to link to.

[vkarpov15]

I would prefer we include an implementation of generateEmbedding() that works with sample_mflix to make the code in this document immediately usable, rather than leaving significant implementation details as an exercise to the reader.

I was able to get the existing sample_mflix data searchable using the following:

const movieSchema = new mongoose.Schema({
  title: String,
  plot: String,
  // Vector embeddings for the plot
  // The stored vectors are 2048-dimensional float32 BSON vectors.
  plot_embedding_voyage_3_large: {
    type: Buffer,  // use Buffer if your embeddings are stored as binData
    validate: {
      validator: v => v == null || v.length === 8192,  // 2048 float32 values, 4 bytes each
      message: 'plot_embedding_voyage_3_large must be nullish or exactly 8192 bytes'
    }
  }
}, { });

// Define a vector search index
movieSchema.searchIndex({
  name: 'vector_index',
  type: 'vectorSearch',
  definition: {
    fields: [{
      type: 'vector',
      path: 'plot_embedding_voyage_3_large',
      numDimensions: 2048,  // Must match your embedding model's dimensions
      similarity: 'cosine'  // or 'euclidean' or 'dotProduct'
    }]
  }
});

const Movie = mongoose.model('Movie', movieSchema, 'embedded_movies');

async function generateEmbedding(input) {
  const voyageAPIKey = process.env.VOYAGE_API_KEY;
  if (!voyageAPIKey) {
    throw new Error('VOYAGE_API_KEY environment variable is required');
  }

  const response = await fetch('https://ai.mongodb.com/v1/embeddings', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${voyageAPIKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      input,
      model: 'voyage-3-large',
      output_dimension: 2048
    })
  });

  if (!response.ok) {
    throw new Error(`Voyage AI request failed: ${response.status} ${await response.text()}`);
  }

  const result = await response.json();
  const embeddings = result.data.map(({ embedding }) => embedding);
  return Array.isArray(input) ? embeddings : embeddings[0];
}

void async function main() {
  await Movie.createSearchIndexes();

  // Your query embedding (example function - see "Generating Embeddings" section below)
  // MongoDB Atlas can generate embeddings automatically, or you can use third-party providers
  const queryEmbedding = await generateEmbedding('vacation to mars');
  const queryVector = Binary.fromFloat32Array(Float32Array.from(queryEmbedding));

  // Perform vector search
  const results = await Movie.aggregate([
    {
      $vectorSearch: {
        index: 'vector_index',                  // Name of your vector search index
        path: 'plot_embedding_voyage_3_large',  // Field containing the vectors
        queryVector,                            // Match the stored float32 BSON vector representation
        numCandidates: 100,                     // Number of candidates to consider (should be >= limit)
        limit: 10                               // Number of results to return
      }
    },
    {
      $project: {
        title: 1,
        plot: 1,
        score: { $meta: 'vectorSearchScore' }   // Include similarity score
      }
    }
  ]);
  console.log(results[0].title); // "Total Recall"
  await mongoose.disconnect();
}();

Using Voyage AI API as described here and Voyage AI API key setup as described here

[RaschidJFR]

Added function definition

[RaschidJFR]

Reworked both guides to share the same semantics and be compatible with mflix dataset

[RaschidJFR]

updated

[RaschidJFR]

Removed from code example.
Added a link to https://mongoosejs.com/docs/guide.html#autoSearchIndex for more info.

[RaschidJFR]

Thanks for catching this. Fixed

[RaschidJFR]

Because with dynamic:true every field gets indexed, making the search index potentially larger than the raw data if not used carefully.

Added a not about this

Source: https://www.mongodb.com/docs/search/performance/index-performance/?deployment-type=atlas#dynamic-vs-static-mappings

[RaschidJFR]

Added a reference to mflix and updated all examples to be compatible with its Voyage version (using plot_embedding_voyage_3_large instead of plot_embedding)

[RaschidJFR]

Reworked the guide to use mflix examples

[RaschidJFR]

With this example agains the mflix dataset, the minimum score is 1.27. Nontheless, I lowered to 0.1 to back the idea of starting low.

[RaschidJFR]

Added function definition