DEV Community: babuvenky76

Build Your Movie Recommendation System Using Amazon Personalize, MongoDB Atlas, and AWS Glue

babuvenky76 — Fri, 12 Apr 2024 07:02:12 +0000

Author:
Siddharth Joshi, (Technical Account Manager at AWS)
Sornavel Perumal (Technical Account Manager at AWS)

Contributor:
Babu Srinivasan (Senior Partner Solutions Architect at MongoDB)

In today's data-driven world, personalized recommendations have become an integral part of enhancing user experiences. With the power of cloud computing and advanced database solutions, building your own personalized movie recommendation system is now more achievable than ever. In this article, we'll explore the integration of MongoDB Atlas, AWS Glue, and Amazon Personalize to create a robust and scalable recommendation engine.

Understanding the components

Before diving into the integration process, let's briefly understand the key components involved in our movie recommendation system:

MongoDB Atlas is a fully managed, cloud-based database service that enables seamless deployment, scaling, and maintenance of MongoDB databases.

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analysis. It helps bridge the gap between our MongoDB Atlas data and the services we'll use for recommendation.

Amazon Personalize is a machine learning service that makes it easy to build, train, and deploy personalized recommendation models. It will analyze the data from MongoDB Atlas and generate personalized movie recommendations for users

Reference architecture

This architecture seamlessly ingests data from MongoDB Atlas, powering personalized recommendations. The AWS Glue Spark Job extracts transforms (filtering, cleaning, joining), and loads data into S3. This prepared data becomes the foundation for your chosen AI/ML service (SageMaker, Personalize, etc.), enabling highly accurate and personalized recommendations.

Prerequisites

This tutorial will be well understood if you have a good understanding of MongoDB Atlas services and Amazon Web Services (AWS), mentioned in the above architecture diagram.

Setting up MongoDB Atlas for movie data

Begin by creating a MongoDB Atlas database to store information about movies, genres, and user interactions. Populate the database with relevant data, ensuring it is well-structured for the recommendation model.
For this article, we will be using the MovieLens dataset.

a. If you do not already have one, you can sign up for a MongoDB Atlas account.

b. Create a database named movielens.

c. Get your connection URI to connect to MongoDB Atlas noted down.

d. Download the MovieLens dataset.

e. Unzip the file locally and run the Python script to upload data to MongoDB Atlas. (Replace and path_to_extracted_files.)

Using AWS Glue for data preparation

AWS Glue comes into play for ETL operations. Create a Glue job to extract data from MongoDB, transform it into a suitable format for training the recommendation model, and load it into an Amazon S3 bucket.

a. Create an S3 bucket to store the processed file from Glue.

b. Store your MongoDB connection properties credentials in AWS Secrets Manager.

c. Create a new AWS Glue Studio job with the Spark script editor option.

From the AWS Glue Studio console, select jobs from the menu and select “Script editor.”

Select the Spark option from the dropdown menu and click Create script.

d. Create an ETL job using Glue. Replace the Python script.

e. Specify input arguments.

Key : Value
--BUCKET_NAME : <bucket_name>
--OUTPUT_FILENAME1 : ratings
--OUTPUT_FILENAME2 : items
--COLLECTION_NAME3 : movies
--COLLECTION_NAME2 : tags
--COLLECTION_NAME1 : ratings
--SECRET_NAME : <name_of_secret>

f. Run the job.

Create a dataset group and an interactions dataset in Amazon Personalize

a. Go to Amazon Personalize in your AWS console.
b. In the left navigation pane, click on Dataset groups.
c. Click the Create dataset group button. Enter movie-datasetgroup as the name for your dataset group. Select Video on demand as the Domain.

In Amazon Personalize, click Create dataset group, provide the name of your dataset, and select the “Video on demand” option. Click Create group.

d. After creating the dataset group, you need to add datasets to it. Click on the dataset group you just created.
e. Click on the Create dataset button, and select Item interactions dataset.

Select the “Item interactions dataset” from the dropdown menu.

f. Select Import data directly into Amazon Personalize datasets as the Import method.

Choose the “Import data directly into Amazon Personalize datasets.”

g. Provide movie-interactions as Dataset name and Schema name.

Give a name to the Dataset and select “Create a new domain schema by modifying the existing default schema for your domain.”

h. To configure your dataset, import the job, select Import data from S3, and provide a name for the import job. Specify the path of ratings.csv in your S3 bucket as the data location, and specify an IAM role that has access to the S3 bucket.

Select “Import data from S3.”

Select the S3 data location.

Create the User dataset

The User dataset is the dataset for all the users listed in the system. MovieLens does not provide a user dataset so we will be using one that has been created for this post. In the real world, this dataset would be coming from your application.

a. Copy the users.csv file and put it on your S3 bucket created earlier.
b. Follow the steps above to create the User dataset.

Select “Users dataset” from the dropdown menu.

Choose “Import data directly into Amazon Peronalize Datasets.”

Provide a name for the Dataset and choose “Create a new domain schema by modifying the existing default schema for your domain.”

c. Ensure that the Schema definition looks like the following:

{
“type” : “record”,
“name” : “Users”,
“namespace”: “com.amazonaws.personalize.schema”,
“fields” : [
       { “name”: “USER_ID”, “type”: “string”},
       { “name”: “SUBSCRIPTION_MODEL”, “type”: “string”,”categorical”: true},
               ],
“version”:”1.0”
}

d. Provide S3 as the data import source and the S3 path for users.csv as the data location.

Select the “Import data from S3” option and provide the name to “Dataset import job name.”

Select the S3 bucket location and provide the IAM role.

Create the Items dataset

The Items dataset refers to a list of all the movies available in our application. Our Glue ETL job has converted the MongoDB collection “movies” into a .csv file in a format usable with Amazon Personalize.
You need to follow similar steps as above to create an Items dataset. For data location, you need to provide the path of items.csv on your S3 bucket.

a. Select the “Items dataset” from the dropdown menu.

b. Select the “Import data directly into Amazon Personalize datasets” option.

c. Provide a name — “movie-item” — to the Dataset name and select the “Create a new domain schema by modifying the existing default schema for your domain” option.

d. Select the “Import data from S3” option and provide the Dataset import job name — “movie-ds-item.”

Select the S3 location for the Data location and provide the IAM Role.

e. Before proceeding to the next step, you should wait until all three datasets become active.

Ensure you can see “3/3 datasets active” in green.

Run data analysis

Now, use Amazon Personalize for analyzing the data imported, meaning Users, Item interactions, and item datasets.
Start the data analysis by clicking on Run data analysis.

Ensure the data analysis run has been completed successfully.

Create recommenders

Create the recommenders after the Domain dataset group is created successfully. A recommender is a Domain dataset group resource that generates recommendations. Use a recommender in the application to get real-time recommendations with the GetRecommendations operation.

a. select Use e-commerce recommenders.

b. For the use case, select Because you watched X and provide a name to the recommender.

c. You can leave Advanced configuration as the default.

d. Review the configuration and click on Create recommenders.

e. Before proceeding to the next step, please wait until the recommender becomes active.

Ensure the status is “Active” for the movie-recommender.

Test recommender

Now that we have created a recommender, we are ready to get recommendations. In a real-world scenario, our application would be sending requests to Amazon Personalize and getting recommendations. For the post, we will test it using the Amazon Personalize console.
a. Go to the Amazon Personalize console.
b. Click on Recommenders under movie-dataset group and select movie-recommender.

c. Click on Test.

d. Enter a valid user ID and movie ID (Item ID), and click on Get recommendations.

e. The recommender will provide the list of recommendations in the form of the movie ID.

In a real-world scenario, your application will map these movie IDs to movie names and will show them as recommendations to users.

Conclusion

In this post, we explored the integration of MongoDB, AWS Glue, and Amazon Personalize to build a personalized movie recommendation system. This powerful combination allows you to leverage the flexibility of MongoDB, the data preparation capabilities of AWS Glue, and the machine learning prowess of Amazon Personalize to deliver a tailored and engaging user experience. As you embark on your journey to enhance user engagement, this integration offers a scalable and efficient solution for building recommendation systems in various domains.

Refer to the following links for further reading:
Writing an AWS Glue for Spark script
MongoDB Atlas database
MongoDB Community Forum

Developer's Guide: Crafting API-Driven Apps with MongoDB Atlas Using AWS CDK, API Gateway, and Lambda

babuvenky76 — Wed, 24 Jan 2024 08:23:47 +0000

Author:

Deepti Chugh (Sr Partner Success SA at AWS)
Bharath S (Senior Partner Solutions Architect ISVs at AWS)

Contributor:
Babu Srinivasan (Senior Partner Solutions Architect at MongoDB)

Welcome to our technical blog, where we unveil a step-by-step guide to deploying a robust REST API powered by Lambda functions, expertly bridging the gap between MongoDB Atlas and AWS, all with the added convenience of automation through the AWS Cloud Development Kit (CDK). Our mission is to empower developers like you to seamlessly integrate MongoDB Atlas with AWS API Gateway, all while implementing authentication via Cognito User Pools. If you're ready to embark on a journey that not only streamlines the process of building modern API-driven applications but also leverages the power of automation, you're in the right place. Let's dive into the details and unlock the potential of this dynamic integration.

What Will You Build?
This solution comprises the following AWS services which get deployed using CDK (Cloud Development Kit):

In the MongoDB SaaS account:
o A MongoDB cluster
o A MongoDB project
o A MongoDB database user
In the AWS customer account:
o Amazon Cognito UserPool - user directory for authentication and authorization
o AWS Secrets Manager – for keeping MongoDB Database Credentials
o Application Programming Interface (API) Gateway – acts as the "front door" for applications to access data, business logic, or functionality from your backend services
o Lambda function – connects to the Mongo DB database using PyMongo which is the Python driver for MongoDB

Reference architecture

In the above figure, the users call the API gateway endpoint to access MongoDB Atlas by invoking the AWS Lambda function. The user is authenticated by Amazon Cognito services. The credentials are stored in AWS Secrets Manager and the entire setup can be automated using the AWS CDK. MongoDB Atlas resides in a distinct Atlas VPC, fully administered by MongoDB. It is accessed securely through a private link for enhanced security.

Implementation Steps

This solution uses AWS CDK to deploy the solution on AWS. The first step involves creating a MongoDB cluster and database and then deploying AWS.

Prerequisites:

AWS CDK
NPM
MongoDB Atlas Account
AWS Account and AWS CLI Installed and Configured
Activate MongoDB Atlas CloudFormation resources in your AWS account with sufficient permissions
Store MongoDB Atlas programmatic API Keys in AWS Secrets Manager. You can find more details about these in MongoDB's tutorial or the GitHub repository

Step 0: Initialize the CDK Project

Open the IDE of your choice — Cloud9, VS Code, etc.
Execute the below commands to initialize the environment.



#Get the application code
    git clone https://github.com/mongodb-partners/Microservice_Application_with_MongoDBAtlas_AWSCDK_APIGW_Lambda.git
    cd aws_mongodb_sample_dir

# If you DONT have cdk installed
    npm install -g aws-cdk



# Make sure you in root directory
    python3 -m venv .venv
    source .venv/bin/activate
    pip3 install -r requirements.txt

Step 1: Deploy MongoDB Atlas and AWS resources

Set up the AWS CLI and connect to the session.
Copy the [MongoDB Atlas Organization ID] and this will be used in the next step.(https://www.mongodb.com/docs/atlas/access/orgs-create-view-edit-delete/#view-organizations)
Run the below commands to install the Python dependencies included with this sample.



#Install Dependencies for Lambda Function
    cd aws_mongodb_sample
    pip install --target ./dependencies pymongo
    cd ..

# Set Environment Variables
    export ORG_ID="<ORG_ID>"
    export MONGODB_USER="<MONGODB_USER>"
    export MONGODB_PASSWORD="<MONGODB_PASSWORD>"

    cdk bootstrap aws://<ACCOUNT_NUMBER>/<AWS-REGION>

Run the below commands to deploy the CDK template.



cdk synth
cdk deploy --all

Copy the API gateway output endpoint from the terminal as you will need this while testing the API gateway. Alternatively, you can copy it from the stack output from the cloud formation in the console.

Step 3: Explore the Deployed Resources

Once the CDK is deployed, go to the AWS Console and verify the resources
1) MongoDB::Atlas::Cluster
2) MongoDB::Atlas::Project
3) MongoDB::Atlas::DatabaseUser
4) MongoDB::Atlas::ProjectIpAccessList
5) Secret for storing ATLAS DB URI
6) Cognito User Pool
7) Lambda
8) API Gateway

Step 4: Test the Resources
1) Navigate to the Cognito user pool and copy the user pool ID and client ID (in the App Integration tab) from the Cognito user pool

2) Open Cloud Shell and create a user with the command below:



aws cognito-idp admin-create-user --user-pool-id  <YOUR_USER_POOL_ID>  --username apigwtest

3) Once you’ve created the user, since it’s created by an admin, we will have to force change the password by running the below command:



aws cognito-idp admin-set-user-password --user-pool-id <YOUR_USER_POOL_ID>  --username apigwtest  --password <YOUR_PASSWORD> --permanent

4) Replace the user pool ID and client ID copied in the above step. Also, replace the password of the user created above.



aws cognito-idp admin-initiate-auth --user-pool-id <YOUR_USER_POOL_ID> --client-id <CLIENT_ID>  --auth-flow ADMIN_NO_SRP_AUTH --auth-parameters USERNAME=apigwtest,PASSWORD=<YOUR_PASSWORD>

5) Copy the ID token created from the above step and run the below command to test the API. Copy the API_GATEWAY_ENDPOINT from the API gateway console --> API Gateway: APIs: ApiGateway (xxxxxx) :Stages



curl --location --request GET 'https://<API_GATEWAY_ENDPOINT>.execute-api.us-east-1.amazonaws.com/dev' --header 'Content-Type: application/json' --header 'Authorization: <ID_TOKEN>'

Conclusion

As we wrap up our journey into the world of modern API-driven applications, we hope this blog has illuminated the path to seamless integration. With AWS CDK, MongoDB Atlas, Cognito, and Lambda at your disposal, you're armed with the tools to craft dynamic, efficient, and scalable applications. The power of these technologies lies in your hands, and we encourage you to roll up your sleeves, dig into the code, and embark on your development adventure. The possibilities are boundless, and your next innovative application could be just a few lines of code away. So, go ahead and explore, experiment, and turn your ideas into reality with the combination of AWS CDK, MongoDB Atlas, Cognito, and Lambda. Your journey is just beginning, and the future of application development is at your fingertips.

Try Out
AWS CDK for MongoDB Atlas,
Amazon Cognito, and
AWS Lambda

Rollback



cdk destroy --all

Cost and Licenses
There is no cost to use this Partner Solution, but you will be billed for any AWS services or resources that this Partner Solution deploys. For more information, refer to the AWS Partner Solution General Information Guide.
This Partner Solution deploys MongoDB Atlas resources with the latest stable MongoDB enterprise version, which is licensed and distributed under the Server Side Public License (SSPL).

Semantic Search API: MongoDB Atlas Vector Search With Amazon Bedrock & AWS Serverless

babuvenky76 — Fri, 19 Jan 2024 08:19:00 +0000

Authors:
Amorosi, Andrea (Senior Solutions Architect at AWS)
Vogel, Pascal (Solutions Architect at AWS)
Doshi, Akash (Solutions Architect at AWS)

Contributor:
Babu Srinivasan (Senior Partner Solutions Architect at MongoDB)

Searching through large volumes of unstructured data to find the most relevant information is critical to many applications. However, traditional keyword-based search approaches often fall short when dealing with complex natural language queries.

Semantic search overcomes this challenge by understanding the meaning and purpose behind search queries. This comprehension improves the accuracy and relevance of search results by taking into account intent and meaning. Semantic search can be used with complex natural language queries and provides a contextual understanding of words and phrases based on different meanings in different situations.

These capabilities make semantic search a powerful approach for many search use cases, including enterprise knowledge, legal and medical documents, e-commerce products, and media libraries.

MongoDB Atlas Vector Search makes it easy to build semantic search by integrating the operational database and vector search into a single, fully managed platform with a native MongoDB interface that leverages large language models (LLMs) through popular frameworks.

Amazon Bedrock provides access to a range of high-performing foundation models (FMs), including LLMs, developed by leading AI companies such as Amazon, AI21 Labs, Anthropic, Cohere, Meta, and Stability AI. Amazon Bedrock is a serverless service that provides access to a variety of foundation models through a single API.

By using Amazon Bedrock to generate vector embeddings and storing them in MongoDB Atlas, you can quickly build powerful semantic search applications. Combining these technologies with cloud-native design patterns unlocks an intelligent semantic search back end that understands the nuances of language. It allows users to query information in natural language and discover highly relevant results — even if the query and keywords don’t match exactly.

With Amazon Bedrock and MongoDB Atlas, you benefit from comprehensive data protection and privacy. You can use AWS PrivateLink to establish private connectivity from these managed services to your Amazon Virtual Private Cloud (Amazon VPC) without exposing your traffic to the Internet.

This tutorial walks through an architecture for a scalable and secure semantic search API built using MongoDB Atlas Vector Search, Amazon Bedrock, and AWS serverless services. The accompanying GitHub repository contains code and detailed deployment details to get you started.

Solution Overview

The solution presented in this tutorial has two main features:

Generating vector embeddings (represented as 1,2,3 and 4 in the diagram)
Performing the semantic search (represented as A, B, and C in the diagram)

To generate vector embeddings:

The Create Embeddings AWS Lambda function can be invoked via an Amazon API Gateway REST API to generate an initial set of vector embeddings for documents stored in the MongoDB Atlas database.
Ongoing database changes are captured and published to an Amazon EventBridge event bus with an Amazon Simple Queue Service (Amazon SQS) queue as the target.
The Ingestion Lambda function receives change events from the SQS queue using Lambda event source mappings. It generates new or updates existing embeddings using the Titan Embeddings model via Amazon Bedrock.
The new or updated embeddings are stored in MongoDB Atlas via the private interface endpoint connection. AWS Secrets Manager is used for secure secret storage.

To perform semantic search:

A. Users submit their search queries to an API endpoint provided by the API Gateway REST API.
B. The Search Lambda function generates an embedding of the search query using the Titan Embeddings model via Amazon Bedrock. To ensure private connectivity, it uses an interface endpoint provided by AWS PrivateLink.
C. The Search function then performs a semantic search on the MongoDB Atlas vector search index using the interface endpoint for AWS PrivateLink. Results are returned to the client through the API Gateway.

The following sections describe these key architectural elements in more detail.

Generating vector embeddings with Amazon Bedrock and Titan Embeddings

This post uses the movies collection in the sample_mflix database as an example to illustrate the presented concepts. You can easily load this database as MongoDB sample data. Each document in the movies collection contains details on a single movie, such as title, runtime length, release date, genre, and IMDb rating. It also contains a plot field with a short summary of the movie’s plot. Let’s assume you want to enable semantic search on this plot field to allow your users to discover movies using natural language queries.

Semantic search relies on vector embeddings which convert words or phrases into numerical vectors of fixed size. As contextually similar words and phrases also produce similar vector representations, these vectors can capture the meaning of a text. Semantically similar words are mapped to proximate points in the vector space which allows semantic search algorithms to identify relevant search results. As a first step, you need to generate vector embeddings for the text stored in the plot field of each document.

Amazon Bedrock supports generating vector embeddings using the Titan Embeddings model (amazon.titan-embed-text-v1). This model can generate embeddings for a maximum input text of 8K tokens and generates vectors with up to 1536 dimensions. Atlas Vector Search currently supports indexing vector embeddings with up to 2048 dimensions.

This solution uses the AWS SDK for JavaScript v3 in the Search Lambda function to connect to the embedding model in Amazon Bedrock using the BedrockRuntimeClient.



import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

const client = new BedrockRuntimeClient();

const inputText = "Text to create embeddings for."

const input = {
  modelId: "amazon.titan-embed-text-v1", 
  contentType: "application/json",
  accept: "*/*",
  body: JSON.stringify({
    inputText,
  }),
};

const command = new InvokeModelCommand(input);
const response = await client.send(command);

After receiving the vector embeddings from Amazon Bedrock, the Lambda function uses the MongoDB driver for Node.js to store the generated vector embeddings for the plot field in a new plot_embedding field in the MongoDB document.

All the Lambda functions used in this solution securely connect from an isolated VPC to Amazon Bedrock and MongoDB Atlas using VPC interface endpoints provided by AWS PrivateLink. This enables access to both MongoDB Atlas and Amazon Bedrock as if they were in your VPC, without the use of an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. The path between a VPC endpoint and an AWS or AWS-based service stays within AWS and does not traverse the Internet.

Indexing vector embeddings and performing the semantic search with Atlas Vector Search

To store the vector embeddings of the plot text in the plot_embedding field, you can use a knnVector type field in MongoDB Atlas. The vector field is represented as an array of numbers (BSON int32, int64, or double data types only).

Next, you need to index the vector embeddings stored in the plot_embedding field of each document. MongoDB Atlas enables you to define a vector search index on knnVector type fields with the following configuration:



{
  "mappings": {
    "dynamic": true,
    "fields": {
      "plot_embedding": {
        "dimensions": 1536,
        "similarity": "cosine",
        "type": "knnVector"
      }
    }
  }
}

To perform search queries on this index, you can use a $vectorSearch aggregation pipeline stage. This search query compares the similarity of the vectors stored in the plot_embedding field with the vector representation of the search query submitted by the user. It uses an approximate nearest neighbor search approach.

A query can then look as follows:



{
  "$vectorSearch": {
    "index": "plot_embedding_index",
    "path": "plot_embedding",
    "queryVector": [<array-of-numbers>],
    "numCandidates": 50,
    "limit": 3,
  }
}

See the Vector Search Queries documentation for a detailed description of fields.

Change data capture with Atlas Triggers and Amazon EventBridge

Data is rarely static. To make new documents and documents where fields are updated searchable by semantic search, you can set up a process for automatically embedding new and re-embedding updated fields. For example, in the case of the movies dataset, you may need to update the plot of some of the movies, which in turn requires an update to the plot_embedding field for the document.

Atlas Triggers allow you to execute server-side logic in response to database events or on a schedule. Database triggers are a type of Atlas trigger that allows you to execute server-side logic whenever a document is added, updated, or removed in a linked Atlas cluster.

There are several ways to configure the types of events that cause a trigger to be executed. First, you can select one or more database change events (INSERT, UPDATE, REPLACE, and DELETE). Second, you can provide a match expression to further filter events based on their properties.

A database trigger can either execute a serverless function with your JavaScript code or send trigger events to an Amazon EventBridge partner event bus.

In the case of this sample application, all INSERT, UPDATE, and REPLACE change events are sent to an EventBridge event bus and placed on an Amazon Queue Service (Amazon SQS) queue. From there, the ingestion Lambda function consumes batches of change events via Lambda event source mappings and creates or updates embeddings for the plot_embeddings document field.

Use a match expression to only forward database events if the document in question either does not have a plot_embeddings field yet or if the plot field has changed:



{"updateDescription.updatedFields.plot":{"$exists":true}}

Serverless semantic search API with Amazon API Gateway and AWS Lambda

Finally, you need a scalable and secure API endpoint that you can integrate with your applications and expose to clients. This solution creates a REST API endpoint using Amazon API Gateway. Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. API Gateway offers multiple authentication options, built-in caching and request validation, and many other features that you can configure to integrate this semantic search solution into your project. As a serverless service, you benefit from automatic scaling, built-in high availability, and a pay-for-use billing model.

Clients send search requests to the /searchendpoint of the REST API and receive a list of relevant search results in response:



curl --request POST \
  'https://<API endpoint>.execute-api.us-east-1.amazonaws.com/prod/search' \
  --aws-sigv4 "aws:amz:us-east-1:execute-api" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
  --header 'Accept: application/json' \
  --data '{ "query": "sports" }' \
  | jq .

The response for this particular request would contain the first three movies, including the _id, title, plot, and score fields:



[
  {
    "_id": "573a1398f29313caabcea388",
    "plot": "Molly is a high school track coach who knows just as much about football as anyone else on the planet. When the football coach's position becomes vacant, she applies for the job, despite ...",
    "title": "Wildcats",
    "score": 0.7063020467758179
  },
  {
    "_id": "573a1397f29313caabce879f",
    "plot": "It started as a friendly meeting between 4 old buddies with their basketball coach and ended up in revealing the truth about their relationship. The meeting forces the five men to reveal ...",
    "title": "That Championship Season",
    "score": 0.6836512088775635
  },
  {
    "_id": "573a1394f29313caabcdf0a6",
    "plot": "Pat's a brilliant athlete, except when her domineering fiance is around. The lady's golf championship is in her reach until she gets flustered by his presence at the final holes. He wants ...",
    "title": "Pat and Mike",
    "score": 0.6823728084564209
  }
]

Under the hood, incoming search requests are routed from the API Gateway to the Search Lambda function using a Lambda proxy integration.

Because embeddings only need to be generated when new data is added or data is updated, event-driven computing with AWS Lambda allows embedding generation to be triggered on-demand rather than running continuously. AWS Lambda is a serverless computing service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.

Scaling and extending the solution

This solution serves as a blueprint that can be enhanced and extended to develop your use cases based on a semantic search with MongoDB Atlas and Amazon Bedrock. Keep the following considerations in mind when scaling the production solution.

The default Amazon Bedrock quotas implement rate limits for the API operations performed in this example application. For instance, the default quotas allow 2,000 requests per minute or 300,000 tokens processed per minute to invoke the Amazon Titan Embeddings model. Depending on the volume and size of your embedding API calls, you may need to configure provisioned throughput to get a higher level of throughput for a fixed cost.

With automatic scaling, built-in high availability, and a pay-for-use billing model, AWS Lambda is well-suited as a computing platform for embedding workloads. To ensure your Lambda functions can handle large numbers of invocations, such as ingesting large amounts of data at once, make sure to manage Lambda function concurrency appropriately. To do this, configure reserved concurrency and provisioned concurrency. For more information about scaling Lambda functions and configuring reserved and provisioned concurrency, see the AWS Lambda Developer Guide.

Consider enabling API Gateway caching to increase the responsiveness of the integration and to optimize the cost of repeat requests. Also, set up access logging for the API Gateway with Amazon CloudWatch to keep a record of who accessed your API endpoint and how. For an overview of security recommendations for API Gateway, see security best practices in Amazon API Gateway.

The integration presented in this tutorial follows security best practices such as storing your MongoDB credentials in Secrets Manager and utilizing IAM to secure access to resources in your AWS account. To protect your MongoDB account, you should regularly rotate your MongoDB credentials and update them in Secrets Manager.

Conclusion

This article demonstrates how to use MongoDB Atlas Vector Search, Amazon Bedrock, and AWS serverless services to build a secure and scalable semantic search API. This approach allows you to not only use MongoDB Atlas to store your data sets but also to unlock more value by using Atlas Vector Search alongside Amazon Bedrock's serverless API integrations.

The associated GitHub repository contains the solution source code and detailed deployment instructions to get you started. Open a GitHub issue to provide your feedback or create a pull request to extend the solution.

See the MongoDB Atlas Vector Search documentation for more information and tutorials.