DEV Community

Cover image for Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK - Part 2 Create and store text and image embeddings
AWS Heroes profile image Vadym Kazulkin
Vadym Kazulkin for AWS Heroes

Posted on • Originally published at vkazulkin.com

Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK - Part 2 Create and store text and image embeddings

#aws #amazonnova #amazons3vectors #java

Introduction

In part 1 of the series, we introduced the goal of this series and introduced Amazon Nova 2 Multimodal Embeddings and Amazon S3 Vectors. In this part, we'll cover creating and storing text and image embeddings. The code examples can be found in my GitHub repository amazon-nova-2-multimodal-embeddings. Please give it a star if you like it, and follow me on GitHub for more examples.

Create and store text embeddings

Let's first declare the most important dependencies to AWS SDK for Java: bedrockruntime, s3vectors, and s3 in the pom.xml :

<dependency>
   <groupId>software.amazon.awssdk</groupId>
   <artifactId>bedrockruntime</artifactId>
</dependency>
<dependency>
   <groupId>software.amazon.awssdk</groupId>
   <artifactId>s3vectors</artifactId>
</dependency>
<dependency>
   <groupId>software.amazon.awssdk</groupId>
   <artifactId>s3</artifactId>
</dependency>
Enter fullscreen mode Exit fullscreen mode

The business logic of our sample application can be found in the AmazonNovaMultimodalEmbeddings.

First, we'll cover creating and storing the text embeddings. The corresponding method is createAndSaveTextEmbeddings :

private static void createAndStoreTextEmbeddings(String text, String key) throws Exception {
   Float[] embeddings = createTextEmbeddings(text, "GENERIC_INDEX");
   putVectors(embeddings, key);
}
Enter fullscreen mode Exit fullscreen mode

Let's look at the creation of text embeddings first. We use a string template to create a JSON request to later be used in the InvokeModelRequest. We set taskType as SINGLE_EMBEDDING, embeddingPurpose as GENERIC_INDEX (as we'll create multimodal embeddings from text, image, audio, and video). After it, we set embeddingDimension as 384. We can also use 4-dimensional sizes to trade off embedding accuracy and vector storage cost: 3072, 1024, 384, and 256. Last, we set the text to create embeddings from. For the complete embeddings request and response schema, I refer to the following article. Below is the complete source code of this method:

private static Float[] createTextEmbeddings(String text, String embeddingPurpose) throws Exception {
 String request = """
    {
        "taskType": "SINGLE_EMBEDDING",
         "singleEmbeddingParams": {
          "embeddingPurpose": {{embeddingPurpose}},
              "embeddingDimension": {{dimension}},
              "text": {"truncationMode": "END", "value": {{text}} }
        }
    }"""
  .replace("{{text}}", "\"" + text + "\"")
  .replace("{{dimension}}", String.valueOf(EMBEDDING_DIMENSION))
  .replace("{{embeddingPurpose}}", "\"" + embeddingPurpose + "\"");

  var eResponse = invokeBedrockModel(request);
  return eResponse.embeddings().getFirst().embedding();
 }
Enter fullscreen mode Exit fullscreen mode

Now we need to invoke the corresponding Bedrock Model. For that, we need to build an InvokeModelRequest object by setting the text to create embeddings from, the JSON request converted to SDKBytes. We also set the model id as amazon.nova-2-multimodal-embeddings-v1:0. Then we use Bedrock Runtime Client to invoke the InvokeModelRequest synchronously and map the JSON response to the EmbeddingResponse object. This object contains the embedding type and the embeddings themselves:

private static EmbeddingResponse invokeBedrockModel(String request) throws Exception {
   var imRequest =   InvokeModelRequest.builder().modelId(MODEL_ID)
  .body(SdkBytes.fromUtf8String(request))
  .contentType("application/json")
  .accept("application/json").build();

   var imResponse = BEDRDOCK_RUNTIME_CLIENT.invokeModel(imRequest);
   return MAPPER.readValue(imResponse.body().asUtf8String(),    EmbeddingResponse.class);
}
Enter fullscreen mode Exit fullscreen mode

Now that we have created our embeddings, let's store them in S3 Vectors.
First, let's create an S3 Vector bucket and index. Please rename the S3 Vector Bucket name in the section private static final String VECTOR_BUCKET = "vk-vector-store" to be unique for you. We build a CreateVectorBucketRequest object and pass the vector bucket name, and use the S3 Vectors Client to create the vector bucket. Then we build a CreateIndexRequest object by setting the S3 vector bucket and index name (in my case, the index name is embeddings), dimension, and distance metric as cosine. Another option is to use euclidean distance metric. Then we use the S3 Vectors Client to create the vector's index. Below is the complete source code of this method:

private static void createS3VectorBucketAndIndex() {
  var cvbRequest = CreateVectorBucketRequest.builder()
   .vectorBucketName(VECTOR_BUCKET).build();

  S3_VECTORS_CLIENT.createVectorBucket(cvbRequest);

  var ciRequest = CreateIndexRequest.builder()
    .vectorBucketName(VECTOR_BUCKET).indexName(INDEX_NAME)
    .dataType("float32")
    .dimension(EMBEDDING_DIMENSION)
    .distanceMetric("cosine").build();

  S3_VECTORS_CLIENT.createIndex(ciRequest);
}
Enter fullscreen mode Exit fullscreen mode

After creating S3 vectors and an index, we can see them in the Amazon S3 Vector buckets section like this:

Now we can store the previously created text embeddings in the S3 Vectors. We first convert embeddings into the VectorData object, which we use in the PutInputVector object, and additionally pass the previously constructed key. Optionally, we can also set the filterable metadata (which we don't use in our example). Then we build the PutVectorsRequest object by passing the S3 Vectors bucket and index (created in the previous step) and the PutInputVector object. Finally, we use the S3 Vectors Client to put the vector into the S3 Vectors. Please note that we can also store the collection of the embeddings in the S3 Vectors in one PutVectorsRequest. This helps to improve the performance of storing all your embeddings at once. We can do it by passing the collection of embeddings as a parameter of the float32 method when building the VectorData object.

Below is the complete source code of this method:

private static void putVectors(Float[] embeddings, String key) {
  var vd = VectorData.builder().float32(embeddings).build();
  var piv = PutInputVector.builder().data(vd)
     .key(key)
     //.metadata(document)
     .build();

  var pvRequest = PutVectorsRequest.builder()
   .vectorBucketName(VECTOR_BUCKET)
   .indexName(INDEX_NAME)
   .vectors(piv).build();

  S3_VECTORS_CLIENT.putVectors(pvRequest);
}
Enter fullscreen mode Exit fullscreen mode

To test creating and storing the text embeddings, we can uncomment these invocations in the main method. The second parameter represents the key used in the PutInputVector object.

public static void main(String[] args) throws Exception {
  createAndStoreTextEmbeddings(AWS_LAMBDA_EMBEDDINGS,"AWS Lambda  Definition");
  createAndStoreTextEmbeddings(AZURE_FUNCTIONS__EMBEDDINGS,"Azure Functions Definition");    
}
Enter fullscreen mode Exit fullscreen mode

Create and store image embeddings

We'll re-use many parts from the process of creating and storing text embeddings described above. The relevant business logic of our sample application can still be found in the AmazonNovaMultimodalEmbeddings.

The relevant method here is createAndStoreImageEmbeddings. The main difference between creating text and image embeddings is that for creating image embeddings, the JSON request contains the image element instead of the text. In the image element, we specify image format as jpeg and the source and s3Location of the image, which is in our case S3 URI. I uploaded both test images, which visually describe AWS Lambda and Azure Functions, to my S3 Bucket specially created for this purpose, declared in the code as private final static String S3_BUCKET = "s3://vk-amazon-nova-2-mme/" (please define your own S3 Bucket for this purpose). Then we iterate over image names defined as private static final String[] IMAGE_NAMES = { "AWS-Lambda", "Azure-Functions" }. In this step, we build S3_IMAGE_URI for each image by using the S3 bucket, image name, and extension.  Then, for each image, we create the JSON request and invoke the Bedrock model to create image embeddings. After it, we store them in S3 Vectors by passing the image name as a key exactly the same way as we did for the text embeddings.

Below is the complete source code of this method:

private static void createAndStoreImageEmbeddings() throws Exception {
  for (String imageName : IMAGE_NAMES) {
    String request = """
    {
        "taskType": "SINGLE_EMBEDDING",
        "segmentedEmbeddingParams": {
             "embeddingPurpose": "GENERIC_INDEX",
             "embeddingDimension": {{dimension}},
             "image": {
            "format": "jpeg",
            "source": {
                "s3Location": {"uri": {{S3_IMAGE_URI}} }
             }
            }
        }
      }"""
     .replace("{{S3_IMAGE_URI}}", "\"" + S3_BUCKET + imageName + IMAGE_EXTENSION + "\"")
     .replace("{{dimension}}", String.valueOf(EMBEDDING_DIMENSION));

   var eResponse = invokeBedrockModel(request);
   putVectors(eResponse.embeddings().getFirst().embedding(), imageName);
   }
}
Enter fullscreen mode Exit fullscreen mode

To test creating and storing the image embeddings, we can uncomment this invocation in the main method.

public static void main(String[] args) throws Exception {
  createAndStoreImageEmbeddings();
}
Enter fullscreen mode Exit fullscreen mode

Conclusion

In this part, we covered creating text and image embeddings with Amazon Nova 2 Multimodal Embeddings and storing them in Amazon S3 Vectors using AWS Java SDK. In the next part of the series, we'll look at audio and video embeddings.

Top comments (0)