Mastering Dynamic Response Streaming Lambda with AWS Bedrock and TypeScript

In the dynamic landscape of serverless architecture, where efficiency and real-time responsiveness reign supreme, the concept of response streaming emerges as a pivotal player. Unlike traditional request-response models, response streaming allows for the incremental transmission of data, enabling a continuous flow of information from the server to the client.

Imagine a scenario where large datasets need to be processed and delivered to end-users in real-time, or where updates to a client application should be instantaneously reflected. Response streaming transforms the serverless paradigm by facilitating the progressive delivery of results, enhancing user experience, and optimizing resource utilization.

In this comprehensive guide, we’ll walk you through the intricacies of setting up AWS Bedrock with AWS CDK, crafting a TypeScript Lambda function for response streaming, and seamlessly integrating it all with AWS CDK. By the end of this journey, you’ll be well-equipped to enhance your serverless applications with unparalleled efficiency and responsiveness.

How to Setup Bedrock with AWS CDK

Check out my previous blog on “Unleashing the Power of Generative AI with AWS Bedrock, AWS CDK, TypeScript” for a comprehensive guide on setting up Bedrock with AWS CDK.

Example Lambda for Leveraging Response Streaming

Lambda response streaming allows your functions to send back a response in chunks, enabling real-time updates and more efficient handling of large datasets. This feature is especially powerful in scenarios where immediate and incremental processing of data is crucial.

Let’s dive into a TypeScript example that showcases the power of response streaming

import {
  BedrockRuntimeClient,
  InvokeModelWithResponseStreamCommand,
} from "@aws-sdk/client-bedrock-runtime";

import { streamifyResponse } from "lambda-stream";

import * as stream from 'stream';
import * as util from 'util';

const { Readable } = stream;
const pipeline = util.promisify(stream.pipeline);

function parseBase64(message: string) {
  return JSON.parse(Buffer.from(message, "base64").toString("utf-8"));
}

const client = new BedrockRuntimeClient({
  region: "us-west-2",
});

export const handler = streamifyResponse(
  async (event, responseStream, _context) => {

    const prompt = 'Can you please what is Pure Function in React?'

    const claudPrompt = `Human: Human:${prompt} Assistant:`;

    const params = {
      modelId: "anthropic.claude-v2",
      contentType: "application/json",
      accept: "*/*",
      body: `{"prompt":"${claudPrompt}","max_tokens_to_sample":2048,"temperature":0.5,"top_k":250,"top_p":0.5,"stop_sequences":[], "anthropic_version":"bedrock-2023-05-31"}`,
    };

    console.log(params);

    const command = new InvokeModelWithResponseStreamCommand(params);

    const response: any = await client.send(command);
    const chunks = [];

    for await (const chunk of response.body) {
      const parsed = parseBase64(chunk.chunk.bytes);
      chunks.push(parsed.completion);
      responseStream.write(parsed.completion);
    }

    console.log(chunks.join(""));
    responseStream.end();
  }
);
export default handler;

How to Configure Lambda Function URL as Response Stream in AWS CDK

Now that we have a TypeScript Lambda function capable of response streaming, the next step is to seamlessly integrate it with AWS CDK. This integration allows for better management and deployment of the serverless infrastructure.

const bedrockResponseStreamingLambda = new aws_lambda_nodejs.NodejsFunction(
      this,
      "BedrockResponseStreamingLambda",
      {
        runtime: aws_lambda.Runtime.NODEJS_16_X,
        handler: "handler",
        entry: path.join(__dirname, "../src/lambda/bedrock-repsone-streaming/index.ts"),
        bundling: {
          forceDockerBundling: false,
          nodeModules:['lambda-stream']
        },
        timeout: Duration.seconds(90),
      }
    );

const lambdaUrl = bedrockResponseStreamingLambda.addFunctionUrl({
      authType: aws_lambda.FunctionUrlAuthType.NONE,
      invokeMode: aws_lambda.InvokeMode.RESPONSE_STREAM
    });

new CfnOutput(this,'LambdaEndpoint',{
      value:lambdaUrl.url
    })

Testing the Configured Lambda Function

Before diving into testing, ensure you’ve deployed your infrastructure by running the cdk deploy command in your terminal.

To make testing even more straightforward, check out this video demonstration that walks you through the process of testing your Lambda function

For those who prefer the command line, you can use the AWS CLI along with CURL to test your Lambda function. Here’s an example CURL command:

curl — request GET “lambda_url" — user AWS_ACCESSS_KEY:AWS_SECRET_KEY — aws-sigv4 “aws:amz:us-west-2:execute-api”

Conclusion

As we wrap up our exploration of AWS Bedrock and Lambda Response Streaming, it’s evident that the synergy between these technologies can elevate your serverless architecture to new heights. By following the steps outlined in this blog, you’ve empowered your applications with enhanced efficiency and dynamic response capabilities.

Checkout my Github repository

Happy Coding :)