Lambda-less outbound HTTP requests on AWS serverless

I would argue that almost all applications make some sort of network requests. Be it to fetch a collection of items or perform a mutation of a resource.

Network as means of transporting data is not exactly 100% reliable. Because of this, engineers employ various strategies to ensure data delivery. The most common of which might be retrying the request itself.

Would it not be nice for us not to write code for all that logic ourselves? Luckily one might leverage various AWS services to do just that – have the service itself worry about retries and data delivery.

Let us explore this topic by looking at a few AWS services and how they enable us to make network requests via configuration rather than code.

You can find all the code samples used in this blog post in this GitHub repository.

Using Amazon EventBridge API destinations

The Amazon EventBridge service took the serverless world by storm from its inception. Managed event bus with rich filtering capabilities was a great addition to the vast (some might argue too vast) family of AWS services.

On March 4, 2021, the Amazon EventBridge feature set expanded to include API destinations. This feature allows for a configuration-only way of making HTTP requests!

Other great developers wrote about this topic, so I will not be re-hashing their excellent work. For a much broader overview of this feature, look no further than this superb blog post (check out part 2 as well!).

If you want to get your hands dirty, I've published an example AWS SAM app that utilizes the API destinations and Amazon Cognito.

Using AWS Step Functions and Amazon API Gateway

It seems like one cannot write a blog post about low-code architectures on AWS without mentioning AWS Step Functions. No wonder! This service lets you integrate with other AWS services by writing declarative, AWS Step Functions specific, ASL code.

In our case, the most relevant integration is the integration with Amazon API Gateway and the underlying configuration of the Amazon API Gateway service. If we configure those pieces correctly, we will be able to make outbound HTTP requests from the AWS Step Functions state machine with configuration-only code.

Using Amazon API Gateway REST API and AWS CDK

Let us walk through the steps required to deploy such integration using AWS CDK as an IaC tool. I will be using an excellent https://webhook.site/ to simulate an external HTTP endpoint.

The first step is to create the Amazon API Gateway REST API.

import { aws_apigateway } from "aws-cdk-lib";

// class definition ...

const api = new aws_apigateway.RestApi(this, "api");

---

The second step is to create a method resource along with the integration. The following code snippet creates a GET Amazon API Gateway endpoint, which, when invoked, will perform a POST request with a static payload.

import { aws_apigateway } from "aws-cdk-lib";

// class and API definition ...

api.root.addMethod(
  "GET",
  new aws_apigateway.HttpIntegration("YOUR_API_ENDPOINT", {
    httpMethod: "POST",
    options: {
      requestTemplates: {
        "application/json": `{
            "static": "body_payload"
          }`
      },
      requestParameters: {
        "integration.request.header.Content-Type": "'application/json'"
      },
      integrationResponses: [{ statusCode: "200" }]
    },
    // Otherwise the mapping template is ignored.
    // The headers are mapped.
    proxy: false
  }),
  {
    methodResponses: [{ statusCode: "200" }]
  }
);

Let us unpack some of the configuration options.

• We need to specify the integrationResponses and methodResponses parameters due to the proxy: false setting.
  Without the proxy:false setting, the Amazon API Gateway would ignore what we have specified in the requestTemplates.

• The statusCode in the integrationResponse and methodResponse has to match. Otherwise, an error will be thrown at method invoke time and NOT deployment time.

• Keep in mind that this is an example. You should add more statusCode mappings for 4xx and 5xx HTTP response cases.

We have only scratched the surface of the Amazon API Gateway (REST API) 's possibilities regarding configuration. For a deep dive, refer to this excellent blog post by Alex DeBrie.

---

The third step is to create an AWS Step Functions state machine with a Task step which will invoke the endpoint we have just created. What follows is a minimal example of doing so.

import { aws_stepfunctions, aws_stepfunctions_tasks } from "aws-cdk-lib";

// Your api definition...

const stateMachineDefinition =
  new aws_stepfunctions_tasks.CallApiGatewayRestApiEndpoint(
    this,
    "callAPIGWTask",
    {
      api,
      method: aws_stepfunctions_tasks.HttpMethod.GET,
      stageName: "prod"
    }
  );

new aws_stepfunctions.StateMachine(this, "stateMachine", {
  definition: stateMachineDefinition
});

When the state machine runs, the AWS Step Functions service will invoke our Amazon API Gateway endpoint. From there, Amazon API Gateway will invoke the external API endpoint according to the configuration we have specified in the previous step.

Gotcha with queryParameters property

There is a particular way the values in the queryParameters of the CallApiGatewayRestApiEndpoint construct have to be formatted. I've spent a considerable amount of time figuring this out, so you might find it helpful as well.

Let us assume your external API requires you to provide a token query parameter. The token is fetched dynamically from the AWS Systems Manager Parameter Store in the state machine. We need to do two things to forward the token from the state machine to the external HTTP endpoint.

The first step is to amend the existing definition of the callAPIGWTask (the example assumes that the result of fetching the token is present under APIKey path).

const stateMachineDefinition =
  new aws_stepfunctions_tasks.CallApiGatewayRestApiEndpoint(
    this,
    "callAPIGWTask",
    {
      api,
      method: aws_stepfunctions_tasks.HttpMethod.GET,
      stageName: "prod",
+     queryParameters: aws_stepfunctions.TaskInput.fromObject({
+        "token.$": "States.StringToJson(States.Format('[\"{}\"]', $.APIKey))"
+      })
    }
  );

The need to use two of the AWS Step Functions intrinsic functions stems from how the integration between AWS Step Functions and Amazon API Gateway handles query parameters – the value has to be a valid JSON array of string values.

The second step is to amend the method resource on the Amazon API Gateway side.

api.root.addMethod(
  "GET",
  new aws_apigateway.HttpIntegration(
    "YOUR_API_ENDPOINT",
    {
      httpMethod: "POST",
      options: {
        requestTemplates: {
          "application/json": `{
                "static": "body_payload"
              }`
        },
        requestParameters: {
          "integration.request.header.Content-Type": "'application/json'",
+         "integration.request.querystring.token":
+           "method.request.querystring.token"
        },
        integrationResponses: [{ statusCode: "200" }]
      },
      proxy: false
    }
  ),
  {
    methodResponses: [{ statusCode: "200" }],
+   requestParameters: {
+     "method.request.querystring.token": true
+   }
  }
);

The above addition tells Amazon API Gateway to forward the token query parameter from the request to the method (which will, in turn, forward it to our external API endpoint).

Using Amazon API Gateway HTTP API and AWS CDK

Utilizing the Amazon API Gateway HTTP API for making outbound HTTP requests works on the same basis as the REST API version. The most significant difference is the configuration depth of the integration. The HTTP API integration is much more streamlined in terms of ways one might configure it.

Whenever I'm faced with choosing between those two options, I always fall back to this helpful documentation page that depicts the differences between those integrations.

To avoid making this blog post overly long, I have decided not to include the configuration steps (very similar to what we have done in the previous section) in this blog post. You could find the implementation in this example GitHub repository.

Using Amazon SNS

This article would not be complete without mentioning the Amazon Simple Notification Service. Along with its filtering capabilities, the service allows for dispatching a large number of messages.

One of the medium of the data transports the service handles is HTTP. Though much less sophisticated than the approaches that we have looked at so far (mainly the inability to transform the incoming payload), it still might come in handy in some scenarios.

The following is an IaC AWS CDK code that declares an Amazon SNS topic and an HTTP subscription.

import { aws_sns } from "aws-cdk-lib";

// class definition ...

const topic = new aws_sns.Topic(this, "topic");
new aws_sns.Subscription(this, "subscription", {
  // Keep in mind the filtering capabilities!
  endpoint: "YOUR_API_ENDPOINT",
  protocol: aws_sns.SubscriptionProtocol.HTTPS,
  topic
});

The most crucial thing to keep in mind is that the endpoint has to be compatible with SNS subscriptions. Do not be surprised when a request is made to the endpoint whenever you deploy your architecture – it is the Amazon SNS requesting the subscription confirmation request!

Using AWS AppSync

I've added this section after feedback from my colleague Ross Williams. Thank you!

AWS AppSync with its managed GraphQL offering is an excellent service for building modern APIs. Used by startups and corporations alike, since its inception allowed developers to achieve tremendous feature velocity.

If you want to learn more about the AWS AppSync, I would recommend getting AppSync Masterclass course.

Service allows us to retrieve data from various data sources. In the context of this blog post, the most relevant of them being the HTTP data source. The HTTP data source and the accompanying HTTP resolver will allow us to make HTTP outbound requests without deploying any AWS Lambda functions.

AWS already has a great tutorial on how to write and deploy the architecture we are interested in, so I will not be repeating their words. Before diving into implementation, though, keep in mind that, to my best knowledge, HTTP resolvers cannot specify any kind of resiliency configuration.

Closing words

Did I miss how one might perform an outbound HTTP request without using AWS Lambda service? (keeping the scope to "serverless" services). Please let me know. I'm eager to learn more!

For more similar content, consider following me on Twitter - @wm_matuszewski.

Thank you for your time.

Comments

[rosswilliams]

AppSync http resolver.

What I want, but I don’t think exists in any service, is to dynamically set the domain at call-time. My guess is a security or abuse prevention need from AWS.

[Wojciech Matuszewski]

Oh, I was unaware of the AppSync HTTP resolver – updating stuff as I write this should land today/tomorrow.