CloudWatch Logging for Web Apps (Part 2)

In the previous post, I covered the CloudFormation template for creating a REST API to log to CloudWatch from a frontend app. This post covers the communication protocol of the REST API.

Overview

Each app has a dedicated log group, created by the CloudFormation template, to make it easy to track all log messages for the app. Log groups contain log streams, which themselves contain chronologically ordered log entries.

The app is responsible for creating the log stream. For single page apps (SPA), such as Blazor WebAssembly, the log stream can be created when the app is loaded. Creating a log stream per app session has the benefit that the log entries show the chronological sequence of operations done by a user.

Once a log stream is created, the app can then send log entries to it. Log entries are sent as a batch operation. After the first batch, the CloudWatch API requires a sequence token for each subsequent batch. The sequence token is obtained in the response from the preceding batch operation.

I will leave the details of how to batch log entries for the next and final post. This post merely focuses on the protocol we will need to implement for it.

API Key

The CloudFormation template from the previous post created an API key using the CloudFormation stack ID to limit access to the REST API. This API key needs to be communicated to the frontend app via a JSON configuration file, for example. By basing the API key on the stack ID, the API key is different for each deployment, but ultimately a malicious actor could load the JSON file with the API key and spam the REST API. Unfortunately, for frontend apps, there is no technique to safely pass an API key without someone else getting a hold of it.

In LambdaSharp, the API key is made of two parts: the stack ID and the build GUID of the app assembly. For this tutorial, I skipped the build GUID part, because it can only be done reliably with tooling and only applies to .NET apps. Note the LambdaSharp approach does not make it safer, only harder for a third party to obtain the API key.

REST API

The logging REST API has two endpoints: one for creating a log stream and another for sending log entries to it.

POST:/.app/logs - Create Log Stream

The POST:/.app/logs endpoint creates a new log stream in the app log group. A log stream is a sequence of log events that originate from an app instance.

There is no limit on the number of log streams that can be created. There is a limit of 50 requests-per-second on this operations, after which requests are throttled.

The log stream name must match the following guidelines:

• Log stream names must be unique within the log group.
• Log stream names can be between 1 and 512 characters long.
• The ':' (colon) and '*' (asterisk) characters are not allowed.

Request Syntax

{
   "logStreamName": "string"
}

Request Parameters

The request accepts the following data in JSON format.

• logStreamName (required): The name of the log stream. Minimum length of 1. Maximum length of 512. Value must match pattern: [^:*]*

Success Response (HTTP Status Code: 200)

On success, the API responds with an empty JSON document.

{ }

Bad Request Response (HTTP Status Code: 400)

On a Bad Request response, the body contains a message describing why the request was rejected. Additional details can be found in the API logs when enabled.

Example: request body is missing required fields

{
    "error": "Invalid request body"
}

Example: request validation error

{
  "error": "1 validation error detected: Value \'\' at \'logStreamName\' failed to satisfy constraint: Member must have length greater than or equal to 1"
}

Internal Error Response (HTTP Status Code: 500)

On an Internal Error response, the body contains a generic message. The actual reason can be found in the API logs when enabled.

{
   "error": "Unexpected response from service."
}

PUT:/.app/logs - Put Log Messages

The PUT:/.app/logs endpoint uploads a batch of log messages to the specified log stream.

The request must include the sequence token obtained from the response of the previous call, unless it is the first request to a newly created log stream. Using the same sequenceToken twice within a narrow time period may cause both calls to be successful or one might be rejected.

The batch of events must satisfy the following constraints:

• The maximum batch size is 1,048,576 bytes. This size is calculated as the sum of all event messages in UTF-8, plus 26 bytes for each log event.
• None of the log events in the batch can be more than 2 hours in the future.
• None of the log events in the batch can be older than 14 days or older than the retention period of the log group.
• The log events in the batch must be in chronological order by their timestamp. The timestamp is the time the event occurred, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
• A batch of log events in a single request cannot span more than 24 hours. Otherwise, the operation fails.
• The maximum number of log events in a batch is 10,000.
• There is a quota of 5 requests per second per log stream. Additional requests are throttled. This quota cannot be changed.

Request Syntax

{
   "logEvents": [
      {
         "message": "string",
         "timestamp": number
      }
   ],
   "logStreamName": "string",
   "sequenceToken": "string"
}

Request Parameters

The request accepts the following data in JSON format.

• logEvents (required): The log events. Minimum number of 1 item. Maximum number of 10,000 items.
  • message (required): The raw event message. Minimum length of 1.
  • timestamp (required): The time the event occurred, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. Minimum value of 0.
• logStreamName (required): The name of the log stream. Minimum length of 1. Maximum length of 512. Value must match pattern: [^:]
• sequenceToken (optional): The sequence token obtained from the response of the previous call. An upload in a newly created log stream does not require a sequence token. Using the same sequenceToken twice within a narrow time period may cause both calls to be successful or one might be rejected.

Success Response (HTTP Status Code: 200)

On success, the API responds with a JSON document containing the sequence token for the next request.

{
  "nextSequenceToken": "49608818592289528730168753288679022865213175397425034930"
}

Bad Request Response (HTTP Status Code: 400)

Example: request body is missing required fields

{
    "error": "Invalid request body"
}

Example: request validation error

{
  "error": "1 validation error detected: Value \'\' at \'logStreamName\' failed to satisfy constraint: Member must have length greater than or equal to 1"
}

Example: The sequenceToken field is either missing or reusing a previous token value

{
  "error": "The given batch of log events has already been accepted. The next batch can be sent with sequenceToken: 49608818592289528730168753288679022865213175397425034930",
  "nextSequenceToken": "49608818592289528730168753288679022865213175397425034930"
}

Internal Error Response (HTTP Status Code: 500)

On an Internal Error response, the body contains a generic message. The actual reason can be found in the API logs when enabled.

{
   "error": "Unexpected response from service."
}

Conclusion - To be concluded...

In this post, we covered the communication protocol for a frontend apps to log to CloudWatch directly. In the next post, we will conclude this series with how to implement it in the frontend.

Happy Hacking!