DEV Community: Manato Takai

Building an Apache Iceberg Log Analytics Platform with S3 Tables and Amazon Data Firehose

Manato Takai — Tue, 23 Dec 2025 00:02:00 +0000

Introduction

Amazon S3 Tables is an AWS-managed object storage service that supports the Apache Iceberg specification and automatically performs table optimization tasks such as compaction in the background. In this article, I explore how to architect a log analytics platform for applications deployed on AWS using S3 Tables and Amazon Data Firehose.

Building an Apache Iceberg Log Analytics Platform on AWS

System Architecture

The system architecture for this implementation is shown in the diagram below. Assuming a containerized application deployed on ECS, I have placed firelens as a sidecar container to serve as the log router. firelens receives logs and forwards them to Amazon Kinesis Data Firehose. Firehose then stores the received logs in S3 Tables in Iceberg format. Finally, the stored Iceberg tables are queried and analyzed using Amazon Athena.

Infrastructure Provisioning

The infrastructure is provisioned using Terraform. I have included a link to the Terraform code repository below for those interested in the implementation details.

https://github.com/manaty226/aws-s3tables-firehose-athena-log-analytics

In this configuration, writing data from Amazon Data Firehose to S3 Tables requires LakeFormation permission settings. However, as of December 2025, Terraform's LakeFormation resource does not support permission configuration for S3 Tables. Therefore, after creating the IAM role for Kinesis Data Firehose with Terraform, you need to execute the following AWS CLI command to configure LakeFormation permissions. Without this configuration, S3 Tables will not be visible from the Firehose stream, causing resource creation to fail.

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier="arn:aws:iam::${ACCOUNT_ID}:role/s3tables-log-demo-firehose-role" \
  --resource "{\"Table\":{\"CatalogId\":\"${ACCOUNT_ID}:s3tablescatalog/${TABLE_BUCKET_NAME}\",\"DatabaseName\":\"logs\",\"Name\":\"some_api_logs\"}}" \
  --permissions "ALL" \
  --region ap-northeast-1

For more details on the LakeFormation permission configuration issue in Terraform, please refer to the following GitHub issue.
https://github.com/hashicorp/terraform-provider-aws/issues/40724

Log Sample Structure

For this implementation, the application container outputs logs in the following JSON format.

{
  "timestamp": "2025-12-20T10:10:21Z",
  "level": "INFO",
  "message": "Sample log message",
  "request_id": "93f7a013-37d1-4793-a73f-b660d78a1f16"
}

When these logs are sent to Amazon Data Firehose via FireLens, metadata is appended, resulting in the following structure.

{
    "container_name": "app",
    "source": "stdout",
    "log": "{\"timestamp\":\"2025-12-20T10:10:21Z\",\"level\":\"INFO\",\"message\":\"Sample log message\",\"request_id\":\"93f7a013-37d1-4793-a73f-b660d78a1f16\"}",
    "container_id": "xxxxxxxxxxxxxxxxx"
}

When data is sent from Amazon Data Firehose to S3 Tables, fields that do not exist in the S3 Tables schema are silently ignored. Therefore, you need to define an Iceberg table schema that accommodates this format in advance. Unlike CloudWatch Logs, where the log schema is parsed at query time after ingestion, with S3 Tables you must standardize the log fields. However, as application developers, we often need to modify log fields depending on specific functions or contexts. Given this, a reasonable approach might be to include the full log content in the log field while defining standardized fields as S3 Tables columns and configuring FireLens to format the logs accordingly. This is still an area I am experimenting with, so if you have any best practices to share, I would appreciate your input.

If you can verify the log data in the S3 Tables preview, the setup is successful. Note that when the JSON fields output by FireLens do not match the S3 Tables column schema, Amazon Data Firehose will not report an error, and the data will simply not be stored in S3 Tables, making debugging particularly challenging.

Querying Logs Stored in S3 Tables

Finally, I query the logs stored in S3 Tables from Athena.

As mentioned earlier, the log body is stored as a JSON string in the log field. Therefore, we need to parse the log field as JSON in Athena to extract individual fields. Below is an example query. I considered creating a View table to streamline analysis for the development team, but currently S3 Tables is recognized as a cross-account Glue Data Catalog, resulting in an error when attempting to execute the CREATE VIEW command. For now, the recommended approach is to save queries like the one below as Named Queries and share them within the team. If anyone knows how to create View tables in this context, I would be grateful for your guidance.

SELECT
    container_id,
    container_name,
    ecs_cluster,
    ecs_task_arn,
    ecs_task_definition,
    source,
    json_extract_scalar(log, '$.timestamp') AS timestamp,
    json_extract_scalar(log, '$.level') AS level,
    json_extract_scalar(log, '$.message') AS message,
    json_extract_scalar(log, '$.request_id') AS request_id
FROM some_api_logs;

Conclusion

In this article, I explored building an application log analytics platform using S3 Tables. While there are still areas that feel somewhat immature such as Terraform not yet supporting LakeFormation permission configuration, the need to set ALL permissions for Amazon Data Firehose, and the inability to create View tables—the potential for significantly lower log ingestion and storage costs compared to CloudWatch Logs is promising. Additionally, the optimization opportunities offered by Iceberg's compaction and index optimization based on data characteristics make this a technology I am looking forward to seeing evolve.

Building a Remote MCP Server with OAuth Authorization Using Amazon API Gateway and Cognito

Manato Takai — Wed, 13 Aug 2025 04:16:09 +0000

Introduction

In the specification of MCP server published at 2025-03-25, a new authorization specification for MCP servers using HTTP transport was proposed. This specification enables access using temporary user credentials via the OAuth 2.1 authorization code flow, allowing AI agents to access the server without embedding long-lived, highly privileged API keys.

Furthermore, the specification published at 2025-06-18 clarified that MCP server should be treated as a resource server, logically separating the authorization server and MCP server.

https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization

However, developers who want to implement a remote MCP server with OAuth authorization still need to provide an OAuth 2.1-compliant authorization server. Specifically, in AWS environments, Amazon Cognito does not support standard protocols or endpoints for dynamic client registration and authorization server metadata. Thus, it is difficult to comply with standard MCP authorization protocol which is required to communicate with general MCP clients like MCP Inspector or VSCode.

Additionally, managing extra compute resources for the authorization server should be undesired for the MCP server developers. Therefore, in this article, I will demonstrate how to implement a remote MCP server with OAuth authorization that works with MCP Inspector and VSCode, leveraging the request/response manipulating features of Amazon API Gateway.

Overview of Authorized MCP Server

The MCP server authorization specification is based on the following four RFCs (including drafts):

For OAuth 2.1, the authorization code flow is sufficient for this use case, and Cognito's standard features can handle it, so details are omitted here.

Protected Resource Server Metadata is used by the MCP server (as a resource server) to indicate the URL of the authorization server it relies on. In RFC 9728, the authorization_servers field is OPTIONAL, but in the MCP specification, it is mandatory to include information about one or more authorization servers. Additionally, when the MCP server responds with HTTP status 401, it must include a WWW-Authenticate header with a value like Bearer resource_metadata="https://resource.example.com/.well-known/oauth-protected-resource", indicating the URL for the resource metadata. This allows MCP clients to follow the resource metadata to obtain the authorization server URL.

Authorization Server Metadata and Dynamic Client Registration are required for MCP clients to interact with the authorization server. Authorization Server Metadata is mandatory, while DCR is optional. If DCR is not supported, some alternative method for registering MCP clients must be provided. However, since MCP Inspector assumes DCR for client registration, DCR is supported in this implementation to ensure standard operation with MCP Inspector.

The overall flow using these specifications is described in the MCP specification:
https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-flow-steps

Implementing an Authorized MCP Server with AWS Managed Services

To implement an authorized MCP server using AWS managed services, Amazon Cognito is used as the authorization server. Cognito supports the authorization code grant with PKCE and the client credentials grant, making it compatible with the MCP server's authorization flow.

However, Cognito does not support authorization server metadata endpoints or DCR. To address this, API Gateway is utilized to supplement the missing specifications.

The conceptual architecture is shown below. API Gateway routes requests to various endpoints. The HTTP API Gateway in the front is a workaround to place well-known endpoints (e.g., https://example.com/.well-known/oauth-protected-resource) directly under the host, as required by MCP clients like MCP Inspector. The REST API Gateway is the main resource for this implementation. Cognito is used as the authorization server, and API Gateway is integrated with AWS services to support DCR. Lambda is used for the MCP server tool implementation via proxy integration. Metadata endpoints are implemented using API Gateway's Mock integration.

By default, API Gateway REST APIs add a stage name to the root path. However, MCP clients like MCP Inspector access the /.well-known/ path directly under the host, as specified in RFC 8414, making it impossible to use REST API mode as-is. HTTP API mode allows defining the stage as $default to place paths directly under the host, but lacks features like Mock integration. Therefore, this architecture is used. In production, only REST API mode with a custom domain is required to set paths directly under the host.

The implementation is available in the following repository:
https://github.com/manaty226/remote-mcp-based-on-aws-managed-services

Implementing Metadata Endpoints with API Gateway Mock Integration

Authorization server and resource server metadata consist of static information such as identifiers and related endpoints. For the MCP server, this information is determined at AWS infrastructure setup time.

API Gateway provides a Mock integration feature for returning fixed values or simple computed results. Using this, you can implement metadata endpoints for the authorization server and resource server.

For example, the minimum required authorization server metadata for the MCP server can be implemented with a mapping template like this. The URLs for the issuer and various endpoints can be statically set using Cognito and API Gateway endpoint information.

{
  "issuer": "<Issuer URL>",
  "authorization_endpoint": "<Authorization Endpoint URL>",
  "token_endpoint": "<Token Endpoint URL>",
  "response_types_supported": ["code"],
  "grant_types_supported": ["authorization_code"],
  "code_challenge_methods_supported": ["S256"],
  "registration_endpoint": "<DCR Endpoint URL>"
}

Similarly, resource server metadata can be implemented as follows:

{
  "resource": "<API Gateway URL implementing the resource server>",
  "authorization_servers": ["<Authorization Server URL>"]
}

Supporting Dynamic Client Registration with API Gateway AWS Integration and Mapping Templates

Amazon Cognito provides an API called CreateUserPoolClient for client registration. However, the request and response formats are proprietary, so you cannot directly comply with the DCR specification.

https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_CreateUserPoolClient.html

To address this, you can use API Gateway's mapping template feature to transform request and response data, enabling DCR-compliant client registration with Cognito.

For example, you can implement the transformation from a DCR request to a Cognito client creation API request as follows:

{
  "ClientName": $input.json('$.client_name'),
  "CallbackURLs": $input.json('$.redirect_uris'),
  "AllowedOAuthFlows": $input.json('$.response_types'),
  "AllowedOAuthFlowsUserPoolClient": true,
  "AllowedOAuthScopes": ["openid", "profile"],
  "SupportedIdentityProviders": ["COGNITO"],
  "UserPoolId": "<Cognito User Pool ID>"
}

API Gateway mapping templates use the Velocity Template Language (VTL), allowing you to reference request body values like $input.json('$.client_name'). You can set these as values in the Cognito API request body. Required Cognito-specific settings can be hardcoded in the mapping template.

https://docs.aws.amazon.com/apigateway/latest/developerguide/models-mappings.html

You can also transform Cognito's proprietary response format to a DCR-compliant response for the MCP client:

{
  "client_id": $input.json('$.UserPoolClient.ClientId'),
  "client_name": $input.json('$.UserPoolClient.ClientName'),
  "redirect_uris": $input.json('$.UserPoolClient.CallbackURLs'),
  "response_types": $input.json('$.UserPoolClient.AllowedOAuthFlows')
}

MCP Tool Implementation

With all authorization features handled by AWS managed services, the resource server functionality (the MCP tool implementation) can be kept lightweight. The awslabs/mcp repository provides a library called mcp-lambda-handler for running MCP tool functions from the Lambda handler's event object.

https://github.com/awslabs/mcp/tree/8d90be5c403af4829a45d8f03093f830ffed6285/src/mcp-lambda-handler

For example, you can implement an MCP server that simply returns a UUID v4 in just a few lines:

import uuid
from awslabs.mcp_lambda_handler import MCPLambdaHandler

mcp_server = MCPLambdaHandler(name="sample-uuid-mcp", version="1.0.0")

@mcp_server.tool()
def get_uuid() -> str:
    """Generate a new UUID v4"""
    return str(uuid.uuid4())

def handler(event, context):
    return mcp_server.handle_request(event, context)

For API authorization, configure the API Gateway Cognito authorizer and add the WWW-Authenticate header to the gateway response for UNAUTHORIZED errors to route to the resource server metadata.

Verification by MCP Inspector

You can use MCP Inspector to verify that the authorized remote MCP server works as expected. Set the Transport Type in MCP Inspector to Streamable HTTP and enter the MCP endpoint of your API Gateway (e.g., https://<your-api-gw-host>/mcp) in the URL form. Then, click "Open Auth Setting" and select either "Quick OAuth Flow" or "Guided OAuth Flow" to start the authorization flow with your remote MCP server.

If successful, you will be able to retrieve metadata, perform dynamic client registration, and execute the authorization code flow to obtain an access token.

You can also connect and invoke tools from the left menu in MCP Inspector, or from AI agents such as VSCode.

Conclusion

In this article, I demonstrated how to implement a remote MCP server with OAuth authorization using AWS API Gateway and Cognito. By using API Gateway mapping templates to bridge protocol and AWS specification gaps, you can implement authorization without additional compute resources.

For simplicity, both the resource server and authorization server were implemented on the same API Gateway, but you can also separate the authorization server functionality using only API Gateway and Cognito, making the MCP server implementation even simpler.