Migrating a TypeScript Serverless API from Serverless Framework to Terraform with Datadog Observability

I originally built this project as a TypeScript CRUD API using the Serverless Framework, AWS Lambda, API Gateway, and DynamoDB.

The application worked, but I wanted to understand the infrastructure at a deeper level instead of relying on one framework to generate most of the deployment configuration for me.

I therefore rebuilt the infrastructure with Terraform and added Datadog observability so I could answer practical operational questions:

• Which Lambda function is slow?
• How much memory does each function use?
• What happens during a cold start?
• Can I follow one request from API Gateway through Lambda to DynamoDB?
• Can application logs be correlated with traces?
• How do I turn dashboard metrics into actionable alerts?
• How do I protect an existing DynamoDB table during terraform destroy?

This article walks through the migration step by step, including the code changes, Terraform structure, Datadog instrumentation, debugging, dashboards, monitors, and cleanup.

Deployment status: The demonstration infrastructure has been destroyed
to avoid ongoing cloud costs. The Terraform configuration remains available
for reproduction.

Source code: Serverless Products API on GitHub

Contents

• Starting from the Serverless Framework application
• Modernizing the Lambda handlers
• Organizing the Terraform project
• Planning and applying the infrastructure
• Securing the Datadog API key
• Debugging the Datadog Extension
• Building the Datadog dashboard
• Adding proactive monitors
• Cost controls and cleanup
• What I learned

Prerequisites

To follow this migration, you will need:

• Node.js 22 and npm
• Terraform
• AWS CLI credentials for a development account
• an existing DynamoDB table named ProductsTable
• a Datadog account and API key
• Postman or another HTTP client

What we are building

The application exposes five endpoints:

Method	Route	Handler
POST	/products	createProduct
GET	/products	listProducts
GET	/products/{id}	getProductById
PUT	/products/{id}	updateProduct
DELETE	/products/{id}	deleteProduct

The final architecture is:

Client / Postman
       |
       v
API Gateway HTTP API
       |
       v
Five Node.js Lambda functions
       |
       v
Amazon DynamoDB

Datadog Lambda Library + Datadog Extension
       |
       +-- Distributed traces
       +-- Enhanced Lambda metrics
       +-- Structured logs
       +-- Dashboards
       +-- Threshold monitors

Final architecture

---

1. Starting from the Serverless Framework application

The original application used serverless.yml to define Lambda functions, API Gateway routes, environment variables, IAM permissions, and packaging.

A shortened example:

#app metadata
org: geocoderserverless 
app: my-serverless-app
service: mini-ecommerce-api-serverless

frameworkVersion: "4"

provider:
  name: aws
  runtime: nodejs18.x
  region: ca-central-1 ## Change the ca-central to the region of choice
  environment: ## This binds your env variable to the dynamoDB product table
    PRODUCTS_TABLE:
      Ref: ProductsTable  
## Lambda permissions.
  iamRoleStatements: 
    - Effect: Allow
      Action:
        - dynamodb:PutItem
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      Resource: arn:aws:dynamodb:ca-central-1:${aws:accountId}:table/ProductsTable ## Change the ca-central to the region of choice
## handler build configuration
package: 
  individually: true ## This causes each lambda function to be built and packaged individually, this was done to save space
  exclude:
    - node_modules/**
    - .serverless/**
    - .git/**
    - test/**
    - README.md
    - package-lock.json

### Handler functions
functions:
  createProduct:
    handler: src/handlers/createProduct.createProduct ## The handler to create products
    events:
      - http:
          path: products
          method: post

  listProducts:
    handler: src/handlers/listProducts.listProducts
    events:
      - http:
          path: products
          method: get

  getProductById:
    handler: src/handlers/getProductById.getProductById
    events:
      - http:
          path: products/{id}
          method: get

  updateProduct:
    handler: src/handlers/updateProduct.updateProduct
    events:
      - http:
          path: products/{id}
          method: put

The Serverless Framework was useful because it provided a concise deployment abstraction. However, I wanted explicit control over the underlying AWS resources and lifecycle.

The migration goals were:

1. Preserve the API behaviour.
2. Preserve the existing DynamoDB table.
3. Move infrastructure ownership to Terraform.
4. Package each Lambda independently.
5. Modernize the handlers to AWS SDK v3.
6. Add structured logging.
7. Add Datadog traces, metrics, dashboards, and monitors.
8. Make cleanup safe and predictable.

---

2. Modernizing the Lambda handlers

Step 1: Move to AWS SDK v3

Before migrating the infrastructure to Terraform, I first updated the application code.

This gave me a working application baseline before introducing infrastructure and observability changes. It also made it easier to determine whether a problem came from the Lambda code, Terraform configuration, or Datadog instrumentation.

The main changes were:

• migrating to AWS SDK for JavaScript v3
• adding TypeScript types for API Gateway events and responses
• validating request payloads
• returning consistent HTTP responses
• adding structured JSON logs
• keeping the DynamoDB client outside the handler for execution-environment reuse

The application contains five handlers:

Handler	DynamoDB operation	Responsibility
createProduct	PutCommand	Creates a product
listProducts	ScanCommand	Returns all products
getProductById	GetCommand	Retrieves one product
updateProduct	UpdateCommand	Updates an existing product
deleteProduct	DeleteCommand	Deletes a product

Rather than reproducing the same initialization and response-handling code five times, I will walk through the complete createProduct handler and then highlight the DynamoDB operation that makes each remaining handler different.

Implementing createProduct

The create handler performs the following steps:

1. Reads and validates the incoming JSON request.
2. Generates a unique product ID.
3. Builds the DynamoDB item.
4. Stores the item using PutCommand.
5. Writes a structured application log.
6. Returns a 201 Created response.

import { randomUUID } from 'node:crypto';
import { ConditionalCheckFailedException } from '@aws-sdk/client-dynamodb';
import { PutCommand } from '@aws-sdk/lib-dynamodb';
import { APIGatewayProxyHandlerV2 } from 'aws-lambda';
import type { CreateProductInput, ProductItem } from '../interfaces/Product';
import dynamoDB from '../utils/DynamoDbClient';
import { getProductsTableName } from '../utils/config';
import { jsonResponse, parseJsonBody } from '../utils/http';
import { logError, logInfo, logWarn } from '../utils/logger';
import { validateCreateProduct } from '../utils/validation';

// Handler definition
export const createProduct: APIGatewayProxyHandlerV2 = async (
  event,
  context,
) => {
  const requestId = context.awsRequestId;
  try {
    const input = parseJsonBody<CreateProductInput>(event.body);
    const validationErrors = validateCreateProduct(input);

    if (validationErrors.length > 0) {
      logWarn('Product validation failed', {
        requestId,
        operation: 'createProduct',
        validationErrors,
      });
      return jsonResponse(400, {
        success: false,
        errors: validationErrors,
      });
    }
    const now = new Date().toISOString();
    const id = randomUUID();

    const product: ProductItem = {
      PK: `PRODUCT#${id}`,
      SK: 'PRODUCT',
      entityType: 'PRODUCT',
      id,
      name: input.name.trim(),
      price: input.price,
      createdAt: now,
      updatedAt: now,
      ...(input.description?.trim()
        ? { description: input.description.trim() }
        : {}),
    };
    await dynamoDB.send(
      new PutCommand({
        TableName: getProductsTableName(),
        Item: product,
        ConditionExpression: 'attribute_not_exists(PK)',
      }),
    );
    // send a success log
    logInfo('Product created', {
      requestId,
      operation: 'createProduct',
      productId: id,
    });
    const {
      PK: _pk,
      SK: _sk,
      entityType: _entityType,
      ...publicProduct
    } = product;

    return jsonResponse(201, {
      success: true,
      data: publicProduct,
    });
  } catch (error) {
    if (error instanceof SyntaxError) {
      logWarn('Invalid JSON request body', {
        requestId,
        operation: 'createProduct',
      });
      return jsonResponse(400, {
        success: false,
        error: error.message,
      });
    }

    if (error instanceof ConditionalCheckFailedException) {
      return jsonResponse(409, {
        success: false,
        error: 'A product with this identifier already exists',
      });
    }
    logError('Product creation failed', error, {
      requestId,
      operation: 'createProduct',
    });
    return jsonResponse(500, {
      success: false,
      error: 'Could not create product',
    });
  }
};

The DynamoDB client is initialized outside the handler in src/utils/DynamoDbClient.ts:

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import { DynamoDBDocumentClient } from '@aws-sdk/lib-dynamodb'

const client = new DynamoDBClient({
  maxAttempts: 3,
})

const dynamoDB = DynamoDBDocumentClient.from(client,{
  marshallOptions: {
    removeUndefinedValues: true
  }
})

export default dynamoDB

AWS Lambda may reuse an execution environment across multiple invocations. Keeping the client outside the handler allows later invocations to reuse it instead of creating a new client on every request.

The product is persisted using PutCommand:

await dynamoDB.send(
  new PutCommand({
  TableName: getProductsTableName(),
  Item: product,
  ConditionExpression: 'attribute_not_exists(PK)',
  }),
  );

The handler delegates structured logging to the shared logInfo utility:

logInfo('Product created', {
  requestId,
  operation: 'createProduct',
  productId: id,
});

The logger serializes the entry as JSON before writing it to CloudWatch. This gives every log a consistent timestamp, level, message, request ID, operation, and contextual fields.

This structure becomes useful later when Datadog correlates the log with the trace that produced it.

Implementing the remaining handlers

The remaining handlers follow the same overall structure, but each uses a different DynamoDB command.

Listing products

The list handler uses ScanCommand:

 const result = await dynamoDB.send(
       new ScanCommand({
       TableName: getProductsTableName(),
       FilterExpression: '#entityType = :entityType',
       ExpressionAttributeNames: {
       '#entityType': 'entityType',
       },
       ExpressionAttributeValues: {
       ':entityType': 'PRODUCT',
       },
       ExclusiveStartKey: exclusiveStartKey,
      }),
      );

A scan is acceptable for this small learning project. For a production table with a large dataset, I would design the access pattern around QueryCommand and an appropriate partition key or secondary index.

Retrieving a product

The get handler uses the product ID to construct the DynamoDB key:

 const result = await dynamoDB.send(
      new GetCommand({
        TableName: getProductsTableName(),
        Key: {
          PK: `PRODUCT#${productId}`,
          SK: 'PRODUCT',
        },
        ConsistentRead: false,
      }),
    );

If no item is returned, the handler responds with 404 Not Found.

Updating a product

The update handler uses UpdateCommand and returns the modified item:

 const result = await dynamoDB.send(
      new UpdateCommand({
        TableName: getProductsTableName(),
        Key: {
          PK: `PRODUCT#${productId}`,
          SK: 'PRODUCT',
        },
        ConditionExpression: 'attribute_exists(PK)',
        UpdateExpression: `SET ${updateExpressions.join(', ')}`,
        ExpressionAttributeNames: expressionAttributeNames,
        ExpressionAttributeValues: expressionAttributeValues,
        ReturnValues: 'ALL_NEW',
      }),
    );

Deleting a product

The delete handler uses DeleteCommand:

 await dynamoDB.send(
  new DeleteCommand({
  TableName: getProductsTableName(),
  Key: {
      PK: `PRODUCT#${productId}`,
      SK: "PRODUCT"
  },
  ConditionExpression: "attribute_exists(PK)",
  }))

The complete implementations of all five handlers are available in the src/handlers directory.

---

3. Building the Lambda bundles

The project targets Node.js 22 and uses esbuild to create one bundle per handler.

The final TypeScript configuration:

{
  "compilerOptions": {
    "target": "ES2022",
    "lib": ["ES2022"],

    "module": "ESNext",
    "moduleResolution": "Bundler",

    "rootDir": "./src",
    "noEmit": true,

    "types": ["node", "aws-lambda"],

    "strict": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "resolveJsonModule": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "noUncheckedIndexedAccess": true
  },
  "include": ["src/**/*.ts"],
  "exclude": [
    "node_modules",
    "dist",
    "infra",
    "src-backup"
  ]
}

The build command:

{
  "scripts": {
    "clean": "rm -rf dist",
    "typecheck": "tsc --noEmit",
    "build": "npm run clean && esbuild src/handlers/*.ts --bundle --platform=node --target=node22 --format=cjs --outdir=dist --minify --tree-shaking=true --sourcemap=external --sources-content=false"
  }
}

Before writing Terraform, I verified the application:

npm run typecheck
npm run build

This gave me a clean baseline and five compiled JavaScript files.

4. Organizing the Terraform project

I separated the Terraform configuration by responsibility:

infra/
├── api.tf
├── dynamodb.tf
├── iam.tf
├── lambda.tf
├── locals.tf
├── logs.tf
├── outputs.tf
├── providers.tf
├── variables.tf
└── versions.tf

Terraform loads all .tf files in the root module together, so this split is for maintainability rather than execution order.

---

5. Defining the functions once

Instead of repeating the same Terraform structure five times, I defined the functions in a local map:

locals {
  functions = {
    createProduct = {
      handler   = "createProduct.createProduct"
      route_key = "POST /products"
    }

    listProducts = {
      handler   = "listProducts.listProducts"
      route_key = "GET /products"
    }

    getProductById = {
      handler   = "getProductById.getProductById"
      route_key = "GET /products/{id}"
    }

    updateProduct = {
      handler   = "updateProduct.updateProduct"
      route_key = "PUT /products/{id}"
    }

    deleteProduct = {
      handler   = "deleteProduct.deleteProduct"
      route_key = "DELETE /products/{id}"
    }
  }
}

This map drives:

• ZIP archives
• Lambda functions
• API Gateway integrations
• API routes
• Lambda permissions
• CloudWatch log groups

Adding an endpoint becomes a controlled change to one source of truth.

---

6. Protecting the existing DynamoDB table

The ProductsTable already existed before the migration.

I referenced it as a data source:

data "aws_dynamodb_table" "products" {
  name = "ProductsTable"
}

This distinction is critical:

• A Terraform resource is lifecycle-managed.
• A Terraform data source is read-only from Terraform's perspective.

Because the table is a data source, terraform destroy does not delete it.

That was one of the most important safety decisions in the project.

---

7. Packaging and creating the Lambda functions

Step 1: Package one handler per ZIP

data "archive_file" "lambda" {
  for_each = local.functions

  type        = "zip"
  source_file = "${path.module}/../dist/${each.key}.js"
  output_path = "${path.module}/build/${each.key}.zip"
}

Independent packages keep deployment artifacts small and allow Terraform to detect function-specific code changes.

Step 2: Create the functions

resource "aws_lambda_function" "product_handlers" {
  for_each = local.functions

  function_name = "${var.project_name}-${each.key}-${var.environment}"
  role          = aws_iam_role.lambda_execution.arn
  runtime       = "nodejs22.x"
  handler       = each.value.handler

  filename         = data.archive_file.lambda[each.key].output_path
  source_code_hash = data.archive_file.lambda[each.key].output_base64sha256

  memory_size = 256
  timeout     = 10

  environment {
    variables = {
      PRODUCTS_TABLE = data.aws_dynamodb_table.products.name
    }
  }

  tags = local.common_tags
}

I first deployed this baseline without Datadog.

That let me verify:

1. Terraform was valid.
2. All routes worked.
3. DynamoDB reads and writes worked.
4. CloudWatch logs worked.
5. Any later issue could be isolated to the observability changes.

---

8. IAM, API Gateway, and log groups

The Lambda role received only the DynamoDB permissions used by the handlers:

data "aws_iam_policy_document" "dynamodb_access" {
  statement {
    sid    = "AllowProductsTableCrud"
    effect = "Allow"

    actions = [
      "dynamodb:PutItem",
      "dynamodb:GetItem",
      "dynamodb:Scan",
      "dynamodb:UpdateItem",
      "dynamodb:DeleteItem"
    ]

    resources = [
      data.aws_dynamodb_table.products.arn
    ]
  }


}

The API Gateway integrations and routes also use the function map:

resource "aws_apigatewayv2_integration" "product_handlers" {
  for_each = local.functions

  api_id = aws_apigatewayv2_api.products.id

  integration_type       = "AWS_PROXY"
  integration_uri        = aws_lambda_function.product_handlers[each.key].invoke_arn
  integration_method     = "POST"
  payload_format_version = "2.0"
}

resource "aws_apigatewayv2_route" "product_routes" {
  for_each = local.functions

  api_id    = aws_apigatewayv2_api.products.id
  route_key = each.value.route_key
  target    = "integrations/${aws_apigatewayv2_integration.product_handlers[each.key].id}"
}

I created the log groups explicitly so retention would not be indefinite:

resource "aws_cloudwatch_log_group" "lambda" {
  for_each = local.functions

  name              = "/aws/lambda/${aws_lambda_function.product_handlers[each.key].function_name}"
  retention_in_days = 7

  tags = local.common_tags
}

---

9. Planning and applying the infrastructure

I initialized, formatted, validated, and planned the project:

AWS_PROFILE=personal terraform -chdir=infra init
terraform -chdir=infra fmt
terraform -chdir=infra validate
AWS_PROFILE=personal terraform -chdir=infra plan -out=tfplan

The first deployment plan showed:

30 to add, 0 to change, 0 to destroy

The most important part was 0 to destroy.

The existing DynamoDB table appeared as a data source, not as a resource to replace or delete.
The plan below shows the proposed infrastructure changes.

After review:

AWS_PROFILE=personal terraform -chdir=infra apply tfplan

Terraform created the Lambda functions, API Gateway routes, IAM resources, permissions, and log groups.

---

10. Verifying the Deployed CRUD API with Postman

POST   /products
GET    /products
GET    /products/{id}
PUT    /products/{id}
DELETE /products/{id}

The workflow was:

1. Create a product.
2. Capture the returned ID.
3. List products.
4. Retrieve the product.
5. Update it.
6. Retrieve the updated version.
7. Delete it.
8. Confirm it no longer exists.
9. Send invalid input and verify a controlled 400 response.

.

Verifying the CRUD Handlers with Postman

Before adding Datadog instrumentation, I tested the Terraform-deployed API through API Gateway using Postman.

The goal was to verify that:

• API Gateway routed each request to the correct Lambda function
• the handlers accepted and validated the request data
• DynamoDB operations completed successfully
• the API returned the expected HTTP status codes
• the product lifecycle worked from creation through deletion

Creating a product

I first sent a POST request to:

POST /products

The request body contained test data:

{
  "name": "Mechanical Keyboard",
  "description": "Wireless mechanical keyboard",
  "price": 89.99
}

The API returned 201 Created together with the generated product ID and timestamps.

The createProduct Lambda returned 201 Created after persisting the new product in DynamoDB.

Below is the DynamoDB table showing the Product item created:

I saved the returned ID as a Postman environment variable so it could be reused by the remaining requests.

productId = <ID returned by POST /products>

Listing Products

The GET /products endpoint returned the current list of products.

Retrieving the created product

I then sent:

GET /products/{{productId}}

The API returned 200 OK and the product created in the previous request.

The getProductById Lambda retrieved the product using its DynamoDB partition and sort keys.

Updating the product

To test the update handler, I sent:

PUT /products/{{productId}}

with the following body:

{
  "description": "Updated wireless mechanical keyboard",
  "price": 99.99
}

The API returned the modified product with a new updatedAt value.

The updateProduct Lambda updated the existing DynamoDB item and returned the new values.

Deleting the product

Finally, I sent:

DELETE /products/{{productId}}

The API confirmed that the product had been deleted.

The deleteProduct Lambda removed the product from DynamoDB.

To verify that the deletion was complete, I repeated:

GET /products/{{productId}}

The API returned:

{
  "success": false,
  "error": "Product not found"
}

with a 404 Not Found status.

The final 404 response confirms that the product was no longer present in DynamoDB.

I also submitted a product-creation request that violated the validation rules. The API returned 400 Bad Request with details explaining why the request was rejected.

---

11. Securing the Datadog API key

The Datadog API key must not be stored in Git, Terraform variables, screenshots, or plaintext Lambda environment variables.

I stored it as an AWS Systems Manager Parameter Store SecureString:

/serverless-products-api/dev/datadog-api-key

A safe shell pattern is:

read -s "DD_API_KEY?Paste the Datadog API key: "
echo

CLEAN_DD_API_KEY="$(printf '%s' "$DD_API_KEY" | tr -d '[:space:]')"

AWS_PROFILE=personal aws ssm put-parameter \
  --name "/serverless-products-api/dev/datadog-api-key" \
  --type "SecureString" \
  --value "$CLEAN_DD_API_KEY" \
  --region ca-central-1 \
  --overwrite

unset DD_API_KEY CLEAN_DD_API_KEY

Terraform constructs the ARN without reading the value:

data "aws_caller_identity" "current" {}

locals {
  datadog_api_key_parameter_name = "/serverless-products-api/dev/datadog-api-key"

  datadog_api_key_ssm_arn = join("", [
    "arn:aws:ssm:",
    var.aws_region,
    ":",
    data.aws_caller_identity.current.account_id,
    ":parameter",
    local.datadog_api_key_parameter_name
  ])
}

The Lambda execution role receives ssm:GetParameter for the specific parameter and kms:Decrypt for the KMS key used to encrypt it.

actions = [
  "ssm:GetParameter"
]

resources = [
  local.datadog_api_key_ssm_arn
]

actions = [
  "kms:Decrypt"
]

resources = [
  data.aws_kms_alias.ssm.target_key_arn
]

---

12. Adding the Datadog Lambda layers

The Datadog library and Extension layer versions depend on the runtime, region, architecture, and current Datadog release.
Because Datadog publishes new versions over time, verify the current Node.js and Extension layer versions before deploying.

The following excerpt shows the Datadog-specific additions to the existing Lambda resource.
I represented them as variables,

variable "datadog_node_layer_version" {
  description = "Datadog Node.js Lambda library layer version."
  type        = number
  default     = 140
}

variable "datadog_extension_layer_version" {
  description = "Datadog Lambda Extension layer version."
  type        = number
  default     = 98
}

The Lambda resource then uses the Datadog wrapper:

resource "aws_lambda_function" "product_handlers" {
  for_each = local.functions

  function_name = "${var.project_name}-${each.key}-${var.environment}"
  role          = aws_iam_role.lambda_execution.arn
  runtime       = "nodejs22.x"

  handler = "/opt/nodejs/node_modules/datadog-lambda-js/handler.handler"

  layers = [
    local.datadog_node_layer_arn,
    local.datadog_extension_layer_arn
  ]

  environment {
    variables = {
      PRODUCTS_TABLE      = data.aws_dynamodb_table.products.name
      DD_LAMBDA_HANDLER   = each.value.handler
      DD_SERVICE          = var.project_name
      DD_ENV              = var.environment
      DD_VERSION          = var.application_version
      DD_SITE             = var.datadog_site
      DD_API_KEY_SSM_ARN  = local.datadog_api_key_ssm_arn
      DD_TRACE_ENABLED    = "true"
      DD_LOGS_ENABLED     = "true"
      DD_ENHANCED_METRICS = "true"

      DD_CAPTURE_LAMBDA_PAYLOAD = "false"
    }
  }
}

DD_LAMBDA_HANDLER tells the Datadog wrapper which original application handler to invoke.

---

13. Debugging the Datadog Extension crash

After the first Datadog deployment, CloudWatch showed:

TELEMETRY Name: datadog-agent State: Subscribed
EXTENSION Name: datadog-agent State: Ready

The application also generated trace and span IDs.

However, the invocation ended with:

Extension.Crash
failed to parse header: InvalidHeaderValue

This indicated:

• the library was instrumenting the Lambda
• the Extension started
• the application operation completed
• telemetry flushing failed afterward

The root cause was a malformed API-key value in Parameter Store.

A hidden newline, surrounding whitespace, or incorrectly copied value can become an invalid HTTP header.

After sanitizing and overwriting the parameter, CloudWatch showed a healthy invocation:

TELEMETRY Name: datadog-agent State: Subscribed
EXTENSION Name: datadog-agent State: Ready
START RequestId: ...
INFO [dd.trace_id=... dd.span_id=...] Product created
END RequestId: ...
REPORT RequestId: ...

The lesson was simple:

A secret can exist and be retrievable while still being invalid for the application consuming it.

The final CloudWatch logs below confirm that the Extension initialized successfully and the Lambda invocation completed normally.

---

14. Verifying end-to-end traces

I generated traffic for all five routes.

Datadog received traces for:

POST /products
GET /products
GET /products/{id}
PUT /products/{id}
DELETE /products/{id}

A POST /products trace showed:

API Gateway
  → Lambda
  → application handler
  → DynamoDB PutItem

This is more useful than an isolated metric because it explains where time was spent during one request.

---

15. Verifying trace-to-log correlation

Inside the Lambda span, Datadog displayed the structured application log:

Product created

The log included:

dd.trace_id=...
dd.span_id=...

The troubleshooting path becomes:

slow or failed request
  → affected span
  → associated log
  → request ID and product ID

Below is an example showing traces, spans and a correlated structured log and its associated metadata:

---

16. Building the Datadog dashboard

I created a dashboard named:

Serverless Products API — Dev

It tracks:

Traffic

• Lambda invocations
• API requests
• API errors

Performance

• average Lambda runtime
• runtime by Lambda function
• API latency percentiles
• cold-start initialization duration

Resources

• peak memory by Lambda function

Serverless overhead

• Datadog Extension post-runtime duration

Cost

• estimated Lambda cost

The main enhanced metrics were:

aws.lambda.enhanced.invocations
aws.lambda.enhanced.runtime_duration
aws.lambda.enhanced.init_duration
aws.lambda.enhanced.post_runtime_duration
aws.lambda.enhanced.max_memory_used
aws.lambda.enhanced.estimated_cost

Making small cost values readable

For a low-volume development workload, the estimated cost was far below one cent.

Four decimal places displayed:

$0.0000

I used a formula:

a * 1000000

and labelled the result as micro-US dollars:

μUSD

For example:

94.13 μUSD

This changes only the display scale, not the underlying cost.

The final dashboard below displays the key metrics described above.

---

17. Adding proactive monitors

A dashboard requires someone to look at it.

A monitor evaluates telemetry and changes state when a threshold is crossed.

I created three monitors.

Monitor 1: High p95 latency

Monitor:    [Dev] Serverless Products API — High Lambda Latency
Scope:      env:dev, service:serverless-products-api
Metric:     p95 trace.aws.lambda latency
Evaluation: MAX over the last 1 hour
Warning:    above 750 ms
Critical:   above 1 second

The observed latency was around 500 ms, so 750 ms represented degradation and 1 second represented a critical threshold.

For sparse development traffic:

Show last known status
Do not require a full window

Monitor 2: High error rate

Monitor:    [Dev] Serverless Products API — High Error Rate
Scope:      env:dev, service:serverless-products-api
Metric:     Lambda APM error rate
Evaluation: SUM over the last 1 hour
Warning:    above 5%
Critical:   above 10%

For this APM error-rate monitor, Datadog used SUM because the underlying request and error metrics are count-based.

A low-traffic limitation is important:

1 failed request / 10 total requests = 10%

A production monitor should also require a minimum request count.

Monitor 3: High memory by function

Monitor:    [Dev] Serverless Products API — High Lambda Memory Usage
Metric:     aws.lambda.enhanced.max_memory_used
Filter:     functionname:serverless-products-api-*
Group by:   functionname
Evaluation: MAX over the last 1 hour
Warning:    above 180 MiB
Critical:   above 205 MiB
Allocation: 256 MiB

Grouping by functionname creates a separate monitor group for every Lambda function.

The notification template includes:

Affected function: {{functionname.name}}

Debugging a no-data monitor

The memory monitor initially showed:

NO DATA
No groups reporting

The dashboard already had memory data, so the telemetry pipeline was working.

The problem was a typo in the filter.

Incorrect:

functionname:serverless-api-*

Correct:

functionname:serverless-products-api-*

After the correction, the monitor displayed:

5 OK

This was a useful reminder that a monitoring failure can come from a query mismatch rather than an application failure.

---

18. Cost controls and cleanup

Before deploying, I configured:

• AWS Budget alerts
• AWS Free Tier alerts
• cost anomaly monitoring
• seven-day CloudWatch retention
• low-volume test traffic

A budget alert is not a hard spending cap.

The strongest control remains removing resources when testing is finished.

Preview destruction:

AWS_PROFILE=personal terraform -chdir=infra plan \
  -destroy \
  -out=destroy.tfplan

Destroy Terraform-managed resources:

AWS_PROFILE=personal terraform -chdir=infra apply destroy.tfplan

The existing DynamoDB table remains because it is a data source.

The manually created SSM parameter must be deleted separately when it is no longer needed:

AWS_PROFILE=personal aws ssm delete-parameter \
  --name "/serverless-products-api/dev/datadog-api-key" \
  --region ca-central-1

The Datadog dashboard and monitors remain because they were created manually.

The screenshots below show the reviewed destruction plan and its successful application.

---

What I learned

Framework abstraction and infrastructure ownership are different skills

The Serverless Framework helped me deploy the original application quickly.

Terraform forced me to understand the individual AWS resources, dependencies, permissions, and lifecycle decisions underneath that abstraction.

The point was not to prove that one tool is universally better. The value was understanding what each tool hides and when explicit infrastructure ownership is useful.

A Terraform plan is a safety mechanism

The most important line in the first plan was:

0 to destroy

Combined with the DynamoDB data source, that gave me confidence that existing data would survive the migration.

Observability should be verified in layers

I verified the Datadog integration in this order:

1. CloudWatch showed the Extension starting.
2. structured logs contained trace and span IDs.
3. Datadog received Lambda spans.
4. traces included DynamoDB.
5. logs appeared inside trace details.
6. enhanced metrics populated the dashboard.
7. monitors evaluated live telemetry.

This made debugging much easier.

No data does not always mean no telemetry

The memory metric existed, but the monitor returned no groups because its filter did not match the real function names.

When a monitor has no data, check:

• tag names
• tag values
• wildcard placement
• grouping
• evaluation window
• full-window requirements
• sparse metric behaviour
• whether edits were saved

Thresholds require context

A threshold should relate to:

• observed baseline behaviour
• allocated resources
• traffic volume
• expected workload
• business impact

The one-hour windows were appropriate for this sparse development workload.

A production environment would likely use shorter sustained windows, minimum request thresholds, and environment-specific policies.

---

Possible next improvements

• Manage Datadog dashboards and monitors through the Datadog Terraform provider.
• Add CI/CD for type-checking, builds, Terraform validation, and deployment.
• Upload source maps for clearer TypeScript stack traces.
• Add automated integration tests.
• Add pagination.
• Replace scans with query-based DynamoDB access patterns where appropriate.
• Add authentication and authorization.
• Add remote Terraform state and locking.
• Add a minimum traffic condition to the error-rate monitor.
• Add deployment markers to correlate releases with performance changes.

---

Final result

The project now demonstrates:

TypeScript development
  → AWS SDK v3 modernization
  → esbuild packaging
  → Terraform infrastructure
  → API Gateway and Lambda deployment
  → DynamoDB persistence
  → secure secret retrieval
  → Datadog tracing and metrics
  → structured log correlation
  → dashboarding
  → threshold monitoring
  → cost-aware cleanup

Instead of stopping when the API returned a successful response, I could now trace the request, inspect the DynamoDB dependency, find the corresponding application log, measure runtime and memory, estimate cost, and define an alert when performance or resource usage crossed a threshold.

That is the difference between deploying an application and operating one.

Article version: v1.0.0-terraform-datadog

---