CloudWatch API key: OpenTelemetry traces without SigV4

Since November 3, 2026, Amazon CloudWatch also accepts OpenTelemetry traces authenticated with a CloudWatch API key: a plain bearer token in the HTTP header instead of a SigV4 signature. It reads like a footnote in the AWS news feed. For anyone running applications outside AWS who still wants to see in CloudWatch what those applications are doing, it is a real relief.

This topic matters to me because observability in mixed environments often gets stuck on authentication. The service on a rented server, the script in a build pipeline or a third-party tool simply cannot sign AWS requests. So the traces end up somewhere else, or nowhere at all. In this article you set up the new path step by step, with a small TypeScript app, the Serverless Framework for the IAM part and a clean strategy for rotation and security. I will also tell you plainly when I would stay away from it.

In short

With a CloudWatch API key, OpenTelemetry clients can send traces to the X-Ray OTLP endpoint using a bearer token, with no SigV4 involved. That helps applications outside AWS and tools that only support a fixed header. The biggest pitfall is the key itself: it is a long-lived secret and needs an expiry date, tight permissions and a rotation plan.

What you will learn in this article

• When an API key makes sense and when SigV4 remains the better choice
• How to enable Transaction Search and create a tightly scoped IAM user
• How to instrument a Node.js app with OpenTelemetry and a bearer token
• How to store, rotate and, in an emergency, block the key

What is new with the CloudWatch API key for traces?

First, a few terms. OpenTelemetry (OTel for short) is an open standard that applications use to produce telemetry: traces, metrics and logs. A trace shows the path of a request through your system, split into individual sections called spans. This data travels over OTLP, the OpenTelemetry protocol, usually via HTTP.

AWS provides its own OTLP endpoint for traces: https://xray.<region>.amazonaws.com/v1/traces. Until now, this endpoint required every request to be signed with SigV4, the signing process AWS clients use to sign each request with their credentials. That works nicely when your code uses the AWS SDK or the AWS Distro for OpenTelemetry (ADOT). Many exporters and third-party tools cannot do it, though.

Now there is a second door. You create an API key for the X-Ray service in IAM and send it as Authorization: Bearer <key>. Any OTLP HTTP client that can set a header is ready to go. The spans end up in the same place as signed spans and are available for Transaction Search, Application Signals and the other CloudWatch features.

Two details matter. First, the key only works for sending spans to the OTLP endpoint. You cannot read anything with it, and all other X-Ray APIs still require SigV4. Second, AWS calls this “CloudWatch API keys” in the announcement, but technically it is a service-specific IAM credential for xray.amazonaws.com. In the IAM console you will find it under “API keys for AWS services”.

When is a CloudWatch API key better than SigV4?

AWS is refreshingly clear here: SigV4 with short-term credentials remains the recommended approach. The API key is the exception for cases where that is not practical. I have sorted the typical situations below.

Situation	My recommendation	Why
Lambda, ECS, EKS or EC2 in AWS	SigV4 via the IAM role	The role provides short-term credentials automatically, and ADOT signs without extra effort.
GitHub Actions or GitLab CI	Check OIDC first, otherwise an API key	With OIDC the pipeline gets short-term AWS credentials, so SigV4 is possible. I show the setup in GitHub Actions OIDC: AWS deploys without access keys.
Server at another hosting provider or on premises	API key with an expiry date	Without an AWS identity, a tightly scoped key is often the most pragmatic option.
Third-party tool with a fixed header field	API key	The tool simply cannot sign requests.
Browser or mobile app	Neither, put a collector in between	A key in client code is public. Let your own collector receive and forward the data.

My decision path: the API key is deliberately the last option.

So for Serverless applications on Lambda, nothing changes. If you already use Powertools or ADOT there, keep doing so. I show what that looks like in TypeScript in Lambda Powertools TypeScript: logs, metrics, traces. The new path gets interesting where your Lambda functions work together with services outside AWS and you finally want to see the full path of a request in one trace.

How traces flow from your server into CloudWatch

Before we build anything, here is the flow at a glance. Your application creates spans, the OpenTelemetry exporter batches them and sends them via HTTPS with a bearer header to the X-Ray endpoint. IAM checks the key and the permissions of the user it belongs to. X-Ray then writes the spans to the aws/spans log group, and Transaction Search makes them searchable.

🎬 Flow: sending traces to X-Ray with a CloudWatch API key – this animated diagram only runs on my site: watch it on sascha-ms.de

Prerequisites

• An AWS account with permissions for IAM, X-Ray, CloudWatch Logs and Secrets Manager
• AWS CLI v2, signed in locally (only for the setup, not for the app)
• Node.js 22 or newer and npm
• Serverless Framework v4 (npm i -g serverless) for the IAM part
• A Region where the endpoint supports API keys. Frankfurt (eu-central-1) is included; the full list is in the X-Ray documentation.

The example is a fictional bike rental. Its booking service runs on a server at another hosting provider and should still deliver its traces to CloudWatch.

Step 1: Enable Transaction Search

The OTLP endpoint for traces only works when Transaction Search is enabled. It stores spans as structured logs in CloudWatch Logs and makes them searchable. If you have already turned it on in your account, simply skip this step.

First, allow X-Ray to write to the two log groups. Replace the account ID and Region with your own values:

export AWS_REGION=eu-central-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

aws logs put-resource-policy --policy-name xray-transaction-search --policy-document "{
  \"Version\": \"2012-10-17\",
  \"Statement\": [{
    \"Sid\": \"TransactionSearchXRayAccess\",
    \"Effect\": \"Allow\",
    \"Principal\": { \"Service\": \"xray.amazonaws.com\" },
    \"Action\": \"logs:PutLogEvents\",
    \"Resource\": [
      \"arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:aws/spans:*\",
      \"arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:/aws/application-signals/data:*\"
    ],
    \"Condition\": {
      \"ArnLike\": { \"aws:SourceArn\": \"arn:aws:xray:${AWS_REGION}:${ACCOUNT_ID}:*\" },
      \"StringEquals\": { \"aws:SourceAccount\": \"${ACCOUNT_ID}\" }
    }
  }]
}"

Then switch the destination for trace segments to CloudWatch Logs and check the result:

aws xray update-trace-segment-destination --destination CloudWatchLogs
aws xray get-trace-segment-destination

You should see "Destination": "CloudWatchLogs" and "Status": "ACTIVE". Allow about ten minutes until the first spans are searchable. By default, AWS indexes 1% of spans as trace summaries free of charge. You can change that share later with aws xray update-indexing-rule.

Step 2: Create the IAM user with the Serverless Framework

An API key always belongs to an IAM user. That is why you create a dedicated user that can do exactly one thing: send spans. AWS offers the managed policy AWSXRayAPIKeyAccess for this. I prefer my own inline policy here because I can additionally restrict it to the public IP address of the server. A stolen key is then worthless anywhere else.

The Serverless Framework also works for pure infrastructure without functions. The template deliberately does not set the value of the key, because a secret does not belong in a CloudFormation template.

# infra/serverless.yml
service: bike-rental-tracing-access
frameworkVersion: "4"

provider:
  name: aws
  region: eu-central-1

params:
  default:
    # Public IP address(es) of the server that sends traces
    allowedIps:
      - 203.0.113.10/32

resources:
  Resources:
    TracesSenderUser:
      Type: AWS::IAM::User
      Properties:
        UserName: ${self:service}-${sls:stage}-traces
        Policies:
          - PolicyName: send-traces-only
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Sid: TracesPerApiKey
                  Effect: Allow
                  Action:
                    - xray:CallWithBearerToken
                    - xray:PutTraceSegments
                  Resource: "*"
                  Condition:
                    IpAddress:
                      aws:SourceIp: ${param:allowedIps}

    TracesApiKeySecret:
      Type: AWS::SecretsManager::Secret
      Properties:
        Name: /bike-rental/${sls:stage}/xray-api-key
        Description: X-Ray API key for OTLP traces (set the value via CLI)

  Outputs:
    TracesSenderUserName:
      Value: !Ref TracesSenderUser
    TracesApiKeySecretArn:
      Value: !Ref TracesApiKeySecret

cd infra
serverless deploy --stage dev

In the outputs you will see the user name bike-rental-tracing-access-dev-traces and the ARN of the still empty secret. If your server sends from changing IP addresses, leave out the Condition and rely more on expiry and rotation.

Step 3: Create the CloudWatch API key and store it safely

Now you create the actual key. This only works with “real” AWS credentials, so via the console or the CLI, never with an API key itself. Always set an expiry date: without --credential-age-days the key never expires.

USER_NAME=bike-rental-tracing-access-dev-traces

API_KEY=$(aws iam create-service-specific-credential \
  --user-name "$USER_NAME" \
  --service-name xray.amazonaws.com \
  --credential-age-days 90 \
  --query ServiceSpecificCredential.ServiceCredentialSecret \
  --output text)

aws secretsmanager put-secret-value \
  --secret-id /bike-rental/dev/xray-api-key \
  --secret-string "$API_KEY"

unset API_KEY

The value from ServiceCredentialSecret is your API key. AWS shows it exactly once and you cannot retrieve it afterwards. That is why it goes straight into Secrets Manager, without a detour via the clipboard or a file. If it gets lost, just create a new one.

On the server you fetch it at startup, for example with a start script that sets the value as the XRAY_API_KEY environment variable. Fetching it requires AWS access, of course. If the server has none, your deployment tool passes the key in as a secret, for example as an encrypted variable of your pipeline.

Step 4: Instrument a Node.js app with OpenTelemetry

Now for the application. You need the OpenTelemetry Node.js SDK and an OTLP exporter. I use the protobuf exporter because it transfers data more compactly than JSON. The endpoint accepts both.

mkdir bike-rental-tracing && cd bike-rental-tracing
npm init -y
npm pkg set type=module
npm i @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto @opentelemetry/api @opentelemetry/resources @opentelemetry/semantic-conventions
npm i -D typescript tsx @types/node

The tsconfig.json is simple. The important part is NodeNext, so that top-level await and ES modules work together:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "outDir": "dist",
    "skipLibCheck": true
  },
  "include": ["src"]
}

Configure the exporter with a bearer header

The tracing.ts file starts the SDK before any other code runs. The crucial part is the headers entry on the exporter. The file also fails immediately if the key is missing. Otherwise missing telemetry tends to surface only when you urgently need it, right in the middle of an incident.

// src/tracing.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from "@opentelemetry/semantic-conventions";

const region = process.env.AWS_REGION ?? "eu-central-1";
const apiKey = process.env.XRAY_API_KEY;

if (!apiKey) {
  // Fail loudly instead of silently sending nothing
  throw new Error("XRAY_API_KEY is missing – traces cannot be sent.");
}

const exporter = new OTLPTraceExporter({
  url: process.env.OTLP_TRACES_URL ?? `https://xray.${region}.amazonaws.com/v1/traces`,
  headers: { Authorization: `Bearer ${apiKey}` },
  timeoutMillis: 10_000,
});

export const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: "bike-rental-booking",
    [ATTR_SERVICE_VERSION]: "1.0.0",
    "deployment.environment.name": process.env.APP_ENV ?? "dev",
  }),
  traceExporter: exporter,
});

sdk.start();

// Flush open spans on shutdown
const shutdown = async () => {
  await sdk.shutdown();
  process.exit(0);
};
process.on("SIGTERM", shutdown);
process.on("SIGINT", shutdown);

Create spans in the application

The sample app books a bike and creates two nested spans along the way. Errors are added to the span with recordException and an error status, so you can filter for them in CloudWatch later.

// src/app.ts
import "./tracing.js";
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { sdk } from "./tracing.js";

const tracer = trace.getTracer("bike-rental-booking");

async function checkAvailability(bikeId: string): Promise<boolean> {
  return tracer.startActiveSpan("availability.check", async (span) => {
    span.setAttribute("bike.id", bikeId);
    await new Promise((r) => setTimeout(r, 40)); // simulated database query
    span.end();
    return true;
  });
}

async function bookBike(bikeId: string, customerId: string): Promise<string> {
  return tracer.startActiveSpan("booking.create", async (span) => {
    try {
      span.setAttribute("bike.id", bikeId);
      span.setAttribute("customer.id", customerId);
      const available = await checkAvailability(bikeId);
      if (!available) throw new Error("Bike is already rented out");
      const bookingId = `B-${Date.now()}`;
      span.setAttribute("booking.id", bookingId);
      return bookingId;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

const id = await bookBike("bike-042", "customer-7");
console.log(`Booking ${id} created`);

// Short-lived processes (scripts, CI jobs): flush explicitly
await sdk.shutdown();

For a long-running server, the SIGTERM handler in tracing.ts is enough. For short-lived processes such as a CI job, call sdk.shutdown() yourself, otherwise the process ends before the exporter has sent its last batch.

Dry run without AWS

Before real data flows, a quick local test shows whether the header arrives at all. A tiny receiver that logs every OTLP request is all you need:

// src/mock-receiver.ts
// Local dry run: accepts OTLP requests and shows whether the header is right.
import { createServer } from "node:http";

createServer((req, res) => {
  const auth = req.headers.authorization ?? "";
  const ok = /^Bearer \S{20,}$/.test(auth);
  let bytes = 0;
  req.on("data", (chunk: Buffer) => (bytes += chunk.length));
  req.on("end", () => {
    console.log(
      `${req.method} ${req.url} | ${req.headers["content-type"]} | ${bytes} bytes | Bearer header: ${ok ? "ok" : "MISSING"}`,
    );
    res.writeHead(ok ? 200 : 403, { "content-type": "application/json" });
    res.end("{}");
  });
}).listen(4318, () => console.log("Mock receiver listening on http://localhost:4318/v1/traces"));

npx tsx src/mock-receiver.ts &
XRAY_API_KEY=testkey-abcdefghijklmnopqrstuvwxyz \
OTLP_TRACES_URL=http://localhost:4318/v1/traces \
npx tsx src/app.ts

The expected output looks like this:

Mock receiver listening on http://localhost:4318/v1/traces
Booking B-1793889441350 created
POST /v1/traces | application/x-protobuf | 1068 bytes | Bearer header: ok

Afterwards, remove OTLP_TRACES_URL, set the real key and AWS_REGION=eu-central-1, and start the app again. The exporter then talks directly to https://xray.eu-central-1.amazonaws.com/v1/traces.

Step 5: Use the OpenTelemetry Collector instead

If several services run on the server, I prefer to put an OpenTelemetry Collector in front of them. It is a standalone process that receives, batches and forwards telemetry. The key then lives in exactly one place, and your applications send to localhost:4318 without authentication. You need the contrib distribution of the collector, because only it includes the bearertokenauth extension.

# collector/otel-collector.yaml
extensions:
  bearertokenauth/xray:
    token: "${env:XRAY_API_KEY}"

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
      grpc:
        endpoint: "0.0.0.0:4317"

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    send_batch_size: 512
    timeout: 5s

exporters:
  otlphttp/xray:
    traces_endpoint: "https://xray.eu-central-1.amazonaws.com/v1/traces"
    compression: gzip
    auth:
      authenticator: bearertokenauth/xray
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 120s

service:
  extensions: [bearertokenauth/xray]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/xray]

The key comes in through the environment variable and never appears in the file. Alternatively, the extension can read it from a mounted file with filename, which suits containers. The retry_on_failure block handles short outages, since AWS explicitly recommends retrying with backoff on a 503.

Step 6: Check the traces in CloudWatch

In the CloudWatch console, open Application Signals and then Transaction Search. Filter by service.name = bike-rental-booking. After a few minutes you should see the spans booking.create and availability.check, including attributes such as bike.id.

If you want more detail, enable CloudTrail data events for AWS::XRay::Trace. Requests with an API key show up there as PutTraceSegments, with the IAM user as the identity and "callWithBearerToken": true in the additional event data. Data events cost extra, but they are worth it during a trial phase.

Rotate a CloudWatch API key without downtime

Each IAM user can hold up to two X-Ray API keys at once, and that is precisely the point: you create the new key, switch over and only then turn off the old one.

1. Create a new key with create-service-specific-credential and write it to Secrets Manager.
2. Restart the collector or application so it picks up the new value.
3. Deactivate the old key and watch for a few minutes whether spans keep arriving.
4. Only then delete the old key.

aws iam list-service-specific-credentials \
  --user-name bike-rental-tracing-access-dev-traces \
  --service-name xray.amazonaws.com

aws iam update-service-specific-credential \
  --user-name bike-rental-tracing-access-dev-traces \
  --service-specific-credential-id <ID-of-the-old-key> \
  --status Inactive

You can automate this as well, for example with the rotation feature of Secrets Manager and a small Lambda function. I show how such a rotation is structured in Secrets Manager rotation with Lambda for API keys. Only the order matters: distribute the new value first, then block the old one.

Common errors and how to fix them

Symptom	Cause	Fix
HTTP 403	Key is invalid, expired or inactive, or the user lacks xray:CallWithBearerToken or xray:PutTraceSegments	Check the status with list-service-specific-credentials, review the policy and IP condition
HTTP 403 only from the server	The IP condition does not match, for example because of NAT or IPv6	Check the actual public address and add it to allowedIps
HTTP 503	Temporary authentication issue	Retry with backoff, in the collector via retry_on_failure
No spans visible	Transaction Search is not active or ten minutes have not passed yet	Run aws xray get-trace-segment-destination
Connection refused	Endpoint configured with http:// instead of https://	The endpoint only accepts HTTPS
Last spans missing	Process ends before the export	Call await sdk.shutdown() at the end

Keeping an eye on cost and security

The API key itself is free. You pay the same as for signed spans: with Transaction Search enabled, AWS bills span ingestion at CloudWatch prices, and stored spans in aws/spans take up log storage. Check the current CloudWatch pricing page before rolling out and set a sensible retention period. With a lot of traffic, sampling in the SDK or the collector keeps every health check from ending up as a trace.

On the security side, an API key is a long-lived secret, which is exactly what AWS wanted to get rid of with roles and short-term credentials. So I stick to a few firm rules. Each application gets its own user, and every key gets an expiry date and lives in a secrets store. With the iam:ServiceSpecificCredentialAgeDays condition you can even enforce a maximum age. If you want to ban API keys across the whole organization, add a deny on xray:CallWithBearerToken via a service control policy.

If a key is compromised, deactivate or delete it immediately. You can also take away all API key usage from the user with a deny policy. According to the documentation, such changes take effect after about five minutes.

My take

I like this change, precisely because AWS positions it so soberly. It closes a real gap: if part of your landscape runs outside AWS, you previously had to operate a collector with its own AWS credentials or switch to a different observability tool. Now a header is enough. That lowers the bar for seeing all services in one trace backend, and that is gold when debugging.

I would push back on CI/CD pipelines, which AWS lists as a typical use case. GitHub Actions and GitLab can obtain short-term AWS credentials via OIDC. If your pipeline can do that, SigV4 is the cleaner path there, and there is one less key to rotate. I mainly see the API key for servers without an AWS identity and for tools that insist on a fixed header. And it has no place in browser or app code.

For Serverless teams the message is relaxed: in Lambda everything stays as it was. But if you monitor durable functions or other workflows that call external services, you can now make their side visible too. What the AWS side of that looks like is covered in Monitoring durable functions: telemetry in TypeScript.

FAQ

Is a CloudWatch API key the same as an IAM access key?

No. An access key consists of a key ID and a secret and can call any permitted AWS API via SigV4. The X-Ray API key is a service-specific credential that only works as a bearer token for the OTLP traces endpoint.

Can I also send logs or metrics with the API key?

The announcement from November 3, 2026 and the documentation describe the API key only for OpenTelemetry traces. For logs and metrics, SigV4 remains the way as of today.

Do I need to change ADOT if I already use SigV4?

No. The ADOT SDKs still sign with SigV4 by default and need no changes. The API key is an additional option, not a replacement.

Does a CloudWatch API key expire automatically?

Only if you set --credential-age-days when creating it; 1 to 36,600 days are allowed. Without this parameter the key never expires. So always set an expiry date.

Your next step

Pick a single service outside AWS that regularly gives you headaches when debugging. Enable Transaction Search, create a user with an IP condition and send the first spans, first as a dry run and then for real. One afternoon is enough. Which service in your landscape is still a blind spot in tracing, and what is stopping you from making it visible?

Sources

• AWS What’s New: Amazon CloudWatch now supports API keys for OpenTelemetry traces (Nov 3, 2026)
• AWS X-Ray Developer Guide: Bearer token authentication for traces
• Amazon CloudWatch: Enable Transaction Search
• OpenTelemetry JavaScript: Exporters
• OpenTelemetry Collector Contrib: Bearer Token Auth Extension
• Amazon CloudWatch Pricing

This post was created with AI assistance.

---

👉 Next step: Not sure which AWS service fits your use case? My AWS service finder asks a few questions and points you to the right one.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en