DEV Community: Sergey Ivanychev

Anatomy of a Claude Code event

Sergey Ivanychev — Fri, 21 Aug 2026 16:06:50 +0000

In one of my previous articles, we discussed using Claude models in AWS Bedrock. One of the advantages of Bedrock was the ability to log supported model invocations made through the bedrock-runtime endpoint. The logged events are a goldmine of essential information for monitoring how you or your users use Claude Code.

The goal of this post is to summarize the useful information you can extract from events and serve as a small reference for monitoring Claude Code usage. I understand if you only want to grasp what is possible, so I’ll start with the TL;DR.

Practical summary

Using Claude invocation logging, you can monitor:

Who uses Claude Code and how much — each event contains the identity of the caller (identity.arn), the model used, and a timestamp.
Usage per device and per coding session — the metadata.user_id field carries device_id and a per-session session_id, so events can be grouped into sessions.
Token consumption, estimated cost, and latency — the message_stop/message_delta output events report input, output, cache-read, and cache-write token counts, as well as invocation latency and time to first byte, for every invocation.
Capabilities exposed to agents — the tools field shows which native and MCP tools each invocation could call, which matters both for security review and for diagnosing context-window pollution by overly verbose MCP servers.
Reasoning effort — output_config.effort (low → max) helps explain unexpectedly high token usage.
The model-visible state — the messages field contains the full conversation state: system prompts, CLAUDE.md contents, user prompts, tool calls, and their results. This is what makes the logs powerful for analysis — and extremely sensitive.

Let’s dig into the structure of the Anthropic API calls made by Claude Code and understand how to extract this useful information from them.

Logging setup

You have multiple options for enabling logging, depending on your setup.

If you use the standard Claude Code setup with an Anthropic subscription, the easiest way is to run an [LLM gateway locally](https://code.claude.com/docs/en/llm-gateway (e.g., LiteLLM) and enable logging there.

If you use the Anthropic API via AWS Bedrock, you can enable model invocation logging. When S3 delivery is enabled, the log files are conveniently delivered to S3 as gzip-compressed JSONL files that are ready to be analyzed.

This feature is not unique to AWS Bedrock — similar capabilities are available through Google Vertex AI, Azure API Management AI gateway logs, LLM gateway logging (e.g., LiteLLM), or LLM observability tools such as Langfuse via instrumentation. You can use one of the solutions in this list, depending on your setup, to collect these logs.

Sensitivity of the data

AI harness logs are very sensitive. If your organization logs the use of Claude Code or other development harnesses, the logs can contain a great deal of sensitive information (leaked tokens and passwords, sensitive prompts, etc.) — everything that appears during AI coding sessions on your developers’ machines.

Anatomy of an AWS Bedrock log payload

Note: Skip this section if you don’t use AWS.

I intentionally left out many fields that are not important for this blog post. Each request contains an ID, the model used, the identity of the caller (represented by an IAM or STS principal ARN), the input, and the output.

{
    "timestamp": "2024-01-15T12:00:00Z",
    "requestId": "abcd1234-5678-efgh-ijkl-mnopqrstuvwx",
    "modelId": "arn:aws:bedrock:us-east-1:073865013699:inference-profile/global.anthropic.claude-haiku-4-5-20251001-v1:0",
    "identity": {
        "arn": "arn:aws:sts::123456789012:assumed-role/MyRole/user@mycompany.com"
    },
    "input": {
        "inputBodyJson": { },
        "inputBodyS3Path": null,
    },
    "output": {
        "outputBodyJson": { },
        "outputBodyS3Path": null,
    },
    ...
}

Useful fields include:

timestamp
requestId — for deduplication.
modelId — the model ID or inference-profile ID used for the invocation. Depending on how the model was invoked, this may be an ARN.
identity.arn — identifies the role and session that called the model. Note that the session name frequently includes the user identifier if you use IAM Identity Center for your users’ AWS logins.

The input payload may be stored in the log entry itself (inputBodyJson), or, if it’s too big, it will be stored as a JSON file in S3, and the entry will contain the path to it (inputBodyS3Path). The same applies to the output.

To conduct analysis and build data sources on top of these events, you need to ensure that inputBodyJson is always present by augmenting it with data from inputBodyS3Path. One approach is to use a streaming process or a Lambda function that resolves inputBodyS3Path when present and writes full-fat events back to S3.

Anatomy of an Anthropic API payload

A typical event contains input and output parts, denoting the LLM input and output. In the following sections, we’ll go through the useful bits in the event payload.

`input.system` field

From the system field, you can extract the following information.

Block 1 contains the Anthropic headers. It includes the version of Claude Code used to invoke the model and whether the model was invoked by a subagent.

Block 2 is always the same, based on my observations.

Block 3 contains the actual system prompt of Claude Code, describing memory configuration, Git setup, available tools (including MCP), the current branch, environment details, and recent commits.

Example system value:

[
  {
    "type": "text",
    "text": "x-anthropic-billing-header: cc_version=2.1.178.2f1; cc_entrypoint=cli; cc_is_subagent=true;",
    "cache_control": null
  },
  {
    "type": "text",
    "text": "You are Claude Code, Anthropic's official CLI for Claude.",
    "cache_control": {
      "type": "ephemeral"
    }
  },
  {
    "type": "text",
    "text": "\nYou are an interactive agent that helps users with software engineering tasks.\n\nIMPORTANT: Assist ...",
    "cache_control": {
      "type": "ephemeral"
    }
  }
]

`input.metadata` field

The contents of this field look like this:

  "metadata": {
    "user_id": "{\"device_id\":\"5e71...\",\"session_id\":\"296...\"}"
  },

Currently, metadata contains a user_id field with device_id and session_id.

device_id identifies the device on which the harness runs; it doesn’t change between different coding sessions.
session_id is maintained within the coding session. For example, if you start a Claude Code session, all events generated by it will contain the same session_id.

`input.output_config` field

Example:

"output_config": {"effort": "high"}

This field contains information about reasoning effort (low/medium/high/xhigh/max). It may be useful for debugging high token usage by some AI agents.

`input.tools` field

The tools field contains the loaded native and MCP tools that the agent can call. It describes which tool schemas were loaded into this particular model invocation, not which MCP servers or tools were connected (see the messages field and its system role).

Using this field, you can:

Monitor capabilities exposed to Claude agents, including security implications.
Diagnose prompt size and cache behavior — some MCP servers are famous for context pollution due to overly sophisticated tool descriptions.

Here’s what the event looks like (I included only one native tool and one custom MCP tool, and trimmed the descriptions):

{
    "tools": [
        # Anthropic native tools.
        # Only "Bash" is included in this example.
        {
            "name": "Bash",
            "description": "Executes a Bash command and returns...",
            "input_schema": {
                "$schema": "https://json-schema.org/draft/2020-12/schema",
                "type": "object",
                "properties": {
                    "command": {
                        "description": "The command to execute.",
                        "type": "string",
                    },
                    "timeout": {
                        "description": (
                            "Optional timeout in milliseconds, up to 600000."
                        ),
                        "type": "number",
                    },
                    "description": {
                        "description": "A clear, concise description of...",
                        "type": "string",
                    },
                    "run_in_background": {
                        "description": (
                            "Set to True to run the command in the background."
                        ),
                        "type": "boolean",
                    },
                    "dangerouslyDisableSandbox": {
                        "description": (
                            "Set to True to override sandbox mode and run "
                            "commands without sandboxing."
                        ),
                        "type": "boolean",
                    },
                },
                "required": ["command"],
                "additionalProperties": False,
            },
            "eager_input_streaming": True,
        },

        # Optionally loaded MCP tools.
        # This example uses a Databricks MCP tool.
        {
            "name": "mcp__databricks__execute_sql",
            "description": (
                "Executes SQL queries in Databricks Unity Catalog. "
                "This tool executes SQL..."
            ),
            "input_schema": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": (
                            "The SQL query to execute. Use fully qualified "
                            "Unity Catalog names (catalog.schema.table) "
                            "for best results."
                        ),
                    },
                },
                "required": ["query"],
            },
            "eager_input_streaming": True,
            "defer_loading": True,
        },
    ],
}

In Claude Code 2.1.178, with my configuration, I observed the following initial root-agent tool set:

Agent
AskUserQuestion
Bash
Edit
Read
Skill
ToolSearch
Workflow
Write

The subagents I inspected received this reduced tool set:

Bash
Read
Skill
ToolSearch

Additional tools are loaded on demand.

Lifecycle of tools

At the beginning of each coding session, Claude Code receives the list of built-in tools and MCP tools in the system prompt (messages field, events with the system role). The schema of a tool describes the tool, its inputs, and its outputs. At the start, only the schemas of the built-in tools are loaded into context. Full schemas of the MCP tools are loaded dynamically using the built-in ToolSearch tool; otherwise, they’d pollute the model context.

The behavior is configurable through ENABLE_TOOL_SEARCH and the individual alwaysLoad settings in the mcpServers configuration of Claude Code.

Example request to load JetBrains MCP tools:

{
    "type": "tool_use",
    "id": "toolu_bdrk_01Sx8LNCXi9boS4p2iqMAzUk",
    "name": "ToolSearch",
    "input": {
      "query": "select:mcp__jetbrains__get_symbol_info,mcp__jetbrains__search_symbol",
      "max_results": 2
    }
  }

Response:

{
    "type": "tool_result",
    "tool_use_id": "toolu_bdrk_01Sx8LNCXi9boS4p2iqMAzUk",
    "content": [
      {
        "type": "tool_reference",
        "tool_name": "mcp__jetbrains__get_symbol_info"
      },
      {
        "type": "tool_reference",
        "tool_name": "mcp__jetbrains__search_symbol"
      }
    ]
}

After the ToolSearch tool is invoked and returns tools, they are added to the tools field, and Claude Code can then call them.

`input.messages` field

The messages field contains the message history of the agentic session. It’s worth highlighting that it is the complete conversation state — it contains all messages within the session, including tool call requests and results, and it always grows while the session is ongoing (until the next compaction). Each subsequent API call contains an increasingly large messages array, resulting in higher token consumption.

Here’s what it looks like:

"messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "<session context reminder>"
        },
        {
          "type": "text",
          "text": "<runtime context>"
        }
      ]
    },
    {
      "role": "system",
      "content": "<runtime-state>"
    },

    // ... actual conversation begins.
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "I'll load the duckdb skill first as instructed, then search the logs."
        },
        {
          "type": "tool_use",
          "id": "toolu_bdrk_01VUdJmzUP5Z3o4Qa85jmpdM",
          "name": "Skill",
          "input": {
            "skill": "duckdb"
          }
        }
      ]
    },
    // ... more messages there.
]

Session context reminder

It contains:

The contents of relevant CLAUDE.md files (including global and project-level files).
Unconditionally loaded Claude rules.
The current date.

Runtime context

It contains:

File metadata about all @ file mentions in the original prompt. For example, if you reference a directory, a listing with file information is included; referencing an individual file includes that file’s contents.
Deferred and MCP tools available for loading through the ToolSearch mechanism.
Descriptions of available agent types, including code review, security, testing, exploration, planning, and general-purpose agents.
Instructions supplied by the connected MCP servers. For example, in my case:
- Brave Search MCP should be used for web searches.
- VictoriaMetrics MCP requires consulting its documentation tool and minimizing query size.

The `output` part of the Anthropic API event

The output body is structured as a series of events:

[
  {
    "type": "message_start",
    ... # type-dependent fields here
  },
  {
    "type": "content_block_start",
    ...
  }
]

The event types that are useful for analytics are message_stop and message_delta.

In AWS Bedrock, message_stop contains the amazon-bedrock-invocationMetrics field with the token consumption of the current invocation, including input, output, cache-read, and cache-write token counts.

{
  "type": "message_stop",
  "amazon-bedrock-invocationMetrics": {
    "inputTokenCount": 6610,
    "outputTokenCount": 503,
    "invocationLatency": 10175,
    "firstByteLatency": 5185,
    "cacheReadInputTokenCount": 0,
    "cacheWriteInputTokenCount": 18378
  }
}

If you use Claude outside AWS, you can extract token consumption from the message_delta events.

Gotchas

The first event within the session (see session_id) can be a probe. For analyses such as “average first message size” or “MCP tools available,” it’s better to filter these events out. Their messages will look like this:

[
    {
      "role": "user",
      "content": "."
    }
]

If you invoke Claude models through AWS Bedrock or Google Cloud outside Claude Code, prompt caching is not enabled by default. Ensure you enable caching where it makes sense; otherwise, your cloud bill will be huge.

Conclusion

Observability of Claude Code is very limited by default, but with minimal setup, you can record LLM invocations and extract many important monitoring signals from them. This rich source of data helps you:

Identify callers and sessions.
Analyze and debug token consumption and latency.
Estimate costs.
Inspect reasoning effort, available tools, and MCP servers.

Invocation logs are powerful precisely because they capture so much — and that is also why they must be handled with great care.

References

Claude Code and Anthropic models on AWS Bedrock: Lessons learned

Sergey Ivanychev — Mon, 29 Jun 2026 18:05:39 +0000

Claude Code is currently one of the major coding AI harnesses, if not the major one. Many organizations choose to use Claude Code not through Anthropic plans but through hosted Anthropic models on third-party cloud providers. AWS Bedrock is an example of a service that provides a large catalog of various LLMs, including the latest Anthropic models such as Opus, Sonnet and Haiku.

Recent articles have found that Claude subscriptions are drastically more economically efficient (examples one, two), so why would anyone prefer to pay more for API billing? There are a few reasons:

Reliability: Anthropic services are still volatile enough to matter for critical operational workflows. At the time of writing, the public Claude status page was showing degraded performance for Claude API and Claude Code during an Opus 4.8 incident; the same status snapshot showed claude.ai at 98.96% uptime and Claude Console at 99.45% uptime over the previous 90 days. In our experience, Bedrock is far more reliable. Its cross-Region inference can automatically route requests to another AWS Region and use compute across Regions to handle unplanned traffic bursts. This can be crucial for latency-sensitive applications.

Compliance and security: Organizations that have compliance requirements related to third-party data processing (GDPR) need to be careful not to send any user or customer information to third-party services. If you already use AWS, then by using Opus on Bedrock, no data leaves the cloud boundary. With minimal setup, the LLM traffic doesn't leave your cloud either.
Logging and observability — using application inference profiles, which are simply pointers to specific models that annotate requests with a predefined set of tags, you can easily track model usage, group by internal user, etc. You also have a way to enable logging of all Bedrock model invocations for security auditing and analysis using model invocation logging

Application inference profile

When a Bedrock caller (AWS SDK, AWS CLI, HTTP) wants to use an LLM, it specifies one of the following:

An AWS-defined inference profile (looks like arn:aws:bedrock:<source-region>:<account-id>:inference-profile/global.anthropic.claude-opus-4-8)
A custom inference profile created by you (looks like arn:aws:bedrock:<region>:<account-id>:application-inference-profile/<application-profile-id>)

Custom application inference profiles are used to "wrap" model use with a predefined set of tags. A typical CloudFormation resource looks like this (docs)

Resources:
  ServiceAOpusProfile:
    Type: AWS::Bedrock::ApplicationInferenceProfile
    Properties:
      Description: Service A uses Opus 4.8
      InferenceProfileName: internal-service-a-opus
      ModelSource:
        CopyFrom: arn:aws:bedrock:<source-region>:<account-id>:inference-profile/global.anthropic.claude-opus-4-8
      Tags:
        - Key: owner-team
          Value: team-x
        - Key: environment
          Value: production
        - Key: service_name
          Value: service-a

Here we specify the description, which model is used (Opus 4.8 in our example), and a list of tags. As soon as the profile is created, you specify its ARN in Claude Code or in the Bedrock invocation.

Permissions

To be 100% sure that the caller can only use the inference profile to invoke the model, you need to set the permissions properly.

# Allow calling our inference profile
- Effect: Allow
  Action:
    - bedrock:InvokeModel
    - bedrock:InvokeModelWithResponseStream
  Resource:
    - arn:aws:bedrock:<profile_region>:<account_id>:application-inference-profile/<profile_id>

# Allow calling the foundation models in all regions (note `*`)
# but only through our inference profile (note `Condition`)
- Effect: Allow
  Action:
    - bedrock:InvokeModel
    - bedrock:InvokeModelWithResponseStream
  Resource:
    - arn:aws:bedrock:*::foundation-model/<model_id>
  Condition:
    StringLike:
      bedrock:InferenceProfileArn: arn:aws:bedrock:<profile_region>:<account_id>:application-inference-profile/<profile_id>

# Allow fetching the inference profile and model metadata descriptions.
# Narrow down the 'Resource' if you need to.
- Effect: Allow
  Action:
    - bedrock:ListFoundationModels
    - bedrock:GetFoundationModel
    - bedrock:ListInferenceProfiles
    - bedrock:GetInferenceProfile
  Resource:
    - '*'

This ensures that the caller must use an inference profile to call the model, and hence the tags will propagate properly.

Tip: you can use the following AWS CLI command to debug Bedrock permissions on a developer machine (it says hello to Opus 4.7):

aws bedrock-runtime converse \
    --region us-east-1 \
    --model-id global.anthropic.claude-opus-4-7 \
    --messages '[{"role":"user","content":[{"text":"Say hello in one short sentence."}]}]' \
    --query 'output.message.content[0].text' \
    --output text

Lesson 1: use application inference profiles for services, not people

If your COO sees these Bedrock usage reports, they will have questions. So use application inference profiles for all workloads in your organization to make spending transparent. Before:

After:

Lesson 2: don't use custom inference profiles for human usage

Claude Code integration with AWS Bedrock supports custom application inference profiles:

export CLAUDE_CODE_USE_BEDROCK=1
export ANTHROPIC_DEFAULT_OPUS_MODEL="arn:aws:bedrock:<region>:<account_id>:application-inference-profile/..."
export ANTHROPIC_DEFAULT_SONNET_MODEL="arn:aws:bedrock:<region>:<account_id>:application-inference-profile/..."
export ANTHROPIC_DEFAULT_HAIKU_MODEL="arn:aws:bedrock:<region>:<account_id>:application-inference-profile/..."

However, it's very cumbersome to manage them in the age of rapidly evolving models because

Frequent updates: Models are released frequently — you'll need to create inference profiles regularly. People tend to prefer to switch to the latest models quickly.
Multiple dimensions: Your organization may have multiple teams — you'd want to create and manage not one but X inference profiles for each LLM model and distribute them somehow
Technical savviness: People in your organization may have different levels of technical skill. Setting up inference profile environment variables or configs may be harder for people who don't work in engineering regularly

How do you track spending then? You have the following options:

Use a third-party LLM gateway: LiteLLM, Portkey, Bifrost to name a few. These services act like proxies between the caller and Bedrock, simplifying authentication, MCP management and letting you set up budgets per user. Go that way if you have extra money!
- Pros: these services have everything out of the box, let you specify concrete budgets, and enforce them in real time.
- Cons: typically, such services are not free when you need any security feature (such as OAuth integration).
Set up a CUR 2.0 export in AWS Cost Explorer and enable caller identity in the export. That way you'll get per-IAM session granularity in the export that will let you compute per-user spending.
- Pros: the figures are 100% accurate and match your cloud bill
- Cons: 24-hour latency

Enable Bedrock invocation logging and track usage in real time
- Pros: near-real-time log delivery with accurate token counts and user identity
- Cons: This is highly sensitive data! You need to restrict access and implement the processing yourself, e.g. using AWS Lambda and a database.

Lesson 3: enable invocation logs

For the purpose of usage tracking, I stripped out much of the information stored in the logs. It looks like this:

{
  "timestamp": "2026-06-16T12:00:56Z",
  "accountId": "...",
  "region": "us-east-1",
  "requestId": "...",
  "operation": "InvokeModelWithResponseStream",
  "modelId": "...",
  "input": {
    ...
    "inputTokenCount": 528,
    "cacheReadInputTokenCount": 1058,
    "cacheWriteInputTokenCount": 0
  },
  "output": {
    ...
    "outputTokenCount": 259
  },
  "identity": {
    "arn": "arn:aws:sts::0123456789:assumed-role/<role-name>/john.doe@yourorg.io"
  },
  "inferenceRegion": "eu-north-1",
  "schemaType": "ModelInvocationLog",
}

See that you have cacheRead, cacheWrite, input and output token counts, and identity.arn identifies the user that made the request. The final cost can be computed if you multiply these figures by the pricing. This approach works if the user uses any AI harness, not only Claude Code, because they all call AWS Bedrock in the end.

In addition to that, the logs store the whole payload of the request, which means that you can track

What models are used by your engineers
What MCP servers and Skills are popular
What the typical context size of the first message is (this helps to determine inefficiencies)
How many messages are in the sessions (this helps to determine if someone abuses the development agent)
Real-time spending alerts sent to Slack

and many more.

Conclusion

Using AWS Bedrock as a provider of Anthropic models for your organization doesn't come cheap, but it provides you with more control over spending, permissions, security and analytical capabilities.

References

Optimizing AWS S3 Costs via Granular Visualization

Sergey Ivanychev — Sat, 20 Jun 2026 11:40:32 +0000

TL;DR: Expressive visualisations help to save money. For example, if you’re using S3 as your cloud storage of choice, then S3 Server access logs show who accessed which S3 object. S3 Inventory regularly exports the metadata of each S3 object in your bucket. These two data sources can be used to answer questions like “How much does this prefix cost per month?”, “How frequently was this prefix accessed for past 30 days?”, or “How much will it cost to transfer this prefix to Glacier?” Go collect and join this data. Then, build treemap visualisations for insights on your spending.

Wow, that’s a huge cloud bill!

Companies build remarkable AI products that store and process a lot of data. Data is the new oil, they say. But how do we make sure we don’t spend too much on oil tanks?

Many data-driven companies have been there. To provide intelligent products, they need to be trained on significant volumes of data. So, they build a data lake. They save every single useful event, message, log, etc., hoping to extract value from it later on. Data engineers build sophisticated pipelines. There are tables that depend on other tables (some get deprecated, and others are unused). Basically, it all eats up a lot of space.

This is one of many ways a data lake can become a data swamp, which is hard to manage and usually costs a lot of money to maintain.

In this article, we’ll explore techniques that can help cloud infrastructure owners figure out how to clean up data. We’ll explore ways of solving problems in AWS and similar techniques that can be replicated in other cloud providers with similar APIs. I’ll add some links to the APIs that can be used if you use Azure or GCP.

Source of inspiration

As a macOS user and a software engineer, I sometimes struggle with my hard drive running out of space. There’s another similar problem: here’s a folder that consumes a lot of space. Why is it so heavy? What consumes that precious disk space?

To solve this problem locally, I use a tool called DaisyDisk. (There are probably many similar tools, even for other operating systems — e.g., WinDirStat of Windows.)

What is so cool about this tool is the visualization technique it uses to represent disk space. It’s called a sunburst graph. Here are some examples of how it can be built using Plotly:

Definition of Done

Building the sunburst visualization for S3 buckets would help us find the most expensive S3 folders to examine their cost and data size.

But we can do even better than that. What if we could see who accessed files from any given S3 prefix, including how frequently? We can build that visualization as well.

It’s also important to note that costs come not only from storage. We also pay for API requests to the storage infrastructure. If a big chunk of the AWS S3 bill comes from API requests, then how can we figure out what causes it? We can build the prefix to request visualization.

In this article, we’ll build:

A sunburst graph of the S3 bucket that visualizes the money we spend on each prefix
A visualization of the usage of each S3 prefix
A visualization of the frequency of requests to each S3 prefix

Data sources

We will use Python + PySpark for data processing and boto3 Python library for accessing the AWS API, but the logic can be replicated in any programming language. Make sure that the data processing framework you use can fit significant amount of logs. (For example, Pandas will choke when analyzing buckets with too many objects.)

To accomplish this task, we need to answer two types of questions for each S3 object:

How much does it cost to store this object per month?
How many requests have been done to this object for the past X# days?

S3 Server access logs and the S3 Inventory will help us answer these questions.

Other cloud providers have similar capabilities, but the APIs and data sources might be different. For example, Azure has blob inventory and storage analytics logging. GCP has storage inventory reports and access logs.

Step 0: Learn about the costs in your AWS region

First, we need to learn how much S3 charges us for requests/storage in our region. These costs differ from region to region. Learn yours on the AWS website.

Another option is to use the API to fetch up-to-date prices.

from mypy_boto3_pricing import PricingClient

pricing: PricingClient = boto3.client('pricing')
pricing.get_products(
    ServiceCode='AmazonS3',
    Filters=[
      {
         "Field": "regionCode",
         "Type": "TERM_MATCH",
         "Value": "us-east-1" # Change to your favourite region.
      }
    ]
)

The S3 pricing is sophisticated, but in this article, we’re interested in two cost components. Depending on the S3 Storage class of the object, we pay different rates for:

Storage (pay for GB per month)
Requests [pay for 1000 requests, GETs, COPYs, Lifecycle transitions and data retrieval requests (change of storage classes)]

For a given object, we know:

its size
its storage class
how many requests of each type were done to it for a past month

From that, we know how much this object costs us per month (the prices from step 0) and if it makes sense to delete or move it to another storage class.

Step 1: Learn about the storage (and storage costs)

The AWS API already supports retrieving all the metadata about specific S3 objects:

import boto3
from mypy_boto3_s3 import S3Client

s3: S3Client = boto3.client('s3')
resp = s3.head_object(
    Bucket='my-bucket',
    Key='some/file.txt'
)
print(resp['StorageClass'], resp['ContentLength'])

This works if you need to fetch this metadata for thousands of objects. But what if your bucket has millions of objects, and you want this metadata to be updated daily? On a daily basis, this would cost us:

object_count / 1000 * cost_of_1000_head_requests

And it will also work painfully slow because of the amount of daily requests we might have to make to fetch all metadata.

Luckily, S3 already provides us with a great tool to dump all of this metadata daily. It’s called S3 Inventory.

You can create per-bucket Inventory configuration in the Management section of S3 bucket. When enabled, S3 generates a report with the following file structure. Each timestamp prefix contains a manifest with a list of files from data/ folders that represent the state of the bucket for a given timestamp. data/contains files. (Note: This is in the format that you picked during configuration. I picked Apache Parquet.)

To read this report, you can use AWS Athena or manually parse the manifest for a given data and read corresponding files. I use Apache Spark to read and process the files (and assume that SparkSession spark exists) because their size can be significant.

from mypy_boto3_s3 import S3ServiceResource
import json

s3 = boto3.resource('s3')

object = s3.Object(
    'some-bucket',
    '.../2023-07-22T01-00Z/manifest.json').get()['Body'].read()
manifest = json.loads(object)

bucket = manifest['destinationBucket'][len('arn:aws:s3:::') :]
file_keys = [f['key'] for f in manifest['files']]
files = [f's3://{bucket}/{key}' for key in file_keys]

# We use Apache Spark to read manifest files. You can use
# any other approach that is suitable for you.
s3_inventory = spark.read.parquet(*files)
# Here we preprocess this data and store the result as parquet dataframe.
(
    s3_inventory.withColumn('parts', F.split(F.col('key'), '/'))
    .withColumn('full_path_length', F.size(F.col('parts')))
    .select(
        F.explode(F.sequence(F.lit(0), F.size(F.col('parts')))).alias('path_length'),
        F.array_join(F.slice(F.col('parts'), F.lit(1), F.col('path_length')), '/').alias(
            'prefix'
        ),
        (F.col('path_length') == F.col('full_path_length')).alias('is_file'),
        F.col('bucket'),
        F.col('storage_class'),
        F.col('intelligent_tiering_access_tier'),
        F.col('size'),
    )
    .write
    .parquet(...)
)

We have a daily updated dataset that contains metadata for all S3 objects in the given bucket. Note that for an object with key a/b/c.txt, the resulting DataFrame will have 4 rows with path_length equal to 0, 1, 2, and 3 along with corresponding prefixes a, a/b, and a/b/c.txt. This is required to group this data frame by prefix and aggregate total sizes and storage costs.

Moreover, the logs contain the following important fields:

bucket and prefix — the object that is being manifested
storage_class — the S3 Object storage class
intelligent_tiering_access_tier — if storage_class is Intelligent-Tiering, this field defines the tier
size — the file size

Step 2: Collect object access data

The S3 bucket Properties section provides S3 Server access logs settings, where you can configure exporting S3 object access events. They are exported as CSV files with about 1 minute latency.

The format of the logs is described here. Here’s how to read the logs using Apache Spark:

RAW_CSV_SCHEMA = schema = StructType([StructField(f'_c{idx}', StringType()) for idx in range(25)])

def read_bucket_logs(spark: SparkSession, bucket: str, path: str, day: date):
    return (
        spark.read.option('header', False)
        .option('delimiter', ' ')
        .schema(RAW_CSV_SCHEMA)
        .csv(os.path.join(path, f'{day.isoformat()}-*'))
        .select(
            f.col('_c0').alias('owner').cast(StringType()),
            f.col('_c1').alias('bucket').cast(StringType()),
            f.to_timestamp(
                f.substring(f.concat(f.col('_c2'), f.lit(' '), f.col('_c3')), 2, 26),
                'dd/MMM/yyyy:HH:mm:ss Z',
            ).alias('time'),
            f.col('_c4').alias('ip').cast(StringType()),
            f.col('_c5').alias('requester').cast(StringType()),
            f.col('_c6').alias('request_id').cast(StringType()),
            f.col('_c7').alias('operation').cast(StringType()),
            f.col('_c8').alias('key').cast(StringType()),
            f.col('_c9').alias('request_uri').cast(StringType()),
            f.col('_c10').alias('http_status').cast(IntegerType()),
            f.col('_c11').alias('error_code').cast(StringType()),
            f.col('_c12').alias('bytes_sent').cast(LongType()),
            f.col('_c13').alias('object_size').cast(LongType()),
            f.col('_c14').alias('total_time_millis').cast(LongType()),
            f.col('_c15').alias('turnaround_time_millis').cast(LongType()),
            f.col('_c16').alias('referer').cast(StringType()),
            f.col('_c17').alias('user_agent').cast(StringType()),
            f.col('_c18').alias('version_id').cast(StringType()),
            f.col('_c19').alias('host_id').cast(StringType()),
            f.col('_c20').alias('signature_version').cast(StringType()),
            f.col('_c21').alias('cipher_suite').cast(StringType()),
            f.col('_c22').alias('authentication_type').cast(StringType()),
            f.col('_c23').alias('host_header').cast(StringType()),
            f.col('_c24').alias('tls_version').cast(StringType()),
        )
        .withColumn('day', f.lit(day))
        .withColumn('bucket', f.lit(bucket))
    )

With this data frame, we can do the same trick as with inventory, where we aggregated the dataset by path and path_length. Note that the logs contain the following important fields:

bucket and prefix — the object that is being manifested
time — the timestamp when the request occurred
requester — the IAM role of the subject, who made the request
operation — the S3 operation

Step 3: Join and visualise

We ended up with two datasets:

Inventory dataset: keyed by bucket and prefix. Contains aggregate storage sizes per each storage class. Costs for storage per month storage_cost_per_month. (Remember Step 0? Multiply the storage sizes by the cost per storage class, and you’ll get the storage cost.)
Server access logs dataset: keyed by bucket and prefix. Contains an aggregate number of Tier 1 and Tier 2 requests per path. requests_cost_per_month costs for requests to prefix per month. (Multiply the number of requests by the cost per 1000 requests from Step 0.) log_access_count_30d is the logarithm of the number of requests for the given prefix.

When we join the datasets by bucket and prefix, we’ll have all the data to visualize the bucket costs. Furthermore, we add a column parent_prefix , which is equal to the folder, that contains the folder/object in prefix (e.g., for prefix=a/b/c parent_prefix=a/b).

Let’s use the plotly library to visualize the storage costs. The following call creates the visualization. Note that you can filter the df with a particular prefix and limit the depth (the number of inner folders) of visualization, as the rendering can be taxing for the web browser.

import plotly.express as px

px.treemap(df,
    names='prefix',
    parents='parent_prefix',
    values='storage_cost_per_month',
    color_continuous_scale='RdBu',
    color='log_access_count_30d',
    branchvalues='total')

The same trick can be done with requests (just replace storage_cost_per_month with requests_cost_per_month):

Did you notice the grey rectangles in the storage costs visualization? Grey comes from the color argument of the px.treemap call and means that the prefix hasn’t been accessed for the past month. These prefixes are great candidates for removal or another storage class transition.

Conclusion

The proposed way of visualization of S3 bucket costs can save a significant amount of funds by making S3 observability more granular. I managed to save dozens of thousands of dollars on cloud bills using these data sources, and at Constructor, we managed to save up to 40% of S3 costs using this technique. I suggest you try it as well.

DEV Community: Sergey Ivanychev

Anatomy of a Claude Code event

Practical summary

Logging setup

Sensitivity of the data

Anatomy of an AWS Bedrock log payload

Anatomy of an Anthropic API payload

input.system field

input.metadata field

input.output_config field

input.tools field

Lifecycle of tools

input.messages field

Session context reminder

Runtime context

The output part of the Anthropic API event

Gotchas

Conclusion

References

Claude Code and Anthropic models on AWS Bedrock: Lessons learned

Application inference profile

Permissions

Lesson 1: use application inference profiles for services, not people

Lesson 2: don't use custom inference profiles for human usage

Lesson 3: enable invocation logs

Conclusion

References

Optimizing AWS S3 Costs via Granular Visualization

Source of inspiration

Definition of Done

Data sources

Step 0: Learn about the costs in your AWS region

Step 1: Learn about the storage (and storage costs)

Step 2: Collect object access data

Step 3: Join and visualise

Conclusion

`input.system` field

`input.metadata` field

`input.output_config` field

`input.tools` field

`input.messages` field

The `output` part of the Anthropic API event