DEV Community: hisatomo futahashi

Getting Started with Bits Agent Builder: An Introduction and Five Real-World Use Cases

hisatomo futahashi — Wed, 26 Aug 2026 00:27:31 +0000

Hi, I'm @futahashi, a Principal Engineer at Nulab.

On August 10, 2026, I gave a talk at Japan Datadog User Group Meetup #20 (JDDUG #20) in Sapporo. The topic was "Getting Started with Bits Agent Builder and Real-World Use Cases." Bits Agent Builder is a feature that lets you build custom agents inside Datadog, the observability platform. This post is a digest of that talk.

On a personal note, the trip was a genuinely rewarding one: I got to return the favor to people who had traveled all the way from Hokkaido to speak in Fukuoka, and I finally met someone from Classmethod whose work I've long admired.

Getting Started with Bits Agent Builder: Real-World Use Cases - Speaker Deck

speakerdeck.com

Why do we need AI agents?

With the rise of generative AI, work like research, summarization, and writing text or code became something a human could delegate and speed up. Tasks too complex for traditional programming came within reach, and I've genuinely felt the amount of toil — repetitive, automatable manual work — shrink as a result.

The next thing that matters is expanding the space where AI works autonomously, without waiting for human instructions. That's where agents come in.

You give an agent a goal, tools, constraints, and trigger conditions. From there, it starts work on its own — not only when a human asks, but when an alert fires or a schedule comes due — making judgment calls as the situation demands, operating tools, and carrying the task through to completion. Chaining together multiple tool operations and processing steps no longer requires assembling a program. In the talk, I described this shift as "growing AI into a worker that acts on its own."

The work best suited to agents, I think, is work that's too complex for static automation but too repetitive for a person to handle every single time. Handing that work off means humans no longer carry every task themselves, and can spend their time on higher-value activities: prioritization, decision-making, communicating with people, and improving the systems around them. The goal isn't simply to make work faster — it's to free humans from the work itself.

What is Bits Agent Builder?

Bits Agent Builder is a custom AI agent that runs inside Datadog, and it became generally available in June 2026. I see three strengths as distinctive:

Rich context: It can use Datadog's rich data — logs, metrics, traces — directly.
Diverse tools: Roughly 3,000 actions, around 50 services, and 4 MCPs. It can operate AWS, Kubernetes, GitHub, Slack, and more.
Flexible triggers: It can be started from around 30 sources, including Workflows, Monitors, and Incidents.

In one sentence: it's an AI agent that runs on Datadog and uses the power of a unified observability platform to move your organization's operations forward.

Unlike an AI agent you fire up ad hoc on your local machine, this one is hosted on Datadog — so the data collected on the platform, the integrations you've already configured, and the wide range of triggers are all shared and reused across the team. Because it runs without depending on any individual's environment, you can make it a durable part of how your organization operates.

Four building blocks for creating and improving an agent

There are three ways to create an agent: from a prompt, from a Blueprint, or from scratch. Once created, you configure and improve it using four building blocks:

Instructions: What you tell the agent. Its role, its procedures, its constraints.
Tools: What the agent can use. If you don't give it a tool, it can't use it.
Automation: When it starts. Schedules, Monitors, Incidents, and so on.
Chat: The conversational interface. Run it, debug it, ask it questions.

Beyond these, you can also choose which model the agent uses, depending on the task.

Instructions aren't something you write once and walk away from. You watch how runs turn out, write down what you learn, and iteratively refine the agent's behavior over time.

Use cases

Bits Agent Builder hasn't been GA for long, but here are several agents I've actually run and found effective. I'm only showing the Slack reports here, but Notebook output is also worth trying — it renders as a clean, rich view.

Use case 1: Cross-organization monitoring of AWS Commitments, with purchase recommendations

AWS Commitments (Reserved Instances, Savings Plans, and the like) are AWS's mechanism for granting discounts in exchange for committing to usage up front — a handful of large purchases each year. In an organization running many AWS accounts, the types of commitments, when they were purchased, and when they expire end up scattered across accounts and services.

If you leave that management to individual teams, the frequency of review and the criteria for renewal decisions vary from team to team. Knowledge of how commitments work and when they apply also varies, so sometimes they aren't even recognized as an option and the opportunity is missed entirely. And when someone transfers to another team or changes roles, the history behind past decisions can disappear with them. Rather than relying on each team's knowledge and attention, you need shared policies and decision criteria, plus a mechanism that manages this continuously across the organization.

AWS's built-in features do surface expiration dates, utilization rates, and purchase recommendations. But organization-wide visibility lives in the management account, and a member account can generally only see its own information. Cross-account investigation requires the right permissions, and tracing actual usage across accounts means bouncing between multiple screens. Pulling all that together, layering your own company's policies on top, deciding what to renew and what to let expire, and getting that decision to the right person — doing all of that end to end with AWS features alone is not straightforward.

So I built an agent that monitors commitments across the entire organization against the same criteria. If all you want is a list, a script is enough. What I delegated to the agent was the judgment: taking usage and organizational policy into account, separating what it recommends renewing from what it doesn't, and reporting to Slack along with the reasoning behind each call. The value here is that finding these opportunities and continuing to review and decide becomes a shared organizational process, rather than something that depends on having a commitments expert or on any one team's effort.

Use case 2: Cross-organization cost trend analysis (Datadog and AWS spend)

Datadog and AWS costs span multiple accounts, services, and teams. If each team only reviews its own slice, the comparison windows and the thresholds for "this looks abnormal" won't line up, and changes that cross team boundaries can slip through. Rather than leaving continuous monitoring to individual teams, it's important to watch across the whole organization with the same lens and the same cadence, as shared work.

So when an anomaly appears in Datadog or AWS cost trends, I have an agent analyze it. Seeing that a bill went up or down is something an invoice or dashboard can tell you; the hard part is judging why it changed and whether it's a change that needs action. When this agent finds an abnormal cost trend, it doesn't stop there — it follows the related data and investigates the cause.

For Datadog cost analysis, it compares daily usage against the prior period to distinguish a temporary fluctuation from a real increase. For AWS cost analysis, it correlates telemetry, events, and logs alongside billing data, pinpointing the resources driving the increase down to the ID or ARN level. I find real value in getting insights that billing data alone could never produce.

What comes back from the agent isn't just the trend, but the insights from its investigation in a consistent format: likely reasons for the increase, the services and resources involved, and the evidence behind its conclusions. Because cross-organization monitoring and initial investigation are guaranteed by the system, whether an investigation happens at all — and how good it is — no longer depends on which team happens to have spare capacity. The people running cross-cutting operations, and the teams that need to act, can skip chasing data point by point and focus on verifying the identified cause and deciding what to do.

Use case 3: A scribe for incident response

Ever been asked "so, where are we right now?" in the middle of an incident? This agent takes the incident response conversation and records a summary, a timeline, action items, and responders in our company's own format. Pulling "what happened, and when did we do what" out of an unstructured conversation that's also full of side chatter is exactly the kind of judgment-heavy work you couldn't write as a program. Communication during response becomes structured, and responders can focus on the substance of fixing the incident.

As an aside, Datadog offers Incident Management, which handles timeline management, postmortem assistance, and more. We haven't adopted it yet, so I used Bits Agent Builder to replicate part of it. I'll skip the details here, but if it sounds relevant to you, it's well worth a look.

We also use Bits Investigation (formerly Bits AI SRE) in our incident response. By having multiple agents take over investigation and reporting, we're building an environment where humans can concentrate on directing the response and making decisions.

For more on how we use Bits Investigation and what's changed with it, see my earlier talk, "The Evolution of Bits AI SRE, Me, and My Organization."

The_Evolution_of_Bits_AI_SRE.pdf - Speaker Deck

speakerdeck.com

Use case 4: Security Signal triage

Plenty of security alerts turn out, after investigation, to need no action at all. But you can't know that until you investigate — so every one of them becomes a burden on a human.

When a Security Signal (Datadog's security event detection) fires, the agent immediately investigates the related information and produces an initial verdict. For example, a signal generated by a burst of AccessDenied errors caused by insufficient permissions was judged as benign — the detection itself was correct, but the activity wasn't malicious — and auto-archived with the reasoning and supporting information attached. Things that can't be confidently declared benign, such as a path traversal detection, get escalated to a human via a Security Case.

As an aside, Datadog also offers Bits Security Analyst. There are some signals it can't handle yet, though, so we're using Bits Agent Builder partly as a point of comparison. I'll leave the details for another time.

Use case 5: WAF log analysis and continuous improvement

A WAF isn't something you deploy, configure once, and forget. Writing a perfect rule set that blocks every attack from day one while never affecting legitimate requests is essentially impossible. You have to analyze real logs, confirm your rules are actually working, and improve them continuously. But finding anomalies in a massive volume of logs — while carefully assessing the blast radius so you don't accidentally block legitimate traffic — is far from easy.

This agent extracts anomalies from WAF logs and prioritizes them, evaluates how effective the existing rules are as defenses, and compiles improvement proposals into a report. The final rule changes are still a human call, made after reviewing the impact on the service; the agent handles the ongoing analysis and proposals. Down the road, this could extend into responding to attacks in real time.

Common pitfalls

Here are some mistakes people commonly hit when building agents. To make real failures easy to grasp, I deliberately exaggerated them in the slides.

Pitfall 1: Vague instructions produce the wrong procedure

This one comes from the cost analysis agent in use case 2. I assumed it would look at the cost metrics — instead, it tried to compute the numbers itself from usage volume and unit prices. The lesson: give the agent explicit instructions, and check the evidence behind its output so you can correct errors.

Pitfall 2: Context overflow

There's a limit to how much information AI can handle at once, and cramming too much processing into a single run leads to unintended behavior or outright errors. The fix is to split the task up and narrow the scope of the output.

What we got out of it

Outcome 1: Work starts moving proactively

Previously, a task would arise, a human would notice it, instruct the AI, and check the result. In other words, work sat still until a human noticed it and picked it up. Agents change that ordering. The task itself triggers the agent, which moves first, and what reaches the human is the investigation results and the points that need a decision. When a security signal fires, the initial investigation is already done before I even notice it.

The value isn't only in reduced hands-on time. Work that had been pushed aside as "we should do this, but nobody has the bandwidth" now starts moving the moment it arises. High-impact decisions stay with humans, but humans no longer carry every first response — they can concentrate on review and decision-making.

Outcome 2: Operations become articulated

Writing Instructions means putting your procedures into words. Ways of working that only existed in someone's head get written down. Once they're in words, the team can review them, improve them, and share the same decision criteria. Operational knowledge that lived with one person becomes an asset the team owns.

And when the agent doesn't behave as expected, that itself reveals where your own procedures and criteria were ambiguous.

Outcome 3: You can start small and grow big

You don't have to hand over all the judgment and execution from day one. Start with a narrow scope and delegate only investigation and recommendations. Humans review the results, and once you're satisfied with the quality, you can gradually expand its permissions and its scope.

The foundation is the three strengths I mentioned at the start. Because you can reuse the context already collected in Datadog, the integrations already connected, and the flexible triggers, you don't need to stand up a new execution environment every time — you can start small and grow both deeper and wider.

Managing usage and cost

In the talk I briefly covered how AI Credits work, and I also got a question from the audience about Bits Agent Builder pricing. Running agents continuously requires visibility into cost, not just capability.

Bits Agent Builder consumes AI Credits. According to Datadog's official pricing page, a single run averages about 3 credits — but note that actual consumption varies with task complexity and the volume of context the model processes.

The Agent Console (Preview) gives you a single place to review usage and cost across Datadog's AI agents (including Bits Agent Builder) and coding agents like Claude Code and Cursor. You can analyze usage over time by user, team, and agent, which makes it easy to pinpoint where consumption is running hot.

AI Credit Limits lets you set monthly caps at both the organization and per-user level. Once a cap is reached, new runs stop — a useful guardrail against unintended charges. Note, though, that these limits are shared across every Datadog AI feature that uses AI Credits, not just Bits Agent Builder, so you'll want to set them with your other usage in mind.

Wrapping up

The value of Bits Agent Builder is that it combines the rich context collected in Datadog, your already-connected tools, and a wide range of triggers — and turns complex, judgment-heavy work that humans used to carry into team operations that keep running on their own.

Three things I hoped to get across in this post:

Agents are a good fit for work that's too complex for static automation and too repetitive for a person to handle every time.
Bits Agent Builder leverages a unified observability platform to make investigation, judgment, and execution part of how your organization operates.
Don't aim for perfection from the start — try something small and feed what you learn back into the Instructions, and you can grow it safely.

That said, handing work to AI doesn't make operational expertise unnecessary. Deciding what evidence to base a judgment on, how far execution should be allowed to go, and under what conditions to hand off to a human — those are human decisions. Checking the evidence and quality of the output and continuously reflecting it in the Instructions is what raises an agent's quality.

I'd love for you to try building a custom agent with Bits Agent Builder to make your own work more efficient — and to share what you learn at a user group. JDDUG is the community for Datadog users, and I help run the Fukuoka side of it. Event announcements go up on the Japan Datadog User Group connpass, so please join us if you're interested.

Last but not least: thank you to everyone who organized JDDUG #20, everyone who came and talked with me at the venue, the Classmethod team for hosting us, and everyone at Datadog!

Datadog Archive Search is now GA — A Hands-on Walkthrough with S3

hisatomo futahashi — Thu, 14 May 2026 13:14:32 +0000

Introduction

Hi, this is futahashi from Japan. In this post, I'd like to share my experience trying out Datadog Archive Search. Logs are something everyone uses on a daily basis, but they are also notoriously hard to design well and easy to overspend on, so I'm sure many of you will appreciate this update as much as I do.

A small note before we begin: I'm not a native English speaker, so please bear with any awkward phrasing. I've done my best to keep the technical content accurate — feedback and corrections are very welcome.

This article focuses on AWS only.

TL;DR

You can stream-query archived logs immediately, without Rehydration
You can also run a Rehydration on top if you need to
You can narrow the scan by time or attribute
Pricing is based on the scanned data size
Each search keeps up to 100k events of search results for 24 hours, for free
Without Rehydration:
- No advanced analysis such as aggregations or visualizations
- No references from other features (Dashboard / Notebook / Log Explorer, etc.)

What changed

"I want to search for logs from three weeks ago" — but the Log Retention Periods on this environment is only 15 days. Situations like this come up sometimes. To deal with this, we humans have historically had the following options:

① Run a Rehydrate

Use Datadog Rehydrate feature to index Archived Logs. It runs as a batch process and takes anywhere from minutes to hours before the logs become searchable. Indexing also costs money.

② Use a different tool (Amazon Athena, etc.)

Use another tool built for analyzing archive logs. You lose Datadog excellent UI, and investigation efficiency drops.

③ Revisit Log Retention Periods

This only takes effect going forward, but depending on the frequency of the use case, revisiting Log Retention Periods is an option. However, the Indexing bill scales with retention, so weighing return on investment is important.

This is exactly where the now-GA Archive Search comes in. There is no batch waiting time like Rehydrate — results stream in incrementally. Before you run, you can preview your query against up to 1,000 sample Archive Logs, so you can verify your filter before kicking off a wasted scan. The billing model is also different from Rehydrate: Archive Search charges only for scan, and indexing through Rehydrate is optional. On top of that, each Archive Search keeps up to 100k events of logs for 24 hours, for free. You can search Archive Logs in Datadog quickly, easily, and intelligently, which is a big win both for investigation efficiency and for cost savings.

To summarize:

	Before (Rehydrate only)	Archive Search
Log retrieval	Batched, delayed	Streaming, fast
Wait time	Minutes to hours	Streamed
Pre-query check	Not possible	Verifiable on a 1,000-event sample
Indexing required	Required	Optional
Pricing axis	Index + Scan	Scan only (Index only if you Rehydrate)
Retention	3–180 days	24h (3–180 days if Rehydrated)
Event count	No limit	Up to 100k events per Archive Search, free for 24h
Example use case	Investigating Archive Logs beyond 24 hours; needing advanced analysis	Simply searching Archive Logs; intelligently narrowing logs before Rehydrate

Overall architecture

Here's an image of Archive Search and the things related to it. This is just my own cheerful mental model, not official information, so please take it as something to give you a feel for how it fits together.

① Logs are ingested into Datadog from a Datadog Forwarder, etc.
② Once logs are ingested, based on the Log Archive config, filtered logs are sent to the S3 bucket used for archiving. At this point, the Role that Datadog uses (usually DatadogIntegrationRole) needs a policy that grants read/write access to the S3 bucket.
③ When you run Archive Search, logs are retrieved from the archive S3 and kept in a dedicated Archive Search index for up to 100k events for 24 hours. This index is not accessible from Log Explorer.
④ If needed, you can Rehydrate to keep the logs in an index that's accessible from Log Explorer.
Notes
- ⑤ The regular Log Index keeps logs based on a config separate from Log Archive.
- ⑥ The reason I draw conventional Rehydration and Rehydration from Archive Search as separate is that, at the moment, Rehydrations triggered from Archive Search don't show up in Historical Views, so they appear to be treated as separate things — and I'm following that.

Let's try it

Here's what I tried with Archive Search, and what I learned. I'm assuming you're starting from creating a new archive, so if you already have an archive, feel free to skip ahead to the permissions section.

S3 bucket and IAM policy

First, if you don't already have an S3 bucket to store the archive logs, create one. The output path for archive logs is configurable, so you can also use an existing suitable bucket.

Next, attach an S3 read/write policy to the Datadog AWS Integration Role. Below is an example — replace the bucket name and prefix with values appropriate to your environment.

Example target bucket

Bucket name: some-datadog-enthusiast-bucket
Prefix: datadog/logs/

Example policy to add:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DatadogUploadAndRehydrateLogArchives",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": [
        "arn:aws:s3:::some-datadog-enthusiast-bucket/datadog/logs/*"
      ]
    },
    {
      "Sid": "DatadogRehydrateLogArchivesListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": [
        "arn:aws:s3:::some-datadog-enthusiast-bucket"
      ]
    }
  ]
}

Archive configuration

Create the archive. The fields are as follows:

Archive Name: The archive name.
Define Which Data To Forward: A filter for which logs to archive.
Set Archive Type: The archive type (Amazon S3 / Google Cloud Storage / Azure Storage).
Configure Bucket
- AWS Account: The AWS account and DatadogIntegrationRole.
- S3 bucket: The name of the archive S3 bucket.
- Storage Class: The object class (Standard / Standard IA / Intelligent Tiering / One Zone IA / Glacier IR).
- Path (Optional): The output prefix.
Advanced Settings
- Compression method: The compression method (ZSTD / GZIP).
- Encryption Type: The encryption type (Default S3 Bucket-Level Encryption / Amazon S3 Managed Keys / AWS Key Management Service).
- Tags in your Archive: A setting to additionally store Datadog Tags with the logs.
- Tags on your Rehydrated Logs: A setting to add specific tags to Rehydrated logs.
- Scan size for your rehydration: The setting for the maximum scan size of Rehydration.

With Define Which Data To Forward, you can configure a filter for the logs you want to archive, so you can avoid archiving unnecessary logs.

For Compression method, ZSTD is recommended when your goal is Archive Search or Rehydrate, since it reduces scan and egress cost. If you also plan to use other tools, GZIP may be an option as well.

You can also specify Storage Class and Encryption Type, which lets you optimize cost and apply appropriate security for your environment.

By running Test Configuration, you can verify whether the role has the appropriate read/write permissions on the configured bucket.

In addition, there are settings called Partition Attributes and Lookup Attributes, which are currently in Preview. These are said to reduce scan size by splitting data into directory hierarchies by attribute, or by skipping unneeded data blocks. I haven't applied for the Preview and couldn't try them out, so I'm leaving them out of this post.

Permissions

To run Archive Search, two kinds of permissions are required. Make sure to grant the necessary permissions for usage.

① Logs Write Historical Views: Required to run Archive Search
② Logs Read Archive: Required to read the archive

Datadog Admin Role and Datadog Standard Role carry both of these permissions, but Datadog Read Only Role only has Logs Read Archive — please be careful. Also, Archive Search results have Restriction Queries applied, so only logs you're allowed to view will be shown. That's reassuring!

Running Archive Search

From the Datadog left-hand menu, open Logs and switch to the Archive Search tab. On this screen, you can configure Archive Search, see a list of past Archive Searches, and start a new Archive Search. You can start one with the New Search button.

By the way, in the settings, you can configure the Rehydration volume limit and Rehydration retention periods as shown below.

Next, to choose what to search, fill in the following:

Archive Name: Specify the target archive.
Filter (Optional): Optionally specify a filter.
Timeframe: The target time range.
Mode
- ① Search: Up to 100,000 events kept for 24 hours, free.
- ② Search & Rehydration: 3–180 days of Indexing.

Specifying a filter lets you keep down the Indexing cost at Rehydration time, and, with Lookup Attributes (currently in Preview), it also enables reductions in scan size.

For choosing the mode, Search seems good for lightweight searches, and Search & Rehydration for use cases that need complex analysis. Even if you choose Search, you can still Rehydrate afterward, so there's nothing to worry about.

By checking Estimated scan size, you can see an estimate of the scan size before you actually run. Preview Log Sample lets you preview up to 1,000 logs from the same partition in the target period. You can click into logs to see details and to validate whether your filter query makes sense. That's a thoughtful touch!

Pressing Search runs Archive Search.

Checking the Archive Search results

Once Archive Search runs, results stream in incrementally and you can quickly search Archive Logs. This is a dedicated Archive Search view, different from Log Explorer. Aggregations and similar analysis aren't available, but filtering is.

In addition, you can show/hide columns from Options.

You can run a Rehydration as needed. With Rehydration, you gain the ability to explore and analyze the logs in Log Explorer. If you need complex analysis, or if you need to retain the logs for a certain period, go ahead and Rehydrate.

Both Archive Search and Rehydration results can be referenced from the Archive Search screen anytime within their retention period. For Rehydrated results, a link to Explorer is also generated — another thoughtful touch.

Notes and caveats

Here are the notes and caveats around using Archive Search.

① No aggregations / visualizations / integration with other features

Archive Search provides a dedicated view that's different from Log Explorer. Here, group-by aggregations, visualizations like pie charts, and integrations with other Datadog features (Dashboard / Notebook / Log Explorer, etc.) are not available. If you want any of those, you'll need to run the Rehydration option to perform Indexing.

② Scan appears to run at the same S3 path level

Archives have the following structure, and the hour directory contains multiple compressed files:

dt=YYYYMMDD/hour=HH/archive_*****

I couldn't find this documented officially, but the scan behavior looks like it operates at the same S3 path level. In other words, all the compressed data for one hour is in scope for the scan. As an extreme example, scanning 00:00–00:01 and scanning 00:00–00:59 have the same scan size, and 00:59–01:00 would be twice that size. As a workaround, you could split your archives based on the ranges you query most often, but in practice that's painful, so I'm looking forward to Partition Attributes and Lookup Attributes.

③ Query Preview is a 1,000-event sample

Query Preview samples appear to show 1,000 archive samples from the same partition that falls within the Timeframe you specified. As mentioned above, partitions are split at the hourly level, so these are 1,000 samples within an hour-sized window. Treat Preview as a feature for validating query syntax and confirming the archive structure — don't misread a zero-result preview as meaning "no matches for that time."

④ Scan stops at 100k events

Archive Search scans stop at 100,000 events per scan. To retrieve everything within a specified range, you'll need to use the results to exclude unneeded logs via a filter, or narrow the timeframe, so the result fits within 100k events, then re-run.

When re-running, the Clone feature is handy. The form gets pre-filled with the original search's filter and timeframe, so you can tweak only what you need and re-submit.

⑤ Cloud-side retrieval and transfer costs are billed separately

In addition to Datadog's scan-based billing, retrieval from S3 cold storage and egress to Datadog are billed by your cloud provider.

⑥ Archive config does not apply retroactively

Archive config applies only to logs archived after the setting is in place. It does not apply retroactively to logs archived in the past, so it needs to be set up in advance.

Pricing

Overview

The price table below uses the AP1 On-demand prices at the time of writing for reference. For the latest and most accurate information, refer to the official Datadog Pricing.

Item	Unit price
Archive Search	$0.07 / GB scanned
Temporary index for Archive Search results	Free (up to 100k events, 24 hours)
Rehydration Scan	$0.13 / GB scanned
Rehydration Indexing	Same as Logs Indexing
Logs Ingestion	$0.13 / GB ingested
Logs Indexing (15 day retention)	$3.19 / 1M log events
Forwarding to S3 / GCS / Azure (archive writes)	Included in Logs Ingestion

Archive Search is billed based on scanned data size. At the moment, you reduce scan size via the Archive & Forwarding filter and the Archive Search Timeframe. Once the Lookup Attribute setting (currently in Preview) is available, you'll also be able to reduce scan via the Archive Search filter. There's also an annual contract option for Archive Search, so if you can predict a certain volume of usage, you can get a commit-based discount. Archive Search results — up to 100,000 events — are kept for 24 hours with no Rehydration charge. That's lovely!

There's no additional charge for writing archive logs. Forwarding to S3 / GCS / Azure Storage is included in the Logs Ingestion price. Forwarding to destinations other than archives, such as external SIEMs and BI vendors, is billed separately, so be careful.

Rehydration pricing has a two-tier structure. One part scales with the size of the compressed logs scanned. The other is the Indexing cost when you index the matched logs, which follows the same pricing as regular Logs Indexing. Also, as mentioned above, Archive Search's scan price is roughly half the price.

Pricing comparison examples

① 15 Day Retention

Assume a service generates 100 GB / 100M log events per month, runs Indexed Logs at 15-day retention, and the data is roughly 10 GB on S3.

Item	Calculation	Monthly
Ingestion	100 GB × $0.13	$13
Indexing (15 Day Retention)	100M × $3.19	$319
Total		$322

② 7 Day Retention + Archive Search

Now consider running Archive Search alongside, so that 7-day retention is enough. Even if you go wild with Archive Search and scan the full 10 GB, the cost is as follows — about a 21% cost reduction. You can see how much of the total cost Indexing accounts for, and how cheap Archive Search is.

Item	Calculation	Monthly
Ingestion	100 GB × $0.13	$13
Indexing (7 Day Retention)	100M × $2.39	$239
Archive Search	10 × $0.07	$0.70
S3 Standard Storage	10 × $0.025	$0.25
S3 Internet Out	10 × $0.114	$1.14
Total		$254.09

③ 7 Day Retention + Archive Search + Rehydration

Under the same conditions, if you Rehydrate 10% of what Archive Search returned, the Indexing portion grows by 10%, so:

Item	Calculation	Monthly
Ingestion	100 GB × $0.13	$13
Indexing (7 Day Retention)	(100M + 10M) × $2.39	$262.9
Archive Search	10 × $0.07	$0.70
S3 Standard Storage	10 × $0.025	$0.25
S3 Internet Out	10 × $0.114	$1.14
Total		$277.99

I went with an extreme example where Archive Search and Rehydration are pushed to the limit, but in practice the scan volume should be much smaller than the full archive, and the Rehydrated index can be kept down further by tuning the time range and queries. You can also pick a shorter Day Retention for Rehydration to make it even cheaper.

Wrap-up

In this post, I covered how to use Datadog Archive Search and the caveats to watch out for. This feature solves the conventional Index and Rehydrate pain points around cost and time while still leveraging existing features, and it brings cost reduction and operational efficiency benefits to many users.

Depending on the use case, you can revisit Log Retention Periods, or even whether you keep logs in Indexed Logs at all, and dramatically cut Indexing cost. It also serves as a clever and inexpensive option as a pre-stage to Rehydrate. You can now narrow down past logs first, then Rehydrate only the parts you actually need.

I'm looking forward to seeing more cheap, fast, and clever evolutions from Datadog.