DEV Community: Tejaswita Soni

OpenTelemetry Observability Guide: How to Optimize Metrics, Logs, and Traces at Scale

Tejaswita Soni — Tue, 09 Jun 2026 09:28:50 +0000

Introduction

Modern cloud-native systems generate an enormous amount of telemetry data every second. Applications, containers, Kubernetes clusters, APIs, databases, and infrastructure components continuously emit metrics, logs, and traces to help engineering teams understand system behavior and troubleshoot issues. While observability has become essential for operating distributed systems reliably, it has also introduced a new challenge: managing the scale, cost, and quality of telemetry.

OpenTelemetry (OTel) has emerged as the industry standard for collecting and processing observability data. It provides a vendor-neutral framework for instrumenting applications and exporting telemetry to different observability backends. However, simply adopting OpenTelemetry is not enough. Without proper optimization strategies, organizations often face excessive telemetry ingestion costs, noisy dashboards, high-cardinality metrics, trace overload, and inefficient debugging workflows.

This article explores practical approaches for optimizing observability using OpenTelemetry. It focuses on metrics, logs, and traces individually while also discussing broader optimization strategies across the telemetry pipeline.

Understanding the OpenTelemetry observability pipeline

OpenTelemetry provides a unified framework for generating, collecting, processing, and exporting telemetry data. At its core, the OTel ecosystem consists of SDKs, instrumentation libraries, collectors, processors, and exporters.

Applications generate telemetry using OpenTelemetry SDKs or auto-instrumentation agents. This telemetry is then sent to the OpenTelemetry Collector, which acts as a centralized telemetry processing layer. The collector can receive telemetry from multiple sources, enrich it with metadata, apply filtering or sampling, and export it to one or more observability backends.

The observability pipeline typically follows this flow:

Application → OTel SDK → OTel Collector → Observability Backend

The OpenTelemetry Collector plays a critical role in optimization because it allows teams to manage telemetry centrally instead of implementing custom logic inside every application. By using processors and exporters efficiently, organizations can reduce unnecessary telemetry volume, improve signal quality, and optimize observability costs.

Common observability challenges at scale

This is where theory meets the reality of running observability in production. As distributed systems grow, observability platforms often become difficult to manage. Before exploring optimization strategies, it is important to understand the common challenges organizations face when operating observability platforms at scale.

Telemetry volume explosion

One of the biggest observability challenges at scale is the sheer volume of telemetry data being generated. Auto-instrumentation makes data collection easy, but it can also produce large amounts of metrics, logs, and traces. Across hundreds of services and thousands of requests per second, telemetry volume can quickly grow to hundreds of gigabytes per day. The problem is that much of this data provides little value during normal operations, yet teams still need to ingest, process, and store it, increasing observability costs and operational overhead.

Poor signal-to-noise ratio

As observability data grows, useful signals often get buried in large amounts of noise, making it harder to identify and troubleshoot real issues. A common result is alert fatigue, where engineers receive so many low-priority or repetitive alerts that important notifications may be overlooked. Similarly, large volumes of routine logs and traces from low-value operations, such as health checks and background jobs, can make it more difficult to find the data that actually matters during an incident.

High-cardinality metrics

Metrics become difficult and expensive to manage when labels contain highly dynamic values such as user IDs, session IDs, or request IDs. Each unique label value creates a separate time series, causing the number of stored metrics to grow rapidly. This phenomenon, known as cardinality explosion, increases storage requirements, consumes more memory, and slows down queries. To keep observability systems efficient and cost-effective, organizations should carefully design metric labels and avoid using unbounded values.

Instrumentation gaps and inconsistency

As systems grow, maintaining complete and consistent instrumentation becomes increasingly difficult. While auto-instrumentation can capture telemetry from common frameworks and libraries, important business workflows often require manual instrumentation. Without it, critical application behavior may remain invisible. At the same time, different teams may use different naming conventions, labels, log formats, and trace attributes. These inconsistencies make telemetry harder to search, correlate, and analyze across services, reducing observability effectiveness and slowing down troubleshooting.

Slow queries and dashboard performance

As telemetry data grows, observability platforms must process and search much larger datasets. Queries that once returned results quickly can become slower, making incident investigation and root cause analysis more difficult. This challenge is often made worse by high-cardinality metrics, long retention periods, and large volumes of logs and traces. Dashboards may also become slower to load and refresh, reducing the effectiveness of monitoring and increasing troubleshooting time during incidents.

Security and compliance risks

Observability data can accidentally contain sensitive information such as authentication tokens, API keys, email addresses, or customer identifiers. As telemetry volume and the number of services grow, identifying and controlling this data becomes increasingly difficult. Organizations must implement measures such as data masking, filtering, access controls, and encryption to protect sensitive information and meet compliance requirements. Failure to do so can lead to security incidents, compliance violations, and increased operational risk.

Cost Unpredictability

For many organizations, observability costs are among the fastest-growing line items in the infrastructure budget and among the hardest to predict. Usage-based pricing models from observability vendors mean that a traffic spike, a new service launch, or a misconfigured log level can double costs overnight. Without clear visibility into where telemetry volume is coming from and which data is actually being queried, it is nearly impossible to make informed decisions about what to keep and what to drop.

Why observability optimization matters

Observability optimization is not just about reducing costs. It is about improving the quality and usefulness of telemetry. Well-optimized observability systems help engineering teams detect incidents faster, reduce mean time to resolution (MTTR), and improve overall system reliability. By reducing unnecessary telemetry noise, teams can focus on meaningful operational signals instead of sorting through excessive data.

Optimization also improves backend performance. Smaller telemetry payloads, lower cardinality, and efficient retention policies lead to faster queries and more responsive dashboards.

From a financial perspective, observability optimization has become increasingly important because telemetry platforms often charge based on ingestion volume, storage, and query usage. Organizations that collect everything without proper governance can experience rapidly growing observability costs. A sustainable observability strategy requires balancing visibility, performance, and cost.

Core principles of OTel optimization

A few best practices can help optimize observability costs, performance, and data quality, regardless of the telemetry signal being collected.

Instrument once, send data anywhere: OpenTelemetry allows a single instrumentation setup to send telemetry to multiple observability platforms through the OTel Collector. This avoids maintaining separate instrumentation for different vendors and simplifies observability management.

Filter unnecessary data early. Not all telemetry data is valuable: Removing noisy logs, low-value traces, and unused metrics as close to the source as possible reduces storage costs, network traffic, and backend processing overhead.

Sample intelligently: Instead of keeping or discarding data randomly, prioritize telemetry that provides the most value. For example, always retain traces for errors and slow requests while sampling routine successful requests at a lower rate.

Use consistent standards: Follow OpenTelemetry semantic conventions and use consistent names for metrics, logs, and trace attributes across services. Standardized telemetry makes dashboards, alerts, and cross-service queries easier to build and maintain.

Metrics optimization in OpenTelemetry

Keep cardinality under control

High-cardinality metrics are one of the most common causes of observability cost and performance problems. Labels should use a limited set of values, such as status codes, regions, or payment methods. Avoid labels that can generate unlimited unique values, such as user IDs, request IDs, or session IDs.

# ❌ High cardinality
payment_counter.add(1, {
    "user_id": user_id,
    "request_id": request_id,
})

# ✅ Low cardinality
payment_counter.add(1, {
    "payment.method": "card",
    "payment.status": "success",
})

If you need detailed information about a specific user or request, traces are usually a better place to store that data.

Use delta temporality

Many backends only need to know how much a metric has changed since the last export interval. Using delta temporality can reduce the amount of metric data transmitted and processed while preserving the same operational insights.

Instead of continuously sending a running total, delta temporality exports only the change since the previous collection interval. This results in smaller metric payloads because only newly collected data is sent during each export cycle. For high-volume metrics, this can reduce network traffic and backend processing overhead while still providing the information needed for monitoring and alerting.

reader = PeriodicExportingMetricReader(
    exporter=exporter,
    export_interval_millis=60_000,
    preferred_temporality={
        Counter: AggregationTemporality.DELTA
    },
)

Drop unused metrics

Auto-instrumentation often generates many runtime and system metrics that teams never use in dashboards or alerts. Storing these metrics adds cost without providing meaningful value. Regularly reviewing and filtering unused metrics at the OpenTelemetry Collector helps reduce ingestion volume and backend storage requirements.

processors:
  filter/metrics:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - "^go\\.gc\\..*"
          - "^python\\..*"
          - "^runtime\\..*"

Reduce collection frequency

Collecting metrics more frequently than necessary increases data volume and storage costs. In many cases, dashboards and alerts do not require second-level granularity. Increasing the export interval from a few seconds to one minute can significantly reduce metric ingestion rates while still providing enough visibility for most monitoring and troubleshooting scenarios.

reader = PeriodicExportingMetricReader(
    exporter=exporter,
    export_interval_millis=60_000  # 1 minute
)

Log optimization in OpenTelemetry

Use structured logging

Structured logs are much easier to search, filter, and analyze than plain text logs. Instead of embedding information inside a message, store important details as separate fields with consistent names across services.

# ❌ Unstructured log
logger.info(
    f"Payment {payment_id} processed for user {user_id}"
)

# ✅ Structured log
logger.info("payment.processed", extra={
    "payment.id": payment_id,
    "payment.method": method,
    "user.id": user_id,
    "http.status_code": 200,
})

With structured logs, finding all failed payments or requests with a specific status code becomes a simple query rather than a complex text search.

Filter and sample low-value logs

Production environments often generate large volumes of DEBUG, TRACE, and routine INFO logs. Storing all of them increases costs and adds noise without providing much operational value.

processors:
  filter/logs:
    logs:
      exclude:
        match_type: strict
        severity_texts:
          - "DEBUG"
          - "TRACE"

For INFO logs, consider sampling to retain only a percentage of records while keeping all WARN and ERROR logs. This can significantly reduce log volume and storage costs.

Correlate logs with traces

Linking logs to traces is one of the most valuable OpenTelemetry capabilities. When logs include trace_id and span_id, engineers can move directly from an error log to the distributed trace that generated it.

with tracer.start_as_current_span("process_payment"):
    logger.error("payment failed", extra={
        "payment.id": payment_id,
        "error.code": error_code,
    })

This provides full request context and makes root cause analysis much faster during incidents.

Redact sensitive data

Not related to the optimization but important to also note that the logs can accidentally contain sensitive information such as email addresses, authentication tokens, API keys, or payment details. Sensitive data should be removed, masked, or hashed before it leaves your infrastructure.

processors:
  transform/redact_pii:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "payment.card_number")

Protecting sensitive information helps organizations meet security and compliance requirements while reducing the risk of data exposure.

Trace optimization in OpenTelemetry

Drop low-value traces

Many traces provide little diagnostic value, such as health checks, readiness probes, and metrics endpoints. Filtering these traces at the Collector can significantly reduce trace volume, especially in Kubernetes environments where these endpoints are called frequently. Removing this noise reduces storage costs and allows teams to focus on meaningful application traffic.

processors:
  filter/traces:
    traces:
      span:
        - 'attributes["http.target"] == "/health"'
        - 'attributes["http.target"] == "/ready"'
        - 'attributes["http.target"] == "/metrics"'

Use tail-based sampling

Rather than sampling traces randomly, tail-based sampling allows decisions to be made after a trace is complete. This makes it possible to retain all error traces, all slow requests, and a small percentage of normal traffic.

processors:
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: keep-slow-traces
        type: latency
        latency:
          threshold_ms: 2000

      - name: baseline-sample
        type: probabilistic
        probabilistic:
          sampling_percentage: 3

Tail sampling requires all spans for a given trace to reach the same Collector instance. If you're running multiple Collectors behind a load balancer, you need to add a load-balancing exporter in front of your tail-sampling Collectors that routes by trace_id.

Add business context to spans

Auto-instrumentation captures infrastructure details such as HTTP requests and database calls, but it cannot understand business operations. Adding custom span attributes provides valuable context for debugging and analysis based on real application behavior rather than infrastructure metrics alone.

with tracer.start_as_current_span("place_order") as span:
    span.set_attribute("order.total_usd", total)
    span.set_attribute("order.payment_method", payment_method)
    span.set_attribute("user.tier", user_tier)

Generate RED metrics from traces

The OpenTelemetry Collector can automatically generate Rate, Error, and Duration (RED) metrics from traces using the spanmetrics connector. This provides SLO-friendly metrics without requiring additional metric instrumentation in application code.

connectors:
  spanmetrics:
    dimensions:
      - name: http.method
      - name: http.status_code
      - name: service.name

When combined with exemplars, RED metrics can be linked directly back to representative traces, making it easier to move from a latency spike or error rate increase to the exact traces responsible for the issue.

Cross-signal optimization strategies

Metrics, logs, and traces provide the most value when they work together. An error log should be linked to the trace that generated it, metric anomalies should point to representative traces through exemplars, and traces should contain the same contextual information as related logs. These connections make it much easier to investigate issues and understand system behavior.

The OpenTelemetry Collector helps establish these relationships centrally within the observability pipeline, reducing the need for individual applications to implement and maintain cross-signal integrations themselves.

Using a single, unified pipeline for all telemetry signals is often easier to manage than maintaining separate pipelines for logs, metrics, and traces. This approach ensures that telemetry is processed consistently, enriched with the same metadata, and routed to the appropriate backends.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resourcedetection, memory_limiter, filter/traces, tail_sampling, batch]
      exporters: [otlp/backend, spanmetrics]

    metrics:
      receivers: [otlp, spanmetrics, prometheus]
      processors: [resourcedetection, memory_limiter, filter/metrics, batch]
      exporters: [otlp/backend]

    logs:
      receivers: [otlp, filelog]
      processors: [resourcedetection, memory_limiter, filter/logs, transform/redact_pii, batch]
      exporters: [otlp/backend]

The resourcedetection processor is particularly useful because it automatically adds infrastructure metadata such as cloud provider information, Kubernetes pod names, and node details to all telemetry signals. Having consistent metadata across logs, metrics, and traces makes correlation easier and improves the overall observability experience.

Common observability anti-patterns

Collecting more data than necessary

A common misconception is that collecting more telemetry automatically leads to better observability. In reality, excessive data increases storage costs, slows down queries, and makes it harder to find useful information during incidents. Before adding a new metric, log field, or span attribute, consider whether it provides actionable value and how often it will actually be used for monitoring or troubleshooting.

Running the Collector without memory protection

The OpenTelemetry Collector processes large volumes of telemetry and can experience memory pressure during traffic spikes. Without proper memory limits, the Collector may become unstable or even crash during periods of high load. Configuring memory protection and resource limits helps ensure that telemetry pipelines remain reliable, especially during incidents when observability data is needed most.

Inconsistent attribute naming

Using different names for the same piece of information across services creates unnecessary complexity. For example, different teams may use variations of the same attribute name, making cross-service searches, dashboards, and queries difficult to maintain. Following OpenTelemetry semantic conventions and enforcing consistent naming standards across teams helps create a more unified and searchable observability platform.

Protocol and backend mismatches

Observability backends do not always support the same protocols and exporters. Using an unsupported protocol can lead to connectivity issues, failed exports, and difficult-to-diagnose errors. Before configuring exporters, verify which protocols and endpoints are supported by your observability platform to avoid integration problems.

Vendor-specific instrumentation

Using vendor-specific SDKs and agents can make applications tightly coupled to a particular observability provider. While this may be convenient initially, migrating to another platform later often requires significant re-instrumentation effort.

OpenTelemetry helps avoid this problem by providing a vendor-neutral instrumentation layer. Applications can be instrumented once and telemetry can then be routed to different backends as requirements change.

Conclusion

OpenTelemetry makes it easy to collect metrics, logs, and traces, but collecting data is only the first step. As systems grow, organizations must ensure their observability pipelines remain scalable, cost-effective, and easy to operate.

Challenges such as high-cardinality metrics, excessive log volume, inefficient trace collection, and rising storage costs are common in large-scale environments. Without proper optimization, observability platforms can become expensive, difficult to query, and less effective during incidents.

By applying techniques such as filtering, sampling, consistent instrumentation, and efficient Collector configuration, teams can focus on the telemetry that provides the most value. The goal is not to collect all possible data, but to collect the right data that helps engineers monitor systems, troubleshoot issues, and maintain reliability at scale.

How to Build a Real-Time DynamoDB to S3 Analytics Pipeline with Apache Iceberg

Tejaswita Soni — Tue, 03 Mar 2026 07:43:23 +0000

Introduction

Data is at the core of almost every modern application, but it is rarely stored in a way that makes it immediately useful. Data engineering focuses on moving data from source systems, shaping it, and storing it so it can be analyzed and queried later. As applications scale and data volumes grow, building reliable and flexible data pipelines becomes an essential part of the system. On AWS, this is usually done using managed services that work well together-such as DynamoDB for storage, Kinesis for streaming, S3 for durable data lakes, and Glue and Athena for cataloging and querying. These services reduce operational effort, but building a dependable pipeline still requires careful handling of streaming behavior and schema changes.

In one of our recent projects, we needed to build a near real-time data pipeline that streamed data from DynamoDB into S3 (S3 Tables). The goal was to make this data available in real time for analytics while keeping the system scalable and future-ready. The pipeline had to handle frequent updates, concurrent writes, and schema changes, all without creating duplicate records when existing data was updated. Building this pipeline gave us a deeper understanding of how these systems behave in real-world, production environments.

In this article, I’ll walk through this DynamoDB-to-S3 streaming setup on AWS, explain the architecture choices, and share some of the practical issues we encountered along the way-along with what worked and what didn’t.

Data Analytics on AWS: Core Services

AWS Glue

AWS Glue provides a centralized data catalog that stores metadata about datasets in S3. It keeps track of table definitions, column types, and partitions. Glue makes it easier for query engines and processing jobs to discover and use data consistently.

Amazon Athena

Athena lets you query data in S3 using standard SQL without provisioning any infrastructure. It is commonly used for analytics, debugging pipelines, and validating data. Athena works directly with Glue tables and supports modern table formats like Iceberg.

Amazon Kinesis Data Streams

Kinesis Data Streams is used to capture high-volume, real-time data from multiple sources. It allows applications to consume and process data as it arrives. This makes it useful when low latency and ordered processing are important.

Amazon Kinesis Data Firehose

Firehose is a fully managed delivery service for streaming data. It buffers, batches, and reliably writes data to destinations like S3. Firehose is often used as the final step to land streaming data into analytics-ready storage.

Amazon MSK (Managed Streaming for Apache Kafka)

Amazon MSK provides a managed Apache Kafka environment without the operational overhead. It is used for complex event-driven architectures where fine control over streaming behavior is needed. MSK integrates well with downstream analytics and storage systems.

S3 Tables (Apache Iceberg)

Apache Iceberg is essentially a way to bring some structure and discipline to data sitting in object storage like S3. Instead of treating data as a bunch of files, Iceberg lets you work with it as a proper table. It keeps track of things like schema, partitions, and table metadata, which makes the data much easier and safer to query as it grows.

One thing Iceberg does really well is handle change. Columns can be added over time, records can be updated or deleted, and queries still return a consistent view of the data. This matters a lot when you’re dealing with streaming or near real-time pipelines, where data keeps changing and the schema isn’t always final from the start.

On AWS, S3 Tables build on top of Iceberg and take away a lot of the heavy lifting. You still get open formats like Parquet under the hood, but AWS manages the table metadata and layout for you. That means fewer moving parts to maintain and fewer things that can go wrong.

For our DynamoDB-to-S3 pipeline, this fits naturally. We needed to keep the latest version of records, support updates without creating duplicates, and still run analytical queries easily. Using Iceberg through S3 Tables gave us exactly that, without turning the storage layer into something we had to constantly manage ourselves.

Architecture Considerations

Why is the Firehose at the Center of This Design?

Before diving into the different approaches, it’s worth calling out one thing upfront: Firehose sits at the center of all of them.

In our case, Firehose was responsible for writing data into S3 Tables in Iceberg format. It handled batching, buffering, and file management, and made the data immediately queryable through Athena. Because of this tight integration, Firehose naturally became the final step in the pipeline.

So instead of thinking about three completely different architectures, it was more accurate to think about three different ways of feeding data into Firehose.

Approach 1: DynamoDB Streams → Lambda → Firehose → S3 Tables (Iceberg)

In this approach, DynamoDB Streams capture data changes and trigger a Lambda function. The Lambda function converts the DynamoDB JSON format into regular JSON and then sends the records to Firehose. Firehose takes care of batching, buffering, and writing the data to S3 Tables in Iceberg format, making it immediately queryable through Athena. This setup keeps the architecture simple, with fewer components to manage and reason about. Since the data is already transformed in Lambda before being sent to Firehose, there’s no need to enable Firehose’s Lambda transformation, which helps keep the pipeline straightforward and easier to operate.

Approach 2: DynamoDB → Kinesis Data Streams → Firehose

Another option we explored was routing DynamoDB changes through Kinesis Data Streams and then forwarding them to Firehose. Kinesis does offer more control around stream processing, scaling, and fan-out, which can be useful in more complex setups. In our case, though, it didn’t simplify the pipeline. The data coming from DynamoDB Streams was still in DynamoDB JSON format, so we had to use a Lambda function to transform it before Firehose could write it to S3 Tables. This meant adding another moving part without really gaining much. It also brought in additional cost and operational overhead, and for our requirements, it didn’t solve any problem that wasn’t already handled in a simpler way.

Approach 3: DynamoDB → MSK → Firehose → S3 Tables

In this approach, changes from DynamoDB are pushed into an MSK (Kafka) topic, and Firehose consumes those messages and writes them into S3 Tables using Iceberg. This can be a good option if Kafka is already being used, since it allows multiple consumers, flexible retention, and clear separation between producers and downstream systems. Firehose fits in neatly here by taking care of batching and delivering the data to S3, while DynamoDB data can be published to Kafka through a custom producer or a Lambda reading from DynamoDB Streams. However, for our use case, this setup added more complexity than we actually needed. Running and managing MSK also increased costs, which made it less practical for a pipeline whose main goal was to stream DynamoDB data into S3 for analytics.

Why did we choose the first approach?

After evaluating all three options, the DynamoDB Streams → Lambda → Firehose approach stood out as the most practical choice.

Kinesis added flexibility, but not without keeping Lambda in the loop for data transformation. MSK provided power we didn’t actually need, at a much higher cost. The first approach kept the architecture simple, serverless, and closely aligned with Firehose’s native integration with S3 Tables.

Most importantly, it let Firehose do what it does best-efficiently writing analytics-ready data to Iceberg-without introducing unnecessary complexity upstream.

Final Architecture

Our goal was to build a near real-time data pipeline that streams data from DynamoDB into S3, making it ready for analytics while handling schema changes and updates safely. The pipeline needed to maintain the latest state of each record, avoid duplicates, and scale for future growth.

At a high level, DynamoDB Streams capture all changes in the source table. A Lambda function transforms these events and manages schema updates, while SQS queues decouple schema changes from data ingestion. Firehose delivers the data into S3 Tables backed by Apache Iceberg, and Athena allows for direct queries on the stored data. Lake Formation ensures secure access to the tables. This setup ensures data consistency, supports schema evolution, and keeps analytics queries accurate and reliable.

Detailed Explanation

DynamoDB and Streams

Data is stored in DynamoDB, and DynamoDB Streams are enabled to capture every insert, update, or delete operation. These streams provide the event source for downstream processing.

Schema Change Lambda Function

The Lambda triggered by the stream has several responsibilities:

Converts DynamoDB JSON records into regular JSON.
Adds an action field to indicate insert/update or delete.
Checks if new columns are present in the record.
Updates the Iceberg table schema using Athena if a new column is detected.
Send a sample record for the new column to Firehose.
Logs schema changes in a DynamoDB tracking table with a status of sent.
Place the record into an SQS queue. If there’s no schema change, the record is simply queued.

Queue Processing Lambda Function

A second Lambda runs on a schedule to process the queued records:

Checks the schema tracking table for any new columns with status sent.
Validates the sample records in the Iceberg table.
If validation succeeds, it deletes the sample records and marks the status as completed.
Polls messages from the SQS queue, sends them to Firehose, and removes them from the queue.
If validation fails, it rewrites the sample records to Firehose to ensure the schema is recognized correctly.

Amazon Firehose

Firehose handles batching, buffering, and delivery of records to S3 Tables in Iceberg format. Based on the action field, it performs upserts or deletes, ensuring that the table always reflects the latest state of the data.

S3 Tables and Apache Iceberg

Iceberg adds a table layer on top of S3 storage, keeping track of schema, partitions, and snapshots. Using Iceberg with S3 Tables simplifies file and metadata management while supporting updates, deletes, and schema evolution without breaking queries.

Athena and Lake Formation

Athena queries the Iceberg tables directly in S3 for analytics. Lake Formation manages permissions, ensuring secure access to the database and tables.

Initial Migration

Before we could rely on the streaming pipeline, we also had to deal with the data that already existed in DynamoDB. Streaming only captures changes from the point it’s enabled, so without a backfill, all historical records would be missing from S3 and Athena.

To handle this, we did a one-time migration of the existing DynamoDB data. For this, we have created a migration utility.

The idea was simple: read the current state of the DynamoDB table and load it into S3 in a format that Athena could query. The approach in the repository helped with scanning the DynamoDB table and exporting all items to S3 in a structured way.

Once the data was exported, we made sure it followed the same shape expected by our streaming pipeline. This was important because we wanted historical data and new incoming changes to be treated the same way. Instead of creating a separate path just for migration, the exported records were pushed through the same logic we already had-schema detection, Iceberg table updates, and upsert handling.

Doing it this way gave us a clean starting point. Athena could immediately query the full dataset in S3, and once the migration was complete, DynamoDB Streams took over for ongoing changes. From that point on, both historical and real-time data lived in the same Iceberg table and behaved consistently during analysis.

Conclusion

This setup helped us move DynamoDB data into S3 in a way that was reliable, scalable, and actually usable for analytics. By combining DynamoDB Streams, Lambda, Firehose, and Iceberg on S3 Tables, we were able to keep the data close to real time while still handling updates, deletes, and schema changes cleanly.

One of the biggest lessons from this work was how important schema handling is in streaming pipelines. Changes don’t always show up when you expect them to, and if you don’t plan for that, data can quietly go missing. Treating schema changes as a first-class part of the pipeline made the whole system far more predictable and easier to operate.

Athena worked well for querying the data because it was simple to set up and fit naturally with S3 Tables. For most use cases, ad-hoc queries, validation checks, and scheduled reports were more than enough.

Overall, this approach struck a good balance. It kept the architecture simple, avoided unnecessary moving parts, and still left room to scale and evolve over time. If you’re building a similar DynamoDB-to-S3 analytics pipeline, this pattern is a solid place to start.

Struggling with schema evolution or duplicate records in your data pipeline? We can help you design, build and deploy a production-grade streaming pipeline tailored to your use case. Get in touch with our team.