Databases are optimized for durable storage: transactions, constraints, consistency, recovery. Amazon OpenSearch Service is optimized for a different job: language-aware search (matching, synonyms, typo tolerance, relevance ranking), deep vector capabilities for hybrid search that combines lexical matching with semantic similarity, and sub-second analytics at scale. Both store data reliably. But each excels at what the other merely tolerates. If your application needs both great transactional writes and great search, you run both. Amazon OpenSearch Ingestion and its zero-ETL integrations now keep them in sync automatically.
The hard part has always been the sync. Your product catalog lives in PostgreSQL. Your session data lives in DynamoDB. Your search experience lives in OpenSearch Service. A customer updates their address, a product goes out of stock, a price changes. That change needs to appear in search results within seconds. Continuously. Without a fragile pipeline that breaks when someone alters a column. OpenSearch Ingestion eliminates that pipeline entirely for Amazon Aurora (Aurora), Amazon RDS (RDS), and Amazon DynamoDB (DynamoDB), with native change data capture and near-real-time synchronization out of the box.
This post walks through why the sync problem is genuinely hard, how OpenSearch Ingestion solves it, and what to expect when you wire it up.
Why Keeping Two Systems in Sync Is Genuinely Hard
The most common first attempt is dual write: the application pushes every change to both the database and OpenSearch Service directly. This works at small scale and feels clean because there is no separate sync system. It breaks once traffic grows. At scale, you are driving thousands of simultaneous connections to OpenSearch Service from your application tier, each with its own overhead. Under load, one write succeeds and the other fails, leaving the two systems out of sync. Retries help, but now your application code is handling distributed transaction semantics that belong in infrastructure, not business logic. Dual write also cannot handle schema migrations, bulk backfills, or replaying historical changes.
The next step teams try is polling. Query the database on an interval, diff against what OpenSearch Service has, push the deltas. This decouples the sync from the write path, which is an improvement. But polling intervals are a tradeoff with no stable answer: too frequent wastes resources, too infrequent means stale results. And the rate of change in a production database is bursty. A product launch triggers thousands of updates in minutes, then nothing for an hour. No fixed interval handles both.
The next level up is change data capture (CDC). MySQL has binary logs. PostgreSQL has logical replication. Read the transaction log, extract the changes, push them to OpenSearch Service. This sounds clean until you start implementing it. You need to parse log formats that differ between database versions, handle schema changes without breaking the pipeline, manage replication slot offsets, deal with connection failures and retries, and ensure exactly-once delivery. What started as "just read the logs" becomes a distributed systems problem with its own failure modes.
Then there is the initial load problem. CDC handles ongoing changes, but what about your existing million rows? You cannot run a massive SELECT against production without locking tables or degrading performance. So you end up with two completely different sync mechanisms: one for the historical backfill, one for the stream. Each has its own failure modes, its own monitoring, its own on-call rotation.
Teams build this infrastructure. It works, mostly. Then someone adds a column and the pipeline breaks. Or a schema migration changes a type and the index mapping rejects the new documents. Or the sync develops a subtle bug that silently drops updates for days before anyone notices. The real cost is not building the pipeline. The real cost is maintaining it indefinitely while the schema underneath keeps evolving.
Amazon OpenSearch Ingestion Now Connects Directly to Your Database
Amazon OpenSearch Ingestion now has native integrations with Amazon Aurora (Aurora), Amazon RDS (RDS), and Amazon DynamoDB (DynamoDB). For Aurora and RDS, it supports MySQL (version 8+) and PostgreSQL (version 16+). For DynamoDB, it uses DynamoDB Streams with point-in-time recovery for initial snapshots. All three provide automatic change data capture and near-real-time synchronization. No Lambda functions, no Kafka clusters, no Glue jobs, no custom code.
The architecture solves both the initial load and the ongoing stream in one pipeline. When you create a pipeline, OpenSearch Ingestion starts with a full snapshot export to S3. This handles the backfill: your database is not locked, the export happens in the background, and OpenSearch Service ingests from S3 at whatever rate makes sense. No production impact.
Once the snapshot is loaded, the pipeline switches to streaming mode. For MySQL, it taps into binary logs (row format, full image). For PostgreSQL, it uses logical replication. Every insert, update, and delete in your database appears in OpenSearch Service within seconds. The pipeline handles offset management, connection recovery, and delivery guarantees. You do not write retry logic. You do not manage replication slots manually.
OpenSearch Ingestion pipelines are configuration-driven. You specify schema mappings, define data mutations (rename fields, drop columns, enrich documents in flight), and control delivery behavior. The pipeline buffers requests for up to 72 hours during downstream outages, retries failed deliveries automatically, and routes undeliverable documents to a dead-letter queue. Default schema mapping is automatic (product IDs become keyword fields, timestamps are typed correctly, text columns are analyzed for full-text search), but you have full control to override any of it through the pipeline configuration.
Our team tested this with a PostgreSQL database containing a product catalog (detailed in Integrating Amazon OpenSearch Ingestion with Amazon RDS and Amazon Aurora). Setup took about twenty minutes: enable logical replication, create a Secrets Manager entry for credentials, define the pipeline in the OpenSearch console. The initial sync happened in the background. After inserting a new record into PostgreSQL, it was searchable in OpenSearch Service within seconds.
That three-second latency is the part that matters. Not the setup. Not the configuration. The fact that changes flow continuously without anyone thinking about it. The pipeline is not a batch job you schedule. It is a live connection that stays current as your database evolves.
Constraints Worth Knowing
The database and your OpenSearch Service domain must be in the same AWS account and region. You can sync one database per pipeline. Multi-AZ database clusters are not supported yet. These are planning considerations, not dealbreakers for most search and analytics use cases. If you run a multi-region architecture, you will need one pipeline per region.
Setup is prerequisite-focused rather than code-focused. For MySQL, enable binary logging with row format and full image. For PostgreSQL, enable logical replication (available in version 16+). For DynamoDB, enable DynamoDB Streams (new and old images) and point-in-time recovery (PITR) for the initial snapshot. Store your database credentials in AWS Secrets Manager. Then define your pipeline using the visual builder in the OpenSearch console or a YAML configuration. You specify which tables to sync and where in OpenSearch Service the data should land. The pipeline handles everything else.
What This Changes
This integration does not solve every data synchronization problem. It solves one specific, common, and expensive one: making your database data searchable and analyzable in real time without building custom infrastructure. If you have a product catalog in RDS and need full-text search with relevance ranking, this is now a configuration task. If you have transactional data in Aurora and need real-time analytics dashboards, you no longer need a team maintaining Kafka and custom consumers. If you have session or user-profile data in DynamoDB and need it searchable alongside your relational data, the same pipeline model applies.
The s/ETL/configuration/ substitution here is real. Teams that were spending months on sync infrastructure can now spend that time on search relevance, query tuning, and the features their users actually see. The plumbing disappears. The search quality work can begin.
If you are currently running custom sync infrastructure between RDS, Aurora, or DynamoDB and OpenSearch Service, or if you have been postponing search capabilities because the integration cost was not worth the effort, the calculation just changed. Enable replication or streams on your database, point OpenSearch Ingestion at it, and your data flows. Continuously. In seconds. Without you maintaining anything in between.
Top comments (0)