DEV Community: Faith Wachukwu

Airbyte vs Estuary for Postgres to Snowflake Replication

Faith Wachukwu — Tue, 10 Nov 2026 20:31:11 +0000

Estuary is the pick when Postgres changes have to land in Snowflake as they happen and one capture has to feed several destinations. Estuary streams changes continuously and can push a Snowflake materialization as fast as possible. Airbyte runs on a schedule: as often as every 5 minutes on Pro and Enterprise Flex, one destination per CDC source.

Estuary vs Airbyte for Postgres to Snowflake: Overview

Estuary captures Postgres changes continuously. Airbyte syncs on a chosen schedule.

Estuary's Postgres connector uses change data capture to continuously capture updates from a Postgres database into one or more data collections. Estuary’s Snowflake connector then uploads the changes to a Snowflake table stage before applying them to the target table in a transaction. Both standard and delta updates are supported, and Snowpipe Streaming is available for the delta bindings.

Snowpipe Streaming is the lowest-latency method Estuary has for loading Snowflake, and it is the default for delta update bindings. Estuary documents exactly-once delivery for every transaction that commits on that path.

That fits teams that need Snowflake tables to track Postgres closely and would rather not run a sync engine to get there.

Airbyte's Postgres connector reads the write-ahead log with logical replication to capture deletes. Its Snowflake destination loads data through Snowflake’s internal stage.

Airbyte fits teams whose downstream consumers tolerate a scheduled cadence, and teams that want to run the sync engine themselves.

Estuary vs Airbyte Pricing for Postgres to Snowflake: pricing options; shortest scheduled interval: 1 hour on Standard, 15 minutes on Plus, 5 minutes on Pro and Enterprise Flex

Primary key on every incrementally synced table; reset and resync after a schema change	Snowflake internal stage	Four credits per GB from a database source on Standard and Plus, as of October 2026	One CDC source to one destination	Airbyte Cloud, or self-managed Airbyte Core

Estuary's row is a set of settings you choose. Airbyte's is a set of intervals you schedule.

Freshness: how often each tool refreshes Snowflake

Freshness decides this pair, and it is the one property you cannot buy your way out of later.

Estuary's Snowflake materialization supports a sync schedule. Its other warehouse materializations do too, including Databricks, BigQuery, Amazon Redshift, MotherDuck, ClickHouse, and Azure Fabric Warehouse. Set it to 0s, and the materialization always pushes updated data as fast as possible.

Airbyte gives each connection three options that allow a sync to run. Airbyte Cloud caps those intervals by plan. The shortest scheduled interval is 1 hour on Standard, 15 minutes on Plus, and 5 minutes on Pro and Enterprise Flex. For both interval-based and cron schedules, Airbyte documents a ±30-minute window for initiating a sync.

If your consumers read Snowflake every few minutes, either cadence is fine. If they need updates pushed as fast as possible, use Estuary's Sync Frequency of 0s.

Postgres source demands and schema behavior

Estuary can automatically add new columns to Snowflake without resetting the pipeline when the columns meet the connector’s criteria, and recommended fields are enabled. Airbyte’s Postgres CDC connector requires a primary key on each table synced incrementally and recommends resetting and resyncing the affected data after a source schema change. For tables without a primary key, Airbyte recommends a full refresh.

Estuary captures from Postgres standby servers on version 16 or later. Each additional capture against the same database needs its own replication slot.

Airbyte can also run Postgres CDC from a read replica of the primary. It supports that starting at Postgres version 16.1 and connector version 3.6.21, provided the replica is configured to allow it. Airbyte also notes that logical replication increases disk space used on the database server.

Estuary normally uses a watermark table during backfills. Read-only capture avoids those writes, but at least one captured table must change regularly. If none does, include a heartbeat table and update it every few minutes.

Cost at Postgres-to-Snowflake volume

The two tools meter different things: Estuary charges per gigabyte and per instance, while Airbyte bills by volume or capacity.

Estuary's Cloud plan costs $0.50 per GB of data, as of October 2026. Connector instances cost $100 each per month, dropping to $50 for each instance after your sixth. Data is counted each time it is captured and each time it is delivered to a destination, including data produced by transformations. Estuary's own example prices 100 GB a month from Postgres to Snowflake at $300.

Airbyte splits its plans by billing model: Standard and Plus bill per use, while Pro and Enterprise Flex are priced by capacity. Airbyte Standard costs $20 a month for the first 5 credits and $5 per additional credit. If this is the only pipeline on the plan, the monthly bill is $1,995: $20 plus $5 for each of the remaining 395 credits.

Airbyte bills the serialized volume it measures, and it warns that this is likely larger than what a direct query against your database shows. The capacity plans go the other way: Airbyte says Pro and Enterprise Flex carry no charges based on data volume.

For the same 100 GB Postgres-to-Snowflake pipeline, Estuary Cloud costs $300 a month, compared with $1,995 on Airbyte Standard. Both of its rates are published, so a Postgres to Snowflake pipeline is arithmetic you can do today. Airbyte's credit plans meter a volume that can exceed the data you can see. Its capacity plans trade volume charges for fixed capacity.

Fan-out, operations, and deployment

One Postgres database, several consumers. This is where the two tools stop looking alike.

Estuary allows one-to-many replication, so a single source can feed warehouses, search systems, microservices, and AI pipelines at the same time. Estuary stores captured data in an Estuary-managed bucket by default, or in your own bucket if you configure one. A new destination can read the retained data without reading it again from Postgres.

Estuary offers more than 200 source and destination connectors, and it offers private and BYOC deployments on annual contracts.

Airbyte's Postgres CDC notes put a hard limit on fan-out. An Airbyte database source using CDC replication can be used with only a single Airbyte destination. That is the constraint to check against your own destination list before you pick a tool.

On Cloud Pro and Enterprise Flex, newly triggered syncs are queued when all committed data workers are in use. Airbyte lists 700 or more connectors across its plans. Airbyte Core is Airbyte's open-source product, which you deploy locally.

Estuary wins on fan-out, because one capture reaches many destinations from change data it already stored.

Verdict: pick Airbyte when, pick Estuary when

Pick Estuary when changes have to be queryable in Snowflake as they happen, and when one Postgres source has to feed more than one destination. Estuary captures changes continuously and pushes them at a frequency you set. Additional destinations can reuse the data Estuary has already captured and retained.

Pick Airbyte when you want to run the sync engine yourself, or when a scheduled cadence is enough, and each CDC source has one destination. Airbyte Core is the product to look at for that, since you deploy a local instance yourself.

For continuous Postgres to Snowflake replication, Estuary is the pick. Its bill is per gigabyte captured and delivered, plus per active connector instance, as of October 2026.

Measure how many gigabytes you capture and deliver in a month, then estimate the cost using Estuary’s published rates. Count how many destinations need the same Postgres changes. With Airbyte Postgres CDC, each additional destination needs a separately configured source that reads those changes again. Estuary captures them once, and each destination can reuse the retained data.

FAQ

Does Airbyte support Postgres CDC into Snowflake?

Airbyte's Postgres connector supports change data capture and replication using the xmin system column. Its Snowflake destination loads data through Snowflake's internal stage and supports automatic schema evolution. The CDC path reads the Postgres write-ahead log with logical replication to capture deletes.

How to Choose Embedding and Reranking Models for Domain-Specific Search

Faith Wachukwu — Thu, 22 Oct 2026 18:24:33 +0000

Quick Answer: Choose embedding and reranking models for domain search on a labelled set of your own queries, not on leaderboard averages. Shortlist by licence, context length, language, and size, measure recall@k for each embedder on your corpus, then add a reranker and keep it only if the measured gain justifies its latency. Run that bake-off on Superlinked Inference Engine: one API for the shortlist, on SIE Cloud or self-hosted, with open weights and shared GPUs.

A lawyer types "indemnification cap" into the contract search your team shipped last quarter. The top result is a well-written paragraph about indemnification in general, from the right kind of agreement, with no cap in it. The clause that actually limits liability to twelve months of fees sits at rank fourteen, below the fold and below the lawyer's patience. The embedder behind it scores near the top of the public leaderboard average. That average, it turns out, was never asked about your documents, which is a common oversight for a number that gets quoted in so many design reviews.

This is the ordinary failure of domain search. General-purpose models learn topic similarity very well. Domain queries often need something narrower: the defined term, the specific obligation, the number that qualifies the clause. A model can find the right neighbourhood and still return the wrong house.

The method below picks models for that job. It sits upstream of a full retrieval evaluation, which the legal retrieval evaluation guide in this series covers in depth.

Step 1: Build the query set from real queries

Start with what people actually type. Pull queries from search logs, support tickets, or the questions subject experts ask each other. Invented queries are cleaner than real ones, and that is exactly the problem: real queries are short, ambiguous, and full of the domain's shorthand.

For each query, record the documents or passages that answer it. This is the labelled set, and it is the most valuable artefact in the whole exercise, because every later decision is measured against it. Keep the labels at the level you retrieve at. If the index stores clauses, label clauses, not whole contracts.

Include the hard cases on purpose. Near-duplicate boilerplate, defined terms with a special meaning, and queries where the correct answer is a number all belong in the set. A query set of easy queries will rate every model highly and tell you nothing.

Step 2: Read the leaderboard for the right subset

Public benchmarks are still the right way to build a shortlist, as long as you read the part that matches your task. MTEB, the Massive Text Embedding Benchmark, covers retrieval, reranking, classification, clustering, and semantic similarity, and it publishes variants such as MTEB(eng, v2) along with domain-specific benchmarks for legal, medical, and code text.

The overall average blends all of those tasks. A model that excels at clustering and similarity can sit high on the average while being ordinary at retrieval. For a search system, sort by the retrieval columns, and then by any domain benchmark that resembles your corpus. Treat the result as a list of candidates, not as a ranking you can trust. Open models you can serve on SIE belong on that shortlist: embedders and rerankers from the catalog, measured on your queries.

Step 3: Shortlist by hard constraints

Before any testing, remove the models you could not ship. Four constraints do most of the work:

Constraint	Question to answer
Licence	Does the model licence permit your commercial use?
Context length	Does your chunk size fit without truncation?
Languages	Does it cover every language in the corpus and in the queries?
Size and cost	Can you serve it at your query volume and latency target?

Three to five embedders and two or three rerankers is a manageable shortlist. More than that and the bake-off turns into a research project. Include one general-purpose baseline, so you can tell whether a domain model is actually earning its place.

Step 4: Test each embedder with recall@k

Embed the corpus with each candidate and run the labelled queries. Recall@k asks a simple question: of the passages that answer the query, how many appear in the top k results? Pick k to match what the next stage sees. If a reranker will look at the top 50, measure recall@50 for the embedder, because the reranker cannot rescue a passage that never reached it.

Look past the average. Split the results by query type and read the failures by hand. A model that wins overall but loses every numeric query may be the wrong choice for a contracts product. Note latency and index size alongside recall, since a larger embedding dimension costs storage for every document you will ever index; models trained Matryoshka-style can be truncated to fewer dimensions, and it is worth measuring recall at the truncated size too.

One setting quietly ruins more bake-offs than any model choice: query instructions. E5 models expect a "query:" or "passage:" prefix, and Qwen3 embedders take a task instruction on the query side. Run each candidate the way its model card says, or the comparison measures your preprocessing, not the model.

Step 5: Add a reranker and measure the change

A reranker, usually a cross-encoder, reads the query and each candidate passage together and scores the pair; late-interaction and listwise rerankers exist too. It is slower than vector search, so it only sees the top of the list, and it often fixes exactly the failure in the opening scene: the right topic ranked above the right answer.

Measure what the user sees. Rerank the top k from each embedder and compare nDCG@10, which rewards the right passage near the top, before and after; MTEB's retrieval tasks report the same metric. Record the reranker's p95 latency at your k as well, because scoring fifty pairs per query is the most expensive thing on the query path. The table below shows the shape of the result you are looking for. The figures are illustrative, invented to show the method, and they describe no real model:

Candidate (illustrative)	nDCG@10, embedder only	nDCG@10, with reranker
General-purpose embedder A	0.61	0.74
Domain-tuned embedder B	0.68	0.76
Smaller embedder C	0.57	0.72

In a result shaped like this, the reranker narrows the gap between embedders, and the smaller model plus a reranker may beat the larger model alone, at lower serving cost once the reranker's per-query cost is counted. Only your own numbers can say whether that happens on your corpus.

Step 6: Decide the representation, and price the reindex

Dense vectors capture meaning. Sparse representations, such as SPLADE-style models, keep exact term matching, which helps with part numbers, statute references, and defined terms. Hybrid search combines the two. Multi-vector models, in the ColBERT family, keep one vector per token and score with late interaction, at the cost of a larger index.

Test the representation that addresses your observed failures, not the most sophisticated one. Then price the decision honestly. Changing the embedder later means re-embedding every document. Stores such as Qdrant and Weaviate let you add the new vector beside the old one in the same collection, which makes the cutover gentler, but the embedding run itself is unavoidable. A reranker, by contrast, can be swapped without touching the index at all.

Superlinked Inference Engine

If candidates are hosted APIs, the bake-off is a loop over API keys. If you self-host them, serving five or six models long enough to test them can dominate the work. SIE, Superlinked's inference engine, shortens that step. Models load on demand and are evicted least-recently-used when GPU memory fills, so switching candidates is a change of model ID in the request rather than a new deployment.

On SIE Cloud, the managed product, a bake-off can use the Cloud models directly: Qwen3-Embedding-4B, Snowflake's arctic-embed-l-v2.0, bge-m3 in sparse and multi-vector form, GTE-ModernColBERT, and the 0.6B and 4B Qwen3 rerankers, each priced per model on the Cloud pricing table. Self-hosted, the wider model catalog adds candidates such as gte-modernbert-base, multilingual-e5-large-instruct, bge-reranker-v2-m3, and mxbai-rerank-large-v2, with a documented path to add other Hugging Face models by config file. Self-hosting is also the mode to choose when the evaluation corpus cannot leave your environment.

The same function scores every candidate. gold maps each query ID to the set of passage indices that answer it; for a real corpus, batch the passage encoding and store the vectors.

import numpy as np
from sie_sdk import SIEClient

client = SIEClient(api_key="sk-sie-…", base_url="https://api.superlinked.com")

def recall_at_k(model, queries, passages, gold, k=50):
    P = np.array([v["dense"] for v in client.encode(model, [{"text": p} for p in passages])])
    P /= np.linalg.norm(P, axis=1, keepdims=True)
    total = 0.0
    for qid, text in queries.items():
        q = np.array(client.encode(model, [{"text": text}], is_query=True)[0]["dense"])
        top = np.argsort(-(P @ (q / np.linalg.norm(q))))[:k]
        total += len(gold[qid] & set(top.tolist())) / len(gold[qid])
    return total / len(queries)

def rerank(query, candidate_ids, passages, model="Qwen/Qwen3-Reranker-4B"):
    items = [{"id": str(i), "text": passages[i]} for i in candidate_ids]
    ranked = client.score(model, {"text": query}, items)
    best = sorted(ranked["scores"], key=lambda r: -r["score"])
    return [int(r["item_id"]) for r in best]

for model in ["Qwen/Qwen3-Embedding-4B", "Snowflake/snowflake-arctic-embed-l-v2.0"]:
    print(model, recall_at_k(model, queries, passages, gold))

Feed each embedder's top 50 into rerank and compute nDCG@10 on the result to fill the Step 5 table.

Honest take: SIE is the production layer for the embedders and rerankers this bake-off is choosing, and for the OCR and extraction models a domain corpus often needs beside them. One API runs every candidate, on SIE Cloud or self-hosted, with open weights and GPUs shared across the shortlist, so a new model is a model ID rather than a new deployment. The labelled query set is still yours to build. Once the numbers are in, the embedder and reranker you keep stay on that same API, in the cloud or on your own cluster.

FAQs

How large must the labelled set be before the numbers mean anything?

Enough to separate the candidates you care about. A few dozen queries will expose large differences, such as a model that fails every numeric query. Distinguishing two strong embedders usually takes a few hundred, spread across query types in proportion to real traffic. Label by the unit you retrieve, include hard negatives, and keep the set fixed while you compare models. Add queries later from production failures, since those are the cases the first set missed.

Should we fine-tune the embedder or just add a reranker?

Add the reranker first. It needs no training data beyond the evaluation set, it can be removed in a single change, and it does not force a reindex. Fine-tuning an embedder can outperform it on a stable, well-labelled domain, but it brings a training pipeline, a new model version to maintain, and a full re-embedding run every time the model changes. Consider fine-tuning when a reranker has plateaued and your labelled set is large enough to train on without overfitting.

Why not use OpenAI embeddings and move on?

Domain queries fail in the way the opening scene describes: a general model ranks the right topic above the right answer. Run the bake-off. Specialist embedders and rerankers on SIE, open weights on one API, are the production layer for that test and for the models you keep. SIE Cloud and self-hosted use the same SDK, so the candidate that passes the labelled set stays where you measured it.

Choosing, then committing

Model choice for domain search is a measurement problem. Build the query set from real traffic, shortlist from the right benchmark columns, test recall on your corpus, add a reranker, and only then settle the representation and the reindex. Run the bake-off on SIE Cloud; the same code runs against a self-hosted cluster for the models its catalog carries.