Getting MySQL to DuckDB usually starts with mysqldump and a DuckDB file that gets copied around. It works for a first look. The file is stale the moment it is written, every copy drifts from the others, and an update made in MySQL never reaches any of them.
Here are the common ways to put MySQL data in front of DuckDB, each with a real use:
- The DuckDB MySQL extension. DuckDB logs in to MySQL and reads the current rows on every query. Nothing is copied, but every notebook that runs it puts load on the production server.
-
mysqldumporSELECT ... INTO OUTFILE, thenCOPYinto DuckDB. A script and a scheduler rebuild a DuckDB file. Freshness is hours to daily, and only full reloads are possible, so updates and deletes never arrive as changes. - Debezium and Kafka, with a consumer that writes Parquet. Changes land within seconds. You run Kafka, Kafka Connect, Debezium and the consumer, and you write the merge logic for updates and deletes yourself.
- Managed extract, load, transform (ELT) into a warehouse, with DuckDB reading the warehouse. Nothing to host, and minutes of freshness depending on the plan. You pay per row, per GB or per connector for a second copy you also pay to store.
This post walks through a fifth route. OLake Go, an open source (Apache 2.0) replication tool, reads the MySQL binary log (binlog) and commits change data capture (CDC) as Apache Iceberg tables into Apache Polaris. Polaris is an open source Iceberg Representational State Transfer (REST) catalog you host yourself, and DuckDB attaches it with one statement.
I ran this on 2026-10-29 with MySQL 8.4, OLake Go v1.1.0, Polaris 1.8.0 on MinIO and DuckDB 1.5.5. Every number below comes from that run.
What the MySQL to DuckDB pipeline looks like
-
Source: MySQL 8.4 with the binlog in
ROWformat, a database calledshop, and two tables,customersandorders. - Pipeline: OLake UI from Docker Compose. The first sync copies the tables in parallel chunks, and every later sync replays only the binlog since the saved position.
- Catalog and storage: Polaris and MinIO in Docker. On AWS, a plain S3 bucket and an Identity and Access Management (IAM) role take MinIO's place.
- Readers: DuckDB first, then ClickHouse 26.8.36 and Trino 484 on the very same tables.
You need Docker and about 40 minutes. DuckDB must be 1.4 or later; it installs the iceberg and httpfs extensions itself on first use.
Step 1: Turn on the binlog and create a reader user
Put these settings in my.cnf (or the parameter group on a managed server) and restart MySQL. log_bin is the one that cannot change without a restart.
server-id = 1
log_bin = mysql-bin
binlog_format = ROW
binlog_row_metadata = FULL
binlog_expire_logs_seconds = 604800
Then create the user OLake Go connects as, and the two sample tables if you want to follow along with the same shape:
CREATE USER IF NOT EXISTS 'olake'@'%' IDENTIFIED BY '<password>';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'olake'@'%';
FLUSH PRIVILEGES;
CREATE TABLE customers (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100), email VARCHAR(200), city VARCHAR(100), tier VARCHAR(20),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);
CREATE TABLE orders (
id INT AUTO_INCREMENT PRIMARY KEY,
customer_id INT, product VARCHAR(100), amount DECIMAL(10,2), status VARCHAR(20),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
FOREIGN KEY (customer_id) REFERENCES customers(id));
Load your own rows. My run started with 8 customers and 40 orders, and the change SQL in step 7 updates customers named Liam Carter and Zara Ali, so include those two if you want it to match.
Now the step most people skip:
ANALYZE TABLE customers, orders;
SHOW VARIABLES WHERE Variable_name IN ('log_bin', 'binlog_format', 'binlog_row_metadata', 'server_id');
SHOW GRANTS FOR 'olake'@'%';
OLake Go plans the parallel snapshot from table statistics. A freshly loaded table has none, and the first sync stops until you run ANALYZE TABLE.
Step 2: Start Polaris and MinIO, then create the catalog
services:
minio:
image: minio/minio
hostname: lake-minio
command: server /data --console-address ":9001"
environment: { MINIO_ROOT_USER: minioadmin, MINIO_ROOT_PASSWORD: minioadmin }
ports: ["9000:9000", "9001:9001"]
polaris:
image: apache/polaris:latest
depends_on: [minio]
ports: ["8181:8181", "8182:8182"]
environment:
AWS_REGION: us-east-1
AWS_ACCESS_KEY_ID: minioadmin
AWS_SECRET_ACCESS_KEY: minioadmin
POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,<root-secret>
polaris.realm-context.realms: POLARIS
Polaris keeps its state in memory in this setup, so it starts empty after every restart. Create the bucket, a catalog called olake, and a principal called olake with one grant:
P=http://localhost:8181
aws --endpoint-url http://localhost:9000 s3 mb s3://lakehouse
TOKEN=$(curl -s $P/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
-d grant_type=client_credentials -d client_id=root -d client_secret=<root-secret> \
-d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)
H=(-H "Authorization: Bearer $TOKEN" -H 'Polaris-Realm: POLARIS' -H 'Content-Type: application/json')
curl -s "${H[@]}" $P/api/management/v1/catalogs -d '{"catalog": {
"name": "olake", "type": "INTERNAL", "readOnly": false,
"properties": {"default-base-location": "s3://lakehouse/olake"},
"storageConfigInfo": {"storageType": "S3", "allowedLocations": ["s3://lakehouse/olake"],
"endpoint": "http://<minio-host>:9000/", "endpointInternal": "http://lake-minio:9000",
"pathStyleAccess": true, "region": "us-east-1"}}}'
curl -s "${H[@]}" $P/api/management/v1/principals -d '{"principal": {"name": "olake"}}'
curl -s "${H[@]}" $P/api/management/v1/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principals/olake/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principal-roles/olake_role/catalog-roles/olake -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles/olake_catalog_role/grants -d '{"type": "catalog", "privilege": "CATALOG_MANAGE_CONTENT"}'
💡 The principal response holds
clientIdandclientSecret, and Polaris shows the secret only once. Save it now; OLake Go and DuckDB both need it.
Keep the MinIO hostname free of underscores. Polaris parses addresses with Java's URI parser, which rejects lab_minio and fails every table creation.
Step 3: Start OLake UI and add the MySQL source
curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d
Open http://localhost:8000 and sign in as admin / password. Go to Sources, click Create Source, pick MySQL, and leave Enable CDC on.
In my run MySQL sat in another container, so the host was host.docker.internal, the database shop and the user olake. Initial Wait Time stayed at 10, and SSL was off for the local test.
Set up source tests the connection and checks binlog_access, log_bin, binlog_format and binlog_row_metadata before saving. A failing check shows the current value next to the recommended one.
Step 4: Add the Polaris destination
Go to Destinations, click Create Destination, pick Apache Iceberg, then Polaris as the catalog type and OAuth2 as the authentication type. The values from my run:
| Field | Value |
|---|---|
| REST Catalog URI | http://host.docker.internal:8181/api/catalog |
| Polaris Catalog Name | olake |
| REST Auth URI | http://host.docker.internal:8181/api/catalog/v1/oauth/tokens |
| REST Credential | <client_id>:<client_secret> |
| REST Scope | PRINCIPAL_ROLE:ALL |
| S3 Endpoint | http://host.docker.internal:9000 |
| AWS Region | us-east-1 |
Clicking Create exchanges the credential for a token, creates a test table through Polaris, writes to the bucket and drops the table again. A failure here is nearly always the secret, the grant or the catalog name.
Step 5: Create the job and run the first sync
Go to Jobs, click Create Job, name it mysql_to_duckdb and pick the source and destination. Under Advanced Settings, set Target Query Engines to DuckDB, plus Trino if it will read the tables too.
OLake Go then offers only the delete formats every listed engine can read. I picked Positional Deletes, the one DuckDB documents.
On the Streams page, tick customers and orders with Full Refresh + CDC and Upsert. Then click Create Job and choose Sync Now from the job menu.
The first sync copied 48 records (2.10 KB) in 17 seconds, container start included. The namespace is the job name plus the source database, mysql_to_duckdb_shop, and the Iceberg DB field at the top of the Streams page renames it before the first run.
Step 6: Query the tables in DuckDB
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs; LOAD httpfs;
CREATE SECRET polaris_secret (
TYPE iceberg,
CLIENT_ID '<client_id>', CLIENT_SECRET '<client_secret>',
OAUTH2_SERVER_URI 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
OAUTH2_SCOPE 'PRINCIPAL_ROLE:ALL'
);
CREATE SECRET store_secret (
TYPE s3, KEY_ID '<access key>', SECRET '<secret key>',
ENDPOINT '<minio-host>:9000', URL_STYLE 'path', USE_SSL false, REGION 'us-east-1'
);
ATTACH 'olake' AS polaris (
TYPE iceberg,
ENDPOINT 'http://<polaris-host>:8181/api/catalog',
SECRET polaris_secret,
ACCESS_DELEGATION_MODE 'none'
);
SHOW ALL TABLES;
SELECT id, name, city, tier FROM polaris.mysql_to_duckdb_shop.customers ORDER BY id;
The S3 secret and ACCESS_DELEGATION_MODE 'none' exist because of the laptop. Polaris vends the MinIO address as host.docker.internal, which only resolves inside Docker. Against real S3, drop both and DuckDB reads with the credentials Polaris vends.
Step 7: Change rows in MySQL and sync again
INSERT INTO customers (name, email, city, tier) VALUES
('Maya Patel', 'maya@example.com', 'Pune', 'silver'),
('Lucas Silva', 'lucas@example.com', 'São Paulo', 'bronze');
INSERT INTO orders (customer_id, product, amount, status) VALUES
(9, 'Laptop', 1299.00, 'placed'), (9, 'Mouse', 25.50, 'placed'),
(10, 'Monitor', 349.99, 'placed'), (3, 'Keyboard', 89.00, 'placed'), (7, 'Webcam', 79.90, 'placed');
UPDATE customers SET tier = 'gold' WHERE name = 'Liam Carter';
UPDATE customers SET city = 'Abu Dhabi' WHERE name = 'Zara Ali';
UPDATE orders SET status = 'delivered' WHERE id IN (1, 4, 7);
Those are 7 inserts and 5 updates. Sync Now read 12 records (563 bytes) from the binlog and finished in 9 seconds.
For customers the commit added one data file with 4 rows, the 2 inserts and the new versions of the 2 updated rows. One positional delete file retired the 2 old versions, and the log says so: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2.
Run the same SELECT again, with no refresh. DuckDB returns 10 rows, one per key: Liam Carter is gold, Zara Ali is in Abu Dhabi, and the two new customers are there.
The _op_type column tells you how each row got there: r from the snapshot, c for an insert, u for an update and d for a delete. OLake Go also adds _olake_id, _olake_timestamp and _cdc_timestamp to every table.
Read the same tables from ClickHouse and Trino
This is where writing Iceberg beats exporting a file. ClickHouse mounts the catalog as one database:
SET allow_database_iceberg = 1;
CREATE DATABASE polaris
ENGINE = DataLakeCatalog('http://<polaris-host>:8181/api/catalog')
SETTINGS catalog_type = 'rest', warehouse = 'olake',
catalog_credential = '<client_id>:<client_secret>',
oauth_server_uri = 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
auth_scope = 'PRINCIPAL_ROLE:ALL', vended_credentials = true, region = 'us-east-1';
SELECT id, name, city, tier FROM polaris.`mysql_to_duckdb_shop.customers` ORDER BY id;
Trino needs one etc/catalog/polaris.properties file with iceberg.catalog.type=rest, the OAuth2 credential, and iceberg.rest-catalog.vended-credentials-enabled=true. Trino 484 returned the same 10 rows.
An orders roll-up matched MySQL to the cent in all three engines: 16 delivered orders for 8659.68, 18 placed for 10522.26 and 11 shipped for 6397.25.
Tips from the run
-
decimalarrives asDOUBLE. The totals above still matched to the cent, but round at query time when exact decimal math matters.datetimeandtimestampbecomeTIMESTAMP WITH TIME ZONEin UTC. -
json,enumandtimearrive asVARCHAR. Parsejsonwith DuckDB'sjsonextension. -
Watch binlog retention. A job paused longer than
binlog_expire_logs_secondscannot resume and needs a fresh snapshot. -
Hot tables can be cached. DuckDB reads Parquet over the network on each query, so
CREATE TABLE local AS FROM polaris...once for a table you query all day. -
Plan compaction. Each sync adds a small data file and a delete file per table, and nothing compacts them for you. Spark's
rewrite_data_filesworks through the same Polaris endpoint.
How the MySQL to DuckDB methods compare
| Method | Freshness | Updates and deletes | What you run |
|---|---|---|---|
| DuckDB MySQL extension | Live, per query | Reads the current row | A MySQL login for DuckDB |
mysqldump or INTO OUTFILE, then COPY
|
Hours to daily | Full reloads only | A script and a scheduler |
| Debezium, Kafka and a Parquet consumer | Seconds | Yes, with your own merge logic | Kafka, Kafka Connect, Debezium, a consumer |
| Managed ELT to a warehouse | Minutes, by plan | Varies by tool | Nothing; priced per row, GB or connector |
| OLake Go to Iceberg on Polaris | CDC on a schedule down to one minute | Yes, as Iceberg row-level deletes | One Docker Compose stack plus Polaris |
When one analyst queries one small table, the MySQL extension or a nightly COPY is less to run than a catalog. Tables without a primary key are another poor fit: inserts replicate, but updates and deletes cannot be matched to a row.
FAQ
Why does the DuckDB secret need ACCESS_DELEGATION_MODE set to none?
On a laptop, the MinIO address Polaris hands out is a Docker-internal name that DuckDB cannot resolve. Turning delegation off makes DuckDB use your own S3 secret with a reachable endpoint instead. On AWS S3 the vended address works everywhere, so both lines go away.
Can DuckDB write back to these tables?
No. The attached Polaris catalog is read-only from DuckDB, and OLake Go is the only writer. Copy a table into a local DuckDB table if you need to transform it in place.
How do I add a column without breaking the job?
Add it in MySQL as usual. OLake Go adds it to the Iceberg table on the next sync, and DuckDB shows it after the next ATTACH. Widening a type such as int to bigint works too, while a narrowing change fails the sync instead of corrupting the table.
Does the change sync rescan the whole table?
No. It opens the binlog at the position saved by the last run and reads only what changed. In my run that was 12 records and 563 bytes, against 48 records on the first sync.
Where to go from here
- Point the source at your real MySQL. Amazon Relational Database Service (RDS), Aurora, Cloud SQL and Azure set the binlog options through a parameter group or flag instead of
my.cnf. - Move Polaris off in-memory state and onto Postgres before anyone depends on it.
- The full guide, with the data type table, the Trino properties and the troubleshooting list, is on olake.io.
If a step does not match what you see, say so in the comments and I will update the post.







Top comments (0)