DEV Community: LakebaseGuru

How Databricks is disrupting Retail and E-Commerce with LTAP direct writes: OLTP and OLAP working together without overwhelming your application

LakebaseGuru — Thu, 15 Oct 2026 17:04:30 +0000

Every retailer runs two businesses on two kinds of databases.

The first is the store itself: carts, checkouts, order status, loyalty balances, store-level inventory, and the price a shopper sees on a shelf tag or product page. That’s OLTP, transactional work that has to answer in milliseconds because a customer is waiting.

The second is the intelligence behind the store: demand forecasts by SKU, store, and day; personalization scores for every shopper; optimized prices; assortment plans; and fraud models. That’s OLAP, analytics over billions of rows and years of history.

Retail has gotten very good at the second business. The trouble has always been getting its output into the first one.

Two engines, one platform

Databricks handles these two kinds of work with two engines that sit side by side on the same platform.

The lakehouse is where analytics and AI happen. It combines the low-cost, open storage of a data lake with the reliability and performance of a data warehouse. Data is stored in open Delta Lake tables on cloud object storage, and it’s processed by engines such as Apache Spark and Databricks SQL, which can scan billions of rows in parallel. This is where a retailer’s sales history, loyalty data, supply chain feeds, and clickstream come together, and where forecasting, pricing, and personalization models get built and run. It’s the OLAP side.

Lakebase is where applications live. It’s a fully managed Postgres database built into the Databricks Platform, designed for the fast, high-concurrency reads and writes that apps need: looking up one order, one shopper’s recommendations, or one item’s price, in milliseconds, thousands of times a second. Because it’s standard Postgres, app teams use the tools, drivers, and SQL they already know. Unlike a separate operational database bolted on from outside, Lakebase is governed by the same Unity Catalog as the lakehouse, so permissions, lineage, and the definition of “customer” or “item” stay consistent on both sides. It’s the OLTP side.

Having both on one platform is the starting point. The hard part, and the subject of this article, is moving large volumes of data from the lakehouse into Lakebase without slowing down the applications Lakebase is serving.

The nightly load problem

Here’s a scenario most retail data teams will recognize.

Every night the lakehouse produces something big. It might be a fresh price file for every item in every store, a new set of recommendation scores for millions of shoppers, or a demand forecast covering tens of thousands of SKUs across a couple of thousand locations. That last one alone runs to hundreds of millions of rows and is rebuilt from scratch every cycle.

All of it has to land in the operational Postgres database the apps actually read from, and that’s where it hurts.

In standard Postgres, every write goes through a single primary instance, the same one serving your storefront, store systems, and mobile app. A bulk load competes directly with live traffic for CPU and memory. Teams end up with familiar workarounds: maintenance windows, loads squeezed into the overnight gap before stores open, overprovisioned databases sized for the load instead of the shoppers, and a separate reverse-ETL tool just to manage the pain.

One such customer, loading about 1 billion rows a day through synced tables, needed more than 8 hours per load cycle and had to overprovision infrastructure just to keep their application from degrading. In retail, 8 hours is the whole window between the last online order of the night and the first store opening in the morning.

The fix: Build the data somewhere else, then hand it over

Databricks announced LTAP Direct Writes in Beta on October 8, 2026. It’s a new way to load data into Lakebase Postgres. The headline numbers are 1 TB in under 5 minutes and up to 147x faster heap loading, without consuming the resources your live application depends on.

With Direct Writes, the heavy construction work moves out of the Postgres primary and into Spark, running in parallel:

Parallel page building in sandboxes. Each Spark executor runs its own isolated Postgres instance and builds native Postgres data pages for its slice of the table. The driver assigns non-overlapping ID ranges so the slices never collide.
Frozen binary COPY. Inside each sandbox, data is loaded with binary COPY and FREEZE. Rows are treated as already committed, so there’s no transaction history to maintain.
Checksums on every page. Workers checksum the pages they build, and Lakebase’s pageserver validates them on ingest. A corrupted page can never become the source of truth.
Index building without re-downloading the table. Workers export key columns and row locations to object storage. Postgres’s standard B-tree builder then builds indexes from those, without pulling the full table back down.
One atomic handoff. The Spark driver writes a manifest and makes a single SQL call. The primary records only a small log entry describing the import, not the terabytes of data. The pageserver claims the files, warms them on the local SSD, and atomically swaps them into the visible table at commit.

The result is that the primary barely notices. Your storefront’s Postgres writes a tiny log entry instead of absorbing a terabyte of inserts.

This is what LTAP (Lake Transactional/Analytical Processing) means in practice. OLAP computes the bulk work it’s built for; OLTP keeps serving customers; and both share a single governed copy of the data under Unity Catalog.

What this changes for retail and e-commerce

Faster loads are nice. For retailers, the bigger change is that bulk refreshes no longer constitute an operational event.

Price and promotion changes stop waiting for a window. A full refresh of store-item pricing can land during the business day without slowing down point-of-sale lookups or the product page. Merchandising teams get to act when they decide, not when the loading schedule allows.

Personalization goes from nightly to whenever. If scoring millions of shoppers produces a big table, reloading it in minutes means you can rescore after a campaign launches, after a weather event, or ahead of a holiday weekend, and the app picks up the new scores right away.

Forecasts reach the systems that act on them. SKU-store-day demand forecasts can be served from Lakebase to replenishment apps, store ordering tools and delivery-promise logic, without a separate serving database and a pipeline to feed it.

Reference data rebuilds stop being scary. Item master, store hierarchies, assortment and planogram tables are rebuilt wholesale all the time. Snapshot syncs with Direct Writes make a full rebuild routine instead of a weekend project.

You can retire some reverse-ETL infrastructure. The write-up puts it directly: large bulk-refreshed datasets like nightly scoring tables, fresh feature sets and rebuilt reference data can stay in Lakebase. They no longer need a separate reverse-ETL tool just to avoid the load penalty.

Peak season gets safer. During Black Friday or holiday weeks, the worst thing a data team can do is compete with shoppers for database capacity. Direct Writes means that the bulk load and the checkout surge aren’t competing for the same CPU.

How it fits with synced tables

Direct Writes is an option on synced tables, which keep a Unity Catalog Delta table available in Lakebase as a read-only Postgres table. Synced tables have three modes, and it matters which one you pick:

Snapshot does a full refresh on every cycle. Direct Writes speeds up every sync in this mode, so it’s the natural fit for price files, forecasts, scores and reference data that get rebuilt wholesale.
Triggered and Continuous use Change Data Feed, so updates after the first load are already incremental. Direct Writes speeds up the initial load, which is often the biggest load you’ll ever do.

That suggests a simple retail split. Use Snapshot with Direct Writes for big tables rebuilt nightly or on demand. Use Continuous for data that changes transaction by transaction, like recent sales for returns lookups or live inventory positions.

Getting started

https://docs.databricks.com/aws/en/oltp/projects/sync-tables#accelerated-sync-with-ltap-direct-writes

The bottom line

Retail has always had a gap between what the business knows and what the storefront can act on. Moving analytics results into operational systems meant risking the customer experience, so it happened less often, more slowly, and with more infrastructure than anyone wanted.

Direct Writes closes much of that gap for the biggest, most disruptive loads. The lakehouse builds the data in parallel; Postgres accepts it in a single atomic handoff; and shoppers at the shelf, in the app, or at checkout never notice.

That’s the shift: OLTP and OLAP working together, on one platform, without the analytics side overwhelming the application side.

Databricks #Lakebase #Retail #Ecommerce #OLTP #OLAP #Postgres #DataEngineering

Originally published on Medium.

I Fixed a Pricing Disaster With a Database Branch

LakebaseGuru — Sat, 10 Oct 2026 13:00:18 +0000

Branch-based restores on Lakebase, told through a retail database.

Picture this. It is 1:47 AM on the Sunday before Thanksgiving week. Your team just pushed a pricing update to the Postgres database behind the e-commerce platform. By 2:15 AM the on-call engineer sees it: a bad join in the migration script overwrote regular prices with clearance prices across 40,000 SKUs. Every minute, orders keep flowing through at the wrong prices.

You have two bad options. Keep selling and bleed margin on the biggest week of the year, or take the storefront offline while the database team runs the classic recovery sequence: provision storage, copy the backup, replay WAL, wait. On a database this size, the waiting is measured in hours.

There is now a third option. Branch-based restores on Databricks Lakebase went GA on October 1, and they change the shape of this night entirely. Instead of a restore procedure, you open a branch at 1:46 AM, one minute before the deploy. Seconds later it exists, with its own compute and its own connection string. You verify the prices are correct, repoint the application, and you are back. The corrupted branch stays right where it is for forensics. The downtime is minutes, and most of those minutes are spent verifying, not waiting.

Why a restore became a branch

The reason this works is architectural. Lakebase separates Postgres compute from storage, and compute ships write-ahead log records to a durable layer that keeps the full history. A restore request maps your timestamp to the corresponding log sequence number, creates a branch pointing at that moment, and attaches compute. Nothing is copied. Copy-on-write means the new branch shares all underlying data through pointers; only new writes take new storage.

The operational detail that matters: the original branch is never touched. It keeps serving whatever is still connected to it while the restored branch comes up alongside. Recovery and investigation stop being the same fraught operation. You bring the past online next to the present instead of replacing the present with the past.

Databricks reports recovering a 100 TB database in seconds this way. Treat that as a vendor number, but the order of magnitude is believable for a simple reason: no data moves, so recovery time stops scaling with database size.

The same operation, four more retail mornings

Once a restore is a branch, it stops belonging to disaster recovery and starts belonging to the weekly routine. Stay in retail for a minute:

The replenishment engine misfires. Your demand forecasting model pushes a bad set of purchase orders overnight. Before the next model run, branch the database. Now you can run the old and new forecasts side by side against identical data instead of arguing about what changed.

Finance needs last Tuesday. An auditor asks what inventory levels looked like at close of business Tuesday. Branch Tuesday, run the queries, drop the branch. No ticket to the DBA team, no snapshot lifecycle to manage.

The AI agent goes sideways. You have an agent adjusting markdowns autonomously, which is exactly the kind of workload that makes experienced operators nervous. Snapshot the branch before the agent runs. If its pricing logic goes off the rails, the undo is a pointer flip, not a recovery project. Agents that write to production data need cheap instant undo the way code agents need git revert. Without it, every agent action is a one-way door.

The migration you were afraid to run. Schema change on the orders table during business hours. Branch first, run the migration against the branch, validate, then decide. The fear premium on routine database work drops toward zero.

The honest caveats

A few things to design around. The history window defaults to seven days and is configurable from two to thirty, so "any point in time" means any point inside your window. A restore covers every Postgres database on the branch, not a single one, and your application must be reconfigured to the restored branch's connection string. It is a new root branch, not an in-place rewind. Keep the window matched to your recovery objectives and treat the connection switch as part of the runbook.

The pattern underneath

First branching gave every pull request its own database. Now restores give every moment in history its own branch. The database is converging on the semantics developers already expect from git: cheap copies, instant reverts, history you can check out. In retail, where the database sits directly under revenue, that convergence is worth more than anywhere else. The teams that internalize it first will ship faster with less fear, because the cost of being wrong keeps dropping toward zero.

I Gave Every Pull Request Its Own Database

LakebaseGuru — Sun, 27 Sep 2026 21:01:29 +0000

I Gave Every Pull Request Its Own Database

Git-style branching for Postgres, wired into Jenkins, with a real repo you can run.

Every team has a dev database story. Ours went like this: one shared Postgres that five developers treated like a timeshare. Someone's migration failed halfway. Someone else was "just testing" against it. By Thursday it had the structural integrity of a sandcastle, and by Friday deploy we were all staring at a migration that had passed CI beautifully against a completely empty schema.

We don't really test migrations. We run migration-shaped SQL against tables with zero rows, then wonder why ADD COLUMN ... NOT NULL locks the orders table for nine minutes in production. The migration wasn't the problem. The test never proved anything.

Why not branch the database too?

We branch code for every pull request without thinking. But the database is this precious, monolithic thing you can't copy, so every PR shares one staging instance and developers queue behind each other.

What if a database branch was as cheap as a git branch?

That's what Databricks Lakebase is built for. It's managed Postgres where storage is separated from compute, and a branch is a first-class object: an instant, copy-on-write clone of production. You get a full, data-inclusive copy in seconds, on an endpoint that scales to zero when nobody's using it. Your throwaway test environment costs nothing while it sleeps.

So I built a complete, runnable Jenkins pipeline around it. Opening a pull request spins up a real database branch, migrations run against production-shaped data, tests assert against real rows, and nothing touches production until a DBA approves the exact migration that already passed CI. The DBA approval step is doing more work than it looks, and I'll explain why below.

The repo is here: https://github.com/harishshenoy552/lakebase-cicd

And if you'd rather watch than read, here's the 9-minute walkthrough:

https://www.youtube.com/watch?v=e2g8YtMcqFg

How the pipeline actually works

There are two paths, and the Jenkinsfile keeps them honest.

Pull request opened: Jenkins creates a branch called ci-pr-42 (or whatever your PR number is) straight from production. Copy-on-write, instant, full data. Then it applies the PR's migration to that branch and runs the test suite against it, not against an empty schema but against your actual rows. When the build finishes, pass or fail, the branch is torn down in post { always }, so nothing lingers and no orphaned branches keep eating storage.

Merged to main: the pipeline does something different. It skips the branch/test stages and parks at a DBA approval gate. A human from the dba-team clicks approve, and only then does the promote stage apply the migration to production. The DBA isn't reviewing SQL in the abstract. They're approving the exact migration artifact that already survived CI against production-shaped data.

Everyone works in one pipeline, off the same migration artifact and the same shape of data.

One thing I made sure of: the Jenkinsfile is just a thin wrapper around shell scripts, so the exact same commands run on your laptop and in CI. create_branch.sh, migrate.sh, test.sh, promote.sh, teardown.sh. What you run locally at your desk is what Jenkins runs.

And Jenkins is just the example here. Underneath it's a set of shell scripts, so the same pipeline works with whatever your team already uses: Azure DevOps, GitHub Actions, GitLab CI, CircleCI, or anything else that can run a shell. Swap the orchestrator, keep the scripts.

The repo even includes screenshots from a real Jenkins run of all five stages, so you can see the whole thing working before you wire it up.

The test that matters most

The demo scenario is an orders service. The seed data customers are Ada Lovelace, Grace Hopper, and Alan Turing.

Migration 002 adds a fulfillment_status column (NOT NULL DEFAULT 'pending', plus an index). And the test suite has this assertion:

def test_existing_rows_were_backfilled(conn):
    # The pre-existing production rows must have been backfilled with the default.
    # This is the assertion that fails on a naive ADD COLUMN NOT NULL without a
    # default - and the reason testing against real data (not an empty table) matters.

It checks that the pre-existing orders actually got backfilled with the default. If your migration was the naive ADD COLUMN NOT NULL without a default, this test fails on the branch, safely, before it ever reaches production. On an empty test schema, that same migration passes without complaint and then causes an outage on Friday.

That one test is the core argument for data-inclusive branches.

There's also a pure-SQL fallback (assertions.sql) for locked-down Jenkins agents where you can't install Python dependencies, and a Liquibase path if you want versioned changelogs with rollback instead of plain SQL files. Pick one per project.

What about the DBAs?

The obvious concern: you're letting developers spin up copies of production?

They're isolated copies, not access to production itself. And the promotion path still goes through a human gate restricted to the DBA group. DBAs actually get more signal than before. Instead of reviewing a SQL file in a ticket queue and guessing at the blast radius, they approve a migration that has already been executed and tested against a copy of their data.

Also worth noting: auth is via short-lived OAuth tokens from a service principal. There is no static database password sitting in a Jenkins credential store.

How it compares

Per-PR data-inclusive copy: Oracle does RMAN restore (hours). SQL Server does restore (slow). Aurora has fast clone (closest). Lakebase is instant copy-on-write.

Idle cost of test env: Oracle and SQL Server charge full license plus compute. Aurora is billed while running. Lakebase scales to zero.

Branch as first-class object: Oracle and SQL Server, no. Aurora has clones, not branches. Lakebase, yes.

Licensing cost of throwaway envs: Oracle is per-core, SQL Server per-core/CAL. Aurora has no license but compute adds up. Lakebase is usage-based with no license cost.

Aurora's fast clone is the closest alternative. The rest are slower, more expensive, or both.

Try it in about 3 minutes

git clone https://github.com/harishshenoy552/lakebase-cicd.git
cd lakebase-cicd
cp .env.example .env   # set your Databricks profile
source .env
./scripts/bootstrap_production.sh            # one-time: project + seed data
BRANCH_ID=ci-pr-42 ./scripts/create_branch.sh
BRANCH_ID=ci-pr-42 ./scripts/migrate.sh
BRANCH_ID=ci-pr-42 ./scripts/test.sh         # real rows, real assertions
./scripts/promote.sh                          # the "merge"
BRANCH_ID=ci-pr-42 ./scripts/teardown.sh

You'll need a Databricks workspace with Lakebase (Autoscaling tier), the Databricks CLI, psql, jq, and Python. The README has the full prerequisites and the Jenkins wiring instructions: multibranch pipeline, service principal, restrict the approval gate to your DBA group.

The point

We spent a decade making code branching instant and free. Databases never got the same treatment, so we built rituals around them instead: the staging database nobody trusts, the DBA ticket queue, the careful Friday deploy.

Copy-on-write branching takes away the usual excuse for skipping all that. A database branch can be as disposable as a git branch, migrations can run against real data, and DBAs get their time back for work that actually needs a DBA instead of rubber-stamping SQL files.

The repo is public and the video is linked above if you want to see it running.

Originally published on Medium: https://medium.com/@harishshenoy552/i-gave-every-pull-request-its-own-database-ac8212f7c713 · Repo: https://github.com/harishshenoy552/lakebase-cicd · Video: https://www.youtube.com/watch?v=e2g8YtMcqFg

DEV Community: LakebaseGuru

How Databricks is disrupting Retail and E-Commerce with LTAP direct writes: OLTP and OLAP working together without overwhelming your application

Two engines, one platform

The nightly load problem

What this changes for retail and e-commerce

How it fits with synced tables

Getting started

The bottom line

Databricks #Lakebase #Retail #Ecommerce #OLTP #OLAP #Postgres #DataEngineering

I Fixed a Pricing Disaster With a Database Branch

Why a restore became a branch

The same operation, four more retail mornings

The honest caveats

The pattern underneath

Further reading

I Gave Every Pull Request Its Own Database

I Gave Every Pull Request Its Own Database

Git-style branching for Postgres, wired into Jenkins, with a real repo you can run.

Why not branch the database too?

How the pipeline actually works

The test that matters most

What about the DBAs?

How it compares

Try it in about 3 minutes

The point