DEV Community

Cover image for Moving one tenant's rows to their own Postgres without rewriting the app
Evgeny Shevtsov
Evgeny Shevtsov

Posted on Edited on

Moving one tenant's rows to their own Postgres without rewriting the app

At some point, you'll be a multi-tenant SaaS on Postgres, and you'll sell to a bank or a clinic.
And their security review will say: "our data should be in our database."

There are two typical responses, and I'm not a fan of either. You can either deploy single-tenant for that customer, which is another thing to upgrade and watch. Or you can rewrite the data layer so that every query knows which database to talk to. For the few customers that asked.

I wrote a third option: LokaDB. It's a Postgres wire protocol proxy. All you change is the connection string in your application. Here's how it works, and more importantly, how much it costs, and what's bad about it.

The idea: a tenant is a partition, and you can have remote partitions

Postgres has two things that play nicely together: declarative partitioning, and postgres_fdw. You can have a partition of a table be a foreign table. This means you can physically have part of a table on another server, but SQL sees it as one table.

LokaDB turns the tables you choose into organization-partitioned tables. It does this in place, t does this in place, without copying rows. For the organizations that are in your cloud database, you have a local partition. For the organization that moved, you have a foreign partition, pointing to their own database.

A simplified end state picture, using LokaDB names:

-- what the application still sees: one table
documents                    -- PARTITION BY LIST (organization_id)
  documents__cloud           -- local: every organization that stayed
  documents__org1            -- foreign: rows live in organization 1's Postgres
Enter fullscreen mode Exit fullscreen mode

Your app still runs the same queries. It doesn't know some row might be in another network.

Why that's not enough: the proxy

If you only do partitioning, then a query without a tenant in it will scan all partitions. This means it scans every customer's database over the network. If a customer is down, then queries fail for everyone. So the interesting part is the proxy in front of Postgres.

It parses every statement with the real Postgres parser (pg_query). It looks for the organization in it: the tenant column, or a row id, or a foreign key, or a relation you declared. It then tells Postgres which organization the statement is for.

If a statement doesn't have an organization in it, it's rejected by default, rather than sent to every customer's database. You can still run admin reports on all customers, but you need someone to review it and allow it.

My rule for it is: "narrow, never widen". The proxy only gets the organization from the statement, so if it's wrong, you'll get an empty or partial answer, rather than someone else's rows.

The cloud never connects into the customer's network

Security teams won't open an inbound port for you. And they're right. So you have an agent in the customer's network that opens an outbound mutual-TLS tunnel to a relay on your side. All traffic to their database goes through this tunnel. Enrollment tokens are one-time. The agent's private key stays on the agent's machine. If you stop the agent, you stop your access. That last part is usually important in the review.

app -> proxy -> cloud Postgres -> relay <== outbound mTLS == agent -> customer Postgres
Enter fullscreen mode Exit fullscreen mode

Moving a customer, while the app writes

The move is online and needs no logical replication, so it works on managed Postgres too. But there's a pause. Reads and writes for all organizations on the moved tables stop for a moment at the switch-over. Seconds, for an organization tens of ms away. It's not zero, so you should plan for that.

You can undo the move. move-back moves the organization back to the cloud. disable undoes the partitioning.

What it costs

These numbers are from 7 separate servers with Postgres 18, fsync on. The workload is primary key and (organization_id, id) index accesses. Your mileage may vary.

If the organization is in the cloud, you're paying 0.2–1 ms per statement. With 32 clients, you're getting 36–80% of a direct connection's throughput (reads went from 66,500 ops/s to 29,800 ops/s). A whole Django app, 16 clients, was 5–10% slower. That's a real cost. Whether that's acceptable depends on your app.

If the organization is moved, you pay for distance. Latency goes up almost linearly with round trip time. At 43 ms:

Operation through postgres_fdw direct path
read 352 ms 133 ms
update 351 ms 89 ms

The direct path exists because postgres_fdw adds round trips. If a statement only touches tables for that organization, you can go direct to their database, through the same tunnel. But an N+1 query that was invisible at 1 ms is the first thing the customer notices at 43 ms.

There's also a limit to scale. You have one partition per organization. Up to ~100 organizations, no noticeable price. At 1,000, reads and updates go down by a third to a half. At 3,000, inserts go down to ~1,400 ops/s. I don't recommend mixed writes at that scale yet.

What it doesn't do

This is the list I'd read first if I was evaluating this.

  • It's not an authorization layer. If your app asks for another organization's document by id, the proxy will route it to where that document is. Who can read what is still your app's job.
  • It only isolates Postgres rows. Nothing else. Caches, queues, search indexes, files, logs, error reports: none of that is covered. Backups made before the move aren't covered either.
  • Statistics was the exception to "rows only in their database", until v0.1.19. A move ran ANALYZE on the foreign partition. So the cloud's pg_statistic had sampled values. And pg_stats could show them through the proxy. I found this when writing cross-tenant tests. Since v0.1.19 the cloud's statistics don't have any values for a moved organization. And the proxy refuses those views. lokadb stats-sync cleans a database moved by an older version. You can still see running statements past the proxy, including literals.
  • On the fdw path, the moved rows go through the cloud Postgres in transit.
  • No atomic commit across the cloud and a customer's database. It's two commits, like any postgres_fdw write.
  • You can't change the organization of a row. I hit this with my own app. It created a draft without a team. Then set the team in a second UPDATE. The proxy refused with moving a row to another organization is not supported. Fix this in the app: create the row with its team from the start.

Also, it's early. It's in production with KeepPlain, my own Laravel SaaS. That's the only production user so far. It's tested end to end on every build, with unmodified Laravel, Django and Go apps. Every defect found gets a regression check. Those tests don't cover other people's schemas, so I'd like people to try it on theirs.

Next: customer chooses what SaaS can see

After the move, the rows live in the customer's database. But your SaaS can still read them in full. The Privacy Engine is for that part. It's not released yet. I'm testing it now. I'll ship it soon as a module. It hasn't run on RDS or with a real customer.

I think it's most relevant for AI services, like an AI recruiting tool. They have to process personal data like resumes. They can't always move their server side to the customer. With this, the model can work on a column like skills. But names, emails, phones only reach it as tokens. There's a catch: a rule applies to the whole column. It doesn't look inside text. If you have a resume in one text column, either the model can see it, or it can't. So this only helps if you have separate columns for the identifying details. If you have a PDF in object storage, this isn't covered at all.

The customer writes a policy file. It sets a rule for each column:

Real value Rule What SaaS sees
anna@acme.com token tok_3fa9…. Equal values get equal tokens
+79161234567 mask **********67
187000 generalize 180000
a note about the client block nothing

The real table moves to a closed schema, lokadb_base. A view with the same name and columns is created in its place. So the app keeps querying documents. Next to each protected column is a shadow copy. In the form that may be shown. A trigger updates the copy on every write. The view only reads the copy. So it can't accidentally let the real value out.

The SaaS also stops holding passwords to the customer's database. It logs in through the agent on three channels. Each has its own secret. app for application queries. app-fdw for queries through the cloud database. ddl for migrations. Only the agent has the real passwords. On the first two channels, it logs in under roles that only see the views. TLS is all the way to the agent. So the relay in the middle only sees ciphertext.

The agent checks what's coming through:

  • Writes. The app can write a new value. It goes to the real table. It's read back as a token. It can't write a value computed from what it saw. salary = salary + 1000 is refused. Otherwise, you'd get a rounded number + 1000 in the database. If an ORM saves the whole row, and sends the same mask back, the real value stays.
  • Migrations. They apply to the real table. The views are rebuilt in the same transaction. New columns are hidden until the customer decides otherwise. Dangerous migrations are refused as a whole. For example, an index on an expression over a protected column.
  • Audit log. Logins, refusals, migrations, statement shapes. No values. The key is with the customer, outside the database. So editing the table won't hide anything. One gap: deleting the very last rows only shows up after the next write by the agent.

If the agent goes down, that organization is unavailable. Others still work. No data is opened up. If the agent can't read the policy for more than a minute, reads continue. Writes are refused until it recovers. To leave, the customer and the SaaS each run a command. The tables go back as they were. But the organization is unavailable between the two.

What it won't do:

  • Whatever a user types into a form or sees on screen still goes through the SaaS's servers.
  • Equal values give equal tokens. So short values like phone numbers can be guessed. You can set those columns to "writes": "deny".
  • There's no limit on how much can be read yet. You can pull out everything the policy allows.
  • Features that need the real value won't work. Search by real email finds nothing. The customer decides what matters more, per column.

The screen is the part I plan to cover with LokaDB Cloud. Something like Google Workspace client-side encryption. It's only a lab prototype so far. It would work like this:

  1. The SaaS serves its page as usual. With tokens. Because it has no other values.
  2. The page embeds our script and a hidden iframe from the customer's agent.
  3. An employee signs in through their company's identity provider. The sign-in token stays in the iframe. The SaaS never sees it.
  4. The SaaS issues a short signed grant. This user, these rows, this agent.
  5. The browser asks the agent directly. The agent checks the grant. The sign-in. The perimeter. A per-employee limit. It writes to the audit log without values. It returns the values.
  6. The script swaps the tokens for values, right in the page. So the values don't go through the SaaS's servers.

So a recruiter would see the real name on the screen. The vendor's servers and model only hold the token. Google keeps the file encrypted on its side. The browser decrypts it with the customer's key. Here, the SaaS has no values at all. Not even encrypted ones. They stay in the customer's database.

It has limits too. Once a value is on the page, the SaaS's own scripts can read it. So the customer has to trust that JavaScript, like Google's. For stricter fields, the iframe draws the value itself. The page can't read it. The employee's browser also has to reach the agent's address. And this only covers showing. What people type into a form still goes through the SaaS.

Try it

lokadb check reads your schema. And the queries your app actually sends. It tells you what blocks a move. What's not supported yet. What will behave differently. It doesn't change anything.

Demo in the browser. Nothing to install. https://play.lokadb.com

Or locally:

git clone https://github.com/lokadb/lokadb && cd lokadb
docker compose -f quickstart/compose.yml up --build
Enter fullscreen mode Exit fullscreen mode

The core is Apache-2.0. https://github.com/lokadb/lokadb. The paid bit is LokaDB Cloud. A panel that drives the open runner inside your network.

If you've had the "keep our data in our database" experience with an enterprise deal, I'd love to hear what you did. What the security team actually checked.

Top comments (0)