DEV Community: Adan Potyondy

Load an on-prem database into a cloud warehouse: 1 million rows, measured

Adan Potyondy — Tue, 10 Nov 2026 14:36:05 +0000

To load an on-prem database into a cloud warehouse, read it in pages inside your network, check and mask each page there, keep the failed rows locally, and send the rest in batches of a few thousand rows per statement. In our test of 1 million rows, rows per insert set the speed. Our first checks caught 55% of the bad rows; after Claude profiled the data and rewrote the checks through the Neblex MCP server, they caught 94%, including in a half of the table they were never shown. The better checks cost about 2 minutes per million rows.

The request we hear often

Moving data from a database inside a company's network to a cloud warehouse, usually as a scheduled ETL job, is a request we hear often. The questions are always the same: how long will it take, will bad rows reach the warehouse, and does personal data leave the building?

Our measurement post covered what the runtime costs your server. This one runs the load: 1 million rows from an on-prem PostgreSQL database into Supabase, checked and masked on the way.

How we tested

Source: PostgreSQL 18 on the test machine, with a test table of 1,000,000 customer records (206 MB).
Problem rows: 18,675 rows, about 1.9%, with a bad email, state or phone, or a test account. Each problem came in one of 6 to 9 forms we see in real data, such as a misspelled domain, "N/A" in the state field or an all-zeros phone, and about 1 in 9 of these rows had a second problem.
Unusual but valid rows: 5,325 rows that should pass, such as a surname starting with "Test", a phone with an extension or a state written in full.
Answer key: a separate table recorded every problem row and every unusual row, so we could score each run.
Runtime: the On-Prem Worker, our local integration runtime, running as a background service on the test machine.
Destination: Supabase, a hosted Postgres database, in us-west-2, through its connection pooler. It stood in for the cloud warehouse.
Line: a home internet connection, 30 to 44 Mbps up. Measured before every timed run: 31 to 110 ms idle latency.

Raw and failed rows stay inside the network; the warehouse gets masked rows and the Neblex cloud gets the run report. Diagram by Neblex.

We kept the error rate low on purpose. In a peer-reviewed study of 75 data quality assessments, 47% of recently created records had at least one critical error; our test had problems in 1.9% of rows, across four fields.

We sampled the runtime's CPU, memory and network traffic throughout, and after each run compared what arrived in Supabase with the answer key. Supabase is Postgres, not a warehouse like Snowflake or BigQuery, which usually load from staged files; the lesson below carries over either way: send fewer, larger loads.

What set the speed: rows per insert

The runtime spent most of each run waiting for the warehouse to accept inserts, so the number of rows per insert decided the time.

Rows per insert	Rows read	Runs	Time	Rows per second (median)
1	500 (sample)	1	195 s	2.6
100	1,000,000	5	21 min 46 s to 22 min 53 s (median 21 min 53 s)	762
1,000	1,000,000	5	9 min 56 s to 10 min 38 s (median 10 min 22 s)	1,608
5,000	1,000,000	5	8 min 18 s to 8 min 48 s (median 8 min 23 s)	1,988
10,000	1,000,000	5	7 min 56 s to 8 min 59 s (median 8 min 37 s)	1,934

Rows per insert set the speed; past 5,000 there was no further gain. Source: Neblex internal tests, October 2026.

Each setting ran five times on the same day. At 5,000 rows per insert, the slowest run took 6% longer than the fastest, and 10,000 overlapped it.

One row per insert took about 0.39 seconds per row, or four and a half days for 1 million rows. Each insert pays a full round trip to the warehouse, which dwarfs the work of inserting one row.

Engineering write-ups show the same shape at higher speeds. In one, run against a database on the same laptop, batching took inserts from about 2,284 to 37,465 rows per second. The PostgreSQL documentation gives the same advice: fewer, larger statements, and COPY where you can use it.

Past 5,000 rows per insert there was no further gain: each insert gets bigger and slower to send and parse. Your best size depends on latency, row width and warehouse, so measure a few sizes rather than picking the largest.

Where the time went

Step timings for one page of 100,000 rows, from the run logs, with our first checks and the rewritten checks described below:

Step	First checks	Rewritten checks
Checks	1.7 s	11.9 s
Normalize email, domain and phone	1.8 s	2.8 s
Route rows to load or quarantine	1.1 s	1.2 s

With the first checks, the 20 inserts took 33.5 of the page's 43 seconds. The rewritten checks added about 11 seconds per 100,000 rows. Inserts still set the pace, but thorough rules are not free: checks and normalizing went from about 8% of the page's time to nearly 30%.

Which checks should run before rows leave the network?

Run the checks that decide whether a row may leave at all; anything you can only fix at the source belongs here. We started with four, each a single rule on one field:

Check	First rule	Bad rows caught (held-out half)
Email	contains an "@"	46%
State	not empty	60%
Phone	at least 10 digits	74%
Test account	on the test domain, or starting with "test"	43%

Together they caught 54.6% of the bad rows in the half we later held out, and wrongly quarantined 5.6% of the unusual but valid rows, such as surnames starting with "Test". Each miss has a plain cause: a misspelled domain still contains an "@", "N/A" is not an empty state, and an all-zeros phone has 10 digits.

The weak rules also hid two masking bugs. Emails with two "@" signs put part of the address in the domain field of 402 warehouse rows, and for phones with an extension, the "last 4 digits" were the extension's.

Improving the rules from a profile of the data

Rules only catch what someone thought to write down, so we did what a data team would do: look at the data, then rewrite the rules. The difference is that Claude did the work through the Neblex MCP server, and nobody edited the flow by hand.

1We split the table into two halves by a hash of each row's id.
2Claude built and ran a profiling flow on the first half through MCP: counts and patterns, not rows. For example, every state value and its count, the shapes of phone numbers, the most common email domains, and values that repeated 4 or more times.
3A second Claude session, given only that profile and our four rules, drafted new rules and listed what they would still miss. It never saw the data, the answer key or how the test data was made.
4Claude wrote the new rules into the flow's check and masking steps through MCP. The checks went in exactly as drafted; the masking fixes were written to the drafted spec.
5We scored the new rules on both halves against the answer key. The second half was never profiled.

The model worked from a summary produced inside the network, never from customer rows, and it changed the same governed flow a person would edit in the designer. One caution if you do this on real data: repeated values can be real phone numbers or emails, so mask them or raise the threshold before the profile goes to a model.

Field	First rule	Rewritten rule (summary)
Email	contains an "@"	exactly one "@", valid characters and domain, not a known misspelled domain or placeholder such as noreply@
State	not empty	a US state code or name; "N/A", "unknown" and similar count as missing
Phone	at least 10 digits	10 digits after dropping a leading 1 and any extension, and not a placeholder such as all zeros
Test account	test domain, or starts with "test"	test domains, "test" or "testuser" as a name or address, keyboard mashes such as asdf@

Claude changed what two steps do; the flow's steps and connections stayed the same. Diagram by Neblex.

	First checks	Rewritten checks
Bad rows caught, profiled half	55.3%	94.4%
Bad rows caught, held-out half	54.6%	93.7% (8,701 of 9,289)
Unusual but valid rows quarantined, held-out half	5.6%	0%
Normal rows quarantined, held-out half	0	0 of 485,638

The held-out half scored within a point of the profiled half, so the rules were not fitted to particular rows. Both masking bugs were fixed.

The rewritten rules still missed some rows in the held-out half:

Domain misspellings not on their list: 62 of 613. The drafted rules themselves suggested an edit-distance check for these.
Test accounts the profile never showed: "demo" and "qa+" addresses, and placeholder names such as John Doe. These were caught in 17%, 14% and 49% of cases.
Emails ending in a dot: the rules strip the dot and load the address. Our answer key counted these as bad. If you count stripping the dot as a fix, the rewritten rules handled about 95% of the bad rows in the held-out half.

One limit: the held-out half had the same kinds of problems as the profiled half. The test shows the rules generalize to new rows, not to kinds of problems nobody has seen yet, which is exactly what the misses above are.

Where do the rows that fail go?

To a quarantine table in the same on-prem database, with their raw values and the reason they failed. No row is dropped, and the raw values stay inside the network.

The raw values are kept because a malformed email can only be fixed by someone who can see it. With the rewritten rules, every run quarantined the same 17,564 rows.

Masking before the load, and what hashing does not do

We kept three things per customer in place of the raw email and phone: a hash of the trimmed, lower-cased email, so two records with the same email still match; the email domain, for company-level reporting; and the last 4 digits of the phone, for support lookups. Names, company, city and state went through unchanged, because the reports needed them. That is a choice each team has to make field by field.

A hash is not anonymisation. The UK Information Commissioner's Office treats pseudonymised data as personal data and advises against hashes that use no salt, pepper or key. The US FTC put it more bluntly: "hashes aren't 'anonymous' and can still be used to identify users".

So use a keyed hash (HMAC) with a secret that stays on your side. Our timed runs used plain SHA-256. In an earlier test, a version built for production use, with a keyed hash and upserts instead of inserts, loaded 1 million rows in 8 minutes. The warehouse can still join on the hash; an outsider with a list of email addresses cannot reverse it.

After each run we checked the warehouse: 982,436 rows, no empty hashes, every phone field exactly four digits, and no "@" in any text column. The Neblex cloud received the run's status, diagnostics, counts and integrity receipts, not records; ask any vendor you evaluate to show this in their run logs.

After the first load: send only what changed

A one-time load is a backfill; after it, a scheduled job should move only the rows that changed. In an earlier production-style version tracking changes by updated_at and id, a run with nothing new took 0.3 seconds and a run with five changed rows took 1.8 seconds.

What the runtime used

CPU averaged 25 to 47% of one core. Upload was 372 to 395 MB per 1 million rows, around 6 Mbps of the 30 to 44 Mbps uplink, so the line was never the limit.

Memory varied more. Across 21 runs of 1 million rows, peaks ranged from 379 to 985 MB, highest at 100 and 1,000 rows per insert, while the median stayed under 210 MB in all but five runs. The runtime returned to 52 MB afterwards; size the machine for the peak. Our measurement post covers how the runtime handles memory as batches grow.

What can go wrong

A slow line stretches the load but should not change the result. Over a poor mobile connection (500 to 950 ms latency), 1 million rows took about 29 minutes instead of about 8, and every row still arrived in the warehouse or quarantine.
A write can fail with its outcome unknown: the warehouse may or may not have committed it. On the poor line, one insert dropped mid-flight and the run stopped for a person to decide instead of retrying blindly.
Rules miss what the profile never showed. Profile again when the source changes.
Incremental loads have edges. A row that later fails its checks stays in the warehouse with its last good values, and deletes in the source are not copied unless you handle them.
Quarantine needs an owner. A quarantine table nobody reviews is a slower way of dropping rows.

How to set it up yourself

1Create the target table in the warehouse with only the masked columns, and a quarantine table in the source database with the row's id, the reason and the raw record.
2Read the source in pages ordered by key (WHERE id > last_id ORDER BY id LIMIT 100000), on a runtime or job inside the network.
3Profile the source before writing rules: distinct values of short fields, shapes of phones and emails, common domains. An AI assistant connected through MCP can run the profile and update the checks.
4Check each row and route failures to the quarantine table with their reason. Score the rules on a part of the data they were not written from.
5Mask the passing rows: a keyed hash of the normalised email, the email domain, the last 4 phone digits, and whatever else your policy requires.
6Group the masked rows into batches of a few thousand and send each batch as one statement. One way in PostgreSQL is to pass the batch as a single JSON parameter and expand it with jsonb_to_recordset:

INSERT INTO customers_masked (id, email_hash, email_domain, phone_last4, city, state)
SELECT r.id, r.email_hash, r.email_domain, r.phone_last4, r.city, r.state
FROM jsonb_to_recordset(CAST(:batch AS jsonb))
  AS r(id bigint, email_hash text, email_domain text, phone_last4 text, city text, state text)
ON CONFLICT (id) DO NOTHING;

After each run, compare counts on both sides: rows read, rows quarantined, rows in the warehouse. Once the first load is done, switch to incremental runs by updated_at.

These checks catch broken values, not duplicates. The data also held 20,000 near-duplicate customers that passed every check and loaded as clean rows; an upcoming post covers finding and merging them.

The takeaway: to load an on-prem database into a cloud warehouse, tune rows per insert first, check and mask inside your network, and write your rules from a profile of your own data, then test them on rows they have not seen.

Sources

Nagle T, Redman T, Sammon D, Assessing data quality: A managerial call to action, Business Horizons 63(3), 2020, pp. 325-337
FTC Office of Technology, No, hashing still doesn't make your data anonymous, July 24, 2024
UK Information Commissioner's Office, Pseudonymisation (guidance)
PostgreSQL documentation, Populating a Database
PostgreSQL documentation, JSON Functions and Operators
Alexander Belanger, The fastest Postgres inserts, Hatchet, May 15, 2025
Connect an AI client (MCP), Neblex Integration Fabric docs

Last checked: November 2, 2026.

Frequently asked questions

How do you load an on-prem database into a cloud warehouse?

Read it in pages by key on a runtime or job inside your network, check and mask each page there, keep failed rows locally, and send the rest in batches of a few thousand rows per statement. After the first load, send only rows that changed.

What batch size should you use for bulk inserts into Postgres over the internet?

Large enough that round trips stop dominating. In our test, 5,000 rows per insert loaded 1 million rows in 8 min 18 s to 8 min 48 s across five runs; 10,000 was no faster; one row per insert ran at 2.6 rows per second.

How do you improve data quality rules?

Profile the data, write rules for the patterns the profile shows, and test them on data they were not written from. In our test, Claude profiled half of a 1 million row table and rewrote the flow's checks through the Neblex MCP server; the new rules caught 93.7% of bad rows in the other half, up from 54.6%, with no false alarms.

Should you mask PII before or after loading it into a data warehouse?

Before, when you can. Masking at the source means the raw values never travel or sit in the warehouse, its backups or its logs.

Is a hashed email address still personal data?

Usually yes. Regulators treat hashing as pseudonymisation, not anonymisation, and an unsalted hash of an email can be reversed by hashing a list of known addresses. Use a keyed hash (HMAC) with a secret kept on your side, or a token.

What should happen to rows that fail validation during a load?

Keep them, with the reason, somewhere they can be fixed. In our test they went to a quarantine table in the same on-prem database, with their raw values, so nothing personal left the network and nothing was silently dropped.

Adan Potyondy leads go-to-market at Neblex. Originally published at neblex.com.

Bad data costs real money. Here is the HubSpot to Supabase workflow that fixed it.

Adan Potyondy — Thu, 05 Nov 2026 19:36:21 +0000

In September I worked with a customer on a data quality problem and recorded a short demo of the general case. Contacts come out of HubSpot, get cleaned and deduplicated, and only the rows that pass land in Supabase.

The interactive demo is on the original post. Here is a separate 6-minute live build of the same flow; it matches duplicates with fuzzy (Jaro-Winkler) matching rather than the exact email match described below.

The problem the customer brought us

Their sales team lives in HubSpot. Their product and reporting live on a Supabase database. A nightly job copied contacts from one to the other, and it copied everything: the same person three times under three spellings of their email, trailing spaces in names, companies typed in four different ways, and records with no usable email at all.

None of that looks dramatic in a CRM. It looks dramatic in the results. Duplicate contacts get counted twice in pipeline reports, two reps reach out to the same person in the same week, and the product team builds features for a customer base that is smaller and cleaner than the one the numbers describe. Bad data does not stay in the database. It shows up in decisions.

What the demo shows

The recording is a walk through the general case, not the customer's data. It uses the Spanish version of the Neblex workspace, which is the language the customer works in; every screen in Neblex Integration Fabric is available in English, German, and Spanish.

Pull the contacts from HubSpot. The HubSpot connector reads contacts with the properties the customer cares about. The flow can run on a schedule or start when a contact is created or changed.
Clean the rows. Emails are lower-cased and trimmed, names lose their stray spaces, and company names are normalized so "Acme", "ACME Inc" and "acme, inc." stop being three companies.
Remove duplicates. The Remove duplicates step matches records on the normalized email and drops the repeats. It can also keep every record and tag each one as unique or duplicate, with a pointer to the match, so a review branch can look at the borderline cases.
Check quality before writing. The Assert data quality step checks the batch against the rules the customer chose, such as required fields and an email address on every row. If a rule fails, the run stops with the failing records and the rule named, so nothing questionable reaches the database and someone can fix the source in HubSpot instead of quietly inheriting the problem downstream.
Insert into Supabase. Only the rows that passed are written to the Supabase table, with the run recorded end to end.

That last point is the one the customer cared about most. Every run shows each step, what stopped the run and why, and how many records were written; with run data kept for replay, the records themselves are there too. If a number looks wrong in a report, the answer is in the run history, not in a guess.

Why we clean at the boundary

It is tempting to fix data quality in the database with a cleanup script that runs once a month. It never stays fixed, because the source keeps producing the same problems. Cleaning at the boundary, in the flow that moves the data, means every record is checked every time and the source system gets the feedback it needs.

It also means the cleanup is not a separate tool. Bulk data processing and data sync and change data capture run on the same platform, with the same connectors and the same audit trail as everything else. When the customer later wants to push a cleaned company list back into HubSpot, that is another flow, not another product.

What this means for your team

If your reporting and your CRM disagree, the fix is rarely a bigger dashboard. It is usually a clean, checked path from one system to the other, with the failures visible instead of silently loaded.

Frequently asked questions

Does the cleanup change the data in HubSpot?

No. The flow reads from HubSpot and writes to Supabase. When a quality rule fails, the run stops and names the records and the rule, so the team can correct them in HubSpot, and the next run picks up the corrected contacts. Writing corrections back to HubSpot would be a separate flow you choose to turn on.

How are duplicates decided?

By the field you choose, or the whole record. In the demo it is the normalized email address. Matching can be exact or fuzzy with a similarity threshold, and instead of dropping duplicates the step can tag them for a review branch.

Can this run as contacts change instead of on a schedule?

Yes. The HubSpot connector has 46 polled triggers, checked as often as every minute, so a flow can start when a contact is created or changed and check the new record on its way through.

Does it work with a plain PostgreSQL database instead of Supabase?

Yes. Supabase is PostgreSQL underneath, and the same flow writes to any PostgreSQL, MySQL, SQL Server, or Snowflake target through their connectors.

Maria Soria works in customer success at Neblex. Originally published at neblex.com.

Headless iPaaS: what an AI assistant should and shouldn't do in your integration platform

Adan Potyondy — Thu, 05 Nov 2026 18:05:46 +0000

A headless iPaaS is an integration platform you can fully use without its visual interface: through an API, a command line, or an AI assistant connected over MCP. Done right, the assistant builds, validates, tests and publishes integrations, and a person keeps the secrets, the production publish and the access grants.

The term is new, and vendors use it for very different things. Some let an assistant run a workflow somebody else built. Others let it build the whole integration and push it to production.

This post compares the designs by what an assistant can actually do at each stage of an integration's life, and what should still need a person. At the end you'll find ten questions to put to any vendor.

What is a headless iPaaS?

An integration platform whose capabilities don't depend on its screens.

The term comes from content management. A headless CMS is a back end that stores content and serves it through an API, with no display layer of its own. Applied to an iPaaS (integration platform as a service), it means anything you can do in the visual builder, you can also do through an API.

What changed in the last year is who calls that API. The Model Context Protocol (MCP), an open standard for connecting AI applications to outside systems, is now supported by Claude, ChatGPT, VS Code, Cursor and others. A platform with an MCP server can be driven from the same assistant your developers already use to write code.

How useful that is depends on what the server exposes.

Three ways a platform can work with an AI assistant

You'll run into three designs, and they differ in how much of the job the assistant can finish.

Let's use one job as the running example, with the same CRM as our HubSpot contacts post. In Claude Code you type: "When a HubSpot deal closes, create the customer in our billing database and post the deal to the #sales Slack channel."

Assistant inside the visual builder

The platform ships its own assistant on the canvas. It turns your sentence into draft steps: a HubSpot trigger, a database insert, a Slack message. Then a person checks the mapping, saves, tests and publishes in the browser.

That works well for people who live in the builder. But you can't use it from the coding assistant you already have open, and it stops at the draft.

Action-only MCP server

The platform exposes actions, or workflows that already exist, as tools. Your assistant can post to Slack or kick off a workflow someone already built, but it can't create or change one.

In our example, the assistant could send a test Slack message. Somebody still has to build the integration itself in the builder.

Full-lifecycle headless platform

The platform exposes the same capabilities its visual builder uses. Your assistant lists the connectors and the connections you already have, reads the HubSpot and database operations, writes the workflow, validates it, tests it on a sample deal, reads the result, fixes the mapping and asks to publish.

A person still steps in at a few deliberate points, covered below. Everything else happens in one conversation, and the result opens in the visual builder like anything else.

What an assistant needs at each stage of the integration lifecycle

Building is one stage of many. An integration gets discovered, built, tested, published, run, debugged and eventually retired. An assistant that only covers the first two leaves the rest to a person.

Stage	What the assistant needs	In-builder assistant	Action-only MCP	Full-lifecycle headless
Discover	List connectors, their operations and schemas, and existing connections	Within the canvas	Only the exposed actions	Yes
Build	Create and edit the workflow definition	Draft only	No	Yes
Validate	Check the definition before saving	When a person saves	No	Yes
Test	Run on sample data without publishing, and read step results	A person runs it	No	Yes
Publish	Make it live, with approval where required	A person	No	Yes, with approval for protected environments
Run and observe	Start runs, read run history and step events	Varies	Trigger only	Yes
Debug	Read the failed step's input and error, fix, re-test	Varies	No	Yes
Retire	Unpublish, delete to a trash, restore	A person	No	Yes

In our example, the problems show up during testing. A deal without a company attached, or a currency the billing table won't accept, shows up in the test run's step events. An assistant that can read those events fixes the mapping and tests again in the same conversation.

One that can't hands you an error message to chase. And after a failed production run, those same events tell you which step to re-run from. We covered why that matters in Replay from the failed step, not from record one.

Guardrails: what should still need a person

Handing an assistant write access to the system that moves your customer data is not a small decision. Keep the high-impact steps with a person.

The standards agree. The MCP specification says there "SHOULD always be a human in the loop with the ability to deny tool invocations," and that servers must implement proper access controls.

OWASP lists "excessive agency" among the top risks for LLM applications: too much functionality, too many permissions or too much autonomy. Its fix is least privilege plus human approval for high-impact actions.

For an integration platform, that comes down to five things:

Secrets stay with a person. The assistant can create a connection, but a person enters the API key or completes the OAuth sign-in in the browser. The secret never passes through the conversation.
Building needs a strong sign-in. Authoring and publishing should require a person's sign-in with a second factor, not a long-lived API key sitting in a config file. A key can be allowed to run workflows. It shouldn't be able to change them.
Access is scoped. Whoever grants access picks the projects and environments the assistant can reach, and what it may do there: operate, author or publish. Revoking it takes effect immediately.
Production publishes wait for approval. The assistant can build and test freely in a development environment. Publishing to production waits until a person approves it in the browser.
Everything is reversible and logged. An edit that collides with someone else's is rejected instead of overwriting it, deletes go to a trash you can restore from, and every run leaves events you can read, with secrets scrubbed.

The assistant does the work. A person keeps access, secrets and the production publish.

None of this stops the assistant from doing the work. In our example, a person steps in exactly three times: to grant access, to enter the HubSpot and database credentials, and to approve the production publish.

Common mistakes when evaluating AI access

Most of them come from treating "it has an MCP server" as the answer.

Only checking the build. A demo that turns a sentence into a workflow is the easy part. Ask to see the assistant test it, read a failed step and fix it.
Letting the assistant hold the keys. If connecting a system means pasting its API key into the chat or a config file, that key now lives in the assistant's context and logs.
Using one environment. If the assistant builds where production runs, every test writes real records. Keep development, test and production separate, and gate the last one.
Granting everything. A grant that covers every project and every permission is exactly the "excessive permissions" OWASP warns about. Start with one project and the author scope.
Assuming the visual builder goes away. It shouldn't. Somebody will review the assistant's work, and most people do that in the builder.

Questions to ask before you let an AI assistant build in your integration platform

They work with any vendor, and the answers tell you which design you're really getting.

Can the assistant create and edit integrations, or only run existing ones?
Does it have the same capabilities as the visual builder, or a subset? Which parts are missing?
Can it test an integration without publishing it, and read each step's input, output and error?
Can it read run history and debug a failed run?
How are credentials entered? Does a secret ever pass through the assistant?
What sign-in does building and publishing require? What can a plain API key do?
How is access scoped: by project, by environment, by action?
What needs a person's approval, and where does that approval happen?
What happens when the assistant and a colleague edit the same integration at the same time?
Can deletes be undone, and is every change and run logged?

What this means for IT and security

Treat the assistant as its own user with its own grant, not as an extension of the developer's account.

Give it one project in a development environment, the author scope, and no publish rights to production. Then watch what it asks a person for. A well-built platform sends you a browser link for every credential and every production publish, and for nothing else.

Where Neblex fits

Neblex Integration Fabric is a full-lifecycle headless platform. The visual builder and the MCP server work with the same building blocks, so an assistant connected to Neblex can do what a person does in the builder, under the same rules.

Any MCP client can connect, including Claude Code, Codex, Cursor, VS Code and Windsurf.
The assistant works across flows, forms, apps, data tables, transform modules, data models, AI agents and connections, not just workflows.
It discovers connectors and their operations, validates before saving, tests before publishing, runs flows, reads secret-scrubbed run events and checks flow health.
Secrets never go through MCP. Creating a connection returns a link where a person enters them.
Building and publishing require a user-approved sign-in with a second factor. A static API key can inspect and run the flows it's granted, and can never create or publish.
On the consent screen, the person picks the scopes (operate, author, publish), the projects or flows, and the environment. Revoking access cuts it off immediately.
Publishing into a protected environment waits for a person's approval in the browser.
An edit that collides with someone else's is rejected, so the assistant re-reads and re-applies its change instead of overwriting. Deletes go to a trash, and a restore re-checks dependencies.

Two things stay in the browser on purpose: entering secrets, and promoting work between environments. That's how we build and test our own flows: from Claude Code, in a development environment, then promoted to production in the GUI.

So before your team connects an AI assistant to its integration platform, find out two things. Can it test and debug as well as build? And which steps still go to a person? Those answers tell you whether a headless iPaaS will speed up your integration work or just move the risk somewhere else.

Sources

Headless content management system, Wikipedia
What is the Model Context Protocol (MCP)?, modelcontextprotocol.io
MCP specification: Tools (2025-06-18), modelcontextprotocol.io
LLM06:2025 Excessive Agency, OWASP Top 10 for LLM Applications, OWASP Foundation, 2025
MCP guide, Neblex Integration Fabric docs

Last checked: October 15, 2026.

Frequently asked questions

What is a headless iPaaS?

An integration platform you can fully use without its visual interface, through an API, a command line, or an AI assistant connected over MCP. The visual builder usually stays; it just stops being the only way in.

Can an AI assistant like Claude Code build integrations in an iPaaS?

Yes, if the platform exposes building and not just running. Check that the assistant can discover connectors, create and validate a workflow, test it and read the run results. Some platforms only let an assistant trigger existing workflows.

Is it safe to give an AI assistant write access to an integration platform?

It can be, if the platform limits what the assistant can reach and keeps the high-impact steps with a person. Look for access scoped per project, a strong sign-in for building, secrets entered by a person, approval before publishing to production, and a full run log.

What's the difference between an iPaaS MCP server and a headless iPaaS?

The MCP server is the connection. What matters is what it exposes. Some only let an assistant run existing actions or workflows. A headless iPaaS exposes the same capabilities as the visual interface, so an assistant can build, test, publish and debug.

Does a headless iPaaS replace the visual builder?

No. An assistant can build an integration that you then open, review and change in the visual builder. Headless adds a way in. It doesn't take one away.

Daniel Wiegand is the founder of Neblex. Originally published at neblex.com.

On-prem runtime security: what leaves your network

Adan Potyondy — Mon, 02 Nov 2026 19:57:17 +0000

Does data leave your network with an on-prem integration runtime? It depends on the design. With the Neblex On-Prem Worker, our local integration runtime (often called an on-prem integration agent), the Neblex cloud gets run status, diagnostics, record counts and integrity receipts over one outbound encrypted connection on port 443. Records go only to the systems each flow is built to reach. AI agents work the same way: an agent, its reasoning model and its embedding model can all run on premises, just as flows do.

The two questions a security review asks

Before an integration runtime goes inside the network, a security review asks what it will send out and what it needs from the outside. Here is how the On-Prem Worker answers both: what it sends, what it listens on, and what it does when the internet drops.

The control plane in the Neblex cloud coordinates; each On-Prem Worker works inside its own network and sends only run reports to the cloud. Records reach the systems a flow uses, including SaaS apps, directly from the On-Prem Worker, never through the Neblex cloud.

How much memory, CPU and disk it uses is covered in What does an on-prem integration runtime cost your server? We measured ours.

What goes to the Neblex cloud?

Run status, diagnostics, record counts and integrity receipts. Record contents stay with the On-Prem Worker, unless a flow shows something to someone through the Neblex cloud: a webhook answer, a form page or an App result.

The On-Prem Worker runs flows next to your systems and keeps records on your disk; one outbound connection carries run reports to Neblex.

The On-Prem Worker opens one outbound, encrypted WebSocket (wss) connection to the Neblex cloud on port 443, over TLS 1.3 with a post-quantum key exchange. It opens no port to the internet.
It proves its identity inside that connection with two keys created on its own machine, which never leave it: the key for the certificate Neblex issues it, and a post-quantum key.
Its only listener on a standalone install is a local console bound to 127.0.0.1, which other machines can't reach. Members of an On-Prem Worker cluster also accept a link from the other members of the same cluster.
Every run on it is reported in local custody: the cloud gets a compact run report, and the records stay with the On-Prem Worker.

What goes to your own systems?

Whatever each flow is built to send, directly. The On-Prem Worker connects to every system a flow uses on that system's own port: HTTPS on port 443 for most SaaS APIs, 5432 for a hosted PostgreSQL database, and the usual ports for local databases, queues and file servers. A flow that writes contacts and deals to a CRM calls the CRM's API; a flow that loads rows into a cloud PostgreSQL database connects to it on port 5432.

Those records leave your network because you chose that destination, and they travel from the On-Prem Worker straight to it, never through the Neblex cloud. Plan outbound firewall rules around your flows' destinations as well as the one connection to the Neblex cloud.

Where are records held during a run?

On your machine. Records do pass through the On-Prem Worker, and a large batch is buffered in its local run journal on your disk until the run is settled. They don't go to the Neblex cloud.

The On-Prem Worker purges the journal once the run is settled, so batch data doesn't accumulate on disk. Neblex doesn't store your records; anything that must persist lives in your own database.

What happens when the internet connection drops?

The run keeps going, and only the status reports wait. The flow executes on the On-Prem Worker, not in the Neblex cloud, so the connection to Neblex isn't in the data path.

The On-Prem Worker keeps reading from and writing to every system it can still reach.
It keeps its run reports in its local run journal and delivers them once it reconnects.
The run's final status reaches the Neblex cloud with the rest of those reports.

A step that needs the Neblex cloud itself, such as a form a person fills in, waits for the connection and then continues. No step fails because the Neblex cloud is out of reach.

Recovering from a failed step, rather than an outage, is covered in Replay from the failed step, not from record one.

The On-Prem Worker also keeps a local, encrypted copy of the scheduled, polling and streaming flows placed on it, so it starts them itself while the cloud is out of reach. Flows started by a webhook, a form or a person in the app come in through the Neblex cloud, so they start once it's reachable again.

Other local runtimes also keep running when their platform is out of reach. Ask any vendor to show you what theirs does.

Can AI agents and their models run on premises?

Yes, the same way flows do. An AI agent runs on the On-Prem Worker as a step in a flow, and both models it depends on can run inside your network too: the large language model (LLM) that does the reasoning, such as an open-weight model behind an OpenAI-compatible or Ollama endpoint, and the embedding model that turns your documents and records into vectors for the agent's knowledge base. Those vectors are stored in your own PostgreSQL database with pgvector, and the agent's long-term memory is stored only in your own database.

Set up that way, an AI step sends no more out of your network than any other step: prompts, model responses, records, documents and embeddings stay on premises, and the Neblex cloud gets the same run status it gets for any other flow.

If an agent uses a hosted provider for either model instead, whatever is sent to that model goes to that provider. Pick one whose data residency fits your rules, or keep the model local.

What to ask any vendor

Does the integration run on your machine, or are records relayed to the vendor's cloud to be processed?
Which ports does the runtime open, inbound and outbound?
Which destinations and ports will your flows need opened outbound, and does any of that traffic pass through the vendor?
What exactly is in its run reports?
What happens to a running job when the internet drops, and to a scheduled one?
Where is an in-flight batch held, and when is it deleted?
Can AI agents, and both the reasoning and embedding models they use, run inside your network the same way flows do?

Ask to see the answers in the runtime's logs and network connections, not only in the documentation.

Sources

On-Prem Workers, Neblex Integration Fabric docs
Offline controls, Neblex Integration Fabric docs
AI agents guide, Neblex Integration Fabric docs
On-Prem Worker architecture and run reporting, as designed and documented by Neblex engineering.

Last checked: October 30, 2026.

Frequently asked questions

Does an on-prem integration runtime need inbound firewall ports?

Not all of them do. The On-Prem Worker opens one outbound, encrypted WebSocket connection to the Neblex cloud on port 443 and opens no port to the internet. On a standalone install, its only listener is a local console bound to 127.0.0.1, which other machines can't reach.

Does my data pass through the vendor's cloud?

It depends on the design. Some runtimes relay records to the vendor's cloud for processing; others run the integration locally. The On-Prem Worker runs the flow on your machine and sends Neblex run status, diagnostics, record counts and integrity receipts, not record contents, unless a flow shows something to someone through the Neblex cloud, such as a form page or an App result.

What happens to a running job if the runtime loses its internet connection?

With the On-Prem Worker, it keeps running. The flow executes on the On-Prem Worker, which keeps its run reports in its local run journal and delivers them after reconnecting. Ask any vendor what their runtime does in this case.

Can AI agents run on an on-prem runtime?

With Neblex, yes, just as flows do. An AI agent runs on the On-Prem Worker, and both its reasoning model (LLM) and its embedding model can run inside your network, with the agent's knowledge base in your own PostgreSQL with pgvector and its memory in your own database. Set up that way, all processing stays on premises and no AI provider outside your network sees your data.

Adan Potyondy leads go-to-market at Neblex. Originally published at neblex.com.

What does an on-premise integration runtime cost your server? We measured ours

Adan Potyondy — Thu, 22 Oct 2026 17:50:12 +0000

We measured on-premise resource usage for the Neblex Worker, our local integration runtime, across 15 everyday flows and three large workloads. Idle, it used about 25 MB of memory. Fourteen of the 15 flows averaged 29 to 94 MB, and a 1 million row load into a cloud database peaked at about 450 MB. CPU stayed low except during heavy matching.

The question IT and security teams ask first

A request we hear often from prospects is to move data from a database inside their network to a cloud warehouse, on a schedule, without opening that database to the internet. That means installing software inside the network: an agent or a runtime. Before anyone discusses the warehouse, IT and security teams ask what that software will cost their servers and what it can reach.

The usual answer is a page of minimum requirements. We wanted to answer with measured numbers, so we ran the Neblex Worker through a week of tests, from small event flows to million-row loads, and recorded what it used. This post gives the numbers, the method and the limits.

This post answers the first question for a local runtime. What the runtime sends out of your network, and what happens when the internet drops, are covered in a separate post.

How we measured on-prem integration agent resource usage

All tests used synthetic or test data, generated by us, on our own hardware, between October 2 and 8, 2026.

Deployments: we ran the runtime two ways, as a background service and in a container capped at 4 CPU cores and 4 GB of RAM. The test machine also hosted the PostgreSQL, Valkey (an open-source, Redis-compatible cache) and Kafka instances it read from and wrote to, so the runtime competed for memory more than it would on a dedicated server.
The workloads: 15 flows deployed to the runtime (events, files, CRM syncs and database loads); a 1 million row load from a local PostgreSQL database into a cloud database, with data-quality checks and masking; fuzzy duplicate matching over about 970,000 records; and, to isolate the effect of batch size, a plain copy from PostgreSQL to Valkey at 1,000 to 5 million rows.
The tools: standard process tools sampled the runtime's memory and CPU every half second, and the container's own memory statistics separated process memory from file cache. Start and finish times came from the runtime's log, and the 15 flows' averages came from the Neblex console.

Your records, connectors and hardware will shape your own numbers; the patterns are what carry over.

What does the runtime use when it's idle?

Very little. The runtime is a single executable, about 74 MB on macOS and 92 MB on Linux x64.

It needs no separate language runtime, such as Java, .NET or Node.js, and it starts no child processes. Builds are published for Linux x64, macOS on Apple Silicon and Windows x64.

Measure (idle)	Value
Install	One executable, about 74 MB (macOS) or 92 MB (Linux x64)
Runtime's own memory	22 to 27 MB
Resident memory (RSS)	27 to 35 MB
CPU	Under 1% of one core
Listening ports	One, on loopback only

For a security reviewer, the small number matters less than what it implies: there is no separate runtime to patch and no helper processes to account for.

What does the runtime use, by kind of flow?

It depends on what the flow does. Memory follows how much the flow holds at once, CPU follows how much it computes per record, and run time follows the destination.

Kind of flow	What we ran	Runtime memory	Runtime CPU (one core = 100%)
Idle	Connected, no runs	22 to 27 MB	Under 1%
Everyday flows	15 flows: MQTT, Kafka and AMQP events, CRM syncs, file transfers, database loads	29 to 94 MB average for 14 flows; 122 MB for one bulk load	0 to 31% average
Load to a cloud database	1 million rows, local PostgreSQL to a cloud PostgreSQL database, with checks and masking	Peak 320 to 445 MB; median 53 to 118 MB	12 to 26% average
Local bulk copy	1,000 to 1 million rows, PostgreSQL to Valkey	41 MB to about 250 MB	About 22% average
Heavy computation	Fuzzy duplicate matching over about 970,000 records	Peak 534 to 947 MB (resident)	93 to 102% average
One very large batch	5 million rows	About 420 MB, plus 1.7 GB of journal on disk	22 to 24% average; brief peaks near one core

The sections below take each row in turn.

What do everyday flows use?

Most use about as much memory as the idle runtime. We deployed 15 different integrations to one Neblex Worker, covering events, file transfers, CRM syncs, cache updates, database loads and an AI step, which gives a broad sample of how the runtime behaves. The Neblex console reports CPU and memory for each flow, averaged across its runs, on synthetic or test data. Run times come from the runtime's log, from start to finish of each successful run.

Flow (what it does)	Avg CPU (one core)	Avg memory	Avg run time
Local Postgres customers to Supabase (bulk load)	19.1%	122 MB	Varies by batch size*
AMQP order events to CRM deals, with dead-letter queue	4.3%	94 MB	0.4 s
Postgres customers: AI industry tagging with an LLM	1.7%	41 MB	19.6 s
Local Postgres customers to Valkey cache	30.9%	40 MB	8.8 s
E-commerce new order to CSV file on FTP (ERP handoff)	2.1%	39 MB	0.7 s
Valkey usage counters to cloud Postgres daily usage, with quarantine	26.8%	38 MB	3.0 s
MQTT machine alarm to work-management task, with quarantine	5.5%	37 MB	1.2 s
MQTT plant telemetry to Postgres history, with quarantine	6.1%	37 MB	0.9 s
Kafka signup events to CRM contacts, with dead-letter topic	4.5%	36 MB	1.0 s
MQTT machine alarm to service-desk maintenance request, with quarantine	4.7%	35 MB	1.4 s
CRM company changes to Valkey account cache	4.3%	32 MB	0.5 s
Postgres customer data-quality check to work-management task	6.1%	31 MB	1.4 s
Postgres new customers to CRM contacts sync	1%	31 MB	0.4 s
FTP contacts file to CRM contacts import	0%	30 MB	0.1 s
Nightly FTP contacts file to Supabase, with data-quality checks	6.1%	29 MB	4.1 s

*A 1 million row load took 402 to 1,240 seconds, from 5,000 down to 100 rows per insert statement.

Fourteen of the 15 flows averaged 29 to 94 MB, close to the runtime's idle memory. Only the bulk load from local Postgres went above 100 MB. The event-driven flows, reading from MQTT, Kafka and AMQP, averaged 4 to 6% of one core. Twelve of the 15 finished in under 5 seconds on average; the AI tagging flow's 20 seconds is mostly spent waiting for the language model, with CPU under 2%.

What does a 1 million row load to a cloud database use?

Less than 450 MB at its peak. Moving data from a database inside the network to a cloud database is the request we hear most, so we ran it at full size: 1 million synthetic customer rows from a local PostgreSQL database into a Supabase table, which is hosted PostgreSQL, over a home internet line (about 175 Mbps down and 44 Mbps up).

Each row went through four data-quality checks, and its email was hashed and its phone number cut to the last four digits before it left the machine. Rows that failed stayed in a quarantine table on the local network. We repeated the load four times, with 100, 1,000, 5,000 and 10,000 rows per insert statement.

The runtime's own memory peaked at 320 to 445 MB across the four runs and spent most of each run near its median of 53 to 118 MB.
CPU averaged 12 to 26% of one core.
Every run loaded exactly 976,000 rows and kept 24,000 in local quarantine. The fastest took 402 seconds.
The internet line was never the limit: uploads averaged about 5 to 7 Mbps of the 44 available.

A later post covers the checks, the masking and how batch size changed the timing.

What does heavy computation cost?

About one CPU core, and more memory. Comparing records with each other is a different kind of work from passing them through, so we ran fuzzy duplicate matching on the runtime: about 970,000 customer records, grouped by zip code, with names and streets scored for similarity inside each group. The candidate pairs stayed in a local table.

First run, 970,000 records: 1,510 seconds. CPU averaged about 102% of one core, peaking at 217%, and resident memory peaked at 534 MB.
Repeat with extra preparation, 976,000 records, on the same machine with the runtime running as a background service: 1,683 seconds. CPU averaged 93%, and resident memory peaked at 947 MB.

If you plan matching, scoring or heavy transformation on the runtime, size for this case rather than for the everyday flows.

Why the destination sets the pace

In every bulk run, the runtime spent most of its time waiting for the destination, not computing.

We built the Valkey flow to write one record per command and wait for each reply, so the local copy ran at roughly 7,500 to 9,400 rows per second. Raising the flow's concurrency setting to 4 and then 64 didn't change that rate. That fits what Valkey's documentation on pipelining describes: when a client sends one command at a time, round-trip time is the limit.
Inserting into the cloud database one row per statement took about 0.35 seconds per round trip, under 3 rows per second. Grouping thousands of rows into each statement brought the full 1 million rows down to under 7 minutes.

Reading was far faster than writing, so millions of rows were waiting to be written at once. The runtime absorbed that gap: its memory stayed flat while the waiting rows sat in its local run journal on disk, covered below.

For large loads, batching writes is the lever, with any agent or runtime. The reassuring part for a server owner is the other side of the same finding: a runtime that is mostly waiting uses little CPU.

How do memory and CPU change as the batch grows?

Memory grows with the batch the runtime holds at once; CPU stays low. To isolate batch size we used the plainest flow we had, a copy from PostgreSQL to a local Valkey cache.

Rows in one run	Duration	Peak memory	Average CPU (one core = 100%)
1,000	about 20 s	about 41 MB (RSS)	not averaged
100,000	12.3 s	about 121 MB (RSS)	about 23%
1,000,000	106 s	about 250 MB*	about 22%

*From a repeat 1 million row run on varied data (116.6 s), measured as the process's own memory including compressed pages.

The 1 million row run took 106 seconds, about 9,400 rows per second. The runtime held about 250 MB during the run and dropped to about 70 MB around 20 seconds after it finished. CPU averaged about 22% of one core at every size.

Where does a large batch live: memory or disk?

On disk. During a 5 million row batch, the runtime's own memory stayed flat at about 420 MB while it wrote the in-flight batch to its local run journal, about 1.7 GB or roughly 350 bytes per row.

The run journal is a durable local record of the batch that lets a run recover from an interruption. The runtime's logs describe it as hash-chained, sealed and flushed to disk with fsync.

The runtime purges it once the run is settled: after that run, its data folder went from about 1.7 GB back to about 1 MB. Neblex doesn't store your records; anything that must persist lives in your own database. Plan disk for the journal, roughly the size of the largest batch you expect to move.

How does the runtime allocate memory and disk?

It sizes itself to the deployment it runs in, rather than to a fixed setting. Installed as a service on a server it reads the machine's resources; in a container it reads the container's limits, not the host's.

Memory sets how much work it accepts. At start, the runtime checks the memory available to it, including a container's limit, and sets how many jobs it will take on. The limit scaled with memory: on the larger test machine it allowed several times as many jobs as in the 4 GB container, and capped at 1 GB it accepted no work at first. CPU didn't change it.
Disk holds what is in flight. It reserves disk for its run journal at start, from about 5 GB to about 25 GB depending on the deployment in our tests, and writes large in-flight batches there, so its memory stays flat while a slow destination catches up.
The journal is emptied as runs settle, so the disk use is temporary.

For sizing, our tests ran comfortably within 4 CPU cores and 4 GB of RAM, and we treat that as the practical minimum for a production install. Add disk for the journal as described above. Published minimum requirements for the runtime are being finalized by our engineering team.

What to plan for when you size it

Size for your heaviest flow, not the average. Everyday flows sit close to idle; matching, scoring and heavy transformation keep about one core busy and can approach 1 GB.
Plan disk for the run journal, roughly the size of the largest batch you expect to move.
Batch your writes. Throughput follows how the flow writes: one record per command waits on every round trip, while grouping rows per statement took a 1 million row load from an estimated four days to under seven minutes.
Account for record size. Our test records were about 275 to 290 bytes; larger records raise memory, journal size and run time in proportion.
Give it room. Capped at 1 GB, the runtime accepted no work at first; 4 CPU cores and 4 GB of RAM is a comfortable starting point.

Whatever on-prem integration agent you evaluate, measure its resource usage before and during a real job, and ask the vendor exactly what the agent sends out of your network.

Frequently asked questions

How much memory does an on-premises integration agent need?

It depends on the agent and the workload.

In our tests on synthetic and test data, the Neblex Worker used about 25 MB idle, 29 to 94 MB on average for 14 of 15 everyday flows, and at most about 450 MB while loading 1 million rows into a cloud database. Heavy duplicate matching peaked near 950 MB. Our tests ran comfortably within 4 CPU cores and 4 GB of RAM.

How much disk does an on-premises integration runtime need?

Enough for its run journal. Large in-flight batches are written there and purged once the run settles; a 5 million row batch took about 1.7 GB, roughly 350 bytes per row. At start the Neblex runtime also reserves journal space, about 5 GB to 25 GB depending on the deployment in our tests. Plan disk roughly equal to your largest batch.

How much CPU does an on-premises integration agent use?

Less than most people expect, because it spends most of its time waiting for the systems it connects. In our tests, everyday flows averaged 0 to 31% of one core and a 1 million row cloud load 12 to 26%. Heavy computation is the exception: fuzzy duplicate matching kept about one core busy.

How do I measure an agent's memory correctly?

Sample the process while a representative job runs. On macOS, use the process footprint, because resident memory leaves out compressed pages. On Linux in a container, look at anonymous memory in the cgroup's memory.stat, because the container total includes reclaimable file cache.

Sources

Neblex internal tests, October 2 to 8, 2026: synthetic data generated by Neblex, on Neblex hardware (a shared test machine, and a container capped at 4 CPU cores and 4 GB of RAM), with timings from the runtime's log and per-flow averages from the Neblex console.
Valkey documentation: Pipelining
Valkey documentation: Bulk loading
Linux kernel documentation: Control Group v2, memory.stat
ps(1) and footprint(1) manual pages.

Last checked: October 7, 2026.

Adan leads go-to-market at Neblex. Originally published at neblex.com.