DEV Community

informat
informat

Posted on

What Happens When the Import Fails Halfway?

On a Tuesday morning in March, the warehouse supervisor at a manufacturing customer called me. Not a ticket. Not an email. A phone call, which in our world means something is on fire.

She had spent two weeks preparing the migration of eleven years of inventory records. Twelve thousand rows, cleaned by hand, in a spreadsheet her team had guarded like a religious document. The import had run. The progress bar had completed. The platform said "Import successful."

Nine hundred rows were wrong. Dates had flipped between formats. A subset of rows had been skipped without any message. And the same three suppliers now existed four times each, because she had re-run the import after the first attempt seemed to stall.

Her question was not technical. It was the question that has stayed with me since: "Why did it tell me it worked?"

That day I learned that import is not a feature. It is a contract. And for most of the platforms I have worked on, including mine, we had written that contract in very small print.

Import looks like a checkbox

In every product plan I have ever seen, import is one line. "Excel import/export: yes." It sits in a feature matrix next to dark mode and CSV download. It demos beautifully: pick a file, watch a progress bar, done.

But that one line hides an entire pipeline. Parse. Validate. Transform. Match against existing records. Deduplicate. Commit. Report. Every one of those stages can fail, and each one fails differently. The checkbox in the feature matrix does not tell you what happens between the progress bar and the database.

Most platforms, mine included for too long, treat the middle of that pipeline as an implementation detail. That is exactly backwards. The middle is the product.

The first decision nobody makes explicitly

The first real design decision in any import system is failure semantics. Do you commit everything, or nothing, or the rows that happened to pass?

All-or-nothing is honest but brutal. Nine hundred bad rows out of twelve thousand means the customer fixes the spreadsheet and runs the whole job again, possibly several times, while the business waits. Best-effort is kinder but dangerous: the import succeeds, the failures are buried in a downloadable log the user will never open, and the data model quietly rots.

We chose best-effort once because it demoed better. Then we spent a week helping the same customer discover which rows had silently failed, by comparing exports against their original spreadsheet. Never again.

The answer, we eventually learned, is neither. It is staged: validate everything first, show the customer exactly what will happen, and only then commit. Which brings up the feature nobody asks for and everybody needs.

Preview is the whole product

A dry-run mode — "show me what this import would do without actually doing it" — is the single highest-leverage feature in this space. It costs a fraction of the effort of the rest of the pipeline, and it changes the emotional experience completely. An import with preview is a negotiation. An import without preview is a gamble.

When we added preview, something unexpected happened. The support tickets did not just drop. The conversations changed. Customers started bringing us their edge cases before running the import, instead of after. "What happens to this row? Is this supplier the same as that one?" Preview turned the import from an oracle into a document.

Matching is where most of the pain lives anyway. Does the row with supplier name "ABC Co." match the existing record "ABC Company Ltd."? Exact string comparison says no. A human says obviously yes. Whatever rule you pick, showing it in a preview lets the customer disagree with the machine before the machine makes the mistake permanent.

Big files are async jobs, whether you planned it or not

The second thing product plans get wrong is scale. In the demo, the import file has forty rows. In production, someone eventually drops in two hundred thousand rows exported from a system older than your platform.

At that size, an import is not a form submission. It is a background job with its own lifecycle: queued, running, partially done, failed, retried. It needs its own status page. It needs to survive the user closing the browser. It needs to not time out the web server, and not block every other user of the tenant while it chews through the file.

We learned this the expensive way, when a customer's import of historical sales data locked their workspace for the better part of an afternoon. The lesson was not "make imports faster." The lesson was: the moment a batch operation can outlive an HTTP request, it must be designed as a job, with progress, with logs, and with a way to cancel it that actually cancels it.

And it must be idempotent. She re-ran the import because it seemed stalled. She should have been able to re-run it safely. Any batch operation in a business system will be re-run — because someone is unsure, because someone got impatient, because someone closed the laptop on a train. Design for the second run, not just the first.

Export is the same problem wearing a nicer shirt

We tend to think of export as the polite sibling. It takes nothing from you. What could go wrong?

An export is a query with a download button, and every failure mode of a query applies. Large exports time out. Exports that stream hold database connections. Exports of filtered views race against concurrent edits. And the file the customer downloads becomes a snapshot that lives in spreadsheets, in email attachments, in places your permission system has never heard of.

Also, encoding. I will simply say: if you have never watched a customer open an export full of unreadable characters because their office computer opened your CSV with the wrong code page, you have not yet supported enterprise software in the wild.

The uncomfortable truth about export is that it is the last thing the customer keeps when they are leaving your platform. The quality of your export decides how gracefully they can go — and, oddly, caring about that is one of the strongest trust signals you can send.

The trust gradient

Here is the pattern I now believe: trust in a low-code platform does not arrive through the modeling tools. It arrives through the data plumbing.

A customer can forgive a form designer that feels clunky. They do not forgive a database that silently dropped four hundred rows. Every modeling feature is about the future, and the future is negotiable. Data integrity is about the past, and the past is not.

The import button is the moment the customer's actual history enters your system. Everything before it is a demo. Everything after it depends on how honestly you handled their eleven years of inventory records.

The uncomfortable conclusion

We spent years making low-code modeling visual, drag-and-drop, approachable. We put that in the pitch deck. Nobody ever put the import pipeline in a pitch deck.

But adoption is decided in the unglamorous places. The customers who stay are the ones whose data made it in intact, whose failed rows came back in a report they could read, whose re-runs did not duplicate anything, whose exports opened correctly on the office computer from 2013.

The uncomfortable part is what that implies about where engineering effort should go. The wizard with the progress bar demos in ninety seconds. The staging, the preview, the job system, the idempotency, the error reports — that is months of invisible work. There is no badge for it, no screenshot for the landing page.

We ship the wizard first anyway, every time, because it demos well. And then a supervisor with eleven years of records calls on a Tuesday, and we learn the same lesson again.

The import is never just a feature. It is the moment your platform becomes responsible for someone's history. Design like it.

Top comments (0)