DEV Community: Hammad Khan

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

Hammad Khan — Wed, 13 May 2026 13:16:43 +0000

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

Bulk imports are a special category of pain. You give the user a CSV uploader, they hand you a file, and your job is to take that file from "blob of text" to "thousands of clean rows in your database." Somewhere in the middle, every assumption you have about your data falls apart.

I rebuilt the bulk import system for a dealership SaaS recently. The redesign covered phone number normalization, VIN format validation, file-format consistency, and the structural decision that makes the whole thing reliable: idempotency.

If you've ever shipped a bulk importer, you've probably hit the same problems. Here's the shape of the design that finally worked.

The four problems

Any bulk import system has to solve four problems:

Validation. Every row has to satisfy schema constraints before it goes into the database. Some constraints are obvious (required fields), some are domain-specific (a VIN must be 17 characters in a specific alphabet).
Normalization. "077 12345" and "+447712345" and "01234 567 890" might all need to become +441234567890 before they hit the database. The user doesn't think about this; you have to.
Error reporting. When 200 of 5,000 rows fail validation, the user needs to know which 200 and why. A flat "import failed" tells them nothing.
Idempotency. When the upload crashes halfway through and the user clicks "Retry," the second run must not double-write the first 2,500 rows.

The fourth problem is the one teams skip. It's also the one that breaks production. Let me start there.

Idempotency: the key decision

The pattern that works is stable per-row keys with an upsert at the DB level.

For each row, compute a stable key from the row's natural identity. For dealership customers, that might be (dealer_id, email_lowercase). For vehicles, (dealer_id, vin). Whatever uniquely identifies the row in the customer's mental model.

function customerKey(row: CustomerRow, dealerId: string): string {
  return `${dealerId}:${row.email.trim().toLowerCase()}`
}

Then use that key as the basis for an upsert:

INSERT INTO customers (dealer_id, email, name, phone, ...)
VALUES ($1, $2, $3, $4, ...)
ON CONFLICT (dealer_id, email) DO UPDATE SET
  name = EXCLUDED.name,
  phone = EXCLUDED.phone,
  ...
  updated_at = NOW();

Now if the import crashes after row 2,500 and the user retries, the first 2,500 rows update themselves in-place (no-op if the data didn't change) and the remaining 2,500 get inserted. Same end state as if the import had run cleanly the first time.

This sounds obvious. Most bulk import code I see doesn't do it. Most code uses INSERT INTO ... VALUES (...) and crashes on the unique-constraint violation when the user retries.

Normalize once, at the entry point

The normalization rule: do it once, as close to the file parsing as possible. Don't sprinkle normalization throughout the codebase.

// import-normalize.ts
import { parsePhoneNumberFromString } from 'libphonenumber-js'

export function normalizePhone(raw: string, defaultCountry: string): string | null {
  if (!raw || !raw.trim()) return null
  // Auto-prepend country code if user typed a local number without +
  const candidate = raw.trim().startsWith('+') ? raw.trim() : raw.trim()
  const parsed = parsePhoneNumberFromString(candidate, defaultCountry as any)
  if (!parsed?.isValid()) return null
  return parsed.format('E.164')  // e.g. "+441234567890"
}

export function normalizeVin(raw: string): string | null {
  if (!raw) return null
  const cleaned = raw.trim().toUpperCase().replace(/[^A-HJ-NPR-Z0-9]/g, '')
  // VIN must be 17 chars; no I, O, or Q allowed
  if (cleaned.length !== 17) return null
  return cleaned
}

export function normalizeEmail(raw: string): string | null {
  if (!raw) return null
  const trimmed = raw.trim().toLowerCase()
  return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(trimmed) ? trimmed : null
}

Three things in here that look small:

Each function returns null on invalid input, not a thrown exception. The caller decides what to do with invalid rows (skip them, surface an error). Throwing inside normalization makes the loop harder to reason about.
Phone normalization uses libphonenumber-js, not a regex. Phone numbers are too varied for regex to handle correctly. The library handles country codes, formatting, valid-number checks, and the auto-prepend logic.
VIN normalization knows about the VIN alphabet. The letters I, O, and Q are not valid VIN characters (they look too much like 1 and 0). Strip them out during cleanup, and reject if the result isn't 17 characters.

The validation pipeline

The pipeline as a whole looks like this:

type ImportResult = {
  validRows: Customer[]
  errors: Array<{ row: number; field: string; reason: string }>
}

async function processImport(file: Buffer, dealerId: string, dealerCountry: string): Promise<ImportResult> {
  const parsed = parseCsv(file)
  const errors: ImportResult['errors'] = []
  const validRows: Customer[] = []

  for (let i = 0; i < parsed.length; i++) {
    const row = parsed[i]
    const rowNum = i + 2  // 1-indexed, plus header

    const email = normalizeEmail(row.email)
    if (!email) {
      errors.push({ row: rowNum, field: 'email', reason: 'invalid email' })
      continue
    }

    const phone = normalizePhone(row.phone, dealerCountry)
    if (row.phone && !phone) {
      errors.push({ row: rowNum, field: 'phone', reason: 'invalid phone' })
      continue
    }

    const name = (row.name ?? '').trim()
    if (!name) {
      errors.push({ row: rowNum, field: 'name', reason: 'required' })
      continue
    }

    validRows.push({ dealerId, email, phone, name })
  }

  return { validRows, errors }
}

A few choices to point out:

continue on validation error, don't return. Process every row, accumulate every error. The user wants to see all 200 errors at once, not the first one.
Row numbers are 1-indexed plus the header. The user's spreadsheet starts at row 1 (header) and the first data row is row 2. Match the user's mental model.
Errors capture row, field, and reason. This is what gets surfaced back to the UI. "Row 47: phone — invalid phone" is what the user needs to fix the spreadsheet.

The atomic write

Now you've got validRows. The write step is a single transactional upsert:

async function writeRows(rows: Customer[]) {
  if (rows.length === 0) return { inserted: 0, updated: 0 }

  return await db.transaction(async (tx) => {
    const result = await tx
      .insertInto('customers')
      .values(rows)
      .onConflict((oc) => oc.columns(['dealer_id', 'email']).doUpdateSet({
        name: (eb) => eb.ref('excluded.name'),
        phone: (eb) => eb.ref('excluded.phone'),
        updated_at: () => sql`now()`,
      }))
      .returning(['id', 'xmax'])  // xmax = 0 means insert, otherwise update
      .execute()

    const inserted = result.filter(r => r.xmax === '0').length
    const updated = result.length - inserted
    return { inserted, updated }
  })
}

Three things:

Single transaction. Either all 4,800 valid rows make it in, or none do. No partial state for the user to discover later.
xmax trick to distinguish inserts from updates. Postgres-specific: when INSERT ... ON CONFLICT DO UPDATE performs an insert, the returning row's xmax is 0. On an update, xmax is non-zero. This lets you tell the user "inserted 3,200, updated 1,600."
updated_at on conflict. Always touch the timestamp, even if no other fields changed. The audit trail wants to know the row was re-imported.

For very large imports (50,000+ rows), this single transaction will become a problem (long-held locks, replication lag). The fix is to batch:

const BATCH_SIZE = 1000
for (let i = 0; i < rows.length; i += BATCH_SIZE) {
  const batch = rows.slice(i, i + BATCH_SIZE)
  await writeRows(batch)
}

Each batch is its own transaction. You lose all-or-nothing across batches, but you gain the ability to scale to large imports. For our dealership use case (rare imports of 5,000-50,000 rows), this trade was right.

What about the "supply-chain" failures?

The original code we replaced loaded its xlsx parser from a CDN. We had a separate write-up on why that's a bad idea (and a separate fix). For the bulk-import context specifically: the parser runs against user-uploaded files. If the parser is malicious, every uploaded file is exfiltrated. Vendor the parser, verify its hash in CI, and don't load it from a remote URL. (Details in the companion article on the xlsx CDN supply-chain hole.)

The user-facing report

The validation step returns errors. The UI needs to show them in a way the user can act on. Two patterns work:

Inline error report. After upload, show "5,200 rows imported, 200 errors. Download error report?" The download is a CSV with the user's original rows plus an error column. They fix the spreadsheet and re-upload.
Reject-on-any-error. If any row has an error, reject the whole upload, surface the errors, and don't write anything. This is right for cases where partial imports are dangerous (vehicle records that have FK dependencies on other tables, for example).

For our customer imports, option 1 was right. For our vehicle imports, option 2 was right. Pick deliberately. Document the choice. Don't let the default be "import what we can, drop the rest silently" — that's how dealers find missing customers six months later.

The full pipeline shape

To summarize the design:

Parse the file into raw rows. No transformation here.
Normalize each field (phone, email, VIN, etc.) to the canonical form.
Validate each row. Collect errors. Don't stop on the first failure.
Decide based on the policy: reject-on-error or skip-bad-rows-and-continue.
Upsert valid rows in a transaction (or batched transactions), using a stable natural key.
Report back to the user: counts, errors, downloadable diff.

Each step is a small, testable function. The pipeline as a whole is reproducible: same file, same dealer config, same result. Crashes mid-way are recoverable because of idempotency.

The takeaway

Bulk imports become reliable once you commit to two ideas: normalize at the boundary, and make every write idempotent via a stable key. Once those two are in place, retries are free, partial failures are recoverable, and "the import broke" stops being a thing your support team has to handle.

The hardest part of building one isn't the validation. It's having the conversation about what your stable key actually is — and being willing to make the user pick one if the data doesn't have one obvious.

From Hours to Seconds: Simplifying RuboCop with rubocop-hk

Hammad Khan — Mon, 25 Aug 2025 07:11:35 +0000

The two-hour setup that drove me crazy

I started what should have been a simple task last Thursday at 3 PM: setting up RuboCop for a new Ruby project. I was still changing configuration files, searching for "best RuboCop rules for Ruby 3.2," and arguing with myself about whether to use double or single quotes by 5 PM. Does this sound familiar?

At that point, I knew it was time to stop. If I'm spending two hours setting up a linter instead of writing code, something is wrong. That night, I began working on rubocop-hk, a pre-configured RuboCop gem that lets you start linting in just 30 seconds.

What is rubocop-hk?

rubocop-hk is a RuboCop configuration gem that works well right away without any changes. It's like RuboCop's opinionated cousin who has already made all the hard choices for you.

RuboCop's default setup takes a long time to customize, but rubocop-hk gives you a setup that has been tested in real-world Ruby projects and improved over time. You can now install it just like any other gem because it's on RubyGems.

What I like best is It's still RuboCop on the inside. You get all the power of RuboCop without having to worry about setting it up. It saves time by making the most boring part of setting up a project a one-minute task.

The problem it solves (or why I made it)

Let's be honest about what adding RuboCop to a project does:

Before rubocop-hk:

Put RuboCop in your Gemfile (5 minutes)
Run it and get 500 or more offenses (10 minutes of crying)
Begin making .rubocop.yml (15 minutes)
Search for "best RuboCop configuration" on Google (30 minutes)
Take rules from different blog posts (20 minutes)
Talk with your team for 45 minutes about how long the lines should be.
Finally, choose a configuration (15 minutes)
Remember that you forgot to set up Rails cops (start over)

Total time: more than two hours of setting up hell.

After rubocop-hk:

Put rubocop-hk in your Gemfile (30 seconds)
Make a small .rubocop.yml file (30 seconds)
Start coding in a consistent way (worth its weight in gold)

The real killer isn't just the first step. It's keeping things the same across a lot of projects. How many times have you copied .rubocop.yml files from one project to another, only to find that each one is a little different and none of them are quite right?

How to Install and Get Started

Are you ready to take back your afternoon? Here's how to get going:

Step 1: Put it in your Gemfile

group :development, :test do
  gem 'rubocop-hk'
end

Step 2: Install the bundle

$ bundle install

Step 3: Make your .rubocop.yml

inherit_gem:
  rubocop-hk: default.yml

# Optional: Add settings that are specific to your project
AllCops:
  NewCops: enable
  TargetRubyVersion: 3.2

Step 4: Start RuboCop

$ bundle exec rubocop

Looking at 23 files
.......................

23 files checked, no crimes found

That's all. That's really all there is to it. In less than a minute, you've set up RuboCop with settings that are ready for production.

Important Features and Benefits

What makes rubocop-hk special isn't what it adds; it's what it takes away from your workflow:

🎯 The Philosophy of No Configuration

The gem follows the Ruby community's best practices right out of the box. No more being stuck on a decision or making changes over and over. It just works.

⚡ Productivity Right Away

Stop arguing about whether to use %w[] or %w() for arrays of words. Based on popular Ruby style guides and how people use it in the real world, rubocop-hk has already made these choices.

🔧 Still Adjustable

Don't like a rule? You can change it in your .rubocop.yml. rubocop-hk gives you the base, but you are still in charge:

inherit_gem:
  rubocop-hk: default.yml

# Your group likes longer lines
Layout/LineLength:
  Max: 120

📦 Batteries Included

The gem has settings for:

Style cops (Ruby idioms that are always the same)
Layout cops (spacing and indentation)
Lint cops (find real bugs)
Metrics cops (code that is easy to understand and change)
Naming cops (making sure that variable and method names are always the same)

To see exactly what you're getting, go to GitHub and look at the full configuration.

Examples of How to Use It in Real Life

Example 1: Putting together a Rails project

inherit_gem:
  rubocop-hk: default.yml

require:
  - rubocop-rails

Rails:
  Enabled: true

AllCops:
  Exclude:
    - 'db/schema.rb'
    - 'vendor/**/*'
    - 'config/initializers/*'

Example 2: GitHub Actions CI/CD Pipeline

name: Linters
on: [push, pull_request]

jobs:
  rubocop:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Run RuboCop
        run: bundle exec rubocop --parallel

Example 3: Slowly getting the team to use it

Start out strict and then loosen up when you need to:

inherit_gem:
  rubocop-hk: default.yml

# Turn off strict cops for a while while the team gets used to it
Metrics/MethodLength:
  Enabled: false

Style/Documentation:
  Enabled: false

Example 4: Fixing Old Code Automatically

# Fix automatically what's safe to repair
$ bundle exec rubocop -a

# Show what needs to be done by hand
$ bundle exec rubocop --only-recognized-offenses

Who Should Use This?

Developers Who Work Alone

Stop wasting time setting things up. Use rubocop-hk and concentrate on adding new features.

Teams that work on development

Stop arguing about the style guide. Make rubocop-hk your team's standard and get back to solving real problems.

Students in Bootcamp and Ruby Beginners

Learn how to write Ruby code well without having to deal with hundreds of cop configurations.

People who keep open source software up to date

Without having to keep up with complicated documentation, give contributors clear style rules. With rubocop-hk, all you have to do is point them to your .rubocop.yml.

Agencies and Consultants

Keep client projects consistent without having to move configuration files around.

Your turn: Start in 30 seconds

The time is running out. You could have already installed rubocop-hk and be writing better Ruby code by the time you finish reading this conclusion.

Get the gem: rubygems.org/gems/rubocop-hk
Star the repo: github.com/hammadxcm/rubocop-hk
Tell us about your experience: Let us know how long it took you to set up by leaving a comment below!

Do you have any questions? Did you find a bug? Do you have a suggestion for how to set it up? This gem was made for the community, by the community. If you have a problem, open an issue on GitHub.

Stop setting things up. Get started with coding. Your future self will be grateful.

What is your RuboCop setup horror story? Please leave a comment below and let me know how much time you've wasted setting up your linter! 👇

If rubocop-hk helped you save time, please like this post and tell your Ruby friends about it. Let's speed up Ruby development for everyone!

Mastering GraphQL Mutations with the graphql-client Ruby Gem

Hammad Khan — Fri, 05 Jan 2024 09:02:34 +0000

GraphQL has become a popular choice for building APIs due to its flexibility and efficiency. When working with GraphQL APIs in Ruby, the graphql-client gem proves to be a powerful tool. In this article, we will delve into the intricacies of using mutations with the graphql-client gem to modify data on the server side.

Introduction to GraphQL Mutations

In GraphQL, mutations are operations used to modify data on the server. While queries are used to fetch data, mutations enable us to create, update, or delete data. This distinction aligns with the principles of the GraphQL schema, where mutations are defined to perform write operations.

Setting Up the graphql-client Gem

Before diving into mutations, it’s crucial to have the graphql-client gem set up in your Ruby project. The gem facilitates easy communication with GraphQL APIs. Here is a basic setup snippet:

require 'graphql/client'
require 'graphql/client/http'

module Api
  HTTP = GraphQL::Client::HTTP.new(ENV['GRAPHQL_API_URL'])
  Schema = GraphQL::Client.load_schema(HTTP)
  Client = GraphQL::Client.new(schema: Schema, execute: HTTP)
end

Make sure to replace 'GRAPHQL_API_URL' it with the actual URL of your GraphQL endpoint.

Executing a Mutation

Executing a mutation with the graphql-client gem is similar to querying data. Let's take an example of a mutation that creates a country. In this case, we have a mutation called CreateCountryMutation:

CreateCountryMutation = Api::Client.parse(<<~'GRAPHQL')   
  mutation($name: String) {
    createCountry(name: $name) {
      id
    }
  }
GRAPHQL

This mutation expects a parameter name of type String and returns the ID of the created city.

To execute this mutation, you can use the following code snippet:

variables = { name: 'United States' }
result = Api::Client.query(CreateCountryMutation, variables: variables)


puts result.data.create_country.id

Replace 'United States' with the desired country name. The result object will contain the response from the server, and you can access the created country's ID using result.data.create_country.id.

Conclusion

The graphql-client gem simplifies the process of working with GraphQL APIs in Ruby, and mutations play a crucial role in modifying data. By understanding how to structure mutations and execute them using the graphql-client gem, you can seamlessly integrate write operations into your Ruby applications.

Explore more about GraphQL mutations, handle errors, and optimize your mutations for efficient data manipulation. The combination of GraphQL and the graphql-client gem opens up a world of possibilities for building robust and flexible APIs. Happy coding!