DEV Community: Tilo Sloboda

Your Ruby CSV Import Ran Successfully — Your Data May Still Be Wrong

Tilo Sloboda — Tue, 31 Mar 2026 20:28:53 +0000

Are you sure that Ruby CSV imported all your data — and correctly? 🤔

I wasn't looking for bugs. I was improving the performance of smarter_csv, then added a new round of tests — including some borrowed from Ruby CSV's own test suite as a sanity check. Then I started thinking through error scenarios. What I found was genuinely surprising — which also led me to realize that smarter_csv needed a reliable mechanism for bad row quarantine.

I found 10 failure modes in Ruby CSV that produce no exception, no warning, and no indication that anything went wrong. Your import runs. Your tests pass. Your data is quietly wrong.

The one that still gets me: CSV's numeric conversion silently converts the ZIP code "00123" into 83 🤯. Not a rounding error — a completely different number, because it interprets leading zeros as octal. ZIP codes, customer IDs, order numbers — all silently replaced with wrong integers that pass every validation, look plausible, and get stored in your database.

Or this: a user uploads a tab-separated file named .csv — your file-type guard passes, Ruby CSV sees no commas, treats each entire row as a single field, and returns what looks like valid data. All column structure is silently gone.

I wrote up all 10, with reproducible examples you can download and run yourself:
👉 10 Ways Ruby's CSV.read Can Silently Corrupt or Lose Your Data

SmarterCSV 1.16 addresses all of these — 1.8×–8.6× faster than CSV.read end-to-end, with a bad-row quarantine system, and instrumentation hooks:
👉 SmarterCSV 1.16 Released

Found something? Issues and feedback and bug reports are always welcome in the GitHub Discussions or Issues.

Do you have a success story you can share? We love hearing back from you.

SmarterCSV 1.16 Released — Faster Than CSV.read, Bad Row Quarantine, Instrumentation, New Features, Improved API

Tilo Sloboda — Tue, 17 Mar 2026 15:16:42 +0000

Coffee time ☕ — this one's a long read. But if you've ever had silent data import bugs ruin your day, it's worth it.

New to SmarterCSV? Start here first:

10 Ways Ruby's CSV.read Can Silently Corrupt or Lose Your Data

Switch from Ruby CSV to SmarterCSV in 5 Minutes

SmarterCSV 1.16 is out — it brings major performance gains, new bad-row quarantine system, instrumentation hooks, significantly expanded API, new features.

gem 'smarter_csv', '~> 1.16'

Contents:

Performance: 1.8×–8.6× Faster Than CSV.read
Instrumentation Hooks
Expanded Read API
Bad Row Quarantine
Quote Handling Improvements
Writer Improvements
Bug Fixes
Deprecations
Links

Performance: 1.8×–8.6× Faster Than CSV.read

The headline number that usually surprises people: SmarterCSV 1.16 returns fully processed symbol-keyed hashes with numeric conversion — and still beats CSV.read (which returns raw string arrays with no post-processing at all):

Comparison	Speedup
vs `CSV.read` (raw arrays)	1.8×–8.6× faster
vs `CSV.table` (symbol keys + numeric conversion)¹	7×–129× faster
vs SmarterCSV 1.15.2	up to 2.4× faster
vs SmarterCSV 1.14.4	9×–65× faster

Measured on 19 benchmark files, Apple M1 Pro, Ruby 3.4.7. The 129× figure is on a 117-column import file where CSV.table's overhead compounds with column count.

¹ The comparison against CSV.table is more apples-to-apples: both produce symbol-keyed hashes with numeric conversion. That's what you actually need in a Rails app — but CSV.table and CSV.read have bugs, including silently mis-handling numbers with leading-zeroes. See 10 Ways Ruby's CSV.read Can Silently Corrupt or Lose Your Data.

What drove these gains

The C extension was overhauled to minimize allocations and per-row overhead in the inner parsing loop. The pure-Ruby path was also improved to build result hashes more directly, with key options cached upfront instead of re-read on every row.

Column selection speedup

When using headers: { only: [...] } to keep a subset of columns, excluded columns are skipped entirely in the C hot path. This is most significant when the columns you need are towards the front of the file. If the columns you want are near the end, the parser still has to read through all the preceding columns before it can skip anything — so the gains will be smaller.

Columns kept	Speedup vs no selection
2 of 500	~16× faster
10 of 500	~8× faster
50 of 500	~3× faster

# Only extract the 3 columns you need from a 500-column file
rows = SmarterCSV.process('wide_export.csv',
  headers: { only: [:id, :name, :email] })

Instrumentation Hooks

SmarterCSV.process('large_import.csv',
  chunk_size: 1000,
  on_start: ->(info) {
    puts "Starting import of #{info[:file_size]} bytes"
  },
  on_chunk: ->(info) {
    puts "Chunk #{info[:chunk_number]}: #{info[:total_rows_so_far]} rows so far"
  },
  on_complete: ->(info) {
    puts "Done: #{info[:total_rows]} rows in #{info[:duration].round(2)}s"
    puts "Bad rows: #{info[:bad_rows]}"
  }
)

Expanded Read API

`SmarterCSV.parse` — parse strings directly

# Before — had to wrap in StringIO
reader = SmarterCSV::Reader.new(StringIO.new(csv_string))

# Now
rows = SmarterCSV.parse(csv_string)

Drop-in equivalent of CSV.parse(str, headers: true, header_converters: :symbol) — with numeric conversion included.

`SmarterCSV.each` / `Reader#each` — row-by-row enumerator

Reader now includes Enumerable:

# Lazy pipeline — process a 10M-row file with constant memory
SmarterCSV.each('huge.csv')
  .lazy
  .select { |row| row[:status] == 'active' }
  .first(100)
  .each { |row| MyModel.create!(row) }

`SmarterCSV.each_chunk` / `Reader#each_chunk`

SmarterCSV.each_chunk('data.csv', chunk_size: 500).each_with_index do |chunk, i|
  puts "Importing chunk #{i}..."
  MyModel.import(chunk)
end

Bad Row Quarantine

Real-world CSV files are malformed. Until now, SmarterCSV raised on the first bad row and stopped — all-or-nothing. 1.16 adds a full quarantine system.

`on_bad_row:`

# :raise (default) — fail fast, same as before
SmarterCSV.process('data.csv')

# :collect — continue and keep the error records
good_rows = SmarterCSV.process('data.csv', on_bad_row: :collect)
SmarterCSV.errors[:bad_rows].each do |rec|
  Rails.logger.warn "Bad row #{rec[:csv_line_number]}: #{rec[:error_message]}"
  Rails.logger.warn "Raw: #{rec[:raw_logical_line]}"
end

# callable — inline handling, no Reader instance needed
bad_rows = []
good_rows = SmarterCSV.process('data.csv',
  on_bad_row: ->(rec) { bad_rows << rec })

# :skip — continue, count available afterwards
SmarterCSV.process('data.csv', on_bad_row: :skip)
puts SmarterCSV.errors[:bad_row_count]   # => 3

Each error record contains:

{
  csv_line_number:     3,
  file_line_number:    3,
  file_lines_consumed: 1,
  error_class:         SmarterCSV::HeaderSizeMismatch,
  error_message:       "extra columns detected ...",
  raw_logical_line:    "Jane,25,Boston,EXTRA_DATA\n",
}

`SmarterCSV.errors` — class-level error access (1.16.1)

Previously, accessing bad_row_count after a class-level call required switching to SmarterCSV::Reader. 1.16.1 exposes errors directly:

SmarterCSV.process('data.csv', on_bad_row: :skip)
puts SmarterCSV.errors[:bad_row_count]   # => 3

SmarterCSV.process('data.csv', on_bad_row: :collect)
puts SmarterCSV.errors[:bad_rows].size   # => 3

SmarterCSV.errors is thread-local — each thread in Puma/Sidekiq tracks its own state independently. It stores the result of the most recent call on the current thread.

⚠️ Fibers: SmarterCSV.errors uses Thread.current, which is shared across all fibers
in the same thread. If you process CSV in fibers (Async, Falcon, manual Fiber
scheduling), use SmarterCSV::Reader directly — its errors are scoped to the instance.

`field_size_limit:` — DoS protection

One unclosed " in a large file causes Ruby's CSV to read the entire rest of the file into a single field — a silent OOM risk. field_size_limit: N raises FieldSizeLimitExceeded as soon as any field or accumulating multiline buffer exceeds N bytes:

SmarterCSV.process('uploads/user_data.csv',
  field_size_limit: 1_000_000,   # 1 MB per field
  on_bad_row: :collect)

`bad_row_limit:` — abort after too many failures

reader = SmarterCSV::Reader.new('data.csv',
  on_bad_row: :collect,
  bad_row_limit: 10)

begin
  result = reader.process
rescue SmarterCSV::TooManyBadRows => e
  puts "Aborting: #{e.message}"
  puts "Collected so far: #{reader.errors[:bad_rows].size}"
end

Other new options

collect_raw_lines: true (default): Include the raw stitched line in bad-row error records. Set to false for privacy or memory savings.
nil_values_matching: regex: Set fields matching the regex to nil. With remove_empty_values: true (default), nil-ified values are removed from the hash. Replaces the deprecated remove_values_matching:.
verbose: :quiet / :normal / :debug: Symbol-based verbosity. :quiet suppresses all output; :normal (default) shows behavioral warnings; :debug adds per-row diagnostics to $stderr. Replaces the deprecated verbose: true/false.

Quote Handling Improvements

`quote_boundary: :standard` (default — minor breaking change)

Previously, a quote character mid-field (e.g. 5'10" or O'Brien) could toggle quoted mode and silently corrupt the row. The new default :standard mode only recognizes quotes as field delimiters at field boundaries — RFC 4180 compliant behavior.

In practice, mid-field quotes were already producing silent corruption in 1.15.x, so this is a bug fix that looks like a breaking change. Use quote_boundary: :legacy only if you deliberately relied on the old behavior.

`quote_escaping: :auto` (default)

MySQL SELECT INTO OUTFILE, PostgreSQL COPY TO, and many Unix tools escape quotes as \" instead of "" (RFC 4180). :auto mode handles both conventions row-by-row without configuration:

# Both of these parse correctly with the default quote_escaping: :auto
rows = SmarterCSV.process('mysql_export.csv')    # uses \"
rows = SmarterCSV.process('excel_export.csv')    # uses ""

Writer Improvements

IO and StringIO support

# Write to any IO object
SmarterCSV.generate($stdout) do |csv|
  csv << {name: "Alice", age: 30}
end

# Write to a StringIO buffer
buffer = StringIO.new
SmarterCSV.generate(buffer) do |csv|
  csv << {name: "Alice", age: 30}
end
csv_string = buffer.string

Generate to String directly

csv_string = SmarterCSV.generate do |csv|
  csv << {name: "Alice", age: 30}
  csv << {name: "Bob",   age: 25}
end

New writer options

SmarterCSV.generate('output.csv',
  encoding:          'ISO-8859-1',
  write_nil_value:   'NULL',
  write_empty_value: '',
  write_bom:         true,        # UTF-8 BOM for Excel compatibility
) do |csv|
  csv << row
end

Streaming mode

When headers: or map_headers: is provided at construction, the Writer skips the internal temp file entirely — the header line is written immediately and each << streams directly to the output. No API change; existing code benefits automatically.

Bug Fixes

1.16.0:

Empty/whitespace-only header cells now auto-generate names (column_1, column_2, …) instead of colliding on "" — fixes #324 and #312
Mid-field quotes no longer corrupt unquoted fields (quote_boundary: :standard)
All library output now goes to $stderr — nothing written to $stdout
Writer temp file no longer hardcoded to /tmp (fixes Windows)

1.16.1:

col_sep in quoted headers was parsed incorrectly — fixes #325 (thanks to Paho Lurie-Gregg)
Quoted numeric fields were not converted to numeric

Deprecations

These options still work but emit a warning — update when convenient:

Old	New
`remove_values_matching:`	`nil_values_matching:`
`strict: true`	`missing_headers: :raise`
`strict: false`	`missing_headers: :auto`
`verbose: true`	`verbose: :debug`
`verbose: false`	`verbose: :normal`

By the Numbers

	1.15.1 → 1.16.1
RSpec tests	714 → 1,410 (+696)
Line coverage	100%
Benchmark files	19
New options	10
New exceptions	2

Switch from Ruby CSV to SmarterCSV in 5 Minutes

Tilo Sloboda — Mon, 16 Mar 2026 17:06:19 +0000

Why switch? → 10 Ways Ruby's CSV.read Can Silently Corrupt or Lose Your Data

In this article we'll explore how easy it is to switch from Ruby CSV to SmarterCSV — often just a single line change.

But we'll also go beyond the basics and look at advanced scenarios where SmarterCSV really shines: parallel processing with Sidekiq, streaming imports directly from S3, production-grade instrumentation, and resumable imports that survive deployments mid-file. These are patterns that Ruby's built-in CSV library can't handle without you building all the plumbing from scratch.

Here's how to make the switch in 5 minutes.

Ruby's built-in CSV library works — but it's slow, and its default output is arrays of arrays, where row data is disassociated from the headers. That means your code has to manually correlate values with column names, introducing risk and boilerplate. The result doesn't lend itself to direct use with ActiveRecord, Sidekiq, or any hash-based workflow — you're always required to do post-processing to get to usable data, re-implementing boilerplate code to clean up data.

SmarterCSV returns Rails-ready hashes with symbol keys, automatic numeric conversion, and whitespace stripping, using sensible defaults to clean up the data — all out of the box. No boilerplate code, and it does it up to 2×–9× faster than CSV.read while returning cleaned-up arrays of hashes.

How Much Faster? 🚀

SmarterCSV is designed for real-world CSV processing — the full pipeline including hash construction, key normalization, and type conversion, not just raw tokenization.

Comparison	Range
vs `CSV.read` (arrays only, no post-processing)	1.7×–8.6× faster
vs `CSV.table` (closest equivalent output)	7×–129× faster

Benchmarks: 19 CSV files (20k–80k rows), Ruby 3.4.7, Apple M1, SmarterCSV 1.16.0.

The CSV.table comparison is the fair one — both return symbol-keyed hashes. CSV.read returns raw arrays, so the post-processing work your application still needs to do is not included in that number, understating the real cost difference.

Step 1: Install

# Gemfile
gem 'smarter_csv'

bundle install

Step 2: The One-Line Switch

Most developers use CSV.read with headers: true, which returns an array of CSV::Row objects with string keys. To get usable hashes, you need to call .map(&:to_h) — and you still have string keys, no type conversion, and no whitespace stripping.

Consider this real-world CSV file — messy headers, extra columns without headers, a trailing comma:

$ cat data.csv
   First Name  , Last Name , Age
Alice , Smith,  30, VIP, Gold ,
Bob, Jones,  25

Before: Ruby CSV

rows = CSV.read('data.csv', headers: true).map(&:to_h)
rows.first
# => {"   First Name  " => "Alice ", " Last Name " => " Smith", " Age" => "  30", nil => ""}
#                                                                                   ^^^ "VIP" and "Gold" silently lost!

Whitespace-polluted keys, Age as a string, and every extra column competes for the same nil key — the last one wins, the rest are silently discarded.

After: SmarterCSV

rows = SmarterCSV.process('data.csv')
rows.first
# => {first_name: "Alice", last_name: "Smith", age: 30, column_1: "VIP", column_2: "Gold"}
#    trailing empty field dropped, no data loss

Clean symbol keys, whitespace stripped, age converted to Integer, extra columns named — no data loss.

That's it. No .map(&:to_h), no header_converters:, no manual post-processing.

Step 3: Know the Differences

SmarterCSV's defaults are designed for real-world use. Here's what changes and what to watch for:

String keys → Symbol keys

CSV.read returns string keys by default. SmarterCSV returns symbol keys, which are more efficient (much lower memory usage and faster), as well as idiomatic for Rails. If you genuinely need string keys, you could still add strings_as_keys: true.

Before: Ruby CSV

rows = CSV.read('data.csv', headers: true).map(&:to_h)
rows.first['name']   # => "Alice"

After: SmarterCSV

# symbol keys (the default)
rows = SmarterCSV.process('data.csv')
rows.first[:name]    # => "Alice"

# string keys — if you don't mind the memory impact
rows = SmarterCSV.process('data.csv', strings_as_keys: true)
rows.first['name']   # => "Alice"

Numeric conversion is automatic

CSV.read returns everything as strings - not ideal for consumption. SmarterCSV converts "42" → 42 and "3.14" → 3.14 automatically.

Watch out for columns where leading zeros matter — ZIP codes, phone numbers, account numbers - and exclude them:

Before: Ruby CSV

rows = CSV.read('data.csv', headers: true).map(&:to_h)
rows.first['age']    # => "30"  (string)

After: SmarterCSV

# numeric strings converted automatically
rows = SmarterCSV.process('data.csv')
rows.first[:age]     # => 30  (Integer)

# Exclude columns where leading zeros matter
rows = SmarterCSV.process('data.csv',
  convert_values_to_numeric: { except: [:zip_code, :phone, :account_number] })

Empty values are removed

SmarterCSV drops key/value pairs where the value is blank, and returns cleaned-up data hashes. CSV.read keeps them as nil.

Before: Ruby CSV

rows = CSV.read('sample.csv', headers: true, header_converters: :symbol).map(&:to_h)
rows[1]   # => { name: "Bob", age: "25", city: nil }

After: SmarterCSV

rows = SmarterCSV.process('sample.csv')
rows[1]   # => { name: "Bob", age: 25 }   ← :city dropped, :age converted

# To keep nil values (match CSV.read behaviour):
rows = SmarterCSV.process('sample.csv', remove_empty_values: false)
rows[1]   # => { name: "Bob", age: 25, city: nil }

Plain Hash, not CSV::Row

CSV.read returns CSV::Row objects — a wrapper around a hash with extra methods. SmarterCSV returns plain Ruby Hash objects, so there's no .to_h needed and no wrapper to unwrap.

Before: Ruby CSV

row = CSV.read('data.csv', headers: true).first
row.class        # => CSV::Row
row['name']      # => "Alice"   (string key)
row.to_h         # => {"name" => "Alice", "age" => "30"}  (still strings)

After: SmarterCSV

row = SmarterCSV.process('data.csv').first
row.class        # => Hash
row[:name]       # => "Alice"   (symbol key, no unwrapping needed)

Quick Reference

Ruby CSV	SmarterCSV	Notes
`CSV.read(f, headers: true).map(&:to_h)`	`SmarterCSV.process(f)`	Symbol keys, numeric conversion, whitespace stripped.
`CSV.read(f, headers: true, header_converters: :symbol).map(&:to_h)`	`SmarterCSV.process(f)`	Drop-in.
`CSV.table(f)`	`SmarterCSV.process(f)`	`CSV.table` returns a `CSV::Table` of `CSV::Row` objects; SmarterCSV returns a plain `Array` of `Hash`.
`CSV.parse(str, headers: true, header_converters: :symbol)`	`SmarterCSV.parse(str)`	Direct string parsing, new in 1.16.0.
`CSV.foreach(f, headers: true) { \	r\	}`
`converters: :numeric`	default	Automatic in SmarterCSV.
`converters: :date`	`value_converters: {col: DateConverter}`	Use explicit format strings — date formats are locale-dependent.
`liberal_parsing: true`	`on_bad_row: :collect`	Explicit quarantine gives you visibility.
`skip_blanks: true`	`remove_empty_hashes: true`	Default in SmarterCSV.
`row.to_h`	`row`	Already a plain Hash.
`row.headers`	`reader.headers`	Available on the `Reader` instance.

Beyond the Basics: What You Unlock

Once you're on SmarterCSV, these features come for free.

Batch processing for large files

SmarterCSV.process('big.csv', chunk_size: 500) do |chunk|
  MyModel.insert_all(chunk)   # bulk insert 500 rows at a time
end

Handle bad rows without crashing

good_rows = SmarterCSV.process('data.csv', on_bad_row: :collect)

puts "#{good_rows.size} imported, #{SmarterCSV.errors[:bad_row_count]} bad rows"
SmarterCSV.errors[:bad_rows].each { |r| puts "Line #{r[:file_line_number]}: #{r[:error_message]}" }

⚠️ Fibers: SmarterCSV.errors relies on Thread.current, which is shared across all
fibers in the same thread. If you process CSV concurrently in fibers (Async, Falcon,
manual Fiber scheduling), use SmarterCSV::Reader directly instead — its errors are
scoped to the instance.

Or use SmarterCSV::Reader directly when you also need access to headers or other reader state after processing:

reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :collect)
good_rows = reader.process

puts "#{good_rows.size} imported, #{reader.errors[:bad_rows].size} bad rows"
reader.errors[:bad_rows].each { |r| puts "Line #{r[:file_line_number]}: #{r[:error_message]}" }

Sentinel values (NULL, N/A, #VALUE!)

rows = SmarterCSV.process('export.csv',
  nil_values_matching: /\A(NULL|N\/A|NaN|#VALUE!)\z/i)
# Matching values are nil-ified and removed automatically

Custom type converters

Date formats are locale-dependent, so SmarterCSV doesn't guess. You supply an explicit format:

require 'date'

rows = SmarterCSV.process('records.csv',
  value_converters: {
    birth_date: ->(v) { v ? Date.strptime(v, '%m/%d/%Y') : nil },
    price:      ->(v) { v&.delete('$,')&.to_f },
    active:     ->(v) { v&.match?(/\Atrue\z/i) },
  })

Row-by-row iteration with full Enumerable

Before: Ruby CSV

CSV.foreach('data.csv', headers: true, header_converters: :symbol) do |row|
  MyModel.create(row.to_h)
end

After: SmarterCSV

SmarterCSV.each('data.csv') do |row|
  MyModel.create(row)   # already a Hash
end

# Full Enumerable — filter, map, lazy
active_users = SmarterCSV.each('data.csv').select { |r| r[:status] == 'active' }
first_ten    = SmarterCSV.each('data.csv').lazy.first(10)

Rails file upload

Accepting a CSV upload in a Rails controller is straightforward — pass the tempfile path directly:

# app/controllers/imports_controller.rb
def create
  file = params[:file]   # ActionDispatch::Http::UploadedFile

  SmarterCSV.process(file.path, chunk_size: 500) do |chunk|
    MyModel.insert_all(chunk)
  end

  redirect_to root_path, notice: "Import complete"
end

No temp file management, no manual header parsing. The uploaded file is processed in streaming chunks, keeping memory usage low regardless of file size.

Renaming headers to match your database

CSV column names rarely match your ActiveRecord attribute names. Use key_mapping: to rename them in one step — the mapping uses the normalized (downcased, underscored) header name as input:

# CSV headers: "First Name", "Last Name", "E-Mail", "Date of Birth"
# After normalization:  :first_name, :last_name, :e_mail, :date_of_birth

rows = SmarterCSV.process('contacts.csv',
  key_mapping: {
    first_name:    :given_name,
    last_name:     :family_name,
    e_mail:        :email,
    date_of_birth: :dob,
  })
# => [{given_name: "Alice", family_name: "Smith", email: "alice@example.com", dob: "1990-05-14"}, ...]

Map a key to nil to drop that column entirely:

key_mapping: { internal_id: nil, created_at: nil }   # these columns won't appear in results

Select only the columns you need

Wide CSV files often have dozens of columns your application doesn't need. Use headers: { only: } to declare upfront which columns to keep — SmarterCSV skips everything else at the parser level, so unneeded fields are never allocated:

# CSV has 50 columns — you only need 3
rows = SmarterCSV.process('contacts.csv',
  headers: { only: [:email, :first_name, :last_name] })
# => [{email: "alice@example.com", first_name: "Alice", last_name: "Smith"}, ...]

Or exclude a known noisy column while keeping everything else:

rows = SmarterCSV.process('export.csv', headers: { except: [:internal_notes] })

Writing CSV from hashes

Before: Ruby CSV

# you manage headers manually, pass arrays
CSV.open('out.csv', 'w', write_headers: true, headers: ['name', 'age']) do |csv|
  csv << ['Alice', 30]
end

After: SmarterCSV

# pass hashes, headers discovered automatically
SmarterCSV.generate('out.csv') do |csv|
  csv << {name: 'Alice', age: 30}
  csv << {name: 'Bob',   age: 25}
end

Advanced Patterns: Where SmarterCSV Really Shines

These are scenarios where Ruby CSV falls short and SmarterCSV makes the solution clean and straightforward.

Parallel processing with Sidekiq

Each chunk is dispatched as an independent background job — the chunk API maps directly onto worker queues:

SmarterCSV.process('users.csv', chunk_size: 100) do |chunk, chunk_index|
  puts "Queueing chunk #{chunk_index} (#{chunk.size} records)..."
  Sidekiq::Client.push_bulk(
    'class' => UserImportWorker,
    'args'  => chunk,
  )
end
# => imports run in parallel across all your Sidekiq workers

Streaming directly from S3

SmarterCSV accepts any IO-like object — so you can stream a CSV directly from S3 without writing a temp file:

require 'aws-sdk-s3'

s3  = Aws::S3::Client.new(region: 'us-east-1')
obj = s3.get_object(bucket: 'my-bucket', key: 'imports/contacts.csv')

SmarterCSV::Reader.new(obj.body, chunk_size: 500).each_chunk do |chunk, _index|
  MyModel.insert_all(chunk)
end

No disk I/O, no temp files, no cleanup. The S3 response body streams directly into the parser.

Production instrumentation

on_start, on_chunk, and on_complete hooks give you full visibility into long-running imports — feed them into your logger, StatsD, or any metrics backend:

SmarterCSV.process('large_import.csv',
  chunk_size: 1_000,

  on_start: ->(info) {
    Rails.logger.info "Import started: #{info[:input]} (#{info[:file_size]} bytes)"
  },

  on_chunk: ->(info) {
    Rails.logger.debug "Chunk #{info[:chunk_number]}: #{info[:rows_in_chunk]} rows " \
                       "(#{info[:total_rows_so_far]} total so far)"
  },

  on_complete: ->(stats) {
    Rails.logger.info "Import complete: #{stats[:total_rows]} rows in #{stats[:duration].round(2)}s, " \
                      "#{stats[:bad_rows]} bad rows"
    StatsD.histogram('csv.import.duration', stats[:duration])
  },
) { |chunk| MyModel.insert_all(chunk) }

Resumable imports with Rails ActiveJob

Rails 8.1 introduced ActiveJob::Continuable — jobs that can pause mid-execution (on deployment or queue drain) and resume exactly where they stopped. SmarterCSV's chunk_index maps directly onto the job cursor:

class ImportCsvJob < ApplicationJob
  include ActiveJob::Continuable

  def perform(file_path)
    step :import_rows do |step|
      SmarterCSV.process(file_path, chunk_size: 500) do |chunk, chunk_index|
        next if chunk_index < step.cursor.to_i   # skip already-processed chunks on resume

        MyModel.insert_all(chunk)
        step.set! chunk_index + 1
      end
    end
  end
end

On interruption after chunk 7, Rails persists the cursor as 8. On the next run, chunks 0–7 are skipped instantly and processing resumes from chunk 8. Ruby CSV has no equivalent — you'd have to implement cursor tracking, row counting, and resume logic yourself.

Bulk upsert — insert or update

For recurring imports where records may already exist, use upsert_all with a unique key. SmarterCSV's hashes pass directly — no transformation needed:

SmarterCSV.process('contacts.csv',
  chunk_size: 500,
  key_mapping: { e_mail: :email },   # normalize header to match DB column
) do |chunk|
  Contact.upsert_all(chunk, unique_by: :email)
  # inserts new records, updates existing ones — all in one query per chunk
end

That's It - Enjoy! ✨

Install the gem, change one line, check the three behavior differences (numeric conversion, empty value removal, plain Hash vs CSV::Row), and you're done.

The rest — batch processing, bad row handling, value converters, column selection — is there when you need it.

gem 'smarter_csv'

GitHub: github.com/tilo/smarter_csv
Docs: Full documentation
RubyGems: rubygems.org/gems/smarter_csv

10 Ways Ruby's CSV.read Can Silently Corrupt or Lose Your Data

Tilo Sloboda — Mon, 16 Mar 2026 17:03:50 +0000

When having to parse CSV files, many developers go straight to the Ruby CSV library — it ships with Ruby and requires no dependencies.

But it comes at the cost of boilerplate post-processing you have to write, test, and maintain yourself. Worse, there are some failure modes that produce no exception, no warning, and no indication that anything went wrong. Your import runs, your tests pass, and your data is quietly wrong.

CSV.read is fine for small, trusted, well-formed files — particularly when you control the source. This article is about what can happen with messy real-world files your partners produce, or users upload — ten reproducible ways CSV.read and CSV.table can silently corrupt or lose data, with examples you can run yourself, and how SmarterCSV handles each case.

Not all ten may be equally surprising — some are odd behavior that bites you anyway, others are genuine traps. All ten are silent.

💡 Want to follow along? Download the example CSV files and run the examples locally.

At a Glance

¹ Issue #4 can be triggered two ways: CSV.table enables converters: :numeric by default (no opt-in required), and CSV.read triggers the same corruption when passed converters: :numeric explicitly. Either way, any leading-zero string field — ZIP codes, customer IDs, product codes — is silently converted to a wrong integer.

² The one case where CSV.table does better than CSV.read: its header_converters: :symbol option includes .strip, so whitespace is removed from headers (#5). Values (#6) are not stripped — CSV.table has the same whitespace-around-values problem. For all other issues CSV.table is identical to or worse than CSV.read.

CSV.table is a convenience wrapper for CSV.read with headers: true, header_converters: :symbol, and converters: :numeric.

The Real Cost of Handling This Yourself

Experienced users of CSV.read know some of these gotchas and handle them in post-processing — but not all of them can be: some are serious bugs that will silently corrupt your data regardless. And even for the ones you can handle, manual post-processing has five hidden costs:

You hand-craft boilerplate for every use case. The right fix for whitespace differs when headers have spaces vs. values have spaces vs. both. Encoding handling depends on the source system. There is no generic post-processing snippet — you write a slightly different version every time.
You have to remember all of it, every time. Every new import, service, or data source needs the same gotchas handled — consistently. But boilerplate doesn't enforce itself. A fix you wrote for one importer doesn't automatically apply to the next. The gotchas don't announce themselves — you only catch them if you remember to look.
Your boilerplate is probably undertested. Post-processing code that wraps CSV.read rarely gets the same test coverage as business logic. Developers don't think of it as the risky part. Data edge cases — files with blank headers, leading-zero IDs, quoted empty fields, mixed encoding — don't make it into the test suite until they cause a production incident. You don't know what your boilerplate misses until a file breaks it.

❓ Do your tests for your CSV wrapper just test the mechanics, or include data corner cases?

Your benchmarks probably don't include the boilerplate code. When you chose CSV.read, you probably looked at raw parsing performance — but did you measure the end-to-end cost of your post-processing? Whitespace stripping, header cleanup, empty normalization: none of that is free. Your end-to-end data pipeline is much slower than what you initially measured. SmarterCSV 1.16 benchmarks at 1.8×–8.6× faster than CSV.read end-to-end — before you even factor in the boilerplate you no longer write.
One library that handles it predictably and performant is worth more than the sum of its parts. The value isn't "these ten cases are covered." It is that you stop maintaining a bespoke cleaning pipeline, stop writing one-off fixes after production surprises, and don't have to worry about test coverage or performance - you can trust that the default behavior handles edge cases sensibly — without silently damaging your data.

Predictable behavior in a well-tested library beats hand-crafted boilerplate that anticipates fewer edge cases.

Why These Failures Are Dangerous

Every single failure in this list is silent. No exception, no warning, no log line — your import completes successfully and your data is quietly wrong. That's what makes these issues so dangerous: they don't surface in tests, they don't cause immediate errors, and they're easy to miss during code review.

The root cause is that CSV.read is a tokenizer, not a data pipeline. It splits bytes into fields and hands them back with no normalization, no validation, and no defensive handling of real-world messiness. Every assumption about what "clean" input looks like is left to the caller.

Issue #4 deserves special mention: CSV.table's default converters: :numeric silently turns "00123" into 83³ and "01234" into 668³ — values that look like perfectly valid integers. ZIP codes, customer IDs, and product codes are quietly replaced with wrong numbers that pass every validation, get stored in your database, and are indistinguishable from real data until someone notices the numbers don't match.

These aren't obscure edge cases. Extra columns, trailing commas, Windows-1252 encoding, duplicate headers, blank header cells, TSV-vs-CSV confusion, leading-zero identifiers, and whitespace-padded values are all common in CSV files exported from Excel, reporting tools, ERP systems, and legacy data pipelines. If your application accepts user-uploaded CSV files, you will encounter these.

The defensive post-processing code required to handle all ten cases correctly — octal-safe numeric conversion, whitespace normalization, duplicate header disambiguation, extra column naming, consistent empty value handling, backslash quote escaping, delimiter auto-detection, encoding detection — is non-trivial to write, test, and maintain. Most applications never bother, because the failures are silent.

³ These aren't rounding errors or truncations — they are completely different numbers. Octal is a base-8 number system from the early days of computing, still used in low-level Unix file permissions and C integer literals. It has no place in CSV data. No spreadsheet, ERP system, or database exports ZIP codes or customer IDs in octal — but Ruby CSV silently assumes that's exactly what a leading zero means.

Read on for a detailed explanation and reproducible example for each issue.

1. Extra Columns Without Headers — Values Silently Discarded

When a row has more fields than there are headers, CSV.read maps every extra field to the nil key. If there are multiple extra fields, they all compete for the same nil key — only the first one survives, the rest are silently discarded.

$ cat example1.csv
   First Name  , Last Name , Age
Alice , Smith,  30, VIP, Gold ,
Bob, Jones,  25

rows = CSV.read('example1.csv', headers: true).map(&:to_h)
rows.first
# => {
#       "   First Name  " => "Alice ",
#           " Last Name " => " Smith",
#                 " Age" => "  30",
#                    nil => " VIP"
#                    ^^^^^^^^^^^^^
#  data from unnamed column with "Gold" is silently lost
# }

Alice's row has 6 fields but only 3 headers. The extra fields " VIP", " Gold", and "" (trailing comma) all land on nil — only the first one wins. No error, no warning.

This is common in real-world exports: tools frequently append audit columns, status flags, or trailing commas that don't correspond to headers.

CSV.table has the same problem.

SmarterCSV: The default missing_headers: :auto auto-generates distinct names for extra columns using missing_header_prefix (default: "column_"). The trailing empty field is dropped by the default remove_empty_values: true setting. No data loss.

rows = SmarterCSV.process('example1.csv')
rows.first
# => {
#     first_name: "Alice",
#      last_name: "Smith",
#            age: 30,
#       column_4: "VIP",
#       column_5: "Gold"
#       ^^^^^^^^^^^^^^^^
#  extra data columns are handled, no data is lost
# }

2. Duplicate Header Names — Second Value Silently Dropped

When two columns share the same header name, CSV::Row#to_h keeps only the first value. Later values are silently dropped.

$ cat example2.csv
score,name,score
95,Alice,87

rows = CSV.read('example2.csv', headers: true).map(&:to_h)
rows.first
# => {"score" => "95", "name" => "Alice"}
#    ^^^ second score (87) silently lost

Common with reporting tool exports that repeat a column (e.g., two date columns both labeled "Date").

CSV.table has the same problem.

SmarterCSV: disambiguates duplicate headers by appending a number directly: :score, :score2, :score3.

rows = SmarterCSV.process('example2.csv')
rows.first
# => {score: 95, name: "Alice", score2: 87}

The default duplicate_header_suffix: "" disambiguates by appending a counter: :score, :score2, :score3.
Use duplicate_header_suffix: '_' to get :score_2, :score_3.
Set duplicate_header_suffix: nil to raise DuplicateHeaders instead.

3. Empty Header Fields — `nil` Key Collision

A CSV file with blank header fields (e.g., name,,age) gives those columns a nil key. Multiple blank headers all collide on nil — same overwrite problem as issue #1, and only the first value survives.

Note: this is distinct from issue #1. Issue #1 is about extra data fields beyond the header count, which get keyed under nil. Issue #3 is about blank cells in the header row itself, which also get keyed under nil.

$ cat example3.csv
name,,,age
Alice,foo,bar,30

rows = CSV.read('example3.csv', headers: true).map(&:to_h)
rows.first
# => {"name" => "Alice", nil => "foo", "age" => "30"}
#    ^^^ "bar" silently lost — both blank headers map to nil, first value wins

CSV.table has the same nil key collision:

rows = CSV.table('example3.csv').map(&:to_h)
rows.first
# => {name: "Alice", nil => "foo", age: 30}
#    ^^^ "bar" still silently lost

SmarterCSV: missing_header_prefix: (default "column_") auto-generates names for blank headers: :column_1, :column_2, etc. No collision, no data loss.

rows = SmarterCSV.process('example3.csv')
rows.first
# => {name: "Alice", column_1: "foo", column_2: "bar", age: 30}

4. `converters: :numeric` Silently Corrupts Leading-Zero Values as Octal

converters: :numeric When numbers have leading zeroes, the result does not just strip them - the entire number is silently converted to a completely different value³ that looks plausible but is incorrect ❌ .

CSV.table enables converters: :numeric by default without any opt-in, triggering the bug by default. CSV.read is safe by default, but triggers the same corruption when converters: :numeric (or converters: :integer) is passed explicitly.

$ cat example4.csv
customer_id,zip_code,amount
00123,01234,99.50
00456,90210,9.99

With Ruby CSV:

# CSV.table — converters: :numeric on by default, no opt-in needed
rows = CSV.table('example4.csv').map(&:to_h)
rows.first
# => {customer_id: 83, zip_code: 668, amount: 99.5}
#    ^^^ "00123" → 83  (octal 0123 = decimal 83)
#    ^^^ "01234" → 668 (octal 1234 = decimal 668)

# CSV.read with explicit converters: :numeric — same result
rows = CSV.read('example4.csv', headers: true, converters: :numeric).map(&:to_h)
rows.first
# => {"customer_id" => 83, "zip_code" => 668, "amount" => 99.5}

"00123" becomes 83. "01234" becomes 668. ZIP codes, customer IDs, order numbers, product codes — any field with a leading zero becomes a completely wrong integer. No exception, no warning. The resulting values look plausible and pass all type validations.

CSV.read without converters is safe — strings are returned as-is:

rows = CSV.read('example4.csv', headers: true).map(&:to_h)
rows.first
# => {"customer_id" => "00123", "zip_code" => "01234", "amount" => "99.50"}

SmarterCSV:

# Default (convert_values_to_numeric: true) — decimal conversion, no octal trap
rows = SmarterCSV.process('example4.csv')
rows.first
# => {customer_id: 123, zip_code: 1234, amount: 99.5}

# convert_values_to_numeric: false — preserves strings exactly, including leading zeros
rows = SmarterCSV.process('example4.csv', convert_values_to_numeric: false)
rows.first
# => {customer_id: "00123", zip_code: "01234", amount: "99.50"}

SmarterCSV's default convert_values_to_numeric: true uses to_i / to_f, which always treats strings as decimal — no octal interpretation. Use convert_values_to_numeric: false when leading zeros must be preserved (ZIP codes, IDs, product codes).

5. Whitespace in Header Names — Silent `nil` on Lookup

CSV.read returns headers exactly as they appear in the file, including leading and trailing whitespace. Code that accesses columns by the expected name silently gets nil.

$ cat example5.csv
 name , age
Alice,30

rows = CSV.read('example5.csv', headers: true).map(&:to_h)
rows.first
# => {" name " => "Alice", " age" => "30"}

rows.first['name']   # => nil  ← silent miss; key is " name ", not "name"
rows.first['age']    # => nil

CSV.table mitigates this: ² the :symbol header converter includes .strip, so whitespace is removed from headers. This is the one issue where CSV.table behaves better than CSV.read.

SmarterCSV:

rows = SmarterCSV.process('example5.csv')
rows.first
# => {name: "Alice", age: 30}

The default setting strip_whitespace: true strips leading/trailing whitespace from both headers and values.

6. Whitespace Around Values — Silent Comparison Failure

CSV.read returns field values exactly as they appear in the file — leading spaces, trailing spaces, and tab characters all preserved. Exporters from fixed-width database systems (Oracle CHAR columns, COBOL-era systems) routinely pad string fields to a fixed width; other tools leave accidental leading spaces. The values look correct when printed, but equality checks silently return false.

This pairs with Example 5 (whitespace in headers): Ruby CSV strips neither headers nor values by default.

$ cat example6.csv
name,status,city
Alice,active  ,New York    ← trailing spaces after 'active'
Bob,inactive,Chicago
Carol, active,Boston       ← leading space before 'active'

rows = CSV.read('example6.csv', headers: true).map(&:to_h)

rows[0]['status']  # => "active  "
rows[2]['status']  # => " active"

rows.select { |r| r['status'] == 'active' }
# => []  ← Alice and Carol are not found. No error raised.

The values look fine in logs and puts output. The bug only surfaces when the comparison silently returns the wrong result.

Workaround: pass strip: true to CSV.read. This correctly strips spaces and tab characters. Note it also strips intentional leading/trailing spaces from any field — including quoted fields where spaces may be meaningful.

CSV.table has the same problem — its :symbol converter strips header names but does not touch field values.

SmarterCSV:

rows = SmarterCSV.process('example6.csv')

rows[0][:status]  # => "active"
rows[2][:status]  # => "active"

rows.select { |r| r[:status] == 'active' }.length  # => 2

strip_whitespace: true (default) strips all leading and trailing whitespace (spaces and tabs) from values. Set strip_whitespace: false to preserve spaces when needed.

7. `nil` vs `""` for Empty Fields — Inconsistent Empty Checks

CSV.read treats unquoted empty fields and quoted empty fields differently:

Unquoted empty (,,) → nil
Quoted empty (,"",) → ""

$ cat example7.csv
name,city
Alice,
Bob,""

rows = CSV.read('example7.csv', headers: true).map(&:to_h)

rows[0]['city']        # => nil   (unquoted empty)
rows[1]['city']        # => ""    (quoted empty)

rows[0]['city'].nil?   # => true
rows[1]['city'].nil?   # => false  ← same semantic meaning, different Ruby type

Both rows have no city. But your code sees two different things. Any check using .nil?, .blank?, .present?, or a simple if row['city'] will behave differently depending on how the upstream exporter happened to quote the empty field. No two exporters agree on this.

CSV.table has the same problem.

SmarterCSV: remove_empty_values: true (default) removes both from the hash. With remove_empty_values: false, both are normalized to "". Consistent either way.

# remove_empty_values: true (default) — both empty cities are dropped from the hash
rows = SmarterCSV.process('example7.csv')
rows[0]   # => {name: "Alice"}
rows[1]   # => {name: "Bob"}

# remove_empty_values: false — both normalized to ""
rows = SmarterCSV.process('example7.csv', remove_empty_values: false)
rows[0]   # => {name: "Alice", city: ""}
rows[1]   # => {name: "Bob",   city: ""}

8. Backslash-Escaped Quotes — MySQL / Unix Dump Format

MySQL's SELECT INTO OUTFILE, PostgreSQL COPY TO, and many Unix data-pipeline tools escape embedded double quotes as \" — not as "" (the RFC 4180 standard). Ruby's CSV only understands the RFC 4180 convention, so a backslash before a quote is treated as two separate characters: a literal \ followed by a " that immediately closes the field.

$ cat example8.csv
name,note
Alice,"She said \"hello\" to everyone"
Bob,"Normal note"

Scenario 1 — crash (at least you know something went wrong):

rows = CSV.read('example8.csv', headers: true)
# => CSV::MalformedCSVError: Any value after quoted field isn't allowed in line 2.

Scenario 2 — silent garbling with liberal_parsing: true:

rows = CSV.read('example8.csv', headers: true, liberal_parsing: true)
rows[0]['note']   # => 'She said \"hello\" to everyone'

No exception. No warning. The note field has extra wrapping quotes and mangled escaping — it won't compare, display, or serialize correctly.

CSV.table has the same problem — and adding liberal_parsing: true makes it silently worse.

SmarterCSV: quote_escaping: :auto (default since 1.0) detects and handles both "" and \" escaping row-by-row. No option required.

rows = SmarterCSV.process('example8.csv')
rows[0]   # => {name: "Alice", note: 'She said \"hello\" to everyone'}
rows[1]   # => {name: "Bob",   note: "Normal note"}

9. TSV File Read as CSV — Completely Breaks ❌

CSV.read defaults to col_sep: ",". When given a tab-delimited file (TSV), it finds no commas and treats each entire row as a single field. The header row becomes one giant key; each data row becomes one giant value. All column structure is silently lost — no error, no warning, and rows.length looks correct.

$ cat example9.csv
name    city    score
Alice   New York    95
Bob Chicago 87

rows = CSV.read('example9.csv', headers: true).map(&:to_h)

rows.length           # => 2  (looks right — but...)
rows.first.keys       # => ["name\tcity\tscore"]  ← entire header is one key
rows.first['name']    # => nil  ← column unreachable
rows.first.values     # => ["Alice\tNew York\t95"]  ← entire row is one value

This can happen when users upload TSV instead of CSV - the file name could still be .csv, so indistinguishable from actual CSV data.

CSV.table has the same problem.

SmarterCSV:

rows = SmarterCSV.process('example9.csv')
# col_sep: :auto detects the tab separator automatically

rows.first
# => {name: "Alice", city: "New York", score: 95}

col_sep: :auto (default) samples the file and detects the actual delimiter. No option required.

10. No Encoding Auto-Detection — Crash or Mojibake

CSV.read assumes UTF-8. CSV files exported from Excel on Windows are typically Windows-1252 (CP1252), which encodes accented characters (é, ü, ñ) differently from UTF-8.

$ cat example10.csv
last_name,first_name
Müller,Hans

The file is saved in Windows-1252 encoding — ü is stored as \xFC, not as UTF-8.

Scenario 1 — crash (the better outcome — at least you know):

rows = CSV.read('example10.csv', headers: true)
# => CSV::InvalidEncodingError: Invalid byte sequence in UTF-8 in line 2.

Scenario 2 — silent mojibake (the worse outcome):

# Specifying the wrong encoding suppresses the error
rows = CSV.read('example10.csv', headers: true, encoding: 'binary')
rows.first['last_name']                # => "M\xFCller"  ← garbled string
rows.first['last_name'].valid_encoding? # => true  ← Ruby thinks it's fine!

The mojibake string passes .valid_encoding?, passes database validations, gets stored, and surfaces as a display bug weeks later in production.

CSV.table has the same problem.

SmarterCSV: file_encoding: accepts Ruby's 'external:internal' transcoding notation; force_utf8: true transcodes to UTF-8 automatically; invalid_byte_sequence: '' controls the replacement character for bytes that can't be transcoded, e.g. ''.

rows = SmarterCSV.process('example10.csv',
  file_encoding: 'windows-1252:utf-8')
rows.first[:last_name]   # => "Müller"

The Alternative

gem 'smarter_csv'

# Before
rows = CSV.read('data.csv', headers: true).map(&:to_h)

# After
rows = SmarterCSV.process('data.csv')

SmarterCSV handles nine of the ten cases out of the box — octal-safe numeric conversion, whitespace normalization, duplicate header disambiguation, extra column naming, consistent empty value handling, backslash quote escaping, and delimiter auto-detection - no surprises! ✅

The remaining one (encoding control) requires explicit opt-in options, but the building blocks are there. No boilerplate, no post-processing pipeline, no silent data loss.

And it's not a trade-off on speed: SmarterCSV 1.16 benchmarks at 1.8×–8.6× faster than CSV.read end-to-end, and up to 129× faster than CSV.table — see the SmarterCSV 1.16 release notes for full benchmark details.

Ready to switch? → Switch from Ruby CSV to SmarterCSV in 5 Minutes

Give SmarterCSV a try and let us know how it performs in your use case — feedback and bug reports are welcome in the GitHub Discussions or Issues.

GitHub: github.com/tilo/smarter_csv
Docs: Full documentation
RubyGems: rubygems.org/gems/smarter_csv

DEV Community: Tilo Sloboda

Your Ruby CSV Import Ran Successfully — Your Data May Still Be Wrong

SmarterCSV 1.16 Released — Faster Than CSV.read, Bad Row Quarantine, Instrumentation, New Features, Improved API

Performance: 1.8×–8.6× Faster Than CSV.read

What drove these gains

Column selection speedup

Instrumentation Hooks

Expanded Read API

SmarterCSV.parse — parse strings directly

SmarterCSV.each / Reader#each — row-by-row enumerator

SmarterCSV.each_chunk / Reader#each_chunk

Bad Row Quarantine

on_bad_row:

SmarterCSV.errors — class-level error access (1.16.1)

field_size_limit: — DoS protection

bad_row_limit: — abort after too many failures

Other new options

Quote Handling Improvements

quote_boundary: :standard (default — minor breaking change)

quote_escaping: :auto (default)

Writer Improvements

IO and StringIO support

Generate to String directly

New writer options

Streaming mode

Bug Fixes

Deprecations

By the Numbers

Further Reading

Links

Switch from Ruby CSV to SmarterCSV in 5 Minutes

How Much Faster? 🚀

Step 1: Install

Step 2: The One-Line Switch

Step 3: Know the Differences

String keys → Symbol keys

Numeric conversion is automatic

Empty values are removed

Plain Hash, not CSV::Row

Quick Reference

Beyond the Basics: What You Unlock

Batch processing for large files

Handle bad rows without crashing

Sentinel values (NULL, N/A, #VALUE!)

Custom type converters

Row-by-row iteration with full Enumerable

Rails file upload

Renaming headers to match your database

Select only the columns you need

Writing CSV from hashes

Advanced Patterns: Where SmarterCSV Really Shines

Parallel processing with Sidekiq

Streaming directly from S3

Production instrumentation

Resumable imports with Rails ActiveJob

Bulk upsert — insert or update

That's It - Enjoy! ✨

10 Ways Ruby's CSV.read Can Silently Corrupt or Lose Your Data

At a Glance

The Real Cost of Handling This Yourself

Why These Failures Are Dangerous

1. Extra Columns Without Headers — Values Silently Discarded

2. Duplicate Header Names — Second Value Silently Dropped

3. Empty Header Fields — nil Key Collision

4. converters: :numeric Silently Corrupts Leading-Zero Values as Octal

5. Whitespace in Header Names — Silent nil on Lookup

6. Whitespace Around Values — Silent Comparison Failure

7. nil vs "" for Empty Fields — Inconsistent Empty Checks

8. Backslash-Escaped Quotes — MySQL / Unix Dump Format

9. TSV File Read as CSV — Completely Breaks ❌

10. No Encoding Auto-Detection — Crash or Mojibake

The Alternative

`SmarterCSV.parse` — parse strings directly

`SmarterCSV.each` / `Reader#each` — row-by-row enumerator

`SmarterCSV.each_chunk` / `Reader#each_chunk`

`on_bad_row:`

`SmarterCSV.errors` — class-level error access (1.16.1)

`field_size_limit:` — DoS protection

`bad_row_limit:` — abort after too many failures

`quote_boundary: :standard` (default — minor breaking change)

`quote_escaping: :auto` (default)

3. Empty Header Fields — `nil` Key Collision

4. `converters: :numeric` Silently Corrupts Leading-Zero Values as Octal

5. Whitespace in Header Names — Silent `nil` on Lookup

7. `nil` vs `""` for Empty Fields — Inconsistent Empty Checks