loading...
JetRockets profile image JetRockets

How to parse CSV with double quote (") character in Crystal

twitter logo github logo ・1 min read

We use microservice written in Crystal to parse large CSV files (about 1.5Gb). Some rows in these files may contain no closed " characters:

,Y,FEDERAL NATIONAL MORTGAGE ASSOCIATION "F,,

With Crystal default CSV parse settings this row and everything after it won't be parsed correctly because DEFAULT_QUOTE_CHAR constant is equal to ". Of course you can override quote_char param in CSV constructor with something that cannot be found in your document.

From my point of view the best is to use zero byte which is '\u0000' in Crystal.

csv = CSV.new(file, headers: true, strip: true, quote_char: '\u0000')
while csv.next
  # ... 
end

Hack!

twitter logo DISCUSS
JetRockets is a technology consulting firm that architects, designs, develops and supports enterprise-level web, mobile and software platforms helping clients achieve their goals and grow their businesses.
Classic DEV Post from Sep 13 '19

Follow Friday: Which DEVs would you recommend following?

Follow Friday: Which DEVs would you recommend following?

Creating your DEV account literally takes 30 seconds and is one of the best moves you can make for a happy and healthy software career.

Get started now ❤️