The supplier sent a file every night.
One line per product,
fields separated by a pipe.
Code, name, price,
discount,
and a note.
The import split each line on the pipe,
checked there were five parts,
and skipped anything else
as malformed.
It ran for months.
Then somebody noticed
that products without a discount
were never updated.
Not wrong.
Not failing.
Just never there.
The log said
a few hundred malformed lines a night,
which everyone had agreed
was the supplier's problem.
The lines were fine.
A product with no discount
and no note
ended in two empty fields.
ABC12|Hinge|4.20||
Five fields,
two of them empty.
Java's split
has a habit
written plainly in its documentation
and nowhere in anybody's memory.
Called with only the separator,
it removes trailing empty strings
from the result.
So that line
came back with three parts,
not five.
An empty field in the middle survives.
An empty field at the end vanishes,
and so does every empty field
before it,
back to the last one with something in it.
The check said three is not five,
and threw the line away.
The data was correct.
The supplier was correct.
The tool quietly decided
that the end of the line
did not count.
Pass a negative limit.
split with the separator and minus one
keeps every field,
empty or not.
And remember it takes a regular expression,
so the pipe needs escaping,
or it splits between every character.
Better still,
stop splitting delimited files by hand.
Use a real parser,
because the next surprise
is a pipe inside a quoted name,
and split has never heard of quotes.
Then make the skip loud.
A rejected line
should be counted,
sampled into the log
with its raw text,
and alarmed on
when the count moves.
A few hundred a night
was the bug,
reporting itself every morning
to nobody.
And write the test
with the boring row.
All fields filled
is the row everybody tests.
The row that ends in nothing
is the one
the supplier sends most.
Empty is a value.
Make sure your code
can count it.
– Serguey Asael Shinder
Top comments (0)