first("abc", typemax(Int)) gives you "abc". first("é", typemax(Int)) gave you "".
Same function, same "give me everything" number, and one accented letter turned the answer into an empty string. I fixed it in JuliaLang/julia#64370, and the cause is a classic that's easy to miss.
The setup
Julia is a programming language that's big in science and data work. first(s, n) returns the first n characters of a string, or the whole string if n is larger. typemax(Int) is the largest whole number an Int can hold, so people pass it to mean "no limit".
MilesCranmerBot reported the bug in issue #64143, and already pointed at a helper called nextind.
Bytes are not letters
Julia counts string positions in bytes. a is one byte. é is two. So "é" is one character sitting in two positions.
first asks nextind "where does the nth character land?", walking forward byte by byte and ticking n down at every new character. When it hits the end with steps left over, it returns i + n, the end position plus the leftover. Julia's docs say that's intended.
Where the number wrapped
For "é" the walk is: byte 1 starts a character, so n goes down by one. Byte 2 is the second half of é, so nothing happens. End of string. Now it adds 2 and typemax(Int) - 1.
That's one more than the largest Int. Julia's Int is a fixed 64 bits, so instead of growing, the value wraps around to the most negative number. That's an integer overflow:
julia> nextind("é", 0, typemax(Int))
-9223372036854775808
first then takes the smaller of "end of string" and that, gets a negative position, and cuts out an empty range.
Why did "abc" survive? Every letter is one byte, so the sum lands exactly on typemax(Int) and never goes over. Pure luck.
The one line fix
I made nextind saturate, which means it stops at the largest value instead of wrapping:
# saturate instead of overflowing for huge `n` such as `typemax(Int)`
return n > typemax(Int) - i ? typemax(Int) : i + n
The comparison is flipped on purpose: computing i + n to check it would overflow in the check itself.
Fixing it inside nextind rather than in first mattered. Iterators.partition, which splits a string into chunks, uses the same pattern and was throwing a BoundsError (an "index out of range" error) with that same negative number. One line fixed both. I added tests for nextind, first, last and partition. The issue also mentioned chop; that's only fixed for plain String input, since a SubString overflows somewhere else, so I flagged it in the PR as a follow up.
vtjnash approved it and shared some history on how this "past the end" behaviour came to be, and IanButterworth merged it.
The lesson
Using the biggest number as "infinity" is handy, but it's still a number. The moment anything gets added to it, you're one step from wrapping. When you see max + something, check the top.
Have you ever been bitten by a "max value means no limit" trick, in Julia or any other language? How did it show up?
The full story, with all the code, is on my blog: https://ashokramsi.hashnode.dev/julia-nextind-saturate
Top comments (0)