DEV Community: geodesic partners

Why softmax returns NaN at a logit of 710, and the one-line fix

geodesic partners — Tue, 03 Nov 2026 09:05:15 +0000

If a training run has ever turned its loss into nan with no warning worth reading, there's a fair chance softmax did it. The formula on the whiteboard is fine. The floating-point version of it isn't, and the fix is one line.

Where it breaks

A float64 holds numbers up to about $1.8 \times 1 0^{308}$ . Since $e^{709.8}$ is already that large, np.exp(710.0) returns inf with a warning. At the other end, np.exp(-746.0) underflows to exactly zero. Logits from a trained network rarely reach those values, but logits during training, in a badly scaled layer, or from a model that has become very confident, do. When they do, the textbook formula

p = np.exp(z) / np.exp(z).sum()

returns inf / inf = nan, and a single nan spreads through every weight it touches on the next update. The loss doesn't creep up first. It just disappears.

The shift

Softmax doesn't change when you add the same constant to every logit, because the constant factors out of the numerator and the denominator alike:

softmax (z - c)_{i} = \frac{e ^{z_{i} - c}}{\sum _{j} e ^{z_{j} - c}} = \frac{e ^{- c} e ^{z_{i}}}{e ^{- c} \sum _{j} e ^{z_{j}}} = softmax (z)_{i}

Choose $c = max (z)$ . Now the largest exponent is $e^{0} = 1$ , so nothing overflows, and the denominator is at least 1, so nothing divides by zero. Small entries may still underflow to zero, but that's the correct answer to sixteen digits: a probability of $e^{- 800}$ is zero for every practical purpose.

Log-sum-exp

Cross-entropy needs log softmax, not softmax. Computing softmax first and then taking the log throws away exactly the information you need: a probability that underflowed to 0 becomes $lo g 0 = - \infty$ . Work in log space throughout instead:

lo g softmax (z)_{i} = z_{i} - lo g j \sum e^{z_{j}} = z_{i} - m - lo g j \sum e^{z_{j} - m}, m = max (z)

The second term is the log-sum-exp. It's a smooth version of the maximum: it's never less than $max (z)$ and never more than $max (z) + lo g n$ . Written with the shift, it's exact and finite for any finite input.

def logsumexp(z, axis=-1):
    m = np.max(z, axis=axis, keepdims=True)
    return m + np.log(np.sum(np.exp(z - m), axis=axis, keepdims=True))

Sigmoid and binary cross-entropy

The sigmoid has the same problem in a different shape. 1 / (1 + np.exp(-z)) is fine for large positive z and overflows inside the exponential for large negative z. The usual repair is to branch on the sign, so that the exponential only ever sees a non-positive argument. For the loss itself there is a closed form that needs no branch at all, and it's the one PyTorch's BCEWithLogitsLoss uses; deriving it's one of the problems below.

If this comes up in an interview

Say early that the textbook formula isn't what runs in production, and prove the shift changes nothing in one line. Move to log space before anyone asks, since a probability of $1 0^{- 400}$ still has a perfectly finite logarithm. And if you can say without being prompted that the gradient of softmax with cross-entropy is just $p - y$ , I'd rate that above the code.

Common mistakes

Subtracting the mean instead of the maximum. The mean doesn't bound the exponent.
Shifting a batch by one global maximum rather than row by row. One very large logit in row 3 then underflows every entry of row 7.
Adding a small epsilon inside the log, np.log(p + 1e-12). It hides the problem, biases the loss, and caps the gradient at the wrong value.

First published here: https://geodesicpartners.co.uk/ml-numerical-stability

Two pointers: stop testing pairs and start ruling values out

geodesic partners — Mon, 02 Nov 2026 09:13:01 +0000

You're given a sorted list of prices and asked whether any two add up to a budget.

for i in range(n):
    for j in range(i + 1, n):
        if a[i] + a[j] == target:
            return (i, j)

Correct, and instant on the ten-element example. Then the interviewer mentions that the list has 100,000 prices and the service answers a thousand of these queries a second.

How bad "every pair" gets

There are n(n−1)/2 pairs.

Array size	Pairs	At 10⁸ a second
100	4,950	instant
10,000	about 50 million	about half a second
100,000	about 5 billion	about a minute
1,000,000	about 500 billion	about an hour and a half

Ten times the input, a hundred times the work. A faster machine won't save you.

The hint in the question

The list is sorted, and the brute force never uses it. It checks 2 + 15 as carefully as 7 + 11, although the order already tells you which pairs are worth looking at.

When a question gives you a property and your solution ignores it, that property is usually the solution. Sorted input, a small alphabet, values under 100, "each element appears at most twice": those are usually hints.

One pass

Put a pointer at each end:

def pair_sum(a, target):
    lo, hi = 0, len(a) - 1
    while lo < hi:
        s = a[lo] + a[hi]
        if s == target:
            return (lo, hi)
        if s > target:
            hi -= 1      # a[hi] is too big for every partner left
        else:
            lo += 1      # a[lo] is too small for every partner left
    return None

On [1, 3, 4, 5, 7, 11] with a target of 10:

Left	Right	Sum	Conclusion
1	11	12	11 with the smallest value is already too big, so 11 is in no pair. Drop it
1	7	8	1 with the largest value left is too small, so 1 is in no pair. Drop it
3	7	10	found

Three sums instead of fifteen pairs.

The brute force keeps asking "is this pair the answer?". The scan asks something better: can this value be in any answer at all? If one comparison can rule a value out for good, try a single pass before anything cleverer.

If you'd rather watch the pointers move, the original has a stepper: https://geodesicpartners.co.uk/coding-two-pointers-why

The sliding window trick, and the one input that breaks it

geodesic partners — Sun, 01 Nov 2026 18:55:46 +0000

Most of us learned the sliding window as a recipe: grow the right edge, shrink the left when things go bad. The recipe's fine. The trouble starts when you don't know why it works, because then you'll happily run it on an input where it's wrong and never notice.

The slow version

Find the longest run of consecutive entries with sum at most K, all numbers positive. The first thing anyone writes:

best = 0
for i in range(n):
    total = 0
    for j in range(i, n):
        total += a[j]
        if total <= K:
            best = max(best, j - i + 1)

That visits about n²/2 pairs. Fine for a thousand numbers, about a minute for 100,000, well over an hour for a million.

Most of those runs were dead on arrival. If a run starting at i already sums past K, making it longer can't fix it, but the inner loop keeps going to the end anyway.

Why the window can skip them

Positive numbers give you two facts. Adding an element on the right can only push the sum up. Removing one on the left can only bring it down.

So if [lo, hi] is over budget, every longer run containing it is over budget too, and the only repair is to move lo right. Once lo has passed an index, no later window wants it back: a window starting further left would contain one you already know is too big.

Both edges only move forward. Each index enters once and leaves at most once, so the whole scan is at most 2n steps.

def longest_run(a, K):
    lo = total = best = 0
    for hi, x in enumerate(a):
        total += x
        while total > K:
            total -= a[lo]
            lo += 1
        best = max(best, hi - lo + 1)
    return best

Here it is on a = [3, 1, 2, 1, 4, 1, 1] with K = 5:

Add	Window	Sum	What happens	Longest
3	[3]	3	fits	1
1	[3, 1]	4	fits	2
2	[3, 1, 2]	6	too big: drop 3, sum 3	2
1	[1, 2, 1]	4	fits	3
4	[1, 2, 1, 4]	8	drop 1 and 2, sum 5	3
1	[1, 4, 1]	6	drop 1, sum 5	3
1	[4, 1, 1]	6	drop 4, sum 2	3

The answer is 3, and the left edge never moved backwards.

The input that breaks it

Allow negative numbers. Shortest run with sum at least 8 in [1, -5, 4, 4]: the answer is [4, 4], length 2.

A window that grows until it reaches 8 adds 1, then −5, then 4, then 4. The running total peaks at 4, never reaches 8, and the window reports that no such run exists. So you get a wrong answer, and nothing tells you.

The second fact is what broke. Dropping the −5 from the left would have raised the sum, so "shrink when invalid, grow when valid" no longer points the right way. That version needs prefix sums and a monotonic deque of candidate starts.

So before I reach for a window I ask one thing: if this window is invalid, is every larger one invalid too? If the answer's no, I use something else.

There's a longer version with a step-through of the window moving here: https://geodesicpartners.co.uk/coding-sliding-window-why

I put my whole coding interview course online: 8 patterns, 24 problems, 6 languages

geodesic partners — Thu, 01 Oct 2026 17:36:24 +0000

Most interview prep is a list of five hundred problems. I think that's the wrong unit to study. Interview questions are built from a small number of patterns, and once you can recognise the pattern and write its template from memory, a new problem mostly comes down to one decision: what to key on, what to keep in the window, what the state of the table is.

So the course is organised by pattern, in this order:

Arrays and hashing
Two pointers
Sliding window
Binary search
Dynamic programming
Stack
Heap (priority queue)
Backtracking

Each module starts with a lecture on why the obvious solution is too slow and what the pattern replaces it with, including an example you can step through one move at a time. Then comes the template, and then the problems, 24 in all, easy to hard. Every problem has an editor in the page, so you can try it before reading anything, and a worked solution in Python, JavaScript, C++, Java, Rust and Haskell. The solutions all run against the same tests, so none of them is a sketch.

A taste of what the lectures focus on

Sliding window is usually taught as a recipe: grow the right edge, shrink the left when the window goes bad. The part worth understanding is why that's allowed. With positive numbers, if a window's sum is over budget, every larger window containing it is over budget too, so the left edge never has to move back.

Allow a negative number and that stops being true. Look for the shortest run with sum at least 8 in [1, -5, 4, 4]: the window adds everything, peaks at a running total of 4, and reports that no such run exists, while [4, 4] is sitting right there. Knowing when a pattern breaks is what interviewers are actually checking.

The timed interview

When you've done a few modules, there's a timed mock interview. The time you get depends on the problem's difficulty, runs and submissions are limited, hints cost you marks, and at the end a report shows where the marks went.