dict.get(key, [None])[0] is not safe, and our green test suite agreed with us

Quick answer: d.get("values", [None])[0] looks like a safe default and isn't. The fallback only applies when the key is missing. If the key is present and holds an empty list, you get [][0] and an IndexError. Our tests passed, our cloud smoke test passed, and the scraper crashed on its very first real run at production settings.

Here is how it got that far, because the interesting part isn't the idiom — it's why three separate gates were green.

The line

group = (taxonomy.get("group") or {}).get("search_values", [None])[0]

Reasonable-looking. taxonomy.get("group") or {} handles a missing or null group. The [None] default handles a missing search_values. Then [0] takes the first item, which is None in the fallback case. Three defensive moves in one line.

It handles every case except the one the API actually produces:

{ "group": { "search_values": [] } }

The key is there. The default never fires. [][0] raises.

The mental model that produces this bug is that .get(k, default) is a general "give me something sensible" operator. It isn't — it is strictly about key presence. An empty list is a present value, and a present value is returned as-is no matter how unhelpful it is.

Why the tests didn't catch it

The parser had unit tests, including a malformed-input test. They used fixtures captured from a real page. Every product on that page happened to be filed under a group — which is the normal case, and exactly why the fixture looked complete.

A fixture is a photograph of one moment of an API's output. It contains the shapes that existed when you took it, and silently asserts that no other shapes exist.

Why the smoke test didn't catch it either

This is the part worth stealing.

Our cloud smoke test clamps every max* input to 3, so a pre-release check costs cents instead of dollars. Sensible. But it means the run fetched three deals, all of which had groups, and reported SUCCEEDED with rows and correct billing. Green.

The shipped default asks for 20 results per query across three queries. Somewhere in those ~47 deals sat exactly one with an empty facet. So the smoke test and the product were testing different programs, and only the cheap one was ever run.

We'd just added a guard for exactly this, which is the only reason the bug was caught: if the harness clamped an input below what the author's own defaults ask for, it now prints DEPTH UNPROVEN and says to fire one deliberate run at the real values. We did. It crashed with IndexError, exit code 91.

Without that run, this ships and is broken for 100% of users on the default settings, while every dashboard shows green.

The second bug, hiding under the first

The parser wrapped its mapping in a handler so that one malformed record skips instead of killing the batch:

except (KeyError, TypeError, ValueError, ValidationError):
    logger.warning("deal %s: skipping", deal_id)
    return None

IndexError is not in that tuple. So the one bad record didn't skip — it propagated out of the parser, out of the scraper loop, and terminated the entire run.

That is the more expensive defect of the two. A single unlucky record taking down a whole batch is, in our experience, the number-one cause of low-success-rate scrapers: the failure looks like "the site blocked us" and is actually "record 31 had an empty array."

Three things worth taking away

1. .get(k, default) is about key presence, not value usefulness. For "first item or nothing", write it explicitly:

   values = (taxonomy.get(key) or {}).get("search_values") or []
   return values[0] if values else None

The or [] catches missing, None, and empty in one move.

1. Your fault-isolation handler is only as good as its exception tuple. Enumerate what your mapping code can actually raise. IndexError belongs there in any code that indexes.
2. If your pre-release check runs smaller inputs than your defaults, it is not testing your product. Either make the check match the shipped defaults, or know precisely which code paths you have chosen not to exercise — and run one deliberate deep pass before you ship.

---

🍋 AppSumo Deals Scraper pulls SaaS lifetime deals into typed rows — name, price, original price, category, deal status, claim percentage and review summary — straight out of the page's own JSON. $5.20 per 1,000 results, and you only pay for rows that land.

FAQ

Why does .get("key", [None])[0] raise IndexError?
Because the default only applies when the key is absent. A key present with an empty list returns that empty list, and indexing it raises.

What's a safer pattern?
values = d.get("key") or [] then values[0] if values else None. The or [] collapses missing, None and empty into one case.

Should IndexError be caught in a per-record handler?
If the record-mapping code indexes anything, yes. Otherwise one malformed record escapes the handler and kills the entire batch.

How do I stop cheap smoke tests from hiding bugs like this?
Compare what the harness ran against what your defaults ask for. If the harness clamped the value down, your deep code paths are untested — fire one run at the real defaults before release.