Wildcard and regex patterns share a specific failure mode: they almost always match more, or less, than the person writing them assumed. The pattern reads correctly in your head. The engine reading it doesn't share your assumptions, and it will match exactly what you wrote, not what you meant.
Step 1: Write down what you actually want matched and excluded
Before touching the pattern itself, list concrete example strings, some that should match, some that shouldn't. This sounds obvious and gets skipped constantly. A pattern meant to catch query-string URLs (?sort=price) needs explicit test cases for URLs with a literal ? in a different context, like an encoded character in a slug, so you can confirm the pattern doesn't also catch those.
Write this list down somewhere durable, not just in your head while you iterate. A plain text file with one example per line, prefixed with MATCH or NO-MATCH, takes two minutes to create and becomes the seed for the automated test suite described further down. Skipping this step is how a pattern ships that technically works on the three examples the author tried and fails on the fourth one a user actually sends.
Step 2: Test the narrowest version first
Start with the most specific pattern that covers your primary case, then broaden it deliberately, testing after each change. Going the other direction, starting broad and trying to add exceptions, tends to produce patterns nobody can fully reason about six months later, because each exception was bolted on to fix one specific test case without re-checking the others.
The narrow-first approach also makes code review easier. A reviewer can look at a tight pattern and confirm it matches the stated intent in a few seconds. A pattern that's been broadened repeatedly with negative lookaheads bolted on for each edge case usually takes longer to verify than to rewrite from scratch, and reviewers often just wave it through instead, which is how subtle bugs survive review.
Step 3: Check greedy vs. lazy quantifiers explicitly
.* and .*? (greedy vs. lazy) produce dramatically different results on strings with repeated delimiters, and this is one of the most common sources of "why did this match the whole string instead of just the part between the first two quotes" bugs. Test any pattern with a quantifier against a string containing the delimiter more than once, not just the simple single-occurrence case that usually gets written first.
A concrete example: matching text inside quotes with ".*" against "first" and "second" greedily matches from the first quote to the last, swallowing both pairs and the word "and" in between. Switching to ".*?" stops at the first closing quote instead. Neither choice is universally correct, it depends on what you're parsing, but you have to test both against a multi-occurrence string to know which one you actually need.
Step 4: Run it against real production data, not synthetic examples
Synthetic test strings are written by the same person who wrote the pattern, which means they tend to avoid the exact edge cases that would actually break it. Pulling a sample of real URLs, log lines, or user input and running the pattern against all of them surfaces mismatches a clean synthetic test suite won't.
If you don't have production data handy, the next best option is user-submitted content that's already messy by nature: free-text form fields, uploaded filenames, or copy-pasted URLs with tracking parameters attached. These sources reliably contain the stray whitespace, mixed casing, and unexpected punctuation that a hand-written test list tends to omit.
Step 5: Confirm the match groups, not just whether it matched
A pattern can technically "match" while capturing the wrong groups, which is often worse than not matching at all, since downstream code will happily extract the wrong substring and keep going. Always inspect the actual captured groups during testing, not just the pass/fail result.
This matters most with nested or optional groups, where a group can be present in the match but empty, or absent entirely and return null depending on the engine. Code that assumes a captured group is always a non-empty string will throw or silently produce garbage the first time that assumption breaks, so test the group values explicitly, not just whether the overall pattern returned true.
Step 6: Test for catastrophic backtracking before the pattern touches user input
If any part of a pattern accepts user-supplied text and runs it through a regex with nested quantifiers, like (a+)+ or (.*)*, you need to test what happens on adversarial input before shipping, not after a server hangs in production. This failure mode is called catastrophic backtracking, or ReDoS (regular expression denial of service): certain input strings cause the engine to try an exponential number of ways to fail a match, and a string of just a few dozen characters can pin a CPU core for seconds or minutes.
The test for this is straightforward: take your pattern and feed it a string designed to almost match but not quite, repeated enough times to be interesting, for example thirty repetitions of a character followed by one character that breaks the match. Time how long the pattern takes to run against it. If the time grows sharply as you add a few more repetitions rather than staying roughly flat, the pattern has backtracking risk and needs to be rewritten, usually by making quantifiers more specific or using atomic groups or possessive quantifiers if your engine supports them. This test costs a few minutes and catches a class of bug that's genuinely dangerous if the pattern ever runs against anything a user controls, like a search box, a filter, or a custom validation rule.
Step 7: Check the pattern against the actual engine flavor it will run in
Not every regex engine agrees on syntax. PCRE, POSIX, and JavaScript's engine all diverge on details like lookbehind support, possessive quantifiers, and how character classes handle Unicode, and a pattern that works when you test it in one context can behave differently, or fail to compile at all, in another. This matters most when the same pattern is duplicated across a frontend written in JavaScript and a backend written in Python or another language, which is common for form validation that's checked client-side and re-checked server-side.
If your pattern needs to work identically in both places, test it in both places, not just the one you happened to write it in first. A quick way to catch this early is running the exact same pattern and test strings through each target language's regex implementation and diffing the results. A mismatch means the pattern needs a compatible rewrite, or the two checks need to tolerate slightly different behavior on purpose. Python's official re module documentation is a solid reference for exactly which syntax it supports and where it differs from PCRE, since some patterns that look identical on paper parse differently between the two.
Step 8: Turn your test cases into an automated suite that runs in CI
Manual testing catches problems the day you write the pattern. It does nothing for the day six months later when someone tweaks the pattern to fix an unrelated bug and breaks a case nobody remembered to check by hand. The fix is turning the match and non-match examples from Step 1 into an actual automated test, however small, that runs whenever the pattern changes.
This doesn't need to be elaborate. A simple test file with an array of { input, shouldMatch } pairs and a loop that asserts each one, wired into whatever test runner your project already uses, is enough. The goal isn't full coverage of every conceivable string, it's a tripwire: if a future edit to the pattern breaks a case that used to pass, the test fails in CI before it ships, instead of surfacing as a support ticket after a user hits it in production. Patterns that guard anything security-relevant, like input sanitization or auth token formats, benefit from this most, since a silent regression there is the kind of bug that doesn't announce itself.
Why this discipline matters beyond regex specifically
This same testing discipline applies almost identically to robots.txt wildcard rules, which use a simpler pattern syntax but fail in the same way: a Disallow: /*? rule looks like it targets query strings specifically, and does, but also matches any path containing a literal ? anywhere, which can catch URLs nobody intended to block. I broke down that exact failure mode and several others in a longer piece here, if you're dealing with crawl directives specifically rather than application code.
I use EvvyTools' Regex Tester for quick iteration since it shows capture groups and match boundaries live as you edit, which speeds up step 5 considerably compared to running a script and printing results manually.
For deeper reference, MDN's regular expressions guide is the best general-purpose explainer, and the regexp info reference site documents cross-engine differences (PCRE vs. JavaScript vs. POSIX) that matter if your pattern needs to work identically across a frontend and a backend written in different languages.
Top comments (0)