I ran my file-type detector over 8,900 real files. It was wrong in ways no test suite would have told me

My file identifier had 260 passing tests and a hand-made benchmark that scored 100%. Then I pointed it at the files already sitting on my PC, and it flagged thirteen files in C:\Windows\System32 as dangerous, called ninety Tcl message files a mismatch, and stopped dead on one .ini file for several minutes.

None of that showed up in the tests, because the tests were made of files I had written to look like the formats I expected. The files on a real Windows machine were written by other people, for other reasons, and they do not care what my extension dictionary says.

This post is about the benchmark that found those problems, what they turned out to be, and the handful of rules that fixed them. It is a follow-up to Catching "invoice.pdf.exe" before you open it, which explained how the detector works. You do not need to have read it. The one-line version: the app reads the first 64 KB of a file, matches signatures, and for text files scores a few hundred regex "features" per language or format.

TL;DR

• Copy real files with their extension removed, ask the detector for the original extension, and count. That single loop, run over ~8,900 files from Windows, Program Files, Python and my dev folders, found everything below.
• Accuracy on extensions the app knows went from 69.9% to 90.1%; on the ~130 extensions ordinary users meet, from 76.5% to 93.0% (that number is "matched the original extension, allowing the aliases described in the harness section", a stricter test than identifying the format; format accuracy is in the results table). Files reported as "unidentified" dropped from 5,408 to 3,533. False alarms on real files went from 105 to 0.
• The false alarms were not bugs in the matcher. They were wrong assumptions in the dictionary: .rs is not always Rust, .msg is not always Outlook, .CSS is sometimes HTML, .so sometimes holds a Windows DLL.
• One Makefile rule, (\w[\w.\-/]*\s*)*$, backtracked exponentially across lines and hung the detector on Appraiser_Data.ini. Timing every file then found four more rules, all starting with \w+, that went quadratic on a single whitespace-free line. Two shapes worth a second look: nested quantifiers with a newline-eating \s, and an unanchored \w+ at the start.
• The most effective scoring change was one sentence: "high confidence" requires two different features to match. Repeating one feature, however many times, never counts as certainty.

The app is closed source (it ships on the Microsoft Store), so there is no repository to link. Every snippet below is lifted from the real code and runs as shown.

---

The harness: 100 lines that replaced my opinions

The idea is embarrassingly simple. Walk a set of directories, pick files, and for each one do two things:

1. Run the detector on the file as it is, and record the verdict (match / mismatch / danger / unidentified).
2. Copy the file to a temp folder with the extension removed, run the detector on the copy, and record which extension it guesses. The original extension is the answer key.

The only non-obvious part is sampling. A Windows box has 9,000 .js files and 8,000 .dll files; without a cap you measure JavaScript and nothing else. So each root gets a per-extension cap, and there is a global cap on top:

ROOTS = [
    (r"C:\Windows\System32", 3, 30),          # (root, max depth, files per extension)
    (r"C:\Windows\Cursors", 1, 40),
    (r"C:\Windows\Fonts", 1, 40),
    (r"C:\Users\me\dev", 7, 25),
    (r"C:\Users\me\AppData\Local\Programs\Python", 7, 30),
    (r"C:\Program Files\Git", 6, 20),
    (r"C:\Program Files\Microsoft Office", 6, 25),
    (r"C:\Program Files (x86)\Windows Kits", 7, 20),
    (r"C:\Users\me\Downloads", 4, 30),
    # ... 35 roots in total
]
GLOBAL_CAP = 90          # per extension, across all roots

def pick():
    global_count, picked = {}, []
    for root, depth, cap in ROOTS:
        local = {}
        for p in walk(root, depth):
            ext = os.path.splitext(p)[1].lstrip(".").lower()
            if local.get(ext, 0) >= cap or global_count.get(ext, 0) >= GLOBAL_CAP:
                continue
            local[ext] = local.get(ext, 0) + 1
            global_count[ext] = global_count.get(ext, 0) + 1
            picked.append(p)
    return picked

That yields about 8,900 files across roughly 600 distinct extensions. The scan itself:

for i, p in enumerate(files):
    rec = {"path": p}
    rec["with_ext"] = slim(inspect_file(p))            # verdict as-is
    if os.path.getsize(p) <= 12 * 1024 * 1024:
        dst = os.path.join(tmp, f"f{i}")               # no extension
        shutil.copyfile(p, dst)
        rec["noext"] = slim(_inspect_file_core(dst))   # what does it guess?
        os.remove(dst)
    out.write(json.dumps(rec, ensure_ascii=False) + "\n")

One JSON line per file, nine minutes per run. Everything else is a second script that groups the lines by extension and prints three things: every file that got a warning or danger verdict, every extension where the content came back unidentified, and the hit rate of the extension guess.

The scoring needs an alias table, or you measure the wrong thing. A .jpeg file guessed as .jpg is right. A .manifest guessed as .xml is right, because it is XML and nothing in the bytes says "manifest". A .dat guessed as anything is unscorable, because .dat means "whatever this app felt like writing", so those are excluded. I ended up with about 250 alias entries, and I kept two totals: "all extensions in the dictionary" and "the ~130 extensions a normal person meets", because the machine is full of developer XML that inflates or deflates the number depending on the day.

That is the whole tool. What it found is the interesting part.

---

What broke, in order of embarrassment

Thirteen dangerous files in System32

The detector's strongest verdict is "danger": the extension says document or text, the contents are an executable. It fired on these:

C:\Windows\System32\cero.rs      danger | dll
C:\Windows\System32\esrb.rs      danger | dll
C:\Windows\System32\pegi.rs      danger | dll
C:\Windows\System32\usk.rs       danger | dll
... 13 files

.rs was in my dictionary as "Rust source". These are Windows game-rating resource files: resource-only DLLs with a .rs extension, present on every Windows install, signed in the system catalog. A user who dropped their System32 folder on the app would have seen thirteen red badges. My 2,400-file sweep from the previous version had not touched System32.

Ninety mismatches that were all Tcl

...\tcl8.6\msgs\af.msg    warn: extension and contents do not match | text, kind=tcl
...\tcl8.6\msgs\ar.msg    warn | text, kind=tcl
... 90 files

.msg meant "Outlook message" in the dictionary. Tcl's message catalogs also use .msg, and any machine with Python installed has ninety of them. The text classifier correctly identified them as Tcl; the dictionary then declared a mismatch because Tcl is not an OLE compound file.

Extensions that mean two things

Once I started looking, this was a category, not an incident:

extension	dictionary said	also is
.rs	Rust source	Windows rating data (DLL)
.msg	Outlook message (OLE)	Tcl message catalog (text)
.ts	TypeScript	MPEG transport stream
.lib	static library (!<arch>)	COPYING.LIB, a license text
.pdb	debug symbols	Protein Data Bank text
.cer / .crt	DER certificate	PEM certificate (text)
.so	ELF shared object	a Windows DLL (Dart/Flutter native assets ship them)
.cs	C#	Tcl, in tix's preference files
.CSS	stylesheet	HTML with an inline <style> (Office ships 25 of these)

The fix was a flag on the dictionary entry, alt_text, meaning "this extension has a text variant; if the contents are text, trust the content classifier and do not call it a mismatch". .msg now shows "Tcl script" for the Tcl files and "Outlook message" for the OLE ones.

Extensions that mean nothing

.dat, .bin, .bak, .tmp, .pak, .cache: 300-odd files where the detector said "the signature does not match the dictionary; the file may be damaged". It was not damaged. It was a .dat file, and .dat has no signature to match. The wording was wrong, and the verdict ("unidentified" with a search button) was unhelpful.

These got a different flag, any. The message became: this kind of file has no fixed signature, so the contents cannot be verified; it was confirmed not to be an executable. When the contents can be identified (an XML .dat, a PNG .bin), the app now says so plainly and gives an OK verdict instead of "the extension might be a mistake". And when the contents are an executable, it is still danger, with one exception below.

Signed DLLs wearing data-file extensions

C:\Windows\System32\OLDFA39.tmp                     danger | dll   (signature: valid, Microsoft)
C:\Program Files\Microsoft Office\...\MSZIP.DIC      danger | dll   (valid, Microsoft)
C:\Program Files\Microsoft Office\...\OUTLFLTR.DAT   danger | dll   (valid, Microsoft)

Windows servicing renames old DLLs to OLDxxxx.tmp; Office keeps DLLs named .DIC and .DAT. All of them carry a valid Authenticode signature or are listed in the Windows system catalog. The app already verifies signatures (offline, via WinVerifyTrust with cache-only revocation), so the rule is: an executable in an unexpected extension is downgraded from danger to "caution: genuine Windows file" if it is catalog-listed, or validly signed and the extension is one of the any kind. Double extensions like invoice.pdf.exe stay danger no matter who signed them; a signature says who built it, not that the name is honest.

---

The regex that hung for ten minutes

The first full run stopped at file 421 and never continued. CPU at 100%, no I/O. The file was C:\Windows\System32\appraiser\Appraiser_Data.ini, 138 KB of this:

[LT_SdbVerDependentIndicatorTables]
IT_ALL_CompatIndicator_NA_SdbVer
IT_ALL_CompatIndicator_NA_DataExpDate
IT_ALL_CompatIndicator_NA_DataExpDateEpoch
...

The text classifier runs every feature regex of every language over the first 24,000 characters. Timing each one found the culprit in the Makefile rules, the one meant to match a dependency line like target: a.o b.o c.o:

r"^\w[\w.\-/]*:\s*(\w[\w.\-/]*\s*)*$"

Two problems compound. \s* matches newlines, so once a line contains a colon, the group happily continues onto the next line, and the next, through the whole 24 KB window. And (\w[\w.\-/]*\s*)* is the textbook catastrophic shape: the inner [\w.\-/]* and the next iteration's \w can split a word at every position, so when the final $ eventually fails (any = or ( will do it), the engine tries every split of every word on every line. With multiline mode, ^ also gives it a fresh start on every line.

The fix keeps the intent and removes the ambiguity:

r"^\w[\w.\-/]*:[ \t]*(\w[\w.\-/]*[ \t]+)*(\w[\w.\-/]*)?[ \t]*$"

Whitespace is now [ \t] only, so the match cannot leave the line, and every repeated element must be followed by at least one space, so there is exactly one way to split a sequence of words. Same file: 0.07 seconds.

Then I grepped the other 400 patterns for *)* and +)+ and \s*). That was the only nested quantifier.

It was not the end of the story, though. Measuring p95 and max runtime for this post turned up a file that took ten seconds: Office's OfficeSharedEntities.bin, which is one line of hash values joined by colons, no whitespace at all. This time the culprit was not nesting but patterns that begin with \w+, such as the TypeScript annotation rule \w+\s*:\s*(string|number|…). Against a single 24,000-character token the engine starts at every position, extends \w+ greedily to the end and backtracks, which is quadratic. Four rules had that shape (TypeScript types, Julia types, the CSS selector rule, the generic name(...); rule). Anchoring the start to a word boundary (\b) or to "line start or just after {, }, ;" took the file from 10 seconds to 0.08.

So, two things I now check for instead of one: nested quantifiers with a newline-eating \s, and patterns that start with \w+ or [^…]+ without their starting position pinned to a boundary. Neither is automatically a bug; both are where the quadratic cases turned out to live. The benchmark now fails if any file takes more than a second, and the test suite feeds the classifier a 24,000-character token with no whitespace.

---

The scoring changes that moved the number

The per-extension hit rate before and after, on the files the benchmark happened to contain:

extension	before	after	what was wrong
.h	28 / 90	88 / 90	C headers read as Markdown
.cs	2 / 20	10 / 20	modern C# not recognised (8 of the rest are Tcl files named .cs)
.vbs	0 / 32	27 / 32	VBScript read as VBA
.ini	57 / 90	88 / 90	INI read as TOML
.cfg	5 / 31	30 / 31	same
.tcl	58 / 90	88 / 90	Tcl read as Makefile or Markdown
.xml	56 / 84	79 / 84	XML with no <?xml declaration
.pem	0 / 34	35 / 35	no PEM detection at all
.css	42 / 90	62 / 90	minified CSS scored zero (the other 25 are HTML)
.js	59 / 90	66 / 90	minified bundles

Five rules explain most of it.

1. "High confidence" needs two different features

Each language is a list of (regex, weight, cap). A file's score for a language is the sum of weight × min(matches, cap). Before, a score of 7 with a 3-point lead over the runner-up meant "high", and high confidence is what the app shows as a firm answer.

Markdown had (r"^#{1,6} ", 3, 3): a heading is worth 3 points, counted up to three times. Any shell script, Perl module or C header with three # comment lines scored 9 points of Markdown from a single feature and was reported, with high confidence, as a Markdown document. The same shape produced "assembly" from C (int was in the mnemonic list), "VBA" from English prose (Range and With as bare words), and "Makefile" from anything indented with tabs.

The scorer now tracks how many distinct features matched:

if best >= 7 and best - second >= 3 and (n_features >= 2 or has_decisive):
    return kind, "high"
if best >= 4 and best > second:
    return kind, "mid"

has_decisive is true when a single feature with weight 8 or more matched: a shebang line, <?php, IDENTIFICATION DIVISION, -----BEGIN CERTIFICATE-----. Those are allowed to be certain on their own. Everything else has to corroborate. This one condition removed more false "high" verdicts than any other change, and it cost nothing on the true positives, because a real Python file has an import and a def, and a real Markdown file has a heading and a link or a code fence.

2. Derived languages may only score on what makes them different

VBA is modelled as "VBScript plus features unique to VBA": if any VBA-specific feature matches, VBA gets its own points plus VBScript's, so it beats the base. The list of VBA-specific features included Sub name(, Option Explicit and With. All three are perfectly normal VBScript. Result: 27 of the 32 .vbs files that ship with Windows were reported as Excel macros.

The corrected list is only things VBScript cannot contain: Dim x As Long (VBScript has no typed Dim), Range(, Cells(, ThisWorkbook., Application.ScreenUpdating. Same pattern for TypeScript over JavaScript (type annotations only), SCSS over CSS ($var: and @mixin, not &), C++ over C. The rule generalises: a derived class in a scoring hierarchy may only earn points for features the base class cannot have, otherwise it wins every tie by construction.

3. Generic shapes are worth one point, and line endings disqualify them

key: value is YAML. It is also every object literal in JavaScript and TypeScript, which is how vite.config.ts became "YAML settings". The YAML key rule now refuses lines that end in {, [, (, , or ;, and lines whose value itself contains : (that is a log line, not a mapping):

(r"^\s*[A-Za-z_][A-Za-z0-9_.\-]*:\s+(?![^\n]*: )[^\n{}\[\](),;]+$", 1, 4)

INI versus TOML was the same fight in the other direction. Both are key = value under [section]. The difference is typed values: TOML quotes strings and writes true, 123, [...]; INI writes bare text, usually with no spaces around =, and comments with ;. Each side now gets points only for its own dialect:

# TOML: typed values, spaces around =, [[arrays]]
(r"^\s*[A-Za-z0-9_.\-\"]+\s*=\s*(\"[^\"\n]*\"|'[^'\n]*'|\d[\d_.:TZ-]*|true|false|\[|\{)\s*(#.*)?$", 2, 4),
(r"^\s*[A-Za-z0-9_.\-\"]+ = ", 1, 4), (r"^\s*\[\[", 3, 1),
# INI: bare values, no spaces, ; comments
(r"^\s*[A-Za-z_][A-Za-z0-9_.\- ]*\s*=\s*[^\"'\[{\n][^\n]*$", 1, 5),
(r"^\s*[A-Za-z_][A-Za-z0-9_.\-]*=\S", 1, 4), (r"^\s*;", 2, 2),

Appraiser_Data.ini (Key=2700, no spaces, bare values) is now INI; Cargo.toml (name = "kantei") is TOML. The 25 .ini files that were TOML are gone.

4. Assembler mnemonics need operands

int, sub, or, and were in the list of mnemonics that score assembly when they start a line. int starts half the lines of a C file; sub starts every Perl function; or and and start sentences in every license text. Two-operand instructions now require a single-token operand followed by a comma, mov eax, 1, and int/sub are gone:

(r"^\s*(mov\w*|lea|cmp|xor|add|sub|and|or|shl|shr|test)\s+[%$\w\[][\w\[\]$%+*.:-]*\s*,\s*[%$\w\[-]", 2, 6),

The general lesson is the same as rule 3: a keyword that is also an English word is not a feature until you attach the syntax that only the language has.

5. Things the text stage was simply not doing

Not scoring mistakes, just gaps the sweep made visible:

• UTF-16 without a BOM. PowerShell 5.1's Out-File and several Windows tools write it. The old rule "a NUL byte in the first 4 KB means binary" sent 74 such files (15 of them .json) to "unidentified". The new check counts NUL at odd positions and printable ASCII at even positions over the first 512 bytes.
• XML and HTML without a declaration. <Project ToolsVersion=...>, <DSAKeyValue>, an HTML file that starts with a comment. Now: strip leading comments, and if the first tag has a matching closing tag or self-closes, it is XML; if it is one of html/head/body/div/script/meta..., HTML.
• CSV with quoted commas. The old check required every line to have the same count of commas. "Naha, Okinawa" broke it. Now the csv module parses the first twelve lines and the file is a table if 80% of rows have the same field count, at least two fields, and the cells are mostly non-empty. That last condition matters: C and TypeScript lines ending in ; looked like two-column semicolon-separated CSV until I required the cells to be filled.
• PEM. -----BEGIN CERTIFICATE----- anywhere in the first 8 KB (CA bundles start with comment lines), plus diff --git, WEBVTT and #EXTM3U.

---

The results, honestly

Same scoring, same files, before and after, split into the metrics people usually ask a classifier for. Where I did not measure something, the table says so.

metric	before (v1.9.0)	after (v1.10.0)	notes
format accuracy, signature-based formats	78.3%	96.2%	real format recovered from the extension-less copy; n = 2,790
format accuracy, text classification	61.0%	83.6%	same; n = 2,621. Almost all of this release's work landed here
extension recovery	69.9% / 76.5%	90.1% / 93.0%	all known extensions / the ~130 everyday ones. The denominator, 5,411, is the known-extension files minus the meaningless extensions (.dat etc.) and files over 12 MB; the "unidentified" count is taken over all files, so the two are different populations and do not add up to the total
false positive rate (benign file marked danger or mismatch)	1.2% (105 / 8,918)	0% (0 / 8,920)	test dummies excluded; the two intended cautions (OEM URL shortcuts) excluded on both sides
false negative rate (malicious file missed)	not measured	not measured	no malware corpus was swept. The crafted disguises in the regression suites (double extensions, %PDF- in a DOS stub, an MSI inside .docx, .lnk launching hidden PowerShell; about 30 cases) still all fire, before and after
runtime p95 / max per file	not measured / one file killed after 10+ minutes	0.07 s / 0.23 s	re-timed over the same 8,900 files (p50 is 0.001 s). Before the \w+ anchoring fix the max was 9.9 s (see the regex section)
unidentified (all files / known extensions)	60.6% / 42.1%	39.6% / 10.5%	most of the all-files remainder is .dat, .pak, .bin and other unverifiable extensions

Extension recovery is a stricter test than format accuracy: the app can identify the format correctly and still miss the extension, because one format maps to many extensions (.jpg/.jpeg, or the dozens of developer extensions that are all plain XML). Read 90.1% as "matched the original extension, allowing the aliases described in the harness section", not as "identified the format".

Denominators differ from the headline "about 8,900" because the rates exclude the six test dummies and about 100 files Windows would not let me read; the two runs differ by two files because temporary files came and went between them.

The false-negative row says "not measured" rather than nothing because this post is about the false-alarm and miss side. For the malicious side, the limits section of the previous post (static inspection does not reach fileless attacks) applies unchanged.

What the remaining 10% of extension recovery is: .dat/.bin/.pak and friends with no signature (excluded from the percentage, but still "unidentified" in the count), tiny JavaScript build artefacts with nothing to grab, developer XML that is correctly identified as XML but whose extension nobody could guess from the bytes, and a few genuinely mislabelled files that the benchmark counts as misses when the app is arguably right.

Bias, stated up front. This is one Windows 11 PC belonging to a developer. It is heavy on Python's standard library, Visual Studio build tools and Office, light on media and design files, and has no Java, Go, PHP or Ruby source at all (those languages are covered by synthetic tests only). Someone else's machine will find a different set of .rs-style surprises. That is the point of the tool, not a caveat about it: the harness is 150 lines and anyone can run it against their own disk.

Why not libmagic? It would have caught some of the signature gaps (CUR, CHM, EVTX and about fifty other formats I added in this release, all of which file has known for decades). It would not have helped with the part that actually failed, which is the dictionary of what an extension is allowed to contain and the text-language scoring. Both of those are the product; the signature list is the easy part. Polyglot files, which satisfy two signatures at once, are handled the way the previous post described: the executable check runs before any "search past the first byte" rule, and the app does not try to be cleverer than that.

---

Takeaways

• A test suite made of files you wrote checks that your rules do what you meant. A pile of files other people wrote checks whether what you meant was true. Both are cheap; only the second one found anything this time.
• Extensions are not a function from name to format. Keep a flag for "has a text twin" and a flag for "means nothing", and let the contents win.
• In a rules-based classifier, count distinct evidence. One feature repeated is a hint, not a conclusion, no matter how many points it adds up to.
• Derived categories score only on what the base cannot have.
• Words that are also English words are not features. Attach the syntax.
• A regex with a nested quantifier, or one that starts with an unanchored \w+, is not wrong by itself, but it deserves a second look: the right input can make it quadratic or worse. Timing every rule on real files is cheap insurance.

The updated detector is version 1.10.0 of File Type & Safety Checker on the Microsoft Store (free, fully offline; nothing about your files leaves your PC).

https://apps.microsoft.com/detail/9PKG5KT1WXR8?hl=en-us&gl=US

If you run the same kind of sweep on your own machine and find an extension that means something I have not listed, I would like to hear about it.

---

About the author

Okinawa Software Lab. I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish file and PDF utilities on the Microsoft Store that follow the same principle: everything happens on your own PC.

• Website: https://okinawasoftwarelab.com/en/