DEV Community: CopperSunDev

Will Your AI Write A Regex That Hangs Your Server?

CopperSunDev — Fri, 04 Sep 2026 16:00:59 +0000

A regex your AI wrote in half a second can put your server on the floor for 27 minutes. That's not hyperbole: it's what happened to Cloudflare in July 2019, and the mechanism behind it, catastrophic backtracking, has landed real CVEs in ajv and minimatch within the last year. CWE-1333 gives the bug a name. The rest of this post gives it a shape you can spot before it ships.

Ask an AI assistant for a validation regex and it will almost always produce something that works on the examples you gave it. Nothing about that request asks the model to think about the worst input a stranger could send. That gap, between correctness on a sample and safety against an adversary, is where ReDoS lives.

What Catastrophic Backtracking Actually Does

BrassCoders treats catastrophic backtracking as a distinct, adversarial failure mode, not a slow-code problem you'd catch by profiling under normal load. Most regex engines, including the ones built into Python, JavaScript, and Java, match by backtracking: when a pattern fails at one point, the engine rewinds and tries the next possible path through the pattern. A regex with nested quantifiers gives the engine an exploding number of equivalent paths to try before it can conclude a string doesn't match.

Take the classic bad pattern (a+)+$. Against the string aaaaaaaaaaaaaaaaaaaaaaaa!, the trailing ! guarantees the whole thing fails to match. But there are many ways to partition 24 a's among the inner and outer + quantifiers, and the engine tries a large share of them before giving up. Add one more a to the input and the work roughly doubles. That's the exponential curve hiding inside a four-character regex.

OWASP's ReDoS reference calls this shape an evil regex: a group with repetition inside it, where that inner group also repeats or offers overlapping alternatives. (a+)+$, ([a-zA-Z]+)*$, and (a|aa)+$ all fit the description. None of them look dangerous in a code review. All of them are.

CWE-1333, MITRE's formal entry for this weakness, defines it plainly: a regular expression whose worst-case computational complexity is inefficient, possibly exponential, in the length of the input. The consequence it lists isn't data exposure or privilege escalation. It's availability, the same category as a crashed process or a full disk, and that's the detail teams tend to underweight when they triage a ReDoS finding against a severity rubric built around leaked data.

The 27 Minutes Cloudflare Lost To One Regex Line

BrassCoders points to the Cloudflare outage as the proof this isn't a lab exercise. On July 2, 2019, Cloudflare pushed a new WAF rule to its entire global edge network at once, skipping the gradual rollout it normally uses. The rule's regex hit catastrophic backtracking against live traffic within minutes.

CPU usage across every server handling HTTP and HTTPS requests spiked toward 100%. The network lost roughly 80% of its traffic. Sites behind Cloudflare, a meaningful fraction of the web at the time, returned 502 errors for 27 minutes before engineers disabled the WAF globally and restored service. Cloudflare's own postmortem, written by then-CTO John Graham-Cumming and published two weeks later, walks through the simplified pattern that caused it and shows the backtracking step count: a 3-character test string took 23 steps to reject, and a 22-character string took 555. The growth between those two numbers is the entire danger of ReDoS in one comparison.

Worth noting: a safeguard meant to cap regex CPU time had been removed during an earlier WAF refactor. The bad regex was necessary for the outage. It wasn't sufficient on its own — the missing backstop was what let it take down the whole network instead of one request.

Why AI Assistants Reach For The Evil Regex Shape

BrassCoders treats the evil-regex shape as a structural symptom of how an AI assistant generates code, not a rare slip. Asked for a validation pattern, the model optimizes for one goal: match every example the prompt implied. Nothing in that goal accounts for a hostile string, and that gap is exactly what produces a nested-quantifier regex indistinguishable from a safe one until an attacker finds the cliff.

Validating an email address, parsing a log line, and stripping whitespace from a nested structure all reduce to the same instruction in a prompt: write a pattern that matches these cases. Nesting a quantifier inside a quantifier is often the shortest route to "matches everything I tried," and it costs nothing on the small inputs a developer tests against. The cost only shows up once a stranger controls what gets fed in.

This is structurally the same gap that produces every other class of AI-generated bug BrassCoders catalogs: the model optimizes for the stated goal, and worst-case behavior was never part of the goal. A performance anti-pattern like an O(N²) loop degrades gracefully as input grows. A ReDoS regex doesn't degrade. It cliffs, going from instant to unresponsive across a narrow range of input lengths, and an attacker only needs to find that cliff once.

The pattern doesn't stay confined to code an AI wrote from scratch, either. It shows up wherever a regex gets built dynamically, string interpolation into a RegExp constructor, a schema validator compiling a user-supplied pattern, a glob matcher expanding a wildcard, because none of those code paths look like "I wrote a regex" in a diff.

The Advisories Already Landing In Your Dependency Tree

BrassCoders treats two recent GitHub Security Advisories as proof this risk already ships in production, not just in a lab example. One reached a vulnerable regex through a JSON Schema validator's dynamic option; the other generated the vulnerable regex entirely at runtime, with no dangerous pattern ever visible in source. Both turned a crafted string of a few dozen bytes into tens of seconds of CPU time on ordinary hardware.

Start with ajv, the JSON Schema validator that sits transitively underneath a large share of published npm packages. It shipped CVE-2025-69873: when its $data option is enabled, an attacker-influenced pattern reaches the RegExp constructor unvalidated. Against a pattern shaped like ^(a|a)*$, a 31-character payload produced roughly 44 seconds of CPU blocking. Each additional character in the payload roughly doubled the run time — the same exponential curve as Cloudflare's, just measured on a laptop instead of an edge network.

minimatch, the glob-matching library that underpins a large share of Node.js file-handling and CI tooling, shipped CVE-2026-27904 for the opposite reason: the vulnerable regex never appears in anyone's source code at all. Nested extglob patterns like *(*(*(a|b))) compile into regexes with nested unbounded quantifiers at runtime. A 12-byte pattern against an 18-byte non-matching input stalled the library's default matching function for over 7 seconds; one more level of nesting pushed the stall toward roughly 64 seconds. If your project accepts a user-supplied glob, through a file-upload filter or a CI config field, you inherit that risk from a dependency you never audited a line of.

Neither of these advisories involved code an AI assistant wrote. They're here because they show the exact same failure shape landing in production, at scale, in libraries downloaded millions of times a week. An AI-generated regex with the same nested-quantifier shape is one crafted input away from the same outcome.

Catching It Before It Ships

BrassCoders bundles Semgrep as one of its 12 scanners, and Semgrep ships a purpose-built analyzer for exactly this weakness class: a rule that flags a catastrophic-backtracking pattern shape without ever executing the regex. It's the same detection engine already doing the SQL-injection and hardcoded-secret work inside a BrassCoders scan, so a team wiring in a custom ReDoS rule isn't reaching for a new tool.

A regex doesn't need to run against a hostile string to be flagged as risky; the pattern shape alone is enough for static analysis to catch. Semgrep ships that analyzer under the name redos, invoked with metavariable-analysis: analyzer: redos inside a custom rule, and it checks a captured pattern against known anti-pattern shapes.

JavaScript and TypeScript projects have a second option that requires no extra CI step at all: eslint-plugin-security, a widely-used ESLint plugin with over 2,300 GitHub stars, ships a detect-unsafe-regex rule in its recommended configuration. It flags a regex that could block the Node.js event loop on the same lint pass that already runs on every commit.

Whichever scanner flags the pattern, the next call is a triage question a deterministic scanner shouldn't try to answer on its own: is this specific regex reachable with attacker-controlled input, and if so, how bad is the exposure? That's a judgment call that needs the surrounding code, not just the pattern text. BrassCoders is built to stay out of that call: it reports the raw pattern match and leaves the reachability analysis to the AI assistant reading its output, the same division of labor it applies to every finding class.

Mitigating What Static Analysis Can't Guarantee

BrassCoders and every other static scanner can only flag the pattern shape, never the runtime behavior, and no static analyzer can promise it caught every vulnerable shape across a large codebase. The same regex that's a curiosity against a 200-character input becomes a production incident against a 200,000-character one, and a length cap upstream of the regex engine is the difference between the two outcomes.

No scanner catches every vulnerable pattern; some regexes only reveal their exponential behavior against specific input shapes that static analysis can't enumerate. Cap the length of anything a regex processes before it reaches the regex engine at all. The cap doesn't fix the pattern, but it bounds the blast radius to something a request timeout can absorb.

For genuinely untrusted input at scale, consider a regex engine that guarantees linear-time matching regardless of pattern shape. RE2 and its language ports trade some regex syntax (backreferences, in particular) for a worst-case bound that makes ReDoS structurally impossible rather than merely unlikely. That's a bigger lift than adding a lint rule, and it's usually reserved for the specific code paths that parse untrusted input directly, not applied wholesale across a codebase.

None of this replaces reviewing the regex your AI assistant just handed you. A pattern with a quantified group inside another quantified group is worth a second look regardless of what wrote it, and now you know the shape to look for.

Install BrassCoders and get Semgrep, the engine behind the redos analyzer, running as one of 12 scanners on every commit: pip install brasscoders.

IDOR and Access Control in AI APIs, by the Numbers

CopperSunDev — Fri, 04 Sep 2026 15:59:11 +0000

OWASP's Top 10 2021 found broken access control in 94% of tested applications — the highest occurrence count of any category, at 318,487 logged instances. The API-specific version of the same bug, Broken Object Level Authorization, ranks first in OWASP's API Security Top 10 2023 and showed up in 27% of real attack traffic against production APIs in Salt Labs' most recent report. An AI assistant asked for a REST endpoint that returns a user's record will write one without hesitation. Whether it checks that the caller owns that record is a separate question, and the data below says it usually never gets asked. This post walks through the prevalence and attack numbers, then draws the line most AI-code-security writing skips: which of these patterns a deterministic scanner can flag, and which require a human, or an AI assistant with real context, to judge.

Broken Access Control Is OWASP's Most Common Finding

BrassCoders treats OWASP's Top 10 2021 ranking as the baseline number for how often this class of bug survives into a tested application. Broken Access Control moved from fifth place in the 2017 list to first in 2021, present in 94% of applications OWASP's contributors tested, with 318,487 total occurrences mapped across 34 separate CWEs — more raw occurrences than any other 2021 category.

The average incidence rate across those 34 CWEs sits at 3.81%, but the ceiling runs far higher: the single most common CWE inside the category peaks at a 55.97% incidence rate among the applications where it was tested. OWASP's A01:2021 Broken Access Control page also logs 19,013 CVEs tied to the category, a volume that puts it ahead of injection, cryptographic failures, and every other 2021 category by raw occurrence count. Access control is a category, not one bug, and IDOR is one of the shapes it takes most often.

The 34 CWEs mapped into A01:2021 span path traversal, forced browsing past access checks, and privilege escalation alongside the identifier-swap pattern IDOR describes. That range is part of why access control rarely shows up as a single named line item on a vulnerability report — it's a family of related gaps, and an IDOR is the member of that family an API endpoint runs into first.

BOLA Is the API-Specific Name for the Same Failure

BrassCoders treats OWASP's API Security Top 10 2023 as the sharper, API-specific lens on the same category. API1:2023 Broken Object Level Authorization ranks first among API risks, and OWASP rates the underlying weakness widespread in prevalence and easy for both an attacker to exploit and a tester to detect, once someone knows to look.

OWASP's API1:2023 entry describes the mechanics plainly: an API exposes an object identifier somewhere in the request, sequential integer, UUID, or plain string, and the server trusts that identifier without checking whether the caller may touch the object behind it. IDOR is the general term for this failure in any application, web page or API alike. BOLA is what it looks like on an API endpoint. The identifier moves from a URL path on a web form to a JSON field or query parameter, and the fix stays identical either way: compare the caller's identity against the resource's owner before the response goes out.

The Attacks Are Already Happening in Production Traffic

BrassCoders treats Salt Labs' Q1 2025 State of API Security Report as evidence that BOLA isn't a finding that only shows up in a pentest write-up. Drawn from 206 surveyed IT and security professionals plus anonymized traffic from Salt Security's own customers, the report found broken object-level authorization responsible for 27% of observed attack traffic, with BOLA and injection attacks combined behind 37% of the production API issues respondents reported.

Two more numbers from the same report frame the stakes. Ninety-nine percent of respondents said they'd hit some kind of API security issue in the past 12 months, and 55% had slowed the rollout of a new application specifically over API security concerns. Salt Labs' report frames this as an attack surface growing faster than most teams' review capacity can keep up with. An AI assistant generating new endpoints every sprint adds straight to that surface. Each one is a fresh object-level check that either exists or doesn't.

Why AI-Generated APIs Skip the Ownership Check by Default

BrassCoders sees the same shape repeatedly across AI-generated CRUD endpoints: a handler that accepts an object ID, fetches the row, and returns it, with no comparison against the caller's identity anywhere in the function. The pattern is not a training defect. It's the natural result of a prompt that specifies what an endpoint returns without specifying who's allowed to ask for it.

Ask an AI assistant for get user by id and it produces exactly that: a route, a database lookup, a response. Ownership is a fact that lives in your application's data model, not in the prompt, and an assistant that hasn't been told which callers should reach which records has no way to add the check unprompted. The result reads as complete code. It passes type-checking and a casual review, because every line does what it appears to do. The missing line, the one comparing the caller's ID against the resource owner, leaves no syntactic gap for a scanner or a reviewer to notice — which is exactly why OWASP rates the underlying weakness easy to exploit and hard to catch by accident.

The gap compounds when the same assistant generates several similar endpoints in one sitting: get by id, update by id, delete by id, list by owner, each one a fresh instance of the same missing comparison. A developer writing that dozen routes by hand might carry the ownership rule forward mentally after the first one. An assistant regenerating each handler from a fresh prompt has no such continuity unless the project's authorization pattern is written down somewhere it can read.

What a Benchmark Like crAPI Actually Tests

BrassCoders treats OWASP's crAPI project as the concrete way to test whether a detection layer or a review process actually catches BOLA, instead of trusting that it would. Short for completely ridiculous API, crAPI is a deliberately vulnerable car-marketplace application, built as a multi-service stack and modeled directly on the OWASP API Security Top 10, with BOLA scenarios included by design.

The project carries more than 1,600 stars on GitHub and ships a documented set of challenges rather than one bug to find. Running a scanner, an AI code-review session, or a manual test plan against crAPI answers a narrower and more useful question than whether it catches IDOR in general: whether it catches this specific object-level authorization gap, in this specific vulnerable app, where the answer is already known. Builders who want a sanity check on their own detection stack have somewhere to point it before trusting that stack against a real API.

What BrassCoders Can and Cannot Flag

BrassCoders does not detect IDOR or BOLA directly, because confirming an ownership check is correct requires knowing the application's authorization rules, and that context lives outside any single file a scanner reads. What BrassCoders' 12 bundled scanners do flag: routes with no visible auth decorator where the framework makes one detectable, mass-assignment-shaped update calls that write every request field to a database row, and hardcoded credentials sitting in the same handler.

Semgrep, one of the scanners inside BrassCoders' OSS core, is the pattern-matching engine behind that structural layer. It can match the shape of a missing decorator or a suspicious field-mapping loop, but it cannot evaluate whether the comparison inside an existing check uses the right field. That evaluation is a judgment about your data model, and BrassCoders hands it to the AI assistant reading its YAML output, or to a human reviewer, rather than guessing at it. Reporting the structural pattern honestly, without pretending to have verified the logic behind it, is the design choice. A wrong demotion here would be worse than an unflagged gap, because it would tell the AI triage layer the finding was already checked when it wasn't.

The same honesty applies in the other direction. A scanner that guessed right most of the time and stayed silent the rest would train an AI triage layer to trust that silence, and the one case where the guess failed is the one that ships. BrassCoders' scanners report what they can verify structurally and nothing more. That's a narrower claim than detecting IDOR outright, but a more honest one — and it's the claim an AI assistant reading the YAML output can actually build on.

What Actually Catches an IDOR Before It Ships

BrassCoders points builders at the same fix OWASP's own testing guidance recommends: write a test that requests every object-returning or object-modifying endpoint as a user who shouldn't have access, and assert a rejection instead of a 200. That test doesn't require a scanner at all. It requires enumerating who owns what, which is exactly the step an AI assistant skips when nobody tells it the answer.

Threat modeling before the code exists, authorization tests that run in CI, and DAST or manual testing against the live app are the three controls OWASP names as the ones that actually reason about intent instead of code shape. None of them are exotic. All three require someone to write down who should reach which resource — a step that's easy to skip under a deadline and impossible for a pattern scanner to reconstruct after the fact.

Practically, that means an authorization test suite grows in step with the endpoint list rather than as an afterthought scheduled for later. Every new object-returning route gets a companion test asserting what happens when someone who doesn't own the object asks for it anyway. That test is cheap to write once the ownership rule is known, and it catches the exact class of bug the numbers above describe before it reaches a review, a scanner, or an attacker.

BrassCoders runs on macOS, Linux, and Windows (WSL2). Install with:

pip install brasscoders
brasscoders scan /path/to/your/api

The OSS core is Apache 2.0 and free. BrassCoders Paid adds AI-powered enrichment for $12/dev/month — 50M tokens included, cancel any time via brasscoders portal.

The CVE Record on Insecure Deserialization in AI Python Code

CopperSunDev — Fri, 04 Sep 2026 15:57:10 +0000

PyYAML's yaml.load carried a CVE rated 9.8 out of 10. PyTorch's torch.load carried one rated 9.3, in 2025, inside a parameter that PyTorch's own documentation called the safe way to load a model. Insecure deserialization, tracked by MITRE as CWE-502, is not a hypothetical risk that AI code generation might someday introduce. It has a CVE history stretching back a decade, and the pattern keeps resurfacing one abstraction layer deeper each time a new ML framework ships its own version of "just load the file."

This matters for anyone reviewing AI-generated Python because the failure mode is invisible at the call site. yaml.load(f) and yaml.safe_load(f) look identical in a diff — one word apart, wildly different security posture. torch.load(path, weights_only=True) reads like a safety flag was already applied. An AI assistant completing a config loader or a model-loading function has no reason to know which of these APIs carries a decade of CVE history behind it; the prompt asked for functionality, and both options satisfy it on safe input.

The Bug That Won't Die: PyYAML's yaml.load

BrassCoders treats CVE-2017-18342 as the reference case for why yaml.load is unsafe by default. PyYAML versions before 5.1 let yaml.load execute arbitrary code on crafted input — no authentication, no user interaction, exploitable over the network. The advisory rates it CVSS 9.8, near the ceiling of the scale.

The GitHub Security Advisory explains the mechanism, and it's straightforward once you see it. PyYAML's full Loader supports Python-specific tags like !!python/object/apply:subprocess.run, which construct and immediately call arbitrary Python objects during parsing. A YAML file is not just data to this loader; it's a set of instructions for building objects, and one of those objects can be a running shell command. PyYAML's fix, shipped in version 5.1 after an earlier attempt in 4.1 got rolled back for breaking compatibility, changed the library's default Loader and pushed developers toward yaml.safe_load, which restricts parsing to plain Python types with no object construction. Eight years after the CVE, yaml.load with the unsafe Loader still shows up in freshly generated code, because the API itself never went away — only the recommendation changed.

The Same Bug, Three Layers Deeper: PyTorch's torch.load

BrassCoders treats CVE-2025-32434 as proof that a documented safe mode can still ship a CWE-502 hole. PyTorch's own documentation recommended torch.load(weights_only=True) as the way to load an untrusted checkpoint without risk. Rated CVSS 9.3, the vulnerability showed that flag alone did not stop remote code execution on PyTorch 2.5.1 and earlier.

The security advisory is worth sitting with for a moment. This wasn't a case of a developer skipping a security flag out of ignorance. This was the flag PyTorch told everyone to use, in code that followed the documentation exactly, still carrying a critical vulnerability. The fix landed in PyTorch 2.6.0 — a version bump, not a code change on the caller's side. A model-loading function generated by an AI assistant that correctly sets weights_only=True looks like defensive code. It is defensive code, against everything except the specific gap this CVE closed, on every PyTorch install before 2.6.0. Version pinning matters here in a way that a code reviewer scanning for the presence of a safety flag would miss entirely.

Even The Framework's Convenience Wrapper Isn't Safe

BrassCoders treats CVE-2026-12484 as evidence the deserialization risk moved into the ML framework's own convenience wrappers, not just raw pickle calls. Keras's TorchModuleWrapper.from_config method calls torch.load with weights_only=False by default, outside an explicit safe-mode context. Rated CVSS 7.8, it was patched in Keras 3.12.3 and 3.15.0.

The full advisory shows what's absent from that vulnerable code path: the word "pickle." Nobody writing or reviewing a call to TorchModuleWrapper.from_config sees a deserialization primitive at all. They see a Keras layer-loading API, three abstraction layers removed from pickle.load, which is itself the primitive torch.load wraps. This is the pattern that makes AI-generated ML code specifically risky compared to AI-generated general-purpose Python: the unsafe call gets buried inside framework glue that reads as routine, safe-looking model plumbing. A reviewer — human or AI — pattern-matching on the literal string pickle will miss every one of these.

Malicious Models Are Already In The Wild

BrassCoders treats JFrog's Hugging Face research as the canonical evidence that pickle-based model deserialization is an active exploitation vector, not a theoretical one. JFrog's security research team found close to 100 malicious models on Hugging Face carrying genuine harmful payloads, with PyTorch pickle files the most common carrier. One model, uploaded by an account the researchers named, opened a reverse shell to an attacker-controlled IP address the moment it was loaded.

The full writeup has a detail worth sitting with too. This wasn't a synthetic test model or a proof-of-concept the researchers built themselves. It was a real file, uploaded to a real public repository, that a real data scientist could have downloaded and loaded into a training pipeline with a single torch.load call. An AI assistant asked to "write a script that downloads and loads the latest checkpoint from this Hugging Face repo" will produce exactly the call that triggers a payload like this one, because nothing in that prompt distinguishes a trusted checkpoint from a hostile one. The model file itself is the untrusted input, and it looks identical to a legitimate one until it's already running.

What The Standard Library Already Warned You About

BrassCoders treats the pickle module's own documentation as the most unimpeachable source in this category — the standard library warns about itself. The documentation states plainly: "The pickle module is not secure. Only unpickle data you trust." It goes further, pointing to json for untrusted data or an hmac signature to verify data hasn't been tampered with.

The pickle module docs carry that warning; the OWASP Deserialization Cheat Sheet names the same Python danger patterns explicitly — pickle, c_pickle, and PyYAML's load method — and adds two general defenses that apply beyond Python: prefer a plain data format that can't encode executable objects, or sign serialized data and reject anything unsigned before deserializing it. Neither source is buried in an obscure security blog. One ships inside the interpreter every Python developer already has installed. When an AI assistant writes pickle.load(untrusted_stream), it isn't missing an edge case the industry hasn't documented. It's contradicting a warning that ships in the same standard-library page it presumably drew the function signature from.

What BrassCoders Catches — And What It Doesn't

BrassCoders' Bandit integration flags pickle.load, pickle.loads, and cPickle calls under rule B301, and unsafe yaml.load calls under rule B506, structurally, on every scan. The detection is pattern-based: any call matching the signature gets flagged, full stop, regardless of whether the surrounding code happens to be safe in this particular instance.

That's a deliberate design choice, not a limitation BrassCoders is trying to hide. Deciding whether a specific pickle.load call actually receives untrusted input requires reading the surrounding code: where the bytes came from, whether they crossed a network boundary, whether a user or an external file supplied them. That's context inference, and BrassCoders doesn't do context inference. It reports the pattern honestly and leaves the "is this one real" judgment to whichever AI assistant reads the finding next, the same division of labor that governs every scanner BrassCoders bundles.

The model-file side of the problem is worth pairing with source-level scanning. ModelScan, from Protect AI, reads Pickle, SavedModel, and H5 model files byte-by-byte to flag unsafe code signatures without executing them — coverage for the file someone downloaded, complementary to BrassCoders' coverage of the call site that loads it.

The ML-Specific Fix: Stop Deserializing Executable Code At All

BrassCoders treats the shift from pickle-based checkpoints to a byte-buffer format as the closest thing this category has to a structural fix rather than a patched flag. Hugging Face's safetensors format stores tensors as raw byte buffers with a JSON header describing shape and dtype — no Python object graph, no __reduce__ method, nothing to execute. Loading a safetensors file cannot run code, by construction, the same guarantee yaml.safe_load gives you for YAML and json.loads gives you for arbitrary Python objects.

Safetensors is a different kind of fix than the four CVEs above required. PyYAML's, PyTorch's, and Keras's fixes each closed one specific hole in a format designed to deserialize executable objects — the next hole in the same design is only a matter of time, which is exactly what happened three times in a row. Safetensors sidesteps the whole design. An AI assistant asked to "load a model checkpoint" has no strong reason to prefer one format over the other unless the prompt or the surrounding codebase steers it there; a requirements.txt pinned to a current framework version and a preference for .safetensors over .bin or .pt checkpoint files where the model publisher offers both closes more risk than either alone. Many popular Hugging Face repositories now publish both formats side by side. The safer one is often already sitting there, just not the default the older tutorial used.

Reproduce The Pattern Yourself

None of the four CVE-class findings above require special access to reproduce. pip install brasscoders, point it at any Python project with a pickle.load, pickle.loads, or bare yaml.load call, and the B301 or B506 finding shows up in the scan output with file, line, and severity. Run it against a project that loads Hugging Face checkpoints and check whether the loading code pins a framework version alongside the safety flag — the version is the part a quick read-through tends to skip.

The full research index entry for this category has the complete CVE list, the primary-source advisories, and the OWASP remediation reference, kept current as new advisories land. The OSS core is free and Apache 2.0 licensed; BrassCoders Paid adds AI-powered enrichment on top at $12/dev/month.

AI Code License Risk From Training-Data Memorization

CopperSunDev — Fri, 04 Sep 2026 15:55:37 +0000

Your AI coding assistant did not write that function from scratch. It predicted the next token, and the token before that, from a model trained on a large slice of public code. Most of the time the result is original enough that nobody thinks twice. Sometimes it is not — the model reproduces a chunk of training data closely enough that the code carries the license of wherever it came from, and nobody in the room knows it.

That gap has a name now: code provenance risk. It is not hypothetical. GitHub has published its own numbers on how often it happens with Copilot. A federal lawsuit over exactly this question has been in litigation since 2022 and is still unresolved. And the underlying mechanism — why some code gets memorized and reproduced while most doesn't — has been measured directly by researchers, not guessed at.

Why Language Models Memorize Training Data At All

BrassCoders points builders to Carlini et al.'s 2022 ICLR paper, "Quantifying Memorization Across Neural Language Models," as the foundational evidence that language models memorize exact snippets rather than only learning generalized patterns. The paper measured three separate drivers of memorization, each with a log-linear relationship to how much verbatim text a model can be made to emit: model capacity, how many times a given example appeared in the training set, and how many tokens of context the model is prompted with.

The duplication finding is the one that matters for license risk. A snippet that shows up once in an obscure repository is unlikely to be memorized. A snippet that shows up thousands of times across public GitHub — a common utility function, a standard boilerplate block, a widely-copied algorithm implementation — is exactly the kind of example a model is statistically most likely to have memorized. And code that gets duplicated that often across public repositories tends to carry a specific, traceable license, because that's how it propagated in the first place.

This is not a claim that most AI-generated code is memorized. Carlini and coauthors were studying the general phenomenon, not code specifically, and the paper predates the current generation of code-focused models by several years. What it establishes is the mechanism: duplication in training data predicts reproduction in output. Any coding assistant trained on public repositories inherits this property to some degree.

How Often This Actually Happens, By GitHub's Own Numbers

BrassCoders treats GitHub's own published data as the most concrete evidence available, because it comes from the vendor measuring its own product rather than a third party estimating from outside. GitHub reports that matches to public code occur in under 1% of Copilot suggestions, using a duplication-detection filter that checks each suggestion's surrounding 150 characters, roughly 65 lexemes, against an index of public code on GitHub.com. The check runs in a 10-20 millisecond budget so it doesn't slow the editor down. When code referencing finds a match, GitHub surfaces the source repositories and their licenses so a developer can decide whether to keep the suggestion, add attribution, or discard it.

Under 1% sounds small. Read the fine print, though: GitHub's own documentation notes matches are far more frequent in empty or nearly empty files than in files with existing surrounding code — the model has less context to steer it toward something novel, so it leans harder on what it has seen before. And a sub-1% per-suggestion rate is not a sub-1% per-year rate for a team. A developer using a coding assistant heavily can accept dozens of suggestions a day. Multiply that across a team, across a year, and a rare event stops being rare in absolute terms.

It's worth being precise about what this number does and doesn't tell you. It's one vendor's own filtered measurement of its own product's suggestions, using its own detection threshold. It says nothing about assistants that don't run an equivalent filter, and it says nothing about code generated in longer sessions where more context accumulates. Treat it as a floor on the true rate, not a ceiling.

The Legal Fight Nobody Has Resolved Yet

BrassCoders tracks Doe v. GitHub, Inc. as the clearest evidence that this question hasn't been settled by anyone with the authority to settle it: a court, a regulator, or a standards body. Filed in the Northern District of California in November 2022 (No. 22-cv-06823-JST), the suit alleges that Copilot reproduces licensed open-source code without the attribution its licenses require, naming GitHub, Microsoft, and OpenAI as defendants. As of 2026 the case is still active, on appeal, unresolved.

According to case commentary published by Syracuse Law Review, the district court dismissed most of the original claims in 2024. The core dismissed claim rested on a DMCA provision protecting copyright management information; the court read that provision as requiring the AI's output to be an identical copy of the original work rather than a modification of it, and found Copilot's output didn't meet that bar. The plaintiffs appealed to the Ninth Circuit later that year, arguing the identicality reading is wrong.

Don't read either side of that ruling as a final answer. A dismissal on a narrow statutory reading isn't a finding that AI-generated code is legally safe, any more than a pending appeal is evidence that it isn't. What the case does establish, plainly, is that the underlying question — does license law's traditional identical-copy standard even fit a system that produces near-verbatim, modified output — hasn't been answered by any appellate court yet. Builders shipping AI-generated code into a product with real IP exposure are operating in that gap today, not after the case resolves.

What Provenance Scanning Actually Checks

BrassCoders points to ScanCode Toolkit, maintained by nexB and the AboutCode project, as a canonical example of what a purpose-built provenance scanner does differently from a security or quality scanner. It walks a codebase looking for license text, copyright notices, and package origin metadata, then emits the result as a structured inventory — JSON natively, or in the SPDX and CycloneDX formats that a software bill of materials (SBOM) uses. That's a different question than "does this code have a SQL injection bug." A security scanner reads code for dangerous patterns. A provenance scanner reads code for where it came from.

The Cybersecurity and Infrastructure Security Agency frames the compliance side of the same problem. CISA defines an SBOM as a nested inventory, a list of ingredients that make up a software component, and treats it as a building block for supply-chain risk management across government and industry. The agency, working with international partners, publishes minimum-elements guidance that sets the baseline fields a compliant SBOM has to record. For a team shipping into a regulated industry or a government contract, an SBOM built by actually scanning the shipped code is the artifact that turns "we think our AI-generated code is fine" into something an auditor can check.

Neither of these tools tells you whether a specific line of AI-generated code was memorized from a specific training example. Nobody can answer that with certainty from the output alone; the training data isn't public for most commercial models. What provenance scanning gives you is the next best thing: a record of what license terms and copyright notices are detectable in what you actually shipped, checked systematically instead of by whichever engineer happened to notice.

Where BrassCoders Fits In This Picture

BrassCoders runs 12 scanners against your codebase — Bandit, Pylint, Pyre/Pysa, Semgrep, ast-grep, detect-secrets, plus custom detectors for secrets, privacy, AI-specific patterns, performance, content moderation, and JavaScript/TypeScript — and every one of them answers a question about what the code does, not where it came from. License and provenance scanning is a fundamentally different detection problem: it's a matching problem against a corpus of known licensed text, not a pattern-matching problem against known-bad code shapes. BrassCoders doesn't attempt it, and this post isn't an argument that it secretly does.

The honest framing matters more than the feature gap. BrassCoders was built as a dumb-but-honest pattern reporter, deliberately: it reports what a deterministic scanner can verify, and leaves the context-dependent judgment calls to whatever AI assistant is reading its output. License provenance is exactly the kind of judgment call that doesn't fit a pattern reporter — whether a 12-line utility function is common enough to be unprotectable, or specific enough to carry real license weight, requires comparing against a corpus BrassCoders was never built to hold. That's ScanCode Toolkit's job, or a commercial SBOM tool's job, run as a separate pass alongside whatever security and quality scanning you already do.

What To Do About It Today

Start with the parts of this that are actually actionable now, before the litigation resolves and before any standards body issues a definitive rule. Run a provenance scanner like ScanCode Toolkit on your codebase periodically, the same way you'd run a dependency audit — it's a separate pass, not a replacement for anything you already run. Treat any AI-generated block that looks unusually idiomatic or complete for a first draft as worth a second look; that's often a sign the model had strong context to draw from, which correlates with memorization. Keep a record of what you find, even a simple one; an SBOM doesn't need to be sophisticated to be useful, it needs to exist.

None of this eliminates the risk. It converts an unknown into something you can reason about, which is the most any team can do with a legal question that's still being argued in front of a federal appeals court.

Why AI Transcription Is Affordable Now

CopperSunDev — Wed, 02 Sep 2026 16:00:04 +0000

BrassTranscripts can price accurate transcription at a few dollars per file because a research breakthrough called self-supervised learning removed the single most expensive ingredient in speech recognition: enormous hand-labeled datasets. Once AI could learn the structure of speech from unlabeled audio and fine-tune on only a little transcribed speech, high accuracy stopped being something you had to pay a premium for — it became the default.

For years, the reason good transcription was expensive had nothing to do with the software running on your file. It was the cost, buried upstream, of paying people to transcribe thousands of hours of audio by hand just to teach the model what words sound like. This post explains how that bottleneck disappeared, why accuracy and affordability now come together instead of trading off, and what that means for the price you pay per file.

Quick Navigation

The labeled-data bottleneck that made transcription expensive
What self-supervised learning changed
The wav2vec 2.0 results, in plain terms
Why affordable no longer means inaccurate
What actually determines your transcript quality now
Frequently Asked Questions

The Labeled-Data Bottleneck {#the-labeled-data-bottleneck}

The historical cost of accurate speech recognition was human labeling, not computing: someone had to listen to thousands of hours of audio and type out every word so a model had examples to learn from. That labeling labor, not the algorithm, is what kept professional transcription priced out of reach for most people.

Traditional supervised speech models learned only from paired examples — an audio clip and its verified transcript. To cover the variety of real speech (accents, vocabularies, recording conditions), you needed a very large paired dataset, and every hour of it had to be transcribed by a person first. That made the training pipeline slow, expensive, and impossible to scale cheaply, and those costs flowed straight through to what customers paid.

Consider what that meant in practice. A single hour of professionally transcribed and verified audio could take several hours of skilled human labor to produce. Multiply that by the thousands of hours needed to train a model that handles many accents and topics, and the labeling budget dwarfs the compute budget. Every provider paid some version of that tax, and it showed up as high per-minute pricing, minimum commitments, and subscriptions. If you want a refresher on the vocabulary in this space, the AI transcription glossary of key terms defines labeling, fine-tuning, and word error rate.

What Self-Supervised Learning Changed {#what-self-supervised-learning-changed}

Self-supervised learning let a model learn the patterns of speech from raw, unlabeled audio first, and only then fine-tune on a small amount of transcribed speech. BrassTranscripts benefits from this shift directly: the most expensive part of building an accurate model — the hand-labeling — was largely replaced by cheap, abundant unlabeled audio.

The landmark demonstration is the paper "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli at Meta AI (FAIR), published at NeurIPS 2020 (arxiv.org/abs/2006.11477). Its core idea is that a model can be pre-trained to understand the structure of speech from unlabeled recordings, so that afterward it needs only a fraction of the transcribed data that older approaches demanded. That inversion — lots of unlabeled audio, a little labeled audio — is the economic hinge that made accurate transcription cheap to deliver at scale.

The wav2vec 2.0 Results {#the-wav2vec-20-results}

wav2vec 2.0 showed both that self-supervised pre-training reaches top accuracy with the full labeled set and that it stays usable with a startlingly small amount of labeled audio. BrassTranscripts points to these numbers because they make the affordability story concrete rather than hand-wavy.

Measured on the standard LibriSpeech benchmark, the model reached a word error rate of 1.8 on test-clean and 3.3 on test-other when fine-tuned on the full labeled set. More striking for the cost argument: with just ten minutes of labeled audio — combined with pre-training on 53,000 hours of unlabeled audio — it still produced a usable 4.8 word error rate on test-clean and 8.2 on test-other. In plain terms, a model taught almost entirely from unlabeled recordings, plus a sliver of transcribed speech, still transcribed clean audio well. Word error rate is simply the percentage of words the system gets wrong, so lower is better; our research summary on transcription accuracy explains how that metric is measured and why it can vary by recording.

Affordable No Longer Means Inaccurate {#affordable-no-longer-means-inaccurate}

Low price and high accuracy stopped being a trade-off because the thing that fell was the training cost, not the quality bar. BrassTranscripts charges a flat per-file rate rather than gating accuracy behind a premium tier, because the underlying research made professional-grade accuracy the baseline rather than an upsell.

This is where older pricing intuitions mislead people. When labeling was the dominant cost, it was reasonable to assume "cheaper transcription" meant "worse transcription." After self-supervised learning, the assumption inverts: the marginal cost of running an already-trained, highly accurate model on your file is low, so charging premium prices for accuracy alone is hard to justify.

It is worth being precise about what the research does and does not claim. The wav2vec 2.0 numbers describe a specific model on a specific English read-speech benchmark, not a guarantee about any given file — a noisy phone recording of three people talking over each other is a harder problem than clean audiobook narration. But the direction is unmistakable: once a model can be built without an enormous hand-labeled corpus, the economics that forced high prices simply are not there anymore. For a fuller comparison of what different providers actually charge and why, see our guide on how to choose an AI transcription service in 2026 and the breakdown of Whisper API pricing, self-hosted versus managed.

What Determines Your Transcript Quality Now {#what-determines-your-transcript-quality-now}

With the model itself already strong, the biggest remaining lever on accuracy is your audio: clear recording, low background noise, and speakers who do not talk over each other. BrassTranscripts runs the same advanced AI transcription on every file, so a cleaner recording improves your transcript far more than any "higher quality" purchase option ever could.

That is genuinely good news for your budget. Instead of paying for tiers, you invest a few minutes in a better recording — a decent microphone, a quiet room, one speaker at a time — and let the model do the rest. Our deeper explainer on what determines transcription accuracy walks through the specific, controllable factors that move the number, most of which cost nothing to fix.

Frequently Asked Questions

Why did AI transcription become cheaper?

AI transcription became cheaper because self-supervised learning let models learn the structure of speech from large amounts of unlabeled audio, then fine-tune on a small set of transcribed speech. This removed the need to pay humans to hand-label thousands of hours of audio, which was historically the largest cost in building an accurate speech recognition system.

What is wav2vec 2.0 and why does it matter?

wav2vec 2.0 is a 2020 speech model from Meta AI that learns speech representations from unlabeled audio before fine-tuning on transcribed speech. It matters because it showed that high-accuracy transcription no longer required massive hand-labeled datasets, which is the economic shift that made affordable, professional-grade AI transcription possible.

Does affordable AI transcription mean lower accuracy?

No. Modern AI transcription is affordable because the training method changed, not because quality was reduced. Research showed models can reach strong accuracy after learning from unlabeled audio, so BrassTranscripts delivers professional-grade accuracy at flat per-file pricing rather than charging more for better results.

How much does BrassTranscripts cost?

BrassTranscripts charges $2.50 for audio files 1-15 minutes and $6.00 flat for files 16 minutes and up, at any length. Automatic speaker identification and four output formats (TXT, SRT, VTT, JSON) are included at that flat rate, with support for 99+ languages and no subscription required.

What most affects my transcript's accuracy today?

Because the underlying AI models are already strong, the biggest remaining factor in transcript accuracy is your audio quality — clear recordings, minimal background noise, and non-overlapping speakers. BrassTranscripts applies the same advanced AI transcription to every file, so improving your recording does more for accuracy than paying a premium tier ever could.

About BrassTranscripts

BrassTranscripts is a pay-per-file AI transcription service: $2.50 for files 1-15 minutes and $6.00 flat for files 16 minutes and up, at any length. Every transcript includes automatic speaker identification and four output formats (TXT, SRT, VTT, JSON), with support for 99+ languages and no subscription required. Upload a file, pay for that file, and download professional-grade results — the affordability comes from the research described above, not from cutting quality.

How AI Adds Punctuation to Transcripts

CopperSunDev — Tue, 01 Sep 2026 16:47:43 +0000

Punctuation is not something a speech recognizer produces on its own. BrassTranscripts starts from what speech recognition actually outputs — an unpunctuated, uncapitalized stream of words — and then a separate punctuation-restoration step adds the sentence boundaries, commas, and capitalization that turn that stream into readable text. Understanding that these are two distinct steps explains why a transcript can nail every word yet still need a light formatting pass, and why punctuation quality is its own dimension of transcript quality.

Quick Navigation

Speech Recognition Produces a Raw Word Stream
Punctuation Restoration Is a Separate Step
How the Punctuation Model Works
Why It Works Across Languages
Why Punctuation Is Its Own Quality Dimension
What This Means for Your Transcripts
Frequently Asked Questions

Speech Recognition Produces a Raw Word Stream

BrassTranscripts begins with core speech recognition, which converts audio into words but nothing more — no periods, no commas, no capital letters, just a continuous lowercase sequence of the words it heard. A recognizer's only job is to map sound to tokens, so its native output looks like "so we met on tuesday and the budget was approved but the timeline slipped" with no structure at all.

That raw form is exactly what you would expect from a system trained to answer one question: what words were spoken? Nothing in that question involves where a sentence ends or whether "tuesday" should be capitalized. Those are decisions about written language, not about sound, which is why they fall outside the recognizer's job. The concepts behind these steps are covered in the AI transcription glossary, which defines the vocabulary of the transcription pipeline.

Punctuation Restoration Is a Separate Step

The readability of a BrassTranscripts transcript comes from punctuation restoration, a dedicated post-processing step that runs after the words are recognized and inserts sentence boundaries, commas, question marks, and capitalization. This step takes the raw word stream as input and rewrites it into structured prose — "So we met on Tuesday, and the budget was approved, but the timeline slipped." — without changing the words themselves.

Treating this as a separate stage is deliberate. The recognizer optimizes for hearing words correctly; the punctuation model optimizes for interpreting how those words group into sentences and clauses. Splitting the work lets each model specialize instead of forcing one system to do two very different jobs at once. The result is that punctuation is applied consistently regardless of how the speaker paused or ran sentences together, and it lands the same way across every output format you download — TXT, SRT, VTT, or JSON.

How the Punctuation Model Works

BrassTranscripts relies on the same class of technique that transcription researchers have converged on: a Transformer-based language model fine-tuned specifically to predict punctuation from unpunctuated text. In their W-NUT 2020 paper "Punctuation Restoration using Transformer Models for High- and Low-Resource Languages," Alam, Khan, and Alam (2020) fine-tuned a Transformer language model — a pretrained encoder followed by a bidirectional LSTM — to label each position in a word stream with the punctuation that belongs there (aclanthology.org/2020.wnut-1.18).

The intuition is that punctuation is a prediction problem over the sequence of words. For every gap between words, the model asks whether that gap should stay empty or hold a comma, a period, or a question mark, and for every word it asks whether it should be capitalized. Because the model reads context in both directions — the words before and after each position — it can tell that a rising, question-shaped clause needs a question mark or that a proper noun needs a capital letter. This is the same interpretive work a human editor does when cleaning up a rough transcript, done automatically at scale.

Why It Works Across Languages

Punctuation restoration is not an English-only trick — the approach generalizes to languages with far less training data available. Alam, Khan, and Alam (2020) evaluated their Transformer approach on both a high-resource language, English, and a low-resource one, Bangla, showing that the same architecture restores punctuation effectively even where large annotated corpora are scarce.

That generality matters for anyone transcribing beyond English. BrassTranscripts applies punctuation and capitalization across 99+ languages, which is only practical because the underlying method does not depend on the enormous datasets that exist for English alone. The demand for non-English transcription is real and growing, as our language usage data shows, and readable output in every one of those languages depends on this step working outside English.

Why Punctuation Is Its Own Quality Dimension

Word accuracy and punctuation accuracy are separate measures because they are produced by separate models solving separate problems. BrassTranscripts can transcribe every word in a sentence correctly while the punctuation model still places a comma in an unusual spot or splits one long spoken sentence into two — the word-error rate and the punctuation quality move independently.

This is why it is a mistake to judge an entire transcript by a single number. A transcript that is flawless word-for-word may still read slightly oddly if punctuation lands imperfectly, and a light formatting cleanup fixes that without touching the words. Our deep dive on what determines transcription accuracy treats word recognition and formatting as distinct factors for exactly this reason, and our research page on transcription accuracy documents the measurable metrics we report. When you know punctuation is its own layer, you know where to look when something reads awkwardly — and that it is usually a two-minute edit, not a re-transcription.

What This Means for Your Transcripts

For practical purposes, BrassTranscripts delivers transcripts with punctuation and capitalization already applied, so you receive readable sentences rather than a raw token stream. You never have to run the punctuation step yourself — it happens automatically before the file reaches you, in every format.

Knowing the pipeline still helps you work faster. If a sentence break lands in an unexpected place, that is the punctuation layer, not a word error, and you can fix it in seconds. If you want to reshape the text further — say, toward clean or intelligent verbatim — you are editing formatting on top of accurate words, which is exactly the workflow described in our guide to verbatim, clean, and intelligent verbatim styles. The words are the hard part, and the machine has already done them; punctuation is the readable finish on top.

Frequently Asked Questions

Does speech recognition add punctuation automatically?

Core speech recognition does not add punctuation — it converts audio into a raw, lowercase stream of words with no periods, commas, or capital letters. Punctuation and capitalization come from a separate post-processing step, called punctuation restoration, that runs after the words are recognized. BrassTranscripts applies this step automatically so the transcript you download reads in proper sentences.

What is punctuation restoration?

Punctuation restoration is a dedicated AI step that reads an unpunctuated word stream and predicts where sentences end, where commas and question marks belong, and which words should be capitalized. It is a distinct task from recognizing the words themselves, which is why a transcript can be accurate word-for-word yet still need light formatting cleanup.

Why does a transcript get the words right but the punctuation wrong?

Word accuracy and punctuation are produced by two different models solving two different problems, so they can succeed or fail independently. The speech recognizer can transcribe every word correctly while the punctuation model still places a comma awkwardly or splits a sentence, because punctuation depends on interpreting meaning and pauses rather than just identifying sounds.

Does punctuation restoration work in languages other than English?

Yes — punctuation restoration generalizes across languages, including low-resource ones. Research by Alam, Khan, and Alam (2020) demonstrated the approach on both English and Bangla, and BrassTranscripts applies punctuation and capitalization across the 99+ languages it supports.

Can I get a transcript without any punctuation applied?

BrassTranscripts delivers transcripts with punctuation and capitalization already applied across TXT, SRT, VTT, and JSON formats, because a punctuated transcript is far more usable for reading, editing, and captioning. If you need the raw token stream for a specialized workflow, the JSON output gives you word-level data you can reformat however you need.

About BrassTranscripts

BrassTranscripts is a pay-per-file AI transcription service — no subscription, no commitment. Pricing is simple: $2.50 for files 1–15 minutes long, and a $6.00 flat rate for files 16 minutes and up, at any length. Every transcript includes automatic speaker identification and arrives in TXT, SRT, VTT, and JSON formats with punctuation and capitalization already applied, across 99+ supported languages. Upload a file, and you get back readable, structured text ready to use.

How Speech Quality Is Measured for Transcription

CopperSunDev — Tue, 01 Sep 2026 16:47:29 +0000

You cannot fix what you cannot measure, and for a long time "good audio" was a matter of opinion. BrassTranscripts starts from a more useful fact: recording quality can be scored before transcription, and that score predicts how accurate the transcript will be. The audio you upload already carries measurable signals of noise, distortion, and dropouts, and each of those signals maps to a specific way a transcript can go wrong.

This post explains how speech quality is actually measured in the field, what the numbers mean, and why a quality score is really an accuracy forecast in disguise. BrassTranscripts maintains a Curated Authority Index of the primary research behind AI transcription, including the audio quality research this article draws on, so you can check every claim at the source.

Quick Navigation

Quality Can Be Scored Before You Transcribe
What a Mean Opinion Score Actually Measures
The Four Dimensions of Speech Quality
Why Quality Predicts Transcript Accuracy
How the Measurement Handles Real-World Audio
How to Use Quality Measurement Before You Upload
Frequently Asked Questions

Quality Can Be Scored Before You Transcribe

BrassTranscripts treats audio quality as a measurable property of the recording, not a subjective impression formed after reading a bad transcript. Modern speech-quality models are non-intrusive, meaning they estimate perceived quality from the degraded recording alone, without needing a pristine reference copy to compare it against.

That distinction matters more than it sounds. Older quality measures required the original clean signal to measure how far a recording had drifted from it, which is useless in the real world where no clean copy of your meeting or interview exists. A non-intrusive model looks only at the file you actually have. The leading example is NISQA, described in "NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets" by Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller of TU Berlin (2021), published at arxiv.org/abs/2104.09494. Because it needs no reference, a score like this can be produced for any recording before it is ever transcribed.

What a Mean Opinion Score Actually Measures

A Mean Opinion Score, or MOS, is a single number that captures overall perceived speech quality, and BrassTranscripts uses it as shorthand for how clear a recording sounds to a listener. Historically the MOS was gathered by asking human listeners to rate audio on a simple scale and averaging their responses; the achievement of modern models is predicting that human rating automatically.

The overall MOS is a useful summary, but a summary hides detail. A recording that scores poorly could be too quiet, too noisy, distorted, or full of dropouts, and the single number treats all of those as the same generic "bad." NISQA was built specifically to reject that oversimplification. Alongside the overall MOS, it predicts four separate quality dimensions, so the score tells you not just that a recording is degraded but in which way it is degraded. That is the difference between a warning light and a diagnosis.

The Four Dimensions of Speech Quality

BrassTranscripts finds the four-dimension breakdown more actionable than any single rating, because each dimension names a defect you can actually go and fix. NISQA predicts Noisiness, Coloration, Discontinuity, and Loudness in addition to the overall MOS.

Each dimension isolates one failure of the recording chain. Noisiness reflects background sound layered over the speech, the hum, chatter, or traffic competing with the voice. Coloration reflects distortion of the frequency balance, the tinny or muffled quality that comes from a poor microphone or aggressive compression. Discontinuity reflects interruptions in the signal, the gaps and dropouts that appear when audio is lost in transit. Loudness reflects level problems, speech that is too quiet or clipped from being too hot. A recording can score well on three and fail the fourth, which is exactly the information a single MOS throws away. The four-dimension model was introduced in the NISQA work by Mittag and colleagues (2021).

Why Quality Predicts Transcript Accuracy

The reason to measure any of this is that input quality is the single biggest driver of transcript accuracy, and BrassTranscripts treats a low quality score as an early warning that the transcript will need scrutiny. Clean input audio consistently produces the most accurate output, a principle so well established it borders on obvious once you have compared a studio recording against a phone call of the same conversation.

What the dimensional view adds is a prediction of the failure mode, not just the failure. A low Discontinuity score suggests packet-loss gaps, the kind of dropout that erases whole words and leaves the transcript missing content entirely. A low Noisiness score suggests background noise, which tends to produce misheard words rather than missing ones, as the model tries to resolve speech buried under competing sound. Coloration problems blur the acoustic distinctions between similar-sounding words, and loudness problems can push quiet speech below the threshold where it registers at all. Each dimension predicts a different way the transcript will disappoint you, which is far more useful than a vague sense that the audio was "not great." Our guide to what determines transcription accuracy covers why the recording matters more than the brand of software.

How the Measurement Handles Real-World Audio

A quality model is only trustworthy if it was tested on the audio people actually record, and BrassTranscripts values the NISQA research precisely because it was built on real conditions rather than laboratory tone. Its corpus contains more than 14,000 speech clips spanning a wide range of distortions, including live recordings made over mobile phone, Zoom, Skype, and WhatsApp.

That coverage is what makes the scores relevant to your uploads. A meeting recorded through Zoom, an interview captured on a phone, and a voice note sent over WhatsApp each degrade in characteristic ways, and a model trained on those exact channels predicts their quality reliably rather than guessing. The crowdsourced datasets behind NISQA were designed to capture this real-world variety, which is why its predictions hold up outside the lab. For a deeper look at the vocabulary around all of this, see our AI transcription glossary.

How to Use Quality Measurement Before You Upload

You do not need to run a research model to benefit from thinking in these terms, and BrassTranscripts recommends listening to your recording through the four-dimension lens before you transcribe. Ask whether the speech is buried in noise, whether it sounds distorted, whether the audio cuts in and out, and whether it is loud and clear, because those four questions map directly to the dimensions research uses to predict quality.

Where you can, fix the problem the diagnosis points to rather than transcribing and hoping. If dropouts are the issue, re-export from the original source or record locally instead of over a call. If noise is the issue, move somewhere quieter or closer to the microphone. Our practical walkthroughs of audio quality secrets for perfect transcription and fixing the audio problems ruining your transcripts cover the specific fixes. When you are ready, the surest test is your own audio: BrassTranscripts shows a 30-word preview of every transcript before purchase, so you can confirm accuracy on the exact file you care about rather than trusting any score in the abstract.

Frequently Asked Questions

Can speech quality be measured before transcription?

Yes. Non-intrusive quality models estimate perceived speech quality from the recording alone, without needing a clean reference copy to compare against. BrassTranscripts treats this the same way a professional would: the recording carries measurable signals of clarity, noise, and dropouts that exist before a single word is transcribed, and those signals predict how the transcript will turn out.

What is a Mean Opinion Score?

A Mean Opinion Score, or MOS, is a single rating of overall perceived speech quality, traditionally averaged from human listeners and now predictable by models. The NISQA research goes further than one number, predicting four separate dimensions alongside the overall score. BrassTranscripts finds the multidimensional view more useful because a bad recording is rarely bad in only one way.

What are the four dimensions NISQA predicts?

NISQA predicts Noisiness, Coloration, Discontinuity, and Loudness in addition to an overall MOS. Each isolates a different defect: Noisiness captures background sound, Coloration captures frequency distortion, Discontinuity captures gaps and dropouts, and Loudness captures level problems. BrassTranscripts maps these to distinct transcript failure modes, so a low score points to the specific problem to fix.

Does a low quality score mean a bad transcript?

Usually, and it also tells you why. A low Discontinuity score suggests packet-loss gaps that erase whole words, while a low Noisiness score suggests background noise that produces misheard words. BrassTranscripts shows a 30-word preview of every transcript before purchase, so users can confirm accuracy on their own audio rather than relying on a score alone.

Where does call and meeting audio fit into quality measurement?

Modern quality models are trained on real communication conditions, not just studio recordings. The NISQA corpus includes clips captured over mobile phone, Zoom, Skype, and WhatsApp, which is why its scores translate to everyday audio. BrassTranscripts sees the same patterns in practice, where the recording channel often matters more than the microphone.

About BrassTranscripts

BrassTranscripts is a pay-per-file AI transcription service with no subscription. Pricing is simple: $2.50 for files 1 to 15 minutes long, and a flat $6.00 for anything 16 minutes and up, at any length. Every transcript includes automatic speaker identification and downloads in TXT, SRT, VTT, and JSON, with support for 99+ languages. Advanced AI transcription handles the recording; you just upload it. Before you pay, a 30-word preview lets you check the quality of your specific transcript, so you can confirm the result on your own audio first. When you are ready, upload a file and see the preview yourself.

Run BrassCoders Automatically in a Claude Code Hook

CopperSunDev — Sat, 29 Aug 2026 23:49:59 +0000

A Claude Code hook is a shell command that Claude Code runs for you at a set moment: after an edit, before a Bash call, when a session opens. Point one at brasscoders scan and the deterministic scan runs on its own. It writes .brass/ai_instructions.yaml to the project, and Claude Code reads that file as context. You stop pasting scanner output into the chat, and you stop remembering to run the scanner at all.

Two existing guides cover what Claude Code does with the YAML and how the same file drives a Cursor session. This one is the wiring: the exact .claude/settings.json entry that fires the scan at the right moment, so the findings are already current when the assistant starts reasoning about your code.

What a Claude Code Hook Actually Runs

BrassCoders plugs into Claude Code as an ordinary command — brasscoders --offline scan . — that Claude Code runs at a lifecycle event defined in .claude/settings.json, not as a plugin or an API. A hook is three nested levels: the event name, a matcher, and the command to run.

Claude Code fires hooks at events like PostToolUse (after a tool runs), SessionStart (when a session opens), and UserPromptSubmit (when you send a message). For tool events, the matcher filters by tool name, so a matcher of Edit|Write runs only after the assistant edits or creates a file. Hooks live in .claude/settings.json for one project or ~/.claude/settings.json for every project you open, and each command hook is a { "type": "command", "command": "..." } entry nested under its event and matcher. The full event list and field reference live in the Claude Code hooks documentation.

The Config: Scan After Every Edit

BrassCoders refreshes its findings after each change when you register a PostToolUse hook matching Edit|Write, so .brass/ai_instructions.yaml reflects the files the assistant just touched. The whole setup is one block in .claude/settings.json.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "cd \"${CLAUDE_PROJECT_DIR}\" && brasscoders --offline scan . --incremental"
          }
        ]
      }
    ]
  }
}

Save it, and Claude Code activates the hook for the current project. After the assistant edits a file, the scan runs, and the YAML on disk is current before the next turn. The ${CLAUDE_PROJECT_DIR} placeholder resolves to your project root, so the scan targets the right directory no matter where Claude Code spawned the command. The --offline flag holds the scan to zero network calls, and --incremental keeps each per-edit run cheap, which the next sections address.

How the Findings Reach the Assistant

BrassCoders hands its output to Claude Code through .brass/ai_instructions.yaml, a severity-sorted YAML file the assistant reads as project context with no paste step. A SessionStart hook can go one step further and print a one-line pointer whose text Claude Code drops straight into the session.

Claude Code surfaces a hook's plain-text stdout as context the model can act on for a few events, SessionStart among them. Run the scan once when the session opens and echo where the results landed:

{
  "hooks": {
    "SessionStart": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "cd \"${CLAUDE_PROJECT_DIR}\" && brasscoders --offline scan . && echo 'BrassCoders wrote .brass/ai_instructions.yaml. Read it before editing.'"
          }
        ]
      }
    ]
  }
}

The echoed line arrives as context the assistant sees at the top of the session, so it knows the findings file exists and opens it before it changes any code. What the assistant opens is a severity-ordered list, one entry per finding:

issues:
  - id: bandit-B608-001
    severity: critical
    file_path: api/views.py
    line_number: 47
    title: SQL Injection via String Formatting
    remediation: Replace with a parameterized query using placeholders
      and a separate values tuple.

Claude Code reads the critical entries first, opens the flagged line, and writes the fix; the hook's only job is to guarantee that list is current. What each field does — severity ordering, line anchors, remediation notes — is the subject of the findings-file walkthrough.

Scan Right Before a Commit

BrassCoders can refresh its findings in the moment before a commit when you attach a PreToolUse hook to the Bash tool and narrow it with the if field to git commit commands. The scan runs, the YAML updates, and the assistant has the current findings in front of it before the commit lands.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "if": "Bash(git commit *)",
            "command": "cd \"${CLAUDE_PROJECT_DIR}\" && brasscoders --offline scan . --incremental"
          }
        ]
      }
    ]
  }
}

The if field uses Claude Code's permission-rule syntax to scope the hook to a subset of Bash calls; the hooks documentation covers the exact matching rules. This is the assistant's local, in-session checkpoint, not a build gate. For a hard gate that blocks a bad commit on the shared branch, a CI step on push and a git pre-commit hook are the right tools, and the every-commit guide has both.

Keep the Per-Edit Scan Fast

BrassCoders's --incremental flag re-scans only the files that changed since the last scan for its file-local scanners and replays cached findings for everything unchanged, so a per-edit hook doesn't pay for a full scan each time. The cross-file scanners, Pysa and ast-grep among them, still run in full for correctness.

On the first run, or when the cache is missing, --incremental falls back to a full scan to seed itself. After that, editing one file re-runs the file-local scanners on that file alone and reuses cached results for the rest. If you'd rather not block the edit loop, set the command hook's async field to true so the scan runs in the background, or drop the per-edit hook and scan once at SessionStart. A one-shot session scan carries no per-edit cost.

Scope the Scan With .brassignore

BrassCoders reads a .brassignore file at the project root and skips the paths listed in it, which keeps a per-edit hook from re-flagging test fixtures or vendored directories every time the assistant saves a file. A focused scan means the assistant reads a shorter, more relevant findings list.

A scan the assistant triggers on every edit is only as useful as it is focused. Point .brassignore at the directories you don't want in the findings — generated migrations, node_modules, a tests/fixtures/ folder full of placeholder credentials — and those paths drop out of every scan the hook runs. The .brassignore tuning guide covers the file format and how it differs from .gitignore.

A Hook Triggers the Scan; It Doesn't Watch

BrassCoders has no daemon and no file watcher; the watch command was removed in version 2.0.9, so a Claude Code hook leaves nothing running between scans. Each time the hook fires, brasscoders scan runs once, writes its YAML, and exits.

The hook is the trigger; the cadence is a choice you make by picking the event. A Claude Code session is one place to run the scanner. Your CI on push and a git pre-commit hook are two others; the every-commit guide has that configuration. Those gates don't replace the Claude Code hook, which keeps your local assistant's context current as you work.

Install the scanner:

pip install brasscoders

Add the PostToolUse or SessionStart block to .claude/settings.json, open Claude Code in the project, and the findings are waiting in .brass/ai_instructions.yaml before you ask for a single change. Apache 2.0, no account, --offline by choice.

LLM Code Reviewer Reliability, by the Numbers

CopperSunDev — Sat, 29 Aug 2026 23:47:04 +0000

How reliable is an LLM when you point it at a diff and ask what's broken? The measured answer: strong at finding bugs, weak at saying the same thing twice. A 2025 benchmark clocked the recall of three frontier models between 0.78 and 0.88 on a set of real security defects, well above what the deterministic scanners in the same test managed. The catch is what those models couldn't do — pin a finding to the right line, or return the same verdict when the prompt ran again.

That pairing is the whole story. An LLM reviewer has real recall and a reliability bill that comes due the moment you try to make it a gate. A gate has one job: mean the same thing on every run. The numbers below come from three studies, and they argue for a division of labor, not a winner.

What the Benchmark Measured

BrassCoders, the scanner that catches what AI assistants structurally miss, runs on a deterministic-plus-LLM split, which is why the study worth reading first is arXiv 2508.04448: it put three LLMs against three deterministic analyzers across 10 real C# projects holding 63 known vulnerabilities.

The three models — GPT-4.1, Mistral Large, DeepSeek V3 — ran against three deterministic analyzers: SonarQube, CodeQL, and SnykCode. Full paper: Large Language Models Versus Static Code Analysis Tools by Damian Gnieciak and Tomasz Szandala, with the models run through the GitHub Models platform.

The defects span SQL injection, cross-site scripting, hardcoded secrets, and command injection. Two caveats ride along: the code is C#, and the deterministic tools tested aren't the ones BrassCoders bundles. What carries across languages is the shape of the result — which class of tool is strong where, and where each one breaks.

Recall: Where an LLM Reviewer Wins

BrassCoders hands its findings to an AI assistant for a reason the recall numbers make plain: in the benchmark, the three models scored recall between 0.78 and 0.88, while the deterministic scanners came in between 0.19 and 0.52. On raw catch rate, the models won.

Recall is the fraction of real defects a tool actually flags. The paper credits the models' lead to it directly: their F1 scores ran roughly 0.75 to 0.80 against 0.26 to 0.55 for the static tools, and the authors attribute that gap to the models' ability to reason across broader code context. A rule fires only when the pattern it encodes matches. A model can flag a defect it has never seen a signature for. That's the recall you want somewhere in your pipeline.

Precision and False Positives: The Bill for That Recall

BrassCoders treats an AI reviewer's output as triage rather than verdict, and the precision numbers show why: the three models scored 0.72 to 0.78 precision, so roughly one flag in four was a false positive, and DeepSeek V3 posted the highest false-positive ratio of the group.

Precision is the fraction of a tool's flags that are real. At 0.75, a quarter of what the model reports is noise a developer has to clear. The fair reading of this benchmark is narrow: the deterministic tools scored lower precision here (0.57 to 0.69), so this isn't a case of static analysis being cleaner. The point is about consistency, because the false-positive rate swings by model, and the paper notes DeepSeek's rate raises the verification effort a developer has to spend. For a check that runs on every commit, a flag rate that's a quarter wrong and model-dependent is friction you feel daily.

Finding Mislocation: Right Bug, Wrong Line

BrassCoders anchors every finding to an exact file and line; the benchmark found the opposite for the models, reporting that all three mislocate issues at line-or-column granularity because of BPE tokenization. A model can name the right bug and still point at the wrong place.

The paper's words: all language models "mislocate issues at line-or-column granularity due to tokenisation artefacts." The cause it names is byte-pair encoding, the step that splits source text into tokens before the model ever reasons about it. Token boundaries don't line up with line boundaries, so the model's sense of where it is drifts. A finding you can't locate is a finding a developer has to re-derive by hand, and it can't drive a line-anchored CI rule or an automatic fix. Location isn't a detail for a gate. It's the whole interface.

Run-to-Run Non-Determinism, Measured

BrassCoders returns byte-identical findings on the same commit every run. LLM reviewers don't, and two studies put numbers on it: a code-review study of four models over 70 commits, repeated five times at temperature zero, found the responses varied every time, and a code-generation study of ChatGPT found 47 to 76 percent of tasks produced no two identical outputs.

The code-review study, Measuring Determinism in Large Language Models for Software Code Review by Klishevich and colleagues (2025), tested GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and LLaMA 3.2 90B Vision. Its finding: "even with temperature minimized, LLM responses varied to different degrees," which the authors call an inherently limited test-retest reliability.

The code-generation study, An Empirical Study of the Non-determinism of ChatGPT in Code Generation by Ouyang and colleagues (peer-reviewed, ACM DOI 10.1145/3697010), ran 829 problems and reported that the share of tasks with zero identical outputs across requests was 75.76, 51.00, and 47.56 percent on its three benchmarks. Its verdict on the common workaround: "setting the temperature to 0 does not guarantee determinism." The reason sits below the API, where floating-point math and GPU batch effects at inference can select different tokens for the same input. A gate that returns a different answer on the same code isn't a gate. It's a suggestion.

Why These Numbers Disqualify the LLM as a Gate

BrassCoders exists to be the gate because a gate needs two properties the data shows LLMs lack: an exact location for every finding, and the same answer on every run. High recall can't rescue a check that can't be reproduced or pinned to a line.

Walk the failure modes. A reviewer that flips between a clean pass and three issues on identical input teaches the team one lesson: ignore it. A flag with no reliable line can't auto-block a merge or feed an auto-fix. A precision near 0.75 means an auto-blocking gate would reject good code about a quarter of the time. Each of these is survivable for a second read; none is survivable for an automated gate. The deterministic scanner runs the other way, with lower recall but the same line-anchored findings on every run. Reproducibility and location are exactly the properties a gate is made of.

The Pattern the Data Points To

BrassCoders runs its deterministic scan on every commit and hands structured YAML to an AI assistant like Claude Code or Cursor for context-aware triage — the split the benchmark's own authors recommend: models early for broad triage, deterministic scanners for high-assurance verification.

That recommendation is a direct quote from the paper: use language models "early in development for broad, context-aware triage, while reserving deterministic rule-based scanners for high-assurance verification." BrassCoders is the reserving-deterministic-scanner half. It bundles 12 scanners — Bandit, Pylint, Pyre/Pysa, Semgrep, ast-grep, detect-secrets, plus six custom detectors — and every finding it emits carries a file, a line, a severity, and an evidence string that reads the same on every run.

The model's recall doesn't get thrown away. It gets applied after the reproducible gate, on a YAML file the assistant reads instead of on raw source it has to re-scan. BrassCoders reports the pattern; the assistant judges the context. That's the division the numbers keep pointing at.

BrassCoders' OSS core is Apache 2.0, free, and makes zero outbound network calls. The Paid plan adds AI-powered enrichment for $12 a month. Point it at a project and read the YAML it writes:

pip install brasscoders
brasscoders scan /path/to/your/project

Scan, Patch, Re-Scan: Verifying AI Bug Fixes

CopperSunDev — Sat, 29 Aug 2026 23:45:44 +0000

Your AI assistant wrote the bug. Now it's proposing the fix, and if you let it, confirming its own fix worked. That last step is where the loop breaks. A verifier has to be independent of the thing it checks, and a language model grading its own patch is neither independent nor repeatable. BrassCoders, the scanner that catches what AI assistants structurally miss, is the independent check: it scans deterministically, so the same code yields the same findings every run, and a re-scan after the patch tells you whether the finding is actually gone.

The loop has three moves: scan, patch, re-scan. The third move is the one most AI workflows skip.

Why an LLM Can't Verify Its Own Fix

BrassCoders re-scans the patched file with the same rules that produced the original finding, so the verification comes from a system that took no part in writing the fix. An LLM asked to confirm its own patch samples a new answer each time, and the same diff can read as fixed on one run and unresolved on the next.

None of this is a criticism of the model's coding. It reflects what the two tools are. A language model generates text by sampling from a distribution, so its output shifts between runs. A rule-based scanner matches patterns in an abstract syntax tree and returns the same hits for the same input. When the writer and the checker are the same sampling process, a green light means the model didn't happen to flag the problem this time, not that the problem left the code.

The measurement backs this up. A 2025 benchmark, Large Language Models Versus Static Code Analysis Tools by Damian Gnieciak and Tomasz Szandala, ran three LLMs against three static analyzers on real vulnerabilities. The models scored higher on F1 through recall, yet they mislocated findings at line-and-column granularity because of tokenization. The authors' recommendation lands on the loop directly: use language models early for broad, context-aware triage, and reserve deterministic rule-based scanners for high-assurance verification.

Step 1: Scan and Read the Findings File

BrassCoders writes its findings to .brass/ai_instructions.yaml, a severity-ranked YAML file that carries a file path, line number, and detecting scanner for every finding. The command brasscoders --offline scan . produces that file with zero outbound network calls.

pip install brasscoders
brasscoders --offline scan /path/to/your/project

The scan runs BrassCoders' 12 bundled scanners against the project and collapses their output into one ranked file. Open .brass/ai_instructions.yaml and the executive_summary block gives you a risk_level and a prioritized recommendation. The security_critical block lists each finding with its file:line and a remediation pointer. That's the input your assistant needs.

Step 2: Hand the Findings to Your AI Assistant

BrassCoders formats .brass/ai_instructions.yaml for an AI assistant to read directly, no copy-paste of source required. An assistant like Claude Code or Cursor opens the file, reads the ranked findings with their file:line and remediation pointers, and proposes patches.

The division of labor is the point. BrassCoders is the deterministic pattern reporter. The assistant is the context-aware layer that reads the finding against the surrounding code and decides how to fix it. For a SQL injection built by string formatting, the assistant swaps the format for a bound parameter. For a hardcoded token, it moves the value to an environment variable. BrassCoders doesn't guess intent, and it doesn't write the patch. It reports what the rules matched and lets the assistant reason.

Step 3: Re-Scan to Verify the Fix

BrassCoders verifies the fix the same way it found the bug: run brasscoders --offline scan . again, and the SQL injection finding that flagged app.py either clears from .brass/ai_instructions.yaml on the next run or it doesn't. Same input, same findings, so a cleared finding means the pattern left the code, not that a model returned a kinder answer.

Run the loop until the re-scan comes back clean on the findings you meant to fix. A residual finding after a correct patch happens, and it's informative. Semgrep's taint rule, for one, keeps firing on a correctly parameterized SQLite call because it tracks user-controlled data to the execution sink without distinguishing the bound-parameter form. That's where the assistant reads the surviving finding, sees the parameter tuple in the code_snippet field, and marks it a false positive. The re-scan narrows the surface. The assistant closes it.

BrassCoders publishes a reproducible version of this drop. In its N=15 AI-code-findings corpus, a Flask endpoint generated from a one-line prompt ships SQL injection at line 22, and the first scan reports three findings on that line from two independent detectors plus Bandit. Apply the parameterized ? placeholder and re-scan, and the input-validation and Bandit findings clear. One Semgrep taint finding remains as the known false positive. Three findings become one triage decision, and the decision takes seconds. The corpus is Apache 2.0 and pinned to a commit, so the before and the after reproduce on your own machine.

Fail the Build When a Critical Survives

BrassCoders exits with a non-zero status (exit code 2) when you pass --fail-on-critical and any critical- or high-severity finding is present, so a CI step or a pre-commit hook fails the build deterministically. The exit code comes from the deterministic scan, so a build that fails today fails tomorrow on the same commit.

Add the flag to the scan step and the gate is one command:

brasscoders --offline scan --fail-on-critical .
# exit code 2 when a critical- or high-severity finding is present; 0 otherwise

For a custom threshold — a specific severity, or a count above some number — read the deterministic .brass/statistics.yaml instead of relying on the exit code:

brasscoders --offline scan .
python3 - &lt;&lt;'PY'
import yaml, sys
stats = yaml.safe_load(open(".brass/statistics.yaml"))
crit = stats.get("distribution", {}).get("by_severity", {}).get("critical", 0)
sys.exit(1 if crit else 0)
PY

A green run means no critical pattern survived the last re-scan. A red run points at the same file and line every time, so the team learns to act on it. That's the difference a deterministic gate makes: the check means the same thing on Monday that it means on Thursday.

Point branch protection at the job and the gate becomes a real merge block. A pull request that reintroduces a critical pattern can't merge until a re-scan comes back clean. The scanner ran the same rules on the same code, so nobody argues with the result.

The Hybrid Pattern the Research Points To

BrassCoders runs the deterministic detection pass and hands structured output to an AI assistant for judgment, the same split a 2025 framework called ZeroFalse reports reaching an F1 of 0.912 on the OWASP Java Benchmark and 0.955 on the OpenVuln dataset. Detection first, model judgment second.

ZeroFalse (Iranmanesh and colleagues, 2025) treats a static analyzer's output as a structured contract, enriches each finding with flow-sensitive traces and CWE-specific knowledge, then has an LLM adjudicate whether it's real. Recall and precision both land above 90% on the two benchmarks. The ordering is what matters: a deterministic tool decides what to look at, and the model decides what it means. The scan-patch-re-scan loop applies the same ordering to remediation. BrassCoders detects and re-verifies. The assistant reasons and fixes.

Run the Loop on Every Fix

BrassCoders makes the loop cheap enough to run on every change. The OSS core is Apache 2.0, free, and offline by default, so scanning and re-scanning cost nothing and send nothing off the machine.

pip install brasscoders
brasscoders --offline scan /path/to/your/project

Fix a finding, run the scan again, and read .brass/ai_instructions.yaml to confirm it's gone. The check that clears your AI's patch should be the one thing in the loop your AI didn't write.

AI Package Hallucination Rate: 96 Names Checked On PyPI

CopperSunDev — Sat, 29 Aug 2026 23:28:09 +0000

Ninety-six Python coding tasks. Ninety-six package names a frontier AI model handed back, each with the confidence it uses for import os. Ninety-five of those names exist on PyPI. One doesn't, and never has. BrassCoders, the scanner that catches what AI assistants structurally miss, ran that check against the live registry, and the single miss is the point of this post.

The probe was small and first-party: 96 package names one commercial frontier model produced across diverse Python tasks, each checked with a single request to the PyPI JSON API. The result was a roughly 1% hallucination rate, one invented name in 96. Set that against the number everyone cites: USENIX Security 2025 measured 19.7% across 16 models. Both numbers are real. This post is about why they differ, and why the low one still matters.

What The Probe Measured

BrassCoders generated 96 package names by prompting one frontier commercial model with 96 well-specified Python tasks, then checked each name against the live PyPI registry. Ninety-five existed. One, python-feature-flags, returned a 404.

The method is a shell loop, not a study. Each task named a concrete job — an async HTTP client, a Postgres driver, a PDF reader, a geocoder, a circuit breaker — and the model returned the package it would pip install. Every name went to https://pypi.org/pypi//json. A 200 means the package is registered and real; a 404 means it isn't. The corpus spanned mainstream libraries and a long tail of niche jobs, because the published research finds hallucination concentrates in the obscure corners.

Call this what it is: a replication in miniature, not a substitute for the large-scale work. It's one model, one pass, 96 names I generated deliberately rather than sampling hundreds of thousands of completions at temperature. The N is tiny next to USENIX's 16-model corpus. What it buys you is a number you can re-derive in about thirty seconds, and a single concrete failure to look at.

The One Package That Didn't Exist

BrassCoders found exactly one invented name, python-feature-flags, and nothing about it looks wrong: lowercase, hyphenated, semantically obvious. It's the kind of dependency you'd approve in a diff without a second look.

Here's the detail that makes it worse. For that same feature-flag task, the model also produced flagsmith, which is a real, published client. So within one job it emitted a real name and an invented one, side by side, with identical confidence. The invented name follows a pattern the registry has trained everyone to trust: python-dotenv, python-dateutil, python-slugify, python-docx are all real packages, so python-feature-flags reads as one more of the family.

That's the mechanic in one example. A language model completes patterns; it doesn't look anything up. A name that fits the shape of real package names gets generated whether or not the registry has ever seen it. The model has no channel to check and no signal that it's guessing. The failure is invisible precisely because the output is well-formed.

Where 1% Sits Against The Published Rates

BrassCoders measured about 1% because the probe used one commercial frontier model on well-specified, mostly-mainstream tasks — the lowest-hallucination corner of the whole problem. The published rates climb from there: 5.2% for commercial models and 21.7% for open-source models in the USENIX Security 2025 study, blending to a 19.7% headline.

The USENIX paper, by Spracklen and colleagues at UT San Antonio and collaborators, ran 16 code-generating models and catalogued 205,474 unique hallucinated package names. That last figure is the one that should stick. Each unique invented name is a slot an attacker can register. The 19.7% is the frequency of the opportunity; the 205,474 is the size of the target list.

Frontier models have improved, and the improvement is measured. A 2026 re-evaluation on the current cohort, The Range Shrinks, the Threat Remains, tested five frontier models across roughly 200,000 prompts and found rates between 4.62% and 6.10% — tighter and lower than the 2025 spread, still well above zero. My deliberate, well-specified single-package prompts land under even that range, which is what you'd expect from the easiest version of the task.

So three numbers, three conditions. My 1% is a floor: a careful model, common tasks. Around 5% is where frontier models sit under controlled testing. The 19.7% is the blended reality once you include open-source models and harder prompts. The rate is a function of model and task, and the plain reading is that it never reaches zero.

Why One Missing Import Is A Supply-Chain Problem

BrassCoders treats a non-existent import as a HIGH-severity finding, the same tier it assigns SQL injection, because an unregistered package name is a live attack surface the moment anyone registers it. A name that returns 404 today is a name an attacker can own tomorrow.

The attack chain is short. An AI assistant suggests a name that doesn't exist. A developer pastes the import in. An attacker who has watched which names models invent registers one on PyPI with a setup.py that runs code on install — Python executes install-time hooks with no sandbox, by design. The next pip install completes the attack. This is slopsquatting, and it exploits the one property that makes AI hallucination different from human typos: the model produces the same plausible name every time anyone asks the same question, so the target list is precomputable.

It's not theoretical. Bar Lanyado at Lasso Security noticed models repeatedly recommending huggingface-cli when the real install is pip install -U "huggingface_hub[cli]". He registered the hallucinated name as a benign, empty package. It drew more than 30,000 downloads over three months, and Alibaba pasted the invented command straight into a public repository README. The payload was harmless because Lanyado made it harmless. An attacker registering that same name would not. The pattern maps onto OWASP A08:2021, Software and Data Integrity Failures, with one added property: the AI-generated name is consistent, which makes the whole attack repeatable.

Run the math on the low rate. A team merging a few hundred AI-suggested imports a week, at 1%, ships a handful of unresolvable names every week. Each one is a coin flip on whether someone got there first.

How BrassCoders Flags It

BrassCoders runs a package-hallucination check that walks each Python file's imports with an AST parse, checks the local environment first, then queries the PyPI JSON API and flags any name that 404s at HIGH severity with its file and line number. The finding carries the package name, the import type, and a note that a malicious package under that name would be a supply-chain risk.

The check is opt-in for a reason. BrassCoders's OSS core is offline-first and makes zero outbound calls by default; the registry lookup is the single path that has to talk to the network, so you turn it on with --check-package-hallucination, and --offline overrides it back off. What leaves your machine is the bare package name, nothing else — no source, no project context, no telemetry. The same check covers npm and pkg.go.dev, so a hallucinated JavaScript or Go import gets caught the same way.

The result is deterministic. The name either resolves on the registry or it doesn't, and there's no model in the loop deciding what counts, so the same imports produce the same flags on every run. That's the division of labor BrassCoders is built on: the scanner reports the fact that a name doesn't resolve, and the AI assistant reading the YAML — Claude Code, Cursor, whatever you run — decides whether to swap the import, remove the code, or accept the risk. BrassCoders is the current release, 2.0.12, Apache 2.0, Python 3.10+.

Reproduce The Probe Yourself

BrassCoders built the probe as a one-line loop anyone can run: pass a list of package names to curl against the PyPI JSON API and flag every 404. On the 96-name corpus below, that loop returns exactly one miss, python-feature-flags.

while read -r name; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "https://pypi.org/pypi/${name}/json")
  [ "$code" = "404" ] &amp;&amp; echo "MISSING: $name"
done &lt; packages.txt

The full corpus, so the number is checkable and not just asserted:

requests httpx aiohttp fastapi djangorestframework pandas polars matplotlib
plotly scikit-learn transformers huggingface-hub tiktoken openai anthropic
langchain llama-index chromadb pinecone-client python-dotenv pydantic-settings
typer rich tqdm faker sqlalchemy alembic sqlmodel psycopg2-binary asyncpg
aioredis boto3 google-cloud-storage pillow pypdf pdfkit python-docx openpyxl
beautifulsoup4 playwright reppy requests-retry pyjwt passlib tenacity slowapi
pybreaker geopy pydub SpeechRecognition kafka-python loguru sentry-sdk rapidfuzz
orjson diskcache arq python-feature-flags flagsmith phonenumbers pycountry
better-profanity python-slugify humanize bleach deepdiff forex-python user-agents
strawberry-graphql tortoise-orm gino icalendar qrcode exifread langdetect
html2text asyncio-mqtt dateparser minio pyarrow python-magic respx email-validator
aiolimiter aioretry cachetools croniter humanfriendly openapi-python-client
sseclient-py jsonschema strictyaml redlock gql opentelemetry-api flag

The shell loop is the manual version, and it only catches what you remember to paste into it. BrassCoders runs the same check against every import in a scan, on every commit, and writes the flagged names into .brass/ai_instructions.yaml for your assistant to act on. Install it and point it at a repo your AI helped write:

pip install brasscoders
brasscoders scan --check-package-hallucination /path/to/your/project

One name in ninety-six is a rate you can live with right up until the week it's the name someone was waiting for.

AI Code Defect Rates: The Q3 2026 Data Report

CopperSunDev — Sat, 29 Aug 2026 23:26:36 +0000

As of Q3 2026, the rates are steady and they aren't small. 45% of AI-generated code samples ship an OWASP Top 10 flaw. AI-assisted pull requests carry 1.7x more issues than human-only ones. Almost one in five packages an AI recommends doesn't exist. BrassCoders, the bug scanner for AI coders, publishes this snapshot every quarter so the number you cite has a date on it.

This is the Q3 2026 edition. Every figure below comes from a named source with linked methodology, and each is labeled vendor report or peer-reviewed so you can weigh it. Five of the six are less than a year old. The next refresh lands in Q4 2026.

The Q3 2026 Rate Table

BrassCoders tracks six defect-and-security rates for Q3 2026, and the headline number is unchanged from earlier 2026 reporting: about 45% of AI-generated code samples still ship an OWASP Top 10 vulnerability. The rest of the table measures density, volume, supply chain, and the perception gap.

Here's the full set, each with its primary source and evidence type:

Metric	Q3 2026 rate	Primary source	Type
AI code samples carrying an OWASP Top 10 flaw	45%	Veracode 2025 GenAI Code Security Report	vendor report
Issues per PR, AI-assisted vs human-only	1.7x (10.83 vs 6.45)	CodeRabbit State of AI vs Human Code Generation	vendor report
Monthly security findings, Dec 2024 to Jun 2025	10x rise	Apiiro	vendor report
Privilege-escalation paths, same window	+322%	Apiiro	vendor report
AI-recommended packages that don't exist	19.7%	USENIX Security 2025	peer-reviewed
Developer speed with AI vs perceived speed	19% slower / felt 20% faster	METR 2025 field RCT	peer-reviewed

Each row gets its own section below, with the sample size and the caveat that rides with it.

Security Vulnerability Rate: 45% Of Samples

BrassCoders scans for the weakness classes Veracode measured, and the Q3 2026 rate holds at 45%: across more than 100 large language models tested on four languages, 45% of generated code samples introduced an OWASP Top 10 vulnerability.

The rate splits hard by language. Java fared worst at 72%. Python came in lowest at 38%, with JavaScript at 43% and C# at 45%. One weakness class stood out on its own: AI tools failed to defend against cross-site scripting in 86% of the relevant samples. The test was objective rather than a survey, each sample checked for a known vulnerability class. Source: Veracode's 2025 GenAI Code Security Report, a vendor report from a company that sells scanning tools, so weigh it as such. It's still the most-cited anchor for the security question, and the answer it gives is that nearly half the time, the code isn't safe.

Defect Density: 1.7x More Issues Per PR

BrassCoders runs in the pre-merge slot where the extra issues surface, the same slot CodeRabbit measured: AI-assisted pull requests averaged 10.83 issues each against 6.45 for human-only PRs, a 1.7x multiplier across 470 open-source GitHub pull requests.

The sample split 320 AI-co-authored PRs against 150 human-only ones. The averages hide a sharper detail: high-issue outliers were far more common on the AI side, which is where a reviewer's afternoon goes. That extra four-or-so issues per pull request is the triage load a human absorbs on every AI-assisted change. Source: CodeRabbit's State of AI vs Human Code Generation report, another vendor figure, so read it in context. The per-PR delta is the part that transfers, and it's what a deterministic first pass cuts before a human looks.

Security-Finding Volume: 10x In Six Months

BrassCoders matters more as commit velocity climbs, and Apiiro's numbers show the slope: by June 2025, AI-assisted repositories were generating over 10,000 new security findings per month, a 10x rise in six months, with privilege-escalation paths up 322%.

The tradeoff is the interesting part. AI assistants cut the easy problems, syntax errors dropped 76% and logic bugs 60%, while the hard ones climbed. Privilege-escalation paths rose 322% and architectural-design flaws 153%. Developers using AI exposed sensitive cloud credentials nearly twice as often as those coding without it. Source: Apiiro's analysis across thousands of developers and tens of thousands of repositories, a vendor report. More code shipped faster means more findings in absolute terms, and the mix shifted toward the severe end. Speed without a gate compounds.

Phantom Packages: 19.7% Don't Exist

BrassCoders flags imports that don't resolve before pip install runs, the defense against a defect class unique to AI: the USENIX Security 2025 package-hallucination study found 19.7% of packages recommended in LLM-generated code did not exist, rising to 21.7% on open-source models.

Commercial models did better at 5.2%, but none reached zero. Across 576,000 code samples the researchers logged 205,474 unique hallucinated package names, and the fakes weren't random: 43% reappeared on every one of ten repeated prompts. Repetition is what makes the attack work. Register a hallucinated name, wait for an AI to recommend it, and you've planted malware, an attack class the researchers named slopsquatting. Source: the USENIX Security 2025 package-hallucination study, peer-reviewed. An import either resolves against the registry or it doesn't, which makes this the cleanest deterministic check in the table.

The Perception Gap: 19% Slower, Felt 20% Faster

BrassCoders closes a gap the data keeps surfacing, that developers trust AI code more than the measurements warrant: in METR's 2025 randomized trial, experienced open-source developers were 19% slower with AI tools while believing they were 20% faster.

They'd expected a 24% speedup going in, and kept believing in a 20% gain even after finishing slower. The trial was small and specific, a cohort of experienced developers working on mature open-source repositories they already knew well, so it doesn't generalize to every team. It points the same direction as the security data, though: confidence in AI output runs ahead of its measured quality. Source: METR's randomized controlled trial, a field study rather than a vendor report. A deterministic gate doesn't argue with the confidence. It checks the code.

What BrassCoders Catches

BrassCoders catches the structural share of these defects for free and deterministically: 12 static-analysis scanners run against your AI-generated Python and JavaScript/TypeScript, emit findings as YAML, and never call out to the network on the Apache 2.0 OSS core.

Match the scanners to the table. Hardcoded secrets and weak crypto (the OWASP A02 class) go to Yelp's detect-secrets plus seven custom format patterns. Injection sinks (A03) go to Pyre's Pysa interprocedural taint analysis, which follows a tainted value across file boundaries the way an LLM's context window can't. Imports that don't resolve, the 19.7% phantom-package class, go to a custom AI-pattern detector that checks each import before pip install runs. BrassCoders produces the same output on the same input every run.

What it doesn't do is judge intent. BrassCoders reports the pattern match and stops; the context-aware call, whether a given match is a real bug or a false positive in this codebase, belongs to the AI assistant reading the YAML. Hand .brass/ai_instructions.yaml to Claude Code or Cursor with "address the critical issues in order," and the model triages the deterministic findings with the context brass deliberately doesn't infer. The free tier covers the scan. The Paid plan at $12/dev/month adds AI-powered enrichment that ranks the findings by project signature.

Why This Report Refreshes Quarterly

BrassCoders refreshes this snapshot every quarter because the denominator keeps moving: AI-assisted commit volume rises, model cohorts turn over, and a 45% rate measured against the 2024 model mix doesn't automatically describe the 2026 one.

The rates have been sticky so far. Veracode's 45% has held across its 2025 and 2026 reporting, and CodeRabbit's 1.7x and USENIX's ~20% likewise. Newer frontier models have narrowed the package-hallucination range without closing it. The per-unit rates move slowly. The volume they apply to moves fast, and Apiiro's 10x-in-six-months finding is that volume signal. A dated report keeps the citation defensible. When you quote a number, quote its quarter. The next BrassCoders edition lands in Q4 2026, and this coverage map lives in the AI Coding Assistant Blind Spots pillar.

Run the scan against your own AI-generated code and see which rows show up:

pip install brasscoders
brasscoders --offline scan /path/to/your/project

The OSS core is free, the scan finishes in under a minute on most projects, and the YAML tells you where your codebase sits against the Q3 2026 table.