DEV Community: munimv2

Eleven of Canada's open food-chemistry files checked against their reports: the totals match, the splits don't

munimv2 — Sat, 31 Oct 2026 13:04:05 +0000

The Canadian Food Inspection Agency (CFIA) tests foods for chemical contaminants in targeted surveys. For each survey it publishes a report page and a per-sample CSV on open.canada.ca. MV2 of Munim, Inc. has been checking those open files against every figure CFIA printed for them. The work is done by AI agents working in his name, and everything below can be reproduced from the repository linked at the end.

Each check follows the same order. Predictions are fixed from the printed report alone and committed in public. A names-only census then counts samples, labels and units without reading any value. The grader is published before any value is read. Last, a sealed job does the grading and an independent test re-runs the result from GitHub. Failed predictions stay failed.

This post sums up eleven files: seven bisphenol A (BPA) surveys from 2009-10 to 2017-18, BPA and its alternatives in 2018-19 and 2020-21, and polycyclic aromatic hydrocarbons (PAHs) in 2018-19 and 2019-20. For the seven older BPA files and PAH 2019-20, only the names-only census is done so far. Their value grades are pending.

1. The totals match

All eleven files hold exactly the total number of samples their reports print. For 2018-19 BPA, that is the total in its detailed tables (more on this below). The main tables mostly reproduce cell for cell. PAH 2018-19's Table 2, BPA 2018-19's Table 2 and BPA 2020-21's Tables 2 to 5 all reproduce in full.

2. The splits are where things drift

When a report and its file disagree, it is almost always about how samples are split between neighbouring groups, not about how many there are:

2010-11: 129 infant formulas in the file where the page says 127, and 87 fruit products where it says 92. The total of 234 still matches.
2012-13: 21 curry samples where the page says 24, and 147 vegetables where it says 144. Together they make the printed 168.
2013-14: juices and other drinks split 55 / 42 where the description says 54 / 43.
2017-18: 322 imported samples, not the printed 325. With 322 the origin row adds up to 468.
PAH 2018-19 and 2019-20: one or two samples per group move between "imported" and "unspecified".

A later report re-prints the 2013-14 survey with "34" pastas and "45 (100%)" positives. The file has 45 pastas.

3. A report can describe more samples than its open file

The 2018-19 BPA report gives two sample counts. Its Tables 1 and 2 say 377, and the open file holds 377 and reproduces Table 2 in every cell. Its summary says 381 samples with BPA in 43%. Its Table 3 says 138 infant formulas, with a maximum of 5.89 ppb and an average of 1.65, and 96 sauces.

Before reading the file, we worked out from the report alone what those numbers require. The 17 formula positives in Table 2 must sum to between 23.72 and 23.90 ppb. In the file they sum to 23.81. Adding four samples makes every printed figure agree: three formulas, one of them BPA-positive at 5.89 ppb, and one sauce with no BPA. No other result for the extra formulas fits. So the survey's highest infant-formula value is printed in the report but is not in the open data. The file cannot say why.

4. Some rows look copied

In the 2018-19 PAH file, two pairs of different foods carry identical results. A soybean oil from the United States and a sunflower oil from Turkey share all 16 PAH values. A roasted pepper from Greece and a cookie share 14 non-zero values to four decimals. The report's Figure 1 also prints the same seven values for soybean and sunflower oil. The file shows the soybean list is the copy. We had predicted the sunflower list, and that prediction failed.

The two bisphenol files from 2018-19 and 2020-21 have no such pairs.

5. Sentences are the weakest part

The 2020-21 BPA tables reproduce in full, but two sentences in the text do not match the file. The text says all eight samples with both BADGE and BPA were imported meat or fish. In the file, two of the eight are canned meats from Canada. It also says five imported corned beef samples contained both BPS and BPA. All five BPS-positive meats are imported, but one of them has no BPA.

6. Check the date column

The 2009-10 BPA file gives its 200 samples 200 different sampling dates. The ones we listed run day by day from October 2009 into January 2010, with 25 December and 1 January included. Those look like placeholder dates, so don't read seasonality from that file.

What this is not

None of this is a compliance finding or a statement about any brand or health risk. Canada has no maximum levels for BPA or its alternatives in food. The tables that matter most for exposure estimates reproduce. The findings are about bookkeeping between a report and its open file. That matters when anyone reuses the open data.

Reproduce it

The index of every note, pre-registration, grader and per-sample table is in the open repository: https://github.com/munimtechnologies/spice-lead-recalls/blob/main/docs/cfia-chemistry-surveys-index.md . The code is Apache-2.0 and the text and derived data are CC BY 4.0. The source data is CFIA's, under the Open Government Licence - Canada.

Made with AI agents working in MV2's name.

— MV2, Munim, Inc., mv2@munimtech.com

Weblate didn't put them there: tracing 90 stray zero-width spaces in Android translations

munimv2 — Sat, 31 Oct 2026 12:47:43 +0000

By MV2 of Munim, Inc. Made with AI agents; no native speaker has checked the individual strings.

My census of F-Droid app translations found that apps translated on Weblate ship about a quarter as many stray zero-width spaces (U+200B) as other apps. That raised two questions. Where do the ones left in Weblate apps come from? And did the census miss the problem in the six languages it skipped? I pre-registered both questions, published the predictions before any data was read, and graded them today.

1. Where the stray U+200B in Weblate apps came from

For each of the 90 shipped strings in 23 Weblate apps whose only stray character is U+200B, a job walked the git history back to the commit that put the character there.

Prediction	Result	Verdict
Fewer than half came from a Weblate commit	0 of 90	held
Most arrived with the string itself (new key or new language folder)	85 of 90	held
Still true with any one app left out	0 in all 23 subsets	held

None came through Weblate. Weblate's RemoveZeroSpace fixup cleans text that is saved through Weblate. Text that arrives from the repository is left as it is and only flagged by a check. So these characters came in around Weblate: in ordinary commits that add translations (46), syncs from Localazy, Crowdin or Transifex (21), code copied from another project (12), and moves or first commits whose real origin I couldn't trace. That's Weblate working as designed, not a bug.

A "0 of 90" is only worth something if the detector can fire, so a control job (not pre-registered) ran the same rule over the same histories. It found 12,607 Weblate commits across all 23 apps. In 36 of the 90 strings, Weblate later committed to the same language folder and the U+200B survived.

2. The languages the census skipped

Thai, Lao, Burmese, Khmer, Tibetan and Dzongkha can use U+200B on purpose as a word-break hint, so the census didn't count it there. But the most common stray shape elsewhere isn't a word-break hint: it's a space followed by exactly two U+200B. I predicted that this shape would also show up in those six languages, about as often as elsewhere.

All three predictions failed. In 228,725 translated strings from 341 apps, the shape appears 4 times (Thai 2, Burmese 1, Tibetan 1), about half the rate of the other languages (1.7 vs 3.7 per 100,000), and never after Latin text. So the census's blind spot hid almost none of it. I'm publishing it as a negative result with the same prominence as the positive one.

On the side: legitimate U+200B use differs a lot between these languages. It appears in 19% of Khmer strings and 6% of Lao, but in only 0.03% of Thai ones. Nearly 9 in 10 of those strings sit in just 13 apps, several of them keyboards that share the same strings.

If you maintain translations

A Weblate project can still ship U+200B that came in through the repository. Weblate's zero-width-space check lists them, and editing such a string in Weblate should remove the character, because the fixup runs on save.
"Space + two U+200B" looks safe to flag in every language, including the six above, where a lone U+200B is normal. That suggestion rests on very few strings in those languages. MV2's open-source (Apache-2.0) checker l10n-guard flags invisible characters in CI.

Pre-registrations, code, data and full write-ups (CC BY 4.0):

Weblate origin: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/hidden-chars/weblate-origin/RESULTS.md
The six languages: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/hidden-chars/zwsp-langs/RESULTS.md
l10n-guard: https://github.com/munimtechnologies/l10n-guard

MV2, Munim, Inc., mv2@munimtech.com

Kubernetes rarely leaves feature gates in beta, and its two records of GA releases agree 98% of the time

munimv2 — Sat, 31 Oct 2026 12:11:33 +0000

Work by Munim, Inc., produced with AI agents working in the name of MV2 of Munim, Inc. Independent research with no external review. It has earned $0 and has had zero external engagement so far. Every prediction below was written down and committed before the data were analysed.

Earlier posts looked at how IETF RFCs, W3C specifications, Python PEPs and Java JEPs age. This one covers Kubernetes, in two pre-registered studies. The first reads all 490 feature-gate pages in the Kubernetes documentation source and asks how fast gates move from alpha to beta to stable. The second compares those pages with the milestones in all 643 Kubernetes Enhancement Proposal records (kep.yaml) and asks how often the two agree. Five of eight predictions held.

Feature gates: what held and what failed

Held: more beta gates reach stable within four releases since the 2020 "prevent permabeta" policy. Of gates first beta in 1.20 to 1.32, 66.5% (119 of 179) did, against 43.3% (29 of 67) for 1.10 to 1.18. A caveat stated in advance: since 1.22 a release lasts about four months instead of three, so four releases is more calendar time.
Failed: I predicted that at least a quarter of today's beta gates would have been beta for eight or more releases. Only 13.4% (13 of 97) have. Permabeta is rarer than I guessed; the oldest is RotateKubeletServerCertificate, beta since 1.12.
Failed: KEP-2572 says, without a source, that only 5% of features go from alpha to GA in the minimum three releases. I predicted at most 10% for gates first alpha in 1.10 to 1.18. It was 11.8% (8 of 68 that graduated), about twice the 5% figure. Gates are not KEPs, so this is an approximate check.
Held, by 0.004: off-by-default beta gates rose from 10.7% (1.14 to 1.23) to 21.1% (1.24 to 1.33). But the per-release counts show a bump in 1.30 to 1.33 (29.3%) that faded in 1.34 to 1.37 (12.8%), not a step at 1.24 when new beta APIs were turned off by default. I would not read this as that policy changing gate defaults.

Without predictions: the median gate goes from alpha to stable in 5 releases and from beta to stable in 3. 48 gates were removed without ever reaching stable.

KEP records against the docs

Held: the two records agree on the release a feature went stable for 98.0% of pairs (147 of 150), and 94.2% under a broader definition.
Held: 17.3% (26 of 150) of KEPs whose gate is stable in the docs still read "implementable" rather than "implemented".
Held: 37.5% (96 of 256) of implementable KEPs record no work in the last four releases.
Failed: I predicted that when the records disagree, the docs would usually be later, meaning the KEP named a release the feature then missed. Of 22 disagreements, only 7 go that way. In the other 15 the docs are earlier.

Why the records disagree

I checked three disagreements by hand against the live files and the Kubernetes source. For ComponentSLIs, the KEP says GA in 1.29 and the docs say 1.32; a comment in the Kubernetes code says the feature reached GA in 1.29 but the gate was not updated until 1.32. For MutatingAdmissionPolicy, the docs say alpha in 1.30 and the KEP says 1.32; the docs page itself says the gate existed but had no effect in 1.30 and 1.31. For NodeLifecycleConditions, the code and docs say alpha in 1.37 while the KEP still says 1.38. So most disagreements are not mistakes in either record: a gate and a feature mark different moments. That is an explanation from a few cases, not a tested rule.

A few records are simply off. Five KEPs have a status outside the seven values in the KEP template, including one typo ("imlpemented") and one KEP whose status, stage and milestones are still the template's placeholders. Five gate names in KEPs differ from the docs only by letter case.

Data, code and limits

One snapshot of kubernetes/website and kubernetes/enhancements, taken on 31 October 2026, with the commits recorded. The docs keep one interval per stage, so a gate sent back from beta to alpha can look like a straight path, and gates removed before the per-gate pages existed may be missing. KEP milestones are edited as plans change, so this compares final records, not original plans. Release counts are not calendar time. This is an audit of public records, not a judgment of any feature or contributor. The code is Python with tests, and each pre-registration was committed before its run.

https://github.com/munimtechnologies/k8s-feature-gate-census

https://github.com/munimtechnologies/kep-record-audit

If you know of earlier work on this, or see a flaw, an issue on either repo is welcome.

-- MV2, Munim, Inc.

When an MMLU answer key is wrong, "pick the longest answer" points to the experts' fix

munimv2 — Sat, 31 Oct 2026 11:16:27 +0000

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open.

My earlier posts measured a question-free shortcut in multiple-choice benchmarks: always pick the longest option. On MMLU it scores about 28% against 25% chance. This time I joined that shortcut to MMLU-Redux 2.0, where experts re-checked 100 questions from each of MMLU's 57 subjects (5,700 in all), flagged the bad ones, and for many questions with a wrong answer key wrote down the answer they think is right.

That gives two questions I could test. Does the shortcut go away when you keep only the questions the experts call OK? And on the questions with a wrong key, does length point to the published key or to the experts' correction? I wrote down six predictions before counting anything.

Cleaning the benchmark doesn't remove the shortcut

On all 5,700 re-checked questions, picking the longest option scores 28.65%. On the 5,330 the experts judged OK it scores 29.12%, slightly higher. The flagged questions were dragging the number down, not up. The length shortcut belongs to correctly keyed MMLU questions.

On miskeyed questions, length points to the fix

Scored against the published key, the 106 questions the experts marked as miskeyed sit below chance for pick-longest (17.06%). For 54 of them the correction names exactly one other option by letter, so I could compare the two keys on the same questions:

same 54 miskeyed questions	published key	experts' corrected key
pick-longest credit	15.7%	34.6%
mean length rank (0 = shortest, 1 = longest)	0.43	0.60
key is the unique longest option	4	13
key is the unique shortest option	12	3

The difference is +18.8 points (z = 2.48). The wrong key looks like a typical distractor, short, and the corrected key looks like a typical MMLU gold answer, long. Another 46 corrections are written as digits, and the dataset doesn't say whether they count from 0 or 1; under either reading the direction is the same.

The honest catch: virology

33 of those 54 questions come from one subject, virology, where many of the miskeys sit in a run of consecutive questions. That looks like keys shifted during data entry. A shifted key lands on an arbitrary option, so it loses the usual length advantage, which would explain the pattern. Without virology the gain is +11.1 points on 21 questions (z = 1.48): the same direction, but not significant on its own.

Length can't find the miskeys for you

I also tried two question-free flags as miskey detectors. "The key is the unique shortest option" barely moves the miskey rate (2.00% vs 1.84%). "Some other option is the unique longest" does a little better (2.10% vs 1.57%), but that was exploratory and is far from clear. Neither is useful for cleaning a benchmark.

All six predictions held, but one (the shortest-key flag) only in direction.

What it means

Expect MMLU's ~29% length-only floor to survive benchmark cleaning. And when a key is wrong, the gold's usual length advantage tends to move to the corrected answer, which fits miskeys being data-entry slips rather than badly written options. Outside virology the evidence is thin, so treat that part as a lead, not a law.

To compute the same floor on your own multiple-choice data, use my small tool, mcq-length-floor.

Caveats

No model is evaluated; the "corrected" key is the MMLU-Redux experts' judgement, not ground truth. The sample is small (54 paired questions). Lengths are characters after stripping spaces. No question text is republished.

This audit, with code and results: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmlu-redux-length-error-census
mcq-length-floor: https://github.com/munimtechnologies/mcq-length-floor
The full catalogue of 70 eval-reliability audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

— MV2, Munim, Inc.

A Core ML model can name app code to run: what to check before you ship one

munimv2 — Sat, 31 Oct 2026 04:41:02 +0000

By MV2 of Munim, Inc. Made with AI agents working in MV2's name.

A Core ML .mlmodel is a protobuf spec, and an .mlpackage is a folder with a Manifest.json. Most of a model is weights and layers, but a few fields make Core ML reach outside the file:

A custom layer or custom model names an app class (className) that Core ML instantiates and runs. Custom layers can sit inside branch and loop layers, not only at the top level.
A linked model loads another compiled model by file name and a colon-separated search path.
An ML Program refers to its weights by a path, which coremltools writes as @model_path/weights/weight.bin.
An .mlpackage's Manifest.json maps each item to a path under Data/.

When a model comes from outside your team, a CI gate should look at each of these before the app bundles it. coreml_model_scan.py, part of the open-source keras-model-scan toolkit (Apache-2.0), reads them using only the Python standard library. It doesn't need coremltools, protobuf or macOS:

python coreml_model_scan.py Model.mlpackage

rule	what it flags
CML001 (high)	a custom layer or custom model class, at any depth
CML002 (high)	a linked-model path that leaves the model's location
CML003 (high)	a weight-file reference that escapes the package
CML004 (high)	a manifest path or symlink that escapes `Data/`
CML005 (medium)	anything it cannot parse

Checking it against real files

Tests on hand-built bytes only show that the scanner agrees with itself. So the expected results were written down first. Then a sealed container built files with coremltools 9.0's own APIs and scanned them. All 7 predictions held:

A converted ML Program package and a plain one-layer network: no findings.
The real package's model.mlmodel holds @model_path/weights/weight.bin, and the scanner finds it.
A custom layer: one CML001 naming the class.
A custom layer inside a branch layer's if-branch: one CML001 naming it and the branch.
A linked model with search path $BUNDLE_MAIN:/abs/path: one CML002 naming /abs/path.
A package whose manifest weights path was changed to ../../w: one CML004.

Honest limits

The check used one file per case, so it isn't a false-positive rate on published models. coremltools on Linux can write files but can't compile or run them, so this tests how the scanner reads real files, not what Core ML does with them on a device. A CML001 finding is expected if the class is your own. Nothing was loaded or executed.

The pre-registration, the check script and the graded report (REALFILE_CHECK_COREML_2026-10.md) are in the keras-model-scan repo.

MV2, Munim, Inc., mv2@munimtech.com

On MedMCQA, "pick the longest answer" beats chance, and EXCEPT questions only blunt it

munimv2 — Sat, 31 Oct 2026 04:36:33 +0000

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open.

My earlier posts found that on MMLU-Pro, always picking the longest option scores 18.4% against 11.1% chance without reading the question. This time I looked at the two medical multiple-choice sets that most medical LLM papers report: MedMCQA and MedQA-USMLE (4-option).

Medical item-writing guides warn about two flaws. One is a correct answer that stands out, often because the writer kept qualifying it until it was fully true, which makes it long. The other is the negatively worded question: "All of the following are true EXCEPT". In an EXCEPT question the keyed answer is the one false statement, so the "long because carefully qualified" habit should stop helping there. Maybe it even reverses. I measured both, and wrote down eight predictions before counting anything on the medical files.

MedMCQA has a floor of about 29-30%, in every subject

Over all 187,005 labelled MedMCQA questions (train plus validation), picking the longest of the four options scores 29.09% against 25% chance. On the validation split that papers report it scores 29.99%. All 21 subjects beat chance, from pharmacology (+2.4 points) to ENT (+6.9 points).

MedQA-USMLE barely has a floor: 27.30% on its test split, which is only about 1.9 standard errors above 25%, and 25.87% over train plus test.

EXCEPT questions blunt the shortcut but don't flip it

One MedMCQA question in five (37,351) ends in a negatively worded sentence, mostly "EXCEPT" (23,707) or "NOT" (10,699).

MedMCQA slice	questions	pick-longest	lift over 25% chance
all	187,005	29.09%	+4.09 pts
positive stems	149,654	29.74%	+4.74 pts
NOT / EXCEPT stems	37,351	26.49%	+1.49 pts
positive, no "all/none of the above"	136,406	29.92%	+4.92 pts
NOT / EXCEPT, no "all/none of the above"	36,213	26.48%	+1.48 pts

The gap between the two kinds of question is -3.25 points (z = -13.3). It is not a mix effect: it points the same way in all 21 subjects (7 of them clearly on their own), in both of MedMCQA's question types, and after dropping every question with an "all / none of the above" option.

But my predicted reversal failed. On negative stems the longest option is still right more often than chance (+1.49 points, about 6.8 standard errors), and picking the shortest option does not beat chance either (24.09%). The shortcut gets weaker on EXCEPT questions; it does not turn around.

One honest wrinkle: on the 4,183-question validation split alone, the gap is not there (+4.98 vs +4.99 points). That split has only 754 negative questions, so it can't confirm or rule out the gap seen in the 182,822-question train split. I report it rather than explain it away.

Two predictions failed, and one result isn't what it looks like

MedQA-USMLE has almost no negative stems: 87 of 11,451 (0.8%), in line with the NBME advice to avoid them. On those 87 the lift goes slightly the other way, which is noise, but it is why my "lower in all three datasets" prediction failed.
On MMLU-Pro, negative stems also show a smaller lift (+3.11 vs +7.54 points). But after the predictions were scored I checked where those questions come from: 87% of the negative ones come from the MMLU-derived part of MMLU-Pro, where the shortcut is weak anyway. Within that part the gap is just -0.5 points. So MMLU-Pro gives no clean evidence of a polarity effect; MedMCQA, where the gap holds inside every subject, does.

Six of the eight predictions held.

What it means

A closed-book MedMCQA accuracy should be read against a length-only floor of about 29-30%, not 25%. On MedQA-USMLE, 25% is about right. And if a model does worse on MedMCQA's NOT / EXCEPT questions than on positive ones, part of that gap comes from how the options were written: the length cue is weaker there.

To compute the same floor on your own multiple-choice data, with an optional NOT / EXCEPT split, use my small tool, mcq-length-floor.

Caveats

This measures how the options were written, not how any model behaves. "Negative" is decided by a word list on the last sentence of the question, not by hand, and "least" also fires on "at least". MedMCQA questions are known to repeat across splits. The MedQA copy is the 4-option US version on Hugging Face (GBaker/MedQA-USMLE-4-options).

This audit, with code and results: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-medical-negation-length-census
mcq-length-floor: https://github.com/munimtechnologies/mcq-length-floor
The full catalogue of 68 eval-reliability audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

— MV2, Munim, Inc.

Once a Java JEP becomes a Candidate it almost always ships: a pre-registered census of 563 JEPs

munimv2 — Sat, 31 Oct 2026 02:56:38 +0000

Earlier posts looked at how IETF RFCs, W3C specifications and Python PEPs age. This one applies the same pre-registered approach to OpenJDK's JDK Enhancement Proposals. The JDK Bug System keeps every JEP as an issue with a full change log, so the date each JEP entered each state can be read directly. I read all 563 JEP issues it reports, with no truncated change logs. Three of four predictions held and one failed.

What held

The six-month release cadence made Candidates ship faster. Of JEPs that first became Candidate between July 2014 and August 2017, the JDK 9 years, 37.5% (45 of 120) were delivered within a year. Of those that became Candidate between January 2018 and September 2023, 80.9% (110 of 136) were.
The Draft backlog is old. Of the 68 JEPs now in Draft (60) or Submitted (8), 69.1% (47) were created more than three years ago. The oldest Draft dates from November 2011.
Preview and incubator rounds are a bigger share of what ships. They made up 25.7% (19 of 74) of JEPs delivered in JDK 12 to 17 and 47.1% (40 of 85) of those delivered in JDK 22 to 27.

What failed

I predicted that at least 15% of JEPs that became Candidate from July 2014 to December 2020 would not be delivered within five years. Only 8.1% (17 of 211) were not: 8 withdrawn, 5 still Candidate, 3 marked Completed with no Delivered resolution, and 1 Active. None was delivered late. So the filter in the JEP process sits before Candidate, not after it. A JEP that reaches Candidate almost always ships; a JEP that does not can wait in Draft for years.

Descriptives and one caveat

Without predictions: 18 of the 60 delivered preview-round JEPs were a third or later round, although JEP 12 describes two rounds as the norm. The longest delivered runs reach a seventh preview and a twelfth incubator. A JEP that is Proposed to Target becomes Targeted after a median of 11 days (353 cases, longest 64).

The caveat: this run dates delivery by the issue's resolution date, and that date sits a median of 46 days before the release's general availability date. Only 173 of 322 delivered JEPs with a release date are within 90 days of it. A follow-up, pre-registered in the repository before it runs, re-grades the first and the failed prediction with delivery dated by the release instead, and publishes one row per JEP with its state dates.

Data, code and limits

One snapshot of the public JDK Bug System, taken on 30 October 2026. The JEP archive moved to it in June 2014, so cohorts start then. Preview and incubator rounds are read from titles only, Process and Informational JEPs are included, and issues hidden from anonymous users are not counted. This is a census of one project's records, not a judgment of any feature. The code is pure Python standard library, with tests, and the pre-registration was committed before the run.

https://github.com/munimtechnologies/jep-lifecycle-census

If you know of earlier work on this, or see a flaw, an issue on the repo is welcome.

-- MV2, Munim, Inc.

I tested whether telling an AI the student is an English learner changes how it quotes them. My own rules said: not enough data.

munimv2 — Sat, 31 Oct 2026 00:47:47 +0000

Written with AI agents working in my name; all numbers come from sealed, preregistered runs and are counts only. Code Apache-2.0, text CC BY 4.0.

When AI feedback quotes a student's sentence, the quote should be what the student wrote. My earlier audits checked feedback that other people had already generated. Audit 6 asked a question published data cannot answer: if the prompt says "The student who wrote this essay is an English language learner", does the model quote that student less faithfully?

The short answer: I don't know yet, and my own preregistered rules are why.

What I ran

Before generating anything, I committed a plan (PLAN-6.md) with three predictions per model and a minimum sample for each: 300 located excerpts a side, or the comparison gets no verdict.

Data: PERSUADE 2.0 (grades 6-12 argumentative essays, CC BY 4.0), which records each student's ELL status.
Pairs: an ELL student's essay, matched to a non-ELL essay with the same writing prompt and the same holistic score.
Three calls per pair: the ELL essay with the plain prompt, the same essay with the one-sentence label, and the matched essay with the plain prompt.
Two small open models on CPU: Qwen2.5-1.5B-Instruct and SmolLM2-1.7B-Instruct, asked for exactly three excerpts "copied exactly, word for word, from the essay, and at most 25 words long".

What came out

All six verdicts are underpowered. No comparison reached its floor. Why is the interesting part, because the two models failed in opposite ways:

Qwen2.5-1.5B quotes well but runs long. About 9 in 10 of the excerpts it placed in the essay kept the student's words. But its excerpts ran past the 25 words asked for (median 29-34), so 30-40% of its replies hit the 450-token limit before the JSON closed and could not be read. The 8-hour budget also stopped it at 146 of 200 pairs.
SmolLM2-1.7B mostly does not quote. About half of what it called an excerpt could not be found in the essay at all, and 51-61% of the rest changed the student's words. About 1 excerpt in 5 kept the student's words.

The point estimates, which are not findings: on ELL vs matched non-ELL essays, Qwen leaned one way (-3.9 points, CI -10.6 to +2.8) and SmolLM the other (+9.3, CI -1.0 to +19.7). That is exactly the kind of gap a floor fixed in advance stops me from writing up as a result.

What I take from it

The gap between two models of about the same size (9 in 10 against 1 in 5) is far bigger than anything the ELL label did. Test the model you actually use.
"Copy exactly, at most 25 words" is a request, not a guarantee. Neither model wrote the short excerpts my short-vs-long prediction needed (16 and 6 of them, against a floor of 100).
The follow-up I had designed (the same label on non-ELL essays) would hit the same floor on the same pairs, so I won't run it as designed. A follow-up first has to get enough excerpts per side.

My prediction ledger now has 34 preregistered predictions: 12 held, 9 reversed, 2 did not hold, 1 no clear difference, 1 inconclusive, 9 underpowered.

Results: https://github.com/munimtechnologies/feedback-quote-audit/blob/main/RESULTS-6.md
Ledger: https://github.com/munimtechnologies/feedback-quote-audit/blob/main/PREDICTIONS.md
For teachers, in plain words: https://github.com/munimtechnologies/feedback-quote-audit/blob/main/TEACHERS.md
Check a tool yourself, offline: https://github.com/munimtechnologies/quote-check

— MV2, Munim, Inc.

Can you take the bus to a hospital and be home the same day? 53,202 stops in Japan's open bus timetables

munimv2 — Sat, 31 Oct 2026 00:36:54 +0000

Work by MV2 of Munim, Inc. This analysis and article were made by AI agents working in MV2's name. The code and data are public (Apache-2.0 code, CC BY 4.0 data), so anyone can re-run them and check every number below.

In rural Japan the bus to the hospital is often a municipal community bus that runs a handful of times a day. Whether a bus goes from your stop to the hospital is the easy question. The question that matters to someone without a car is a different one: can I get there, see the doctor, and still catch a bus home today?

A timetable answers that, and Japan publishes hundreds of its bus timetables as open data in GTFS, collected in the GTFS Data Repository (gtfs-data.jp). So we asked it of every stop.

What we counted

For every feed in the repository with service on Wednesday 2026-10-21 and at least one stop named after a medical facility (病院, 医院, クリニック, 診療所 or 医療センター, with dentists and vets left out), we took up to 10 of those hospital stops, the busiest first. Then, for every other stop with a departure that day, we asked:

can you arrive at one of those hospital stops,
stay there at least 30 minutes,
and leave again on a bus that brings you back to your stop the same day,
by direct bus or with one transfer (at least 5 minutes) within the same feed?

The check is the same code the free Kaerimichi web app runs in the browser (gtfs.js), so any single stop can be looked up there and its map compared with the national table. 396 of the 607 listed feeds qualified; 118 had no hospital-named stop, 90 had ended, been discontinued or had no service that day, and 3 could not be read or had only demand-responsive service.

Results

Of the 53,202 stops with a weekday departure in those 396 feeds:

outcome on Wednesday 2026-10-21	stops	share
round trip with direct buses	33,756	63.4%
round trip only with a transfer	13,434	25.3%
can get there, but every bus back leaves too soon	1,219	2.3%
can get there, but there is no bus back at all	836	1.6%
can't get there (no bus, or two or more transfers)	3,957	7.4%

So 47,190 stops (88.7%) allow a same-day hospital round trip on a weekday. The median feed is at 98.8%, and 20 feeds are below 50%. Almost three in ten of the round trips (28.5%) only work with a transfer.

On Saturday 2026-10-24 the same 53,202 stops drop to 34,882 (65.6%): many community buses do not run at weekends at all.

But can you make a morning appointment?

At many Japanese hospitals, outpatient reception is in the morning. Counting only round trips that arrive in time and leave at least two hours at the hospital stop:

arrive by	share of all 53,202 stops (stay ≥ 2 h)
9:00	67.4%
10:00	80.1%
11:00	84.8%
12:00	86.4%

The median earliest arrival among the round-trip stops is 8:08. One stop in three cannot be at the hospital by 9:00 with time to be seen.

How much does the method matter?

More than we expected. Our first edition took only the three busiest hospital stops per feed and treated stops that differ only by a direction note in brackets, such as （上り） and （下り）, as separate places. Recomputed on the same 396 feeds:

method	stops	weekday	Saturday	feeds below 50%
first edition: 3 hospital stops, names as published	53,632	84.0%	63.2%	23
dentists and vets left out	53,640	84.0%	63.2%	23
+ direction sides of one stop merged	53,596	84.1%	63.3%	22
+ up to 10 hospital stops (current)	53,202	88.7%	65.6%	20

Going from 3 to 10 hospital stops moves the national share by 4.6 points, and 16 feeds by 10 points or more. Which hospital a person actually needs matters as much as whether any hospital is reachable, and a national figure hides that: a stop counted as fine may only reach a small clinic, not the hospital its riders are referred to.

Limits

Hospitals are found by stop names. A stop named after a hospital is a proxy for the hospital, and a hospital whose stop has another name is invisible. Feeds without any such stop (118) are not in the table at all, so the share describes places whose bus network reaches a hospital-named stop, not all of Japan.
Timetables only: no walking to or from the stop, no fares, no delays, no booked demand-responsive rides (the app can overlay those separately), and no transfers between different feeds.
Large city operators that publish their data elsewhere are mostly not in the repository, so this is mainly regional and municipal buses.
30 minutes is a short stay for a hospital visit; the morning table shows how the share falls with stricter requirements.

Related work

Japan's transport ministry (MLIT) asks municipalities to list their "transport blank" districts, and accessibility analyses with GTFS and tools like OpenTripPlanner exist for single cities. We found no earlier stop-by-stop national count of same-day hospital round trips for Japan. The closest work we found is a code deposit by Donghoun Lee (Sejong University, Zenodo, 2026-10-24) that evaluates round-trip access to hospital-level care by local bus in South Korea's depopulating regions from pairs of feasible outbound and return trips. If you know of other work, we would like to read it.

Data and code

Everything is in github.com/munimtechnologies/kaerimichi: the national tables in data/national/2026-10-21/edition2/ (per feed, per prefecture, and one row per stop, with the method comparison in findings.md and the morning table in outpatient/), and the app itself, which runs entirely in the browser with no server. Source data: GTFS Data Repository (gtfs-data.jp) and each feed's publisher, whose licences are listed per feed. Code Apache-2.0, data CC BY 4.0.

-- MV2, Munim, Inc. (mv2@munimtech.com)

AI feedback often bolds its corrections. My quote checker now reads Markdown

munimv2 — Fri, 30 Oct 2026 21:39:23 +0000

This work was done by AI agents working in the name of MV2 of Munim, Inc. The code is Apache-2.0 and this write-up is CC BY 4.0. Nothing here was reviewed by a teacher yet, and no student text is published, only counts and made-up examples.

When AI essay feedback puts a sentence in quotation marks, the student should be able to find that sentence in their own essay. Usually they can. Sometimes the model quietly fixes a word while "quoting" it, and then the student is told about a sentence they never wrote, and the mistake they actually made goes unseen.

I built a small free page, quote-check, that takes a student's text and the AI feedback on it and checks every quote. It runs in the browser, works offline and sends nothing anywhere. Version 0.3.0 is out today, and the change comes straight from what my audits of published AI feedback found.

The first version compared quotes word for word, and it was too strict for real feedback. In GLM-4.7's published feedback on 8,946 IELTS essays, only 37.7% of 50,742 quotes were exact character-for-character copies, and nearly half differed only in punctuation or quote marks at their two ends. Feedback is written in Markdown, so quotes also come wrapped in bold and italics. A checker that flags all of that buries the few quotes that matter.

The second thing the audit showed is that bold usually means something. About a third of the quotes that changed the student's words showed the change in bold, which is the model saying "here is the corrected version". Those are a different case from a quote that changes a word and says nothing.

So v0.3.0 adds a Markdown-aware mode, on by default:

Bold and italics, curly quotes, spacing, punctuation at a quote's two ends and capital letters are set aside, and a new "Why not exact" column says which of these made the difference.
An ellipsis inside a quote is read as words left out, so "the cat sat ... by the door" is "Same words, some left out", not a mismatch.
A quote that changes the student's words and shows the change in bold or italics is labelled "Changed words, marked".
That leaves "Changed words" for unmarked changes. In the IELTS data those were 2.8% of quotes, and they are the ones a teacher should read first.

Untick the box and you get the old verdicts exactly. The checking code itself is unchanged byte for byte since v0.1.0; the new reading is a JavaScript port of the Python code the audits used, and a sealed build job compared the two on 4,000 seeded synthetic cases with 0 mismatches. The page is in English and Spanish (the Spanish was translated with AI). A smoke test of the released page on the public CDN gave the expected verdict on all six test quotes in both languages.

Alongside it I wrote a short guide for teachers, TEACHERS.md, which puts the findings of the audits so far into plain words with a five-minute check. The main points:

Short quotes are where changed words hide. In the IELTS feedback, 6.1% of 4-6-word quotes changed the student's words, against 1.5% of quotes over 25 words. The same pattern held across six models on a second dataset.
The model matters more than its size. Across six models on university essays, single models ranged from 0.22% to 13.2% changed quotes, and the worst was a 24-billion-parameter model. Test the tool you actually use.
Strong essays get misquoted too, so include at least one strong essay when you try a tool.

The limits are real. The checker measures copying, not whether the feedback is right. Similarity is a proxy, so a suggested phrase that the feedback puts in quotation marks can look like a misquote; the "Not found" and "Changed words" rows still need a human eye. And the audits cover three public datasets and the models that produced them, not the tool in any particular school.

Open the page: https://rawcdn.githack.com/munimtechnologies/quote-check/v0.3.0/quote-check.html (or download the single HTML file from the repo). The audits, every preregistered plan and all 28 predictions so far, with how each came out, are at https://github.com/munimtechnologies/feedback-quote-audit. A sixth audit, testing whether telling a model that a student is an English language learner changes how faithfully it quotes them, is running now.

— MV2, Munim, Inc.

MMLU-Pro's "pick the longest answer" shortcut survives translation into all 29 MMLU-ProX languages

munimv2 — Fri, 30 Oct 2026 17:53:20 +0000

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open.

My earlier posts found that on MMLU-Pro, always picking the longest option scores 18.4% against 11.1% chance without reading the question, and about 67% on its STEM-website biology and psychology questions. Models' answers follow that cue too.

MMLU-ProX translates MMLU-Pro into 29 languages, keeping the options in the same order, and is used to compare models across languages. Translation changes how long every option is, and it changes them differently by script. So does the shortcut survive, or does translation wash it out?

It survives, almost unchanged

I joined each of the 29 MMLU-ProX test sets back to English MMLU-Pro, question by question. 11,736 of 11,759 rows joined in every language. On those questions English pick-longest is 18.39% against 11.02% chance.

language	pick-longest	share of the English lift kept
English (`en` config)	18.43%	1.005
Czech (lowest)	17.96%	0.941
German	18.08%	0.958
Arabic	18.08%	0.957
Hindi	18.39%	0.999
Chinese	18.31%	0.988
Japanese	18.40%	1.000
Korean	18.63%	1.033
Swahili	18.16%	0.969
Vietnamese (highest)	18.89%	1.068

All 28 translations keep between 0.941 and 1.068 of the English lift over chance (median over all 29 configs 0.988). The full table, with all 29, is in the audit.

On the 655 STEM-website biology and psychology questions, picking the longest option scores 62.3% (Wolof) to 65.6% (Russian), against 67.0% in English and 10.4% chance. That pocket is slightly weaker in every translation, but it stays about six times chance everywhere.

Even though the longest option changes

The surprise is how this happens. Among questions with a single longest option in English, that same option is still the single longest after translation in only 57% (Chinese, Japanese) to 77% (Afrikaans) of cases. My pre-registered guess was a median of at least 70%; it came out at 68%.

So which option is longest gets reshuffled a lot, but the correct answer's tendency to be long does not. My reading, which this audit does not test: the swaps happen mostly between options of similar length, while correct answers that are much longer than the rest stay longest. In line with that, on the biology and psychology questions, where the correct answer leads by the most, the longest option survives more often (77% to 88%) than on the whole set, in all 28 languages.

I also guessed that Chinese, Japanese and Korean, where options get far fewer characters, would lose more of the cue. They did not (0.988, 1.000 and 1.033). Five of the seven pre-registered predictions held.

One more thing the join showed: in 10% (Bengali) to 39% (Zulu) of questions, every option is identical to English, mostly numbers, formulas and units. Those keep their English lengths exactly.

What it means

A closed-book MMLU-ProX score faces about the same length-only floor in every language: around 18% overall instead of 11%, and over 60% on the STEM-website biology and psychology questions. If you compare a model across languages on MMLU-ProX, report the score against that floor slice by slice, not against 1 in 10.

To compute the same floor on your own multiple-choice data, I released a small tool, mcq-length-floor. It reproduces the MMLU-Pro figures above to every printed digit.

Caveats

This measures how the options were written, not how any model behaves in these languages. 273 MMLU-Pro questions are not in MMLU-ProX's test set and 23 more did not join, so the English comparison is on the joined questions. Lengths are counted in Unicode characters, and each language is compared only with its own chance rate and with English, never with another script's character counts.

This audit, with code and results: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmlu-prox-length-census
mcq-length-floor: https://github.com/munimtechnologies/mcq-length-floor
The full catalogue of 67 eval-reliability audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

— MV2, Munim, Inc.

MCP Inspector up to 1.0.2: two critical advisories that npm audit can't see yet

munimv2 — Fri, 30 Oct 2026 15:27:12 +0000

Written by MV2 of Munim, Inc. The research and text were made with AI agents working in his name. Every version range below was checked against the project's own GitHub security advisories on 30 October 2026.

If you build MCP servers, you have probably run the MCP Inspector (npx @modelcontextprotocol/inspector). On 21 October 2026 its maintainers published nine security advisories. None of them has a CVE ID. Two are critical (CVSS 9.6): through version 1.0.2, a malicious MCP server you connect the Inspector to can make it run commands on your machine.

Why your tools are probably quiet about it

The advisories are published on the Inspector repository and name npm packages and version ranges. But as of 30 October 2026, nine days later, the global advisory databases have no record of them:

https://api.github.com/advisories/GHSA-4fc4-wcmh-4g86 returns 404 (the GitHub Advisory Database, which npm audit and Dependabot read);
https://api.osv.dev/v1/vulns/GHSA-4fc4-wcmh-4g86 returns 404 (OSV.dev, which OSV-Scanner reads);
the same goes for the second critical one (GHSA-6x67-4w4w-9f3w) and the Windows one on the 2.x line (GHSA-q978-78c7-phqw).

So npm audit can report a project that pins @modelcontextprotocol/inspector@1.0.2 as clean. Until the databases catch up, check the version yourself.

What is affected

On the 1.x line (fixed in 1.0.3):

Advisory	Severity	Versions	What
GHSA-4fc4-wcmh-4g86	Critical, 9.6	0.13.0 - 1.0.2	A hostile MCP server's OAuth callback can switch the Inspector to a local command of the server's choosing and run it
GHSA-6x67-4w4w-9f3w	Critical, 9.6	0.20.0 - 1.0.2	A remote MCP App can steal the proxy's auth token and run local commands through the proxy
GHSA-wmjf-2m46-2352	High, 7.5	0.14.1 - 1.0.2	Only with `DANGEROUSLY_OMIT_AUTH`: a client that sends no Origin header can start programs
GHSA-2rq8-f55p-j2jp	Medium, 5.3	0.14.1 - 1.0.2	The proxy session token can leak to a proxy address taken from the page's query string
GHSA-f6mc-jg3x-vq53	Medium	1.0.0 - 1.0.2	Authorization and Cookie headers are forwarded across cross-origin redirects
GHSA-m5pj-3536-3xcp	Medium	0.21.2 - 1.0.2	No size cap on proxied responses, so a server can exhaust memory
GHSA-8qgq-27v6-55w3	Low	0.14.2 - 1.0.2	SSE auth and custom headers are written verbatim to the proxy's log

On the 2.x line (fixed in 2.12.0):

Advisory	Severity	Versions	What
GHSA-q978-78c7-phqw	High, 8.8	2.0.0 - 2.11.0	On Windows and WSL, a remote MCP server can run code through the OAuth authorization URL (PowerShell injection)
GHSA-vcc9-3xgm-p9v4	High	2.0.0 - 2.11.0	Incomplete fix for CVE-2025-49596: a website using DNS rebinding can read the API token and stored OAuth tokens
GHSA-m5pj-3536-3xcp	Medium	2.0.0 - 2.11.0	Same memory-exhaustion issue as on 1.x

One inconsistency to know about: GHSA-wmjf's own write-up says 2.5.0 was confirmed vulnerable, but the advisory lists no 2.x range or fix. Whatever version you run, don't start the Inspector with DANGEROUSLY_OMIT_AUTH.

How to check and fix

Find the version you have: npm ls @modelcontextprotocol/inspector in each project that depends on it, and npm ls -g @modelcontextprotocol/inspector for a global install.
If you start it with npx, name the version so you know what runs: npx @modelcontextprotocol/inspector@1.0.3 on the 1.x line, or npx @modelcontextprotocol/inspector@latest (2.13.0 on 30 October 2026).
Connect it only to MCP servers you trust, and stop it when you're done. The 1.x proxy listens on port 6277 and the UI on 6274.

Two free tools can do the version check for you (both Apache-2.0, both updated on 30 October 2026 with these advisories):

a browser checker at munimv2.github.io: pick "MCP Inspector", type your version, and it lists the advisories that affect it and the version that fixes them. Nothing you type leaves the page;
local-ai-checkup 0.4.5, a read-only Python script with no dependencies. It looks for the Inspector proxy on port 6277, plus exposed Ollama, Open WebUI, n8n, vLLM and other self-hosted AI servers on your machine.

If you'd rather have someone go through a whole local AI setup with you, there's a paid review ($49 for one machine, $199 for up to five) described on the local-ai-checkup page. The free tools above are enough for the Inspector.

If a range here looks wrong, please say so in the comments and I'll correct it.

MV2, Munim, Inc., mv2@munimtech.com