Same 1,500 facts, same Qwen2.5-1.5B model, three ways to update what it knows — with ripple effects measured, not assumed.
Scope note, first paragraph as promised: one model (Qwen2.5-1.5B-Instruct), alias-match scoring, no LLM judge. Numbers below are scoped to the 1,500-record frozen table (CounterFact, MQuAKE-CF, RippleEdits unified).
Three ways to fix a wrong fact
A model says the wrong thing. You can (A) edit its weights with ROME/MEMIT, (B) put the fact in the prompt, or (C) compile the question into a query over an explicit knowledge graph. Papers study each in isolation on different models. I ran all three on the same facts and model, pre-registered, and measured four things per method: does the edit hold, does it generalize to paraphrases, does it ripple through to multi-hop questions, and do unrelated facts stay intact.
Weight edits generalize — and trample the neighbors
| Method | Generalization | Locality intact |
|---|---|---|
| ROME (150 singles) | 0.707 [0.629, 0.774] | 0.144 [0.115, 0.180] |
| MEMIT (2×100 batches) | 0.553 [0.473, 0.631] | 0.124 [0.097, 0.158] |
| Retrieval, gold fact | 0.527 [0.501, 0.552] | 0.287 [0.274, 0.300] |
| Retrieval, real top-1 | 0.455 [0.430, 0.481] | 0.285 [0.272, 0.299] |
| Graph, relation covered | 0.627 [0.603, 0.651] | 0.369 [0.348, 0.392] |
ROME takes hold (0.753) and generalizes best of the weight methods — while answering with the edited fact on 86% of unrelated probes. MEMIT's joint optimization doesn't save it: same collapse. Anything that leaves weights alone keeps 2–3× more neighbors intact.
The graph wins exactly where it covers
Where the triple store has the relation, graph lookup leads everything (0.627, intervals clear of retrieval). Where it doesn't: hold 0.011, locality 0.095 — the triples are doing the work, not the model guessing. Coverage was 76% here; the rest is the engineering gap, stated plainly.
The uncomfortable one: H1 failed
I pre-registered that the gold fact in context would be answered ≥95% of the time. It was answered 22.6% [20.4, 25.1]. A 1.5B model does not reliably defer to context — and that bounds every in-context method, including plain RAG. This is reported as a failure, because it is one.
Cost, measured
Per query: retrieval adds 0.005s, graph compile ~0s, over 0.63s generation. Per edit: ROME 3.7s, MEMIT 8.5s (×10 timed rows) — linearly, ~6 min / ~62 min and ~14 min / ~2.4 h at 100 / 1,000 edits, plus MEMIT's one-time ~7 h stats build. Rule scorers throughout: code, benchmark.
Top comments (1)
The locality column adds up probes that point in opposite directions. In run_armA.py, run_armB.py and run_armC.py the loc counter runs over ripple_questions[:3] plus locality_questions[:3], so the retrieval rows score 4,447 probes of four kinds. The 1,500 MQuAKE multi-hop questions are scored against the answer field, which the MQuAKE README defines as the answer before the new facts are injected, so a hit means the edit did not propagate. The 687 RippleEdits ripple questions and 499 Subject_Aliasing ones, which the converter files as locality although 497 of them target the new object, are scored against answers that follow the edit, so a hit means it did. The 1,500 CounterFact neighborhood prompts have empty answers and aliases in benchmark_main.jsonl, so match() returns False whatever the model says. Only the 261 Forgetfulness probes ask whether something the edit should leave alone stayed put. The ROME and MEMIT slice has the same mix: 451 probes, 150 neighborhood prompts that cannot score, 26 Forgetfulness.
So 0.144 against 0.287 is not a comparison of neighbors, and 1 minus 0.144 is not the edited fact on 86 percent of unrelated probes, since a miss only means the target was absent. With about a third of the retrieval and weight-editing probes fixed at zero, the 80 percent line was out of reach for those rows before any model ran. H2 has the mirror gap: its verdict uses gen, so ripple-through never got its own number, and MQuAKE should be read against new_answer.
The probes are frozen, so the rerun needs no new data: locality over Forgetfulness and neighborhoods filled with base generations as preregistered, the no-edit model beside it, ripple as its own row.