DEV Community

hao li
hao li

Posted on

contradiction-judge: the memory auditor your AI agents are missing

#ai #agents #memory #python

AI agents remember things. Over weeks of conversations they accumulate facts:
you are 20, you love pizza, you live in Pasadena. But memory systems almost
never check whether those facts agree with each other. When a new memory
says "you are 21" and an old one says "you are 20", one of them is wrong —
stale entry, misheard sentence, upstream bug — and nobody notices.

I built contradiction-judge
(pip install contradiction-judge) to fix that. Feed it a JSON file of memory
entries; it returns every contradicting pair plus an arbitration suggestion
telling you which entry should win.

contradiction-judge check memory.json
Enter fullscreen mode Exit fullscreen mode 
{
  "entries": 2,
  "pairs_checked": 1,
  "contradictions": [
    {
      "entry_a": "m1",
      "entry_b": "m2",
      "kind": "numeric",
      "engine": "rules",
      "explanation": "'I am 20 years old.' vs 'I am 21 years old.': person.age cannot be both 20.0 and 21.0.",
      "arbitration": {
        "strategy": "recency",
        "recommendation": "keep m2, drop or correct m1",
        "reason": "Both entries are timestamped; the newer record (m2) is more likely to be current."
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Two layers

Rules (stdlib only, zero dependencies). Numeric conflicts ("20 years old"
vs "21 years old", unit-aware: "240 months old" == "20 years old"), date
conflicts ("birthday May 5, 2006" vs "2006-06-01"), factual conflicts (name,
location), and polarity conflicts ("I love pizza" vs "I hate pizza", "I am a
student" vs "I am not a student").

A pluggable LLM judge hook for the subtle stuff rules can't see:

from contradiction_judge import check_memories, set_judge

def gpt_judge(text_a, text_b):
    # call your model; return {"contradiction": bool, "explanation": str}
    ...

set_judge(gpt_judge)
report = check_memories(entries)  # semantic pairs also get judged
Enter fullscreen mode Exit fullscreen mode

The hook only runs on pairs the rules did not flag, so you pay for LLM
calls only where they add value. (--hook module:func on the CLI.)

How it differs from memgovern

If you've seen my memgovern
project: that one is a governance framework — provenance tracking, source
trust levels, write policies, retention rules. It answers "should this
memory be trusted and kept?"

contradiction-judge is the auditor. It answers "do these memories
contradict each other, and which one should win?" — semantic
fact-contradiction detection with arbitration suggestions. Same roadmap
family (the LLM-judge idea came straight from memgovern's roadmap), but a
different job. Govern with memgovern, audit with contradiction-judge.

Design notes

  • Recency wins, by default. When both entries are timestamped, the newer record is recommended — facts change, and location or preference conflicts may reflect a genuine move rather than an error. Without timestamps, the pair is flagged for manual review.
  • CI-friendly: --fail exits 1 when any contradiction is found.
  • 15 smoke tests, all green before release.

On the roadmap: embedding-based topic alignment ("the Bay Area" vs
"San Francisco"), confidence scores, and an auto-fix mode that rewrites the
stale entry instead of just flagging it.

Repo: https://github.com/hahahahahahahahah6/contradiction-judge
PyPI: https://pypi.org/project/contradiction-judge/

Top comments (1)

Collapse
 
galagent profile image
Qingfeng •

the recency-wins-by-default arbitration is the part i'd watch most. on a local agent that edits files, the newest write is often the wrong one because the agent loaded a stale copy earlier and wrote it back after a newer hand edit, so mtime recency points at the stale content. one cheap guard is keeping a content hash per entry and refusing arbitration when two entries have different hashes but the same source-event id, since that usually means a write-back collision rather than a real fact change. flagging those for manual review instead of letting recency decide saved me a few bad merges, and running the llm hook only on pairs the rules missed keeps cost sane on long memory files.