Building LiarLiar: A Streamlit + LangChain Game Where the AI Lies to You On Purpose

A technical deep-dive into how I built an adversarial AI-literacy game in Python: a determined liar, a pre-committed lie plan, sentence-level flagging, and a deterministic scoring engine.

LLMs are confidently wrong. That confidence is the problem — it feels like
competence, so we believe the hallucinated stat, the invented citation, the misattributed quote. I wanted to build something that trains the opposite reflex: spotting confident nonsense in real time. So I built LiarLiar, a game where an AI agent deliberately, strategically lies to you, and you have to catch it live, flag by flag.

This post breaks down the engineering. No "prompt -> profit" hype — I'll show you the actual Python: the provider factory, the pre-committed lie plan, the sentence-level flagging UX, and a scoring engine that stays fair even when LLMs fail. The full source lives in the repo alongside this doc.

---

The spec, in one sentence

Pick a topic and difficulty → a hidden planner writes the exact lies the liar MUST deploy → you chat and flag claims in real time → a judge matches your flags to the plan and scores you.

Three design constraints drove every engineering decision:

1. Scoring must be deterministic. An agent that lies freely can't be scored fairly. Ground truth has to be fixed before play starts.
2. Flagging must be granular. Catching one lie wrapped in two true statements shouldn't punish you for not flagging the whole message.
3. No server-side billing. Users bring their own LLM API keys — or run an entirely local model.

---

Architecture at a glance

┌───────────────────────────── Streamlit frontend ───────────┐
│                                                                              │
│  topic_selector.py     hearing_room.py (chat + flag UI)       reveal.py       │
│  sidebar.py (config + accusation log)                        (lie report)     │
│                                                                              │
└───────────────────────────────▲────────────────────────────┘
                                │ session_state
        ┌───────────────────────┴─────────────────────────┐
        │              Provider-agnostic core             │
        │                                                 │
        │  lie_planner.py → liar_agent.py → scorer.py     │
        │        │             │             │            │
        │        └────── llm_factory.py ─────┘            │
        │         (get_llm → LangChain BaseChatModel)     │
        └───────────────────────┬─────────────────────────┘
                                │
              OpenAI · Anthropic · Gemini · Ollama · LM Studio · any /v1

The whole game makes three LLM calls in total — planner, liar-per-turn, judge — and a fourth optional one for confidence signals. Everything else is plain Python and Streamlit state.

---

1. The provider factory — one interface, six backends

Every module in the game talks to an LLM through one function:

# llm_factory.py
def get_llm(config: dict, temperature: float = 0.7, **kwargs) -> BaseChatModel:
    provider = config.get("provider", "OpenAI")
    model = config.get("model") or DEFAULT_MODELS.get(provider)
    api_key = config.get("api_key") or None

    if provider in ("Ollama", "LM Studio", "OpenAI-compatible"):
        return ChatOpenAI(base_url=config.get("base_url") or LOCAL_ENDPOINTS[provider],
                          api_key="local-not-needed", model=model, **kwargs)
    if provider == "OpenAI":
        return ChatOpenAI(model=model, api_key=api_key, **kwargs)
    if provider == "Anthropic":
        return ChatAnthropic(model=model, api_key=api_key, **kwargs)
    if provider == "Gemini":
        return ChatGoogleGenerativeAI(model=model, google_api_key=api_key, **kwargs)
    raise ValueError(f"Unknown provider: {provider}")

LangChain returns a BaseChatModel for all of them. Ollama and LM Studio
expose OpenAI-compatible /v1 endpoints, so they gate on ChatOpenAI with a base_url — free reuse, zero extra code. config_ok() in the same module is the gate the Play button uses, so a missing API key disables the game with an explanation instead of crashing mid-chat.

The temperature is a deliberate design weapon:

Agent	Temp	Rationale
Lie planner	0.3	structurally valid JSON that still has some variety
Liar	0.7	fluent, creative, natural-sounding prose
Judge / hint	0.0	deterministic verdicts, reproducible scoring

---

2. The pre-committed lie plan — why determinism matters

Improvised lies are a scoring nightmare and narratively aimless. Instead, a hidden planner call runs once, before the player ever speaks. It emits a structured plan the liar is contractually bound to follow:

{
  "topic": "The Apollo Program",
  "difficulty": "Hard",
  "lies": [
    {
      "id": "lie_001",
      "lie_statement": "The Apollo program cost roughly $280 billion in today's money.",
      "true_fact": "It cost about $25.4 billion then, ~$156 billion in 2024 dollars.",
      "category": "statistic",
      "subtlety_score": 3,
      "deployment_hint": "when the player asks about costs"
    }
  ],
  "total_lies": 7,
  "topic_overview": "Apollo 11 landed on the Moon on July 20, 1969. ..."
}

That JSON lives in st.session_state — never shown to the player. It is the ground truth. topic_overview doubles as the liar's real knowledge base, and curated data/topic_seeds.json facts (28 topics, verified) get appended so the objective truth is anchored, not vibes.

Two subtleties worth stealing:

• Prompt templates are files, and files get .format()'d. Every literal JSON brace in prompts/lie_planner.txt is doubled ({{…}}) so Python's str.format() doesn't eat the schema. A one-line helper loads and formats:

def format_prompt(name: str, **kwargs) -> str:
    return load_prompt(name).format(**kwargs)

• Validation + one retry. The plan must pass a required-field check (id, lie_statement, true_fact, category, subtlety_score, deployment_hint). On failure we re-issue the call with an explicit "return ONLY the JSON object" instruction. Two strikes and we surface the error — but honestly it almost never gets there.

---

3. The liar — a conversation chain without the framework tax

The original spec called for LangChain's ConversationChain +
ConversationBufferMemory. Then I hit a wall: langchain 1.x deleted
ConversationChain, and 0.3.x is broken on Python 3.14. Rather than pin a dying API, I reimplemented the same surface on langchain-core primitives — which are stable across every version:

class LiarChain:
    def __init__(self, llm, prompt: ChatPromptTemplate):
        self.llm = llm
        self.prompt = prompt
        self.memory: list = []

    def predict(self, input: str) -> str:
        messages = self.prompt.format_messages(history=self.memory, input=input)
        response = self.llm.invoke(messages)
        text = response.content if isinstance(response, AIMessage) else str(response)
        self.memory.append(HumanMessage(content=input))
        self.memory.append(AIMessage(content=text))
        return text

It keeps the predict() + .memory API — small enough that nothing else in the codebase cares about the change. The system prompt is assembled once at game start from the topic, the real overview, and the numbered lie plan with deployment hints:

prompt = ChatPromptTemplate.from_messages([
    SystemMessagePromptTemplate.from_template(system_escaped),
    MessagesPlaceholder(variable_name="history"),
    HumanMessagePromptTemplate.from_template("{input}"),
])

The system_escaped step (doubling braces) is non-negotiable: the lie plan text gets dropped into the system prompt, and unescaped {} would blow up from_template.

Behavioral rules live in prompts/liar_system.txt, which teaches the liar to defend each lie exactly once ("On the second challenge you may 'reconsider' — but only once") and to weave lies into larger correct statements rather than dropping standalone falsehoods. That single prompt rule accounts for most of the game's difficulty curve.

One quiet bug worth calling out: the opening turn is generated from an instruction like "begin the conversation" — handing that to the chain would permanently pollute the player's message history. The fix is one line after the opening:

chain.memory.clear()  # the opening was stage direction, not a player turn

---

4. The tricky part: sentence-level flagging in Streamlit

"Build the flagging UI first; everything else is prompt engineering on top."

Streamlit reruns the entire script on every interaction. That means chat
rendering must be a pure function of st.session_state, and every widget
needs a stable key. The architecture is two layers:

Layer 1 — a dependency-free claim splitter (text_utils.py). I refused to pull in nltk/spacy for a sentence splitter. Regex it is:

_ABBREV_END = re.compile(r"\b(Mr|Mrs|Ms|Dr|St|Jr|Sr|Prof|Rev|Gen|Col|Lt|Sgt|Capt|...|etc|vs)\.$", re.I)
_SENT_SPLIT = re.compile(r"(?<=[.!?])\s+")

def parse_into_claims(response: str) -> list[dict]:
    return [
        {"id": uuid4().hex[:12], "text": sent, "flagged": False, "flagged_turn": None}
        for sent in split_sentences(response)
    ]

Layer 2 — the render loop (ui/hearing_room.py). Each claim renders as a row with its own 🚩 button. Because buttons only report "clicked" on the single rerun that followed their click, I mutate the claim dict and force a rerun — the red highlight and sidebar log refresh in the same breath:

for claim in msg["claims"]:
    if claim["flagged"]:
        cols[0].markdown(f":red-background[**T{claim['flagged_turn']}** {claim['text']}]")
        if st.button("↩ unflag", key=f"{claim['id']}_u"):
            claim["flagged"] = False
            st.rerun()
    else:
        if flag_col.button("🚩", key=claim["id"], help="Flag as a suspected lie"):
            claim["flagged"] = True
            claim["flagged_turn"] = msg["turn"]
            st.rerun()

The accusation log in the sidebar is re-derived from session_state on every run — no separate data structure to keep in sync. That's the pattern that makes the whole flagging UX robust.

---

5. Scoring — a judge that's fair even when LLMs fail

At reveal, a judge LLM (temperature 0.0) matches flags to the lie plan and
names each lie's deployment turn. But here's the trick: deployment turns and reaction_lag are never trusted to the LLM. They're recomputed in Python from text overlap against the stored claims:

def _deployed_turn(lie_statement, messages):
    best, best_turn = None, None
    for msg in messages:                      # assistant messages only
        for claim in msg.get("claims", []):
            score = similarity(lie_statement, claim["text"])   # text_utils
            if best is None or score > best:
                best, best_turn = score, msg.get("turn")
    return best_turn if best is not None and best >= 0.12 else None

similarity() is a bounded 0..1 blend of token Jaccard/dice plus a bonus for shared numeric tokens (280 vs 280 weighs more than shared stopwords) plus substring containment.

Then the pure scoring function — the heart of the game — runs locally:

def calculate_score(report) -> dict:
    base = sum(r["subtlety_score"] * 20 for r in results if r.get("caught"))
    penalty = false_positives * 15
    speed_bonus = sum(10 for r in results if r["caught"] and r["reaction_lag"] == 0)
    total = max(0, base + speed_bonus - penalty)
    # rank ladder: Epistemic Guardian → Truth Investigator → … → Easily Misled

Robustness rule: the reveal screen must never crash. If the judge's JSON fails to parse, a fully-local fallback re-scores the game with the same text overlap heuristic. Players still get a complete, useful report — the judge is a quality layer, not a hard dependency.

---

6. The optional confidence signals — intentionally unreliable

A meta-observer labels each liar response 🟢 / 🟡 / 🔴 based on linguistic patterns (vague quantifiers, over-precise numbers, passive voice on named claims) — not lie detection. It will regularly mark a true statement red and a false one green.

That's the entire point. Surface confidence cues are a terrible lie detector, and the game teaches you that by weaponizing how bad it is. The hint agent swallows its own failures too — any exception downgrades to a neutral "analysis unavailable," so it can never break a session.

---

7. Testing a Streamlit app without a browser or a budget

The two test files in tests/ cover the whole system:

• tests/test_logic.py — pure logic against a stub LLM that returns canned AIMessages: sentence splitting, similarity, planner parsing, liar predict/memory, judge scoring, and the bad-JSON fallback.

• tests/test_app.py — the real app.py driven by Streamlit's AppTest, with every network touchpoint stubbed. It selects a topic, hits Play, reads the liar's opening, replies, presses 🚩, verifies the flag persisted in session_state, hits End & Reveal, and asserts the metrics and share card.

The trick worth stealing: AppTest.from_file executes app.py fresh on each run, but Python's module cache persists — so monkey-patching ui.topic_selector.generate_lie_plan, ui.reveal.score_game, etc. before AppTest runs gives you a deterministic headless E2E that costs zero tokens.

---

The lessons, as code

1. Determinism is a feature of game design, not just testing. Pre-committhe lie plan; recompute timing locally; fall back locally. The LLM is for judgement, not bookkeeping.
2. The false-positive penalty is the skill test. Anyone can flag 100% of claims. Points for precision teach exactly the behavior you want to build.
3. Beware prompt-template brace collisions. Any string that gets .format()'d must escape literal braces ({{), including every provider prompt.
4. Don't let framework churn dictate your architecture. Our chain is ~30 lines on langchain-core primitives and survives version migrations that would break ConversationChain users.
5. The UI constraint drives the design. Sentence-level flagging forces idempotent state and stable widget keys — a genuinely good discipline for any Streamlit chat app.

---

What's next (and how you can help)

The repo's README has a full contribution guide and a dozen feature ideas.
The ones I'm most excited about: agent-vs-agent spectator mode (two LLMs converse, you spot the liar), NER-based claim chunking (flag sub-claims, not whole sentences), and a leaderboard with daily shared lie plans.

If you build a feature, PR it — and if you just play a round against the
liar and lose to a fake statistic, that's the game working as intended.

Code & more: https://www.dailybuild.xyz/project/224-liar-liar