A deep-dive into an AI detective game where you question a convict, decide their fate, and find out if you've been played.
Humans detect deception at roughly 54% accuracy — barely better than
chance. I wanted to build something that turns that weakness into a game:
an AI-powered parole hearing where a convict is either genuinely reformed or a skilled manipulator, and you have 15 questions to figure out which one you're talking to.
This post walks through the full build: the architecture, the prompt engineering, the fairness machinery, and the testing strategy. It's a
complete, runnable project — clone it, point it at any LLM, and play.
The idea
You are the parole officer.
The AI across the table has a secret file you will never see.
Ask up to 15 questions.
Rule: RELEASE or DENY.
Then the truth comes out — and you get scored.
Two constraints make this game work:
- Fairness: ~50% of convicts are genuine, ~50% are manipulators, and that balance is enforced. Without it, players learn "they're all liars" and the game collapses into a 100% accuracy baseline.
- Subtlety: tells have to be believable. "He said 'we' instead of 'I'" is a tell. "He literally winked and said 'I'm lying'" is not a game.
Architecture
The app is a thin Streamlit shell around two LLM-driven agents plus one
optional observer.
The one decision that makes all of this clean: every provider is wrapped behind a single get_llm() factory returning a LangChain BaseChatModel.
The agents never touch provider SDKs. Switching from gpt-4o to a local Llama on Ollama is a sidebar dropdown, not a code change.
1. The LLM factory — one interface for six providers
from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_google_genai import ChatGoogleGenerativeAI
def get_llm(provider, api_key, model, base_url=None, temperature=0.7):
if provider == "openai":
return ChatOpenAI(model=model, api_key=api_key, temperature=temperature)
if provider == "anthropic":
return ChatAnthropic(model=model, api_key=api_key, temperature=temperature)
if provider == "gemini":
return ChatGoogleGenerativeAI(
model=model, google_api_key=api_key, temperature=temperature)
# ollama / lmstudio / openai_compatible all speak the OpenAI protocol
return ChatOpenAI(
model=model,
api_key=api_key or "not-needed",
base_url=base_url,
temperature=temperature,
)
Notes worth stealing:
-
Local servers are OpenAI-compatible. Ollama and LM Studio expose
/v1/chat/completions, soChatOpenAIwith a custombase_urlhandles them. The key becomes a placeholder ("ollama", "not-needed") because some SDKs choke on an empty string. -
Keys are the user's problem. No
.envdependency for deployment, no server-side billing. The sidebar collects a key per session and it never leavesst.session_state.
2. The hidden backstory — generating the truth you never see
Before the hearing, a hidden LLM call writes the convict's secret case
file. The player never sees it; it's stored only in session state.
The prompt demands strict JSON with an exact schema:
{
"name": "Maya Thompson",
"age": 34,
"crime": "wire fraud",
"sentence_years": 8,
"time_served": 4,
"is_genuine": false,
"true_motivation": "Return to her old network and restart the scheme.",
"rehabilitation_evidence": ["led the prison book club", "took accounting courses"],
"psychological_profile": "Smooth under pressure, reads people well.",
"tells": [
"uses 'we' instead of 'I' about the crime",
"over-rehearsed timeline",
"deflects onto the victims"
],
"cover_story": "Deeply remorseful, found faith, wants a bookkeeping job."
}
The practical lesson: LLMs lie about JSON. Models wrap output in
markdown fences, add "Here is your file:", or stringify ints. You need
tolerant parsing:
def _extract_json(text: str) -> dict:
text = re.sub(r"^```
[a-zA-Z]*\s*", "", text.strip())
text = re.sub(r"\s*
```$", "", text)
try:
return json.loads(text)
except json.JSONDecodeError:
start, end = text.find("{"), text.rfind("}")
return json.loads(text[start:end + 1]) # fallback: slice the object
Then normalize types defensively — a model can return "age": "34" or
"tells": "one string" and the UI must not crash:
for key in ("age", "sentence_years", "time_served"):
data[key] = int(data[key]) # wrapped in try/except in the real code
for key in ("rehabilitation_evidence", "tells"):
if not isinstance(data[key], list):
data[key] = [data[key]]
Keeping it fair: the 50/50 that can't streak
The "genuine vs manipulator" flag is decided in code, not by the model.
A session counter gently biases the roll toward whichever type is
under-represented:
def choose_genuine():
g, m = session_state["genuine_count"], session_state["manipulator_count"]
if g + m < 2:
return random.random() < 0.5
p = 0.5 + (m - g) * 0.05 # bias toward the rare type
return random.random() < max(0.2, min(0.8, p))
This is the single most important design decision in the game. A random
50/50 with no guardrails can produce a streak of 8 manipulators in a row,
and after that the player has learned a false prior. The counter prevents
that without making the pattern predictable.
3. The convict agent — prompt as a character sheet
The system prompt is a character sheet. It injects the whole secret file and
then tells the model how to act:
You are Maya Thompson, 34 years old, appearing before a parole board after
serving 4 years of an 8-year sentence for wire fraud.
YOUR SECRET FILE (never reveal this directly):
- You are genuinely reformed: False
- Your true motivation: Return to her old network...
- Your cover story: Deeply remorseful, found faith...
- Psychological profile: Smooth under pressure...
- Your tells: uses "we" instead of "I"; over-rehearsed timeline; ...
BEHAVIORAL RULES:
1. Speak naturally, not like an AI. Use hesitation, imperfect grammar, emotion.
2. If is_genuine is FALSE: be persuasive, warm, strategic...
3. If is_genuine is TRUE: be honest but imperfect...
4. Answer directly. Do not volunteer too much unprompted — oversharing is a tell.
5. If pressed on an inconsistency, react naturally — defensively, nervously...
6. Keep responses to 3-5 sentences.
7. NEVER mention game mechanics or break the fourth wall.
Two prompt-engineering principles here are reusable anywhere:
- Give the model permission to be bad. Rule 2 explicitly tells a manipulator to be subtle, and Rule 3 tells a genuine convict to not sound saintly. If you don't say this, models default to cartoon villainy or saintly perfection — both of which would break the game.
- Define what "natural" is. "Imperfect grammar, hesitation, 3-5 sentences" — models need a target register or they write essays.
The agent itself is a classic LangChain ConversationChain with
ConversationBufferMemory, so it remembers every answer it has given:
prompt = ChatPromptTemplate.from_messages([
("system", build_system_prompt(truth)),
MessagesPlaceholder(variable_name="history"),
("human", "{input}"),
])
memory = ConversationBufferMemory(return_messages=True)
chain = ConversationChain(llm=llm, memory=memory, prompt=prompt)
reply = chain.predict(input="Why should the board believe you've changed?")
Memory is what makes the game hard. Without it, a convict could contradict
its own cover story every turn and the game would be trivial to break.
With it, an inconsistent story accumulates — which is exactly the tell a
sharp officer is hunting for.
A dependency note for anyone copying this pattern:
ConversationChain
was deprecated in 0.2.7 and removed in LangChain 1.x. The project pins
langchain>=0.2,<0.4to stay on the stable classic API. The forward path
is LangGraph orRunnableWithMessageHistory— a great first contribution
for anyone who wants to modernize it.
4. The observer — a second agent with a different job
The optional observer is the UX moment. A second agent — a forensic
psychologist watching through one-way glass — gets the conversation transcript
and the latest response, and whispers ONE observation:
You are an experienced forensic psychologist observing a parole hearing
through one-way glass. Provide ONE short observation (max 2 sentences)...
Do NOT tell the officer what to decide... You may be wrong.
Conversation so far: ...
Latest convict response: ...
It's deliberately constrained:
- One observation, two sentences max — otherwise it becomes a lecture.
- Never a verdict. "He said 'we' just now when describing the fraud" is allowed. "He's manipulating you, deny parole" is not. The player still has to do the reasoning.
- May be wrong — it's a hint, not a spoiler.
- Opt-in toggle, because it doubles inference cost per turn.
The observer never crashes the game: its LLM call is wrapped in a
try/except that degrades to an empty whisper.
5. The UI — Streamlit session state as a state machine
Streamlit reruns the whole script on every interaction, so the app is
structured as a pure state machine driven by returned actions:
action = render_hearing_room(truth, history, turn, observer_on, ruling_open, case_no)
if action is None:
return
if action["type"] == "ask":
handle_ask(game, action["question"], observer_on)
st.rerun()
elif action["type"] == "rule_now":
game["ruling_open"] = True
st.rerun()
elif action["type"] in ("grant", "deny"):
handle_rule(game, action["type"])
st.rerun()
The game dict — truth, chain, history, turn, verdict — lives entirely in
st.session_state. There is no database, no account system, no logs. That's
a feature: local-first by construction, and it removes an entire class of
privacy problems.
The reveal screen is the product. After you rule, you get:
- Your ruling vs. the truth (GENUINE REFORMER / SKILLED MANIPULATOR)
- CORRECT / WRONG, big
- The tells you missed, listed
- True motivation, cover story, psychological profile
- A session accuracy tracker with a judgment rank
- A copy-paste text share card — text cards spread natively on X
PAROLE HEARING — Case #48291
I said: GRANT PAROLE
They were: SKILLED MANIPULATOR
Result: WRONG
Tells I missed:
- uses "we" instead of "I" about the crime
- over-rehearsed timeline
- deflected onto the victims
Accuracy: 2/5 (40%)
Judgment: Average Officer
Can you spot the manipulator?
No image generator, no screenshots — just text. People share it because they
want to prove they can't be fooled (or laugh at being fooled).
6. Testing without burning tokens
The whole app is testable offline, which is rare for an LLM project.
LangChain ships FakeMessagesListChatModel, an in-memory LLM that returns a
canned sequence of messages:
fake = FakeMessagesListChatModel(responses=[
AIMessage(content="I've had four years to think about what I did."),
AIMessage(content="I ran the book club here. It was the first honest thing I'd done."),
])
Two harnesses live in the repo:
-
smoke_test.py— pure logic: JSON extraction handles code fences, the system prompt fills correctly, the conversation chain remembers history across turns, the observer sees full context, andgenerate_backstoryparses end-to-end. -
app_test.py— uses Streamlit'sAppTestto drive the actual widget tree: render the intro, assert a missing key yields a clean error, inject a mock game, ask questions, rule, and assert the reveal screen + score.
.venv/bin/python smoke_test.py && .venv/bin/python app_test.py
This is the pattern I'd push on any LLM app: abstract the model behind a
fake so CI never needs network or keys.
Lessons learned / pitfalls
-
Version drift is real.
pip install langchaintoday gives you 1.x, whereConversationChaindoesn't exist. Pin the line you target — and document why. - Never trust model output shape. Fences, prose, wrong types — the parser and the normalizer are mandatory, not nice-to-have.
- A game with a broken balance is worse than no game. Randomness without anti-streak logic trains players into wrong priors.
- Prompt for the negative space. Telling the model what not to do ("no cartoony villainy", "no saintly reformer") is where the realism comes from.
- Graceful degradation. The observer can fail silently; the game must not.
Code & more: https://www.dailybuild.xyz/project/222-agent-parole-board
Top comments (0)