DEV Community: 494

When Simpler AI Decisions Worked Better: What I Changed in My Jev-Powered Game Judge

494 — Sun, 04 Oct 2026 16:43:52 +0000

I have been building a small lateral-thinking puzzle game called Sideways.

The basic interaction is simple.

The player sees a mysterious situation and asks questions until they discover what really happened.

For example:

Player:
Is it light?

Game Master:
Yes.

Or:

Player:
Does the wake-up mechanism make sound?

Game Master:
No.

The Game Master is powered by Jev.

I deliberately did not want a general-purpose chatbot generating explanations.

I wanted something much closer to this:

Natural-language input
        ↓
Probabilistic semantic decision
        ↓
Typed result
        ↓
Deterministic TypeScript
        ↓
Game behavior

After several iterations, I realized that this is almost like building probabilistic IF statements over natural language.

And I also learned something I did not expect:

Making those semantic IF statements more sophisticated actually made the game worse.

The fix was not to make the AI smarter.

The fix was to make the decision smaller.

The original idea: AI understands meaning, code controls behavior

The architecture follows a simple rule:

AI = semantic judgment
Code = deterministic policy
User = consequential intent

The model does not directly decide which UI to show.

The browser does not receive the hidden puzzle solution.

The model produces a bounded semantic judgment, and TypeScript turns that judgment into a product result.

For questions, the public result can be:

YES
NO
PARTLY
IRRELEVANT
UNCLEAR

For theory submissions:

SOLVED
PARTIAL
NOT_SOLVED

This separation still feels right to me.

The mistake was not the architecture itself.

The mistake was how much semantic structure I tried to extract from one player sentence.

Version 2: the judge became too clever

In Version 2, a Question judgment looked roughly like this:

{
  wellFormed,
  relevant,
  evidence: {
    entailed,
    contradicted,
    undetermined
  },
  mixedClaims: {
    supported,
    contradicted
  }
}

Most of those values were probabilities between 0 and 1.

Then TypeScript applied rules such as:

if (wellFormed < 0.8) {
  return "REPHRASE";
}

if (relevant <= 0.2) {
  return "IRRELEVANT";
}

if (relevant < 0.8) {
  return "NOT_ENOUGH_INFORMATION";
}

if (
  mixedClaims.supported >= 0.8 &&
  mixedClaims.contradicted >= 0.8
) {
  return "PARTLY";
}

if (evidence.entailed >= 0.8) {
  return "YES";
}

if (evidence.contradicted >= 0.8) {
  return "NO";
}

The Solution Judge was even more detailed:

{
  coreMechanism,
  causalRelation,
  supportingInsight,
  contextualError,
  competingMechanism
}

Again, deterministic TypeScript combined these scores.

The goal was reasonable.

I wanted to distinguish:

"There is a light."
→ PARTIAL

from:

"The light wakes the guest."
→ SOLVED

and from:

"There is a light, but vibration wakes the guest."
→ NOT_SOLVED

Conceptually, Version 2 was much more expressive.

The code also looked disciplined.

And the software tests looked excellent

Version 2 passed:

880 unit/API tests
82 desktop/mobile E2E tests
Leak scanning
Lint
Type checking
Production build

I had calibration data.

I had a separate engineering-generalization corpus.

Hidden grading metadata stayed server-side.

Provider calls remained bounded.

Everything looked good.

Then I used the real model.

The real game felt worse

One puzzle is about a guest waking without sound or physical contact.

The hidden mechanism is a visual light signal.

I tried this:

is it light?

Version 2:

Not enough information

Then:

is it light that wakes the guest?

Version 2:

Not enough information

I submitted the exact idea as a theory:

is it light that wakes the guest?

Version 2:

Not quite yet.

That was already worrying.

Then I tried:

does the wake up mechanism make sound?

Result:

Please rephrase

And even:

does the wake up mechines make sound?

produced:

Please rephrase

The word mechines is obviously a typo.

But the intended meaning is still easy for a human to recover.

At this point the architecture was technically clean but the actual Game Master felt strangely rigid.

What may have gone wrong

There was probably no single cause.

In fact, Version 3 changed several things at once, so I cannot claim that any one of them independently fixed the problem.

But four design problems became obvious.

1. The 0.8 thresholds may have been too conservative

This was my first suspicion.

Suppose Jev had actually returned something like:

wellFormed = 0.74
relevant = 0.92
contradicted = 0.89

for:

does the wake up mechines make sound?

That would mean the system understood quite a lot.

But my code did this first:

if (wellFormed < 0.8) {
  return "REPHRASE";
}

So all the useful information below that gate became irrelevant.

The probability was continuous.

My application policy turned it into a cliff.

0.7999 → rejected
0.8000 → accepted

A lower threshold might have improved the result.

I still think this is a plausible explanation.

But Version 2 had several thresholds, so lowering one would not necessarily solve the whole problem.

2. Several uncertain decisions were connected with hard gates

The deeper problem was not just that 0.8 might have been too high.

There were several such boundaries.

The system effectively became:

IF recoverable enough
AND relevant enough
AND supported enough
AND not contradicted enough
AND ...
THEN YES

Each semantic dimension could be individually reasonable.

But the final behavior depended on all of them interacting correctly.

I had replaced one uncertain AI decision with several uncertain AI decisions and connected them using deterministic cliffs.

That made the overall product more brittle.

3. I decomposed things that humans understand together

Consider:

does the wake up mechines make sound?

A human probably does not consciously evaluate:

grammar quality
→ relevance
→ referent resolution
→ semantic truth

as separate numerical decisions.

We do something more like:

"mechines" probably means "mechanism"
        ↓
They mean the wake-up mechanism
        ↓
They are asking whether it makes sound
        ↓
No

It is one semantic interpretation.

Version 2 decomposed that interpretation into multiple scores.

That was elegant from a TypeScript perspective.

It may have been unnatural from a semantic-decision perspective.

4. The judge knew the scenario, but not explicitly what mystery was being solved

This became one of the most useful changes.

Take:

is it light?

As isolated English, it is ambiguous.

But this is not isolated English.

The player is solving a specific mystery.

For the quiet-alarm puzzle, the actual question being investigated is approximately:

What causes the sleeping guest to wake
when the alarm makes no sound
and nothing touches the guest?

A human Game Master naturally interprets:

is it light?

as:

Is light what causes the guest to wake?

Version 2 had the scenario and hidden answer, but it did not have an explicit representation of the mystery focus.

That changed in Version 3.

Version 3: one semantic decision per judge

Instead of trying to improve Version 2 by adding more examples, more thresholds, or more scores, I removed complexity.

The Question Judge now makes one categorical semantic decision.

Conceptually:

type QuestionSemanticClass =
  | "SUPPORTED"
  | "CONTRADICTED"
  | "MIXED"
  | "UNKNOWN"
  | "IRRELEVANT"
  | "AMBIGUOUS";

Jev returns one probability distribution over those choices.

Then TypeScript performs a deterministic mapping:

SUPPORTED
→ YES

CONTRADICTED
→ NO

MIXED
→ PARTLY

UNKNOWN
→ NOT ENOUGH INFORMATION

IRRELEVANT
→ NOT RELEVANT

AMBIGUOUS
→ PLEASE REPHRASE

There is no active:

wellFormed >= 0.8

gate anymore.

There is no chain of independent semantic Nouls.

There is one semantic classification.

The Solution Judge was simplified in the same way

Version 2 had five independent signals.

Version 3 has one classification:

type SolutionSemanticClass =
  | "CORE_CAUSAL_EXPLANATION"
  | "CORE_WITH_CONTEXT_ERROR"
  | "SUPPORTING_INSIGHT_ONLY"
  | "COMPETING_WRONG_MECHANISM"
  | "NO_MEANINGFUL_INSIGHT";

The product mapping remains deterministic:

CORE_CAUSAL_EXPLANATION
→ SOLVED

CORE_WITH_CONTEXT_ERROR
→ PARTIAL

SUPPORTING_INSIGHT_ONLY
→ PARTIAL

COMPETING_WRONG_MECHANISM
→ NOT_SOLVED

NO_MEANINGFUL_INSIGHT
→ NOT_SOLVED

So I did not abandon semantic distinctions.

I reduced the number of separate decisions required to produce them.

I also added `mysteryFocus`

Every puzzle now includes a small server-only description of what the player is trying to explain.

For example:

What causes the sleeping guest to wake
when the alarm makes no sound
and nothing touches the guest?

This does not contain the answer.

It only gives the judge the same contextual frame that a human Game Master naturally has.

That helps with short language such as:

is it light?
does it buzz?
is it moving?

without creating hard-coded phrase exceptions.

I shortened the shared rubric

Another change was less visible but important.

Earlier rubrics had gradually accumulated semantic instructions and special distinctions.

Version 3 intentionally moved back toward a smaller question:

Which semantic category best describes this player's statement in this puzzle?

The rubric still tells the model to tolerate:

non-native English
minor spelling mistakes
missing articles
telegraphic wording
ordinary shorthand
natural pronoun resolution

But it does not ask for several independent measurements of those properties.

Again:

Decide less.

What happened after the simplification?

The difference was immediate.

Here are real protected-preview results.

Before: Version 2

is it light?
→ Not enough information

is it light that wakes the guest?
→ Not enough information

does the wake up mechanism make sound?
→ Please rephrase

does the wake up mechines make sound?
→ Please rephrase

is it light that wakes the guest?
[Theory]
→ Not quite yet.

Now compare that with Version 3.

After: Version 3

is it light?
→ Yes

is brightness involved?
→ Yes

something bright wake him?
→ Yes

This is especially important.

The grammar is poor:

something bright wake him?

But the intended meaning is recoverable.

The new judge treated it that way.

Then:

does the wake up mechines make sound?
→ No

The typo no longer caused the system to reject the whole question.

Similarly:

does it buzz?
→ No

and:

is there vibration?
→ No

Short questions also became usable.

Mixed statements started behaving correctly too

I tried:

something bright wake him?
does the wake up mechines make sound?

The result was:

Partly

with the UI:

Some of that is right, but another part isn't.

This is exactly what PARTLY was intended to mean.

Not uncertainty.

Not closeness.

Actual mixed semantic truth.

Most importantly, concise correct theories started solving the puzzle

Question:

does light wake him?

Result:

Yes

Then I reused the exact same text as a theory:

does light wake him?

Result:

You got it.

Another phrasing:

the room gets bright and that wakes him

Question:

Yes

Theory:

You got it.

That is much closer to how a human Game Master should behave.

Wrong causal mechanisms still failed

Simplification did not mean blindly accepting anything containing the correct keyword.

For example:

lamp is there but vibration wakes him

Question:

No

Theory:

Not quite yet.

That matters because false SOLVED results are much worse for this game than conservative partial judgments.

So far, Version 3 became more tolerant without immediately destroying the distinction between correct and incorrect causal explanations.

The behavior also generalized to another puzzle

I tested a different puzzle involving an apparently empty frame whose contents seem to change.

Some examples:

is the frame big
→ Not relevant

is it an animal
→ No

is it a TV
→ No

is it a machine
→ No

Then:

it changes its color with sun light
→ Yes

Submitted as a theory:

it changes its color with sun light
→ You got it.

Again, the grammar is not perfect.

But the causal idea is correct.

And the judge accepted it.

Version 3 is not perfect

There is still some instability around boundary cases.

For example, during one session:

is it the sunrise and sunset
→ Yes

while similar wording later produced:

is it sunrise and sunset
→ Not enough information

That tells me the system is not magically deterministic at the semantic level.

Nor should I expect it to be.

The important difference is that the major player-facing failures changed from:

"I clearly understand what you're asking,
but please rephrase."

to occasional disagreement around genuinely fuzzy boundaries.

For a proof-of-concept game, that is a much better failure mode.

So what actually fixed it?

The honest answer is:

I do not know which individual change fixed it.

Version 3 changed several things together:

Removed multiple continuous semantic gates

Removed the 0.8/0.2 decision chain

Replaced many Nouls with one Choice distribution

Added mysteryFocus

Shortened and generalized the rubric

Made language recovery part of the single semantic classification

Any one of those may have contributed.

The improvement is evidence that the overall architecture became better.

It is not a controlled experiment proving that 0.8 alone was the problem.

A useful future experiment would compare Version 2 under several threshold policies.

That could answer:

Was Jev already producing useful semantic probabilities that my application was throwing away?

I think that is entirely possible.

Jev started feeling less like "adding AI" and more like building probabilistic IF statements

This project changed how I think about decision models.

I am not really building a chatbot.

I am building something conceptually closer to:

if (semanticCondition) {
  doSomething();
}

Except the condition is not:

x > 10

It is:

Does this player's sentence semantically express
a proposition supported by the hidden explanation?

Jev estimates that semantic condition.

TypeScript decides what happens next.

That is why I increasingly think of this pattern as:

probabilistic IF statements over natural language

or:

an AI-powered switch/case

Version 2 accidentally turned that simple idea into a giant conditional expression.

Version 3 moved it back toward a small semantic switch.

There is also a hidden engineering cost: calibration

Another lesson was that API pricing is not the only cost of using a decision model.

For this project I needed to repeatedly:

design semantic categories
test real player wording
test spelling mistakes
test short questions
test false mechanisms
test incomplete solutions
check false positives
check false negatives
change policies
retest on protected Preview

That is human labor.

Jev reduced one kind of complexity for me.

I did not have to parse arbitrary assistant prose into application state.

But that complexity did not disappear.

Some of it moved into:

decision design
calibration
threshold selection
test-play
behavioral evaluation

I would therefore not say:

Jev requires more engineering work than a traditional LLM.

I have not run a controlled comparison that proves that.

A more accurate statement is:

In this project, Jev moved a meaningful part of the engineering effort from output handling to decision design and calibration.

That is an important cost to account for.

849 tests still did not replace test-play

Version 3 currently passes:

849 unit/API tests
82 desktop/mobile E2E tests
Leak scanning
Lint
Type checking
Build

Those tests are valuable.

They prove things such as:

the parser rejects invalid distributions
the deterministic mapping works
private grading data does not reach the browser
provider call counts remain bounded
the UI still behaves correctly

But they still cannot fully prove:

A human writes an unseen sentence
        ↓
The model understands it as intended

That requires actual behavioral testing.

For AI systems I now think about validation as two separate layers:

Layer 1
Software correctness

Layer 2
Model behavior

A green CI pipeline is excellent evidence for Layer 1.

It is not sufficient evidence for Layer 2.

Why I am stopping here

There are still things I could tune.

I could add more categories.

I could add confidence gates.

I could add more examples.

I could create exceptions for phrases like:

sunrise and sunset

I am deliberately not doing that.

That path is exactly how Version 2 became complicated.

For this PoC, Version 3 is now a strong freeze candidate.

The remaining requirement is not:

perfect semantic classification

It is:

Natural short player language usually works.

Minor grammar and spelling errors usually work.

Correct causal explanations can solve the puzzle.

Wrong causal explanations do not solve it.

Hidden information stays hidden.

The game remains fun.

That is enough.

What if this still breaks later?

I already have a fallback.

Simplify even further:

YES
OTHER

OTHER would intentionally collapse:

NO
UNKNOWN
IRRELEVANT
AMBIGUOUS
possibly MIXED

That would lose information.

But if it produced a better game experience, I would seriously consider it.

Again:

The goal is not to extract the maximum amount of semantic information from the model.

The goal is to ask for the smallest useful decision.

Final takeaway

The most surprising lesson from building Sideways has been this:

More semantic structure does not automatically produce better AI behavior.

Version 2 looked more sophisticated.

It had more probabilities.

More semantic axes.

More explicit thresholds.

More detailed grading logic.

And worse real gameplay.

Version 3 asked Jev to do less:

One semantic Choice
        ↓
One probability distribution
        ↓
Deterministic TypeScript switch

And the Game Master got noticeably better.

So the principle I am taking forward is:

Do not ask AI to make a more complicated decision than your product actually needs.

Or even shorter:

Understand enough.

Decide less.

That turned out to be a much better architecture for this game.

When More Semantic Signals Made My AI Judge Worse: Building a Lateral Thinking Game with Jev

494 — Sun, 04 Oct 2026 12:48:10 +0000

When More Semantic Signals Made My AI Judge Worse

I have been building a small web game called Sideways.

It is a lateral-thinking puzzle game: the player sees a mysterious situation and tries to discover what really happened by asking questions.

A typical interaction should feel almost trivial:

Player:
Is it light that wakes the guest?

Game Master:
Yes.

Or:

Player:
Does the wake-up mechanism make sound?

Game Master:
No.

The interesting part is that the Game Master is powered by AI.

I chose Jev because I did not want a chatbot generating long answers. I wanted a decision layer that could take application state, make a typed judgment, and let ordinary TypeScript decide what the product should do.

Jev describes itself as a decision model rather than a chat model. Its API works with application state and typed questions such as Choice, Score, and Noul, returning structured results and probabilities rather than conversational prose.

That sounded almost perfect for this game.

But I discovered something unexpected:

The more semantic intelligence I tried to extract from the model, the worse the actual game experience became.

This article is about that failure.

And why I am now making the AI do less.

The Architecture I Wanted

From the beginning, I did not want the model to control the application directly.

The architecture was:

Player text
    ↓
Probabilistic semantic judgment
    ↓
Typed output
    ↓
Deterministic TypeScript policy
    ↓
Player-facing result

The principle was simple:

AI = semantic judgment
Code = deterministic policy
User = consequential intent

The model should understand language.

My code should decide what happens next.

This has several advantages.

The browser never needs to receive the hidden solution.

The model does not get to decide arbitrary UI behavior.

The public API can expose only simple results such as:

YES
NO
PARTLY
IRRELEVANT
UNCLEAR

while keeping the private reasoning signals on the server.

For solution attempts, the public results are similarly constrained:

SOLVED
PARTIAL
NOT_SOLVED

Conceptually, I still believe this architecture is correct.

The problem was how much semantic structure I asked the model to produce.

Version 1: Simple Evidence

The first Question Judge was relatively simple.

For a player's question, the model estimated something like:

{
  wellFormed: number,
  relevant: number,
  evidence: {
    entailed: number,
    contradicted: number,
    undetermined: number
  }
}

TypeScript then converted these probabilities into a product result.

For example:

if (wellFormed < 0.8) return "UNCLEAR";
if (relevant <= 0.2) return "IRRELEVANT";
if (entailed >= 0.8) return "YES";
if (contradicted >= 0.8) return "NO";

return "UNCLEAR";

This had an important property:

A low probability of YES did not automatically mean NO.

An unknown fact could remain unknown.

That part worked well.

But actual gameplay revealed another problem.

Natural player language is messy.

People write things like:

light wake him?

does it make sound

maybe lamp?

They make spelling mistakes.

They omit articles.

They use pronouns.

They do not write propositions like lawyers.

So I tried to make the judge more sophisticated.

Version 2: More Semantic Intelligence

In Version 2, the Question Judge expanded into:

{
  wellFormed,
  relevant,
  evidence: {
    entailed,
    contradicted,
    undetermined
  },
  mixedClaims: {
    supported,
    contradicted
  }
}

This allowed a new result:

PARTLY

For example:

There is no alarm. Only a lamp wakes the guest.

contains both a false claim and an important true claim.

I did not want that to be treated as simply NO.

For solution checking, I went even further.

Instead of judging a theory with a simple checklist, the model returned five semantic dimensions:

{
  coreMechanism,
  causalRelation,
  supportingInsight,
  contextualError,
  competingMechanism
}

The idea was reasonable.

Consider these three theories:

There is a light.

The light wakes the guest.

There is a light, but vibration wakes the guest.

They should not receive the same result.

I wanted:

"There is a light."
→ PARTIAL

"The light wakes the guest."
→ SOLVED

"Light exists, but vibration wakes the guest."
→ NOT_SOLVED

So I separated:

identifying the important object;
understanding the causal mechanism;
making a harmless contextual mistake;
asserting a competing wrong mechanism.

Architecturally, this looked much better.

And the Offline Tests Looked Excellent

The implementation passed:

880 unit/API tests
82 desktop/mobile E2E tests
Leak scans
Lint
Type checking
Build
Full repository checks

I also created separate calibration and engineering-generalization corpora across all six puzzles.

The browser did not receive private grading data.

The hidden solution remained server-only.

The number of provider calls remained bounded.

Everything looked clean.

But there was one very important limitation.

Most of those tests proved:

semantic signal
→ parser
→ deterministic policy
→ API result
→ UI

They did not prove that the real model would infer the expected semantic signal from previously unseen player language.

That distinction turned out to matter a lot.

Then I Played the Game with the Real Model

I deployed the new judge to a protected preview environment and started typing ordinary questions.

The results were surprising.

Example 1

Player:
is it light?

Result:
Not enough information

Example 2

Player:
is it light that wakes the guest?

Result:
Not enough information

Then I submitted the exact same sentence as a theory:

Theory:
is it light that wakes the guest?

Result:
Not quite yet.

A human Game Master would almost certainly understand the intended idea.

The hidden solution is that a visual light signal wakes the sleeping guest.

Example 3

Player:
does the wake up mechanism make sound?

Result:
Please rephrase

Even this typo-heavy version:

does the wake up mechines make sound?

also produced:

Please rephrase

That was a serious UX problem.

The grammar is imperfect, but the semantic intent is obvious.

Example 4

Player:
Instead of an alarm, lights from a device wakes the guest.

Question result:
Not enough information

Theory result:
Not quite yet.

Again, the wording is imperfect.

But the important causal insight is clearly present.

Why Did More Structure Make Things Worse?

I cannot infer Jev's internal reasoning from its output alone.

So the following are hypotheses about my decision architecture, not claims about the internals of the model.

Several problems stood out.

1. The Semantic Dimensions Were Not Really Independent

I had separated concepts like:

well-formedness
relevance
truth
mixed claims
causal understanding

because it made the TypeScript architecture easier to reason about.

But humans do not necessarily understand a sentence in that order.

Take:

does the wake up mechines make sound?

A human probably performs something closer to:

"mechines" is probably "mechanism"
        ↓
The player means the wake-up mechanism
        ↓
They are asking whether it makes sound
        ↓
I know the answer
        ↓
No

This is one integrated semantic interpretation.

I had converted it into several semi-independent judgments.

That created more opportunities for one signal to disagree with the others.

2. Probabilities Became Hard Cliffs

The intention behind probabilities was to avoid brittle Boolean decisions.

But then my code contained rules such as:

if (wellFormed < 0.8) {
  return "REPHRASE";
}

Imagine the model essentially understands the sentence but produces:

wellFormed = 0.77

The rest of the semantic evidence no longer matters.

The application suddenly says:

Please rephrase.

A continuous probability had become a discrete cliff.

Adding more semantic dimensions meant adding more places where such cliffs could occur.

The model did not necessarily need to be dramatically wrong.

My policy only needed one signal to fall on the wrong side of a threshold.

3. I Was Asking a Small Decision Model to Solve a Large Semantic Task

Jev's documentation emphasizes focused, well-scoped typed decisions. Choice is designed for predefined classification, while Noul represents a yes/no judgment whose probability can be interpreted by application code.

But my Question Judge was effectively asking the model to do all of this at once:

Recover imperfect language
Resolve pronouns
Determine the semantic proposition
Determine relevance
Compare with hidden truth
Detect multiple claims
Separate supported and contradicted claims
Distinguish uncertainty from contradiction

The individual fields looked small.

The total semantic task was not.

I had decomposed the output without necessarily decomposing the difficulty.

4. The Judge Knew the Scenario, but Not the "Mystery Focus"

This turned out to be especially interesting.

Consider:

is it light?

As an isolated English sentence, it is ambiguous.

What is "it"?

The alarm?

The mechanism?

The object?

But in a lateral-thinking game, a human Game Master has another piece of context:

What mystery is the player currently trying to explain?

For the quiet-alarm puzzle, that focus is roughly:

What causes the sleeping guest to wake
without sound or physical contact?

Given that context:

is it light?

has a very natural interpretation:

Is light the cause?

I had provided the scenario and hidden reference, but not an explicit representation of the question being investigated.

That may have encouraged overly literal ambiguity handling.

5. My Solution Judge May Have Confused Completeness with Correctness

The Version 2 Solution Judge required strong scores for both:

coreMechanism
causalRelation

before returning SOLVED.

That sounds reasonable.

But consider:

The light wakes the guest.

This is extremely short.

It also contains both the mechanism and causal relationship.

If the model gives:

coreMechanism = high
causalRelation = medium

the deterministic policy returns PARTIAL.

From the application's perspective, that is a false negative.

The player has solved the mystery.

The grading representation made the answer look less complete than it actually was.

Version 3: Make the Model Do Less

So I am now testing a different architecture.

Instead of asking for many independent semantic scores, each judge will return one categorical probability distribution.

For Question:

type QuestionSemanticClass =
  | "SUPPORTED"
  | "CONTRADICTED"
  | "MIXED"
  | "UNKNOWN"
  | "IRRELEVANT"
  | "AMBIGUOUS";

Then ordinary TypeScript maps it:

SUPPORTED
→ YES

CONTRADICTED
→ NO

MIXED
→ PARTLY

UNKNOWN
→ NOT ENOUGH INFORMATION

IRRELEVANT
→ NOT RELEVANT

AMBIGUOUS
→ PLEASE REPHRASE

There is still probabilistic information.

But there is only one semantic decision.

No chain of:

0.8
0.8
0.2
0.8
...

gates.

The Solution Judge Is Being Simplified Too

Instead of five independent dimensions, I am testing one semantic classification:

type SolutionSemanticClass =
  | "CORE_CAUSAL_EXPLANATION"
  | "CORE_WITH_CONTEXT_ERROR"
  | "SUPPORTING_INSIGHT_ONLY"
  | "COMPETING_WRONG_MECHANISM"
  | "NO_MEANINGFUL_INSIGHT";

TypeScript then maps:

CORE_CAUSAL_EXPLANATION
→ SOLVED

CORE_WITH_CONTEXT_ERROR
→ PARTIAL

SUPPORTING_INSIGHT_ONLY
→ PARTIAL

COMPETING_WRONG_MECHANISM
→ NOT_SOLVED

NO_MEANINGFUL_INSIGHT
→ NOT_SOLVED

The important distinction remains.

There is a light.

should not equal:

The light wakes the guest.

And neither should equal:

The vibration wakes the guest.

But the model no longer needs to independently estimate five continuous semantic dimensions before my application can make that distinction.

I Am Also Adding a "Mystery Focus"

Each puzzle will have a small server-only field such as:

What causes the sleeping guest to wake
without sound or physical contact?

This is not the hidden answer.

It simply represents the question raised by the public scenario.

The hope is that it helps resolve natural shorthand:

is it light?

does it make sound?

was he covered?

without adding phrase-specific exceptions.

That distinction matters.

I do not want:

if text === "is it light?"

I want:

short player language
+
current mystery
→ recoverable semantic proposition

And If Version 3 Still Fails?

Then I plan to simplify again.

Possibly all the way to:

YES
OTHER

For Question judging:

YES

would mean:

This player proposition is supported by the hidden explanation.

Everything else would collapse into:

OTHER

That would intentionally stop distinguishing among:

NO
UNKNOWN
IRRELEVANT
AMBIGUOUS

This loses information.

It may still produce a better game.

That is the part of this experiment I find most interesting.

The Product Does Not Need the Maximum Amount of AI Information

It is tempting to think:

More semantic signals
→ more information
→ better product decisions

My experience so far suggests something more like:

More semantic signals
→ more uncertain boundaries
→ more policy interactions
→ potentially worse UX

The best AI interface may not be the one that extracts the most information from a model.

It may be the one that asks the model for the smallest decision the product actually needs.

Another Lesson: Offline Tests and Semantic Accuracy Are Different Things

I had hundreds of passing tests.

They were useful.

They caught:

schema mistakes;
API regressions;
accidental hidden-data leaks;
extra provider calls;
incorrect deterministic mappings;
broken UI behavior.

I absolutely want those tests.

But:

synthetic semantic signals
→ correct product output

does not prove:

previously unseen human language
→ correct semantic signal

Those are different systems.

For AI products, both need testing.

I now think of them as two separate validation layers:

Layer 1:
Software correctness

Layer 2:
Model behavior under real language

A green CI pipeline proves the first.

It does not automatically prove the second.

Why I Still Like the Typed-Decision Architecture

None of this has made me want to replace the system with a free-form chatbot.

Actually, the opposite.

I still want:

AI
→ bounded semantic decision

TypeScript
→ application policy

Jev's API is explicitly designed around application state and typed decisions, with probability distributions that software can consume.

The lesson for me is not:

Typed decisions are too simple.

It is:

I should respect their simplicity.

If the application needs one classification, asking for one well-designed Choice may be better than constructing a small semantic ontology and connecting it with threshold gates.

Where the Experiment Stands

At the time of writing:

v1
Simple probabilistic evidence
        ↓

v2
More semantic dimensions
        ↓

Real-model testing exposed brittle behavior
        ↓

v3
One categorical semantic distribution per judge
+ explicit mystery focus
        ↓

Currently being tested

If v3 performs well on previously unseen wording, I will freeze the Judge and move on to production hardening.

If it does not, I will test the YES / OTHER design rather than adding more semantic complexity.

That constraint is intentional.

At some point, engineering discipline means stopping optimization.

Final Takeaway

The most useful lesson from this project so far is surprisingly simple:

Do not ask the AI to make a more complicated decision than your product actually needs.

A sophisticated semantic representation can be elegant in TypeScript.

It can have clean types.

It can have excellent unit tests.

It can even look more theoretically correct.

And still produce a worse experience for the person actually playing the game.

Sometimes the better AI architecture is not:

Understand more.

It is:

Decide less.

I will update this article once the simplified Version 3 Judge has been tested against the real model.

About the project

Sideways is an experimental lateral-thinking puzzle game built with Next.js, TypeScript, and Jev.

The project explores a specific architecture:

Probabilistic Semantic Judgment
→ Typed Output
→ Deterministic TypeScript Policy
→ Product Result

The next question is how small that semantic judgment can become while still producing a good game.

DEV Community: 494

When Simpler AI Decisions Worked Better: What I Changed in My Jev-Powered Game Judge

The original idea: AI understands meaning, code controls behavior

Version 2: the judge became too clever

And the software tests looked excellent

The real game felt worse

What may have gone wrong

1. The 0.8 thresholds may have been too conservative

2. Several uncertain decisions were connected with hard gates

3. I decomposed things that humans understand together

4. The judge knew the scenario, but not explicitly what mystery was being solved

Version 3: one semantic decision per judge

The Solution Judge was simplified in the same way

I also added mysteryFocus

I shortened the shared rubric

What happened after the simplification?

Before: Version 2

After: Version 3

Mixed statements started behaving correctly too

Most importantly, concise correct theories started solving the puzzle

Wrong causal mechanisms still failed

The behavior also generalized to another puzzle

Version 3 is not perfect

So what actually fixed it?

Jev started feeling less like "adding AI" and more like building probabilistic IF statements

There is also a hidden engineering cost: calibration

849 tests still did not replace test-play

Why I am stopping here

What if this still breaks later?

Final takeaway

When More Semantic Signals Made My AI Judge Worse: Building a Lateral Thinking Game with Jev

When More Semantic Signals Made My AI Judge Worse

The Architecture I Wanted

Version 1: Simple Evidence

Version 2: More Semantic Intelligence

And the Offline Tests Looked Excellent

Then I Played the Game with the Real Model

Example 1

Example 2

Example 3

Example 4

Why Did More Structure Make Things Worse?

1. The Semantic Dimensions Were Not Really Independent

2. Probabilities Became Hard Cliffs

3. I Was Asking a Small Decision Model to Solve a Large Semantic Task

4. The Judge Knew the Scenario, but Not the "Mystery Focus"

5. My Solution Judge May Have Confused Completeness with Correctness

Version 3: Make the Model Do Less

The Solution Judge Is Being Simplified Too

I Am Also Adding a "Mystery Focus"

And If Version 3 Still Fails?

The Product Does Not Need the Maximum Amount of AI Information

Another Lesson: Offline Tests and Semantic Accuracy Are Different Things

Why I Still Like the Typed-Decision Architecture

Where the Experiment Stands

Final Takeaway

About the project

I also added `mysteryFocus`