LLMs Generate. Jev Decides. Software Should Know the Difference

Bringing TypeSafe's System One model into a real Java application with The Pipeline Framework

For the last few years, we have been solving an extraordinary range of software problems with essentially the same primitive:

send some context to a large language model and ask it to generate the answer.

That has worked relatively well, but it has also encouraged us to turn problems that are not fundamentally generative into generation problems.

Classification becomes generation.

Routing becomes generation.

Selecting one item from a known catalogue becomes generation.

Determining whether a condition holds becomes, you guessed it: generation.

And then, because a generative model is free to generate almost anything, we spend an increasing amount of effort constraining it:

Return exactly one of these values.

Do not invent identifiers.

Return exactly this JSON structure.

Do not include an explanation.

Do not wrap the result in another object.

Never return anything outside the supplied catalogue.

Structured output has made this considerably better. But there is still something slightly peculiar about the underlying architecture.

We are using a machine designed to generate an open-ended sequence of tokens, and then asking it very politely not to be open-ended.

TypeSafe's Jev presents a fascinating alternative.

And when we integrated Jev into The Pipeline Framework and then used it in a real invoice-processing application, something became very clear:

some of the work we were giving to an LLM was never an LLM problem in the first place.

---

System One: AI as a decision function

TypeSafe describes Jev as the first of its System One Models: models designed not primarily to generate language, but to make fast, structured decisions that software can consume directly.

Instead of an open-ended generation operation, the abstraction is closer to a bounded decision function.

flowchart LR
    S[State] --> G[Generative model]
    G --> T[Open-ended tokens]

versus:

flowchart LR
    S[State] --> D[Bounded questions]
    D --> J[System One model]
    J --> R[Probabilistic decisions]

The distinction is profound.

TypeSafe currently exposes three particularly useful decision shapes: Choice, Noul, and Score.

A Choice selects from a finite set of alternatives and returns the corresponding probability information.

A Noul evaluates a proposition probabilistically.

A Score evaluates something against an ordered scale.

Several independent questions can be evaluated against the same state in one request.

The output is not primarily prose intended for a human reader. It is information intended for software.

That changes what the model is being asked to do.

Consider the difference.

A generative approach says:

Read this invoice and tell me which property it belongs to. Return exactly one ID from this list and do not invent another one.

A bounded decision instead defines:

Which property?

Choice:
    PROPERTY_A
    PROPERTY_B
    PROPERTY_C

In the first case, selecting a valid identifier is a prompt requirement.

In the second, it is part of the decision domain.

That is a much stronger contract.

---

We happened to have exactly the right application

At Mokapot Labs we maintain a small application called Invoice Assistant.

It is built with The Pipeline Framework (TPF), an open-source framework for constructing typed processing applications.

The application receives an invoice and a property catalogue, extracts invoice information, determines which property the invoice belongs to, optionally performs visual analysis when the supplier cannot be established from text, presents the result for human confirmation, and then performs the required external effects.

Its pipeline includes ordinary computation, AI inference, branching, a human Await boundary and replay-safe Commands.

Before Jev, its text-analysis stage looked roughly like this:

flowchart TD
    A[Invoice] --> B[Extract document text]
    B --> C[Gemma 12B]
    C --> C1[Extract supplier/invoice nr/amount]
    C --> C4[Classify supplier evidence]
    C --> C5[Select property]
    C --> C6[Explain recommendation/w note]
    C1 --> D[Route supplier evidence]
    C4 --> D
    C5 --> D
    C6 --> D
    D -->|Sufficient| E[Review]
    D -->|Insufficient| F[Vision model]

This worked.

It was not disastrously slow. It was not unreliable enough to force a redesign.

But the LLM call was doing several fundamentally different jobs.

And our prompt showed it.

---

The old prompt contained the clue

Part of the original prompt said:

Classify supplier evidence with exactly one qualitative
supplierEvidenceStatus:

EXPLICIT_TEXT
STRONG_TEXTUAL_IDENTITY
INSUFFICIENT

Another part said:

Select exactly one property ID present in the supplied
catalogue.

Do not invent, rewrite or normalize property IDs.

Read those requirements again in the context of a System One model.

They describe Choices.

We had an LLM generating a result and a prompt instructing it to behave as though the output space were closed.

But the output space really was closed.

For supplier evidence, there were three possible answers.

For the property recommendation, the possible answers were precisely the properties already supplied to the application.

The problem was not:

Generate a property identifier.

It was:

Choose one of these properties.

That difference sounds small.

Architecturally, it is enormous.

---

But not everything was a Choice

This is where the distinction becomes useful rather than ideological.

The application also needs to determine:

supplier      = "Some arbitrary company name"
invoiceNumber = "INV-2026-18473"
totalAmount   = 68.52

Those are open-world values.

The supplier can be a string we have never encountered before.

The invoice number is arbitrary.

The amount is arbitrary.

These are extraction problems, and our existing generative LLM remains well suited to them.

So we did not replace the LLM with Jev.

We split the problem according to its semantics.

The result became:

flowchart TD
    A[Invoice] --> B[Extract document text]
    B --> C["Generative LLM<br/>Gemma"]
    C --> C1[Supplier]
    C --> C2[Invoice number]
    C --> C3[Total amount]

    C1 --> D[Prepare DecisionRequest]
    C2 --> D
    C3 --> D

    D --> J["Jev<br/>System One"]
    J --> J1["Supplier evidence<br/>Choice"]
    J --> J2["Property<br/>Choice"]

    J1 --> P[Probabilistic judgments]
    J2 --> P

    P --> R[Deterministic application policy]
    R -->|Sufficient evidence| H[Review]
    R -->|Insufficient evidence| V["Existing vision model<br/>VLM"]

The generative model generates.

The decision model decides.

The application governs what happens next.

That separation turned out to be more important than simply changing model providers.

---

The LLM prompt got dramatically smaller

The revised generative step now has a much narrower job:

Extract only the supplier, invoice number and total amount
from the invoice evidence.

And, perhaps more revealingly:

Do not classify supplier evidence,
select a property,
explain a choice,
generate a note...

The LLM is no longer responsible for everything that happens to involve semantic understanding.

It is responsible for the part of the problem that genuinely requires open-ended extraction.

That is an important architectural lesson.

"Uses AI" is not a sufficient reason for two operations to belong to the same model call.

Their semantic shapes matter.

---

The System One decision is ordinary Java

There was another complication.

TypeSafe currently publishes official Python and JavaScript/TypeScript SDKs, but not an official Java SDK.

Invoice Assistant is a Java application.

We could have waited.

We could have introduced Python into the application.

We could have written a thin application-specific HTTP client.

Instead, this became an opportunity to establish something more reusable in The Pipeline Framework:

a provider-neutral Java protocol for bounded AI decisions.

The application constructs an ordinary DecisionRequest.

For supplier evidence, it declares:

new DecisionQuestion(
    "supplierEvidence",
    DecisionQuestionType.CHOICE,
    "Judge only the strength of textual evidence identifying the invoice supplier.",
    List.of(
        new DecisionCriterion(
            "EXPLICIT_TEXT",
            "The issuer or supplier is directly labelled or named."),
        new DecisionCriterion(
            "STRONG_TEXTUAL_IDENTITY",
            "Several consistent textual identity cues identify the issuer."),
        new DecisionCriterion(
            "INSUFFICIENT",
            "Text is absent, generic, conflicting, or ambiguous.")))

The property decision is constructed dynamically from the application's actual property catalogue:

result.properties().forEach(property ->
    properties.add(
        new DecisionCriterion(
            property.id(),
            property.displayName()
                + "; "
                + property.canonicalAddress()
                + "; aliases: "
                + String.join(", ", property.aliases()))));

Then:

new DecisionQuestion(
    "property",
    DecisionQuestionType.CHOICE,
    "Select the supplied property best supported by the invoice evidence.",
    properties)

This is one of my favourite consequences of the change.

Previously we told the LLM:

Do not invent property IDs.

Now the set of possible property IDs is the decision.

The safety property moved from prose into structure.

---

One state, multiple judgments

The application constructs a state containing the information Jev actually needs:

extractedFacts
invoiceText
originalFilename
properties

and submits both questions together.

flowchart TD
    S["Decision state<br/>extractedFacts<br/>invoiceText<br/>originalFilename<br/>properties"]
    S --> R[DecisionRequest]
    R --> Q1["supplierEvidence<br/>Choice"]
    R --> Q2["property<br/>Choice"]
    Q1 --> J[Jev]
    Q2 --> J
    J --> A1["Supplier evidence judgment<br/>choice + probabilities"]
    J --> A2["Property judgment<br/>choice + probabilities"]

That maps naturally onto the System One model.

The application is not conducting an agent conversation with the model.

It is not asking one question, parsing the answer, constructing another prompt and asking another question.

It describes the state and the bounded judgments it requires.

The model evaluates them and returns structured results.

That is a much more application-shaped interaction.

---

The model judges. Software governs.

This may be the most important architectural property of the whole integration.

Jev does not decide whether the pipeline should execute visual analysis.

It does not decide whether the application should ask a human.

It does not execute another capability.

It doesn't become the workflow engine.

It returns judgments.

The application then applies policy.

flowchart TD
    J[Jev] --> P["Probabilistic judgments"]
    P --> A["Deterministic application policy"]
    A -->|Text evidence accepted| R[Review]
    A -->|More evidence required| V[Vision analysis]

This boundary matters.

A probabilistic model is excellent at answering questions such as:

Which supplied property is most strongly supported by this evidence?

It should not automatically acquire authority over:

What should the business process do next?

Those are different concerns.

TPF makes that distinction very natural.

---

This is where The Pipeline Framework gets interesting

At first glance, integrating a new AI model might sound like a framework feature:

kind: jev

We deliberately did not do that.

Jev is not a new kind of pipeline operation.

From TPF's perspective, Jev observes something the pipeline does not currently know.

That makes it a Query.

TPF's semantic rule is simple:

known execution-local data   → carry it
fresh external observation   → Query
external side effect         → Command
deferred external completion → Await

Whether the external observation came from PostgreSQL, an HTTP API, an LLM or a System One decision model does not fundamentally change that semantic boundary.

So the pipeline contains:

- name: Judge Invoice
  kind: query
  cardinality: ONE_TO_ONE
  input: InvoiceJudgmentRequest
  output: InvoiceJudgmentResult
  using: invoice-judgment-model
  operation: decide
  operationVersion: 1

and the provider binding is:

invoice-judgment-model:
  provider: decision.query.jev
  version: 1
  config:
    model: typesafe/jev-1.13
    connection: openrouter-primary

That is all the pipeline needs to know.

There is no Jev step kind.

There is no System One workflow engine.

There is no Jev-specific branch operator.

There is simply another typed external observation.

---

Provider-neutral protocol, provider-specific implementation

This distinction is crucial for avoiding framework lock-in.

The application does not model its domain using Jev's Python SDK classes.

Its canonical pipeline types refer to:

<tpf.decision.DecisionRequest>
<tpf.decision.DecisionResult>

The application code constructs:

DecisionRequest
DecisionQuestion
DecisionCriterion
DecisionQuestionType

The provider happens to be decision.query.jev today.

The conceptual layering is therefore:

flowchart TD
    A[Application domain]
    A --> P["TPF bounded-decision protocol<br/>DecisionRequest / DecisionResult"]
    P --> Q["TPF Query<br/>decide / v1"]
    Q --> X[Provider adapter]
    X --> J[Jev]
    X -. future .-> O[Another bounded-decision engine]

This is substantially different from writing an unofficial Java clone of TypeSafe's SDK.

TPF is defining the capability the application needs.

Jev is implementing it.

That leaves the application architecture independent of one vendor's client library.

---

Java gets a native System One path in the process

This has an interesting practical consequence.

TypeSafe's official SDKs currently target Python and JavaScript/TypeScript.

TPF applications can nevertheless use Jev from Java through ordinary typed application code.

There is no requirement for a Python sidecar.

flowchart TD
    J[Java application] --> DR[DecisionRequest]
    DR --> Q[TPF Query]
    Q --> A[decision.query.jev]
    A --> V[Jev]
    V --> RS[DecisionResult]
    RS --> J

Nor does application code need to manually construct vendor HTTP payloads.

And because the Java-facing contract is provider-neutral, this integration is useful beyond Jev itself.

It establishes bounded probabilistic decisions as an application capability in the Java ecosystem rather than merely exposing one vendor endpoint.

---

The output contract became better too

The old property recommendation contained:

propertyId
explanation

The new result contains:

propertyId
confidence
probabilities[]

Supplier evidence similarly carries:

status
confidence
probabilities[]

This represents a significant change in what the application expects from AI.

flowchart LR
    subgraph Before
        A1[AI] --> A2[Answer]
        A1 --> A3[Prose explanation]
    end

    subgraph After
        B1[AI] --> B2[Judgment]
        B1 --> B3[Confidence]
        B1 --> B4[Probability distribution]
    end

We had originally added explanations partly as a crutch while introducing the first real LLM Query into the application.

But prose explanation and model uncertainty are not the same thing.

A convincing explanation does not necessarily mean a model is confident.

A terse answer does not necessarily mean it is uncertain.

For application decisions, explicit probabilistic information is often much more useful.

The application can inspect it.

Policy can act on it.

Telemetry can record it.

Humans can see uncertainty where useful.

And future policy changes do not require rewriting a prompt merely to change what confidence means operationally.

---

Dead generative work disappeared

The refactoring also exposed functionality that no longer justified its existence.

The old model generated a short mnemonic note.

In practice, it was not used.

So it disappeared.

The recommendation explanation had largely existed to make early LLM behaviour inspectable.

It disappeared too.

This is another benefit of decomposing model responsibilities.

Large prompts have a tendency to accumulate requirements because adding another sentence feels cheap:

While you're there, also generate...

But inference responsibilities then become coupled together.

Once the application explicitly separates extraction, decision, policy and presentation, each output has to justify why it exists.

That is healthy architecture, AI or otherwise.

---

TPF's Functional Core / Imperative Shell model survives AI

One of the recurring problems in agentic systems is that the model gradually absorbs application architecture.

The model decides what to call.

The model decides whether to retry.

The model decides what state matters.

The model decides when the workflow is finished.

Eventually the "application" becomes a prompt wrapped around a tool registry.

TPF deliberately takes another direction.

The pipeline owns composition.

Canonical types own application contracts.

Queries own fresh observations.

Commands own external effects and, optionally, deferred completion.

Ordinary application functions own deterministic business policy.

AI fits inside those boundaries rather than replacing them.

Invoice Assistant demonstrates that particularly well.

The model can judge:

supplierEvidence =
    STRONG_TEXTUAL_IDENTITY

confidence =
    0.x

but ordinary application code determines what that means for the workflow.

Likewise, Jev may select a property, but it does not archive the invoice.

Archiving is an external effect and therefore remains a TPF Command.

Human property confirmation does not become an agent loop waiting in memory; that, remains an Await boundary.

flowchart TD
    P[Typed TPF pipeline]

    P --> Q[Query]
    P --> C[Command (and await)]

    Q --> O[Fresh observation]
    C --> E[External effect]

    O --> L[Generative LLM]
    O --> J[System One / Jev]
    O --> X[Database / API / other provider]

AI becomes part of software architecture rather than a replacement for it