Why AI Shouldn't Be the Decision Engine

Decisy — Fri, 18 Sep 2026 15:47:08 +0000

There's a pattern I've been thinking about for a while.

We keep getting better at making AI systems answer questions, and then we increasingly ask those same systems to make decisions.

The transition feels natural.

If a model can read a company's documents, understand its data, summarize the situation, identify risks, and explain possible options, why not just ask it what the company should do?

The problem is that the last step is fundamentally different from the others.

Generating an answer isn't the same thing as computing a decision.

And I think that distinction is going to become increasingly important as AI systems become more capable.

A decision is more than a prompt

Consider a relatively simple question:

Should we build this software ourselves or buy it from a vendor?

An LLM can produce a very reasonable answer almost immediately.

It might identify:

development cost
licensing cost
implementation time
customization requirements
engineering capacity
vendor dependency
maintenance cost
strategic importance

All useful.

But we're still missing something.

What are the relationships between these variables?

Suppose engineering capacity affects the expected delivery date.

Delivery date affects the opportunity cost.

The opportunity cost affects the economic value of building internally.

Vendor pricing affects the long-term cost of buying.

Expected customization affects both implementation cost and maintenance effort.

Now we have a system of dependencies.

Changing one assumption can change several downstream variables, which can ultimately change the decision.

That's not just text generation anymore.

It's a computational problem.

Where LLMs are genuinely useful

None of this means LLMs don't belong in decision systems.

They absolutely do.

In fact, they're particularly useful near the beginning of the process.

A business problem usually starts as messy human language:

"We're spending too much time maintaining this system, our engineering team is already stretched, and the vendor has offered us a new pricing model. Should we continue investing internally?"

There's a lot buried in that sentence.

An LLM can help extract things such as:

the decision being considered
possible alternatives
constraints
assumptions
relevant variables
objectives
missing information
questions that need clarification

That's a valuable job for an LLM.

It turns an unstructured problem into something that can be analyzed.

But once the problem has been structured, I don't think the language model should necessarily remain responsible for the mathematics.

Computation should be deterministic

Imagine that we're evaluating two alternatives using a financial model.

We have:

estimated implementation cost
recurring operating cost
expected revenue impact
probability of failure
time to deployment
discount rate

We calculate expected value under several scenarios.

If we run the same analysis tomorrow using exactly the same inputs, we should get the same result.

If another analyst runs it, they should get the same result.

If we change the probability of failure from 10% to 20%, we should be able to see exactly how the result changes.

That sounds obvious.

But it's surprisingly different from asking an LLM to reason about the same problem in natural language.

A model can produce a different explanation or conclusion based on subtle changes in context, model version, temperature, prompting, or even how the information was presented.

That's perfectly acceptable for language generation.

It's much harder to defend when we're talking about a financial calculation or statistical test that is supposed to be reproducible.

The algorithm shouldn't be hidden either

There's another problem with treating an LLM as the decision engine.

How did it decide which analytical method to use?

Was it comparing means?

Running a regression?

Using a Monte Carlo simulation?

Performing sensitivity analysis?

Calculating expected value?

Something else?

These methods answer different questions and make different assumptions.

A decision system should be explicit about that.

Ideally, you should be able to inspect something like:

Decision
   │
   ├── Objective
   │
   ├── Alternatives
   │
   ├── Assumptions
   │
   ├── Variables
   │
   └── Dependencies
          │
          ▼
     Analytical Method
          │
          ▼
       Execution
          │
          ▼
       Validation
          │
          ▼
        Result

The exact implementation can vary.

The important part is that the reasoning isn't trapped inside a black box.

Reproducibility matters

This becomes particularly important when decisions have consequences.

Suppose a company makes a significant investment decision today.

Six months later, someone asks:

"Why did we make this decision?"

A useful decision system should be able to answer more than:

"The AI recommended it."

You should ideally be able to reconstruct:

what information was available
what assumptions were made
which data was used
which analytical methods were applied
what scenarios were considered
what validation was performed
how sensitive the conclusion was to important assumptions

In other words, the decision should have a history.

That history is just as important as the final answer.

Where AI fits

I think a useful architecture looks something like this:

Human
  │
  │ Problem / Context
  ▼
AI-assisted extraction
  │
  │ Structured decision model
  ▼
Decision representation
  │
  ├── Assumptions
  ├── Variables
  ├── Constraints
  └── Dependencies
  │
  ▼
Deterministic analytics
  │
  ├── Statistical methods
  ├── Optimization
  ├── Simulation
  └── Sensitivity analysis
  │
  ▼
Validation
  │
  ▼
Synthesis
  │
  ▼
Human decision

The LLM isn't removed.

It's given a more appropriate role.

It helps translate between human language and structured analysis.

The analytical engine does the computation.

The validation layer challenges the result.

And the human remains responsible for deciding what to do.

There's a subtle benefit to this separation

It also makes the system easier to improve.

If an analytical method isn't appropriate, we can replace it.

If an assumption is wrong, we can change it.

If new data becomes available, we can rerun the analysis.

If the validation identifies a weakness, we can investigate it.

The components remain inspectable.

That's much harder when the entire process is essentially:

prompt → model → answer

The latter is incredibly powerful for many tasks.

But complex decisions deserve more structure.

This is the direction I'm interested in

I'm increasingly interested in what happens when we stop thinking about AI as the thing that makes the decision and start thinking about AI as one component of a broader decision system.

A system where:

AI handles ambiguity and unstructured information.

Algorithms handle computation.

Data provides evidence.

Validation challenges the analysis.

Humans provide judgment and accountability.

That separation doesn't make AI less useful.

I think it makes it considerably more useful.

Because the goal shouldn't be to have an AI system that confidently tells us what to do.

The goal should be to build systems that help us understand why a decision makes sense, what it depends on, and what could change it.

That's a much more interesting engineering problem.

DEV Community: Decisy