DEV Community

Kanwal Oswal
Kanwal Oswal

Posted on

Stop Asking LLMs “Does This Pass?” — Turn Policies Into Executable Rules Instead

#java #businessrules #python #llm

If you’ve worked with LLMs in real systems, you’ve probably tried something like this:

“Here’s a policy document. Here’s some input data.

Does this meet the policy?”

It works surprisingly well… at first.

But as soon as you move beyond demos, a few problems start to show up:

  • Results vary depending on phrasing or context
  • It’s hard to explain why a decision was made
  • The only audit trail is prompt + response
  • Re-running the same input doesn’t always guarantee the same output

This becomes a real issue in domains like:

  • financial services
  • compliance
  • eligibility systems
  • underwriting

Where decisions need to be consistent, explainable, and auditable.

The Core Problem

The issue isn’t that LLMs are bad.

It’s that we’re using them for the wrong part of the workflow.

We’re asking them to evaluate policies repeatedly at runtime, instead of using them to extract structured logic once.

A Different Approach

Instead of:
Policy Document + Input Data → LLM → Decision

What if we did:
Policy Document → LLM → Structured Rules
↓
Deterministic Engine
↓
Decision + Trace

This is the idea behind AeroRule.

👉 GitHub: https://github.com/kanwaloswal/AeroRule

What is AeroRule?

AeroRule is a lightweight rules engine designed around:

  • JSON-based rule definitions
  • CEL (Common Expression Language) for conditions
  • deterministic evaluation
  • traceable outputs

It’s built to be:

  • safe for LLM-generated rules
  • easy to integrate (Java + Python)
  • suitable for audit-heavy environments

The goal is to decouple business logic from code while keeping it safe and explainable.

Example Rule

Here’s what a rule looks like:

{
  "id": "CREDIT-001",
  "description": "Minimum credit score requirement",
  "condition": "customer.creditScore >= 680",
  "onSuccess": { "action": "PASS" },
  "onFailure": { "action": "DECLINE" }
}
Enter fullscreen mode Exit fullscreen mode

Simple, readable, and structured.
The condition is written in CEL, which is:

  • safe (non-Turing complete)
  • fast
  • easy to sandbox

Why Traceability Matters

In many systems, it’s not enough to know what happened.

You need to know:

  • what inputs were used
  • what condition was evaluated
  • whether it matched
  • what action was taken
  • how long it took

This is critical for:

  • debugging
  • audits
  • compliance reviews

Rule Sets: Real Policies, Not Just Rules

Real-world policies aren’t a single condition. They are a sequence of checks.

AeroRule supports RuleSets, for example:

{
  "executionStrategy": "GATED",
  "rules": [
    { "condition": "customer.creditScore >= 680" },
    { "condition": "customer.annualIncome >= 40000" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

With strategies like:
ALL → evaluate everything
GATED → stop on first failure

This lets you model real decision flows like loan approvals.

Where LLMs Fit In

LLMs are still incredibly useful — just in a different role.

Instead of evaluating decisions, they:

👉 convert natural language → structured rules

AeroRule includes a demo app that:

  • takes a policy document
  • generates rules using an LLM
  • links rules back to source text
  • allows interactive evaluation

This creates a pipeline like:

Human Policy → LLM → JSON Rules → Deterministic Engine → Traceable Decision

If you’re interested, check it out:
👉 https://github.com/kanwaloswal/AeroRule

Feedback, ideas, and contributions are welcome.

Top comments (0)