DEV Community

Cover image for 🛡️ DecisionShield: An AI That Audits Your Decision Before You Make It
Praveen Raj Thulasi S
Praveen Raj Thulasi S

Posted on Edited on

🛡️ DecisionShield: An AI That Audits Your Decision Before You Make It

#ai #gemma #opensource #hackathon

Don't ask AI what to choose. Ask AI if you're ready to choose.

We use AI every day to make decisions.

Which technology should we use?

Should we migrate our database?

Which cloud platform should we choose?

Which vendor is better?

Should we build or buy?

The problem is that AI assistants are often very good at giving us an answer — even when the information behind the decision is incomplete.

That led us to a different question:

What if AI didn't immediately tell us what to choose, but first checked whether we actually had enough evidence to make the decision?

That's the idea behind DecisionShield.

🔗 Project Links

Resource Link
💻 GitHub Repository https://github.com/Praveen-Raj-Thulasi/DecisionAI
🎥 Demo Video https://drive.google.com/file/d/1WADhxrbvxrPhcrXLVeSE0215SyG1gwe5/view?usp=sharing

🤔 The Problem

Imagine asking an AI:

"Should our startup migrate from PostgreSQL to MongoDB?"

A conventional AI assistant might respond:

"MongoDB could be a better choice because it provides flexibility and horizontal scalability."

Sounds convincing.

But what if we don't know:

  • What our query patterns look like?
  • How many transactions require strong consistency?
  • Whether complex joins are important?
  • What the migration downtime tolerance is?
  • What the three-year cost looks like?
  • Whether our team has the required expertise?

The AI may have given us a reasonable answer.

But we still weren't ready to make the decision.

This is the problem DecisionShield tries to solve.

💡 The Idea

DecisionShield is an AI-powered Decision Readiness & Risk Audit Platform.

Instead of immediately recommending an option, DecisionShield analyzes the decision itself.

It separates the information into:

  • ✅ Facts
  • ⚠️ Assumptions
  • ❓ Unknowns
  • 📌 Claims
  • 🚨 Risks
  • 🔍 Missing Evidence
  • 🧪 Stress-Test Scenarios

Then a deterministic decision engine calculates a Decision Readiness Score.

The result isn't simply:

"Choose Option A."

Instead, it might say:

Decision Readiness: 64/100 — NOT READY

Three critical pieces of evidence are missing before this decision should be finalized.

That's a very different approach to AI-assisted decision making.

🔄 How DecisionShield Works

The complete workflow is:

                USER DECISION
                     │
                     ▼
             Decision Context
                     │
                     ▼
              ┌─────────────┐
              │   Gemma 4   │
              │  Reasoning  │
              └──────┬──────┘
                     │
                     ▼
          Structured AI Analysis
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
      Facts     Assumptions      Risks
        │            │            │
        └────────────┼────────────┘
                     │
                     ▼
             Missing Evidence
                     │
                     ▼
          Deterministic Scoring
                     │
                     ▼
          Decision Readiness Score
                     │
                     ▼
             Stress Testing
                     │
                     ▼
           Decision Stability
                     │
                     ▼
             Decision Audit
Enter fullscreen mode Exit fullscreen mode

🧠 Why Not Just Build Another AI Chatbot?

This was one of the most important design decisions behind the project.

A traditional AI assistant looks like:

User Question
      ↓
     LLM
      ↓
Text Recommendation
Enter fullscreen mode Exit fullscreen mode

DecisionShield looks like:

Decision
   ↓
Evidence Extraction
   ↓
Classification
   ↓
Risk Analysis
   ↓
Missing Evidence
   ↓
Deterministic Scoring
   ↓
Stress Testing
   ↓
Decision Audit
Enter fullscreen mode Exit fullscreen mode

The goal isn't to replace the human decision maker.

The goal is to improve the quality of the reasoning and evidence available to the human decision maker.

🧩 Core Features

1. Decision Workspace

The user starts by describing the decision.

They can provide:

  • Decision title
  • Decision description
  • Options
  • Context
  • Requirements
  • Constraints
  • Existing evidence

For example:

Decision:
Should our startup migrate from PostgreSQL to MongoDB?

Options:
• Stay with PostgreSQL
• Migrate to MongoDB

Context:
Our startup currently has approximately 2 TB of data
and expects significant growth over the next three years.

Constraints:
• Small engineering team
• Limited migration budget
Enter fullscreen mode Exit fullscreen mode

2. AI Evidence Analysis

Gemma analyzes the decision context and separates information into different categories.

Facts

Information directly supported by the provided context.

✓ Current database is PostgreSQL
✓ Current data volume is approximately 2 TB
✓ Engineering team is small
Enter fullscreen mode Exit fullscreen mode

Assumptions

Statements that may be true but aren't sufficiently supported.

⚠ MongoDB will automatically solve scalability problems.
⚠ Migration will reduce operational complexity.
Enter fullscreen mode Exit fullscreen mode

Unknowns

Important information that wasn't provided.

? Query workload patterns
? Transaction requirements
? Data consistency requirements
? Peak traffic
Enter fullscreen mode Exit fullscreen mode

Claims

Statements that require verification.

? "MongoDB will be cheaper over three years."
Enter fullscreen mode Exit fullscreen mode

This classification becomes the foundation of the rest of the analysis.

3. Decision Readiness Score

DecisionShield calculates a score from 0–100.

The score considers multiple dimensions such as:

Evidence Quality
Requirement Coverage
Evidence Completeness
Risk Coverage
Assumption Load
Unknown Information
Decision Stability
Enter fullscreen mode Exit fullscreen mode

The important architectural decision here is:

Gemma does not generate the final score.

Instead:

Gemma
  ↓
Semantic Analysis
  ↓
Structured JSON
  ↓
Pydantic Validation
  ↓
Python Decision Engine
  ↓
Readiness Score
Enter fullscreen mode Exit fullscreen mode

This makes the scoring process deterministic and reproducible.

Example:

                 DECISION READINESS

                       64
                     /100

        ┌───────────────────────────┐
        │        NOT READY          │
        └───────────────────────────┘

Evidence Quality       72%
Requirements           80%
Risk Coverage          55%
Evidence Completeness  61%
Decision Stability     68%
Enter fullscreen mode Exit fullscreen mode

4. Missing Evidence Detection

One of the most important features is asking:

"What information are we missing that could materially change this decision?"

For the PostgreSQL vs MongoDB example, DecisionShield might identify:

🔴 CRITICAL

1. Query workload analysis
2. Transaction requirements
3. Data consistency requirements

🟠 HIGH

4. Migration downtime tolerance
5. Three-year infrastructure cost
Enter fullscreen mode Exit fullscreen mode

But it doesn't stop there.

For each missing piece of information, the system generates a verification action.

For example:

Missing Evidence:
Expected peak traffic

Verification Action:
Estimate current peak requests per second and project
expected traffic over the next three years.
Enter fullscreen mode Exit fullscreen mode

This converts AI analysis into an actionable workflow.

5. Risk Analysis

DecisionShield identifies risks associated with the decision.

Each risk can include:

  • Severity
  • Likelihood
  • Impact
  • Confidence
  • Supporting evidence

Example:

🚨 HIGH RISK

Complex relational workloads

MongoDB may introduce additional complexity
if the application's workload depends heavily
on complex relational queries and joins.
Enter fullscreen mode Exit fullscreen mode

The backend then incorporates risk information into the deterministic readiness calculation.

6. Decision Stress Testing

A decision shouldn't only be evaluated under today's conditions.

So DecisionShield allows users to ask:

"What happens if the assumptions change?"

Example scenarios:

Scenario 1

Traffic increases 10×.

Scenario 2

The infrastructure budget decreases by 50%.

Scenario 3

Strict data residency becomes mandatory.

Scenario 4

The engineering team loses its existing database expertise.

Scenario 5

Data volume doubles.

The system analyzes how these scenarios affect the decision.

📉 Decision Stability

Suppose the current readiness score is:

72 / 100
Enter fullscreen mode Exit fullscreen mode

Stress testing might produce:

Current Decision       72
10× Traffic            68
50% Budget             65
New Compliance         43
Team Expertise Loss    61
Enter fullscreen mode Exit fullscreen mode

The system can then identify:

⚠️ Decision Stability: LOW

because one plausible change significantly alters the decision's readiness.

This is important because a decision can have a high current score but still be fragile.

🧠 Why Gemma 4?

Gemma is not being used as a generic chatbot inside DecisionShield.

It is used as the semantic reasoning layer.

Gemma performs tasks such as:

Unstructured Decision Context
            ↓
      Gemma 4 E4B
            ↓
     Semantic Analysis
            ↓
 ┌──────────┼──────────┐
 ▼          ▼          ▼
Facts   Assumptions   Risks
 ▼          ▼          ▼
Unknowns  Claims  Missing Evidence
Enter fullscreen mode Exit fullscreen mode

The backend then converts this structured analysis into measurable decision metrics.

Separation of responsibilities

Component Responsibility
Gemma 4 Semantic reasoning
Pydantic AI output validation
Python Decision Engine Deterministic scoring
FastAPI Backend orchestration
React Visualization
User Final decision

This separation was intentional.

⚙️ Technical Architecture 

                         ┌─────────────────────┐
                         │   React + Vite UI   │
                         │                     │
                         │ Decision Workspace  │
                         │ Dashboard           │
                         │ Evidence Explorer   │
                         │ Stress Testing      │
                         │ Audit Report        │
                         └──────────┬──────────┘
                                    │
                                REST API
                                    │
                                    ▼
                         ┌─────────────────────┐
                         │       FastAPI       │
                         │      Backend        │
                         └──────────┬──────────┘
                                    │
                    ┌───────────────┴──────────────┐
                    │                              │
                    ▼                              ▼
          ┌──────────────────┐           ┌──────────────────┐
          │   Gemma 4 E4B    │           │ Decision Engine  │
          │                  │           │                  │
          │ AI Reasoning     │           │ Deterministic    │
          │ Classification   │           │ Scoring          │
          │ Risk Discovery   │           │ Stability        │
          └────────┬─────────┘           └────────┬─────────┘
                   │                              │
                   └──────────────┬───────────────┘
                                  ▼
                         ┌──────────────────┐
                         │ Decision Analysis│
                         │   Structured     │
                         │     Result       │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │ Decision Audit   │
                         │     Report       │
                         └──────────────────┘
Enter fullscreen mode Exit fullscreen mode

🛠️ Tech Stack

Frontend

  • React
  • Vite
  • Tailwind CSS
  • Recharts
  • Framer Motion

Backend

  • Python
  • FastAPI
  • Pydantic
  • Uvicorn

AI

  • Gemma 4 E4B
  • 4-bit quantization
  • Local inference

Architecture

  • REST API
  • Structured JSON
  • Deterministic scoring engine
  • In-memory decision store for the MVP

💻 Running Locally

Prerequisites

You will need:

  • Python 3.x
  • Node.js
  • npm
  • NVIDIA GPU recommended for local Gemma inference
  • Approximately 16 GB RAM
  • Approximately 6 GB VRAM for the target E4B Q4 setup

Clone the Repository 

git clone https://github.com/Praveen-Raj-Thulasi/DecisionAI/
cd DecisionShield
Enter fullscreen mode Exit fullscreen mode

Backend 

cd backend

python -m venv .venv
Enter fullscreen mode Exit fullscreen mode

Windows 

.venv\Scripts\activate
Enter fullscreen mode Exit fullscreen mode

Linux/macOS 

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Install dependencies:

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Create environment configuration:

cp .env.example .env
Enter fullscreen mode Exit fullscreen mode

Configure the Gemma model path and other required settings.

Start the backend:

uvicorn app.main:app --reload
Enter fullscreen mode Exit fullscreen mode

🎨 Frontend 

cd frontend

npm install

npm run dev
Enter fullscreen mode Exit fullscreen mode

🧪 Demo Scenario

For the first demonstration, we use:

Should our startup migrate from PostgreSQL to MongoDB?

The system begins with a simple decision.

It then progressively reveals why the decision isn't as straightforward as it initially appears.

Decision
   ↓
Facts
   ↓
Assumptions
   ↓
Unknowns
   ↓
Risks
   ↓
Missing Evidence
   ↓
Readiness
   ↓
Stress Test
   ↓
Decision Stability
Enter fullscreen mode Exit fullscreen mode

This demonstrates the central philosophy of DecisionShield:

A confident decision isn't necessarily a well-supported decision.

🔬 Example Analysis

Input 

Our startup has 2 TB of PostgreSQL data.

We expect rapid growth and believe MongoDB
will make scaling easier.

Our team is small and we want to reduce
operational complexity.
Enter fullscreen mode Exit fullscreen mode

DecisionShield might identify:

Facts 

✓ PostgreSQL is currently used
✓ Data volume is approximately 2 TB
✓ Team size is limited
✓ Growth is expected
Enter fullscreen mode Exit fullscreen mode

Assumptions 

⚠ MongoDB will automatically improve scalability
⚠ MongoDB will reduce operational complexity
Enter fullscreen mode Exit fullscreen mode

Unknowns 

? Query patterns
? Transaction requirements
? Consistency requirements
? Peak workload
? Migration downtime
Enter fullscreen mode Exit fullscreen mode

Risks 

🚨 Complex relational queries
🚨 Migration downtime
🚨 Learning curve
Enter fullscreen mode Exit fullscreen mode

Missing Evidence 

🔍 Query workload analysis
🔍 Transaction requirements
🔍 Three-year cost comparison
Enter fullscreen mode Exit fullscreen mode

Result 

Decision Readiness

64 / 100

⚠ NOT READY
Enter fullscreen mode Exit fullscreen mode

Instead of blindly recommending a database, DecisionShield tells the user what they should investigate first.

🏗️ Project Structure 

DecisionShield/
│
├── frontend/
│   ├── components/
│   ├── pages/
│   ├── services/
│   └── ...
│
├── backend/
│   ├── app/
│   │   ├── api/
│   │   ├── ai/
│   │   ├── engine/
│   │   ├── schemas/
│   │   ├── services/
│   │   └── ...
│   │
│   ├── tests/
│   └── requirements.txt
│
├── docs/
│
├── README.md
└── ...
Enter fullscreen mode Exit fullscreen mode

🔐 Reliability by Design

A major design principle of DecisionShield is separating AI reasoning from numerical decision logic.

We don't want:

Gemma:
"I think the decision readiness is 87."
Enter fullscreen mode Exit fullscreen mode

Instead:

Gemma
 ↓
Facts / Assumptions / Risks / Unknowns
 ↓
Validated JSON
 ↓
Python Decision Engine
 ↓
Readiness Score
Enter fullscreen mode Exit fullscreen mode

This provides a more reproducible scoring process and makes the system easier to inspect and debug.

🧪 Demo Mode

Local AI models can sometimes introduce practical problems during a hackathon:

  • model loading
  • VRAM limitations
  • CUDA issues
  • inference failures
  • environment configuration

To keep the application demonstrable, DecisionShield includes a fallback/demo provider.

              AI Analysis
                   │
          ┌────────┴────────┐
          │                 │
       Gemma 4          Demo Provider
          │                 │
          └────────┬────────┘
                   ▼
            Same JSON Schema
                   ▼
          Same Decision Engine
Enter fullscreen mode Exit fullscreen mode

This means the rest of the application doesn't depend on whether the local model happens to load successfully during a demonstration.

The application clearly identifies whether the result came from Gemma or demo mode.

🌍 Why This Could Matter

Decision making isn't only about having more information.

It's about knowing:

  • which information is reliable
  • which assumptions are untested
  • which information is missing
  • which risks matter
  • which scenarios could invalidate the decision

DecisionShield tries to turn those questions into a repeatable workflow.

The goal is not:

AI makes the decision.

The goal is:

AI helps humans understand whether their decision is ready to be made.

🚀 Future Roadmap

DecisionShield is currently an MVP.

Possible future improvements include:

📄 Evidence Ingestion

Upload:

  • PDFs
  • documents
  • spreadsheets
  • reports

and automatically extract evidence.

🔗 Evidence Provenance

Track where every fact came from.

Fact
 ↓
Source
 ↓
Document
 ↓
Page / Section
Enter fullscreen mode Exit fullscreen mode

🧠 Advanced Decision Graphs

Represent relationships between:

Evidence
   ↓
Assumption
   ↓
Risk
   ↓
Requirement
   ↓
Decision
Enter fullscreen mode Exit fullscreen mode

👥 Collaborative Decision Rooms

Allow teams to collaboratively audit a decision.

📚 Decision History

Track how decisions change over time.

🌐 External Knowledge

Use trusted external sources to verify missing evidence.

🧪 Advanced Simulations

Introduce more sophisticated scenario and sensitivity analysis.

🏢 Domain-Specific Audits

Create specialized templates for:

  • software architecture
  • business decisions
  • procurement
  • hiring
  • cloud infrastructure
  • product strategy

⚠️ Limitations

DecisionShield is a decision-support system.

It does not guarantee that a decision is correct.

AI-generated classifications may be imperfect.

Missing evidence suggestions may not be exhaustive.

Stress tests are scenarios, not predictions.

The Decision Readiness Score is an analytical indicator, not an objective measure of truth.

For high-stakes decisions, users should consult qualified professionals and verify important evidence independently.

🌱 Open Source

DecisionShield is being developed as an open-source project.

The goal is to make the architecture transparent and allow developers to experiment with:

  • local AI
  • Gemma
  • structured reasoning
  • deterministic scoring
  • decision intelligence
  • human-in-the-loop systems

👥 Team

Team Name

*[Wakie Wakie]

Members

  • [Naveensri V] — [Backend]
  • [Prithivi Raj T J] — [Database]
  • [Praveen Raj Thulasi S] — [AI]
  • [Gopika B] — [Frontend]

Responsibilities

Member Contribution
Praveen Raj Thulasi S AI / Gemma Integration
Naveensri V Backend / FastAPI
Gopika B Frontend / UI
Prithivi Raj T J Architecture / Testing

🏆 Hackathon

Built for:

Hacktoberfest Hack Day Coimbatore x (INIT Club & IDEA Club)

Additional Challenge:

Gemme 4

🎥 Demo

Watch the complete DecisionShield demonstration:

https://youtu.be/Y3VlhbDvYRg

The demo covers:

Create Decision
      ↓
AI Analysis
      ↓
Evidence Classification
      ↓
Readiness Score
      ↓
Missing Evidence
      ↓
Stress Testing
      ↓
Decision Stability
      ↓
Final Audit
Enter fullscreen mode Exit fullscreen mode

🔗 Links

GitHub

https://github.com/Praveen-Raj-Thulasi/DecisionAI

Demo Video

https://drive.google.com/file/d/1WADhxrbvxrPhcrXLVeSE0215SyG1gwe5/view?usp=sharing

Presentation

https://docs.google.com/presentation/d/1s8DROAmsQw_Jsu0aPcsOp6fnWyoJju-1/edit?usp=sharing&ouid=110537926822090636746&rtpof=true&sd=true

Team

Wakie Wakie

🙌 Final Thoughts

The most interesting part of building DecisionShield wasn't getting an AI model to generate a recommendation.

It was asking a different question:

What if the most useful thing AI can tell us isn't what to choose — but what we're missing before we choose?

That idea became DecisionShield.

Instead of:

"What should I choose?"
Enter fullscreen mode Exit fullscreen mode

we ask:

"Do I have enough evidence to choose?"
Enter fullscreen mode Exit fullscreen mode

And if the answer is no, the system tells us why.

Built with ❤️ using open-source AI and Gemma 4.

Top comments (2)

Collapse
 
ai_adam profile image
ai_adam •

Cái góc nhìn "audit quyết định" thay vì "đưa ra quyết định" này rất thực tế. Đã bao nhiêu lần thấy team (mình inclusive) ném context vào LLM xin output ngay, mà quên rằng AI không chịu rủi ro — người ký tên vào PR/merge mới chịu.

Một pattern mình hay áp dụng: trước khi hỏi "nên chọn A hay B?", mình ép model liệt kê assumptions đang ngầm hiểu, failure modes của từng option, và reversibility của quyết định. Nếu model không chỉ ra được ít nhất 2-3 rủi ro cụ thể (không phải generic), thì context còn thiếu hoặc vấn đề chưa đủ chín để quyết định.

Cái khái niệm "readiness checklist" trong bài nhắc mình nhớ những pre-mortem meeting — chỉ khác là làm async với AI thay vì kéo cả team vào phòng. Hiệu quả hơn hẳn so với việc hỏi "what do you think?" rồi nhận một câu trả lời an toàn, middle-ground.

Curious: bạn có test trường hợp AI over-flag rủi ro (false positive) không? Đó là nỗi lo mình thấy khi đưa gate này vào CI/CD decision flow — dev sẽ bypass nếu noise quá lớn (site: labagent .tech)

Collapse
 
praveen007 profile image
Praveen Raj Thulasi S •

That's a very valid concern. A decision gate that flags every possible risk can quickly become noise, and developers may eventually stop trusting it.

My approach with DecisionShield is to separate the LLM's semantic analysis from the deterministic readiness scoring, rather than letting the model make the final decision on its own.

I think the next step is to evaluate false positives using a set of known decision scenarios and measure how often the system flags risks that aren't actually relevant. For a CI/CD integration, I'd also distinguish between advisory warnings and blocking conditions, reserving hard blocks for genuinely critical issues with clear evidence.

I haven't validated the false-positive rate in a real CI/CD pipeline yet, so I wouldn't claim the problem is solved. But I agree that measuring signal-to-noise ratio and tracking developer overrides would be essential before making this a mandatory gate.

Curious how you'd approach the threshold: would you prefer a configurable risk tolerance per repository, or a fixed threshold with severity-based overrides?

Some comments may only be visible to logged-in visitors. Sign in to view all comments.