DEV Community

Cover image for Building an AI Assistant That Actually Understands Company Policy
Shushant Rishav
Shushant Rishav

Posted on • Edited on

Building an AI Assistant That Actually Understands Company Policy

#ai #python #productivity #rag

Managing customer complaints in footwear retail is deceptively complex. Orders, warranty windows, product defects, inventory constraints, and outlet-specific rules intersect, creating a decision space where speed, accuracy, and consistency are all hard requirements.

STRIDE Assistant is an AI-driven complaint resolution system designed to automate this process while ensuring strict policy compliance, accountability, and human oversight.

This document explains why STRIDE Assistant exists, how it works, and how AI can be combined with deterministic policy logic to deliver reliable and auditable decisions in production systems.

Why STRIDE Assistant Exists

A customer messages support:

“My shoe tore 4 days after purchase.”

Without automation, a staff member must manually:

  • Verify the order and purchase date
  • Check warranty eligibility
  • Interpret company policy
  • Confirm inventory availability
  • Decide between return, replacement, repair, or rejection

Multiply this by hundreds of tickets per day, and the process becomes slow, inconsistent, and error-prone.

STRIDE Assistant addresses this by combining:

  • AI-driven semantic analysis to understand customer complaints
  • Deterministic policy enforcement grounded in business rules
  • Human-in-the-loop escalation for ambiguous or high-risk cases
  • Immutable audit logging for compliance and traceability

The result is fast, accurate, and accountable complaint handling—without over-reliance on generative AI.

Architecture Overview 

Customer Message
        |
Session Verification (Order + Phone + JWT)
        |
Semantic Analysis (Intent Detection)
        |
Clarification Step (if required)
        |
Policy Retrieval
        |
Decision Engine (Policy + Order + Inventory)
        |
+-----------------------------+
| FINAL DECISION / TICKET     |
+-----------------------------+
        |
     OR Manual Review → Ticket Created
Enter fullscreen mode Exit fullscreen mode

Tools and Models Used

Component Tool / Model Purpose
Semantic Analysis & RAG LLaMA 3 (8B) via Ollama Intent detection and generation of professional, policy-safe customer responses
Embeddings SentenceTransformer (all-MiniLM-L6-v2) Converts complaints and policy chunks into vectors for semantic retrieval
API Layer FastAPI Secure endpoints for customers, staff, and admins (JWT-based)
Database PostgreSQL Stores orders, tickets, staff actions, and audit logs
Cache Redis Low-latency access to prefetched order data
Logging Structured Python logging Captures system events, AI decisions, and staff actions

Why this setup?

  • LLaMA 3 (8B) hosted via Ollama enables fully local, private inference with strong instruction-following capability.
  • Embeddings enable deterministic policy retrieval without relying on probabilistic generation.
  • The system explicitly separates language understanding from decision authority.

Dual-Logic Workflow

STRIDE Assistant deliberately separates reasoning from enforcement using two complementary logic layers:

1. Policy Retriever (RAG Layer)

  • Interprets the complaint semantically
  • Retrieves relevant policy rules
  • Produces an initial suggested action

2. Decision Engine (Deterministic Layer)

  • Validates the suggestion against:

    • Order history
    • Warranty duration
    • Product category
    • Inventory availability
    • Past customer interactions

  • Produces an authoritative outcome that may:

    • Approve
    • Suggest repair
    • Require inspection
    • Reject the complaint

Complaints are rejected when:

  • Warranty is expired
  • Damage indicates misuse
  • Repeated requests violate policy limits
  • Required order data is missing

RAG Orchestration

The RAG pipeline coordinates both logic layers:

  • Aggregates signals across multiple turns
  • Requests clarification when signals conflict
  • Escalates ambiguous cases to inspection
  • Produces a final ticket with explicit reasoning notes

This prevents hallucinated decisions and ensures traceability.

Code Snippet: Combining Logic Outputs 

policy_signal = retriever.get_policy_signal(user_text, order)
decision_signal = engine.make_decision(
    user_text,
    order,
    inventory_available
)

signals = [policy_signal, decision_signal]
final_ticket = resolve_final_ticket(signals)
Enter fullscreen mode Exit fullscreen mode

Every ticket is generated only after policy compliance, historical context, and system constraints are validated.

Authentication & Access Control

STRIDE Assistant uses JWT-based authentication across all actors:

User Credentials Access
Customer order_id + phone Submit complaints, track status
Staff user_id + password Act on tickets, add remarks
Admin role-based credentials Audit logs, system oversight

Benefits

  • Stateless and scalable
  • Role-based enforcement
  • Every action is attributable and logged

Staff & Admin Logging

  • Staff actions (approve, reject, escalate, close, note delays) are recorded securely
  • Logs are immutable and stored in staff_action_log
  • Admins have read-only visibility 
log_staff_action(staff.id, ticket_id, action, remarks)
Enter fullscreen mode Exit fullscreen mode

This ensures full accountability for both AI and human decisions.

Example End-to-End Workflow

  1. Customer: “My shoe tore after 4 days.” → Policy retriever suggests REPLACEMENT
  2. Decision engine validates eligibility
  3. Clarification requested if needed
  4. Signals are aggregated
  5. Final ticket is resolved as one of: RETURN, REPLACEMENT, REPAIR, PAID_REPAIR, INSPECTION, or REJECT
  6. Inventory constraints are checked
  7. Ambiguous cases escalate to inspection
  8. Every ticket includes a system-generated reasoning note
  9. Staff actions are logged immutably

Production Considerations

  • Thread-safe LLM access via Ollama
  • Transaction-safe ticket creation
  • Redis-based order prefetching
  • Role-based access control
  • Structured logging for observability

Future Improvements

  • Multi-model ensembles for higher signal accuracy
  • Analytics dashboards for staff performance
  • Versioned policy ingestion
  • Predictive ticket prioritization
  • GPU-backed inference for scale

Conclusion

STRIDE Assistant demonstrates how AI can augment human decision-making without replacing it.

By combining:

  • Semantic understanding via LLaMA 3
  • Deterministic policy enforcement
  • Structured escalation
  • Immutable audit logs

STRIDE delivers explainable, compliant, and production-safe complaint resolution, a blueprint for enterprise AI systems built to operate under real-world constraints.

Author: Shushant Rishav
Repository: STRIDE_Assistant_DEMO

Top comments (0)