DEV Community

Alp Allovi
Alp Allovi

Posted on

Introducing Ayeixa Sentience Eval: Catching Raw JSON Leaks & Robotic Clichés in Agent Dialogue

#ai #testing #opensource #typescript

Introducing Ayeixa Sentience Eval: Catching Raw JSON Leaks & Robotic Clichés in Agent Dialogue

When autonomous agents convert structured internal states and tool execution payloads into natural language dialogue, formatting breakdowns can occur: unparsed JSON delimiters leak into output text, robotic conversational clichés degrade user experience, and assertions deviate from context.

Ayeixa Sentience Eval (@ayeixa/sentience-eval) is a standalone assertion harness for testing, benchmarking, and guarding behavioral cleanliness in autonomous conversational agents.

1. Core Architecture

Sentience Eval is composed of four inspection engines:

  1. RawJsonLeakDetector: Identifies raw JSON structures, unbalanced brace delimiters, and unrendered system delimiters matching configured regex patterns.
  2. HallucinationDetector: Compares response claims against ground-truth context to flag ungrounded assertions.
  3. BehavioralCleanlinessHarness: Enforces conversational naturalness rules by detecting forbidden robotic clichés, prompt leaks, and meta-tokens.
  4. SentienceBenchmarkRunner: Executes evaluation batches and computes a weighted cleanliness index (0–100).

2. Implemented Capabilities & Test Verification

Verified with hermetic unit tests:

  • Raw JSON Detection: Detects tested unescaped JSON state and delimiter leaks (tests/json-leak.test.ts).
  • Cleanliness Suite: Flags tested cliché markers and delimiter imbalances (tests/cleanliness.test.ts).
  • Benchmark Runner: Weighted penalty scoring calculation (tests/benchmark.test.ts).

Verification: 6/6 hermetic unit tests passing (0 failures).

3. Local Quick Start 

git clone https://github.com/alpallovy/ayeixa-sentience-eval.git
cd ayeixa-sentience-eval
npm ci
npm run build
npm test
Enter fullscreen mode Exit fullscreen mode

Usage Example 

import { SentienceBenchmarkRunner } from './src';

const runner = new SentienceBenchmarkRunner();
const evaluation = runner.evaluate({
  output: "Here is your task: {\"internalState\": true, \"status\": \"RUNNING\"}",
  groundTruthContext: ["The user requested task status."]
});

console.log("Cleanliness Score (0-100):", evaluation.cleanlinessScore);
console.log("Violations Detected:", evaluation.violations);
Enter fullscreen mode Exit fullscreen mode

4. Limitations & Contributing

  • Pre-release v0.1.0-alpha. Focuses on structural and lexical patterns covered by its rule set.
  • Public npm publication is pending.
  • Contributions welcome! Check out good first issue tags on GitHub.

License: MIT

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

Raw JSON leaks are a useful eval target because they are easy to recognize and hard to excuse. I would treat them as contract failures, not style failures: the assistant left the user-facing protocol and exposed an internal shape. That makes the metric much more actionable.