DEV Community

Cover image for Sub-35ms Typed AI Decisions Without Token Generation Or Hallucinations
Dmytro Bilukha
Dmytro Bilukha

Posted on

Sub-35ms Typed AI Decisions Without Token Generation Or Hallucinations

#ai #python #machinelearning #performance

Demo Animation

Modern AI pipelines often burn compute using 8B+ parameter generative models just to answer questions like:

  • "Is this support ticket urgent?"
  • "Does this comment violate moderation policies?"
  • "Should this request route to the billing or tech support department?"

Autoregressive generation for classification introduces high latency (waiting for token-by-token decoding), unpredictable JSON formatting errors, and unnecessary costs.

laya-cli is a high-performance CLI and daemon wrapper around Laya — providing typed decisions (choice, score, noul) in a single forward pass without autoregressive token generation.

Key Capabilities

  1. Deterministic Typed Outputs in One Pass:
    • choice: Discrete labels with calibrated probabilities.
    • score: Ordinal expected values (e.g. priority 1 to 5).
    • noul: Probability that an assertion is true.
  2. Speed: ~33ms for single predictions, ~7ms per item when batched.
  3. Resident Daemon: Run laya-cli serve to keep weights resident in memory, eliminating cold starts for subsequent CLI or sub-process calls.
  4. Unix Pipeline Compatibility: Native streaming support for JSON and JSONL feeds.

Quick Installation 

uv tool install laya-cli
# or: pip install laya-cli
Enter fullscreen mode Exit fullscreen mode

Usage Examples

Interactive Categorization 

laya-cli predict "User reports database timeout after running migrations" --preset triage
Enter fullscreen mode Exit fullscreen mode

Streaming Pipeline 

cat incoming_events.jsonl | laya-cli classify --preset moderation --format jsonl >> audited.jsonl
Enter fullscreen mode Exit fullscreen mode

Links & Community

Top comments (0)