When something goes wrong with your LLM
in production, you shouldn't have to
check a dashboard to find out.
Today I shipped Slack webhook support
to Ajah — two types of alerts, both
fire-and-forget, zero latency added.
Cost spike alerts:
When a feature's daily LLM spend exceeds
the configured threshold, Ajah fires a
formatted Slack message:
🚨 Cost Alert — Ajah
Feature: chat
Cost today: $4.23
Threshold: $2.00
Model: gpt-4o
Deduplication is built in — one alert
per feature per day maximum, using a
Redis SetNX with 24h TTL.
Risk alerts:
When a response is flagged — hallucination,
RAG contradiction, claim density, or
overconfidence — Ajah fires a Slack alert
with the risk level, scores, and exact
reason strings.
⚠️ Risk Alert — Ajah
Feature: support-bot
Risk Level: high
Hallucination Risk: 0.78
Grounding Score: 0.31
Reasons: Response contradicts source document
Both use the webhook_url configured per
feature in the Settings page. One URL,
both alert types. Configure in 30 seconds.
Self-hosted. MIT license.
→ github.com/VigneshReddy-afk/ajah
→ useajah.com
Top comments (0)