If you're running LangChain or LlamaIndex
agents in production, you're missing
critical signals.
You know what your agent said.
You don't know what it cost per step.
You don't know when it hallucinated.
You don't know when it reversed a position
under pressure across a long conversation.
Today I shipped two integrations for Ajah
that fix this — a LangChain callback handler
and a LlamaIndex observer. Both are single-file
drops into your existing project.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SETUP (2 MINUTES)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Clone and start Ajah:
git clone https://github.com/VigneshReddy-afk/ajah
cd ajah
cp .env.example .env
docker-compose up -d
Dashboard live at localhost:3000.
Gateway at localhost:8080.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
LANGCHAIN INTEGRATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Copy examples/langchain/ajah_callback.py
into your project. Then:
pip install langchain-openai langchain-core
from ajah_callback import AjahCallbackHandler
handler = AjahCallbackHandler(
gateway_url="http://localhost:8080",
feature_name="my-agent",
user_id="user-123",
)
llm = ChatOpenAI(
base_url="http://localhost:8080/v1",
api_key="your-groq-key",
model="llama-3.3-70b-versatile",
callbacks=[handler],
model_kwargs={
"extra_headers":
handler.get_extra_headers("step-1")
},
)
What you get automatically for every call:
Cost attribution — how much each agent
step costs in USD, tracked per feature
and model in real time.
Hallucination risk — every response
scored async using local ML models.
Zero latency added to your agent.
Claim density detection — flags responses
that make many specific claims on
low-context prompts. Catches a class
of hallucination that embedding similarity
misses.
Narrative drift detection — compares
claims across session turns. Flags when
your agent reverses a position under
pressure. Critical for long-running agents.
RAG verification — if you pass source
documents, every response is verified
against them. Contradictions flagged
before they reach users.
Full session trace — visual step tree
in the dashboard showing every turn,
cost, latency, and quality score.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
LLAMAINDEX INTEGRATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Copy examples/llamaindex/ajah_observer.py
into your project. Then:
pip install llama-index llama-index-llms-openai
from ajah_observer import AjahObserver
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
observer = AjahObserver(
gateway_url="http://localhost:8080",
feature_name="rag-pipeline",
user_id="user-123",
)
observer.register()
Settings.llm = OpenAI(
api_base="http://localhost:8080/v1",
api_key="your-groq-key",
model="llama-3.3-70b-versatile",
additional_kwargs={
"extra_headers":
observer.get_extra_headers(
"step-1-query")
},
)
Every RAG query now gets full observability
— grounding scores, contradiction detection,
cost tracking, and session tracing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT YOU SEE IN THE DASHBOARD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
After running your agent:
Sessions page — visual step tree showing
every LLM call in your agent run, grouped
by session ID with per-step cost and latency.
Warnings page — any hallucination flags,
RAG contradictions, claim density alerts,
or narrative drift detected across your
session turns.
Traces page — live feed of every call
with quality scores, PII detection,
and RAG verdicts.
Overview — cost by feature and model,
quality trend over time.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Both integrations are in the repo under
examples/langchain/ and examples/llamaindex/.
Self-hosted. No data leaves your server.
MIT license. Free forever.
→ github.com/VigneshReddy-afk/ajah
→ useajah.com
Top comments (0)