You know the drill. QA opens a Word doc, types "the bot should ask for the order number if it's missing," and tests the agent by hand. Meanwhile, Dev builds against an ever-mutating PR description. And PO has nothing to sign off on that isn't prose or code.
We solved this for APIs 20 years ago with OpenAPI. A shared contract. Machine-readable.Tool-independent. Human-auditable.AI agents have no equivalent. Until now.
What ABS is — in 30 seconds
Agent Behavior Specification (ABS) is a YAML format for describing observable agent behavior — messages, tool calls, hand-offs, UI interactions — independent of your LLM provider, framework, or tool protocol.
One file. Three stakeholders. Two modes: descriptive and executable.
session: Customer checks order status
behaviors:
- actor: user
action: says
content: "Where is my order #8291?"
- actor: assistant
action: calls
target: Orders API
- actor: assistant
action: informs
content: "Your order is on the way"
evaluations:
- type: contains
value: "on the way"
shell
That's it. Three fields per step — actor, action, content/target. A PO reads it as a behavioral contract. QA runs it as a test. Dev uses it as acceptance criteria.
In this article, you'll build and run your first agent spec — in 10 minutes.
1. Install the CLI
npm install -g abslang
# or: pip install abslang
Verify:
abslang --version
# abs v0.1.0
2. Scaffold a project
abslang init my-agent-tests
cd my-agent-tests
You get:
.
├── abs.config.yaml
├── sessions/
│ └── order-status.abs.yaml # Example session
└── datasets/
└── order-status.jsonl # 3 test cases
**3. Don't want to write YAML? Chat it.**
This is the feature that saves you 80% of the learning curve:
abslang chat
🤖 ABS Assistant — describe the agent behavior you want to test
You: A customer reports a damaged item. The agent should verify the order, process a refund, confirm the amount and reference, and offer further help.
Assistant: I'll draft a refund flow with tool calls, step-level evaluations, and chain checks…
[generates complete .abs.yaml]
Uses OPENAI_API_KEY, ANTHROPIC_API_KEY, or DEEPSEEK_API_KEY — whichever you have set. It knows the full ABS v0.1 spec and generates validated YAML instantly.
4. Write a session by hand
Here's a refund flow across three conversational turns. No tool calls — this works with any agent:
session: Damaged item → refund (multi-stage evaluation)
behaviors:
# ── Turn 1: the agent classifies intent ──
- actor: user
action: says
content: "I received a damaged item, I want my money back. Order #8291."
- actor: assistant
action: clarifies
content: "I understand your order #8291 arrived damaged. I'll help you get a refund."
# ── Turn 2: resolution ──
- actor: user
action: says
content: "Yes please, how long will it take?"
- actor: assistant
action: informs
content: "Refund of €47.50 approved. Reference: R-5512. You'll receive it in 3-5 days."
capture:
refundId: "R-5512"
# ── Turn 3: closing ──
- actor: user
action: says
content: "Great, thanks."
- actor: assistant
action: confirms
content: "You're welcome! Is there anything else I can help with?"
Six behaviors, three turns. Descriptive only so far — no evaluations yet.
5. Add evaluations at every stage
This is where ABS earns its place. Add an evaluations: block to any step:
- actor: assistant
action: clarifies
content: "I understand your order #8291 arrived damaged. I'll help you get a refund."
evaluations:
- type: llm_judge
criteria: |
1. Correctly classifies the intent as a refund request
2. References the order number #8291
3. Acknowledges the damage (not a simple return)
4. Takes ownership of the resolution
# ...
- actor: assistant
action: informs
content: "Refund of €47.50 approved. Reference: R-5512. You'll receive it in 3-5 days."
capture:
refundId: "R-5512"
evaluations:
- type: contains
value: "R-5512"
- type: llm_judge
criteria: |
1. States the exact refund amount (€47.50)
2. Provides the reference number R-5512
3. Sets a clear timeline (3-5 days)
4. Professional and empathetic tone
Now add chain evaluations — properties of the entire trace:
evaluations:
- type: sequence
order:
- { actor: assistant, action: clarifies }
- { actor: assistant, action: informs }
- { actor: assistant, action: confirms }
- type: variable_consistency
variable: refundId
sequence checks that the three stages happen in order. variable_consistency catches a subtle but deadly bug — the agent silently swapping one refund ID for another mid-conversation.
6. Run it
abslang run sessions/refund.abs.yaml --agent http://localhost:8080/chat
Your agent needs one HTTP endpoint:
POST /chat
{ "messages": [{ "role": "user", "content": "..." }] }
That's it. OpenAI-compatible. The Runner plays the user, captures the trace, and runs all evaluations:
┌──────────────────────────────────────────────────────┐
│ ABS — Results │
├──────────────────────────────────────────────────────┤
│ Session: Damaged item → refund │
│ Result: ✅ PASSED │
│ Steps: 6/6 matched · 5/5 evaluations passed │
├────┬──────────────────────────────────┬────────┬─────┤
│ 1 │ user says "damaged item..." │ → │ sent│
│ 2 │ assistant clarifies │ ✅ │match│
│ │ └─ llm_judge: intent check │ ✅ │ pass│
│ 3 │ user says "how long?" │ → │ sent│
│ 4 │ assistant informs │ ✅ │match│
│ │ ├─ contains "R-5512" │ ✅ │ pass│
│ │ └─ llm_judge: quality check │ ✅ │ pass│
│ 5 │ user says "thanks" │ → │ sent│
│ 6 │ assistant confirms │ ✅ │match│
│ │ └─ llm_judge: closing check │ ✅ │ pass│
│ C │ sequence: clarifies→informs→conf │ ✅ │ pass│
│ C │ variable_consistency: refundId │ ✅ │ pass│
└────┴──────────────────────────────────┴────────┴─────┘
7. Scale it: one session, 200 test cases
Replace hardcoded values with {{placeholders}}:
dataset:
id: cases
path: cases.jsonl
behaviors:
- actor: user
action: says
content: "{{cases.userMessage}}"
# ...
Use your dataset file:
{"userMessage": "Item damaged, order #8291. Refund please.", ...}
{"userMessage": "Order #3412 arrived broken. I want my money back.", ...}
{"userMessage": "Wrong item in box #5567. Refund.", ...}
Run it:
abslang run session.abs.yaml --agent $URL --dataset cases.jsonl
Three rows, three runs, one aggregated report. Works with 3 rows or 300.
The anti-lock-in architecture
The industry standard couples agent execution and evaluation in one platform. Deploy to their infra, they run it, they evaluate it. Change platforms → rewrite everything.
ABS separates them:
Industry today: [agent execution + evaluation] in one platform → vendor lock-in
ABS: [agent] in your infra → [trace] → [evaluation] wherever you want
Your agent runs on your infrastructure. The evaluator receives only {type, input, context, response, threshold} and returns {passed, score, reason}. Any provider — Azure, LangSmith, Galileo, a local Ollama instance — implements the same adapter interface in an afternoon.
Your session file never changes. Only the --adapter flag.
What's shipped
- TypeScript + Python CLI — abslang init, chat, run, report
- Built-in evaluators — exact_match, contains, regex, schema, tool_call, llm_judge, Groundedness, Relevance, sequence, eventually, never, count, within, variable_consistency
- Agent adapters — OpenAI, Anthropic, Gemini (add yours with one function)
- Visual designer — React drag & drop, built into the docs site
- VSCode extension — visual editor + ▶ Run button (Work in progress)
- JSON Schema — normative, validated at parse time
Try it
npm install -g abslang
abslang init
abslang chat
GitHub: fvinciarelli/abslang — stars appreciated ⭐
Docs: fvinciarelli.github.io/abslang
v0.1, open for review. If you're building or testing agents, I'd love your feedback.
Top comments (0)