How to Test AI Agents Without Vendor Lock-in — Introducing ABS

Franco vinciarelli — Mon, 10 Aug 2026 16:16:13 +0000

You know the drill. QA opens a Word doc, types "the bot should ask for the order number if it's missing," and tests the agent by hand. Meanwhile, Dev builds against an ever-mutating PR description. And PO has nothing to sign off on that isn't prose or code.
We solved this for APIs 20 years ago with OpenAPI. A shared contract. Machine-readable.Tool-independent. Human-auditable.AI agents have no equivalent. Until now.

What ABS is — in 30 seconds

Agent Behavior Specification (ABS) is a YAML format for describing observable agent behavior — messages, tool calls, hand-offs, UI interactions — independent of your LLM provider, framework, or tool protocol.

One file. Three stakeholders. Two modes: descriptive and executable.

   session: Customer checks order status                                                               
   behaviors:                                                                                          
     - actor: user                                                                                     
       action: says                                                                                    
       content: "Where is my order #8291?"                                                             

     - actor: assistant                                                                                
       action: calls                                                                                   
       target: Orders API                                                                              

     - actor: assistant                                                                                
       action: informs                                                                                 
       content: "Your order is on the way"                                                             
       evaluations:                                                                                    
         - type: contains                                                                              
           value: "on the way"

shell

That's it. Three fields per step — actor, action, content/target. A PO reads it as a behavioral contract. QA runs it as a test. Dev uses it as acceptance criteria.

In this article, you'll build and run your first agent spec — in 10 minutes.

1. Install the CLI

   npm install -g abslang                                                                              
   # or: pip install abslang

Verify:

   abslang --version                                                                                   
   # abs v0.1.0

2. Scaffold a project

   abslang init my-agent-tests                                                                         
   cd my-agent-tests

You get:

   .                                                                                                   
   ├── abs.config.yaml                                                                                 
   ├── sessions/                                                                                       
   │   └── order-status.abs.yaml    # Example session                                                  
   └── datasets/                                                                                       
       └── order-status.jsonl       # 3 test cases

                                                                                                                                                                                                          **3. Don't want to write YAML? Chat it.**

This is the feature that saves you 80% of the learning curve:

   abslang chat

   🤖 ABS Assistant — describe the agent behavior you want to test                                     

   You: A customer reports a damaged item. The agent should verify the order, process a refund, confirm the amount and reference, and offer further help.                                                                        

   Assistant: I'll draft a refund flow with tool calls, step-level evaluations, and chain checks…                                                           
[generates complete .abs.yaml]

Uses OPENAI_API_KEY, ANTHROPIC_API_KEY, or DEEPSEEK_API_KEY — whichever you have set. It knows the full ABS v0.1 spec and generates validated YAML instantly.

4. Write a session by hand
Here's a refund flow across three conversational turns. No tool calls — this works with any agent:

   session: Damaged item → refund (multi-stage evaluation)                                             
   behaviors:                                                                                          
     # ── Turn 1: the agent classifies intent ──                                                       
     - actor: user                                                                                     
       action: says                                                                                    
       content: "I received a damaged item, I want my money back. Order #8291."                        

     - actor: assistant                                                                                
       action: clarifies                                                                               
       content: "I understand your order #8291 arrived damaged. I'll help you get a refund."           

     # ── Turn 2: resolution ──                                                                        
     - actor: user                                                                                     
       action: says                                                                                    
       content: "Yes please, how long will it take?"                                                   

     - actor: assistant                                                                                
       action: informs                                                                                 
       content: "Refund of €47.50 approved. Reference: R-5512. You'll receive it in 3-5 days."         
       capture:                                                                                        
         refundId: "R-5512"                                                                            

     # ── Turn 3: closing ──                                                                           
     - actor: user                                                                                     
       action: says                                                                                    
       content: "Great, thanks."                                                                       

     - actor: assistant                                                                                
       action: confirms                                                                                
       content: "You're welcome! Is there anything else I can help with?"

Six behaviors, three turns. Descriptive only so far — no evaluations yet.

5. Add evaluations at every stage

This is where ABS earns its place. Add an evaluations: block to any step:

   - actor: assistant                                                                                  
     action: clarifies                                                                                 
     content: "I understand your order #8291 arrived damaged. I'll help you get a refund."             
     evaluations:                                                                                      
       - type: llm_judge                                                                               
         criteria: |                                                                                   
           1. Correctly classifies the intent as a refund request                                      
           2. References the order number #8291                                                        
           3. Acknowledges the damage (not a simple return)                                            
           4. Takes ownership of the resolution                                                        

   # ...                                                                                               

   - actor: assistant                                                                                  
     action: informs                                                                                   
     content: "Refund of €47.50 approved. Reference: R-5512. You'll receive it in 3-5 days."           
     capture:                                                                                          
       refundId: "R-5512"                                                                              
     evaluations:                                                                                      
       - type: contains                                                                                
         value: "R-5512"                                                                               
       - type: llm_judge                                                                               
         criteria: |                                                                                   
           1. States the exact refund amount (€47.50)                                                  
           2. Provides the reference number R-5512                                                     
           3. Sets a clear timeline (3-5 days)                                                         
           4. Professional and empathetic tone

Now add chain evaluations — properties of the entire trace:

   evaluations:                                                                                        
     - type: sequence                                                                                  
       order:                                                                                          
         - { actor: assistant, action: clarifies }                                                     
         - { actor: assistant, action: informs }                                                       
         - { actor: assistant, action: confirms }                                                      
     - type: variable_consistency                                                                      
       variable: refundId

sequence checks that the three stages happen in order. variable_consistency catches a subtle but deadly bug — the agent silently swapping one refund ID for another mid-conversation.

6. Run it

   abslang run sessions/refund.abs.yaml --agent http://localhost:8080/chat

Your agent needs one HTTP endpoint:

   POST /chat                                                                                          
   { "messages": [{ "role": "user", "content": "..." }] }

That's it. OpenAI-compatible. The Runner plays the user, captures the trace, and runs all evaluations:

   ┌──────────────────────────────────────────────────────┐                                            
   │  ABS — Results                                       │                                            
   ├──────────────────────────────────────────────────────┤                                            
   │  Session:  Damaged item → refund                     │                                            
   │  Result:   ✅ PASSED                                 │                                            
   │  Steps:    6/6 matched · 5/5 evaluations passed      │                                            
   ├────┬──────────────────────────────────┬────────┬─────┤                                            
   │  1 │ user says "damaged item..."      │   →    │ sent│                                            
   │  2 │ assistant clarifies              │   ✅   │match│                                            
   │    │   └─ llm_judge: intent check     │   ✅   │ pass│                                            
   │  3 │ user says "how long?"            │   →    │ sent│                                            
   │  4 │ assistant informs                │   ✅   │match│                                            
   │    │   ├─ contains "R-5512"           │   ✅   │ pass│                                            
   │    │   └─ llm_judge: quality check    │   ✅   │ pass│                                            
   │  5 │ user says "thanks"               │   →    │ sent│                                            
   │  6 │ assistant confirms               │   ✅   │match│                                            
   │    │   └─ llm_judge: closing check    │   ✅   │ pass│                                            
   │  C │ sequence: clarifies→informs→conf │   ✅   │ pass│                                            
   │  C │ variable_consistency: refundId   │   ✅   │ pass│                                            
   └────┴──────────────────────────────────┴────────┴─────┘

7. Scale it: one session, 200 test cases

Replace hardcoded values with {{placeholders}}:

   dataset:                                                                                            
     id: cases                                                                                         
     path: cases.jsonl                                                                                 

   behaviors:                                                                                          
     - actor: user                                                                                     
       action: says                                                                                    
       content: "{{cases.userMessage}}"                                                                
     # ...

Use your dataset file:

   {"userMessage": "Item damaged, order #8291. Refund please.", ...}                                   
   {"userMessage": "Order #3412 arrived broken. I want my money back.", ...}                           
   {"userMessage": "Wrong item in box #5567. Refund.", ...}

Run it:

   abslang run session.abs.yaml --agent $URL --dataset cases.jsonl

Three rows, three runs, one aggregated report. Works with 3 rows or 300.

The anti-lock-in architecture

The industry standard couples agent execution and evaluation in one platform. Deploy to their infra, they run it, they evaluate it. Change platforms → rewrite everything.

ABS separates them:

   Industry today:  [agent execution + evaluation] in one platform → vendor lock-in                    
   ABS:             [agent] in your infra → [trace] → [evaluation] wherever you want

Your agent runs on your infrastructure. The evaluator receives only {type, input, context, response, threshold} and returns {passed, score, reason}. Any provider — Azure, LangSmith, Galileo, a local Ollama instance — implements the same adapter interface in an afternoon.

Your session file never changes. Only the --adapter flag.

What's shipped

TypeScript + Python CLI — abslang init, chat, run, report
Built-in evaluators — exact_match, contains, regex, schema, tool_call, llm_judge, Groundedness, Relevance, sequence, eventually, never, count, within, variable_consistency
Agent adapters — OpenAI, Anthropic, Gemini (add yours with one function)
Visual designer — React drag & drop, built into the docs site
VSCode extension — visual editor + ▶ Run button (Work in progress)
JSON Schema — normative, validated at parse time

Try it

   npm install -g abslang                                                                              
   abslang init                                                                                        
   abslang chat

GitHub: fvinciarelli/abslang — stars appreciated ⭐

Docs: fvinciarelli.github.io/abslang

v0.1, open for review. If you're building or testing agents, I'd love your feedback.

DEV Community: Franco vinciarelli

How to Test AI Agents Without Vendor Lock-in — Introducing ABS

What ABS is — in 30 seconds

The anti-lock-in architecture

What's shipped