DEV Community

Cover image for Building Domain-Specific AI Agents Through Configuration, Not Code
Arvindkumar Akula
Arvindkumar Akula

Posted on

Building Domain-Specific AI Agents Through Configuration, Not Code

#agents #llm #tooling #ai

Building Domain-Specific AI Agents Through Configuration, Not Code

I built a framework where you can create specialized AI agents (like a Compliance Reviewer or Travel Planner) by writing a YAML file instead of coding. Same core, infinite possibilities. Built entirely with Kiro IDE's powerful features.

The Problem: Building AI Agents is Too Code-Heavy

Want to build a compliance reviewer AI? Write hundreds of lines of code.

Need a travel planning assistant? Write hundreds more lines.

Want to add a customer support bot? You guessed it - more code, more complexity, more maintenance.

What if you could create a new AI agent just by writing a configuration file?

That's exactly what I set out to build during the Kiroween Hackathon 2025.

Introducing Agent Skeleton Framework

Agent Skeleton is a configuration-driven framework for building domain-specific AI agents. The same core framework powers completely different specialized agents:

🔍 Compliance Reviewer Agent

  • Reviews documents for regulatory compliance
  • Identifies policy violations with severity ratings
  • Provides specific regulation citations (FLSA, GDPR, OSHA)
  • Generates structured compliance reports

✈️ Travel Planner Agent

  • Creates personalized travel itineraries
  • Provides cost estimates and budget breakdowns
  • Suggests activities based on preferences
  • Offers weather forecasts and local insights

🎯 The Magic: Just YAML

Creating a new agent requires only a YAML file:

domain:
  name: "customer-support"
  description: "AI assistant for customer support"

personality:
  tone: "friendly"
  style: "helpful"

tools:
  allowed:
    - "ticket_search"
    - "knowledge_base_search"
    - "order_lookup"

constraints:
  - "Only respond to customer support questions"
  - "Always be empathetic and patient"
  - "Provide step-by-step solutions"
Enter fullscreen mode Exit fullscreen mode

That's it! No Python, no TypeScript, just configuration.

How It Works: The Architecture

1. Configuration-Driven Core

The framework has a single core that adapts based on YAML configuration:

┌─────────────────────────────────────┐
│         Agent Core                  │
│  • Loads domain config              │
│  • Filters tools by domain          │
│  • Applies personality              │
│  • Enforces constraints             │
└─────────────────────────────────────┘
              ↓
┌─────────────────────────────────────┐
│      Domain Configuration           │
│  • compliance-reviewer.yaml         │
│  • travel-planner.yaml              │
│  • customer-support.yaml            │
└─────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

2. MCP (Model Context Protocol) for Pluggable Tools

Tools are organized into swappable toolsets:

Compliance Toolset:

  • document_parser - Parse and chunk documents
  • policy_search - Search policy database
  • regulation_lookup - Find regulations

Travel Toolset:

  • destination_search - Find destinations
  • weather_lookup - Get weather info
  • price_estimator - Estimate costs
  • currency_converter - Convert currencies

The beauty: Tools are filtered by domain. The travel agent literally cannot access compliance tools - enforced at the code level, not just prompts.

3. Guardrails That Actually Work

One of the biggest challenges was preventing agents from answering out-of-scope questions.

The Problem:

User: "Plan a trip to Paris"
Compliance Agent: "Sure! Here's a 3-day itinerary..." ❌
Enter fullscreen mode Exit fullscreen mode

The Solution:
Place scope restrictions at the TOP of configuration:

constraints:
  - "SCOPE RESTRICTION: Only respond to compliance questions"
  - "If asked about travel, respond: 'I am a Compliance Reviewer. 
     For travel planning, please use the Travel Planner agent.'"
Enter fullscreen mode Exit fullscreen mode

Result:

User: "Plan a trip to Paris"
Compliance Agent: "I am a Compliance Reviewer agent specialized 
in regulatory compliance. For travel planning, please use the 
Travel Planner agent." ✅
Enter fullscreen mode Exit fullscreen mode

LLMs pay more attention to early content - this simple positioning change eliminated 90% of out-of-scope responses.

Built Entirely with Kiro IDE

This project showcases 5 major Kiro features working together:

1. 📋 Spec-Driven Development

I followed Kiro's complete spec workflow:

Requirements Phase:

  • Created requirements.md with EARS (Easy Approach to Requirements Syntax) patterns
  • Defined 12 major requirements with user stories and acceptance criteria
  • Established clear system boundaries

Design Phase:

  • Wrote comprehensive design.md with architecture decisions
  • Defined component interfaces and data models
  • Specified MCP tool architecture

Implementation Phase:

  • Generated tasks.md with 18 step-by-step tasks
  • Executed tasks incrementally with Kiro's assistance
  • Validated each step before proceeding

Impact: Caught architectural issues early, provided clear success criteria, enabled seamless resumption of work.

2. 🎯 Steering Documents

Created steering docs to guide behavior:

Base Steering (base_agent_behavior.md):

## Core Principles
- Always respect domain constraints
- Never fabricate information
- Acknowledge uncertainty when appropriate

## Response Format
- Provide structured, clear responses
- Use appropriate formatting
- Include reasoning for decisions
Enter fullscreen mode Exit fullscreen mode

Domain-Specific Steering:

  • compliance_specific.md - Formal tone, citation-focused
  • travel_specific.md - Friendly tone, creative suggestions

Impact: Consistent behavior without hardcoding personality.

3. ⚡ Vibe Coding

Used Kiro's AI assistance for rapid development:

Most Impressive Generation:
Asked Kiro to create the complete MCP toolset architecture. In one session, it generated:

  • Base MCPToolset class
  • Tool registry with validation
  • Two full domain toolsets (compliance and travel)
  • Proper error handling and JSON schema validation
  • 400+ lines of production-ready code

Impact: Built complete framework with API, CLI, and Web UI in days, not weeks.

4. 🔌 MCP Integration

Implemented Model Context Protocol for pluggable tools:

class MCPToolset:
    def __init__(self, name: str, description: str):
        self.name = name
        self.description = description
        self.tools: List[Tool] = []

    def register_tools(self, registry: ToolRegistry):
        for tool in self.tools:
            registry.register_tool(tool)
Enter fullscreen mode Exit fullscreen mode

Benefits:

  • Add new tools in 15 minutes
  • No core code changes needed
  • Domain-based tool permissions
  • Easy testing with mocks

5. 🔗 Agent Hooks

Implemented event-driven callbacks:

@hook("after_tool_call")
def log_tool_execution(context):
    logger.info(f"Tool: {context['tool_name']}, "
                f"Duration: {context['execution_time_ms']}ms")

@hook("after_response")
def track_metrics(context):
    metrics.record(context['execution_time_ms'])
Enter fullscreen mode Exit fullscreen mode

Impact: Extensible architecture with clean separation of concerns.

Real-World Demo: Compliance Review

Let me show you the framework in action:

User Query:

"Review this employee handbook for overtime policy compliance"

What Happens Behind the Scenes:

  1. Planning Phase: 
   Step 1: Parse document using document_parser
   Step 2: Search for overtime policies using policy_search
   Step 3: Lookup FLSA regulations using regulation_lookup
   Step 4: Compare requirements and generate report
Enter fullscreen mode Exit fullscreen mode

  1. Execution Phase:

    • Document parsed into chunks
    • Relevant policies found (Employee Overtime Policy - POL-001)
    • FLSA Section 7 regulation retrieved
    • Comparison performed

  2. Response:

   COMPLIANCE REVIEW REPORT

   Executive Summary:
   The employee handbook's overtime policy is COMPLIANT with 
   federal requirements under FLSA Section 7.

   Findings:
   ✅ Policy correctly states 1.5x rate for hours over 40/week
   ✅ Proper reference to FLSA requirements
   ✅ Clear definition of non-exempt employees

   Regulation References:
   - FLSA-Section-7: Fair Labor Standards Act - Overtime

   Severity: LOW (No violations found)
Enter fullscreen mode Exit fullscreen mode

Total Time: 3-5 seconds

The Technical Stack

Backend:

  • Python 3.10+ with FastAPI
  • Pydantic for data validation
  • OpenAI/Anthropic LLM support
  • In-memory and persistent storage

Frontend:

  • Next.js 14 with TypeScript
  • TailwindCSS for styling
  • Real-time step visualization
  • Domain switching with state isolation

Core Framework:

  • Planner: Goal decomposition and execution
  • Memory: Short-term and long-term strategies
  • Evaluation: Response validation and revision
  • Steering: Behavior guidance system
  • Hooks: Event-driven callbacks
  • Tools: MCP-based registry

Challenges and Solutions

Challenge 1: Tool Response Formatting

Problem: Tool results were showing as raw JSON instead of human-readable text.

Example:

{'policies': [{'id': 'POL-001', 'title': 'Employee Overtime Policy', 
'content': '...', 'relevance_score': 0.25}]} ❌
Enter fullscreen mode Exit fullscreen mode

Solution:
Modified the response synthesis logic to always format tool results through the LLM, even for single-step plans.

Result:

Based on your query about employee compliance, here are the 
top requirements:

**Employee Overtime Policy (POL-001)**
All non-exempt employees must be paid overtime at 1.5x their 
regular rate for hours worked over 40 in a workweek. ✅
Enter fullscreen mode Exit fullscreen mode

Challenge 2: Domain Switching UX

Problem: When switching between agents, chat history persisted, causing confusion.

Solution:

  1. Clear chat history on domain switch
  2. Clear execution steps and state
  3. Show system message indicating fresh start
  4. Separate memory contexts per domain

Result: Clean separation between domains with clear visual feedback.

Challenge 3: Performance

Initial: 10-15 seconds per query
Optimized: 3-5 seconds per query

Optimizations:

  • Switched to GPT-3.5-turbo (3x faster than GPT-4)
  • Disabled evaluation layer (saves 2-3s)
  • Reduced memory context (5 items vs 10-15)
  • Minimal verbosity in responses

Adding a New Agent: Customer Support Bot

Want to see how easy it is? Here's how to add a customer support bot:

Step 1: Create YAML (5 minutes) 

# domains/customer-support.yaml
domain:
  name: "customer-support"
  description: "AI assistant for customer support"

personality:
  tone: "friendly"
  style: "helpful"
  verbosity: "balanced"
  characteristics:
    - "Empathetic and patient"
    - "Solution-oriented"

tools:
  allowed:
    - "ticket_search"
    - "knowledge_base_search"
    - "order_lookup"

constraints:
  - "SCOPE RESTRICTION: Only respond to customer support questions"
  - "Always acknowledge customer frustration"
  - "Provide step-by-step solutions"
Enter fullscreen mode Exit fullscreen mode

Step 2: Create Toolset (15 minutes) 

# tools/customer_support_toolset.py
class CustomerSupportToolset(MCPToolset):
    def __init__(self):
        super().__init__("customer-support", "Customer support tools")
        self.tools = [
            self._create_ticket_search(),
            self._create_kb_search(),
            self._create_order_lookup()
        ]
Enter fullscreen mode Exit fullscreen mode

Step 3: Register (2 minutes) 

# api/main.py
customer_support_tools = CustomerSupportToolset()
customer_support_tools.register_tools(tool_registry)
Enter fullscreen mode Exit fullscreen mode

Step 4: Update UI (5 minutes) 

// ui/web/components/DomainSelector.tsx
const domains = [
  { id: 'compliance-reviewer', name: 'Compliance Reviewer' },
  { id: 'travel-planner', name: 'Travel Planner' },
  { id: 'customer-support', name: 'Customer Support' }, // NEW
];
Enter fullscreen mode Exit fullscreen mode

Total Time: ~30 minutes

Compare that to building a customer support bot from scratch (weeks of work)!

Key Learnings

1. Spec-Driven Development Works

Before: Jump into coding, refactor constantly, miss requirements.

With Kiro's Specs:

  • Requirements phase catches issues early
  • Design phase prevents architectural mistakes
  • Task phase provides clear roadmap
  • Implementation is smooth and predictable

Lesson: Upfront planning saves time overall.

2. Steering Documents Are Powerful

Discovery: Placing scope restrictions at the TOP of steering docs is critical.

Why it matters:

  • LLMs pay more attention to early content
  • Guardrails need maximum visibility
  • Prevents hallucinations effectively

Lesson: Document structure affects AI behavior significantly.

3. Configuration > Code for Flexibility

Insight: Configuration-driven architecture enables rapid iteration.

Benefits:

  • Changed agent personality in seconds
  • Added new tools without core changes
  • Adjusted guardrails without redeployment
  • Tested different configurations easily

Lesson: Separate configuration from logic for maximum flexibility.

4. MCP Protocol Scales

Without MCP:

  • Adding 10 tools = modifying core 10 times
  • Risk of breaking existing functionality
  • 2-3 hours per tool

With MCP:

  • Adding 10 tools = 10 independent toolsets
  • Zero risk to existing tools
  • 15 minutes per toolset

Lesson: Abstraction layers are worth the initial investment.

What's Next

Short-Term (Next Month)

  1. More Domain Examples

    • Customer Support Agent
    • Code Review Agent
    • Data Analysis Agent
    • Content Writer Agent

  2. Visual Configuration Builder

    • Drag-and-drop tool selection
    • Visual personality customization
    • Real-time validation
    • Export to YAML

  3. Enhanced Memory

    • Vector search for semantic retrieval
    • Cross-session persistence
    • Memory summarization

Medium-Term (Next Quarter)

  1. Multi-Agent Collaboration
    • Agent-to-agent communication
    • Task delegation between agents
    • Collaborative problem solving

Example:

   User: "Review this contract and plan a business trip"
   → Compliance Agent reviews contract
   → Travel Agent plans trip based on contract dates
   → Combined response delivered
Enter fullscreen mode Exit fullscreen mode

  1. Streaming Responses

    • Stream LLM responses as they generate
    • Show tool execution progress
    • Reduce perceived latency

  2. Tool Marketplace

    • Share custom toolsets
    • Rate and review tools
    • One-click installation

Long-Term (Next Year)

  1. Enterprise Features

    • Role-based access control
    • Audit logging
    • Multi-tenancy support
    • SSO integration

  2. Agent Analytics

    • Usage metrics per agent
    • Performance dashboards
    • Cost tracking
    • A/B testing framework

Try It Yourself

The framework is open source and ready to use:

GitHub: [Link to repository]

Quick Start:

# Clone the repository
git clone <repository-url>
cd agent-skeleton

# Install dependencies
pip install -e .

# Set up environment
cp .env.example .env
# Add your OpenAI/Anthropic API key

# Start the API
uvicorn api.main:app --reload

# Or use the CLI
python agent_cli.py --domain travel --goal "Plan a weekend getaway"
Enter fullscreen mode Exit fullscreen mode

Documentation:

  • Complete setup guide in README.md
  • Architecture details in ARCHITECTURE.md
  • Kiro usage documented in KIRO_USAGE.md
  • 7 PlantUML diagrams showing system flows

Conclusion

Building AI agents doesn't have to be code-heavy. With the right architecture and tools, you can create specialized agents through configuration alone.

Key Takeaways:

  1. Configuration-driven architecture enables rapid iteration
  2. MCP protocol provides clean, scalable tool integration
  3. Guardrails require intentional design (position matters!)
  4. Kiro's features work synergistically for powerful results
  5. Spec-driven development catches issues early and maintains quality

The Agent Skeleton Framework demonstrates that with proper abstraction and configuration, you can build production-ready AI agents in days, not weeks.

What would you build with this framework? Drop a comment below! 👇

Resources

Built with ❤️ using Kiro IDE for Kiroween Hackathon 2025 🎃

Top comments (0)