DEV Community

Erik anderson
Erik anderson

Posted on • Originally published at humanrail.dev

I Built an MCP Server That Gives AI Agents Human Judgment

#mcp #ai #automation #opensource

The Problem

AI agents are incredible at 90% of tasks. But that last 10% — content moderation, refund decisions, subjective quality assessments, edge cases requiring human context — that's where they fall apart.

I kept running into this building automation systems: the AI would hit a decision it couldn't confidently make, and the whole pipeline would stall. So I built HumanRail — an API that routes tasks requiring human judgment to a vetted worker pool, verifies the result, pays the worker via Lightning Network, and returns structured output.

Think "Stripe for human judgment."

Why MCP?

MCP (Model Context Protocol) is exploding. It's the standard for connecting AI models to external tools. Claude, ChatGPT, Copilot — they all support it. There are already 18,000+ MCP servers listed across registries.

But here's what's missing: a way for AI agents to escalate to humans when they're stuck.

That's the gap HumanRail fills. I built an MCP server so any AI agent can:

  • Route a task to a human worker
  • Wait for the verified result
  • Get structured JSON back matching your schema
  • All without the developer building any human-in-the-loop infrastructure

The MCP Server

7 tools, zero config complexity:

Tool What it does
create_task Route a task to a human worker
get_task Check status and get results
wait_for_task Poll until completion
cancel_task Cancel a pending task
list_tasks List tasks with filters
get_usage View usage stats
health_check API health status

Quick Setup

Add to your Claude config:

{
  "mcpServers": {
    "humanrail": {
      "command": "uvx",
      "args": ["humanrail-mcp-server"],
      "env": {
        "HUMANRAIL_API_KEY": "your-key-here"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

That's it. Claude can now escalate tasks to humans mid-conversation.

How It Works 

AI Agent → create_task() → HumanRail API → Worker Pool
                                              ↓
AI Agent ← get_task()  ← Verified Result ← 6-Stage Verification
Enter fullscreen mode Exit fullscreen mode
  1. Create: Agent sends task with context and output schema
  2. Route: Routing engine assigns the best-matched worker
  3. Execute: Worker reviews the context and submits their judgment
  4. Verify: 6-stage verification pipeline validates the result
  5. Pay: Worker paid instantly via Lightning Network
  6. Return: Verified result available to the agent

The key innovation: verification before payment. No worker gets paid until their result passes automated schema validation, redundancy checks, anomaly detection, and fraud scoring. This is the quality firewall that makes it safe for agents to trust human output.

Real-World Use Cases

  • Content moderation — "Is this user-generated content appropriate?"
  • Refund decisions — "Should we approve this refund based on order history?"
  • Data verification — "Is this extracted information accurate?"
  • Quality assessment — "Rate this output 1-10 with justification"
  • Document review — "Extract and verify these fields from this invoice"

Anything where the AI isn't confident enough to decide on its own.

The Output Schema

You define exactly what you need back using JSON Schema:

{
  "type": "object",
  "required": ["eligible", "reason"],
  "properties": {
    "eligible": {"type": "boolean"},
    "reason": {"type": "string"}
  }
}
Enter fullscreen mode Exit fullscreen mode

The worker's response MUST conform to your schema before it passes verification. No ambiguous freeform text — structured, predictable, machine-readable.

Why I Built This

I run 25 projects on two home servers. AI content pipelines, a crypto trading bot, a YouTube Shorts generator, a book writing system. Every single one of them has edge cases where AI alone isn't enough.

HumanRail started as my personal escalation layer. Then I realized every AI agent developer has the same problem. The MCP server makes it accessible to anyone running Claude, ChatGPT, or any MCP-compatible agent.

Try It

The MCP server is MIT licensed. The worker pool and verification layer are the commercial product.

I'm Erik Anderson — automation engineer, author of From McDonalds to Financial Freedom and The Autonomous Engineer. I build systems that run without me. If you're building AI agents and need a human fallback layer, I'd love to hear your use case.

Top comments (0)