DEV Community

Cover image for How I Built an MCP Server That Lets Claude Control ComfyUI
Dmytro Bilukha
Dmytro Bilukha

Posted on

How I Built an MCP Server That Lets Claude Control ComfyUI

#opensource #webdev #ai #javascript

Demo Animation

ComfyUI is arguably the most powerful node-based runtime for generative diffusion models (SDXL, Flux, Stable Video Diffusion). However, working with ComfyUI usually involves either:

  1. Dragging virtual cables across hundreds of spaghetti nodes in a browser canvas, or
  2. Manually crafting fragile, deeply nested JSON workflow payloads.

With Anthropic’s Model Context Protocol (MCP), there is a better way: treat workflow construction as an agentic planning problem.

I built and open-sourced mcp-comfy-ui-builder. Here is how it works under the hood.

The Architecture

To make an LLM like Claude reliably build ComfyUI workflows without hallucinating invalid connections or crashing the GPU, the MCP server provides 3 core subsystems:

1. Embedded Knowledge Base & Pre-Flight Checks

Instead of blindly sending prompts, the agent first runs get_system_resources. This tool reads local GPU VRAM and memory, returning safe boundaries (max resolution, batch sizes).

The server bundles knowledge/base-nodes.json (62+ default nodes and compatibility rules). For custom nodes, npm run sync-nodes discovers and indexes them dynamically from your live runtime.

2. Declarative Workflow Construction

Rather than writing raw graph JSON, the model invokes high-level semantic tools:

  • build_workflow with pre-tuned templates (txt2img, txt2img_flux, img2img, inpainting, upscale)
  • add_node and connect_nodes for dynamic alterations
  • validate_workflow to ensure all type slots (MODEL, CLIP, VAE, LATENT) match before queueing

3. Real-Time WebSocket Execution

Standard HTTP polling against /prompt and /history adds 1–2 seconds of latency.

mcp-comfy-ui-builder maintains a persistent WebSocket connection to ComfyUI. During generation, the server receives sub-100ms progress events, showing node-by-node updates in Claude's UI before downloading outputs via list_outputs.

Quick Setup

1. Install & Build 

npm install mcp-comfy-ui-builder
npm run build
Enter fullscreen mode Exit fullscreen mode

2. Configure Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "comfy-ui-builder": {
      "command": "node",
      "args": ["/ABSOLUTE/PATH/TO/mcp-comfy-ui-builder/dist/mcp-server.js"],
      "env": {
        "COMFYUI_HOST": "http://127.0.0.1:8188"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Now open Claude Desktop and say:

"Generate a futuristic cyberpunk skyline at sunset with neon reflections, upscale 2x."

Links

Top comments (0)