How I Built a Visual AI Agent Builder with Tool Calling — Architecture, Tools, and Lessons

AI agents are the most exciting thing happening in software right now — but most agent tutorials stop at "call the LLM in a loop." The part that's actually hard is giving the agent real tools, watching what it does with them, and trusting it enough to let it run.

I hit all three walls while building AgentForge, a visual AI agent builder with tool calling, real-time execution visualization, and run analytics. This post covers the architecture, the tool-calling design that makes it work, and what I'd do differently next time.

What I built

AgentForge is a full SaaS-style product where you:

• Create agents with custom system prompts, model, and temperature
• Attach tools from a library of 8 built-in tools across 6 categories
• Chat with agents and watch every tool call execute in real time — inputs, outputs, and durations
• Track every run with token usage, duration, and tool-call details
• Compare agent performance on a dashboard with charts

It's live here: agentforge-next.netlify.app

Tech stack

Layer	Technology
Framework	Next.js 16 (App Router)
Language	TypeScript (strict)
Database	SQLite + Prisma 7
Styling	Tailwind CSS v4
Components	shadcn/ui
Charts	Recharts
Icons	Lucide React
Deployment	Netlify

I'm a full-stack developer at RA Technologies, and this stack is exactly what we use for client SaaS and AI products — one codebase, strict types, and a database that doesn't fight you at the start.

The tool-calling design (the heart of the product)

The core problem: an LLM gives you a structured request to call a function, and you have to execute it safely and return something the model can understand. Everything else is engineering around that contract.

The tool interface

Every tool implements the same contract:

export interface Tool {
  name: string;
  category: ToolCategory;
  description: string;
  parameters: JSONSchema;
  execute(input: Record<string, unknown>): Promise<ToolResult>;
}

export interface ToolResult {
  ok: boolean;
  data?: unknown;
  error?: string;
  durationMs: number;
}

The registry

Tools are registered once and reused everywhere — the agent config screen, the chat executor, and the library page:

export const tools: Tool[] = [
  webSearch,      // category: "search"
  calculator,     // category: "utility"
  codeExecutor,   // category: "code"
  weather,        // category: "data"
  urlFetch,       // category: "network"
  dbQuery,        // category: "data"
  emailSender,    // category: "communication"
  fileReader,     // category: "utility"
];

export function getToolsByCategory(): Record<ToolCategory, Tool[]> {
  return tools.reduce((acc, tool) => {
    (acc[tool.category] ??= []).push(tool);
    return acc;
  }, {} as Record<ToolCategory, Tool[]>);
}

The execution loop

The agent loop is deliberately simple: send the conversation with the tool schema, read the tool calls from the response, execute, append results, repeat:

while (true) {
  const response = await model.chat.completions.create({
    model,
    messages,
    tools: tools.map(openAISchema),
    tool_choice: "auto",
  });

  const toolCalls = response.choices[0].message.tool_calls;
  if (!toolCalls?.length) {
    messages.push(response.choices[0].message);
    break;
  }

  messages.push(response.choices[0].message);
  for (const call of toolCalls) {
    const tool = registry[call.function.name];
    const started = performance.now();
    const result = await tool.execute(JSON.parse(call.function.arguments));
    messages.push({
      role: "tool",
      tool_call_id: call.id,
      content: JSON.stringify(result.data ?? { error: result.error }),
    });
  }
}

The 8 tools

Tool	Category	What it does
web_search	Search	Real-time web information
calculator	Utility	Math operations
code_executor	Code	Runs JavaScript safely
weather	Data	Weather forecasts
url_fetch	Network	Fetches URL content
db_query	Data	Queries databases
email_sender	Communication	Sends emails
file_reader	Utility	Reads file contents

The lesson here: tools with side effects (email, DB, code) need confirmation UX, while read-only tools (search, weather, fetch) can run automatically. AgentForge separates them by category so the UI can treat them differently.

Real-time tool-call visualization

The most requested feature by far. When an agent runs, the chat stream shows each tool call as a card with input → output → duration, updating live over SSE.

// Simplified SSE emission
export async function POST(req: NextRequest) {
  const encoder = new TextEncoder();
  const stream = new ReadableStream({
    async start(controller) {
      for (const step of runAgent(agent, message)) {
        controller.enqueue(encoder.encode(`data: ${JSON.stringify(step)}\n\n`));
      }
      controller.close();
    },
  });
  return new Response(stream, {
    headers: { "Content-Type": "text/event-stream" },
  });
}

This single feature is why people trust the product: an agent you can watch is an agent you can debug.

Run history and analytics

Every run is persisted — tokens, duration, and every tool call with its inputs and outputs. The dashboard then answers real questions:

• Which tools get used the most?
• Which agents burn tokens fastest?
• Which runs fail and where?

What I'd do differently

1. Schema-first from day one. I hand-wrote the tool parameter schemas. If I rebuilt this, I'd generate them from Zod types so the TypeScript type and the LLM schema can never drift.
2. Sandbox the code executor harder. It's isolated, but real deployments need per-user sandboxes (or WASM) — not just per-process isolation.
3. Add multi-agent orchestration earlier. Single agents are easy; the interesting problems start when agents delegate to each other.

Try it

• Live demo: agentforge-next.netlify.app
• Source: github.com/Raja-Abbas/AgentForge (MIT)
• Built at RA Technologies — we build SaaS and AI products like this for startups and businesses worldwide.

If you're building agents, I'd love to hear how you handle tool execution and side effects in the comments.

---