I Built an Open-Source Studio for Building, Testing, and Deploying AI Agents

Mohammad Joud Julius — Thu, 01 Oct 2026 10:38:09 +0000

I Built an Open-Source Studio for Building and Shipping AI Agents

Building an AI chatbot is easy.

Shipping one that has tools, knowledge, memory, observability, evaluations, human handoff, multiple model providers, workflows, and an actual interface your users can interact with is a different problem.

That gap is what led me to build Chatbot Studio.

It's an open-source platform for building AI agents and then publishing those agents as website chatbots or connecting them to channels such as WhatsApp.

This isn't a SaaS announcement or a sales pitch. The project is MIT licensed, self-hostable, and available on GitHub:

GitHub: https://github.com/judejulius/ChatbotStudio

The problem I wanted to solve

A lot of AI projects begin with something like this:

response = client.chat.completions.create(
    model="...",
    messages=[...]
)

Then reality arrives.

You need tools.

Then retrieval.

Then credentials.

Then streaming.

Then conversation state.

Then rate limits.

Then evaluations.

Then traces because something went wrong in production.

Then a UI.

Then somebody asks:

"Can we put this on the website?"

And someone else asks:

"Can customers talk to a human if the AI gets stuck?"

At that point you're no longer building a prompt around an LLM.

You're building an agent platform.

That is the problem Chatbot Studio is trying to explore.

The core design: the agent and the chatbot are different things

One architectural decision became especially important while building this.

An agent should not be the same object as its presentation layer.

The agent owns things such as:

model provider
model
system instructions
tools
MCP servers
knowledge
memory
skills
guardrails
sandbox settings
human-in-the-loop behavior

A published chatbot owns the things that belong to the channel:

appearance
allowed domains
usage limits
launcher configuration
welcome experience
suggested prompts
publishing state

In other words:

Model + Knowledge + Tools
          |
          v
        Agent
          |
     test / evaluate
          |
          v
   Published Chatbot
          |
   Website / Channel

The chatbot isn't a duplicated agent configuration.

It's a channel sitting on top of the agent.

That means I can improve an agent's instructions, knowledge, model, or tools without recreating every chatbot using it.

The agent remains the source of truth.

Building agents

The main Agent Studio lets you configure and test an agent before publishing it.

Currently the platform supports provider families including:

OpenAI
Anthropic
Google Gemini
Groq
OpenRouter
Ollama
OpenAI-compatible endpoints

An agent can then be connected to knowledge bases, tools, MCP servers, memory, skills, guardrails, and other runtime controls.

The test chat runs against the saved agent configuration.

That part matters.

I didn't want a playground where the test environment was secretly different from the thing that eventually gets deployed.

The goal is:

configure -> test -> evaluate -> publish

rather than:

prototype -> rewrite everything -> deploy something different

MCP support

One part I've been particularly interested in is Model Context Protocol support.

Chatbot Studio can connect agents to MCP servers using:

stdio
SSE
streamable HTTP

This makes external capabilities much easier to attach to an agent without baking every integration directly into the application.

The broader architecture becomes something like:

                 +----------------+
                 | Knowledge Base |
                 +-------+--------+
                         |
+----------+      +------v------+
| MCP Tool +----->+             |
+----------+      |    Agent    |
                  |             |
+----------+      +------+------+
| API Tool +------------+
+----------+             |
                         v
                  Conversation

For me, MCP is interesting because it moves agent tooling toward a more interoperable ecosystem instead of every project inventing its own tool interface.

RAG and knowledge bases

Agents can also be connected to reusable knowledge bases.

The backend currently handles sources including:

text
URLs
PDF
DOCX
Markdown
CSV
JSON

Documents are processed into chunks and embeddings that can be retrieved during conversations.

The important idea here was making knowledge a reusable platform resource rather than stuffing documents directly into one chatbot implementation.

One knowledge base can therefore become part of a broader agent configuration.

The chatbot is an actual Web Component

For deployment on websites, I wanted the exported chatbot to be as framework-independent as possible.

So the primary integration is a browser-native custom element:

<script
  type="module"
  src="https://your-domain.com/widget.esm.js?widget=wgt_..."
></script>

<chatbot-widget></chatbot-widget>

The component uses a Shadow DOM so the host website's CSS doesn't unexpectedly destroy the chatbot UI—and the chatbot doesn't leak its styles back into the host application.

The element also exposes a small JavaScript API:

const chatbot = document.querySelector("chatbot-widget");

chatbot.open();
chatbot.send("I need help with an order");
chatbot.close();
chatbot.reset();

And it emits events such as:

chatbot-ready
chatbot-error
chatbot-handoff

Because the core is a Web Component, framework integrations can stay fairly thin.

Chatbot Studio can generate integrations for:

native HTML
React
Vue
Angular
WordPress

Instead of maintaining five completely different chatbot implementations, they all revolve around the same browser-native element.

The visual chatbot customizer

I also wanted customization to use the real renderer.

The editor therefore mounts the same widget implementation that gets exported.

You can customize things such as:

launcher
window
header
greeting
suggested prompts
agent message bubbles
visitor message bubbles
composer
loading indicator
scroll controls
footer
dark/light behavior
lead capture
knowledge citations
streamed reasoning
human handoff
usage limits
allowed domains

This avoids a problem I've seen in visual builders where the editor preview looks one way and the actual embedded component behaves differently.

The preview and the shipped widget share the same rendering path.

Human handoff

AI shouldn't have to pretend it can solve every problem.

So Chatbot Studio also has human handoff.

Instead of handing a conversation off to an undefined "human", conversations can be routed into queues such as:

General Support
Technical Support
Sales
Billing

Administrators control which users belong to which queues.

The AI can hand the conversation to an appropriate queue, after which a person can take ownership of it.

Once handed over, the assistant stops trying to answer the conversation as though nothing happened.

This required treating handoff as application state rather than just another tool response.

Agent teams and visual workflows

Not every task belongs inside one large agent.

The project therefore also supports agent teams and visual workflows.

The workflow side supports concepts including:

DAG execution
conditional paths
input mapping
human approval
persisted runs
scheduled execution
execution traces

The UI uses XYFlow for the visual workflow editor.

This lets you move from:

User -> Agent -> Response

toward workflows such as:

                    -> Research Agent -----
                   /                       \
Input -> Classifier                          -> Final Agent
                   \                       /
                    -> Internal Data Tool --

And because workflow runs are persisted and traced, there is something to inspect when an orchestration path doesn't behave as expected.

Evals and observability

One lesson from working with LLM applications is that manually chatting with an agent is not a sufficient testing strategy.

You eventually need repeatable evaluations.

Chatbot Studio includes evaluation suites so test cases can be run repeatedly instead of relying only on intuition.

There is also observability around things such as:

token usage
cost
latency
tool calls
traces
errors
sessions
visitor feedback

There is also a prompt optimization workflow where generated improvements can be reviewed before being accepted.

I deliberately wanted that last step to stay human-controlled.

An optimizer can propose a prompt change.

It shouldn't silently decide that production needs a new personality at 3 AM.

The project has also grown beyond browser chatbots.

There is a WhatsApp channel implementation using a Node.js bridge around Baileys.

It handles things including:

QR pairing
authentication persistence
inbound messages
outbound messages
quoted replies
typing state
debounce windows
audio transcription
optional generated voice replies

So the architecture is increasingly becoming:

                      +--> Website Widget
                      |
Provider -> Agent -----+--> WhatsApp
           |
           +-------------> Workflows
           |
           +-------------> Agent Teams

The channel shouldn't define the intelligence.

The agent should.

Under the hood

Chatbot Studio isn't built as one giant application.

The main pieces are:

frontend/
backend/
widget/
wa-bridge/

Frontend

The Studio UI is built with:

Next.js 16
React 19
TypeScript
Tailwind CSS
Zustand
XYFlow

Backend

The API and agent runtime use:

Python 3.12+
FastAPI
Pydantic
Motor
MongoDB
APScheduler
MCP

Widget

The embeddable chatbot is a separate JavaScript package built with esbuild.

WhatsApp bridge

WhatsApp transport runs as a separate Node.js service.

Infrastructure

The default Docker setup brings together:

Next.js
FastAPI
MongoDB
WhatsApp bridge

with an optional nginx SSL profile.

This separation has made it much easier to reason about where functionality actually belongs.

Running it locally

The project is designed to be self-hosted.

You'll need:

Node.js 20+
Python 3.12+
uv
MongoDB 7

Clone the repository:

git clone https://github.com/judejulius/ChatbotStudio.git
cd ChatbotStudio

Create your environment configuration:

cp .env.example .env

Install everything:

npm run install:all

Start MongoDB:

npm run mongo

Then run the development stack:

npm run dev

The frontend will be available on:

http://localhost:3000

and FastAPI's API documentation on:

http://localhost:8000/docs

There's also a Docker Compose setup:

docker compose up --build

Make sure you replace the placeholder secrets in .env before running a real deployment.

Security became part of the architecture

Once a platform stores model-provider credentials, MCP credentials, visitor conversations, and authentication data, security can no longer be an afterthought.

The project includes mechanisms around:

encrypted provider credentials
encrypted MCP secrets
JWT authentication
optional TOTP two-factor authentication
short-lived widget sessions
domain allowlists
rate limiting
visitor IP hashing
role-based access
queue-based conversation access

There is still always more security work to do in a project like this, but I've tried to make those concerns architectural rather than something added after everything else.

What I learned building this

The biggest lesson was that building an AI agent is usually not the hardest part.

The hard part is everything surrounding it.

A useful agent system needs answers to questions like:

Where do credentials live?

How are tools registered?

How is knowledge retrieved?

How do I test changes?

How do I inspect failures?

How do I embed this into another application?

What happens when the AI gets stuck?

Who takes over?

How do I control usage?

Can I change providers?

Can I run it myself?

Once you start answering those questions, you're no longer writing a chatbot script.

You're designing infrastructure around intelligent software.

That's the space I want Chatbot Studio to explore.

Why open source it?

Because I think the interesting part of AI agents right now isn't another closed chat interface.

It's figuring out the infrastructure patterns that make agents useful outside demos.

Things like:

MCP
agent orchestration
reusable knowledge
evaluations
observability
human escalation
portable widgets
multiple providers
self-hosting

These are problems developers should be able to inspect, modify, argue about, and improve.

Chatbot Studio is MIT licensed, so you can clone it, study it, change it, break it, rebuild parts of it, or use ideas from it in your own projects.

I'd like developer feedback

The project has become much larger than the original chatbot-widget idea, and there are plenty of areas I want to keep improving.

If you work with LLM applications, MCP, agent orchestration, RAG, frontend infrastructure, FastAPI, or self-hosted AI systems, I'd especially like feedback on the architecture.

I'm interested in hearing:

What would you change?
What integrations are missing?
Which parts should be separated further?
Where would you simplify the architecture?
What features would make it more useful for your own agent projects?

And if you find something broken, opening an issue is even better.

The repository is here:

https://github.com/judejulius/ChatbotStudio

If the project is useful or the architecture gives you ideas, a GitHub star is appreciated.

More importantly, I'd like to see what other developers build with it.

DEV Community: Mohammad Joud Julius