Under the Hood: Teaching an LLM to Draw in Excalidraw — an Agent Loop in 100 Lines

The Setup

One day I asked ChatGPT to draw an org chart. Do you know what it did? It gave me ASCII art:

        CEO
       /   \
   VP Eng  VP Prod
    /  \      |
  FE   BE    QA

Cute, but not what I wanted.

I wanted a real diagram: rectangles, arrows, colors, zoom, and the ability to move elements around manually. Not an image embedded in a chat response, not ASCII art, but a proper interactive canvas.

I could have reached for LangChain, MCP, a ready-made agent framework, and hidden everything behind several layers of abstractions. But I was interested in a different question: what is the minimum needed to turn an LLM into an agent that does not just answer with text, but changes the state of the outside world?

In my case, that “outside world” was an Excalidraw canvas.

I wrote a small agent loop from scratch. Important clarification: not the whole project, of course, but the agent core itself — roughly 100 lines. Around it there is a server, a model layer, WebSocket sync, a UI, and a lot of small engineering details. But the actual “agentic” part lives in that loop. No LangChain, no MCP, no special agent libraries.

Spoiler: the agent did not just draw the org chart. In one run, it drew the chart, inspected the result, noticed that the arrows were wrong, deleted them, recreated them with proper bindings, and only then said “done”.

Here is how it works under the hood.

The Idea: Don’t Ask for an Image, Give the Model Tools

A normal chat with an LLM looks like this:

user prompt -> LLM -> text answer

An agent loop adds one important ingredient: tools.

In other words, we do not ask the model “draw me a diagram” and hope that it somehow magically creates an image. We give it functions. For example:

• get_scene — inspect what is already on the canvas;
• batch_create_elements — create a batch of elements at once;
• update_element — fix or adjust the result;
• done — finish the task.

This is not the full list, just an illustration of the idea. I will show the complete tool set later.

The model does not draw by itself. It says: “call batch_create_elements with these arguments”. My code executes that call and changes the canvas state.

The Agent Core

In simplified form, the whole agent loop looks like this:

import { runTool, TOOLS } from './tools.js'

export async function agent(prompt: string, sessionId = 'default') {
  const log: string[] = []
  const messages: any[] = [
    { role: 'system', content: systemPrompt },
    { role: 'user', content: prompt },
  ]

  for (let turn = 0; turn < 15; turn++) {
    const response = await fetch(LLM_URL, {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${LLM_KEY}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: LLM_MODEL,
        messages,
        tools: TOOLS.map((tool) => ({
          type: 'function',
          function: {
            name: tool.name,
            description: tool.description,
            parameters: tool.parameters,
          },
        })),
        tool_choice: 'auto',
      }),
    })

    const data = await response.json()
    const msg = data.choices[0].message
    // msg may contain plain text, tool_calls, or both
    messages.push(msg)

    if (isDone(msg)) {
      return { turns: turn, log }
    }

    for (const toolCall of msg.tool_calls || []) {
      const args = JSON.parse(toolCall.function.arguments || '{}')
      const result = await runTool(toolCall.function.name, args, sessionId)

      messages.push({
        role: 'tool',
        tool_call_id: toolCall.id,
        content: JSON.stringify(result),
      })
    }
  }

  return { turns: 15, log }
}

This is not the production-ready version. The real code also has a timeout, HTTP status checks, LLM error handling, and safe JSON parsing. But the essence is exactly this.

There are three steps in the loop:

1. Send the LLM the conversation history and the tool descriptions.
2. Receive the model’s response: plain text, tool calls, or both.
3. Execute the tool calls and add their results back into the message history.

After that, the model receives a new context and can take the next step.

That is almost all the “magic” of agentic behavior.

How the Model Knows Which Tools Exist

Each tool is described in a format similar to JSON Schema. For example, creating an element:

{
  name: 'create_element',
  description: 'Create a new Excalidraw element. For arrows, use startElementId/endElementId to bind to shapes.',
  parameters: {
    type: 'object',
    properties: {
      type: {
        enum: ['rectangle', 'ellipse', 'diamond', 'arrow', 'text', 'line']
      },
      x: { type: 'number' },
      y: { type: 'number' },
      width: { type: 'number' },
      height: { type: 'number' },
      text: { type: 'string' },
      backgroundColor: { type: 'string' },
      strokeColor: { type: 'string' },
      startElementId: { type: 'string' },
      endElementId: { type: 'string' },
    },
    required: ['type', 'x', 'y'],
  },
}

The LLM sees this description and understands which arguments it can pass. For example:

{
  "type": "rectangle",
  "x": 320,
  "y": 80,
  "width": 160,
  "height": 70,
  "text": "CEO",
  "backgroundColor": "#a5d8ff"
}

The model does not need to know the full internal Excalidraw format. It works through a more convenient tool layer, and the server turns those parameters into real canvas elements.

How the Model “Sees” the Canvas

The important part: the model does not see the image.

When I write that “the model sees” or “the model understands”, that is, of course, a readability shortcut. It does not look at the canvas with eyes. It receives a textual description of the scene:

[rectangle] "CEO" at (350, 50)
[rectangle] "VP Engineering" at (130, 230)
[rectangle] "VP Product" at (570, 230)
[arrow] from CEO to VP Engineering
[arrow] from CEO to VP Product

This is what the get_scene tool is for. When the model calls it, the server walks through the elements in the current session and returns a compact description.

This gives us a simple feedback loop:

draw -> get_scene -> analyze -> fix -> get_scene -> done

This is the step that turns a JSON generator into an agent. Without feedback, the model just fires once with a set of elements. With feedback, it can inspect the result and adjust it.

Why batch_create_elements Matters More Than It Seems

At first I thought a single create_element tool would be enough. The model would create elements one by one: a rectangle, some text, an arrow, another rectangle, another arrow.

That quickly becomes inconvenient:

• too many round trips to the LLM;
• more chances to make a mistake halfway through;
• harder to link elements together;
• diagrams take longer to create.

So I added batch_create_elements: the model can send the whole set of elements in a single tool call.

For example:

{
  "elements": [
    { "id": "ceo", "type": "rectangle", "x": 350, "y": 50, "text": "CEO" },
    { "id": "eng", "type": "rectangle", "x": 150, "y": 220, "text": "VP Engineering" },
    { "id": "prod", "type": "rectangle", "x": 550, "y": 220, "text": "VP Product" },
    { "type": "arrow", "startElementId": "ceo", "endElementId": "eng" },
    { "type": "arrow", "startElementId": "ceo", "endElementId": "prod" }
  ]
}

This improves the quality of the result a lot. The model stops thinking in individual shapes and starts operating on the diagram as a whole: structure, connections, and relative layout all at once.

The Most Annoying Detail: Arrows

Creating a rectangle is easy. Creating text is easy. Even the background color is easy.

Arrows were the annoying part.

If you simply create a line from point A to point B, it may look fine visually. But it is not a real Excalidraw arrow bound to shapes. If you later drag a rectangle, the arrow stays where it was.

I wanted the agent to create diagrams that could still be edited manually. So arrows had to be bound to elements.

For that, the model passes not start and end coordinates, but startElementId and endElementId:

{
  "type": "arrow",
  "startElementId": "ceo",
  "endElementId": "vp-eng"
}

The server then calculates points on the edges of the shapes and fills in Excalidraw bindings:

function edgePoint(el, targetX, targetY, gap) {
  const cx = el.x + el.width / 2
  const cy = el.y + el.height / 2
  const dx = targetX - cx
  const dy = targetY - cy

  const hw = el.width / 2
  const hh = el.height / 2

  const scale = Math.abs(dx) * hh > Math.abs(dy) * hw
    ? hw / Math.abs(dx)
    : hh / Math.abs(dy)

  return {
    x: cx + dx * scale + (dx > 0 ? gap : -gap),
    y: cy + dy * scale + (dy > 0 ? gap : -gap),
  }
}

For a rectangle, this finds the intersection of the line “shape center -> target” with the rectangle edge. For ellipses and diamonds, the formulas are slightly different; I omitted those branches here for brevity. The idea is the same: do not force the model to calculate geometry. Hide that inside the tool.

This is an important principle: a good tool should be convenient for the model, not for the internal format of the library.

What the Agent Can Do

In the current version, the agent has around two dozen tools. It can:

• create elements;
• create elements in batches;
• read the scene;
• update elements;
• delete elements;
• group and ungroup;
• align and distribute;
• lock elements;
• create a scene snapshot;
• restore a snapshot;
• control the viewport;
• read a small design guide;
• finish the task with done.

But in practice, it mostly uses four or five tools:

1. get_scene — inspect what is already on the canvas;
2. batch_create_elements — draw the main diagram;
3. create_element — add a single element or arrow;
4. update_element — fix the result;
5. done — finish.

That is also a useful lesson: the model does not need a huge API. It needs a small set of well-described actions.

The Agent That Fixed Itself

The most interesting moment happened while testing a three-level org chart.

I asked the agent to draw a company structure. In that run, the signal came from the textual scene: after calling get_scene, the model saw that arrows existed as standalone elements, but did not look like bound relationships between shapes via startElementId and endElementId. So it decided to delete them and recreate them as properly bound arrows.

The log looked roughly like this:

🤖 Let me check the current scene
  🛠 get_scene({})

🤖 Canvas is empty. Creating org chart...
  🛠 batch_create_elements({ 6 rectangles, 5 arrows })

🤖 The arrows are wrong — they're not bound to shapes. Let me fix that.
  🛠 delete_element({ id: "arrow-1" })
  🛠 delete_element({ id: "arrow-2" })
  🛠 delete_element({ id: "arrow-3" })
  🛠 delete_element({ id: "arrow-4" })
  🛠 delete_element({ id: "arrow-5" })

🤖 Recreating arrows with proper bindings
  🛠 create_element({ type: "arrow", startElementId: "ceo", endElementId: "vp-eng" })
  🛠 create_element({ type: "arrow", startElementId: "ceo", endElementId: "vp-prod" })
  🛠 create_element({ type: "arrow", ... })
  🛠 create_element({ type: "arrow", ... })
  🛠 create_element({ type: "arrow", ... })

🤖 Let me verify the result
  🛠 get_scene({})

🤖 Done! Here's your org chart.

Of course, this does not mean the model will always do this. It is not a formal guarantee, and it is not magic.

But it shows an important thing: if you give the model tools, execution results, and a way to inspect state, it can sometimes behave like a simple autonomous worker:

inspect -> act -> verify -> fix -> finish

To me, this is where the line between “an LLM with functions” and “an agent” starts to appear. Not in the number of frameworks, but in the presence of a feedback loop.

Where the Real Problems Started

The agent core is small. That does not mean the whole project is simple.

The complexity just moves out of the agent loop and into other places.

1. Scene Format Matters More Than It Seems

The model does not see the canvas. It only sees what get_scene returns. If the description is too poor, the model cannot understand what is wrong. If it is too verbose, the context gets polluted quickly.

You need a format that is informative enough without turning into a full Excalidraw JSON dump.

2. Tool Descriptions Are Part of the UX

Tools are not only for your code. They are also for the model.

A bad tool description is like a bad UI for a human. The model will call the function incorrectly, confuse arguments, or forget important fields.

For example, for arrows I had to explicitly write: “use startElementId/endElementId to bind to shapes”. Without that hint, the model often tried to draw arrows using coordinates.

3. Batch Operations Are Almost Mandatory

If the agent draws a large diagram one element at a time, it burns too many iterations. That is why batch_create_elements became one of the most important tools.

4. Real State Must Live on the Server

If the canvas is open in the browser, the agent runs on the server, and the elements exist only in frontend state, things break quickly.

That is why the project has a simple model layer: sessions, elements, snapshots, and WebSocket notifications. This is no longer “an agent in 100 lines”, but without it the agent has nothing stable to work with.

5. A Public Demo Is Dangerous Without Limits

If /api/agent is exposed to the internet, anyone can start burning your LLM API key. I will not go deep into security here — that is a separate article — but the minimum checklist is:

• token guard;
• rate limit;
• prompt length limit;
• timeout on LLM requests;
• careful error handling.

This part is boring, but you cannot safely put this kind of project on a VPS without it.

Bonus: What If the Agent Looks at Itself?

I tried adding this line to the system prompt: “You are an AI agent that draws diagrams. Your TypeScript code lives in src/agent.ts.”

The response was funny. The agent answered in English; the meaning was roughly:

I am an AI agent running in a loop. I have a set of tools, and I call functions to draw diagrams. My code is in agent.ts: I could read it and explain how I work, but I would rather just draw the diagram.

It did not start philosophizing about itself. It kept working on the task.

In a way, that is the right behavior. A good agent does not need to reflect on its own nature. It should move the state toward the goal.

The Takeaway

This project is not really about Excalidraw.

Excalidraw is just a convenient sandbox where you can immediately see the result of the agent’s actions. The same approach could be used for GitHub issues, the filesystem, CI/CD, a CRM, a calendar, or almost anything else.

Here are my main takeaways.

1. The Agent Loop Is Simpler Than It Seems

In minimal form, it is this:

There is no mandatory hidden magic inside.

2. Tools Matter More Than the Framework

LangChain, MCP, and other tools can be useful. But agentic behavior does not appear because of them. It appears when the model has clear actions and feedback.

3. The Model Can Fix Mistakes If It Can See the Result

Not always. Not perfectly. But it can.

If the agent can inspect the state after its own actions, it gets a chance to notice a problem and make the next iteration better.

4. “100 Lines” Is Only the Core

Yes, the agent loop really is small. But the whole project is larger: model layer, tools, WebSocket, UI, Excalidraw element format, arrow binding, deployment, and safety limits.

And that is fine. The agent loop is small. The useful environment around it is not.

Code

The project is open source:

https://github.com/mikhail-angelov/excalidraw-agent

It is based on mcp_excalidraw by yctimlin: I reused ideas around the canvas, the UI, and the element format. I wrote the agent loop from scratch to understand what is really happening under the hood.