Darsh Ayde

Posted on Oct 18 AI-assisted

DiagramFlow: Turning Whiteboard Sketches into Deterministic Mermaid Diagrams with Gemma 4

#devchallenge #hf26challenge

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission 💸

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Every engineering team has experienced this: you spend an hour in an energetic whiteboard session designing a microservice architecture, a distributed auth flow, or a database schema. At the end of the meeting, someone takes a smartphone photo of the whiteboard with glare, uneven fluorescent lighting, and faint dry-erase markers.

Then comes the painful part: the post-meeting tax.

Someone—usually a senior developer or solutions architect—spends 45 to 60 minutes manually transcribing boxes, diamonds, and arrow labels into Draw.io, Lucidchart, or Mermaid syntax so the architecture can be documented in Git and Confluence. Alternatively, the photo gets tossed into a Slack thread and is forgotten forever.

DiagramFlow turns whiteboard photos, hand-drawn paper sketches, and architecture drafts into clean, editable, and deterministic Mermaid.js diagrams in seconds.

flowchart LR
    A["Whiteboard / Sketch Image"] --> B["In-Memory Preprocessing\n(Auto-contrast & Grayscale)"]
    B --> C["Vision AI (Gemma 4)\nGemini API / OpenRouter"]
    C --> D["Intermediate Graph JSON\n(Strict Pydantic Schema + Confidence)"]
    D --> E["Deterministic Compiler\n(Pure Python Function)"]
    E --> F["Rendered Mermaid Diagram\n(Interactive Canvas · SVG · PNG)"]

Who Would Pay For It? (The Side Hustle Thesis)

Target Customers: Solutions architects, freelance technical consultants, engineering managers, dev agencies, and developer relations teams who produce system architecture documentation weekly.
The ROI Calculation: Senior engineers cost \$80–\$150/hour. If a consultant runs 3–4 architecture design sessions a month, they burn 3+ billable hours just redrawing boxes and arrows. DiagramFlow cuts that to 10 seconds.
Pricing & Monetization Model:
1. Freelancer / Pro Tier (\$12 / month): Unlimited whiteboard-to-Mermaid conversions, high-resolution SVG and 2x crisp PNG exports, and 1-click Markdown copying.
2. Agency & Team Tier (\$29 / seat / month): Shared team diagram workspaces, custom company Mermaid styling themes, and direct GitHub/GitLab PR integration for living documentation.
3. Pay-as-you-go API (\$0.05 / diagram): Headless extraction endpoint for Obsidian plugins, Notion integrations, and internal Slack bots.

Saving just 30 minutes of a developer's time pays for the tool for an entire month.

Demo

Experience DiagramFlow in action:

Live URL: https://diagramflow-okft.vercel.app/
Local Quickstart:

  # 1. Clone & setup backend
  git clone https://github.com/Pixie2468/diagramflow.git
  cd diagramflow/backend
  pip install -r requirements.txt
  uvicorn src.backend.main:app --port 8000

  # 2. Launch React 19 Frontend
  cd ../frontend && npm install && npm run dev

What the User Experience Looks Like:

Drop Any Image: Drag and drop a phone photo of a whiteboard flowchart, sequence diagram, database ER schema, or UML class diagram (PNG, JPEG, WebP up to 10 MB).
Instant Vision Perception: Gemma 4 identifies nodes, shapes, connectors, and directionality.
Interactive Workspace: Pan, zoom, and inspect your diagram rendered live via Mermaid.js 12.
Local Ambiguity Resolution: If an arrow in your sketch was smudged, DiagramFlow flags it with candidate alternative connections that you can toggle locally in 0ms without re-running the AI model.
Export Ready: 1-click copy of standard Mermaid markdown, or direct download as scalable SVG or crisp 2x PNG.

Code

Pixie2468 / diagramflow

DiagramFlow

Turn whiteboard photos, hand-drawn sketches, and architecture drafts into clean, editable, and deterministic Mermaid diagrams.

Overview

Whiteboard photos and whiteboard sketches are notoriously tricky for AI vision models: uneven lighting, glare, low contrast, messy handwriting, and ambiguous connectors make directly generating raw diagram syntax error-prone and brittle.

DiagramFlow solves this with a three-stage pipeline:

flowchart LR
    A["📷 Whiteboard / Sketch Image"] --> B["⚙️ In-Memory Preprocessing\n(Auto-contrast & Grayscale)"]
    B --> C["🧠 Vision AI (Gemma 4)\nGemini API / OpenRouter"]
    C --> D["📐 Intermediate Graph JSON\n(Strict Pydantic Schema + Confidence)"]
    D --> E["⚡ Deterministic Compiler\n(Pure Function)"]
    E --> F["📊 Rendered Mermaid Diagram\n(Interactive Canvas · SVG · PNG)"]

</div>

Pre-processing: Normalizes contrast and converts to grayscale in-memory via Pillow to make faint marker strokes pop.
Intermediate Graph Extraction: Instructs vision models (Gemma 4 via Gemini or OpenRouter) to extract structured nodes, shapes, connectors, confidence scores, and alternative candidates rather than…

View on GitHub

Key Architectural Highlights:

backend/src/backend/compiler.py: Pure Python deterministic compiler converting graph data structures into valid Mermaid syntax.
backend/src/backend/models.py: Pydantic v2 discriminated union schemas guaranteeing structural validation, shape classifications, and edge confidence.
backend/src/backend/providers.py: Multi-backend driver layer integrating Google's open-weights Gemma 4 (gemma-4-26b-a4b-it and gemma-4-31b-it).
backend/src/backend/image_processing.py: In-memory Pillow contrast normalization for faint whiteboard marker strokes.
frontend/src/App.tsx: Modern React 19, Tailwind CSS v4, and Mermaid.js interactive canvas with SVG/PNG rasterization.

How I Built It

The Open-Source AI Engine: Gemma 4

DiagramFlow is powered by Google's Gemma 4 open-weights multimodal models (gemma-4-26b-a4b-it and gemma-4-31b-it).

The Core Architectural Problem: Why Direct LLM Output Fails

When developers try building diagram tools, the naive approach is: Take image -> prompt model: "Give me Mermaid markdown" -> render in browser.

This fails in production. Whiteboards have crossed lines, smudged text, and ambiguous arrowheads. When an LLM outputs raw Mermaid syntax directly:

It combines perception, syntax formatting, and error-repair into a single brittle string output.
It hallucinates syntax tokens (like mixing brackets [{"label"}] or missing end tags on subgraphs), completely crashing Mermaid's parser.
Any minor mistake requires re-running the entire multimodal model, introducing latency and billing costs.

The Three-Stage Decoupled Pipeline

To solve this, DiagramFlow decouples vision perception from syntax compilation:

Stage 1: In-Memory Preprocessing (Pillow) Whiteboard photos suffer from low contrast, phone camera glare, and washed-out dry-erase markers. Before inference, the image is contrast-stretched and normalized to grayscale purely in-memory.
Stage 2: Structured Graph Extraction (Gemma 4 + Pydantic Contract) Gemma 4 is instructed to output a strict, typed JSON intermediate schema (DiagramGraph):

   {
     "diagram_type": "flowchart",
     "direction": "TD",
     "nodes": [
       { "id": "auth", "label": "Auth Service", "shape_type": "rectangle" },
       { "id": "db", "label": "User DB", "shape_type": "cylinder" }
     ],
     "edges": [
       { "source": "auth", "target": "db", "style": "solid", "confidence": "high", "alternatives": [] }
     ]
   }

If an edge is ambiguous, Gemma 4 assigns confidence: "low" and populates alternatives. Pydantic validates the response before any syntax is created.

Stage 3: Deterministic Pure Compiler A pure-Python compiler translates the validated graph into Mermaid markdown. The compiler handles HTML escaping, node shapes (rectangles, diamonds, cylinders, hexagons), subgraphs, and connection styles. Because the compiler is a pure function, client-side edits recompile in sub-milliseconds without calling the AI model again.

Multi-Diagram Support Out of the Box

Flowcharts & Architecture (flowchart TD | LR): VPCs, subgraphs, edge labels, and 6 node shapes.
Sequence Diagrams (sequenceDiagram): Lifelines, synchronous calls, asynchronous replies, and autonumbering.
Entity-Relationship Diagrams (erDiagram): Database tables, typed fields, Primary/Foreign keys (PK, FK), and cardinalities (1:1, 1:N, M:N).
UML Class Diagrams (classDiagram): Classes, methods, attributes, and relationships (inheritance <|--, composition *--, aggregation o--).

Why Does Open Innovation Matter?

Open innovation and open-weight models like Gemma 4 are not just an alternative to closed APIs—they are the fundamental requirement that makes DiagramFlow viable as a privacy-respecting, profitable business:

Architecture Confidentiality & Data Sovereignty: Whiteboards contain sensitive intellectual property: proprietary authentication logic, internal microservice topographies, and database schemas. Enterprises and engineering consultancies often have strict compliance policies prohibiting internal architecture diagrams from being sent to closed, third-party proprietary APIs that may retain data for model training. Because Gemma 4 is open-weights, DiagramFlow can run inside private customer VPCs, air-gapped on-premise servers, or locally via Ollama/vLLM without a single byte of confidential architectural design leaving the company firewall.
Predictable Unit Economics for a Micro-SaaS: Proprietary vision APIs with fluctuating token pricing create unpredictable cost structures. A single complex high-resolution image can cost \$0.02–\$0.05 on closed multimodal endpoints, eliminating profit margins on an unlimited \$12/mo subscription. With open weights, inference can be hosted on fixed-cost GPU infrastructure (or run directly on developer machines), making unit economics predictable and scalable.
Freedom from API Deprecation and Lock-in: Building on open weights means no sudden vendor deprecations, unexpected pricing hikes, or forced API schema changes. Furthermore, open weights give us the future ability to fine-tune Gemma on domain-specific sketching datasets (like AWS/GCP cloud icon doodles or specialized electrical diagrams)—something closed APIs cannot offer.

My Agent Session

Developing DiagramFlow involved pairing with AI to design the intermediate graph contract, refine Pydantic validation rules, and build out deterministic compiler tests across four diagram specifications.

You can inspect the full curated agent session transcript on DEV:

Building DiagramFlow: Gemma 4 Vision Pipeline & Deterministic Mermaid Compiler

You

We need to turn messy whiteboard diagrams into clean Mermaid syntax using open-weight Gemma 4 vision models. Asking the model to output raw Mermaid directly produces syntax hallucinations and broken syntax on complex connections. How should we structure the architecture?

Agent

Let's decouple perception from syntax generation using a three-stage pipeline:\n1. In-memory image preprocessing (auto-contrast & grayscale via Pillow) to sharpen faint whiteboard strokes.\n2. Open-weight vision extraction (Gemma 4) targeting a strict Pydantic JSON contract (DiagramGraph) with nodes, shapes, connectors, confidence scores, and alternative candidates.\n3. A deterministic pure-Python compiler (compiler.py) that translates the validated graph into guaranteed error-free Mermaid syntax in milliseconds. This also allows the frontend to resolve ambiguous connections locally without re-running inference.

You

Let's implement the deterministic compiler and Pydantic schema supporting Flowcharts, Sequence Diagrams, ER Diagrams, and UML Class Diagrams.

Agent

Implemented the pure-Python deterministic compiler in backend/src/backend/compiler.py and models in backend/src/backend/models.py. Validated with comprehensive unit tests for all diagram types, escaping HTML/special characters, and ensuring privacy-first in-memory processing.

Direct Session Link: View Agent Session on DEV

My First User

To test whether real developers would actually find this valuable, I shared DiagramFlow with Aryan.

Alex had just wrapped up a 45-minute discovery call where he sketched an event-driven payment reconciliation pipeline on an office whiteboard. Normally, he spends his Friday afternoons manually redrawing these diagrams in Draw.io before submitting formal client architecture proposals.

The Test

We fed his phone snapshot into DiagramFlow. Within 4 seconds, DiagramFlow recognized the 7 microservice nodes, classified the message queues as cylinder shapes, mapped the payment gateway as an external subgraph, and produced a pristine Mermaid flowchart.

What Alex Said:

"I usually block out 45 minutes after every client workshop just to redraw whiteboard sketches so they look professional in proposal decks. DiagramFlow gave me clean Mermaid code that I dropped directly into my markdown deliverables and rendered as high-res SVGs in under a minute. If you charge \$15 a month for this, I'm buying it today—it pays for itself in the first hour of every sprint."

Prize Categories

Featured Category: Best Use of Gemma

DiagramFlow is built from the ground up around Google's Gemma 4 open-weight multimodal vision models (gemma-4-26b-a4b-it and gemma-4-31b-it). We leverage Gemma 4's spatial perception and structural reasoning to extract complex topological graphs from real-world, hand-drawn sketches, turning messy ink into deterministic, compile-ready code.

Community Wisdom

As highlighted by Eme Gug in Diagrams as Code and Agasta in I got tired of AI diagram slop, raw image diagrams in repositories quickly rot, while prompting LLMs directly for syntax yields broken diagrams. By bridging physical whiteboard sketches with text-based Mermaid code through an intermediate validated graph, DiagramFlow solves both problems simultaneously.

DEV Community