I built a tool that turns diagrams into something blind students can touch. Here's what broke.

Last weekend I did a hackathon (Banana Hacks 2026, an image AI hackathon). I wanted to build something that actually helps someone, not just another image generator.

So I built a tool called TouchSight. You give it any diagram, like a heart from a biology textbook, and it gives back three things:

• a tactile sheet you print on swell paper. The black lines puff up so you can feel them, and the labels are real Braille
• a 3D model you can spin, tap to hear each part, or 3D print
• an explore mode where you slide your finger over the picture and it tells you what you're touching

Live demo: https://touchsight-flame.vercel.app
Code: https://github.com/DhruvalAnandkar/touchsight

Why I built it

In science class the diagram is usually the lesson. A blind student can't see it, so someone has to remake it by hand as a tactile graphic. That takes a long time, so a lot of diagrams just never get made.

I wanted to see how much of that I could automate in one weekend.

How it works

It's a pipeline with 8 steps:

1. Rectify – if it's a phone photo of a page, OpenCV straightens it
2. Understand – Gemini vision figures out what the diagram is and names each part
3. Segment – OpenCV's GrabCut cuts out each part
4. Simplify – smooth the shapes so they're easy to feel
5. Texture – give each part a pattern (dots, lines, etc.)
6. Braille – draw the labels as real Braille dots
7. Relief – build the 3D model
8. Export – PDF, SVG and STL files

Backend is Python + FastAPI. Frontend is React + TypeScript, Three.js for 3D, and the browser's speech API for voice. It's on Vercel and Render, and GitHub Actions runs the tests on every PR.

Honestly, the models were the easy part. Everything around them is where things broke.

What broke (and how I fixed it)

1. The AI called heart chambers "blue polygon"

My first version asked Gemini to name the parts of a heart. Here is what it gave me:

"Upper left section"
"Faceted heart shield"
"Europe polygon" (for a world map)

That's technically correct but useless. A blind student needs to hear "left ventricle", not "blue polygon".

What fixed it:

• I told the model the labels are for a blind student learning by touch, and to use the word a teacher would use
• I added a "What is this diagram?" box. A teacher always knows this, so it costs nothing
• If the answer still has color or shape words in it, the code sends it back once and asks again

Big lesson for me: checking the AI's output matters as much as the prompt.

2. Real illustrations fell apart

On simple drawings, normal edge detection worked fine. On a realistic heart drawing with shading and little vessel lines, it cut the heart into tons of tiny pieces.

The fix was to use both halves together. Gemini already tells me roughly where each part is (a box).

I give those boxes to GrabCut and GrabCut finds the exact edges. After that, the realistic heart came out as 7 correctly named parts: superior vena cava, aorta, pulmonary trunk, and all 4 chambers.

3. The Earth's crust ate the mantle

I tested a diagram of Earth's layers. The mantle just disappeared.

The crust's box was the biggest, so it contained all the other layers, and its mask swallowed the mantle. The fix: if one part's box contains another one, the outer part only keeps the pixels nobody inside claimed. So it turns into a ring. I added a test with 3 circles inside each other that has to come out as 3 regions.

4. Touch has its own rules

This was the part I didn't expect. Things that look fine on screen feel wrong under a finger:

• Double borders. Each region drew its own outline, so two parts touching made two lines next to each other. With your finger that feels like two walls. Now every shared border is drawn only once.
• Same texture next to each other. My first version gave three touching heart chambers the same pattern, so you couldn't tell them apart by touch. The bug: I was checking which parts touch using the simplified shapes, which have small gaps between them. Now I check on the original masks, and touching parts always get different patterns.
• Textures touching the edge. I leave a 2 mm gap so your finger can find the outline.
• Braille as real dots. Not a font. 1.5 mm dots at standard spacing. If a label doesn't fit inside a part, it gets a number and the full name goes in the legend.

5. Browsers won't talk until you click

I wanted the app to talk as soon as it opens. Browsers block that until the user interacts with the page. So the app starts with a big "tap anywhere to start" screen. Also, many blind users already use their own screen reader, so the voice guide can be turned off, and every button has proper labels.

6. Free tier limits

I used Gemini's free tier, which has a daily limit per model. I hit it in the middle of testing. So now the code tries a list of models. If one says "limit reached" or the server is busy, it moves to the next one right away. I also made sure a failed answer never gets saved to the cache, so one bad moment doesn't stick forever.

What it can't do yet

• Only Grade 1 Braille (no contractions yet)
• I haven't tested it with blind students or their teachers. That's the most important next step, and I can't do it alone
• Really busy diagrams still need a human to decide what to simplify

If you can help

If you're a teacher of visually impaired students, or know one, I'd really love feedback.

If you are a developer, PRs are welcome! Whether you know OpenCV, Three.js, or just want to help improve the tactile generation logic, check out the repo on GitHub. Let's make science accessible for everyone.

Thanks for reading!