Intellisketch : AI powered drawing tool 🖌️

IntelliSketch: Building an AI-Powered Mathematical Sketching App

Recently, I was inspired by a video on Apple's official YouTube channel showcasing an AI-powered calculator app called Math Notes for iPad. The app's ability to evaluate drawn mathematical equations and graphical questions in real-time intrigued me. So, I decided to build my own version using the Gemini API and ReactJS. I call it IntelliSketch.

• 🔗 Live Demo
• 📂 GitHub Repository

The Development Process

Frontend

• ReactJS
• Tailwind CSS
• Excalidraw npm package
• Deployed on Vercel

Backend

• ExpressJS
• Multer
• Sharp
• Google Generative AI npm package
• Deployed on Render

Initially, I considered building the drawing app from scratch. However, after discovering Excalidraw, I decided to use their npm package for the drawing canvas. With the canvas sorted, it was time to implement the AI functionality.

How It Works

1. The user draws on the canvas.
2. When ready to evaluate, the user clicks the "Evaluate with AI" button.
3. The app takes a screenshot of the canvas.
4. The screenshot is sent to the Gemini API for evaluation.
5. Gemini analyzes the image and returns a text description.
6. Results are displayed in a modal.

Testing and Results

The results were impressive! The Gemini AI API successfully solved mathematical equations and interpreted abstract drawings. Here are some examples:

Example 1: Evaluation of a Cricket Wagon Wheel

Example 2: Solving Quadratic Equations

Example 3: Interpreting Abstract Drawings

The AI Prompt

Here's a simplified version of the prompt I used to instruct the Gemini AI API:

Analyze the given image containing mathematical expressions, equations, or graphical problems. Respond based on the content:

1. For simple mathematical expressions: Solve and return the answer.
2. For equations: Solve for variables and return values.
3. For graphical problems: Describe the problem and provide the result.
4. For non-mathematical images: Provide a description.
5. Handle edge cases with appropriate error messages.
6. Use the PEMDAS rule for solving mathematical expressions.
7. Utilize user-assigned variables when applicable.

Conclusion

Building IntelliSketch was an exciting journey that showcases the power of AI in enhancing mathematical understanding and problem-solving. The Gemini AI API's capabilities in interpreting and solving various mathematical and graphical problems are truly impressive.

I'd love to hear your thoughts on this project! Have you worked on similar AI-powered tools? What do you think about the intersection of AI and education? Let's discuss in the comments below!

---