Reviving My AI-Powered Draw and Learn Game - What If an AI Could Look at Your Drawing and Actually Respond?

Krish Panchani — Fri, 05 Jun 2026 09:58:06 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

AI Playground: Draw, Create, and Explore is an AI-powered draw and learn game that runs entirely in the browser. You draw on a canvas, and Google's Gemini AI - one of the most capable vision models available today - looks at your drawing and responds: it scores your art, guesses what you drew, or writes the next chapter of a story based on what you created.

The core idea is simple: drawing is fun, and AI feedback makes it more interesting. Whether you're an artist or someone who draws stick figures and calls them dragons, the AI reacts to your specific drawing, not a generic response.

There are three game modes, each offering a completely different experience:

Creative Quest - The Main Learning Game
This is the heart of AI Playground. Each session gives you 10 drawing missions spread across 5 themed fantasy worlds (Forest, Crystal Caves, Ancient Ruins, Sky Realm, Dragon Peak). Before each chapter, Gemini AI generates a unique mission - something like "Draw a magical door hidden in an ancient forest" - and gives you a timer. You draw it, hit Submit, and Gemini actually looks at your drawing using its vision model and scores it on three dimensions:

Accuracy (50%) - How well does the drawing match the mission?
Creativity (30%) - Did you bring something original to it?
Effort (20%) - Did you put real work into the drawing?

Those three scores combine into a final score, which translates into XP. Complete all 10 chapters and you've finished an adventure.

Artful Guesswork - Can the AI Guess What You Drew?
Draw literally anything - a cat, a spaceship, your lunch. Gemini tries to identify it and gives you a confidence score (0-100%). You then rate whether it was right or wrong. No timer, no pressure. It's a surprisingly fun way to see how AI "sees" drawings. Every submission gives you 8 XP just for playing.

Artful Stories - Draw the Next Chapter
Your drawing becomes part of a story. You draw something, Gemini generates a short story chapter (max 150 words) narrating what happens based on what you drew, then you draw again for the next chapter. The whole illustrated story gets saved. It's creative writing + drawing in a loop. Each chapter earns 15 XP.

Wrapped around all three modes is a full product layer that makes the game feel complete: Google Sign-In, a player profile with your XP and level, a full adventure history log, a community art gallery showing everyone's drawings, and a live leaderboard ranking the top 20 players.

Tech Stack:

Layer	Technology
Frontend	React 19, Vite, Tailwind CSS, Fabric.js (canvas), Framer Motion
State & UX	Zustand, React Context, react-hot-toast
Backend	Node.js, Express, Zod validation
Database	MongoDB (Atlas) via Mongoose
AI	Google Gemini 2.5 Flash - vision for scoring, text for missions and stories
Auth	Google OAuth 2.0 → server-verified JWT (7-day expiry)

🔗 GitHub Repository: github.com/Krish-Panchani/ai-playground

Demo

🔗 Live Demo: ai-playground-krish.vercel.app

Screenshots

Home - Choose Your Mode

Once signed in, this is where every session starts. Three animated mode cards, each with an illustration and a one-line pitch. No tutorials to sit through, no settings to configure - just pick a mode and start drawing.

Skill Level Selection - Before a Creative Quest

Before starting a Creative Quest adventure, you choose your skill level. This isn't just a label - it directly controls how much time you get per mission. Beginners get 4 minutes per chapter; Expert players get 2.5 minutes. The AI also uses this context when generating missions, so a Beginner won't get asked to draw something highly detailed in a short window.

Creative Quest - Mission in Progress

This is the core loop. The AI-generated mission sits at the top - something like "Draw a phoenix rising from golden flames." The countdown timer is running. The drawing canvas is in the center with a full toolbar: pen, marker, spray, shapes, eraser, color picker, brush size, and opacity slider. At the bottom a progress bar shows which chapter you're on out of 10. The whole thing is one focused screen - no distractions.

Mission Results - AI Scores Your Drawing

After submitting, Gemini's vision model analyzes the actual pixel content of your drawing against the mission prompt. You get three individual scores - accuracy, creativity, and effort - plus a short written comment that Gemini generates specifically about what you drew. The weighted final score and XP earned appear below. Then the next mission loads automatically.

Artful Stories - Drawing Becomes a Story

In Stories mode, each drawing you make becomes the next chapter in an illustrated narrative. You draw something, Gemini writes a short story continuation based on what it sees in your drawing, and the full text appears alongside your art. Then you draw the next chapter. The complete illustrated story - all drawings and all Gemini-written text - is saved to your profile and visible in the gallery.

Player Profile

Your profile aggregates everything: total XP, current level (XP ÷ 100), global rank on the leaderboard, total drawings submitted, and adventures completed across all three modes. The XP bar shows exactly how far you are to the next level. Below that, a recent activity section shows your last few sessions at a glance.

Art Gallery - Community Drawings

All drawings from all players - Creative Quest missions, Guesswork submissions, and Story illustrations - show up here in one feed. It turns a solo game into a shared experience. Seeing someone else's interpretation of the same mission prompt you just struggled with is consistently entertaining, occasionally impressive, and sometimes very funny.

Leaderboard - Top Players by XP

The top 20 players ranked by total XP. Since XP accumulates from all three modes, it rewards players who explore the full game - not just the one who grinds a single mode. In the original version this was a hardcoded array in the component. Now every row is a real player from MongoDB.

The Comeback Story

Where this project started

This began at a hackathon. The idea was straightforward: what if Gemini could score your drawings and make the experience feel like a game? I had a weekend, a Gemini API key, some coffee, and an idea I thought was worth building.

The hackathon demo worked. You could draw something, submit it, and Gemini would respond. People who watched it laughed, engaged with it, thought it was fun. Then the weekend ended, I pushed the final commit, and the repo sat there untouched for months.

It wasn't that the idea was bad - I still thought it was interesting. It was that the code had never been meant to last beyond the demo. I knew it. And every time I thought about picking it back up, opening the codebase felt like opening a drawer full of things you've been meaning to deal with.

The GitHub Finish-Up-A-Thon gave me the framing I needed: don't build something new, just finish what you started.

What the original version actually looked like

Watch the original version here: youtu.be/A24KG36sfW4

The video shows the before state better than I can describe it, but here's the technical picture:

Gemini API keys were exposed in the browser. All AI calls happened inside React helper files (src/helpers/). The API key lived in a .env variable that Vite bakes into the client JavaScript bundle. Anyone who opened the browser's DevTools and looked at the network requests - or just read the compiled JS - would have had full access to the key.

Firebase was wired directly on the client. Auth, Firestore reads, Storage uploads - all happening in React components with the Firebase client SDK. There was an architecture document that listed backend API routes, but every single one was marked 501 Not Implemented. The server layer existed only on paper.

Creative Quest was a single AI call. You'd draw something, it would fire one Gemini prompt, return a response, and that was it. Reload the page and everything was gone - no session, no saved score, no progress. The progress bar was always at 100% because there was no actual progress to track.

The leaderboard was hardcoded data. Not connected to a database, not real players. Just an array I wrote into the component to make the demo look more complete.

Errors crashed the experience. When Gemini returned a 503 "model overloaded" error - which happens during peak hours - the user would see a popup containing raw JSON: {"error":{"code":503,"status":"UNAVAILABLE","message":"The model is overloaded. Please try again later."}}. There was no retry, no fallback, no user-friendly message.

Mission timers were whatever Gemini decided. Sometimes Gemini returned a 45-second timer. You cannot finish a drawing of anything in 45 seconds. The timer wasn't normalized or validated on the server - it went straight to the countdown component.

The idea behind the project was solid. The implementation was a hackathon draft that had never moved past "it works in the demo."

What I fixed, refactored, and built from scratch

This is the full list of what changed during the Finish-Up-A-Thon sprint.

Restructured the entire project into a client-server monorepo.
The frontend (React) and backend (Express + MongoDB) now live in the same repo but are fully separate: the React app in the root, the API server in /server. The Vite dev server proxies /api requests to localhost:8080 so local development feels seamless, but in production they're distinct services.

Moved all Gemini API calls to the server.
Zero AI calls happen in the browser anymore. The frontend sends drawing data to the Express API, and the server calls Gemini, handles retries, and returns results. The API key never reaches the client. This also made it possible to add retries and fallbacks - neither of which is viable when the AI is called from the browser.

Replaced Firebase client auth with Google OAuth → JWT.
A player clicks "Sign in with Google," the Google OAuth popup opens, and the resulting ID token is sent to the server. The server verifies the token using google-auth-library, upserts the user in MongoDB, and returns a 7-day JWT. The client stores that JWT in localStorage and attaches it as a Bearer header on every API request. Nothing sensitive runs on the client side.

Rebuilt Creative Quest as a proper game loop with saved state.
Each playthrough creates a GameSession document in MongoDB. That document tracks:

Which world you're in (5 worlds, 10 chapters total)
Your current chapter number
The AI-generated mission for each chapter
Your score and XP for each completed chapter
Session status: active, completed, or abandoned

Missions are generated per-chapter, and the previous mission prompts from that session are passed back to Gemini each time - so it won't repeat a prompt you've already seen in the same run. That duplicate-mission prevention was a small detail that made a big difference in playtesting.

Fixed mission timers with server-side normalization.
Before a timer reaches the client, it goes through a normalization function that accounts for your chosen skill level and which chapter you're on:

Beginner: 240 seconds base
Intermediate: 210 seconds
Advanced: 180 seconds
Expert: 150 seconds
Later chapters get a small time bonus (up to 60s extra)
Hard clamp: no mission can be shorter than 2 minutes or longer than 6 minutes

No more 45-second missions.

Added retry logic and fallback missions for when Gemini is overloaded.
When Gemini returns a 503 or rate limit error, the server retries automatically - up to 4 times, with increasing delays (700ms → 1.4s → 2.8s → 5.6s). If all four attempts fail, the game doesn't crash. Instead, it pulls from a pool of 20 hand-curated fallback missions so the game loop continues uninterrupted. The player sees a small toast notification explaining the situation in plain language, not a JSON error.

Rewrote the XP and scoring system server-side.
XP is no longer a client-side variable that disappears on refresh. Every score is computed on the server using this formula:

Final Score = (Accuracy × 0.5) + (Creativity × 0.3) + (Effort × 0.2)
XP Earned = max(10, round(Final Score ÷ 2))
Level = floor(Total XP ÷ 100)

After submitting a drawing, the frontend calls the profile endpoint to refresh the user's XP, level, and rank - so the numbers you see on your profile always reflect what's actually stored in the database.

Replaced every alert() and raw JSON error with human-readable toast messages.
There is a formatError.js utility on the frontend that pattern-matches server error responses to player-friendly strings. Gemini overloaded? → "Our AI helper is busy right now. Please wait a minute and try again." Network failure? → "Couldn't connect to the server." Expired session? → "Please sign in again to continue." These are surfaced as dismissible toast notifications, never as popups or raw JSON.

Built all four product pages that were missing entirely.

Profile - aggregates XP, level, global rank, total drawings submitted, adventure history
Adventure Log - browse all past game sessions with their status and chapter-by-chapter results
Art Gallery - community feed of every drawing submitted across all three modes
Leaderboard - top 20 players by XP, updated in real time (actual database queries, not hardcoded)

Before vs. After - Side by Side

The biggest change was Creative Quest going from a one-shot demo to a real game loop:

	Original Version	Finished Version
Missions	Same static prompt each run	AI-generated, unique per chapter, no repeats
Timers	Whatever Gemini returned (45s possible)	Server-normalized by skill level (120–360s)
Scoring	Client-side, no persistence	Server-computed, saved to MongoDB
XP	Reset on page reload	Persistent, synced to player profile
Progress	Progress bar stuck at 100%	Real chapter count (2/10, 5/10, etc.)
AI failure	Hard crash, raw JSON popup	Retry → fallback mission → friendly toast
Auth	Firebase client SDK	Google OAuth + server-verified JWT
Leaderboard	Hardcoded array	Live database query, real players
Profile	Didn't exist	Full stats, history, rank, XP bar
API keys	Exposed in browser bundle	Server-side only, never sent to client
Error UX	`alert()` popups	react-hot-toast with plain-language messages

Key Code Changes

Scoring formula - moved from client to server:

// server/src/services/progression.service.js
export const computeCreativeQuestScore = ({ accuracy, creativity, effort }) => {
  const finalScore =
    (Number(accuracy) || 0) * 0.5 +
    (Number(creativity) || 0) * 0.3 +
    (Number(effort) || 0) * 0.2;

  const xpEarned = Math.max(10, Math.round(finalScore / 2));
  return { finalScore: Math.round(finalScore), xpEarned };
};

export const levelFromXp = (xp) => Math.floor((Number(xp) || 0) / 100);

Mission timer normalization - so no mission is ever unfair:

// server/src/utils/mission-timer.js
const SKILL_SECONDS = { Beginner: 240, Intermediate: 210, Advanced: 180, Expert: 150 };

export const normalizeMissionTimer = (requestedSeconds, skillLevel = "Beginner", chapterNumber = 1) => {
  const skillBase = SKILL_SECONDS[skillLevel] ?? 210;
  const chapterBonus = Math.min(60, (chapterNumber - 1) * 5);
  const parsed = Number(requestedSeconds);
  if (!Number.isFinite(parsed) || parsed <= 0) return skillBase + chapterBonus;
  return Math.max(120, Math.min(360, Math.round(parsed)));
};

Gemini retry - because the AI is occasionally busy:

// server/src/utils/gemini-retry.js
export const isGeminiRetryableError = (error) => {
  const status = error?.status ?? error?.error?.status;
  if ([429, 500, 502, 503, 504].includes(Number(status))) return true;
  const text = [error?.message, error?.error?.message, JSON.stringify(error)]
    .filter(Boolean).join(" ").toLowerCase();
  return text.includes("unavailable") || text.includes("high demand") ||
         text.includes("resource_exhausted") || text.includes("overloaded");
};

The Gemini SDK surfaces errors in different shapes depending on whether the failure came from the HTTP layer, the SDK's internal parsing, or the response body. Checking all three forms means the retry logic actually catches all the cases - not just the obvious one.

Player-friendly error messages - because users don't speak HTTP:

// src/lib/formatError.js
const errorPatterns = [
  {
    test: (text) => text.includes("high demand") || text.includes("503") || text.includes("resource_exhausted"),
    message: "Our AI helper is busy right now. Please wait a minute and try again.",
  },
  {
    test: (text) => text.includes("network") || text.includes("failed to fetch"),
    message: "Couldn't connect to the server. Check your connection and try again.",
  },
  {
    test: (text) => text.includes("401") || text.includes("authentication"),
    message: "Please sign in again to continue.",
  },
];

My Experience with GitHub Copilot

I want to give an honest account of this because the Finish-Up-A-Thon specifically asks about it, and "Copilot wrote my whole project" is both untrue and uninteresting.

Here's the honest version: Copilot handled the work that usually kills side projects - the tedious, repetitive, unglamorous 30% - so I could spend my energy on the parts that actually required judgment.

The real places where Copilot made a difference

Building out the backend API layer.
Once I set up the first authenticated Express route - requireAuth middleware, controller function, Mongoose query, standardized JSON response - Copilot saw the pattern and replicated it consistently for every subsequent route I needed to add. Profile aggregation, gallery queries, adventure log, leaderboard, game session endpoints - the structural scaffolding was right on the first suggestion most of the time. I wasn't typing the same boilerplate eight times; I was reviewing and adjusting what Copilot generated.

This alone probably saved three or four hours. Route scaffolding isn't hard, but doing it eight times in a row while trying to maintain consistency across middleware, error handling, and response shapes is exactly the kind of work that makes you lose momentum.

Migrating dozens of Firebase calls to the new API client.
The migration from Firebase client SDK calls to a centralized src/lib/api.js with JWT headers touched almost every page and component in the frontend. Every component that previously called Firebase directly needed to be updated to use the new API client pattern instead.

Copilot saw the pattern in each file - the old Firebase call, the new api.get() or api.post() shape - and predicted the correct replacement. What would have been a full evening of tedious find-and-replace work took a couple of hours, and the replacements were consistent in a way that's easy to miss when you're doing it manually and getting tired.

Writing the error-handling layer.
The formatError.js utility and all the react-hot-toast wrappers are the kind of work that's easy to skip because it's not a feature - it just makes the app not feel broken. Copilot drafted the initial pattern-match structure and the toast helper function signatures. I adjusted the tone of each message to fit the game's friendly voice, but the structure and the enumeration of error cases were Copilot's first suggestion.

Without Copilot, this section would probably have been "good enough" error handling - maybe one or two cases covered, everything else falling through to a generic message. With it, every meaningful failure scenario has a specific, useful, plain-language response.

Keeping the UI components visually consistent.
AI Playground has a dark-themed game UI with gradient headings, card components, stat blocks, progress bars, animated overlays, and a drawing HUD. All of these needed to look like they belonged together - same border radius, same color tokens, same responsive breakpoints. Copilot kept the Tailwind class composition consistent across MissionHUD, GameLoadingOverlay, profile stat cards, and the drawing toolbar as I built each piece. Small thing, but the visual coherence of the final UI is partly a result of Copilot staying consistent when I didn't have time to constantly cross-reference.

The moment Copilot actually surprised me

The most valuable thing Copilot did wasn't fast typing - it was catching an edge case I would have missed.

I was debugging the Gemini retry logic. My first version just checked if (error.status === 503), which felt like the right thing to do. The Gemini API returns 503 when overloaded, so catch that status and retry.

When I described the problem to Copilot - "missions are silently failing when Gemini is overloaded, but my catch block doesn't seem to be catching everything" - its response wasn't just "add more status codes." It explained that the Gemini SDK surfaces failures in at least three different shapes depending on where in the call stack the error originates:

Sometimes you get error.status (HTTP layer error)
Sometimes you get error.error.status (SDK wrapper error)
Sometimes neither field is set and the only signal is in the serialized error body as keywords like "resource_exhausted" or "overloaded"

So the final retry check looks at all three forms. Without that, I would have had a retry policy that looked correct in code review and silently missed half of the actual failure cases. I would have found it eventually - probably during a playtest at midnight - but I didn't have to.

Where I made the decisions myself

Copilot does not know your game. It doesn't know when a failed drawing should trigger a retry versus abort the mission. It doesn't know what XP formula feels fair to a casual player. It doesn't know that passing previous mission prompts back to Gemini is how you prevent repeated missions. It doesn't know when "the AI is overloaded" should mean "try again" versus "load a fallback and keep playing."

These are judgment calls - about user experience, game design, player frustration curves, and what a good session arc feels like. I made all of those.

The auth security decisions were also entirely manual. What runs server-side versus client-side, how the JWT payload is structured, what requireAuth must reject and why - none of that was delegated. Getting that wrong has real consequences.

The most productive workflow I found with Copilot was giving it file-level tasks with enough context to be specific: "Add a GET /users/me/profile endpoint that aggregates the user document with drawing count, session history, and leaderboard rank from these three MongoDB models." That's specific enough for Copilot to generate something structurally useful. Vague prompts return vague code.

Copilot as a finishing partner, not a code generator

The Finish-Up-A-Thon challenge is about reviving things you started. The biggest obstacle to reviving a stalled project usually isn't lack of ideas - it's the activation energy required to push through the unglamorous work before you get to the part that's fun again.

Copilot lowered that activation energy. It handled the boilerplate, kept the patterns consistent, and caught at least one real bug I would have shipped. I handled the architecture, the game design, the security decisions, and the user experience. Both parts were necessary to finish the project.

What This Sprint Taught Me

One complete loop beats ten half-finished features. The thing that made AI Playground feel done was having a complete, working arc in Creative Quest: sign in → generate mission → draw → submit → Gemini scores → XP saves → profile updates. Every page I built after that felt like bonus. A complete loop is what separates "cool demo" from "actual game."

Move AI to the server early. The moment Gemini left the React layer, everything else became unlocked - retries, fallbacks, auth gating, rate limiting, security. None of that is possible when the AI runs client-side. This was the single structural change that made the most things possible.

Good error messages are part of the product, not an afterthought. Players don't read HTTP status codes. Someone who spent two minutes carefully drawing a dragon should not be told {"error":{"code":503,"status":"UNAVAILABLE"}}. "Our AI helper is busy right now, we loaded a backup mission for you" is a completely different experience. The underlying failure is identical. The player's perception of the app is not.

The "before and after" framing actually works. The Finish-Up-A-Thon mandate - show us the arc - forced me to close loops instead of opening new ones. Every time I wanted to add a new feature, I asked whether it contributed to a clear completion story. If not, it went on a list for later. That constraint made the sprint productive in a way that open-ended "work on your project" never would have been.

Run It Locally

git clone https://github.com/Krish-Panchani/ai-playground.git
cd ai-playground

# Install frontend and backend dependencies
npm install
npm --prefix server install

# Set up environment files
cp .env-example .env
cp server/.env-example server/.env

Frontend .env (project root):

VITE_GOOGLE_CLIENT_ID=your-google-oauth-web-client-id.apps.googleusercontent.com

Backend server/.env:

PORT=8080
NODE_ENV=development
CLIENT_ORIGIN=http://localhost:5173
MONGODB_URI=mongodb+srv://<user>:<pass>@<cluster>/<db>?retryWrites=true&w=majority
MONGODB_DB_NAME=ai_playground
GEMINI_API_KEY=your-gemini-api-key
GEMINI_VISION_MODEL=gemini-2.5-flash
GOOGLE_CLIENT_ID=your-google-oauth-web-client-id.apps.googleusercontent.com
JWT_SECRET=any-long-random-string-at-least-32-characters
JWT_EXPIRES_IN=7d

# Start backend (terminal 1)
npm run server:dev    # runs on http://localhost:8080

# Start frontend (terminal 2)
npm run dev           # runs on http://localhost:5173

You can run without a GEMINI_API_KEY - the server will return mock AI responses so you can test the full game loop, profile, leaderboard, and gallery without needing a Google Cloud account.

Conclusion

AI Playground started as a weekend hackathon idea that actually resonated with people - an AI that looks at your drawing and reacts to it specifically, not generically. The problem was that the prototype was never meant to survive past the demo. API keys in the browser, a hardcoded leaderboard, a game mode that was really just a single API call, errors surfacing as raw JSON popups. It worked in the moment and fell apart the second the moment ended.

The GitHub Finish-Up-A-Thon gave me a reason to go back and do it properly. A real backend, a real auth flow, a real game loop with persistent state, proper error handling, and all four product pages that were missing. The core idea - drawing + AI feedback = fun - turned out to hold up. It just needed the infrastructure around it to make it feel like a product instead of a proof of concept.

GitHub Copilot was the right tool for this specific challenge. Finishing a stalled project isn't about having more ideas - it's about grinding through the repetitive work until the interesting parts are reachable again. Copilot handled the repetitive parts (route scaffolding, migration boilerplate, UI consistency, error utilities) and caught at least one real edge case in the Gemini retry logic that I would have shipped as a silent bug. That freed me up to focus on the parts that actually required judgment: the game design, the auth security, the user experience, and the AI integration details.

If you've got a project sitting in a GitHub tab that you keep meaning to come back to - this challenge is worth trying. The "before and after" framing forces you to actually finish something, and finishing something that already has momentum is a very different experience than starting from scratch.

Built with GitHub Copilot for the GitHub Finish-Up-A-Thon Challenge.
Repository: github.com/Krish-Panchani/ai-playground

DEV Community: Krish Panchani

Reviving My AI-Powered Draw and Learn Game - What If an AI Could Look at Your Drawing and Actually Respond?

What I Built

Demo

Screenshots

The Comeback Story

Where this project started

What the original version actually looked like

What I fixed, refactored, and built from scratch

Before vs. After - Side by Side

Key Code Changes

My Experience with GitHub Copilot

The real places where Copilot made a difference

The moment Copilot actually surprised me

Where I made the decisions myself

Copilot as a finishing partner, not a code generator

What This Sprint Taught Me

Run It Locally

Conclusion

Growing as a Developer Through Hacktoberfest 2023

Intro

Highs and Lows

Growth

Conclusion