DEV Community: Mongoo (Mungunshagai Tum)

Amibuddy Live homework agent app - Gemini Live agent hackathon

Mongoo (Mungunshagai Tum) — Mon, 16 Mar 2026 22:56:57 +0000

My Daughter's Doodles Came to Life: Building an AI Tutor with Gemini & SAM 2

GeminiLiveAgentChallenge.

By a Mom Developer

The Spark: First Grade Jitters

My daughter just started first grade here in Japan. It’s a huge milestone—the shiny new randoseru (backpack), the excitement of new friends, but also the daunting reality of homework.

Watching her sit at her desk, I noticed two things: she loved drawing, but she found studying to be a chore. She would doodle monsters and princesses on the corners of her math worksheets instead of solving the problems.

That’s when it hit me. What if those doodles could help her study?

As a developer (and a mom), I decided to build a solution. I wanted to bridge the gap between her creativity and her education. I wanted to bring her drawings to life, not just as animations, but as intelligent, interactive study partners.

Introducing AmiBuddy

AmiBuddy is the result of that late-night "mom coding." It’s an AI-powered application that transforms static children's drawings into "Living AI Agents."

Demo & Links

📺 Watch the Demo:

https://youtu.be/PFbhIW4gZxs

🚀 Try the App: AmiBuddy Live Site

Architecture Overview

I built AmiBuddy using a hybrid architecture: a React Native (Expo) frontend for the best mobile experience, and a Python (FastAPI) backend on Cloud Run for heavy AI processing.

graph LR
    subgraph Frontend ["Frontend Layer (React Native / Expo)"]
        direction TB
        MobileClient["📱 Mobile App"]
        WebClient["💻 Web App"]
    end

    subgraph External_AI ["External AI Services"]
        direction TB
        Gemini["✨ Gemini 2.5 Flash (Vision/Reasoning)"]
        ElevenLabs["🗣️ ElevenLabs (TTS)"]
    end

    subgraph Cloud_Run ["Backend (Cloud Run)"]
        direction TB
        OrchestratorAPI["🚀 Orchestrator API"]
        SAM2["🧩 SAM 2"]
        RiggingAgent["🦴 Rigging Agent"]
    end

    Frontend -.->|Direct API Call| Gemini
    Frontend -.->|Direct API Call| ElevenLabs
    Frontend ==>|Upload Image| OrchestratorAPI

    OrchestratorAPI --> SAM2
    OrchestratorAPI --> RiggingAgent
    RiggingAgent -.->|Query Structure| Gemini

Technical Deep Dive: The "Living Agent" Pipeline

The core magic happens in the backend, where we turn a static image into a rigged character.

Phase 1: Structural Understanding (Rigging Agent)

Before we can animate a drawing, we need to understand what it is. Where is the head? Where are the hands?

I used Google Gemini 2.5 Flash for this. Unlike traditional object detection models that need training, Gemini has incredible "zero-shot" understanding. I simply prompt it to find the joints and body parts.

Code Highlight: The Rigging Prompt (agents/rigging.py)

I specifically ask Gemini to be "generous" with bounding boxes to ensure we don't crop off limbs.

        prompt = """
        Analyze this character drawing for skeletal animation.
        Provide a JSON object with the following structure:
        {
            "joints": {
                "neck": [x, y],
                "left_shoulder": [x, y],
                ...
            },
            "parts": {
                "head": [ymin, xmin, ymax, xmax],
                "left_arm": [ymin, xmin, ymax, xmax],
                ...
            }
        }
        IMPORTANT: For "parts", ensure the bounding boxes are GENEROUS and INCLUSIVE. 
        Include the ENTIRE limb, even if it overlaps with the body slightly. 
        Do not crop off hands, feet, or joints.
        RETURN ONLY JSON.
        """

Phase 2: Surgical Extraction (Segmentation Agent)

Once Gemini tells us where the parts are (roughly), we need to cut them out with pixel-perfect precision.

For this, I used Meta SAM 2 (Segment Anything Model 2). The magic here is the hand-off: we use the bounding boxes from Gemini as the "prompt" for SAM 2.

Code Highlight: From Gemini Box to SAM 2 Mask (agents/segmentation.py)

        for part_name, box in parts.items():
            # Convert normalized coordinates from Gemini to pixels
            ymin, xmin, ymax, xmax = box
            input_box = np.array([xmin * width, ymin * height, xmax * width, ymax * height])

            # Add padding to ensure SAM 2 gets the whole context
            padding = 10 
            input_box[0] = max(0, input_box[0] - padding)
            ...

            # Ask SAM 2 to predict the mask for this box
            masks, scores, _ = self.predictor.predict(
                point_coords=None,
                point_labels=None,
                box=input_box[None, :],
                multimask_output=False,
            )

            # The result is a perfect alpha mask for that body part!

Phase 3: Life & Voice

Finally, the frontend (LiveBuddy component) takes these cut-out parts and animates them using simple sine waves (math-based animation) to simulate breathing and floating.

The Study Buddy Agents

Once the character is "alive," it needs to be "smart." I implemented two more agents on the Frontend side.

Phase 4: Homework Analysis (Gemini Vision Agent)

When the child uploads a homework sheet, it’s not enough to just OCR the text. The AI needs to understand the intent and difficulty to explain it to a child.

Code Highlight: Homework Analysis (src/services/geminiService.ts)

I use a structured JSON prompt with Gemini Vision to extract metadata about the homework. Note how I switch the prompt language based on the user's locale!

    const prompt = locale === 'en'
      ? `Analyze this homework image for a young student.
         Provide a JSON response with:
         - description: A short, encouraging summary of what the homework is about (in simple English).
         - topics: A list of key topics found (e.g., "Addition", "Kanji").
         - difficulty: estimated difficulty ("Easy", "Medium", "Hard").
         Keep the tone friendly and encouraging.
         Return ONLY valid JSON.`
      : `この宿題の画像を子供向けに分析してください。
         以下のJSON形式で返してください：
         - description: 宿題の内容についての短く励ますような説明（やさしい日本語で）。
         - topics: 見つかった主要なトピックのリスト（例：「たしざん」、「かんじ」）。
         ...`;

    const result = await model.generateContent([
      prompt,
      { inlineData: { mimeType, data: base64 } } // Multimodal Input
    ]);

Phase 5: Voice Conversation (Speech-to-Text & TTS)

For a 6-year-old, typing is a barrier. AmiBuddy is voice-first.
I used Gemini 2.5 Flash again for Speech-to-Text (transcription) because it handles mixed languages (Japanese/English) better than traditional STT engines, and ElevenLabs for the character's voice.

Code Highlight: Context-Aware Chat (src/services/voiceConversationService.ts)

The AI needs to know what homework we are looking at to answer questions correctly.

    // Build conversation context
    const contextPrompt = locale === 'en'
      ? `You are a friendly teacher helping a child with homework.
         Homework context: ${homeworkContext}

         Answer the child's question simply and kindly.
         Keep answers short (2-3 sentences).`
      : `あなたは子供の宿題を手伝う優しい先生です。
         宿題の内容: ${homeworkContext}

         子供の質問に対して、わかりやすく、優しく答えてください。
         答えは2-3文で簡潔にしてください。`;

Workflow Summary

sequenceDiagram
    participant User as 👤 User
    participant API as 🚀 Backend
    participant Gemini as ✨ Gemini
    participant SAM2 as 🧩 SAM 2
    participant Storage as ☁️ Firebase
    participant Eleven as 🗣️ ElevenLabs

    Note over User, Storage: 1. Character Creation
    User->>API: Upload Drawing
    API->>Gemini: "Rig this character"
    Gemini-->>API: Skeleton Data
    loop Each Part
        API->>SAM2: "Segment this part"
        SAM2-->>API: Part Mask
    end
    API-->>User: Living Character!

    Note over User, Eleven: 2. Study Session
    User->>Gemini: Upload Homework Image
    Gemini-->>User: "This looks like Math!"
    User->>User: Ask by Voice: "How do I do this?"
    User->>Gemini: Transcribe Voice -> LLM Answer
    Gemini-->>User: "Try counting backwards!"
    User->>Eleven: Speak Response
    Eleven-->>User: Character speaks!

Why This Matters

For my daughter, AmiBuddy turned homework from a struggle into play. She’s excited to show her "buddy" what she learned today.

For me, it’s a testament to how AI can be personal. We often talk about AI in terms of productivity or business, but its potential to connect with us emotionally—to nurture a child's curiosity—is what excites me the most.

I’m open-sourcing parts of this journey to help other parents and developers build their own magic. Because sometimes, the best way to predict the future is to invent it for your kids.

Building Frankenstein Laboratory: Automating Legacy PHP Migration with Kiro IDE

Mongoo (Mungunshagai Tum) — Fri, 05 Dec 2025 20:32:32 +0000

The Problem That Haunts Japanese Companies

After two years working at a traditional Japanese company, I've witnessed a crisis that affects countless organizations across Japan: legacy PHP projects that refuse to die.

Every year, teams struggle with migration projects. Some critical systems still run on PHP 5 - a version that reached end-of-life in 2018. Developers spend months manually rewriting code, risking bugs with every line. Business stakeholders grow frustrated as new features take a backseat to maintenance. Technical debt compounds year after year.

This isn't just my company's problem - it's an industry-wide crisis. Japanese companies, known for their stability and long-term thinking, often maintain systems for decades. But that stability becomes a prison when the technology underneath becomes obsolete.

During the Kiro Halloween Hackathon, I decided to tackle this head-on. The result? Frankenstein Laboratory - a tool that brings dead PHP code back to life by automating migration to modern Python.

What I Built: Two Projects in One

The solution has two components:

1. PHP Migration Tool

A web application that automates PHP-to-Python migration:

Clone any GitHub repository
Analyze PHP code structure automatically
Generate production-ready Python/FastAPI code
Create OpenAPI specifications
Generate property-based tests
Download everything as a deployable package

2. Strangler Studio Demo

A working reference implementation of the Strangler Fig pattern:

Legacy PHP application (student request system)
Modern Python API (FastAPI equivalent)
Nginx gateway for intelligent routing
Zero-downtime incremental migration
Comprehensive test suite

The Technical Architecture

Migration Tool Stack

Frontend (React + Vite):

// Four-stage wizard interface
const stages = [
  'Source',    // GitHub URL or ZIP upload
  'Analyze',   // PHP code analysis
  'Transform', // Python code generation
  'Extract'    // Download generated code
];

The UI features a Frankenstein laboratory theme with animated logo, electric effects, and cinematic design. Six simultaneous CSS animations create a living, breathing interface:

.frankenstein-logo {
  animation: 
    float-logo 6s ease-in-out infinite,
    glow-pulse-logo 3s ease-in-out infinite,
    electric-spark 8s ease-in-out infinite;
}

Backend (FastAPI + Python):

The backend handles the heavy lifting:

@app.post("/api/clone-github")
async def clone_github_repo(request: GitHubRepoRequest):
    """Clone and analyze GitHub repository"""
    # Shallow clone for performance
    git.Repo.clone_from(
        repo_url,
        extract_dir,
        branch=request.branch,
        depth=1,  # Only latest commit
        single_branch=True
    )

    # Analyze PHP code
    analyzer = PHPAnalyzer()
    analysis = analyzer.analyze_directory(extract_dir)

    return {
        "routes": analysis['routes'],
        "models": analysis['models'],
        "dependencies": analysis['dependencies']
    }

The analyzer uses AST parsing to understand PHP code structure, identifying:

Routes and controllers
Data models and relationships
Database queries
Dependencies and imports

Then it generates Python code with:

FastAPI endpoints with proper type hints
Pydantic models for data validation
OpenAPI specifications
Property-based tests using Hypothesis

Strangler Pattern Implementation

The demo showcases safe, incremental migration:

Docker Compose Orchestration:

services:
  gateway:
    image: nginx:alpine
    ports: ["8888:80"]
    depends_on: [legacy-php, new-api]

  legacy-php:
    build: ./legacy-php
    ports: ["8080:80"]

  new-api:
    build: ./new-api
    ports: ["8000:8000"]

Nginx Gateway Routing:

# Route /api/* to new Python API
location /api/ {
    proxy_pass http://new-api:8000;
}

# Route everything else to legacy PHP
location / {
    proxy_pass http://legacy-php:80;
}

This allows incremental migration - move one endpoint at a time from PHP to Python while both systems run simultaneously.

Property-Based Testing: The Secret Weapon

Traditional unit tests check specific examples. Property-based tests verify universal properties across ALL inputs.

Example: Gateway Routing Correctness

#!/bin/bash
# Property 1: Gateway routes requests to correct backend

for i in {1..100}; do
    # Generate random path
    if [ $((RANDOM % 2)) -eq 0 ]; then
        PATH="/api/requests"
        EXPECTED="new-api"
    else
        PATH="/requests"
        EXPECTED="legacy-php"
    fi

    # Test routing
    curl -s "http://localhost:8888$PATH"
    # Verify correct backend handled request
done

Python Property Test with Hypothesis:

from hypothesis import given, strategies as st

@given(st.integers(min_value=1, max_value=7))
def test_api_returns_matching_id(request_id):
    """For ANY valid ID, returned object should have matching ID"""
    response = client.get(f"/requests/{request_id}")
    assert response.status_code == 200
    assert response.json()["id"] == request_id

This runs 100+ iterations with random inputs, catching edge cases you'd never think to test manually.

Working with Kiro: The Development Process

Spec-Driven Development

For the Strangler Studio demo, I used Kiro's spec-driven approach:

1. Requirements (EARS format):

WHEN a request is made to /api/requests 
THEN the Gateway SHALL route it to the new Python API

WHEN a request is made to /requests 
THEN the Gateway SHALL route it to the legacy PHP application

2. Design with Correctness Properties:

Property 1: Gateway routing correctness
For any request path, the gateway routes to the correct 
backend based on the path prefix.

3. Task Breakdown:

- [ ] 1. Set up Docker Compose with three services
- [ ] 2. Implement Nginx gateway routing
- [ ] 3. Create legacy PHP application
- [ ] 4. Create new Python API
- [ ] 5. Write property-based tests

Kiro executed tasks one by one, automatically referencing requirements and implementing correctness properties as tests.

Vibe Coding for Rapid Prototyping

For the migration tool UI, I used conversational iteration:

Me: "Create a Frankenstein laboratory theme with electric blue and toxic green. Add animations: floating logo, electric glow pulse, and lightning sparks."

Kiro: Generates 200+ lines of CSS with six simultaneous animations, GPU-accelerated properties, and hover effects

This flexibility was perfect for UI polish and rapid feature additions.

Steering Documents: Project-Specific Intelligence

I created four steering files in .kiro/steering/ that were automatically included in every conversation:

testing-standards.md:

- Use Hypothesis for property-based testing
- Minimum 100 iterations per test
- Tag format: # Feature: {name}, Property {n}: {description}
- Test patterns: invariants, round-trips, idempotence

halloween-theme.md:

- Electric Blue: #00d4ff
- Toxic Green: #76ff03
- Font: Cinzel (headings), Inter (body)
- No cheap emojis or generic Halloween clichés

This transformed Kiro from a general assistant into a domain expert for my project. Every response was consistent with project conventions without me repeating myself.

Technical Challenges and Solutions

Challenge 1: Docker Port Conflicts

Problem: Port 80 already allocated on host system.

Solution: Changed gateway to port 8888, updated all documentation. Created PORT_CHANGES_SUMMARY.md to track the change.

Challenge 2: Git Not Found in Docker

Problem: Backend container couldn't clone GitHub repositories.

Solution: Updated Dockerfile to install git:

RUN apt-get update && apt-get install -y git
ENV GIT_PYTHON_REFRESH=quiet
ENV GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git

Challenge 3: Vercel Serverless Limitations

Problem: 10-second timeout on free tier fails for large repos.

Solution: Implemented shallow cloning (depth=1), optimized analysis, documented upgrade path to Pro tier ($20/month for 60-second timeout).

Challenge 4: Logo Not Showing After Deployment

Problem: Added logo after Docker build, so it wasn't in the container.

Solution: Created rebuild scripts and documentation explaining Docker's immutable nature. Containers capture files at build time - changes require rebuilds.

The Results

What I Accomplished:

Complete end-to-end migration tool
Working Strangler pattern demo
Property-based test suite
15+ documentation files
Production-ready code quality
Deployed to Vercel in 15 minutes

Code Quality Metrics:

Type safety throughout (Pydantic models, TypeScript)
Comprehensive error handling
Security headers and CORS configuration
100+ property-based test iterations
OpenAPI contract validation

Development Speed:
What would normally take weeks took days, thanks to Kiro's assistance.

Key Learnings

1. Property-Based Testing is Powerful

Hypothesis finds edge cases you'd never think to test. It requires different thinking (properties vs examples) but catches more bugs.

2. Spec-Driven Development Ensures Quality

Clear requirements prevent misunderstandings. Correctness properties ensure quality. Task-by-task execution is systematic.

3. Steering Documents are Game-Changers

They transform Kiro from a general assistant into a domain expert. Every response follows project conventions automatically.

4. The Strangler Pattern Works

Incremental migration is safer than big-bang rewrites. The gateway pattern allows gradual transition with zero downtime.

5. Documentation is a Feature

Good docs are as important as good code. Users need multiple entry points: quick start, detailed guide, troubleshooting, reference.

Real-World Impact

This solves a real problem for Japanese companies (and companies worldwide) trapped by legacy PHP. The tool provides:

Automated analysis - No more manual code review
Intelligent generation - Production-ready Python code
Safe migration - Strangler pattern with zero downtime
Comprehensive testing - Property-based tests ensure correctness

Companies can now modernize without fear. No more months-long rewrite projects. No more choosing between maintenance and innovation.

What's Next

Immediate improvements:

Support for Laravel, Symfony, WordPress frameworks
Database schema extraction and migration
AI-powered code optimization
Multi-language support (Ruby, Java, Node.js to Python)

Long-term vision:

Japanese language UI and documentation
Integration with Japanese cloud providers
Enterprise features (SSO, audit logging)
Open-source core with plugin ecosystem

The ultimate goal: Make legacy PHP migration so easy that Japanese companies can modernize without fear.

Try It Yourself

GitHub Repository: https://github.com/mongonsh/Strangler-Studio-with-PHP-Migration-Tool

Deploy to Vercel: Follow the guides in the repo - you can have it running in 15 minutes.

Tech Stack:

Frontend: React, Vite, TailwindCSS
Backend: FastAPI, Pydantic, GitPython, Hypothesis
Demo: PHP 8, FastAPI, Nginx, Docker
Testing: Pytest, Hypothesis, Bash scripts
Deployment: Vercel (serverless)

Conclusion

This hackathon taught me that the future of software development isn't human OR AI - it's human AND AI working together.

I provided the vision: solve the legacy PHP problem plaguing Japanese companies.

Kiro provided the execution: production-ready code, comprehensive tests, beautiful UI, thorough documentation.

Together, we built something that would have taken weeks in just days.

Legacy PHP doesn't have to be a death sentence. With the right tools, even the oldest code can be reborn.

"It's alive!" ⚡

*Built with Kiro IDE during Kiroween *

#kiro #hackathon #php #python #fastapi #migration #legacy #strangler-pattern #property-based-testing #ai-assisted-development

What do you think? Would you use a tool like this for your legacy projects? Drop a comment below!