DEV Community

Cover image for [Google AI Studio Multimodal Challenge] Whispurr the ghost diner
cutieyunny-tech
cutieyunny-tech

Posted on

[Google AI Studio Multimodal Challenge] Whispurr the ghost diner

#devchallenge #googleaichallenge #ai #gemini

This is a submission for the Google AI Studio Multimodal Challenge*

What I Built

I built "Whispurr: The Ghost Diner," an interactive mini-game that leverages the multimodal capabilities of the Gemini API. The project aims to create a dynamic narrative experience where the story changes based on the player's actions. The player controls a ghost that must navigate between different zones, with each zone triggering a unique AI response.

Demo

https://codepen.io/nad-Yunny/pen/wBKRxmW

<!DOCTYPE html>









Ghost Diner: Cabaran Multimodal AI



<br>
@import url(&#39;<a href="https://fonts.googleapis.com/css2?family=Inter:wght@400;700&amp;display=swap&amp;#x27;"&gt;https://fonts.googleapis.com/css2?family=Inter:wght@400;700&amp;amp;display=swap&amp;#39;&lt;/a&gt;);&lt;br>
body {<br>
font-family: &#39;Inter&#39;, sans-serif;<br>
background-color: #0d1117;<br>
color: #c9d1d9;<br>
display: flex;<br>
justify-content: center;<br>
align-items: center;<br>
height: 100vh;<br>
margin: 0;<br>
overflow: hidden;<br>
flex-direction: column;<br>
}<br>
.container {<br>
display: flex;<br>
flex-direction: column;<br>
gap: 20px;<br>
align-items: center;<br>
padding: 20px;<br>
border-radius: 12px;<br>
background-color: #161b22;<br>
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);<br>
max-width: 90%;<br>
}<br>
canvas {<br>
border: 2px solid #2f363d;<br>
border-radius: 8px;<br>
background-color: #010409;<br>
width: 100%;<br>
height: auto;<br>
max-width: 500px;<br>
max-height: 500px;<br>
}<br>
.message-box {<br>
width: 100%;<br>
max-width: 500px;<br>
height: 200px;<br>
background-color: #010409;<br>
border: 2px solid #2f363d;<br>
border-radius: 8px;<br>
padding: 15px;<br>
overflow-y: auto;<br>
display: flex;<br>
flex-direction: column-reverse;<br>
font-size: 14px;<br>
}<br>
.message-box p {<br>
margin: 5px 0;<br>
word-wrap: break-word;<br>
}<br>
.message-box .spinner-container {<br>
display: flex;<br>
justify-content: center;<br>
align-items: center;<br>
}<br>
.spinner {<br>
border: 4px solid #f3f3f3;<br>
border-top: 4px solid #3498db;<br>
border-radius: 50%;<br>
width: 30px;<br>
height: 30px;<br>
animation: spin 1s linear infinite;<br>
}<br>
@keyframes spin {<br>
0% { transform: rotate(0deg); }<br>
100% { transform: rotate(360deg); }<br>
}<br>
@media (min-width: 768px) {<br>
.container {<br>
flex-direction: row;<br>
justify-content: space-around;<br>
}<br>
}<br>






















<br>
// Mengkonfigurasi parameter permainan dan elemen DOM<br>
const canvas = document.getElementById(&#39;gameCanvas&#39;);<br>
const ctx = canvas.getContext(&#39;2d&#39;);<br>
const messageDisplay = document.getElementById(&#39;messageDisplay&#39;);<br>
const loadingSpinner = document.getElementById(&#39;loadingSpinner&#39;);</p>
<div class="highlight"><pre class="highlight plaintext"><code> // Konstanta untuk model Gemini API
const API_URL = https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-preview-05-20:generateContent?key=; 
// Status permainan
let player = { x: 50, y: 50, size: 20, speed: 5, color: '#4caf50' };
let keys = {};
let lastZone = 'none';
let timeInZone = 0;
let lastUpdateTime = Date.now();
const reactionTime = 10; // Masa dalam saat sebelum reaksi

// Definisi zon permainan
const playfulZone = { x: 50, y: 50, width: 100, height: 100, color: 'rgba(76, 175, 80, 0.3)', name: 'Playful Zone' };
const darkZone = { x: 350, y: 50, width: 100, height: 100, color: 'rgba(244, 67, 54, 0.3)', name: 'Dark Zone' };
const hintZone = { x: 200, y: 350, width: 100, height: 100, color: 'rgba(255, 193, 7, 0.3)', name: 'Hint Zone' };

// Simulasi data imej untuk dihantar ke Gemini. Ini adalah data Base64 dummy.
const simulatedImageData = {
    inlineData: {
        mimeType: "image/png",
        data: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=" // Dummy 1x1 pixel PNG
    }
};

// Fungsi untuk memaparkan mesej ke skrin
function showMessage(text, isUser = false) {
    const p = document.createElement('p');
    p.textContent = text;
    p.style.color = isUser ? '#60c7f2' : '#c9d1d9';
    messageDisplay.prepend(p);
    // Hadkan jumlah mesej
    while (messageDisplay.childElementCount &amp;gt; 20) {
        messageDisplay.removeChild(messageDisplay.lastChild);
    }
}

// Fungsi untuk menunjukkan atau menyembunyikan pemuat
function showSpinner(show) {
    loadingSpinner.classList.toggle('hidden', !show);
}

/**
 * Fungsi untuk memanggil Gemini API
 * @param {string} prompt Teks untuk dihantar kepada model
 * @param {object | null} imageData Data imej yang disimulasikan (pilihan)
 */
async function callGeminiAPI(prompt, imageData = null) {
    showMessage("Gemini sedang menjana...", true);
    showSpinner(true);

    try {
        const parts = [{ text: prompt }];
        if (imageData) {
            parts.push(imageData);
        }

        const payload = {
            contents: [{ parts: parts }],
            generationConfig: {
                responseMimeType: "text/plain"
            }
        };

        let response = null;
        for (let i = 0; i &amp;lt; 3; i++) {
            try {
                response = await fetch(API_URL, {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify(payload)
                });
                if (response.status !== 429) {
                    break;
                }
                await new Promise(resolve =&amp;gt; setTimeout(resolve, Math.pow(2, i) * 1000));
            } catch (error) {
                 await new Promise(resolve =&amp;gt; setTimeout(resolve, Math.pow(2, i) * 1000));
            }
        }

        if (!response || !response.ok) {
            throw new Error(`Ralat API: ${response?.statusText || 'Tiada respons'}`);
        }

        const result = await response.json();
        const text = result.candidates?.[0]?.content?.parts?.[0]?.text;
        if (text) {
            showMessage(text);
        } else {
            showMessage("Ralat: Kandungan respons tidak dijangka.");
        }

    } catch (error) {
        console.error('Ralat semasa memanggil Gemini API:', error);
        showMessage(`Ralat: ${error.message}`);
    } finally {
        showSpinner(false);
    }
}

// Fungsi untuk melukis elemen pada kanvas
function draw() {
    ctx.clearRect(0, 0, canvas.width, canvas.height);

    // Lukis zon
    ctx.fillStyle = playfulZone.color;
    ctx.fillRect(playfulZone.x, playfulZone.y, playfulZone.width, playfulZone.height);
    ctx.fillStyle = darkZone.color;
    ctx.fillRect(darkZone.x, darkZone.y, darkZone.width, darkZone.height);
    ctx.fillStyle = hintZone.color;
    ctx.fillRect(hintZone.x, hintZone.y, hintZone.width, hintZone.height);

    // Lukis label zon
    ctx.fillStyle = '#c9d1d9';
    ctx.font = '16px Inter';
    ctx.fillText(playfulZone.name, playfulZone.x, playfulZone.y - 5);
    ctx.fillText(darkZone.name, darkZone.x, darkZone.y - 5);
    ctx.fillText(hintZone.name, hintZone.x, hintZone.y - 5);

    // Lukis pemain
    ctx.fillStyle = player.color;
    ctx.beginPath();
    ctx.arc(player.x, player.y, player.size / 2, 0, Math.PI * 2);
    ctx.fill();
}

// Fungsi untuk mengemas kini keadaan permainan
function update() {
    // Mengemas kini kedudukan pemain
    if (keys['ArrowUp'] || keys['w']) player.y -= player.speed;
    if (keys['ArrowDown'] || keys['s']) player.y += player.speed;
    if (keys['ArrowLeft'] || keys['a']) player.x -= player.speed;
    if (keys['ArrowRight'] || keys['d']) player.x += player.speed;

    // Mengelakkan pemain keluar dari sempadan kanvas
    player.x = Math.max(0, Math.min(canvas.width, player.x));
    player.y = Math.max(0, Math.min(canvas.height, player.y));

    // Logik interaksi zon
    const currentTime = Date.now();
    const deltaTime = (currentTime - lastUpdateTime) / 1000;
    lastUpdateTime = currentTime;

    const currentZone = getZone();
    if (currentZone &amp;amp;&amp;amp; currentZone.name !== lastZone) {
        lastZone = currentZone.name;
        timeInZone = 0;
        showMessage(`Anda telah memasuki ${currentZone.name}.`);
        if (currentZone.name === playfulZone.name) {
            callGeminiAPI("Hasilkan naratif yang pendek dan ceria tentang zon bermain hantu.");
        } else if (currentZone.name === darkZone.name) {
            callGeminiAPI("Hasilkan naratif seram yang pendek, misteri, dan menakutkan tentang bisikan di kegelapan.");
        }
    } else if (!currentZone &amp;amp;&amp;amp; lastZone !== 'none') {
        lastZone = 'none';
        timeInZone = 0;
        showMessage("Anda telah meninggalkan zon.");
    } else if (currentZone) {
        timeInZone += deltaTime;
        if (timeInZone &amp;gt;= reactionTime) {
            timeInZone = 0; // Reset pemasa untuk mengelakkan mesej berterusan
            if (currentZone.name === playfulZone.name) {
                callGeminiAPI("Hasilkan reaksi pelanggan yang lucu dan sabar tentang hantu yang perlahan, mungkin dengan mengeluh.");
            } else if (currentZone.name === darkZone.name) {
                callGeminiAPI("Hasilkan petunjuk misteri dan menakutkan tentang laluan rahsia, atau benda yang hilang.");
            }
        }
    }
}

// Mendapatkan zon semasa di mana pemain berada
function getZone() {
    if (isColliding(player, playfulZone)) return playfulZone;
    if (isColliding(player, darkZone)) return darkZone;
    if (isColliding(player, hintZone)) return hintZone;
    return null;
}

// Fungsi untuk mengesan perlanggaran antara pemain dan zon
function isColliding(player, zone) {
    const playerLeft = player.x - player.size / 2;
    const playerRight = player.x + player.size / 2;
    const playerTop = player.y - player.size / 2;
    const playerBottom = player.y + player.size / 2;
    return playerRight &amp;gt; zone.x &amp;amp;&amp;amp; playerLeft &amp;lt; zone.x + zone.width &amp;amp;&amp;amp;
           playerBottom &amp;gt; zone.y &amp;amp;&amp;amp; playerTop &amp;lt; zone.y + zone.height;
}

// Logik klik kanvas untuk interaksi dengan zon petunjuk
canvas.addEventListener('click', (e) =&amp;gt; {
    const rect = canvas.getBoundingClientRect();
    const clickX = e.clientX - rect.left;
    const clickY = e.clientY - rect.top;

    const currentZone = getZone();

    // Dapatkan jarak antara klik dan pusat pemain
    const playerCenterX = player.x;
    const playerCenterY = player.y;
    const dx = clickX - playerCenterX;
    const dy = clickY - playerCenterY;
    const distance = Math.sqrt(dx * dx + dy * dy);

    // Jika klik berada dalam zon petunjuk
    if (currentZone &amp;amp;&amp;amp; currentZone.name === hintZone.name) {
        // Hantar prompt yang lebih spesifik berdasarkan kedudukan klik pemain
        const prompt = `Pemain telah klik di Zon Petunjuk. Hasilkan petunjuk naratif berdasarkan kedudukan pemain sekarang (${Math.round(player.x)}, ${Math.round(player.y)}) dan imej ini. Imej ini menunjukkan kunci yang diletakkan di atas meja.`;
        callGeminiAPI(prompt, simulatedImageData);
    } else if (currentZone) {
        showMessage(`Anda berada di ${currentZone.name}. Klik hanya berfungsi di Zon Petunjuk.`, true);
    } else {
        showMessage("Anda tidak berada di mana-mana zon. Sila bergerak ke zon dahulu.", true);
    }
});

// Mengendalikan input papan kekunci
window.addEventListener('keydown', (e) =&amp;gt; {
    keys[e.key] = true;
});

window.addEventListener('keyup', (e) =&amp;gt; {
    keys[e.key] = false;
});

// Loop utama permainan
function gameLoop() {
    update();
    draw();
    requestAnimationFrame(gameLoop);
}

// Memulakan permainan apabila tetingkap dimuatkan sepenuhnya
window.onload = function() {
    showMessage("Gunakan kekunci anak panah atau WASD untuk bergerak. Masuk ke zon untuk berinteraksi.");
    gameLoop();
};
Enter fullscreen mode Exit fullscreen mode

&lt;/script&gt;
</code></pre></div>
<p></body><br>
</html></p>

<p>How I Used Google AI Studio<br>
I used Google AI Studio to interface with the Gemini API. I leveraged the API to act as the &quot;brain&quot; behind the game. Instead of using a static, pre-written script, Gemini generates narratives and hints in real-time. This allows the game to provide a unique experience each time it&#39;s played, highlighting the power of generative AI for dynamic content creation.</p>

<p>Multimodal Features<br>
I implemented the multimodal capabilities of Gemini 2.5 Flash to create a richer user experience.</p>

<ul>
<li>Text &amp; Location-Based Narrative Understanding:

<ul>
<li>When the player enters the Playful Zone, the AI generates a cheerful and humorous narrative, sometimes including patient &quot;customer&quot; reactions.</li>
<li>When the player moves to the Dark Zone, the AI generates a scary narrative and mysterious whispers, completely changing the game&#39;s atmosphere. This demonstrates Gemini&#39;s ability to adapt tone and context based on user input.</li>
</ul></li>
<li>Image Understanding for Hints:

<ul>
<li>In the Hint Zone, I demonstrate multimodal capability by sending both text and an image input to Gemini. While the image is a simulation, this process shows how the AI can &quot;see&quot; a visual input (e.g., &quot;a key on a table&quot;) and generate a relevant narrative hint to help the player progress. This makes the interaction more meaningful and connected to the in-game visuals.</li>
</ul></li>
</ul>

Top comments (0)