NeoPet: An AI Virtual Pet That Talks Back — From X Celsior '26 Win to HackHazards '26
The Problem
Virtual pets have existed since Tamagotchi, but they've always been dumb state machines: feed it, it's happy; ignore it, it's sad. There's no actual interaction — just a hunger bar pretending to be a relationship. I wanted to build a virtual pet that a user could genuinely talk to — one with a persistent personality, real conversational memory, and a voice, rather than a sprite that reacts to button presses.
NeoPet started as a submission for X Celsior '26, where it won first place, and was later cleaned up and resubmitted as a secondary project for HackHazards '26 alongside StudyMate AI, since both projects share a philosophy: AI interfaces should feel like a conversation, not a menu.
Tech Stack
- Frontend/Framework: Next.js — server-side rendering for fast initial loads and clean routing between pet states/screens
- Backend & persistence: Firebase — auth, real-time database for pet state (mood, stats, conversation history), and hosting
- LLM layer: Groq (LLM inference) — powers the pet's personality and conversational responses with low enough latency to feel like a real-time chat rather than a loading spinner
- Voice output: ElevenLabs TTS — gives the pet an actual distinct voice rather than generic robotic TTS, which matters enormously for something meant to feel like a companion
- Voice input: Web Speech API — browser-native STT, kept lightweight since this is a web app and didn't need a heavier custom pipeline
Implementation
The core architecture treats the pet as a stateful character, not a stateless chatbot wrapper. Every conversation turn updates persisted state in Firebase — mood, recent topics discussed, "relationship" progress — which then gets fed back into the LLM prompt on the next turn so the pet's responses stay consistent with its established personality and history, instead of resetting to a blank slate every message.
The voice loop (Web Speech API for input, ElevenLabs for output) was the deliberate design centerpiece: talking to a pet and having it respond in a distinct voice is what makes it feel alive versus reading text in a chat bubble. Getting Groq's response speed low enough that the ElevenLabs TTS call didn't introduce an awkward silence between "user finishes talking" and "pet starts responding" was the main engineering focus — a virtual pet that pauses for two seconds before speaking breaks the illusion immediately.
Challenges Faced
Making the pet feel consistent, not just responsive. An LLM without persisted context will happily contradict itself between sessions — remembering nothing about a prior conversation. Solving this meant treating Firebase not just as a stats database but as the pet's actual memory: summarizing recent interactions and injecting that summary into the system prompt so the pet "remembers" the user across sessions rather than greeting them like a stranger every time.
Latency stacking across three services. STT (browser) → LLM (Groq) → TTS (ElevenLabs) is three network round-trips before the pet can "speak." Each individually is fast, but stacked naively they add up to something that feels laggy. This pushed toward starting the TTS request as soon as a usable chunk of the LLM response was available rather than waiting for the full response to complete.
Scope discipline. It's easy for a virtual pet project to sprawl into a dozen half-finished mechanics (feeding, mini-games, customization). The version that won X Celsior '26 deliberately cut everything that wasn't in service of the core loop: talk to the pet, have it respond in character, in voice, with memory. Everything else got left for a later phase rather than diluting the demo.
From X Celsior to HackHazards
Winning X Celsior '26 validated the core concept, but the HackHazards '26 resubmission wasn't just a re-upload — it involved cleaning up the README, restructuring the project description into Devpost's format, and presenting it as a secondary, complementary project to StudyMate: two different applications of the same underlying belief, that AI products should feel conversational and persistent rather than transactional.
1st place, X Celsior '26. Submitted as a secondary project for HackHazards '26.
Top comments (0)