Smart Cultural Lanterns: we taught Timișoara's street lamps to tell stories

Mario Mihaila — Sun, 15 Nov 2026 10:50:41 +0000

What We Built

The memories hidden in Timișoara’s squares were waiting to be told at UniHack 2026’s Immersive Arts track. We created Smart Cultural Lanterns – handcrafted lanterns equipped with an ESP32 board, a smart bulb and a speaker, placed in the cultural squares of the city. Each of the lanterns can be interacted with in two different ways: you can listen to the stories recorded by other people in the same place, or even record a new memory for that place and have the lantern read it out loud to other people walking through the square later.

The lantern is the main character of this experience. Your phone is simply a remote control.

The flow, end to end:

Scan the QR code of the lantern fixed at location (/f/unirii). The session recorded will be bound to the lantern. The URL is geofenced to location Uniirii square so you will be prompted to sign in when you are physically present in the square.
Log in (Auth0) and start recording by pressing the button to upload the raw audio recordings (of whatever length) in whatever language you speak.
A single Gemini API call will now in turn perform the 4 following actions on the newly uploaded raw audio: 1) transcription of the audio 2) content moderation (profanity/PII/spam –> reject or flag for human moderation) 3) language detection 4) translation of recorded story into Romanian and English.
If the entry is approved, the backend generates two MP3s: one for your own cleaned-up voice for the language you spoke in (more on that below), and one in ElevenLabs TTS for the translation.
The story enters the lantern's play queue. The ESP32 polls the backend, downloads the MP3 over plain HTTP, sets the bulb's color over local UDP, and plays your story into the square.

The part we're proudest of is a small UX touch we call sealed stories: after you submit, your story's text stays blurred behind a stardust animation on your phone. It only unseals once the physical lantern has actually spoken it. Your story belongs to the square before it belongs to your screen.

The stack: FastAPI + asyncpg on the backend, React + Vite on the frontend, PostgreSQL (Tiger Data) for storage and time-series presence analytics, Auth0 for identity, Gemini for the audio pipeline, ElevenLabs for voice, and an ESP32 with a MAX98357A I2S amp driving the speaker and a WiZ bulb.

What we learned the hard way

A hackathon with real hardware on real networks teaches you things no tutorial will:

An ESP32 on hotspot WiFi is a brutal code reviewer. Our /poll endpoint originally ran 4–5 sequential DB queries. Each round trip to the cloud database cost 200–500 ms, so the endpoint took up to ~4.8 s — and the firmware's HTTP timeout was 3 s. The lantern just saw HTTP -11 and went silent. The fix: collapse the whole poll (fetch next story, mark it, bump last_seen, count presence) into one CTE statement. One round trip, ~0.4 s, lantern happy.
Networks fail in layers. We lost two hours to a demo setup where everything was configured correctly — except the laptop's hotspot IP had silently changed underneath all of it. And on my dev machine, HTTP/2 POSTs to Gemini's API hang forever while HTTP/1.1 sails through. When a request hangs with zero bytes received, suspect the protocol before your code.
Delivery guarantees aren't just for distributed-systems papers. A story that gets lost between the database and a speaker in a public square is a person's memory, gone. So the queue became at-least-once: polling marks a story delivering, the ESP32 must POST an ack after playback actually starts, and a sweeper requeues anything unacked after 75 s, up to 3 attempts.

Demo

🌐 Live: smart-lanterns.duckdns.org

(Heads-up: the recording flow is GPS-geofenced to the actual squares in Timișoara — by design, you have to be there. The rest of the app is browsable from anywhere.)

The Smart Cultural Lantern in the square during testing

Partner Technologies

Auth0 gates recording (you can listen anonymously, but your voice in a public square is attached to an identity). The integration itself via @auth0/auth0-react was quick; the educational part was everything around it. Our WSL2 dev clock ran a few seconds behind real time, so freshly minted tokens failed validation with ImmatureSignatureError — the iat claim was "in the future." (Fix: a small leeway in JWT validation.) And we silently bounced logins for an hour because the Auth0 application wasn't authorized for our API in the dashboard — the lesson being: surface auth errors in the UI immediately, never swallow them.

Google Gemini is the heart of the pipeline, and the design decision that paid off most was one call instead of four. We send the raw audio with a prompt asking for a single JSON object: transcript, language, moderation verdict (approved / rejected / pending_review with a reason), and translations. One call means one failure mode, one latency budget, and a moderation decision that actually has the full context of what was said. We hit one infrastructure quirk worth sharing: the official Python SDK's gRPC transport hung indefinitely on our network, so we talk to Gemini over plain REST with httpx — about 30 lines, fully debuggable, zero magic.

ElevenLabs does two very different jobs. The obvious one: TTS for the translated version of each story. The less obvious one: audio isolation. We decided a storyteller's own voice should play for the language they spoke in — a grandmother's story about Piața Unirii should sound like her, not like a synthetic narrator. Street recordings from a phone are noisy, so originals go through ElevenLabs' audio-isolation model (with an ffmpeg afftdn denoise fallback), then get loudness-normalized to −14 LUFS so TTS and human recordings sit at the same volume on the lantern's speaker. TTS is only the fallback for languages the storyteller didn't speak.

Tiger Data (PostgreSQL) holds everything: users, lanterns, stories, the play queue, and a time-series table of presence pings (the frontend pings every 15 s; the lantern shows how many people are "with it" right now, and the bulb's mood follows the crowd). Using raw asyncpg instead of an ORM felt risky under time pressure but turned out to be the enabler for the single-CTE poll endpoint — when you own the SQL, collapsing five queries into one is an evening's work, not a framework fight.

Hackathon Experience

UniHack 2026 in Timișoara, Immersive Arts track. Our team split cleanly: my teammate owned the hardware — ESP32 firmware, I2S audio, driving the WiZ bulb over local UDP — and I took the frontend from "works" to "beautiful" (Romanian-first UI, dark mode, a recording fallback for iPhones, which of course record audio/mp4 instead of what every other browser produces), as well as the backend, the AI pipeline, and the deployment.

The moment I'll remember: the first full end-to-end test. Phone in hand, standing over a lantern, recording a story — and a minute later hearing it come back out of the speaker in the storyteller's own voice, with the bulb glowing. Everything between those two moments — the auth, the moderation, the translation, the queue, the firmware — had to work at once, and it did.

The other thing hardware hackathons give you that pure-software ones don't: the bug is sometimes physical. Not in your code, not in your config — in the clock of your VM, in the hotspot deciding you deserve a new IP, in a 3-second timeout compiled into a chip on the other side of the table. You learn to debug the whole system, down to the air.