Traycer: What a returned dining-hall tray can tell us

Alan Tai — Tue, 06 Oct 2026 00:02:16 +0000

This is a submission for the MLH x DEV Writing Challenge.

What I Built

We kept seeing food come back on trays at our college dining halls. Someone had bought it, cooked it, and served it, but by the dish return it was headed for the trash. We wanted to know which foods people were leaving behind and whether a kitchen could use that information to change the next meal.

At MHacks 2026, I built Traycer with Aslan Wang and Anton Angeletti. It connects a phone camera to a live food-waste dashboard, then lets a kitchen explore menu changes in a 3D cafeteria simulation. There are also two fictional AI characters you can text or call: a food-waste consultant and a chef.

My contributions covered the idea, frontend, computer vision, SpacetimeDB integration, Notability notebook, and domain. The finished experience brought our work together: capture a returned tray, review what the camera found, look at the waste totals, and test a possible change.

Getting a tray counted once

A phone streams camera frames over WebSockets to a Python FastAPI backend. YOLO-World and OpenCV detect food and containers; ByteTrack and our tracking logic help track the same tray across frames. OpenAI vision refines the food identification and estimates how much is left. Staff can accept or reject a capture in the review interface.

That tracking matters because a tray doesn't disappear after one frame. Counting each detection as a new tray would make the dashboard wrong even if every detection looked convincing. We had to handle the identity of the tray over time as well as the food on it.

One detector test picked up the paper beside a plate and missed the pizza label. We annotated the actual image in Notability and noted trying a tighter crop. Having the failed output beside the proposed fix made it much easier to explain what we were testing.

Keeping the dashboard and simulation consistent

SpacetimeDB stores the tray observations and photos. When an observation changes, a reducer replaces that tray's food rows and recomputes the food and service totals in one transaction. The dashboard shows leftovers by food, with estimated cost, carbon, and water impact.

The simulation has its own SpacetimeDB module. A scheduled reducer ticks the shared world every 200 milliseconds, managing movement, queues, seats, conversations, and waste. When a simulated student needs to make a decision, the module writes a job row for an AI worker. A Node.js worker uses OpenAI for those decisions, and React Three Fiber renders the cafeteria through live subscriptions.

This gave the AI diners a shared world to act in. Their choices could differ, while the database kept the state and waste accounting consistent. The worker or 3D client could reconnect without needing to recreate the world.

We compare a baseline menu and a proposed menu using the same seeded student cohort, and repeat baseline runs to understand variability. The simulation is a way to explore a change before cooking it. A later real service, measured by the camera, is how we'd check whether that change helped. The environmental and cost figures are estimates, and a simulation result isn't a measured reduction in a real kitchen.

Demo

The walkthrough follows the path from camera captures to the waste dashboard and cafeteria simulation. It also shows the conversational side: kitchen staff can ask the consultant about waste patterns or talk to the chef about a recipe and portion change.

Partner Technologies

Gemini API: explaining the live counts

We used gemini-2.5-flash through the Gemini API with structured JSON output. Gemini reads the actual capture totals, including the tray count, food left behind, and food types, and writes a short explanation of the main leftover and one small experiment to try.

The dashboard calculates its own estimated carbon, water, and cost figures. Gemini provides the language around those figures. In one live run, it identified bagels as the main leftover and suggested an experiment beside the dashboard's existing totals.

This was useful because a busy kitchen operator needs to decide what to do with a number. The summary updates while captures come in, so the operator can read it during service.

ElevenLabs and Relay: talking through a change

Relay supports texting and voice and video calls with our fictional consultant and chef. Shared tools let the characters read camera reports, save recommendations, and retrieve simulation results. Pipecat and ElevenLabs support the voice conversations.

We used ElevenLabs' preset voices George for the chef and Eric for the consultant. The chef and consultant have different roles: one proposes recipes and portion changes, while the other helps explain the waste patterns. SQLite preserves the evidence and connects recommendations to their tests.

Calls brought their own debugging problems. A connection could succeed without carrying audio or video, and interruption detection sometimes stopped a reply before it finished. Network routing and turn-taking needed attention alongside the agent tools.

A .tech domain for the project

We chose trayitforward.tech, a reference to “pay it forward.” What comes back on one tray can inform what a kitchen prepares for the next meal. The domain opens our landing page and introduces that workflow.

Hackathon Experience

We built Traycer at the University of Michigan's MHacks in Ann Arbor on October 3–4, 2026. Aslan, Anton, and I were bringing together systems with very different failure modes: a live camera, independent AI diners, and agents people could call.

During a late-night team meeting, we used Notability's recording and note sync to keep the discussion connected to our handwritten notes. We put architecture sketches, build checklists, detector outputs, and dashboard screenshots in the same notebook. Missed food detections and stale totals were much easier to discuss when we could point at the actual output.

The project taught us how to track objects across frames, give vision models clearer instructions, and keep incoming captures synchronized with a dashboard. Bringing the pieces together also meant making sure the dashboard, conversational agents, and simulation could refer to the same findings and saved tests.

I'm proud that we finished a shared experience where each teammate's work contributed to the next step. We also gave our simulated diners sponsor merch and U-M gear. Go Blue!

DEV Community: Alan Tai