The Problem
Ballet is beautiful. It's also completely unreadable if you didn't grow up with it.
A viral YouTube Short changed how we thought about this. Hand-added captions translating dancer movements into plain language had people who'd never cared about ballet suddenly hooked. Someone said they're autistic and this was the first time ballet made sense to them.
We wanted to automate that. Motion Lore generates synchronized narrative subtitles for ballet videos — drop in a YouTube URL or upload a file, get back a subtitle track that reads the choreography like a story.
Here's how we built the backend on AWS + Vercel.
The Architecture
Two-Pass LLM Pipeline
A single model looking at raw video footage with zero context produces mediocre subtitles. So we split it into two passes:
- Groq (llama-3.3-70b) reads the video title, identifies the ballet, and enriches it with narrative context — characters, plot, setting.
- That context gets injected into Gemini 2.5 Flash's system prompt before it analyzes the video. Gemini isn't guessing blind anymore — it knows it's watching Giselle, not some random dance clip.
The quality difference is noticeable.
DynamoDB — Two Tables, Two Jobs
ballet-subtitles is content-addressed by SHA-256 hash of the video URL or file. The same video uploaded by a thousand different users gets processed exactly once and cached forever. Zero redundant Gemini calls.
ballet-jobs is our async job state machine — queued → processing → done → failed. The key decision here was moving job state out of memory and into DynamoDB early. In-memory state dies on restart and can't scale across instances. DynamoDB means our FastAPI backend is fully stateless.
S3 for Uploads
User-uploaded videos land in S3 temporarily. Once Gemini processes them via the Gemini File API, they're deleted. YouTube URLs skip S3 entirely — Gemini has a native URL handler that takes them directly, which saved us an entire download pipeline.
FastAPI on AWS
Stateless, horizontally scalable, no server-side sessions. All state lives in DynamoDB. Spin up as many instances as needed.
Vercel for the Frontend
React frontend deployed on Vercel. The stateless FastAPI backend made this pairing clean — Vercel handles edge delivery and the frontend talks directly to the AWS backend over REST. No friction.
The Tricky Parts
Gemini + S3: Gemini can't pull from S3 directly. We had to route all uploads through the Gemini File API as an intermediate step — not obvious from the docs.
Job state at scale: First pass used in-memory job tracking. Worked fine locally, broke immediately under any real load or restart. DynamoDB fixed this completely.
Context poisoning: Early on, bare file hashes were being passed straight to Gemini as identifiers. No title, no context. The subtitles came back generic and disconnected from the actual ballet. Adding the Groq classification gate — which fires before any Gemini call — fixed the quality problem at the root.
What We'd Do Differently
Honestly, we'd add the DynamoDB job state table on day one instead of migrating to it mid-build. The in-memory approach feels fine until it suddenly isn't.
What's Next
- Opera and classical Indian dance forms
- Community subtitle ratings to improve future generations
- Embeddable widget for ballet company websites
Built for #H0Hackathon.
Top comments (0)