This journey started a week before the hackathon, long before I knew I’d be leading a global dev team. I’m a junk hauler by trade, running my own deck art business on the side. I’m not a developer. I design with Canva and sell skateboards online. One day, a custom order pushed me to try something more advanced in Photoshop. Five YouTube tutorials later, I still couldn’t figure it out.
That frustration sparked a wild idea: What if AI could guide me inside the software itself?
I started tinkering in Replit, but when I saw the Bolt.new hackathon—offering access to cutting-edge tools like ElevenLabs and Gemini—I decided to go all in.
The original idea was called Forge: a gamified learning platform for creators, where users would earn XP for using an AI mentor (in “Forge Mode”) and compete in creative challenges judged by the community. Think: “Design a landscaping logo in Photoshop” or “Make a classical remix of a rap song in Pro Tools.” Players could win gold, unlock avatar skins, and eventually compete for paid opportunities.
But around Day 17, I hit a wall. I realized this was bigger than I could build alone. So I posted in the Bolt Discord and that’s when everything changed.
Within hours, I had a global team:
• Swara (Backend, India)
• Maruf (Backend, Bangladesh)
• Tobi (UI/UX, Nigeria)
We jumped into a group chat and immediately got to work. Our priority: making “Forge Mode” functional. The technical vision was ambitious:
Real-Time Screen Awareness: Every three seconds, a screenshot is taken from the user’s screen and sent through an n8n workflow.
Visual Analysis with Gemini: The screenshot goes to Gemini Vision, which interprets the screen and returns a structured JSON of what’s visible.
AI Tutoring via ElevenLabs: This JSON is passed to Percival, our ElevenLabs conversational agent, as context for answering the user’s question.
Software-Specific Knowledge via RAG: Percival uses a RAG database we created, containing in-depth docs and tutorials across 12 major creative tools, to provide expert-level answers tailored to the software in use.
The backend was like having Google Vision, ChatGPT, and a software mentor rolled into one.
Tobi and I worked on the frontend and UX while Swara and Maruf tackled backend pipelines. With 5 days left, time was slipping. Tobi suggested we drop the gamification and avatars to focus entirely on the AI tutor—the true MVP. We renamed the app OmniVeo, from the Latin “to know” and “to see.”
We ran into serious GitHub merge issues, so I created a shared Bolt.new account so everyone could contribute in turns. It wasn’t ideal, but it worked. With less than 12 hours remaining, we hopped on a late-night global call. I stayed up all night, screen-sharing with my team while preparing for my 7:30am job.
At 8am, while I was hauling junk in California, I got a message: It works.
Maruf had solved the final integration. I tested it live in Photoshop. The AI could see my screen, understand my context, and answer with precision using our RAG-powered system. It was working exactly as envisioned.
Swara had been facing issues merging the backend workflow into the production project on Bolt, but after hours of troubleshooting and team collaboration, everything finally aligned.
The final hours were chaos. Tobi was finalizing the UI. I was trying to finish a demo video during a lunch break. My computer crashed during the upload. We re-rendered and submitted with 8 minutes to spare.
It wasn’t perfect. But it was real. It worked. And we did something that’s never been done before. We created an AI tutor that can see your screen, understand your problem, and guide you through complex software in real time.
None of this would’ve been possible without Bolt.new. The platform gave a non-technical person like me the tools—and the global collaborators—to take a raw idea and turn it into a working product in under 30 days.
Now, we’re taking OmniVeo into production and working on making it scalable, affordable, and available to the world.
Top comments (0)