Pratyay Banerjee

Posted on Jul 28

Wynnie 🦄 — 𝘠𝘰𝘶𝘳 𝘚𝘩𝘰𝘱𝘱𝘪𝘯𝘨, 𝘯𝘰𝘸 𝘰𝘯 𝘈𝘶𝘵𝘰𝘱𝘪𝘭𝘰𝘵!

#devchallenge #assemblyaichallenge #ai #api

AssemblyAI Voice Agents Challenge: Business Automation

𝗖𝗮𝘁𝗲𝗴𝗼𝗿𝘆 𝗦𝘂𝗯𝗺𝗶𝘀𝘀𝗶𝗼𝗻: Business Automation Voice Agent

Note: We'd also like to consider our project under the Real-Time Voice Performance track.

Participants: @neilblaze & @achalbajpai.

Video ▶️

ㅤ

What we built 🤔

Wynnie is your smart autonomous AI shopping companion / agent that revolutionizes how people shop online through simple natural language. It's like having a personal shopping genie 🧞 that actually understands what you want and handles everything automatically, starting from finding products to getting the best deals, it got all covered! We built this using AssemblyAI's speech recognition that can detect over 50 languages on the fly! ⚡

💡 The whole idea behind a smart system like this is that it eliminates all the tedious shopping work while making everything accessible to literally anyone - whether you speak English, Hindi, Spanish, or whatever. Plus it scales incredibly well because each user gets their own personalized shopping brain. Most importantly, it caters to this huge mass of elderly people who have money to spend and need to shop, but they're being shut out by friction designed for digital natives. That's millions of potential customers just sitting there, frustrated and underserved, whom we (as well as business owners) care about and thus we're serving what they want!

🏠Homepage: https://wynnie-v1.vercel.app
🔗Try it out here: https://wynnie.vercel.app

ㅤ

How it works? 💣

Users simply sign up using Google OAuth, and we handle the same via Firebase. Once they’re in, they land on the dashboard, and from there, they can ask for anything, either by typing it out or just talking. If it’s voice, AssemblyAI's speech recognition kicks in to transcribe everything in real-time, even down to the word-level timestamps and formatting. That transcription is then piped to OpenAI, which pulls out what the user wants, any key details, and even picks up on tone or emotion. Then our AI agents jump in, digging through SERP APIs and Perplexity Sonar to find the best product matches, factoring in things like location, coupons, and what they’ve liked before. Once the picks are ready, they hit the inventory, coupons get auto-applied using Synphase, and payments are seamlessly handled through UPI LiteX. All of it’s tracked and stored in Supabase, keeping everything clean, secure, and seamless.

ㅤ

App Repository 🔗

Wynnie.AI 👉 https://github.com/achal-b/wynnie [Open source on GitHub]

ㅤ

Features 🎠

Autonomous AI shopping buddy that delivers true performance!

AssemblyAI's Voice recognition that works with 50+ languages automatically!

Scales like crazy with our multi-agent orchestrated architecture

Real-time product hunting using Perplexity AI (Sonar) & SERP API

Smart deal finder that optimizes your cart automatically!

Intelligent delivery planning with eco-friendly routing

Works offline as a Progressive Web App (PWA)!

Speaks your language - literally any of 50+ languages, with AssemblyAI

Knows who's talking with speaker identification & automatically isolates background noise.

Clean, modern interface built with Next.js & Tailwind

Google OAuth SSO via Google Firebase

Live price tracking and bundle suggestions!

True AI recommendations that finetuned to user's experience!

Supabase backend for blazing fast performance!

Seamless payments via UPI-Litex are highly secure & E2E encrypted!

Comes with batteries & has CI/CD via GitHub actions.

Saves you 💰 + tons of time!

Works for everyone - accessibility first!

Privacy-focused and GDPR* compliant!

ㅤ

System Architecture 📊

Privacy & Security 🔐

Wynnie deals with your shopping data and payment info, which is pretty sensitive stuff. We've gone overboard on security to make sure everything stays locked down and 100% GDPR compliant.

All communication happens over encrypted channels, and we use Supabase's built-in security features. Voice data gets processed securely through AssemblyAI's endpoints. Down the road, we're planning full end-to-end encryption for everything.

Background 📜

Here's the thing - online shopping is still a pain for way too many people! Language barriers, confusing interfaces, and just the overwhelming number of choices make it really hard for people to find what they actually need and get good deals. 😔

Source: https://retailwire.com/discussion/are-retailers-making-it-too-tough-for-seniors-to-shop-online

ㅤ

Most shopping sites basically dump you into this maze where you have to:

Navigate complex menus, compare tons of products manually, figure out which deals are actually good, and somehow optimize everything yourself. If you don't speak the main language perfectly, or if you have accessibility needs, or if you just find tech interfaces confusing - you're pretty much out of luck. It's 2025 and we're still making people work way too hard just to buy stuff.

The core problem is that e-commerce platforms are built like databases instead of conversations. Nobody shops by filling out forms, rather we shop by talking about what we need.

ㅤ

Traditional E-commerce Pain Points	Wynnie's AI Solution
🟠 Users manually search for products	🟢 AI-driven intent detection + voice/text input
🟠 Overwhelming product listings	🟢 Personalized, context-aware recommendations
🟠 No clarity on best deals or coupons	🟢 Auto-applied coupons via Synphase Scraper
🟠 Complex checkout flows	🟢 Streamlined voice-first ordering system
🟠 Static dashboards and limited insights	🟢 Dynamic dashboard with conversational UX
🟠 No real-time decision feedback	🟢 LLM-as-Judge provides on-the-fly optimization
🟠 Limited customer engagement	🟢 Conversational agents tailored to user needs
🟠 Siloed services & fragmented UX	🟢 Unified AI Orchestrator with agent collaboration

ㅤ

Instead of making people navigate complex websites, Wynnie lets you just talk naturally about what you're looking for. The AI figures out your intent, researches products automatically, finds the best deals, and presents you with optimized options. It's like having a really smart friend who knows everything about shopping!

Wynnie changes the game by understanding natural conversation, making smart decisions independently, and handling all the tedious optimization work automatically! 👪

Beyond just understanding what you say, Wynnie does the heavy lifting with real-time product research through Perplexity AI, automatic deal optimization, and smart delivery planning.

The kicker? Users don't need to learn anything new — just talk naturally about what you want! This makes shopping accessible to everyone, regardless of language, tech skills, or physical abilities.

We're aiming for shopping that's fast, smart, and genuinely helpful through AI conversations that understand context, preferences, and optimize for the best outcomes automatically! ✨

Our mission is making online shopping work for everyone through natural AI conversations that connect people with exactly what they need through Intelligent Shopping Automation.

Snapshots 🖼️

Why AssemblyAI?

Voice recognition is make-or-break for accessible shopping. AssemblyAI's Universal Speech Model gives us the accuracy and language support we need without the headaches.

AssemblyAI automatically detects what language someone's speaking from a list of 50+, figures out who's talking when, and gives us word-level timing. This lets us build shopping experiences that actually work for real people having real conversations.

// AssemblyAI Voice Processor for Wynnie 🦄

async transcribeAudio({
  file,
  speech_model = 'universal',
  language_code,
  punctuate = true,
  format_text = true,
  speaker_labels = false,
  speakers_expected,
}: AssemblyAITranscriptionRequest): Promise<AssemblyAITranscriptionResponse> {
  try {
    const audioUrl = await this.uploadAudio(file);
    const transcriptionJob = await this.startTranscription(audioUrl, {
      speech_model, language_code, punctuate,
      format_text, speaker_labels, speakers_expected,
    });
    const completedTranscription = await this.pollForCompletion(transcriptionJob.id);
    return completedTranscription;
  } catch (error) {
    console.error('Error transcribing audio with AssemblyAI:', error);
    throw error;
  }
}

AssemblyAI processes all our voice input and turns natural speech into shopping intent. The multi-language support means we can help users regardless of what language feels most comfortable to them.

Also, thanks for the $50 credits which helped us quickly get started! 🙏🏻

Our AI agent system (powered by OpenAI's GPT-4o mini) generates smart yet fast shopping recommendations where AssemblyAI handles the voice input, and our orchestrator coordinates specialized agents for finding products, optimizing deals, and planning delivery. Everything works together seamlessly! 🙂

We dove deep into AssemblyAI's advanced features like speaker diarization and confidence scoring. Building voice-first interfaces was definitely a learning curve since most of us come from traditional web development. We had to study voice interaction patterns and accessibility guidelines from scratch.

Besides AssemblyAI, we learned tons about autonomous agent architectures, real-time AI coordination, and building systems that actually scale. 🌟

ㅤ

Design 🎨

We were heavily inspired by the revised version of Double Diamond design process, a model popularized by the British Design Council, which not only includes visual design, but a full-fledged research cycle in which you must discover and define your problem before tackling your solution & then finally deploy it.

Discover: Understanding why current shopping experiences fail so many people.

Define: Figuring out what an autonomous shopping agent actually needs to do.

Develop: Building the multi-agent system that handles real conversations.

Deliver: Launching with PWA support and continuous learning from real users.

We used Figma extensively, focusing on voice interaction flows and accessibility patterns. Our friend Praveen did the user testing which helped us refine how the AI responds and when it asks for clarification.

CREDITS

Design Resources: Figma Community, Web Accessibility Initiative
Icons: Lucide React, accessibility-focused icon sets
Typography: Manrope and other system fonts for maximum readability

Challenges we ran into 😤

Building an autonomous shopping agent brought some really interesting technical challenges, especially around coordinating multiple AI services in real-time.

The biggest headache was getting all our AI services to work together smoothly without creating noticeable delays. We needed AssemblyAI for voice processing, Perplexity AI for product research, and OpenAI for reasoning, all of them working in harmony. Orchestrating these different agents while maintaining fast response times required some creative prompt engineering and smart fallback strategies.

Getting voice recognition accuracy right across different accents and speaking styles was trickier than expected. AssemblyAI's auto-detection helped a ton, but we still had to fine-tune confidence thresholds and build intelligent fallback mechanisms.

Real-time optimization without sacrificing privacy was another puzzle. We ended up doing as much processing as possible on the client side while using secure API calls for the AI services.

We're really proud of creating a shopping experience that genuinely works across languages and provides intelligent optimization. The multi-agent architecture successfully handles complex shopping tasks without human intervention! :)

ㅤ

What's next? 🚀

Wynnie has serious potential to change how people think about online shopping. We want this to be the thing that finally makes e-commerce work for everyone, regardless of language, tech comfort, or physical abilities!

What we're building next:

Predictive Shopping: AI that suggests things before you even ask
Visual Product Search: Point your camera at something and find it online
Group Shopping: Shop with friends and family through shared conversations
Sustainability Scoring: See the environmental impact of your purchases
Cross-Platform: Native mobile apps and smart speaker integration

We're excited to expand language support, improve AI accuracy, and connect with more online retailers!

ㅤ

End Notes 🙌🏻

Huge thanks to DEV for hosting this challenge and the AssemblyAI team for excellent documentation and API design, and to the open-source community for inspiration and support! 🙌