MenuMaster: Your Menu Anxiety Solution

Zakeer — Sat, 13 Sep 2025 10:50:44 +0000

This is a submission for the Google AI Studio Multimodal Challenge

What I Built

Picture this: You're on a first date at that trendy new restaurant downtown. The menu arrives, and suddenly you're staring at "Quiche Lorraine" and "Spinach and Feta Stuffed Chicken" wondering what exactly you're about to order. Your date looks confident, but inside you're panicking. Sound familiar?

Whether you're an introvert who dreads asking servers about menu items, an extrovert exploring a new cuisine, or simply someone who wants to make smart dining choices, MenuMaster is your discreet dining companion that turns menu confusion into confident ordering.

MenuMaster solves the universal problem of menu anxiety by instantly analyzing any restaurant menu through your phone's camera and providing personalized recommendations based on your preferences, dietary restrictions, and dining goals. No more awkward moments, no more ordering regrets, just confident dining experiences.

Demo

Try the Applet:

You can try out the deployed applet here: https://menumaster-181501076904.us-west1.run.app/

Video Demo:

Watch a demonstration of the applet in action: https://vimeo.com/1118329445

Additional Demos:

For a more in-depth look at the different sections and features, you can view multiple demo videos in this folder: https://drive.google.com/drive/folders/1UJEJuQFrmcoNfRZzhlX1VA52d8cy1mwc?usp=sharing

How I Used Google AI Studio

MenuMaster is powered by Google AI Studio, using Gemini 2.5 Pro as its intelligent core. It's a true multimodal app that combines sight with reasoning to solve a common problem.

Here’s how it works:

Vision: When you upload a menu photo, Gemini’s powerful vision capability instantly reads and understands all the text, from dish names to prices.
Language & Reasoning: Next, it processes your personal preferences (like "mildly spicy" or "gluten-free"). It then uses its advanced reasoning skills to connect your needs with the menu items it just read.

In short, Google AI Studio allowed me to create a simple tool that sees what you see, understands what you want, and gives you a clear, confident recommendation.

Multimodal Features

Menu Upload & Analysis
Simply snap a photo of any menu, and MenuMaster instantly understands every dish and price. This core vision capability removes the first barrier to a great dining experience.

Goal-Based Recommendations
Tell the app your goal—whether you're exploring new flavors, sticking to a budget, looking for healthy options, or just need a quick lunch—and it tailors suggestions to fit the occasion.

Smart Recommendations
By combining the menu analysis with your personal tastes (like spice level and protein choice), the app delivers truly personalized dish suggestions that you're likely to love.

Dietary Alerts
The app automatically flags dishes containing allergens or ingredients you want to avoid, providing peace of mind and making dining safer for those with restrictions.

Dish Decoder
Confusing culinary terms are translated into simple, easy-to-understand explanations, complete with key ingredients and taste profiles, so you always know what you're ordering.

Cultural Insights
To enrich the dining experience, the app provides interesting facts about the cuisine, local dining etiquette, and the history behind certain dishes, turning a meal into a cultural exploration.

Built with Google AI Studio • Deployed on Google Cloud Run • Powered by Gemini 2.5 Pro

GlobalCare AI: International Medical Voice Assistant

Zakeer — Sun, 27 Jul 2025 17:22:58 +0000

This is a submission for the AssemblyAI Voice Agents Challenge

What I Built

GlobalCare AI is a specialized Domain Expert Voice Agent for medical consultations. Users speak their medical concerns, and the system provides AI-powered medical analysis with voice responses, supporting multiple languages and country-specific emergency protocols.

Domain Expert Voice Agent Category
This submission addresses the Domain Expert Voice Agent prompt with:

Medical Domain Expertise: Recognition of medical terminology and symptoms
Emergency Triage: Automated urgency assessment (Critical/High/Medium/Low)
Cultural Adaptation: Country-specific emergency numbers and medical protocols
Voice-to-Voice Workflow: Complete audio input to audio response system
Multi-language Support: 7 languages across 6 countries

Core Problem Addressed
Many people face language barriers and accessibility issues when seeking medical guidance. GlobalCare AI provides voice-first medical consultation that works across languages and cultures.

Demo

Assembly AI Demo.mov - Google Drive

drive.google.com

GitHub Repository

Zakeertech3 / Assembly_AI_Hackathon

🏥 International AI Medical Voice Assistant

Advanced voice-to-voice medical consultation with emergency triage across 6 countries in 7 languages

A complete medical AI system that processes voice input, analyzes symptoms using advanced AI, and provides spoken medical guidance with emergency triage capabilities. Built for global healthcare accessibility with support for multiple languages and country-specific emergency protocols.

🖼️ System Overview

Complete medical voice consultation interface with country selection, voice recording, and real-time medical analysis

Emergency alert system with country-specific protocols and multilingual support

System performance dashboard showing real-time metrics and component status

Advanced medical analysis results with urgency assessment and voice response generation

✨ Features

🚨 Emergency Triage - Sub-5 second critical symptom detection
🎙️ Voice-to-Voice - Complete audio workflow with AI voice responses
🌍 International - 6 countries with localized emergency protocols
🗣️ Multilingual - 7 languages with cultural medical adaptation
💊 Drug Safety - Real-time medication interaction checking
⚡…

View on GitHub

Technical Implementation & AssemblyAI Integration

1. Medical Vocabulary Boosting for Domain Expertise

Medical terminology requires exceptional accuracy - a misrecognized medication name could have serious consequences. We implemented comprehensive vocabulary boosting using AssemblyAI's word_boost feature:

class MedicalSpeechProcessor:
    def __init__(self, knowledge_base):
        self.knowledge_base = knowledge_base
        self.medical_terms = self._load_medical_terminology()

    def _load_medical_terminology(self):
        """Load comprehensive medical vocabulary for AssemblyAI boosting"""
        return {
            "symptoms": [
                "chest pain", "shortness of breath", "nausea", "vomiting", 
                "dizziness", "headache", "fever", "cough", "fatigue", 
                "abdominal pain", "palpitations", "syncope"
            ],
            "medications": [
                "acetaminophen", "ibuprofen", "aspirin", "amoxicillin", 
                "metformin", "lisinopril", "atorvastatin", "amlodipine", 
                "metoprolol", "sertraline", "fluoxetine", "alprazolam"
            ],
            "body_parts": [
                "heart", "lungs", "stomach", "liver", "kidney", "brain",
                "spine", "shoulder", "knee", "ankle", "throat", "chest"
            ],
            "medical_conditions": [
                "diabetes", "hypertension", "asthma", "heart disease", 
                "pneumonia", "bronchitis", "arthritis", "depression"
            ]
        }

    def configure_transcriber_settings(self, language_code="en", medical_focus=True):
        """Configure AssemblyAI with medical-specific vocabulary boosting"""

        # Flatten all medical terms for vocabulary boosting
        all_medical_terms = []
        if medical_focus:
            for category in self.medical_terms.values():
                all_medical_terms.extend(category)

        # AssemblyAI supports up to 1000 boosted terms
        config = aai.TranscriptionConfig(
            language_code=language_code,
            punctuate=True,
            format_text=True,
            speech_model=aai.SpeechModel.best,  # Use highest quality model
            word_boost=all_medical_terms[:1000],  # Boost medical vocabulary
            boost_param="high"  # Maximum boosting for critical terms
        )

        return config

2. Multilingual Medical Processing

GlobalCare AI supports 7 languages across 6 countries. We mapped user-friendly language names to AssemblyAI's language codes and configured each for medical accuracy:

def process_medical_audio(audio_data, country, language):
    """Process medical audio with language-specific configuration"""

    # Map user language selection to AssemblyAI language codes
    language_mapping = {
        "English": "en_us", "Spanish": "es", "Hindi": "hi",
        "Telugu": "te", "Japanese": "ja", "Arabic": "ar", "Mandarin": "zh"
    }

    language_code = language_mapping.get(language, "en")

    # Configure AssemblyAI for the specific language with medical focus
    config = speech_processor.configure_transcriber_settings(
        language_code, medical_focus=True
    )

    transcriber = aai.Transcriber(config=config)

    # Process audio with medical vocabulary boosting
    transcript = transcriber.transcribe(audio_file_path)

    return transcript

3. Real-Time Audio Processing Pipeline

Our system processes live audio recordings from Gradio's microphone input. We implemented a robust pipeline that converts numpy audio arrays to AssemblyAI-compatible formats:

def process_audio_file(self, audio_file_path, language="en", country="USA"):
    """Complete audio processing pipeline with AssemblyAI"""
    try:
        # Step 1: Configure AssemblyAI for medical accuracy
        config = self.configure_transcriber_settings(language, medical_focus=True)
        transcriber = aai.Transcriber(config=config)

        # Step 2: Process audio with Universal-Streaming
        print(f"Processing audio with AssemblyAI: {audio_file_path}")
        transcript = transcriber.transcribe(audio_file_path)

        # Step 3: Handle AssemblyAI response
        if transcript.status == aai.TranscriptStatus.error:
            return {
                "success": False,
                "error": transcript.error,
                "text": "",
                "confidence": 0.0
            }

        # Step 4: Extract medical information from transcript
        result = self._process_medical_transcript(transcript, language, country)
        return result

    except Exception as e:
        return {
            "success": False,
            "error": str(e),
            "text": "",
            "confidence": 0.0
        }

def _process_medical_transcript(self, transcript, language, country):
    """Process AssemblyAI transcript for medical analysis"""
    text = transcript.text.lower()

    # Extract medical entities from transcribed text
    medical_entities = self._extract_medical_entities(text)

    # Detect emergency keywords in transcript
    emergency_detected = self._detect_emergency_keywords(text)

    # Assess urgency based on transcript content
    urgency_level = self._assess_urgency(text, emergency_detected)

    # Get country-specific medical information
    country_info = self.knowledge_base.countries_data.get(country, {})

    return {
        "success": True,
        "text": transcript.text,
        "processed_text": text,
        "confidence": getattr(transcript, 'confidence', 0.85),
        "language": language,
        "country": country,
        "emergency_detected": emergency_detected,
        "urgency_level": urgency_level,
        "medical_entities": medical_entities,
        "country_info": country_info,
        "requires_immediate_attention": urgency_level in ["Critical", "High"]
    }

4. Emergency Detection with AssemblyAI Transcripts

We implemented intelligent emergency detection that analyzes AssemblyAI's transcription output for critical medical keywords:

def _detect_emergency_keywords(self, text):
    """Detect emergency indicators in AssemblyAI transcript"""
    emergency_keywords = [
        "chest pain", "can't breathe", "difficulty breathing", 
        "heart attack", "stroke", "unconscious", "severe bleeding",
        "overdose", "poisoning", "choking", "seizure"
    ]

    detected_keywords = []
    for keyword in emergency_keywords:
        if keyword in text:
            detected_keywords.append(keyword)

    return detected_keywords

def _assess_urgency(self, text, emergency_keywords):
    """Assess medical urgency from AssemblyAI transcript"""
    critical_indicators = [
        "can't breathe", "chest pain", "heart attack", "stroke",
        "unconscious", "severe bleeding", "overdose", "choking"
    ]

    high_indicators = [
        "severe pain", "difficulty breathing", "bleeding", 
        "high fever", "rapid heartbeat", "confusion"
    ]

    # Check for critical emergencies first
    for indicator in critical_indicators:
        if indicator in text:
            return "Critical"

    # Check for high urgency symptoms
    for indicator in high_indicators:
        if indicator in text:
            return "High"

    # Medium urgency if any emergency keywords detected
    if emergency_keywords:
        return "Medium"

    return "Low"

5. Audio Format Handling for Gradio Integration

We implemented robust audio processing to handle Gradio's numpy audio format and convert it for AssemblyAI processing:

def convert_gradio_audio_for_assemblyai(audio_data):
    """Convert Gradio numpy audio to AssemblyAI-compatible format"""
    if isinstance(audio_data, tuple) and len(audio_data) == 2:
        sample_rate, audio_array = audio_data

        # Create temporary WAV file for AssemblyAI
        temp_audio_file = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")

        try:
            import scipy.io.wavfile as wavfile
            import numpy as np

            # Convert audio to proper format
            if audio_array.dtype != np.int16:
                audio_array = np.clip(audio_array, -1.0, 1.0)
                audio_array = (audio_array * 32767).astype(np.int16)

            # Write WAV file compatible with AssemblyAI
            wavfile.write(temp_audio_file.name, sample_rate, audio_array)

        except ImportError:
            # Fallback using wave module
            import wave
            import numpy as np

            with wave.open(temp_audio_file.name, 'wb') as wav_file:
                wav_file.setnchannels(1)  # Mono audio
                wav_file.setsampwidth(2)  # 16-bit
                wav_file.setframerate(sample_rate)

                if audio_array.dtype != np.int16:
                    audio_array = np.clip(audio_array, -1.0, 1.0)
                    audio_array = (audio_array * 32767).astype(np.int16)

                wav_file.writeframes(audio_array.tobytes())

        temp_audio_file.close()
        return temp_audio_file.name

    return None

6. Integration Architecture

# Complete AssemblyAI integration workflow
def complete_voice_medical_workflow(audio_data, country, language):
    """End-to-end workflow using AssemblyAI Universal-Streaming"""

    # Step 1: Convert audio format
    audio_file = convert_gradio_audio_for_assemblyai(audio_data)

    # Step 2: Configure AssemblyAI for medical domain
    language_code = get_language_code(language)
    config = configure_medical_transcription(language_code)

    # Step 3: Transcribe with AssemblyAI
    transcriber = aai.Transcriber(config=config)
    transcript = transcriber.transcribe(audio_file)

    # Step 4: Process medical content
    medical_analysis = analyze_transcript_for_medical_content(transcript)

    # Step 5: Generate response
    return create_medical_response(medical_analysis, country, language)

AssemblyAI Universal-Streaming serves as the critical foundation that enables GlobalCare AI to accurately understand medical terminology across multiple languages, making voice-first healthcare accessible globally. The vocabulary boosting feature is particularly crucial for medical applications where precision in recognizing drug names and symptoms can have significant implications for patient care.

Teammate: @tanmaiyeevadloori

DEV Community: Zakeer

MenuMaster: Your Menu Anxiety Solution

What I Built

Demo

How I Used Google AI Studio

Multimodal Features

GlobalCare AI: International Medical Voice Assistant

What I Built

Demo

Assembly AI Demo.mov - Google Drive

GitHub Repository

Zakeertech3 / Assembly_AI_Hackathon

🏥 International AI Medical Voice Assistant

🖼️ System Overview

✨ Features

Technical Implementation & AssemblyAI Integration