DEV Community

omkar
omkar

Posted on

ClimateIQ AI – 11 Mux Features Power Climate Action for 2B+ Users

#devchallenge #muxchallenge #showandtell #video

DEV's Worldwide Show and Tell Challenge Submission 🎥

This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux

What I Built

ClimateIQ AI is a production-grade climate intelligence platform transforming satellite data into actionable insights for farmers, cities, and communities worldwide.

🌍 The Full Platform (Beyond Video)

ClimateIQ already has deep climate features:

Module Features
🌾 Smart Agriculture Multi-agent AI Crop Advisor (SoilAgent + WeatherAgent + MarketAgent), Carbon Footprint Calculator (IPCC emission factors), Solar Irrigation Planner (NREL API)
🗺️ Climate Map 6 satellite layers (NASA FIRMS fires, AQI, temperature, precipitation, NREL solar, NDVI vegetation), AI Climate Alert System (heatwave/flood/AQI warnings)
🏙️ Urban Sustainability Gemini Vision Waste Scanner, Smart Home Tracker (water/electricity), Solar ROI Calculator, Eco-Advisor with weather-based tips
📚 Climate Education Storyblok CMS learning modules, AI-generated quizzes, climate news feed
♿ Accessibility Keyboard shortcuts, voice navigation, high contrast themes

Tech Stack: Next.js 14, TypeScript, Supabase, Google Gemini AI, Mapbox, NASA FIRMS, NREL PVWatts, Planet API, SoilGrids

📹 NEW: Community Hub (Powered by Mux)

The missing piece was community-generated content. Rural communities couldn't share drought videos. Cities couldn't broadcast awareness campaigns. Climate educators needed multilingual reach.

Mux solved this with 11 integrated features:

  • 📹 Climate Stories – Upload video testimonials with auto-generated captions
  • 🌍 Climate Live – Low-latency streaming for real-time expert consultations
  • 🌐 AI Translation – Captions in Hindi, Spanish, French for global reach
  • 🎙️ Audio Dubbing – ElevenLabs voice cloning + Mux multi-track audio

My Pitch Video

Demo

🔗 Live Demo: https://climate-iq-git-feature-omkardongres-projects.vercel.app/

📦 GitHub: https://github.com/omkardongre/climate-iq/tree/feature

Test Credentials:

  • Email: test@gmail.com
  • Password: 12345678

🔓 Without Login: View videos, watch with captions, switch dubbed audio tracks, watch live streams

🔐 With Login: Upload videos, generate chapters, translate captions, dub audio, go live

How to Test Mux Features (All from the UI)

  1. Upload Video: Go to Community Hub → Click "Share Your Story" button → Select a video → Watch it upload with progress bar
  2. AI Chapters: After upload completes, click on your video → Click "Generate AI Chapters" button → See chapters appear below the player
  3. Translate Captions: Click "Translate Captions" button → Select Hindi/Spanish from dropdown → After ~40 seconds, check the CC menu in the video player
  4. Dub Audio: Click "Dub Audio" button → Select language → Wait ~2 minutes → Check the Audio/Language menu in the player settings (gear icon)
  5. Live Streaming: Click "Go Live" button → Copy the RTMP URL and Stream Key → Use OBS or any streaming software to broadcast

The Story Behind It

The Climate Data Gap: Satellites collect terabytes of climate data daily. But farmers facing crop failures, cities battling pollution, and communities seeking education can't access it. ClimateIQ bridges this gap.

Why Video Was Missing:

  • A community in India showing drought damage creates more impact than statistics
  • Live broadcasts enable real-time expert consultations during crises
  • 2B+ potential users need content in their language

Why Mux:

  • We needed auto-captions for accessibility
  • Low-latency for real-time consultations
  • Multi-language support for global reach
  • Production-ready infrastructure in days, not months

Technical Highlights

11 Mux Features Integrated:

# Mux Feature Implementation
1 @mux/mux-player-react All video playback with ClimateIQ theming
2 @mux/mux-uploader-react Direct uploads (no backend file handling)
3 Auto-Generated Captions generated_subtitles in upload config
4 Live Streaming API muxVideo.liveStreams.create()
5 Low Latency reduced_latency: true for live broadcasts
6 Image API Thumbnails + animated GIFs on hover
7 @mux/ai Chapters Official SDK for chapter generation via Gemini
8 Text Tracks API assets.createTrack() for translated captions
9 Audio Tracks API assets.createTrack({ type: 'audio' }) for dubs
10 Master Access Temporary signed URLs for AI processing
11 Mux Data Analytics Custom dimensions: climate_topic, user_region

Use of Mux (Additional Prize Category)

Mux isn't just hosting videos – it's the backbone of our entire community feature.

🎬 Beyond Basic Hosting

1. AI-Powered Chapters (@mux/ai SDK)

import { generateChapters } from '@mux/ai/workflows';
const chapters = await generateChapters(assetId, 'en', { provider: 'google' });
Enter fullscreen mode Exit fullscreen mode

Climate videos get auto-chapters like "Impact on Agriculture" or "Mitigation Strategies" – helping users navigate long content.

2. Multi-Track Audio Dubbing

await muxVideo.assets.createTrack(assetId, {
  type: 'audio',
  url: supabaseAudioUrl,
  language_code: 'hi',
  name: 'AI Dubbed (hi)'
});
Enter fullscreen mode Exit fullscreen mode

ElevenLabs generates dubbed audio → we add it to the Mux asset → users switch languages natively in the player.

3. Low-Latency Live Streaming

await muxVideo.liveStreams.create({
  reduced_latency: true,
  reconnect_window: 60,
  generated_subtitles: [{ language_code: 'en', name: 'English (auto)' }]
});
Enter fullscreen mode Exit fullscreen mode

Users broadcast with low-latency. Experts provide real-time advice during live sessions.

4. Auto-Captions + AI Translation Pipeline

  1. Upload with generated_subtitles → Mux generates English captions
  2. Fetch VTT from https://stream.mux.com/{playbackId}/text/{trackId}.vtt
  3. Gemini translates to Hindi/Spanish
  4. Upload to Supabase + add via assets.createTrack()

📊 What Makes This Deep Integration

We're not just hosting videos. We're using:

  • Official @mux/ai SDK for chapter generation
  • Multi-Track Audio for dubbed languages
  • Text Track API for translated captions
  • Master Access for AI processing pipelines
  • Low-Latency Live Streaming for real-time advice
  • Custom Analytics Dimensions for climate insights

🌍 Real Impact

User How Mux Helps
Rural Communities Upload climate impact videos with auto-captions
Experts Watch live broadcasts, give real-time advice
Global Audience Content in Hindi, Spanish, French via AI translation
Non-Readers Audio dubbing for accessibility

Mux handles encoding, CDN, multi-track, and AI – so we focus on climate impact.

❗ By submitting this project, I confirm that my video adheres to Mux's terms of service: https://www.mux.com/terms

Top comments (1)

Collapse
 
omkar598 profile image
omkar •

Due to a last-minute submission rush, the demo video playback speed is faster than intended. Viewers are requested to watch the video at a slower speed for better clarity, as updates to the submission or video are not permitted after the deadline.
Thank you for your understanding.