This is a submission for the AssemblyAI Challenge : No More Monkey Business.
What I Built | 🚨ReportSOS 🆘
Help! Help! Emergency here! 🗣
Can you imagine how hectic it must be for dispatchers to handle and interpret hundreds of emergency calls every single day? That’s why I created 🚨ReportSOS, an app designed to streamline the chaos!
With 🚨ReportSOS, emergencies get sorted faster and smarter. The app lets users report incidents in just a few taps, providing crucial details like location, type of emergency, and even summaries. It’s like giving dispatchers a superpower—cutting through the noise and delivering the right help to the right place, right on time. Whether it’s a fire, an accident, or a cat stuck in a tree, 🚨ReportSOS has your back! 🆘✨
Demo
🚨ReportSOS Demo
Admin page :
🚨ReportSOS Demo Admin
Username: reportagent
Password: assemblyai
Assembly API Key: your own api key
User Feature
▶Start Page
To use 🚨ReportSOS, users can either press the "Help Me" button or shake their phone vigorously until it vibrates 4 times.
🎙Voice Recorder
Try using the microphone to report emergencies around you. You can also manually upload reports directly from your file manager.
🏠Location
Access to your location is necessary to determine your precise position, ensuring that emergency services or responders can reach you quickly and efficiently.
...And many more! Check out the full details in the video above.
Dispatcher and Emergency Team Feature
🔑Authentication Page
Includes fields for a username, password, and AssemblyAI API key.
This page ensures secure access by verifying user credentials and API keys before granting access to the system.
🖥Dashboard Page
This feature includes a "Latest Emergency" section that highlights the most recent emergencies.
On the right side of this section, users can find detailed information such as the location description, images, audio, a summary of the event, emergency status, casualty details, key points, and more.
This layout ensures users have a clear and organized view of all critical information in one place, enabling faster understanding and response. Additionally, the section is designed to be interactive, allowing users to drill down into specific details or share reports directly.
Code
Frontend Code
React + Vite
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
Currently, two official plugins are available:
- @vitejs/plugin-react uses Babel for Fast Refresh
- @vitejs/plugin-react-swc uses SWC for Fast Refresh
Backend Code
Express API Starter
How to use this template:
npx create-express-api --directory my-api-name
Includes API Server utilities:
-
morgan
- HTTP request logger middleware for node.js
-
helmet
- Helmet helps you secure your Express apps by setting various HTTP headers. It's not a silver bullet, but it can help!
-
dotenv
- Dotenv is a zero-dependency module that loads environment variables from a
.envfile intoprocess.env
- Dotenv is a zero-dependency module that loads environment variables from a
-
cors
- CORS is a node.js package for providing a Connect/Express middleware that can be used to enable CORS with various options.
Development utilities:
-
nodemon
- nodemon is a tool that helps develop node.js based applications by automatically restarting the node application when file changes in the directory are detected.
-
eslint
- ESLint is a tool for identifying and reporting on patterns found in ECMAScript/JavaScript code.
-
jest
- Jest is a delightful JavaScript Testing Framework with a focus on simplicity.
-
supertest
- HTTP assertions made easy via superagent.
Setup
npm install
Lint
npm run
API & Technology 💻
- React for building a sleek and interactive frontend.
- react-voice-visualizer for recording voice
- react-webcam for capture image
- Tailwind CSS to make styling a breeze
- Express.js powering the backend like a boss.
- Neon as our go-to database—because who doesn’t love a fast, modern DB?
- OpenStreetMap: Using Overpass API to find nearby emergency locations and Nominatim for reverse geocoding magic.
- AssemblyAI, because transcribing audio and handling LLM tasks is their superpower.
Journey
How can I implement LeMUR in Assembly AI?
- Firstly, I convert audio to text and use entity detection and IAB category detection to identify topics. For the example, here i using AssemblyAI Playground to test the audio.
- From the transcript that already been converted before, I utilize LeMUR with a custom prompt as follows:
{
"context": "This is an emergency call for assistance.",
"answer_format": `{
"activities":"very short sentence",
"summary": "",
"key_points": [],
"tone": "",
"casualty":"",
"status":"Critical"|"Urgent"|"Non-Urgent"
}`
}
The custom prompt above is designed to ensure that the output stays within a JSON structure, making it easy to parse and integrate into the application.
So here the example of output
{
"activities": "Multiple vehicle accidents on I-75 in poor visibility conditions.",
"summary": "Caller reports a series of vehicle accidents occurring on I-75 in very low visibility conditions. At least 4 vehicles are involved, including a truck. The situation is chaotic and rapidly evolving.",
"key_points": [
"Multiple accidents occurring on I-75",
"At least 4 vehicles involved, including a truck",
"Very poor visibility making it hard to see",
"Situation is ongoing and chaotic"
],
"tone": "Panicked and distressed",
"casualty": "Unknown number of injuries",
"status": "Critical"
}
The data is stored in a PostgreSQL database using the Neon service to avoid retrieving duplicate data, thereby saving API credits.
Additional Feature⚡
To complete this app and turn it into a superapp for reporting emergencies, here’s what I’ve done:
Media Integration
This app uses both camera and voice to provide a complete report for dispatchers, ensuring that all the critical details are captured accurately. I’m using React-Webcam to enable users to capture images or videos of the emergency situation. For voice reporting, I’m using React-Voice-Visualizer to record and transcribe spoken information.Emergency Mapping
I use React-Leaflet to display a spatial map. Additionally, I leverage the Overpass API from OpenStreetMap.org to search for nearby emergency offices, such as police stations, fire stations, and hospitals, using the following query:
[out:json][timeout:25];
nwr["amenity"~"^(hospital|fire_station|police)$"](south, west, north, east);
out geom;
Thus, the 🚨ReportSOS application is equipped with comprehensive features, making it a complete solution for reporting emergency situations. 😎
- User Information
After users report an emergency they are experiencing, the application will provide examples of first aid steps that can be taken immediately. It will also display the nearest emergency offices, such as hospitals, police stations, and fire stations.
Team Member
Just me and Myself😁✌🏻
Thanks for reading! It’s been an exciting experience trying out Assembly AI—a whole new venture for me. Apologies if there are any shortcomings in the app; of course, it can be developed and improved further. I hope you enjoyed exploring it as much as I did! Goodbye! 😊👋
Top comments (3)
An interesting idea, and usage case, although emergency call taking systems are pretty advanced and integrated already (even if they aren't yet exposing alot of that to the callers).
One item I would raise would be that your solution needs to be able to tolerate errors/limitations without blocking the caller from progressing. If, for instance, I don't give permission to use my camera, at the moment my progress dead-ends - I cannot complete the interaction if I either don't want to send a picture, or if my phone is not capable of doing so.
Hopefully, your system also starts processing the report from the first possible point in the interaction, with additional info like the photo just being added to the report as that info becomes available. With real emergency dispatchers, as soon as they know the location and the rough details, they are already passing the job to the nearest suitable and available assets. Seconds matter.
Thanks for reading🙌🏻
This is good! Great job 💪🏽