A private AI copilot for phone calls that spots scams as they happen and runs entirely on your laptop.
Inspiration
Most of us have received a call like this: "I'm calling from your bank, your account is blocked, please share the OTP." Sometimes it's a fake courier, sometimes it's someone claiming to be the CBI saying you're under "digital arrest". These scams work because they happen in real time, under pressure, with no one around to help.
Existing protections act either before the call (spam-number labels) or after the damage is done (reporting on 1930 or cybercrime.gov.in). Nothing helps during the conversation, which is exactly when the decision is made. Cloud call assistants could help, but sending private phone calls to a server isn't something people should have to accept.
Small open-weight models can now run on an ordinary 8 GB laptop, so we wanted to build a real-time call assistant that keeps everything private.
What it does
CallPilot AI sits alongside a phone call on your laptop:
Two-lane live transcript. Your voice (laptop mic) and the caller's voice appear side by side as you speak.
Live scam detection with evidence. It flags requests for OTPs, UPI PINs, passwords, remote-access apps, KYC updates, "digital arrest" threats, parcel scams and money transfers. Each alert quotes the caller's exact words.
Context, not keywords. "Never share your OTP" isn't flagged; "Tell me your OTP" is.
Hang-up banner. When a call clearly matches a scam pattern, it shows one clear action: hang up, call your bank on its official number, and report on 1930.
Private assistant. Ask questions during the call ("Do I have to share the code?"). Answers are visible only to you and grounded in the conversation or your own notes.
Auto-suggested replies on normal work calls, based on your notes.
After-call summary. A summary, a list of what the caller asked for, and follow-up tasks you can approve.
Shareable call report for family, your bank or the police. Private notes are excluded.
Live phone calls. A phone joins by scanning a QR code, over an iPhone Personal Hotspot. No accounts or cloud services are needed.
The caller never sees any AI output, and nothing leaves your laptop.
How we built it:
Frontend: React, TypeScript and Vite.
An AudioWorklet captures both audio lanes, and WebRTC carries the live phone call.
Backend: Python with FastAPI, bound to localhost only. The phone uses a separate HTTPS guest app with no access to transcripts or AI output.
Speech-to-text: whisper.cpp with Whisper base.en, plus Silero VAD to split speech into sentences.
AI: Qwen3 1.7B through Ollama handles live scam classification, the private assistant and the summary. Gemma 4 E2B can optionally write the after-call summary.
Safety design:
Rules flag risky phrases instantly.
The model only classifies intent and must cite transcript line IDs.
The warning text comes from a vetted table, so the model never writes safety advice itself.
Engineering:
CI on every pull request.
Over 150 automated tests, including tests against the real local models.
A one-command setup (./start), tested on fresh macOS and Linux machines.
Challenges we ran into
8 GB of RAM. Running two models during a call caused constant reloading, and alerts took up to 21 seconds. We redesigned around a single small model.
Latency. Having the model write explanations made alerts slow (13.6 s). Asking it only to classify, with advice from a fixed table, brought this down to about 5.8 s.
Small models make mistakes. One answered "Yes, you must share the OTP", and another drifted into Chinese mid-sentence. We added deterministic guards in code to block both.
Whisper on echo and silence. It looped or invented text, so we added filtering and switched to Silero VAD.
Phone mic access without a router or domain. Browsers require HTTPS for the mic, so we created a local certificate authority restricted to the hotspot network only.
Accomplishments that we're proud of
20/20 on our scam test set, with 0 missed scams and 0 false alarms.
The whole pipeline (speech, AI and summaries) runs offline on an 8 GB laptop.
The caller can never see the AI. This is enforced by the architecture and covered by tests.
We benchmarked six small models on the same machine and chose based on the results.
Judges can run it with a single command: ./start.
What we learned
Measuring beats guessing. Our assumption that a bigger model would be better was wrong on 8 GB.
For safety-critical output, prompts aren't enough. You need checks in code.
Generation time, not prompt processing, dominates latency on small models.
Privacy is a design choice that shapes everything from network binding to what the phone is allowed to load.
What's next for CallPilot AI
Support for Hindi, Telugu and Hinglish speech.
Running directly on Android phones.
Optionally alerting a trusted family member during a suspected scam call.
A larger, more diverse scam test set.
Built with
python · fastapi · react · typescript · vite · webrtc · whisper.cpp · ollama · qwen3 · gemma-4 · silero-vad · onnxruntime · numpy · scipy
Links
GitHub: https://github.com/ManoharPaturi/techie-cooks-callpilot-ai
Team
Techie Cooks: Manohar P (Team Lead), Sainath B, Pushpak K, Laasya B
Top comments (0)