No data or access was compromised.
On the 20th of May at 6:45 AM, I got a phone call alert from Spike.sh from our Twilio integration about a surge in calls.
Scammers abused our phone verification by otp generating thousands of phone calls. This qualifies as Toll fraud
Timeline of how it went down
7 AM 👨💻
Within the next 15 minutes, I was awake and glaring into my screen trying to find out what the hell is happening.
I learn about these keys facts -
40 new registrations in the last one hour using disposable email addresses, mainly Mailinator (.com and .net)
Our Twilio account had been recharged which was unusual
So far, I was knee-deep with $107 bill on Twilio because scammers had made about hundreds of phone calls to a number in Moldova
Also, Twilio, being awesome, detected suspicious activity and blocked several phone calls by that time. I had emails from their Fraud detection team. Thank you Twilio
7:15 AM 😧
After digging into logs, I learn that the scammers were generating phone calls from our Phone verification section. We make phone calls to verify your phone number after signing up on Spike. I knew this was not the best idea going in too... 🙁 We now send SMS to verify instead of making a phone call.
There were two such sections. Both of them now look like this -
7:30 AM 💸
I got a call from our credit card company who inform us we have been charged about $870 in total so far 😭
We have barred making calls to Moldova and some other regions scammers had been targeting using Twilio's GeoPermissions
By default, Twilio blocks voice call to some high-risk areas.
And this is us blocking Moldova.
All in all, about 2400+ phone calls were made so far.
7:45 AM 🤖
Integrate Google reCaptcha on the dashboard. Block spam user's accounts. I was personally very sceptical if things would go back to normal.
When you do this, make sure to authorise the request on the backend too.
8:00 AM 🚫
So far, I have made about 20 deployments to production, thanks to GitHub Actions for automating that.
Replace phone call verification with SMS verification. This took some time along with testing.
9:00 AM 📖
Revoke all Twilio tokens and read their guide again on toll frauds. Are we missing anything?
Facepalm - Twilio has explicitly written in their docs to not use a phone call for verification via OTP and that's exactly what we did 🤦♀️
We released some of our old phone numbers and bought new ones. This didn't take long.
10:00 AM 🧟♂️
Deploy to production but we still see a number of signups coming in via bots using Mailinator email addresses.
11:00 AM 🕵️♀️
Our search for a service API to tell disposable email addresses apart has begun.
Shortlisted:
I would like to know of the possible ways to block disposable email addresses from registering in my website.
For simplicity, let's take the example where the registration form of the website is done with HTML and PHP.
Any ideas, solutions or suggestions would be greatly appreciated.
The problem is creating newer disposable email addresses is easy. It's perhaps too much to expect that one service would cover all of them.
Interesting stat - A ton of our users were using Google OAuth, so we made the bold move.
Allow only Google OAuth. Remove login and signup using Email/Password combinations.
1:00 PM 👮♂️
Time to take extra measures -
- Activate Cloudflare DDoS protection with I'm under attack mode.
Cloudflare is pretty awesome at this. When you visit Spike.sh, Cloudflare will try to determine if you are human or a bot. Sometimes, a test also appears for verification using hcaptcha.
- We installed Needle.sh, which provides a security layer with their NPM module. In their dashboard, we noticed that security scanners have been probing our web properties for vulnerabilities multiple times a day. This made us realise the importance of using security tools for our web products from an early stage, obviously.
Rate limiting was super helpful and easy to setup. I have seen tons of 429 response status after we activated this.
Setup usage triggers on twilio
3:00 PM 🤞
All of our measures are now live. Now - it's all about monitoring carefully.
Scammers were unable to create new accounts. Yipee!
Some literally created accounts using Google OAuth and tried to make more calls but over 99% of them ended up in either 401 or 429 response codes.
What did we learn?
- Setup critical alerts, had I not gotten the phone call from Spike.sh, I easily would have gotten a bill of more than $5000 by the time I woke up.
- Security is super important. Never too early to start.
- Use the right tools for the right job, we are using Cloudflare and Needle.sh
- Better safe than sorry
Resources
These are some good reads I came across during the entire ordeal. Hope this helps you -
- Cloudflare DDoS
- Needle.sh
- SSL error with Letsencrypt
- https://block-temporary-email.com
- SO question
- Detecting headless browsers
- Impossible to detect headless browsers
This doesn't cover all the steps we have taken to stop abuse or DDoS.
Top comments (4)
Thanks for sharing your experience and the details of this event, I think we can all learn from it. Glad you were able to pick up on it relatively quickly too, it certainly appears like your site caught interest for abuse and they were targeting you for some time.
I had to deal with scammers and malicious actors when I was building one of my projects and wrote about it here on dev. I shared the code I wrote to help others that are dealing with the same problem or likely will.
Blocking temporary emails and using reCAPTCHAv3 has very effective for preventing abuse on my site. Thanks again for sharing your experience and advice.
Great job 👏
Is there any service that would essentially only tel us reliably if given email is disposable or not?
Wow, swift work to head off this abuse. Sorry you found yourself in the firing line of scammers like this.
As a suggestion, Twilio SendGrid has an email validation API which will return whether an email address is disposable, along with a bunch of other data and a confidence score for the address as a whole. That might be of use to you.
I wasn't aware of SendGrid's email validation. We use SendGrid already for all transactional emails, will use this add on too. Thanks