DEV Community: Spike.sh

Scammers signed up, scammed us of $870 in 15 minutes.

Kaushik Thirthappa — Tue, 26 May 2020 13:16:59 +0000

No data or access was compromised.

On the 20th of May at 6:45 AM, I got a phone call alert from Spike.sh from our Twilio integration about a surge in calls.

Scammers abused our phone verification by otp generating thousands of phone calls. This qualifies as Toll fraud

Timeline of how it went down

7 AM
7:15 AM
7:30 AM
8 AM
9 AM
10 AM
11 AM
1 PM
3 PM
What did we learn?
Resources

7 AM 👨‍💻

Within the next 15 minutes, I was awake and glaring into my screen trying to find out what the hell is happening.

I learn about these keys facts -

40 new registrations in the last one hour using disposable email addresses, mainly Mailinator (.com and .net)
Our Twilio account had been recharged which was unusual
So far, I was knee-deep with $107 bill on Twilio because scammers had made about hundreds of phone calls to a number in Moldova
Also, Twilio, being awesome, detected suspicious activity and blocked several phone calls by that time. I had emails from their Fraud detection team. Thank you Twilio

After digging into logs, I learn that the scammers were generating phone calls from our Phone verification section. We make phone calls to verify your phone number after signing up on Spike. I knew this was not the best idea going in too... 🙁 We now send SMS to verify instead of making a phone call.

There were two such sections. Both of them now look like this -

7:30 AM 💸

I got a call from our credit card company who inform us we have been charged about $870 in total so far 😭

We have barred making calls to Moldova and some other regions scammers had been targeting using Twilio's GeoPermissions

By default, Twilio blocks voice call to some high-risk areas.

And this is us blocking Moldova.

All in all, about 2400+ phone calls were made so far.

7:45 AM 🤖

Integrate Google reCaptcha on the dashboard. Block spam user's accounts. I was personally very sceptical if things would go back to normal.

When you do this, make sure to authorise the request on the backend too.

8:00 AM 🚫

So far, I have made about 20 deployments to production, thanks to GitHub Actions for automating that.

Replace phone call verification with SMS verification. This took some time along with testing.

9:00 AM 📖

Revoke all Twilio tokens and read their guide again on toll frauds. Are we missing anything?

Facepalm - Twilio has explicitly written in their docs to not use a phone call for verification via OTP and that's exactly what we did 🤦‍♀️

We released some of our old phone numbers and bought new ones. This didn't take long.

10:00 AM 🧟‍♂️

Deploy to production but we still see a number of signups coming in via bots using Mailinator email addresses.

11:00 AM 🕵️‍♀️

Our search for a service API to tell disposable email addresses apart has begun.

Shortlisted:

Jun 11 '12 Comments: 4 Answers: 11

I would like to know of the possible ways to block disposable email addresses from registering in my website.

For simplicity, let's take the example where the registration form of the website is done with HTML and PHP.

Any ideas, solutions or suggestions would be greatly appreciated.

Open Full Question

The problem is creating newer disposable email addresses is easy. It's perhaps too much to expect that one service would cover all of them.

Interesting stat - A ton of our users were using Google OAuth, so we made the bold move.

Allow only Google OAuth. Remove login and signup using Email/Password combinations.

1:00 PM 👮‍♂️

Time to take extra measures -

Activate Cloudflare DDoS protection with I'm under attack mode.

Cloudflare is pretty awesome at this. When you visit Spike.sh, Cloudflare will try to determine if you are human or a bot. Sometimes, a test also appears for verification using hcaptcha.

We installed Needle.sh, which provides a security layer with their NPM module. In their dashboard, we noticed that security scanners have been probing our web properties for vulnerabilities multiple times a day. This made us realise the importance of using security tools for our web products from an early stage, obviously.

Rate limiting was super helpful and easy to setup. I have seen tons of 429 response status after we activated this.
Setup usage triggers on twilio

3:00 PM 🤞

All of our measures are now live. Now - it's all about monitoring carefully.

Scammers were unable to create new accounts. Yipee!
Some literally created accounts using Google OAuth and tried to make more calls but over 99% of them ended up in either 401 or 429 response codes.

What did we learn?

Setup critical alerts, had I not gotten the phone call from Spike.sh, I easily would have gotten a bill of more than $5000 by the time I woke up.
Security is super important. Never too early to start.
Use the right tools for the right job, we are using Cloudflare and Needle.sh
Better safe than sorry

Resources

These are some good reads I came across during the entire ordeal. Hope this helps you -

This doesn't cover all the steps we have taken to stop abuse or DDoS.

Excerpt of 7 regrets from creator of Deno about Node.js

Kaushik Thirthappa — Fri, 22 May 2020 06:37:48 +0000

✨ This post has been copied from the YouTube comment with some edits.

1. 🤷‍♂️ No Promises

Promises were added in June 2009 but removed "foolishly" in Feb 2010
Promises are the necessary abstraction for async/await.
It's possible unified usage of promises in Node would have sped the delivery of the eventual standardization and async/await.

2. 👮‍♀️ Security

V8 by itself is a very good security sandbox
Node apps outside of the browser shouldn't need to have all the permissions like writing to disk and network.
Example: Your linter shouldn't get complete access to your computer and network.

3. 👷‍♀️ The Build System (GYP)

Build systems are very difficult and very important.
V8 (via Chrome) started using GYP and Node uses the same.
Later Chrome dropped GYP for GN. Leaving Node the sole GYP user.
GYP is not an ugly internal interface either - it is exposed to anyone who's trying to bind to V8.
It's an awful experience for users. It's this non-JSON, Python adaptation of JSON.
The continued usage of GYP is the probably largest failure of Node core because there are just too many wrappers to make it work.
Instead of guiding users to write C++ bindings to V8, I should have provided a core foreign function interface (FFI)

4. 🗄 Package.json

Isaac, in NPM, invented package.json (for the most part)
But I sanctioned it by allowing Node's require() to inspect package.json files for "main"
NPM in node distribution means it is the de-facto standard now. Also remember, NPM, a centralized repository is now privately controlled.
package.json now includes all sorts of unnecessary information. License? Repository? Description? It's boilerplate noise.
If only relative files and URLs were used when importing, the path defines the version. There is no need to list dependencies.

Too much boilerplate noise

5. 🗃 node_modules

Complicated module resolution algorithm.
vendored-by-default has good intentions, but in practice just using $NODE_PATH wouldn't have precluded that.
Deviates greatly from browser semantics. can't undo now

6. 🧩 require("module") without ".js" extension

Needlessly less explicit.
Not how browser javascript works. You cannot omit the ".js" in a script tag src attribute.
The module loader has to query the file system at multiple locations trying to guess what the user intended.

7. 🧟‍♂️ index.js

Inspired from index.html and though index.js is cute. A default file to be loaded might reduce complexity but...
It needlessly complicated the module loading system.
It became especially unnecessary after require supported package.json

✨ If you think something is kinda cute and not completely necessary then don't add it. This comes with experience.

Ryan Dahl created Deno with all the above things kept in mind. Deno is focused on security without having to give access to everything, kept simple, comes with Typescript enabled.

This is Ryan's talk 👉

Monitoring your cron jobs is critical

Kaushik Thirthappa — Tue, 19 May 2020 08:50:05 +0000

At Spike, we have a few supercritical cron jobs.

Some use cases of cron for us has been -

Send weekly email reports
Billing reminders
DB backups
Bill organisation for the right amount based on active members every month (flexibility to add/remove users and reflect on billing is super important)

Basically cron is important and we need to be alerted when they fail.

There are several tools that will help you with cron monitoring. Here is a comparison chart

What should you setup?

Timeouts
Insert cron expression to check if cron has executed
Alerting upon failure

Crontab.guru is the best cron expression validator out there. It will convert your expression into human-readable format and validate it for you.

How to setup?

Once you have your expression, you can use one of the cron monitoring tools to validate if the cron was executed and run successfully or not.

We ourselves use Healthchecks the most and have absolutely no complains in the past year of using it. Quite sure the other products are great too.

Give it a spin and let us know.

ps: Spike integrates with healthchecks.io very easily. We alert you so you can stay on top of your cron jobs execution.

Simple git aliases for daily purpose

Kaushik Thirthappa — Sat, 16 May 2020 04:54:12 +0000

On average, every developer commits their code atleast once a day.

Having some git aliases help you quickly make changes and push your code.

No more typos

One of my daily frustrations in the past 🤬 -

Avoid common pitfalls are super important, it helps increase productivity in the long term.

git status
git commit
git push
git add and commit
git add, commit and push
git stash
git stash apply
git push --set-upstream
git checkout

ps: These are aliases I have been using personally, feel free to create and share your own ☺️

1. git status 👉 gs

Before -

git status

After -

gs

Setup code -

alias gs='git status'

2. git commit 👉 gc

Before -

git commit -m 'Adding integration test cases'

After -

gc 'Adding integration test cases'

Setup code -

alias gc='git commit -m $2'

3. git push 👉 gp

Before -

git push

After -

gp

Setup code -

alias gp='git push'

4. git add . && git commit -m 👉 gac

Before -

git add .
git commit -m 'Adding integration test cases'

After -

gac 'Adding integration test cases'

Setup code -

alias gac='git add . && git commit -m $2'

5. git add . && git commit -m && git push 👉 gacp

Before -

git add .
git commit -m 'Adding integration test cases'
git push

After -

gacp 'Adding integration test cases'

Setup code -

alias gacp='git add . && git commit -m $2 && git push'

6. git stash 👉 gst

Before -

git stash

After -

gst

Setup code -

alias gst='git stash'

6. git stash apply 👉 gsta

Before -

git stash apply

After -

gsta

Setup code -

alias gsta='git stash apply'

6. git push --set-upstream origin 👉 gpst

This syntax is the one I forget the most 🤷‍♀️

Before -

git push --set-upstream origin integration-tests

After -

gpst integration-tests

Setup code -

alias gpst='git push --set-upstream origin $1'

6. git checkout 👉 gco

Before -

git checkout staging

After -

gco staging

Setup code -

alias gco='git checkout $1'

These are the aliases that I use everyday for the past few years. No typos, no errors and super handy.

⚡️Tools and products we use at Spike - part 2

Kaushik Thirthappa — Mon, 11 May 2020 01:45:28 +0000

We believe in outsourcing complex jobs to specific products. This a small list of other tools and products we use.

👈 First part on all the tools and products used by engineering.

1. Intercom

We got into Intercom's early stage program to access the entire platform. We mainly use this for customer support using Live chat and user onboarding.

2. GitBook

After having tried multiple products to host our documentation, we finalised GitBook. It's intuitive, has a great UI and super flexible. Our customers love our docs.

3. 1Password

We didn't think we would use this as much as we do. This is by far the product we use the most other than Gmail. We recommend orgs and individuals to use 1Password to stay safe and secure.

4. Notion

Roadmap, sprints, extensive documentation for features and team collaboration all on Notion. It's literally capable of everything our teams needs in one place.

5. Headway

We display our changelog on the website and on the dashboard. There are mixed opinions about public changelog on the internet but for us, it's easy to keep our customers up to date about the latest changes on our platform.

6. Figma

Frankly, the only reason we started with this was because it was free. (not kidding). However, Figma has played a HUGE role in how we work. At Spike, we take design very seriously and Figma has been an ultimate solution.

7. SendX

We use SendX for our promotional emails. Easy, affordable and perhaps the best support of all the other products we have used. Super SME friendly.

⚡️Tools and products we use at Spike - part 1

Kaushik Thirthappa — Mon, 11 May 2020 01:45:08 +0000

Every organisation uses a multitude of a number of tools to support their business. We will try and cover all of the ones we use at Spike

This post has all the DevTools or say products used by engineering.

Second part on all the marketing, support products we use 👉

PS: not covering our tech stack here

Servers

1. AWS

This is pretty clear. AWS is reliable and has over 160+ products 😳. From deployments, hosting, storage and monitoring. All of it on AWS.

// Detect dark theme var iframe = document.getElementById('tweet-1013959156678578176-838'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1013959156678578176&theme=dark" }

2. Vercel (formerly Zeit.co)

We use this for all of our serverless needs. Why didn't we directly use AWS lambda instead? Well, only because the setup on Vercel is so damn easy. Highly recommend them.

3. MongoDB Atlas

For all our DB hosting needs. We haven't moved all of it yet but soon we will finish this migration.

Monitoring

1. Cloudwatch

We monitor a bunch of things from network I/O, CPU, Disk space, Autoscaling, etc.

2. Healthchecks.io

We have a few cron jobs running at different schedules. While setting cron in Linux is easy, we still monitor it via Healthchecks to make sure it is actually running successfully. You can set your cron expression and they will make sure they are getting a ping ensuring it's all good. Super awesome tool.

3. Uptime Robot

Uptime monitoring to make sure all of our endpoints are up and running. Some endpoints everyone should monitor 👉

Website
Dashboard
DB endpoint
Cache endpoint

Good alternative is Apex Ping

4. Datadog

For our Application Performance Monitoring. When incidents happen, we want to make sure that alerts reach to you within one second. Datadog helps give us great insights on how our application peaks at certain times.

Why Datadog and not New Relic?

// Detect dark theme var iframe = document.getElementById('tweet-1257936993956192259-421'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1257936993956192259&theme=dark" }

5. Nginx Amplify

To monitor our web server. Nginx is clean, easy to setup and has a lot of plugins you can use.

6. Sentry

For all our error monitoring. They provide a ton of features and we have hardly ever been disappointed. Integrates with most languages and frameworks, catches all exceptions on serverless too 👏

tip: send additional data and use tags to identify your errors better with Sentry

CI/CD

1. Buddy.works

We rely the most on Buddy for one simple reason - the UI.

You don't have to spend a lot of time configuring on Buddy or even writing YAML files to get it running. This is easily the most underrated tool we use.

2. Travis CI

This is in our wishlist. We have been writing a lot of test cases in the past couple of months and in the near time we will be integrating Travis. shoutout to Kent 👉 for building testingjavascript.com

Kent C. DoddsFollow

My name is Kent C. Dodds and I'm a Remix Co-Founder, JavaScript engineer, and teacher. I'm also active in the open source community. I like my family, JavaScript, and React.

DevTools

1. Ngrok

Clean, succinct and does the job well to expose our localhost urls so we can build integrations we ease. We have reserved about 4 subdomains on ngrok so we don't have to spend time on reconfiguring the webhook urls for dev environment.

2. VS Code Live Share

Great for remote teams, we usually have a Google Meet call and pair program many-a-times using VS Code Live Share.

3. Twilio

Easy to integrate with extensive documentation that you will actually read. Easily a very reliable platform for us to alert customers via Phone and SMS.

4. Sendgrid

We send all email notifications using Sendgrid. Reliable, easy to use SDK and so far not that expensive.

Non-developer tools are next in this short series.

Why we built Spike? An alternative to PagerDuty and Opsgenie you will love 💙

Kaushik Thirthappa — Thu, 07 May 2020 03:38:51 +0000

First things first - we are starting to write our thoughts on Dev.to 🎉

Incident management for dev teams is a slightly confusing term. The core of the term is managing incidents properly.

We think of incidents as an issue that will or is currently affecting your customers to use your product. We understand that no matter the size of your business, this affects you

A simple overview

Before Spike

Incident occurs
😡 frustration mounts
Contacts support (or you)
🤷‍♀️ they have no idea why this happened
You 👩‍💻 identify there is an incident
Get back to support or to customer
Customer's wait time for resolution begins ⏳

After Spike

Incidents happen
We alert instantly via phone call/sms
You resolve them
customers stay unaffected 😇

Most other players are majorly focused on Enterprises.

When you build for enterprises your product becomes bloated with tons of features for customisations and loose ends that it becomes difficult to use if you are not an enterprise

Spike is built on simplicity and reliability at it's core and completely focused on SMBs.

Simplicity

No really, how do you focus on simplicity?

Making it easy to use
Designed for dev teams
Building features that our customers will actually use

Honesty

This is expected from every product you use but it's kinda rare.

No fake promises to seal the deal
No charging based on number of incidents or alerts
No constant pings to upgrade your plan. We only have plan
Cancel your plan with a click of a button
Your data is yours, privacy is super important.

Flat pricing

One plan for the entire platform with access to all features makes a lot of sens to us.

Why?

Benefit from the entire platform
Unlimited integrations. Monitor for incidents across the entire platform
Unlimited alerts. Imagine not getting alerts when your customers are affected only because you need to upgrade... Yikes!!
Stop guessing your bill at the end of the month

No matter the size of your business, when incidents occur they affect your customers.

Bootstrapped

Yup! We are bootstrapped. We have a small handful of paying customers over the past couple of months.

No VCs, no pressure translates to 👉 focusing on customers and growing at our own pace

More about us 👉 🌏Spike, Twitter

Quick note -
We will continue to share our experiences on tech and product as we grow Spike here. Let us know what you would like to hear.

DEV Community: Spike.sh

Scammers signed up, scammed us of $870 in 15 minutes.

Timeline of how it went down

Allow only Google OAuth. Remove login and signup using Email/Password combinations.

Excerpt of 7 regrets from creator of Deno about Node.js

1. 🤷‍♂️ No Promises

2. 👮‍♀️ Security

3. 👷‍♀️ The Build System (GYP)

4. 🗄 Package.json

5. 🗃 node_modules

6. 🧩 require("module") without ".js" extension

7. 🧟‍♂️ index.js

Monitoring your cron jobs is critical

What should you setup?

How to setup?

Simple git aliases for daily purpose

Table of contents

1. git status 👉 gs

2. git commit 👉 gc

3. git push 👉 gp

4. git add . && git commit -m 👉 gac

5. git add . && git commit -m && git push 👉 gacp

6. git stash 👉 gst

6. git stash apply 👉 gsta

6. git push --set-upstream origin 👉 gpst

6. git checkout 👉 gco

⚡️Tools and products we use at Spike - part 2

1. Intercom

2. GitBook

3. 1Password

4. Notion

5. Headway

6. Figma

7. SendX

⚡️Tools and products we use at Spike - part 1

Servers

1. AWS

2. Vercel (formerly Zeit.co)

3. MongoDB Atlas

Monitoring

1. Cloudwatch

2. Healthchecks.io

3. Uptime Robot

4. Datadog

5. Nginx Amplify

6. Sentry

CI/CD

1. Buddy.works

2. Travis CI

Kent C. DoddsFollow

DevTools

1. Ngrok

2. VS Code Live Share

3. Twilio

4. Sendgrid

Why we built Spike? An alternative to PagerDuty and Opsgenie you will love 💙

A simple overview

Simplicity

Honesty

Flat pricing

Bootstrapped