How I made Instagram scraper on Firebase Console

Mladen Ružičić — Wed, 20 Feb 2019 12:34:40 +0000

The last couple of days I read about Instagrams new Graph API and was thinking about some creative ways of using it. To my surprise, they did not support anything related to user registration or at least account username availability. After some research, I saw one “tool” doing exactly that - checking if a provided text is a valid string and available Instagram username.
But now, I don't care about Graph API - I want to figure out how did they make it, without API!

Research

Of course, the first thing I tried was to inspect their source code and check network requests. All I could see was that it’s something hosted on Heroku. No help. After that, I went to Instagram official sign up page and inspected their code. instagram.com/accounts/web_create_ajax/ - I googled it - turns out it’s not publicly available.

Ok, now I want to create the tool myself. Why? Because I was the one googling "check Instagram username availability" a few days ago, so I hope I am not the only one who refuses to go to the official site to do that. Expectations? Coding all day and learning about new technologies.

Idea

After investigating the behavior of their form validation, my first idea was a NodeJS script, running puppeteer, populating Instagrams official sign up form and waiting for success or error element to show up on the screen. (I wrote some e2e tests at work last week, so I'm totally into this at the moment).
The plan for puppeteer is:

Navigate to the Instagram signup page
Click username input and fill in some text
Click body (to trigger blur event validation check)
Observe DOM and return whether the field is valid

e2e - for different needs

This is what I came up with, and - it was working!

Note: Since the puppeteers waitForSelector method will throw if an element does not appear in the DOM, I will assume it does not exist, and the username input field is valid.

Now, when I know the script works, let's deploy it!

Adjustments for Firebase Functions

For quite some time, I wanted to make practical use of Firebase Functions. This was a perfect moment to try it. I had some experience using Firebase before, so I went to Firebase Console, created a new project and initialized it locally.

There are two types of Firebase Functions:

The ones you call from Firebase app (an app hosted on Firebase, or an app authenticated with Firebase)
The ones you want anybody to access - through HTTP requests.

I wanted both. One for my web app, and the other for everyone else.

First things first. Create a new project on Firebase Console, go to Functions tab and click "Get Started" with functions.
Second thing, install Firebase CLI locally:

npm install -g firebase-tools

Authenticate to your Firebase account

firebase login

And initialize a new project, answering the on-screen questions (defaults are just fine)

firebase init

It is important to configure your app to use the Firebase project you created a minute ago. Use firebase use --add.

Function - onRequest

Okay. Now let's make this function available through an HTTP request. We must use onRequest. Arguments are the same as for Express.js - request and response, meaning the Request object gives you access to the properties of the HTTP request sent by the client, and the Response object gives you a way to send a response back to the client.
You can easily create a new Express app and export it as a Firebase Function, but that would be overkill for this use case.

Notice the CORS wrapper - required for accessing this function from outside our Firebase app domain.

We can test it right now using Postman

GET https://us-central1-your-project-name.cloudfunctions.net/check?username=awesome_username_99

and confirm it's working - Status 200 OK

{
    "available": true
}

Function - onCall

Now, the easier part - export a callable function to use from within Firebase app.

Web App - httpsCallable

What about testing this one? I want a simple webpage. Actually, validating input in vanilla javascript took me a while, but I liked it. If you spend most of your time using angular/vue/react/whatever, you probably forgot what it takes to check and set a validity to form elements, at least I did. I'll skip the boring part (link to the source code will be at the end of text).

Once I was happy how my form looked I deployed it to Firebase Hosting and tested.

firebase deploy

I took it a step further - bought an SEO-friendly domain name, set up Google Tag Manager and Google Analytics Goals, to actually track how many visitors (if I get any) hit available username.

P.S. I actually spent more time getting familiar with Google Tag Manager and writing this blog post, than I did coding. xD

You can see this app live at https://instagram-username.firebaseapp.com/ and the source code at GitHub.

Update February 27.

Instagram asked me to stop using domain containing their name (two days after I registered instagram-username.com), so I did. Demo still available on firebase subdomain.

DEV Community: Mladen Ružičić