DEV Community: Haider Ali Punjabi

Creating a secure Wordle using Serverless Functions

Haider Ali Punjabi — Tue, 01 Feb 2022 08:33:26 +0000

Introduction

Wordle has been viral on Twitter for the past few weeks. It's a really simple game and has been really successful. But, people were able to reverse engineer it and find out what the words are going to be in the future. I wanted to try and make one that can't be reverse-engineered that way. I did make one, but since there are hundreds of "How to make Wordle" tutorials out there, I will focus only on the "how to make it a bit more secure" part. I will use Serverless Functions for it.

Note: Serverless Functions can cost a bit if your Wordle becomes viral.

What's wrong with Wordle?

I won't say anything is wrong with Wordle. It's just a choice that its developer made. I am sure if Josh Wardle wanted, he could have made it more secure, and if I was in his shoes, I would have also made it as he has.

Why? Because using Serverless Functions is costly if the traffic is too much.

Robert Reichel wrote a good article on Reverse Engineering Wordle which explains how Josh Wardle's Wordle determines the words on the client-side.

At this point, we've done enough digging to know how Wordle is choosing the word of the day. We know that Wordle uses a client-side date-based algorithm to determine which word to use from a static wordlist. Each day is predictable so long as we have all of the code pieced together

-- Reverse Engineering Wordle | Robert Reichel

What do I mean by secure?

By "secure", I mean that no one would be able to know what tomorrow's (or the day after, or any day in the future) word is. One can always know what today's word is by playing the game once, seeing the word and playing it again in a different browser. Or if you know how to, you can send a request to the API and it will tell you today's word. What will this do? This will prevent bots like "The Wordlinator" that spoil the game for others.

Also, another advantage of using this method is. No matter where in the world you are playing from, everyone will get the new word at the same time because the selection will be based on the clock of the server, and not of the client. Many Wordle games have this issue where some people start getting the new word earlier than the rest of the world because it is 12:00 am of the new day for them, and the rest of the world is still on the previous day.

How to do it?

I won't write about how to make the entire Wordle game, but just the API / Serverless Functions Part. It can be deployed on any platform you like. I deployed mine on Next.js and Vercel.

Requirements:

A list of words from which each day's word will be chosen. It's better if this is a subset of a larger list of words that decide which word is accepted and which isn't. The word acceptance logic and the large list can be client-side. (It would be better as it will reduce the load on API and may save money). The smaller list of words is never loaded on the client.

Logic:

The logic for it is pretty simple. We will make an API Route that:

Loads the list of words from which we select a word each day
Calculates the number of days since some fixed data (e.g, the day the app/game is launched).
Selecting and responding with the word from the list of words using the calculated difference.

Code:

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction  
import { DateTime } from 'luxon';  
import type { NextApiRequest, NextApiResponse } from 'next'  
import { GameData } from '../../lib/interfaces';  
import gameWords from "../../data/selected.json"; // The list of words

// Function to calculate the difference between today and and a fixed date  
function getIndex():number {  
    let start = DateTime.fromFormat("31/01/2022","dd/mm/yyyy").setZone("UTC+5:30").startOf("day")  
    let today = DateTime.now().setZone("UTC+5:30").startOf("day")  
    return today.diff(start,'days').get('days');  
}

export default function handler(req: NextApiRequest,res: NextApiResponse<GameData>) {  
    let id = getIndex();  
    res.status(200).json({  
        id: id,  
        word: gameWords\[id\]  
    });  
}

Also, if you want, you can make another API Endpoint that returns the time left for next word.


import { NextApiRequest, NextApiResponse } from "next/types";  
import {DateTime} from "luxon";

export default function handler(  
    req: NextApiRequest,  
    res: NextApiResponse<number>  
  ) {  
    let t = DateTime.now().setZone("UTC+5:30").startOf('day').plus({days:1}).valueOf()  
    res.status(200).send(t)  
  }

With these two APIs, you can make a Wordle game that is a bit more secure.

References:

Also

Dynamic Twitter Header using Python

Haider Ali Punjabi — Fri, 10 Sep 2021 11:06:23 +0000

Twitter Headers are a craze among the tweeps of Tech Twitter, especially the ones "Made With CSS". Even though I am fairly comfortable with CSS, I lack the creativity to make a good CSS header with it, and near impossible to make anything on par with the CSS geniuses there. (Check out @Prathkum, he is a CSS Wizard and one of the inspirations behind my header projects).

Since "Made With CSS" wasn't possible, I decided to take a different route. Python. Python is something I use daily, from automating simple tasks to making bots for my amusement. Given my love for Python, it was the obvious choice for my Twitter Header Experiments.

The first one I made was similar to the one I will explain in this blog. It displayed my Twitter Statistics (Followers, Following, No. of Tweets, Lists I am on, My Twitter Age, etc). Even though it was cool and something that I used for a month or so, I later changed it to a humourous cryptocurrency one.

The second one like I said, was a cryptocurrency joke which updated the USD rates every minute. (I have that same joke on Reddit). I used this header for a while but later changed it to some static image.

Then, a couple of days ago I saw a cool "Made With CSS" header and thought about making another dynamic one. I also realised that I hadn't made a tutorial about it. So, here I am, killing two birds with one stone.

The Idea

The idea is really simple, make a Dynamic Twitter Header that display's the likes and retweets of the Tweet announcing the header. Since the header will change when users interact with the Tweet, more users will interact with it, just to see the header update. I was expecting the likes/retweets on the tweet to increase after I publish this blog, but it has already become one of my top tweets while I am typing this.

Although I hadn't planned it initially, after receiving feedback from what little audience I have on my Twitter, I also added the Latest 3 Followers to the header.

Demo

// Detect dark theme var iframe = document.getElementById('tweet-1435219303465324548-71'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1435219303465324548&theme=dark" }

Before starting the actual tutorial, you can check my header on my Twitter (@HAliPunjabi) and the related Tweet

Prerequisite - Twitter Developer Account and App

Before writing the actual code, you are going to need a Twitter Developer Account and then create a Twitter App. Using the created App's Keys and Tokens, you will be able to update your header and get the data from Twitter about your followers and likes/retweets on any tweet.

Go to Twitter Developer Account and apply for access. It shouldn't take too long to get approved.
Go to Overview and Create a New App. Name the App whatever you want, and copy and save the API Key and Secret they will provide. I will refer to them as CONSUMER_KEY and CONSUMER_SECRET from now on.
Change the App Permissions to Read and Write (or Read and Write and Direct Messages if your script needs it).
Go to the Keys and Tokens section and generate Access Token and Secret. Copy these as well and save them somewhere. I will refer to them as ACCESS_TOKEN and ACCESS_TOKEN_SECRET from now on.

Writing the script

To make the tutorial (and the code) easy, I won't be using any pre-made images but will generate everything from scratch using Python.

Setup

Create a directory where you will store the code
Create a file .env that will store our Keys and Tokens. This is how its contents should look

ACCESS_TOKEN=<ACCESS_TOKEN>
ACCESS_TOKEN_SECRET=<ACCESS_TOKEN_SECRET>
CONSUMER_KEY=<CONSUMER_KEY>
CONSUMER_SECRET=<CONSUMER_KEY>

Create a directory called fonts and download SourceCodePro-Regular.ttf from Google Fonts into it

Code

haideralipunjabi / twitter-header-script

Dynamic Twitter Header using Python

Twitter-Header-Script

Python Script to generate a Twitter Header with Like / Retweet Statistics of a Tweet, and 3 latest followers

Demo

Sample Generated Image

Also, I am going to be using this as my header on Twitter for a while, using this tweet.

Usage

Download the requirements pip install -r requirements.txt
Create a directory called fonts and download SourceCodePro-Regular.ttf from Google Fonts into it
Create a Twitter app from Twitter Developer Dashboard
Rename .env.sample to .env and fill it with appropriate values from your Twitter App
Change the PINNED_TWEET_ID variable in main.py to the ID of the tweet you want to use
Run the script to generate the header and upload it
Run it at regular intervals (e.g, using cronjobs) to make sure the header is always updated

View on GitHub

The whole code is available on Github, so I will explain only the important parts of it.

Variables and Constants to be used later on in the code

# Load environment variables from .env file
load_dotenv()

# Location of this file
parent_directory = os.path.dirname(os.path.abspath(__file__))

# Colors to be used in the header
COLORS = {
    "FOREGROUND": "#000000",
    "BACKGROUND": "#FFFFFF",
}

# Fonts to be used in the header
FONTS = {
    "TITLE": ImageFont.truetype(os.path.join(parent_directory, "fonts/SourceCodePro-Regular.ttf"), 48),
    "SUBTITLE": ImageFont.truetype(os.path.join(parent_directory, "fonts/SourceCodePro-Regular.ttf"), 30),
    "NUMBERS": ImageFont.truetype(os.path.join(parent_directory, "fonts/SourceCodePro-Regular.ttf"),  144),
    "FOOTER": ImageFont.truetype(os.path.join(parent_directory, "fonts/SourceCodePro-Regular.ttf"),  24),
}

# ID of the Tweet to be used
PINNED_TWEET_ID = "1435219303465324548"

Authentication to Twitter

# Authenticate to Twitter and return the API object
def get_twitter_api():
    auth = tweepy.OAuthHandler(
        os.getenv("CONSUMER_KEY"), os.getenv("CONSUMER_SECRET"))
    auth.set_access_token(
        os.getenv("ACCESS_TOKEN"), os.getenv("ACCESS_TOKEN_SECRET"))
    return tweepy.API(auth)

Fetching required data

# Fetch the status and followers data
def get_status_data(api, status_id):
    status = api.get_status(id=status_id)
    followers = api.followers(
        count=3, skip_status=True)    # 3 Latest Followers
    return {
        "likes": status.favorite_count,
        "retweets": status.retweet_count,
        "followers": [{
            "username": follower.screen_name,
            "photo": follower.profile_image_url_https
        } for follower in followers]        # The username and profile picture only
    }

Drawing the header. The coordinates are mostly hardcoded around the 1500x500px dimension of the header. Only the followers' images and the rectangle around them is calculated based on the width of usernames.

# Draw the Header
def draw_header(data):
    # Create a new Image 1500 x 500 with background color
    img = Image.new('RGB', (1500, 500), color=COLORS['BACKGROUND'])
    d = ImageDraw.Draw(img)  # Draw variable for drawing on the image
    # The rectangle which contains the test "My pinned tweet has"
    d.rectangle((30, 30, 680, 110), None, COLORS["FOREGROUND"], 5)
    # Text - My Pinned Tweet has
    d.text((355, 70), "My pinned tweet has",
           fill=COLORS["FOREGROUND"], font=FONTS["TITLE"], anchor="mm")

    # Likes and Likes Count
    d.text((500, 370), "LIKES",
           fill=COLORS["FOREGROUND"], font=FONTS["TITLE"], anchor="mm")
    d.text((500, 250), str(data["likes"]),
           fill=COLORS["FOREGROUND"], font=FONTS["NUMBERS"], anchor="mm")
    # Retweets and Retweets Count
    d.text((1000, 370), "RETWEETS",
           fill=COLORS["FOREGROUND"], font=FONTS["TITLE"], anchor="mm")
    d.text((1000, 250), str(data["retweets"]),
           fill=COLORS["FOREGROUND"], font=FONTS["NUMBERS"], anchor="mm")

    # Text - Latest Followers
    d.text((1300, 50), "Latest Followers",
           fill=COLORS["FOREGROUND"], font=FONTS["SUBTITLE"], anchor="mm")

    # Keeping track of widest line of text
    max_width = d.textsize("Latest Followers", font=FONTS["SUBTITLE"])[0]

    # Drawing the followers text and image
    for idx, follower in enumerate(data["followers"]):
        username_width = d.textsize(
            follower["username"], font=FONTS["SUBTITLE"])[0]
        if username_width + 50 > max_width:
            # 50 = Width of image (30) + gap between image and text (20)
            max_width = username_width + 50
        d.text((1320, 90 + 40*idx), follower["username"],
               fill=COLORS["FOREGROUND"], font=FONTS["SUBTITLE"], anchor="mm")
        # Download image
        profile_image = Image.open(io.BytesIO(
            requests.get(follower["photo"]).content))
        # Resize image
        profile_image = profile_image.resize((30, 30))
        # Paste Image
        img.paste(profile_image, (1280 - (username_width//2), 75 + 40*idx))

    # Draw rectangle around followers
    d.rectangle((1300 - (max_width/2) - 20, 30, 1300 +
                (max_width/2) + 20, 210), None, COLORS["FOREGROUND"], 5)

    # Footer Text
    d.multiline_text((750, 465), "Like / Retweet my pinned tweet to see my header update\nCheck pinned thread for more details",
                     fill=COLORS["FOREGROUND"], font=FONTS["FOOTER"], align="center", anchor="ms")
    # Save the image
    img.save(os.path.join(parent_directory, "header.png"))

Driver Code

# Driver Code
if __name__ == "__main__":
    api = get_twitter_api()     # Get the Authenticated Api
    # Draw the header using data of PINNED_TWEET_ID
    draw_header(get_status_data(api, PINNED_TWEET_ID))
    api.update_profile_banner(os.path.join(
        parent_directory, 'header.png'))  # Upload the header

Execution

You can run this script to update the header. I will suggest keeping this as a cronjob, for at least every two minutes to get the best results. I am hosting this on my Raspberry Pi, but you could try hosting it on Heroku, or some other service.

Other Related Resources

Buy Me a Coffee

Testing PWAs on mobile devices during development

Haider Ali Punjabi — Mon, 21 Dec 2020 11:05:08 +0000

PWA? What is that?

Progressive Web Applications (PWAs) are web applications (duh!) that use emerging web APIs and modern technologies to make the web app behave similar to a native application on any platform / operating system. They can be installed on your phone or PC, and even be distributed through App Stores.

I will link some articles related to PWAs at the end.

The capability to be installed on a mobile device is very important for a PWA, and to test those mobile-specific features even more important. PWAs can only be served over Secure Contexts (HTTPS). It isn't a big deal today due to services like Let's Encrypt, which provide free SSL Certificates. The HTTPS restriction isn't a problem but makes testing the PWA difficult. Most browsers treat localhost as a Secure Context, and testing PWA on your own machine isn't a problem. The problem arises when you access your web app from your phone (over local network), and it won't work because usually, you won't have an SSL Certificate for debugging on your own machine. One of the solutions to this problem is to get an SSL certificate, but it can be a complicated process for those not having experience with such stuff.

In this blog post, I will explain two methods which will help you test your PWAs on your phone.

Chrome Port Forwarding

Host a site on your machine, access the content from an Android Device (over Android Debugging Bridge). By using Port Forwarding, the browser on your mobile will be able to access the site on its own localhost, thus in a Secure Context.

Most of the tutorials on internet use Wired ADB, which discourages many people from using this method. I will be using Wireless ADB, which isn't very difficult to setup.

ADB over Network

Go to your Phone Settings > System > Developer Options (This might vary in different phones, so if it is not the same in your's, look it up on the internet)
Turn on Android Debugging and ADB over Network.
Note the IP Address and Port shown under ADB over Network
Install ADB on your computer
Go to your command-line / command prompt and enter adb connect <ip-address>:<port> (Use the IP Address and Port from Step 3)
When connecting for the first time, you will need to authorize the connection on your phone.
Your device should be connected to your PC over WiFi.

Using Chrome Port Forwarding

Make sure your development server is running on your PC
Go to chrome://inspect/#devices. You should see your device along with a Connected status indicator
Enable Port Forwarding
Click Add Rule
In the Device Port, enter the port number on which you want to access the site on your device
In the Local Address field, enter the address and port of your development server. (e.g, localhost:1313)
Click Add

Port Forwarding should be setup and you will be able to access your server on your Android Device on localhost:port (which you set in Step 5.)

Pros vs Cons

Pros:

Faster Deployment and Testing
Already using ADB and Chrome, which will also be used if you want to use the Dev Tools on your mobile device

Cons:

Can only be used for Android Devices
The devices need to be on the same network

Netlify Dev

Netlify Dev is part of the netlify-cli, and is a great choice if you are already using netlify-cli, or need to showcase your work live without deploying it to production. Netlify Dev also allows you to use many features from the Netlify Ecosystem like Netlify Functions, Custom Headers, etc.

Setup

First of all, you will need to setup netlify-cli and authorize it. You will need an account on Netlify for it.

Here's a link to a detailed tutorial

Install netlify-cli npm install netlify-cli -g
Authenticate
netlify login
Authorize it in the browser window that opens.
Setup your repository using
a. Automated Setup (if your repository is on Github)
netlify init
b. Manual (for other Git Providers, or if you wise to do it like this)
netlify init --manual
You can do a lot of stuff with this cli, but I will skip directly to Netlify Dev
To start a Netlify Dev server, you can use netlify dev , but since we want to Start a Public Live Session, we will use
netlify dev --live
You will get a URL that looks like https://clever-cray-2aa156-6639f3.netlify.live/. This URL will be accessible by everyone on internet.

netlify-cli has many more features you should read about in the documentation.

Testing your PWA on your mobile

Now that you can access the PWA on your mobile, you need to find a way to test it properly. The Dev Tools available in PC Browsers help in debugging and testing, but unfortunately, I have yet to find a similar feature on phone. Fortunately, Chrome provides a solution by allowing us access to the Dev Tools for a page on mobile devices using ADB.

Setup ADB, as shown in the Chrome Port Forwarding
In chrome://inspect/#devices, you will see a list of all pages opened on your mobile device.
Click on the inspect button to launch the Dev Tools for that page

References

Connect with me

Friendly reminder: String.replaceAll is only supported on 83.7% of devices

Haider Ali Punjabi — Sun, 13 Dec 2020 16:35:26 +0000

I just found out after one of my friends couldn't access my NextJS web app. Finished my free trial of BrowserStack on it.

Can I Use replaceAll ?

Badges for the terminal - Python

Haider Ali Punjabi — Mon, 24 Aug 2020 09:00:23 +0000

Hello everyone!

I've just ported the quirky node-js library cli-badges over to Python. The author of that library:

.ltag__user__id__187971 .follow-action-button { background-color: #000000 !important; color: #ffffff !important; border-color: #000000 !important; }

Manolo Edge

I don't know... I just code and they pay me!

nombrekeff https://manoloedge.com/

Post about the NodeJS Library

Badges for the terminal

Manolo Edge ・ Aug 22 ・ 1 min read

#showdev #javascript

Python Port that I made:

haideralipunjabi / cli-badges

Quirky little python package for generating badges for your cli apps.

Quirky little python library for generating badges for your cli apps.

Inspired & Python Port of cli-badges - nombrekeff

Getting Started

Installing

As usual, you need to install from PIP:

$ pip install cli-badges

Usage

This is a simple example, using badges to display test results:

from cli_badges import badge
failedBadge = badge("failed",'2',messagebg='red')
skippedBadge = badge('skipped', '1', messagebg='yellow',messagecolor='black')
successBadge = badge('success','8', messagebg='green',messagecolor='black')
print(failedBadge, successBadge, skippedBadge)

The above would output something similar to the terminal:

You could also create a donate badge with a link (if supported):

from cli_badges import badge
donateBadge = badge('❤️ donate', 'ko-fi', messagelink='https://ko-fi.com/logginjs');
print(donateBadge)

You can also only…

View on GitHub

Installing

As usual, you need to install from PIP:

$ pip install cli-badges

Usage

This is a simple example, using badges to display test results:

from cli_badges import badge

failedBadge = badge("failed",'2',messagebg='red')
skippedBadge = badge('skipped', '1', messagebg='yellow',messagecolor='black')
successBadge = badge('success','8', messagebg='green',messagecolor='black')


print(failedBadge, successBadge, skippedBadge)

The above would output something similar to the terminal:

You could also create a donate badge with a link (if supported):

from cli_badges import badge

donateBadge = badge('❤️ donate', 'ko-fi', messagelink='https://ko-fi.com/logginjs');
print(donateBadge)

You can also only show the label:

from cli_badges import badge

onlyLabel = badge('❤️ donate', '')
print(onlyLabel)

Example output is a mock, console output will vary slightly from terminal to terminal.

Badge Structure

A badge is conformed of a label and a message <label>:<message>. Each segment can be customized, by changing bg color, text color and style.

Available Options

Option	Value	Default
label	String	`''`
message	String	`''`
messagebg	Color	blue
labelbg	Color	dark_gray
messagecolor	Color	white
labelcolor	Color	white
labelwidth	Integer	label length + 2
messagewidth	Integer	label length + 2
labelstyles	Array of Styles	`[]`
messagestyles	Array of Styles	`[]`
labellink	URL	`''`
messagelink	URL	`''`

Colors

cli-badges uses colored internally for managing colors, you can check the list of available colors there.

Styles

cli-badges uses colored internally for managing styles, you can check the list of available styles there.

Available Styles

bold
dim
underlined
reverse
hidden

Links

You can output badges with a link attached to it, that can be clicked in some terminals. labellink option will add the link to the label, while messagelink option will add the link to the message.

⚠︎ cli-badges will only output link if its supported by your terminal.

See this for information on supported terminals

badge('with', 'link',  labellink='https://link.com', messagelink='https://link2.com');

Redesigning My Website - Automation, Custom SSG & Optimisations

Haider Ali Punjabi — Sun, 02 Aug 2020 06:46:35 +0000

Link to the redesigned website

The Old Design

Over the years, my portfolio website has gone through many changes. I always liked my website as simple as possible, with the possibility of editing my resume without opening the code. I have always preferred using HTML/CSS & JS to make websites if possible. I will, of course, use a proper SSG if the project needs it (mostly Hugo for blogs). I am also not against using some framework like React or Vue, but I only use them in projects with a huge number of components.

Also, although I know CSS, the quality of any design I make from scratch has an inverse relation with the number of components. Thus, I prefer using premade themes as a base and then customizing them to my needs.

The older version of my website used Astral theme from HTML5UP. I had customized it to some extent (not a lot, the theme was great but I never spent enough time on it). I hosted my resume on Github Gists and embedded it on the website. After some minor CSS changes, it looked as if it was part of the website itself.

There were a number of problems in that design, and I had to eventually change it someday. Among the problems, the things that I hated the most were:

Extremely Heavy: The design loaded a lot of resources (Font Awesome, etc) that made it extremely slow to load.
No CMS: I never got the time to integrate the website with a CMS, and I always had to edit the code to make any change.

I was always planning to fix these two issues, as soon as some inspiration struck me for the new design.

Inspiration for the New Design

The inspiration came when Florin Pop did a live stream on Twitch where he did a developer portfolio review. Among those portfolios, I found a design that I liked a lot. The said design was of CasperTheGhost’s portfolio website.

What I liked about his website was:

Limited Colors: It had only 3 colours, one purplish for the background and grey and a white for the text
Repeatable design: All the sections of the website had the same base design. Adding or removing sections won’t destroy the design of the website.
Lightweight: Very few external resources are used, and I could remove whatever I didn’t want.

Before proceeding further, you should check out the website because I am going to mention different parts of it

Planning

After deciding on the design, I spent some time deciding how to make the website work. If I did minimal changes to the source, I would have to do a lot of changes manually and keep doing them manually in the future. I knew that I was going to add a variety of stuff to the portfolio which can change frequently, so having a CMS was a must. I decided that the CMS would make store all the data in JSON files somewhere in the code and I would load the data into HTML files.

For the loading part, I had two choices. One, I could use JavaScript and its fetch() method to load the JSON data. The advantage of this method is that no building or generating is required, and the files could be used as-is. The disadvantage, of course, was the effect on performance and that the website would have almost nothing that isn’t loaded via JavaScript. I had used this method a lot in the past but wasn’t too fond of it due to its disadvantages.

On the other hand, I could use a Static Site Generator that would put the data into the HTML files during the build process. The client will only need to download the HTML file which already has all the data in it. The advantage, of course, would be the increase in performance, but I don’t like using Static Site Generators for such a simple task. Static Site Generators are usually made for generating much bigger and complicated sites (many are specially tuned for generating a blogging website from markdown files).

I finally decided on writing a Python Script that would act as my custom Static Site Generator. Jinja2 is a great Python module for generating files from templates. I had already used similar scripts in other projects of mine. This is a great way to add a common part (header, footer, navigation bar, etc) of code to a multi-paged website. I was also able to generate a multilingual web app using a similar technique.

Deciding to use a Python Script allowed me to make a lot of stuff editable from the CMS (Name, Open Graph Information, Colors, etc).

The CMS

Before starting the work on the website, I had to decide what CMS I was going to use. I have previously used both Forestry and Netlify CMS, but only for Hugo Blogs. I wanted to use Netlify CMS because I could access it from mydomain.com/admin (I forgot how to access Forestry CMS once). No matter how much I tried, I couldn’t get it to work for editing single data files. I switched to Forestry and setup the required front matter and data files.

Structure of Data

Since the whole website works on this data, I think it would be appropriate to define its structure first.

At the time of writing this, I have 5 different JSON files for various sections of the website.

Settings.json This contained the settings for the website (colours, google analytics code, etc), Open Graph Details, and some stuff that is used in the landing section (Name, Subtitle, Social Media Links, etc)
backpack.json, foss-contributions.json, timeline.json, projects.json Each of these contained an array of items to be used in different sections. You can see the actual structure in the code below. For the icons, I was originally using the Font Awesome class names but changed it to the path to increase performance. (More on this later)

backpack.json

    {
      "name": "Python",
      "icon": "assets/svgs/brands/python.svg"
    },

foss-contributions.json

    {
      "link": "https://github.com/processing/p5.js"
    },

timeline.json

    {
      "title": "Polybar Module",
      "time": "June 2020",
      "description": "<p><a rel=\"noopener noreferrer\" href=\"https://github.com/HackeSta/polybar-speedtest\">polybar-speedtest</a> - speedtest.net module for Polybar<br></p>"
    },

projects.json

    {
      "name": "Tweet2Pic",
      "buttons": [
        {
          "name": "Play Store",
          "link": "https://play.google.com/store/apps/details?id=org.hackesta.tweet2pic",
          "icon": "assets/svgs/brands/google-play.svg"
        }
      ],
      "description": "Tweet2Pic is a small tool, which can be used to share your Twitter Tweets in the form of an image.",
      "tags": [
        "assets/svgs/brands/android.svg",
        "assets/svgs/brands/java.svg"
      ]
    },

Data from the Internet

Apart from the data which I am entering via the CMS, there were some sections that required the script to fetch data from different sources on the internet. Most importantly, I use the Github API data for the Open Source section, which lists the number of repositories (source) I have, the numbers of stars they have, and the number of forks I have. I use the Github API for the “Projects I Contribute to” Section as well. In its data file (foss-contributions.json), I am storing only the repository name of the items. The script then uses Github API to fetch the description, number of stars and forks of each project.

def get_github_data(username,num):
    repos = []
    page = 1
    while True:
        data = requests.get(GITHUB_API_URL%(username,page)).json()
        repos.extend(data)
        if len(data) < 100:
            break
        page += 1
    source_repos_len = len(list(filter(lambda repo: repo["fork"]==False,repos)))
    stargazers = reduce(lambda a,b: a+b["stargazers_count"],repos,0)
    forks = reduce(lambda a,b: a+b["forks"],repos,0)
    most_popular = list(sorted(repos, key=lambda repo: repo['stargazers_count'],reverse=True))[:num]
    return {
        "github":{
            "repo_count": source_repos_len,
            "stargazers": stargazers,
            "forks": forks,
            "most_popular": most_popular
        }
    }
def get_foss_contributions():
    contributions = json.load(open(FOSS_CONTRIBUTIONS_DATA,"r"))["contributions"]
    contributions_data = []
    for contribution in contributions:
        api_url = contribution["link"].replace("github.com","api.github.com/repos")
        data = requests.get(api_url).json()
        contributions_data.append(data)
    return {
        "contributions": contributions_data
    }

I also configured my blog to output a JSON listing recent blog posts along with their featured/open graph image. The script fetches this JSON file, then downloads the data for four recent posts, along with the images. The images are then resized to a smaller size and stored in a folder. Downloading the images, resizing them to a smaller size and converting them to WEBP format really helped in maintaining the performance of the website.

def get_blog_posts(num):
    data = requests.get(BLOG_API_URL).json()
    os.system("rm -rf assets/img/blogs")
    os.system("mkdir assets/img/blogs")
    for post in data[:num]:
        filein = f'assets/img/blogs/{post["data"]["image"].split("/")[-1]}'
        fileout = filein.rsplit(".",1)[0] + ".webp"
        wget.download(post["data"]["image"],out=filein)
        post["data"]["image"] = fileout
        im = Image.open(filein)
        im.thumbnail(BLOG_IMAGE_SIZE,Image.ANTIALIAS)
        im.save(fileout)

Optimizations

I wanted my website to be as lightweight as possible. To make sure I am doing everything right, I used the Lighthouse Test to check for errors, and get ideas for optimizations. Apart from issues that were fixable easily (missing attributes, missing icons, etc), the most important part of the optimization was to reduce the number & size of requests. As I already had minimal JS, I focused on optimizing the CSS files and reducing the number of requests.

Optimizing CSS

Two Node Packages, PurgeCSS (to delete unused CSS, although there wasn’t much to delete) and PostCSS + cssnano (to minify and bundle the CSS) were more than enough to optimize the files. I added both of them to the Travis Build Process to automate it.

Also, Netlify does provide a feature to minify and bundle resources, which should also work similarly.

Optimizing Font Awesome

Apart from other places in the website, the “My Backpack” section heavily used Font Awesome. Font Awesome files are huge in size and each icon had a request associated to it. To overcome this, I stopped using Font Awesome the normal way (adding fa* fa-* classes to elements) and took advantage of using Jinja2 to import the SVG icons into the HTML wherever required. After generating the final page using Jinja2, all the SVGs are present as inline SVGs and are loaded as part of the HTML document. This removed the size of the helper script (approx 1.4mb) and removed a good number of requests.

Optimizing Favicons

I originally used another Python script to generate Favicons but they were being requested numerous times during the page load. After some research, I came across this favicon generator site which generated beautiful icons and reduced the number of requests to just 2 requests.

Final Score

Resources

Read this on my blog

Automating Android Games with Python & Pytesseract: Sudoku

Haider Ali Punjabi — Wed, 24 Jun 2020 16:45:36 +0000

Introduction

I made a Python Script to Automate a Sudoku Game on Android after watching Engineer Man's Videos on Youtube doing the same for different games.

The script can be divided into 5 parts

Connecting to an Android device using ADB, and getting the screenshot of the game from it
Using Pillow to process the screenshot for pytesseract
Using pytesseract to extract the Sudoku Game Grid to a 2D List in Python.
Solving the Sudoku Game
Sending the solved input to your Android Device using Python

Out of the 5, I will be focusing mostly on 2,3 & 5 as 1 & 4 are topics that have been extensively covered.

Link to the game I automated: https://play.google.com/store/apps/details?id=com.quarzo.sudoku

The complete code is available on the following repository:

Github: haideralipunjabi/sudoku_automate

You can also watch the script in action on:

Libraries Used

Tutorial

1 (a). Using ADB to Connect to your Device

Most of the tutorials on internet use Wired ADB, which discourages many people from using this method. I will be using Wireless ADB, which isn't very difficult to setup.

Go to your Phone Settings > System > Developer Options (This might vary in different phones, so if it is not the same in your's, look it up on the internet)
Turn on Android Debugging and ADB over Network.

Note the IP Address and Port shown under ADB over Network
Install ADB on your computer
Go to your command-line / command prompt and enter

adb connect <ip-address>:<port>

Use the IP Address and Port from Step 3

When connecting for the first time, you will need to authorize the connection on your phone.
Your device should be connected to your PC over WiFi.

1 (b). Using ADB with Python (pure-python-adb)

You can define the following function to connect to the first ADB device connected to your computer using Python

from ppadb.client import Client

def connect_device():
    adb = Client(host='127.0.0.1',port=5037)
    devices = adb.devices()

    if len(devices) == 0:
        print("No Devices Attached")
        quit()
    return devices[0]

We will be using this function later to return an instance of ppadb.device.Device which will be used to take a screenshot, and send input to your device.

1 (c). Taking a Screenshot and saving it

pure-python-adb makes it very easy to capture a screenshot of your device. The screencap function is all that you need to get the screenshot. Use Pythons File IO to save it to screen.png\

def take_screenshot(device):
    image = device.screencap()
    with open('screen.png', 'wb') as f:
        f.write(image)

2. Processing the screenshot with Pillow

In the captured screenshot, the accuracy of any OCR will be very low. To increase accuracy, I used Pillow to process the screenshot so that it only shows the numbers in black color on a white background.

To do that, we first convert the image to grayscale (or single channel) using image.convert('L'). This will make the convert the colors to shades of greys (0-255).

After this, we need the numbers (which are the darkest, or very near to black) in black color, and the rest in white. For this, we use image.point() so that all the greys > 50 become white (255) and the rest (numbers) become 0. I also increased the Contrast and Sharpness a bit to be on the safer side.

def process_image(image):
    image = image.convert('L')
    image = image.point(lambda x: 255 if x > 50 else 0, mode='L')
    image = ImageEnhance.Contrast(image).enhance(10)
    image = ImageEnhance.Sharpness(image).enhance(2)
    return image

3. Extracting the numbers from the image using pytesseract

Using pytesseract on the whole image might give us the numbers, but it won't tell us in which box the number was present. So, I use Pillow to crop each box and then use pytesseract on the cropped images. Before using pytesseract, I defined some functions to give me the coordinates of each box and to give me a cropped image of each box.

Since Sudoku has a 9x9 grid, I use two for loops from 0 to 8 to loop over each box. The pytesseract wasn't accurate enough on the default configuration and I had to pass the config --psm 10 --oem 0.

The --psm argument defines the Page Segmentation Method. 10 stands for Treat the image as a single character. This seemed most appropriate since I am passing cropped images of each box.
The --oem argument defines the OCR Engine Mode. 0 stands for Legacy Engine Only.

The following function will extract the numbers from the passed image and return a 9x9 2D List with the numbers.

def get_grid_from_image(image):
    grid = []
    bar = Bar("Processing: ", max=81)
    for i in range(9):
        row = []
        for j in range(9):
            digit = pytesseract.image_to_string(
                get_box(image, i, j), config='--psm 10 --oem 0')
            if digit.isdigit():     # If pytesseract returned a digit
                row.append(int(digit))
            else:
                row.append(0)
            bar.next()
        grid.append(row)
    return grid

4. Solving the Sudoku Game

Now that we have the 9x9 Sudoku, we need to solve it. Solving Sudoku is a topic that has been covered a lot, and I also copied this code from geeksforgeeks.org.

Here's the geekforgeeks article on Sudoku

5. Sending the solved input to your Android Device using Python

To send the input, I first filtered out the input from the solved Sudoku Grid,i.e, only send the values which were missing. I used the get_coords function from earlier to get the coords of each box and then calculated their centres. I sent a touch at that centre using ADB, and then sent over the solution.

def automate_game(org_grid, solved_grid):
    for i in range(9):
        for j in range(9):
            if org_grid[i][j] == 0:     # If the box was blank in the game
                x1, y1, x2, y2 = get_coords(i, j)
                center = (x1 + (x2 - x1)/2, y1 + (y2-y1)/2)     # Calculating the center of the box (to select it)
                solution = solved_grid[i][j]
                device.shell(
                    f'input touchscreen swipe {center[0]} {center[1]} {center[0]} {center[1]} 5')
                device.shell(f'input text {solution}')

Running the code

All the code that I wrote is in functions and they are called one by one. Note that the grid that I get in step 3 isn't passed directly to step 4. I use deepcopy to create a copy of it, so that I can compare the solved grid with the unsolved/original one in step 5.

if __name__ == "__main__":
    # Connect the device using ADB
    device = adb.connect_device()
    # Take Screenshot of the screen and save it in screen.png
    adb.take_screenshot(device)
    image = Image.open('screen.png')
    image = process_image(image)        # Process the image for OCR
    org_grid = get_grid_from_image(image)      # Convert the Image to 2D list using OCR / Pytesseract
    solved_grid = deepcopy(org_grid)        # Deepcopy is used to prevent the function from modifying the original sudoku game
    solve_sudoku(solved_grid)
    automate_game(org_grid, solved_grid)        # Input the solved game into your device

References

The Elder Scrolls V: Skyrim Special Edition - Analysis of Dialogues

Haider Ali Punjabi — Thu, 04 Jun 2020 13:19:57 +0000

Read this on my blog

Background

After my last visualisation of Harry Potter data, I decided to use some other data to create word clouds. I am a huge Skyrim fan and always wanted to learn how to use xEdit Scripts. As a result, here I am with another Word Cloud.

Disclaimer: A lot of the code is shared between my previous project and this one

Visualisations

Getting the data

I don't like opening my Windows installation (I have a dual boot setup, and use Manjaro mainly), and looked around the internet for some sort of data dump of Skyrim Dialogues. Unfortunately, I couldn't find any and then decided to extract the data myself. I had recently formatted my Windows Partition so had to reinstall the game. It also provided the benefit that no mods would pollute the data. (I had over 150 mods before the format). I downloaded the latest xEdit and used the Export dialogues.pas script that comes with it to export all the dialogues. (It took me 22:05 minutes).

I am going to look into other data I can extract this way, and maybe make some other stuff.

Processing the data

In the CSV, there were two columns of data I was interested in RESPONSE TEXT & TOPIC TEXT. Response Text was the larger one, with over 40k unique dialogues. Topic Text had only around 5.5K unique dialogues and also needed some additional processing. Topic Text contained some game constants such as RoomCost , HorseCost, and other prices, which had to be filtered out. I did all that in csv_to_json.py.

Counting the Words

Like the previous visualisation, I used nltk's stopwords corpus, along with a modified version of 20k most common words by Google. Interestingly, the modifications I did for Harry Potter were valid for Skyrim as well, because there is no dialogue with names like Harry, Ron, Arthur, etc and they share words like vampires, magic, etc.

I counted both RESPONSE TEXT & TOPIC TEXT data separately and then merged them into a single file count.json

Additional Tip: progress is a great Python Package to show progress in your scripts.

Making the WordCloud

I used pretty much the same process as the last visualisation. I changed the maximum font size to depict the variation properly and used a custom font this time.

To make the WordCloud, I used the wordcloud package. For the mask, I used Skyrim Logo Vector. For the font, I used Sovngarde font.

Making the Graph

I initially planned on making a set of graphs from the data, but wasn't able to due to two reasons:

Some of the data was weird. Argneir had the highest dialogue count due to dialogues of many NPCs (including General Tullius, I think) is assigned to him.
Some of the data doesn't; produce interesting visualisations. Nords have the highest dialogue count, and after the difference between the first few races and the remaining is so huge that a lot of the races aren't visible.

Since I had already made this, I thought of sharing it here, in case someone is interested in the image or its code.

Future Plans

I will look into making custom scripts (if someone already has them, do share it with me) to extract other interesting data from Skyrim and see what I can do with them.

References:

Harry Potter Books & Fanfiction - An Analysis of Words

Haider Ali Punjabi — Mon, 01 Jun 2020 16:07:58 +0000

Background

On 28th May, a post on r/dataisbeautiful inspired me to learn how to make Word Clouds myself. Being a huge Harry Potter fan, the data I was going to use was obvious. Using the Books seemed too simple so I decided to scrape 250 stories from Fanfiction.net , and make a Word Cloud from that data. I posted my first attempt on r/dataisbeautiful, and based on the feedback I received, I decided to write this blog.

The entire source code (except the data files & output files) is available here

Attempt 1

There are many approaches I could have taken to prepare the data. I decided to download the stories first and then do the processing on the local files due to my slow & unreliable internet.

Scraping Fanfiction

I used simple Python + BeautifulSoup combination to scrape the stories form Fanfiction.net. I sorted the stories based on their Favorite Count, and filtered them to stories having more than 100k words. (Link to the URL). I scraped first 10 pages, (each page has 25 stories) resulting in 250 stories. It took me a total of 10 hours (7 on one day, and 3 on the next) to scrape all the stories.

Processing the Data

Taking hints from the original post, I used nltk to tokenize the stories, and removed the common words from the nltk English Stopwords Corpus. This was my first attempt at doing anything like this, and the process was taking 3-4 minutes per story initially. After some optimization, I was able to reduce the time to 1-2 minutes per story. I talked to a friend about the problem, and he suggested me to try multiprocessing. After adding multiprocessing, I had the idea of distributing the load over two CPUs (my laptop and a Raspberry Pi 4B). I copied the script and 25% of the stories over to the Pi and started the job.

Additional Tip: screen is a good utility to do long jobs over SSH

It took me an hour to the processing. I didn't want to do the processing again if I needed to remove some more words so I decided to store the word frequency data into json files. (Really helpful in future)

Making the Word Cloud

I took a look at wordcloud Python Package and copied the code from its examples to generate the word cloud.

To make the mask image, I downloaded some images from the Internet and used Inkscape to fix them.

Feedback

After posting the first attempt over at Reddit & Twitter, I received a lot of feedback. Common among them were the queries about why is Daphne more frequent and why is Ron less frequent (I will answer both later), suggestions to remove more words to focus it more on Harry Potter related words, and to show some other visualisations, especially ones comparing the books and fanfiction.

Attempt 2

Finding more stopwords

In my first attempt, I used the nltk English stopwords corpus, which is just 179 words. I searched for a bigger list and ended up using a customised 20,000 most common words list from google-10000-english repository. What were the customisations? I had to remove some words (like magic, magical, wand, wards, vampire, etc) and some names (Harry, Ron, Fred, Arthur, etc) from the 20k list so that they aren't removed from my analysis. Storing the results of the processing from my first attempt into json files saved me from spending another hour of processing. I just removed the necessary keys from each data file.

Harry Potter Canon Books

I also downloaded the text versions of the 7 books from somewhere on the Internet, sanitised them a bit, and applied the same process as the fanfiction stories to generate their data. Using that data, I was able to compare the occurrence of some words in fanfiction vs canon. Since I had the data and the code, I decided to make their corresponding word clouds as well.

Visualisations from Attempt 2

Frequently Occurring Words in Top 250 Fanfictions

Average Frequency of Occurrence of Words per book or story

Top 20 Most Occurring Words in Fanfiction Stories
Top 20 Most Occurring Words in Harry Potter Books (excluding Top 20 from Fanfiction Stories)

Code to generate the above visualisations

Important Results

Who is Daphne? Why is she so popular in Fanfiction?

Hermione’s name was called. Trembling, she left the chamber with Anthony Goldstein, Gregory Goyle, and Daphne Greengrass. Students who had already been tested did not return afterward, so Harry and Ron had no idea how Hermione had done.

-> Harry Potter and the Order of the Phoenix

Daphne Greengrass is an almost non-entity in canon, and a blank slate for fanfiction writers. In canon & most fanfictions, she is the sister of Astoria Greengrass (another almost non-entity) who becomes the wife of Draco Malfoy. In fanfictions, she is usually a Slytherin due to her ambitions & cunningness & not because of being a Pureblood Supermasict. Her family is depicted as Light or Grey, and support "Lord Potter". She is a popular pairing in Independent Harry stories.

Her being a blank slate character-wise is a boon for writers who want to write an OC without explicitly mentioning it.

Video explaining Daphne Greengrass and her popularity

What happened to Ron?

Ron is an almost opposite of Daphne. JKR wrote Ron in such a beautiful manner that many fanfiction writers are unable to write a good Ron. In canon, Ron is flawed but is also very funny, brave and loyal to his friends. In fanfictions, especially where Harry is very different to canon (Independent, Super-Powered, Lord Potter, etc), Harry usually ignores Ron (if diverging before Hogwarts) or the author does a lot of Ron bashing to justify Harry breaking up their friendship.

Bonus Visualisations

The 7 Canon Books:

Philosopher's Stone

I tried to use an image of 9 3/4 . The word Quirrell and Griphook are frequent in this book and will lose their frequency in future books.

Chamber of Secrets

I used an image of Dobby the Free Elf. You will words like Dobby, Lockhart (I hate that guy), Polyjuice, Parseltongue, Mandrakes, Mudbloods making an appearance in this book.

Prisoner of Azkaban

I used an image of Prongs (James' Marauder Nickname, and Animagus form. Harry's Patronus) for this book. Words like Lupin, Sirius, Pettigrew, dementors, Crookshanks, Patronus start appearing.

Goblet of Fire

I tried to use an Image of the Triwizard Trophy. Words like Cedric, Beauxbatons, Crouch, Durmstrang start appearing.

Order of the Phoenix

Tried to use an image of a Phoenix. Umbridge is very popular in this book.

Half Blood Prince

I used an image of the Half Blood Prince for this book. Apart from the usual, Slughorn is the most common word in this book.

Deathly Hallows

I used an image of the Deathly Hallows for this. You will see "wand" becomes very used due to "Elder wand". Hallows, Cloak, Wandmaker appear. Also, Griphook is back.

Future Plans

I am planning to scrape AO3 in the future to do some more analysis. I might also create some other Word Clouds from other popular books.

Reference:

Read this on my blog

Receive Netlify Notification on Telegram

Haider Ali Punjabi — Sun, 17 May 2020 12:44:47 +0000

Introduction

Netlify is one of the worlds largest hosting providers for JAMStack Websites. All of my websites are hosted on it. Some of them use manual deploys as they are deployed via CI. I have disabled auto-publishing for some of the websites because I like to preview them before publishing. Since a CI is used during deploy, the time taken for the website to deploy on Netlify is random. I wanted to be notified whenever a deploy was ready to be published and I tried many different techniques for it. Initially, I used Outgoing Notifications with IFTTT. With that, I was able to receive a notification on my mobile whenever the deploy was ready. It worked fine, but I wanted to receive a notification on my PC, not my mobile, which will open the deploys page of the website when I click on it.

For a project of mine, I learnt how to make Telegram Bots. With that, I was able to create another bot that messaged me whenever a website was ready to deploy.

Setup

Follow this tutorial to create the Telegram Bot. Note the API Token you receive. In the following steps, it will be referred to by $token
The bot will need to know your chat id. To find that, we need to set up a temporary webhook. Webhook.site is a great service to test webhooks online. Go to Webhook.site and copy the unique URL. (Referred by $unique_url)
Open the following URL in your browser https://api.telegram.org/bot$token/setWebhook?url=$unique_url

Now, send a message to your bot from your Telegram account. In the Webhook.site window, you will receive a new request which looks like:

       {
         "update_id": 213123213,
         "message": {
           "message_id": 16,
           "from": {
             "id": 987654321,
             "is_bot": false,
             "first_name": "John",
             "last_name": "Doe",
             "username": "johndoe",
             "language_code": "en"
           },
           "chat": {
             "id": 987654321,       // <----- chat_id
             "first_name": "John",
             "last_name": "Doe",
             "username": "johndoe",
             "type": "private"
           },
           "date": 1589716177,
           "text": "Jd"
         }
       }

Note the chat["id"] from the previous step. It is the chat id you need. (Referred by $chatid)
You can delete the webhook now
https://api.telegram.org/bot$token/deleteWebhook
Now, go to your website's settings on Netlify. In Build & Deploy > Deploy Notifications, add a new Outgoing Webhook.
Select the required event (Details of all events), and the following in URL field.
https://api.telegram.org/bot$token/sendMessage?chat_id=$chatid&text=$message

The $message is the message you want to receive. I send the name of the website, event and the link to the website's deploys page.
The set up is done. Now, you will receive a Telegram Message whenever a deploy is ready.

References

Read this on my blog

Using CI to update Static Websites.

Haider Ali Punjabi — Tue, 28 Apr 2020 09:41:55 +0000

Background

Sometimes, our websites require data that isn't updated in realtime but updated after every few hours, days or maybe even weeks. In such cases, requesting the data from some backend or API on every request will be a waste of resources. Client-side caching of data might help a bit but isn't the best solution. Also, you might end up reaching your API Requests limit.

In July 2019, I made a project - HPFanfiction Recommender (Read about it in a previous blog post), where my backend generates the required data and then triggers a Travis CI build which downloads the data and stores it in the repository. This method had a serious drawback. It used to commit 2 - 3 MB of files every 4 hours. As a result, when I tried to Pull the repo in February 2020, the size was huge. As a temporary fix, I added --amend to the Travis Commit command but I am not sure how well it worked.

At the end of March 2020, I started a project - covidkashmir.org, which also needed to make some changes to repo during CI but I didn't want those changes committed back to the repo, only deploy them to Netlify. My initial idea was to download netlify-cli during the CI build and then use it to deploy the changes without committing them back. To my surprise, I found out about Travis CI DPL v2 and its Netlify Drop Deployment. In a future blog, I will write the details about covidkashmir.org 's requirements. In this blog, I will write about how to set up Travis CI DPL v2 and its Netlify Drop Deployment for any static site.

Setting Up Travis CI DPL v2 and its Netlify Drop Deployment

In your code/repository

In your repository, add a .travis.yml file which will act as the configuration file for Travis.
Configure .travis.yml as per your needs. (Travis CI Documentation)
In your .travis.yml add the following lines to set up Travis CI DPL v2 and its Netlify Drop Deployment

       deploy:
         provider: netlify
         site: $NETLIFY_SITE    # Add in Travis CI Settings
         auth: $NETLIFY_AUTH    # Add in Travis CI Settings
         dir: .
         edge: true # opt in to dpl v2
         prod: true     # for deploying to production

On Netlify

For New Sites

Deploy your site using netlify-cli or Neltify Drop, and don't link your site to Git

For existing Sites

If your site is already linked to a git repository, apply here to unlink it

We need an unlinked site because it will deploy twice (once without the changes from Travis) otherwise.

Go to the Sites Settings > General and note the API ID. We will need it later.
Go to User Settings > Applications and create a new Personal Access Token. Note it as well.

On Travis CI

Link Travis CI to your Repository
Go to the repository's settings and add the following environment variables:
1. NETLIFY_SITE : API ID from Step 3 of the Previous Section
2. NETLIFY_AUTH: Personal Access Token from Step 4 of the Previous Section

Now, when you push your code, it will trigger the build on Travis, generate your data and deploy to Netlify.

Read this on my blog

Adding View Count to your JAMstack Website with JavaScript and Google Analytics

Haider Ali Punjabi — Fri, 14 Feb 2020 13:36:39 +0000

Introduction

As JAMStack is becoming more and more popular, many people (especially developers) are shifting their websites (and blogs) to JAMStack and use some Static Site Generator (Jekyll, Hugo, Nuxt, Next, Gatsby, etc).

My blog is based on Hugo and hosted on Netlify. My previous post got more views than I expected and I spent the next two days checking my Analytics App to keep track of the views. Then it occurred to me that I should try and add a view counter to my blog. I post all my articles simultaneously to dev.to and Medium, and both of them tell me how many views my article has got.

It took me around 4 hours to add the views counter, but it won’t take you as much time because I experimented with lots of stuff.

My only goal was to have a counter with good accuracy which updates the values without rebuilding the entire site.

After searching for a while, I concluded that using Google Analytics data instead of adding a new mechanism would be best.

I spent around two hours messing with Google Core Reporting API v3 because I wanted to avoid the Google Analytics Reporting API v4’s API KEY requirement. When I was almost finishing up the changes, I read somewhere that v3 API would stop functioning this year.

After wasting 2 hours on v3, I knew I had no choice other than using v4. While looking up stuff related to v4, I came across Google Analytics Spreadsheet Add-on, which allows you to export Google Analytics data to a Google Spreadsheet. After a lot of experimenting, following multiple tutorials and numerous visits to Stack Overflow, I was able to export my Analytics Data to Google Sheets.

Then I had to figure out how to get this data to my website. I decided on using Hugo’s inbuilt functions and variables to achieve this (a very bad decision in hindsight). I spent another hour meddling through Hugo’s documentation to show the page view data on my website. After everything was done, and I was about to commit the changes, I decided to check the size of the data I was loading from Google Sheets. To my surprise and shock, I couldn’t find the entry for the CSV file in my browsers Network tab. Then it occurred to me that the data is being fetched during build, something which I wanted to avoid. Scraping all the changes, I decided to use JavaScript to make this work. Since my website doesn’t have jQuery, and I didn’t want to add it for such a small task, I wrote all the code in vanilla JavaScript (which I could have written better) and had it working finally.

The following tutorial would only cover getting the data from Analytics to Google Sheets and fetching that data in vanilla JavaScript. I won’t show how to add that to your website as every website structure is different.

Tutorial

Part a) Setting Up Google Sheets + Analytics Add-On

Follow this guide to install the add-on to your Google Sheets.
Select "Add-ons" > "Google Analytics" > "Create a New Report" from the menu bar.
Name it something, and select your Analytics View.
Under configuration options, choose Pageviews (ga:pageviews) as your metric.
The Dimensions will depend on the structure of your website. Page Path Level 1 will only return the data for Pages and Folders in the root of your website. Page Path Level 2 will return data for Pages and Folders in some folder in your root. Use the following table to decide your dimension.

My blog posts are present in the posts/ directory, so I decided to use both Page Path Level 1 & Page Path Level 2 dimensions. By using both 1 & 2, I get both posts/ & my blog post URL.

If you use the Page dimension instead of Page Path n, your data will also include Query Strings and the data would be divided for every unique URL.
Under Segments, I selected All Users. but you could also use New Users to get unique views only.
Create the Report and you will see your Configuration Sheet will be generated.
Change the Start Date to a date before your first blog post (in YYYY/MM/DD format), and change the End Date to 'today'
You should remove the 1000 limit if you want to get the data for more than 1000 posts.
1. Create another empty sheet which will act as output. Share this sheet with Edit Permission enabled. Copy the Share Link.
2. Paste the copied Share Link into the Configuration Sheet in the Spreadsheet URL Row.
3. Select "Add-ons" > "Google Analytics" > "Run reports" from the menu bar.
4. It will show you a Report Status Popup, which should tell you that your report completed successfully.
5. Select "Add-ons" > "Google Analytics" > "Schedule Reports" from the menu bar.
6. Select "Enable Reports to run automatically" and run it "every hour", and save it.
7. Go back to your output sheet and go to "File" > "Publish to Web"
8. Under "Published content & settings", select you Sheet Name and check "Automatically republish when changes are made". Then click "Start Publishing".
9. Choose "Comma-separated values (.csv) as output format and copy the given link"

Part b) Getting the data on your blog

On the page where you want to show the view data, add a JavaScript file (or edit an existing JavaScript File for that page).
Use Fetch API to get the CSV data and parse it using CSVToArray function from this StackOverflow Answer


url  = "URL which you copied in Step 18. of part A" 
fetch(url).then((response)=>{     
    return response.text() 
}).then((text)=>{ 
    data = CSVToArray(text) 
    viewMap = {} 
    for(let i = 15; i < data.length; i++){
         viewMap[window.location.href + data[i][0].slice(1,-1) + data[i][1]] = parseInt(data[i][2])     

    } 
})

You might need to adjust the code according to your needs

You will have a dictionary with your Post URLs as Key and their Views as values.
Use them according to your website structure. (You can look at my blog's code if you need more help)