DEV Community: Priyansh Jain

Project Overkill?

Priyansh Jain — Sun, 28 Jun 2020 21:20:38 +0000

I just got this dumb idea here at 2:36 AM IST, it's called Project Overkill.

Basically, use the most overkill software/tool/library for everything in this project (the opposite of YAGNI principle

For example, we can have a very simple CRUD app.
But for this, I'll use

fault tolerant k8s/openshift and dockerised GraphQL based APIs that are in the form of separated microservices in different languages (for Auth, processing, mailer) with load balancers
Kafka as the message queue for these microservices
Kong as the API gateway
Cassandra/some blockchain db as the database complete with sharding and replication and encryption
VueJS for the sign-up page, react for post-login actions
Monitoring via Prometheus, grafana
logging via ELK stack
complete CI/CD using Jenkins scripted pipelines
DDos protection via cloudflare and recaptcha for POST call pages

What ideas do you people have for creating the dumbest most overkill project?
I believe it would be a good learning experience, going hands-on with all of these tools

GPU rendered desktop apps/assets

Priyansh Jain — Thu, 26 Sep 2019 06:05:20 +0000

Hello!

I've been playing a lot of games now, and most of these games have insane menu design - extremely responsive and not using any native widgets. Which led me to the thought - why not make a library for creating GUIs that uses the GPU for rendering the screens?

Any ideas?

Hands-on with distributed file systems and storage virtualization!

Priyansh Jain — Tue, 30 Apr 2019 16:32:47 +0000

We had a course this spring, Virtualization, which had a project component.
You must've heard about virtualization, how things look unified from one end but are an orchestration of multiple services at the other end. We've multiple types of virtualization - ones that range from virtualizing entire Operating Systems (dual boots / Oracle VirtualBox) to virtualizing the desktops. I had to select a topic for the project and on some research found storage virtualization interesting.

Here's the source code for the project.

Storage Virtualization is practically providing a unified view on a cluster of storage devices, managed in such a manner that we have data safety and size/performance optimization.

A decent issue with having a storage solution is data de-duplication, which means trying to remove unnecessary copies of files that take up your resources. And I intended to solve the issue with my project(a distributed file system) while getting hands-on experience with managing multiple servers and implementing a management system within it.

The stack

The web framework I am most comfortable working with is Express(Node.js), and hence chose the same coupled with EJS templating for the frontend views.
I also decided to make use of docker for containerized services and using docker swarm to orchestrate them.

Development Methodology

I wrote some general upload code in express using multer for managing them, then packaged it into a docker image. The base image was the LTS Node.js one, and I used docker-compose to do it in an organized manner.
Docker swarm is used to manage the cluster, and provides fault tolerance by having replicas and whatnot. What I wanted here was easy to perform hostname mappings for my storage servers.
The dockerfile I used for building the image

FROM node:10

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm install --only=production

# Bundle app source
COPY . .

We see that I set the working directory as /usr/src/app. During development, I mounted the current folder as a volume, so nodemon would automatically detect all the changes and reflect them immediately. The development docker-compose file -

version: "3.2"

networks:
  testnet:
    external:
      name: testnet

services:
  server1_server:
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      # placement:
      #   constraints:
      #     - node.hostname == your-hostname-2-here
    image: presto412/storev1
    hostname: server1.example.com
    command: ./node_modules/.bin/nodemon
    volumes:
      - "/tmp/uploads:/tmp/uploads"
      - ".:/usr/src/app/"
    environment:
      - SELF_HOSTNAME=server1.example.com
    ports:
      - "3000:3000"
    networks:
      testnet:
        aliases:
          - server1.example.com

First attempt

Since I had developed on blockchain platforms during the past year, my initial go-to logic was to make use of not block-chaining itself, but the tech that lies underneath it, Distributed Ledger Technology.

The general process flow -

The file is uploaded via a form
On upload, the backend hashes the file contents by making use of the sha1 library. I also considered using xxHash but didn't implement it because speed wasn't a concern for a small indie project.
It maintains a mapping JSON of the following structure

{
  "0347f8b35b22104339b5f9d9d1c8d3f0251b6bdc": {
    "details": {
      "name": "JKSUCI_AuthorAgreement (2).pdf",
      "mimetype": "application/pdf",
      "date": "Sun, 07 Apr 2019 23:55:04 GMT",
      "size": 27087
    },
    "backups": [
      {
        "hostname": "bangalore.storage.com",
        "path": "/tmp/uploads/528518c7e306e2f04e34117a5b245e6f"
      },
      {
        "hostname": "amsterdam.storage.com",
        "path": "/tmp/uploads/600405f52831535e54aaf590f2b14052"
      }
    ]
  }
}

So here you can see that the key is the hash of the file, and the value is where it is stored and some metadata.
This new file's hash is now checked with the above JSON, and if it exists, the system denies the upload and doesn't save the file.
Otherwise, it randomly selects two servers to store it in and updates the mapping as well. This mapping is then updated for every other server, thus making it a distributed ledger implementation.

At this point, the project was ready and showed the implementation for the first review. Professor suggested to include some meaningful changes, and the one I liked most was delivering files by geographical proximity to the server.

Improvements

I moved to the cloud this time. Created 5 droplets on digitalocean, with each one having a separate data centre - New York, Toronto, Singapore, Bangalore, Amsterdam.
I changed the architecture a bit this time - had a central server that served the frontend and maintained metadata of each file stored(I know that it is a bottleneck, but was enough for a class project)

What I also did, was hash the files in the client side, by making use of the FileReader API coupled with the Forge library. Here's some code for it.

  <script type="text/javascript">
    function submitFile() {
      var form = document.getElementById("uploadFileForm");
      var reader = new FileReader();
      reader.onload = function() {
        var fileContent = reader.result;
        var md = forge.md.sha1.create();
        md.update(fileContent);
        var fileContentHash = md.digest().toHex();
        document.getElementById("fileHash").value = fileContentHash;
        $.ajax({
          type: "POST",
          url: "/checkHash",
          contentType: "application/json;charset=utf-8",
          data: JSON.stringify({
            fileHash: fileContentHash
          }),
          success: function(response) {
            console.log(response);
            console.log("sresponse", response.success);
            if (!response.success) {
              form.submit();
            } else {
              console.log("not submtting form");
              alert("File already exists");
            }
          },
          error: function(response) {
            console.log("fresponse", response);
          }
        });
      };
      const file = document.getElementById("fileItem").files[0];
      reader.readAsText(file);
    }
  </script>

So now, only the file hash was being sent to the server as an ajax, and if the hash wasn't present, the file was freshly uploaded.

When the central server received the file, it selected two storage servers to store it in and then updated its metadata file.

So this time, when a new request for a file comes in, I used APIs like ipstack and distance24 to determine what location(city, preferably) the IP is from, and how far the city was from each of the storage servers. The servers are then sorted by distance, and the a URL is generated for the closest server, and returned. When the user clicks on the link, the file is downloaded.

Docker and Docker Swarm have been extremely helpful in this scenario, since I can simply specify the deployment constraints, pull and distribute images and so on.

What was your first project with distributed systems? Let me know in the comments below. Here's the source for the project.

Thanks for reading!

Priyansh Jain
Github | LinkedIn

Deciding a database architecture for a Social Networking use-case?

Priyansh Jain — Wed, 05 Sep 2018 20:42:11 +0000

So I'm in the process of writing a Node-backed server, and one major part of the application is the Social Network Platform. I don't plan on using MySQL, even though it's reliable.

After some research, I stumbled upon multiple articles stating why you should use one over the other or with the other. It does make sense that using a graph database is the best option, better than a simple RDBMS/conventional NoSQL.

As of now, I've got these options with me

MongoDB for document storage, and using mongo-connector in collaboration with neo4j for storing relationships, likes and stuff. Looks good to me, as some part of the application has already been developed with mongo, wouldn't have to rewrite everything.
OrientDB as a whole package from the ground up
Gremlin
Apache Cassandra(No idea how this works, any help could save me some time :D)

I'll have you know that the social network relationship is but a part of other functions the application is supposed to execute, the other one being a chat interface, along with real-time mapping.

Any advice would be much appreciated!

How do you scale a nodejs real-time API to concurrently serve a million users?

Priyansh Jain — Sun, 05 Aug 2018 20:05:57 +0000

So I recently got a project that requires real-time location + chat data streaming, and I wanted to know the right steps to take.

Most articles I've read online state to spawn up multiple servers and set up a load balancer like nginx+pm2.

However, I wanted to make this thing DevOps ready, with CI/CD and best practices, which I am not able to judge from the articles. Containerisation, database optimisation, and everything.

Would love to learn the right way!

Can react be used as a utility full stack web app?

Priyansh Jain — Mon, 16 Jul 2018 11:03:36 +0000

I was just wondering, that since react is/has

ES6 compatible
NPM support
Request Support

Consider the case where we have a UI based interface to some data that is in a different website, where we use APIs/requests to fetch data and render it to the frontend. Can react be used as a standalone frontend+backend solution for small task stuff?

Since react has it's own routing mechanism, can the processing that an express app does be done via imported libraries in react itself? Assume that the size transferred in requests is very less and processing the response is basic string manipulation/html parsing, and there is no requirement for a database, the client's storage should do.

Why did i think of this? No need to maintain a server that acts as a middleware to the react app.

EDIT: For such an app, compile the code into react native, and every user who uses the app is their own server. Possible?

Thoughts?

Google Summer of Code - can any organizations help me contribute?

Priyansh Jain — Sat, 30 Jun 2018 04:25:14 +0000

I'm still an undergrad, and the GSoC program has fascinated me. I've browsed a lot of organizations, and just wanted to know if there are any people on Dev.to who could guide me to contribute more to their open source projects featured on GSoC.
I can work with Python, C/C++, JavaScript, and Golang.

Do let me know if you can help!

Resources for Getting into DevOps?

Priyansh Jain — Fri, 22 Jun 2018 06:14:46 +0000

I was very recently introduced to using Docker and Docker Swarm, and I got interested in cluster management and horizontal scaling. How the nodes balance the load was fascinating to me. Can anyone provide some resources like books or online courses to seriously get into DevOps?

How do you get a decent estimate on the time it will take to complete a task?

Priyansh Jain — Fri, 15 Jun 2018 16:55:02 +0000

I have had certain situations where I gave how much time I thought it would take, usually fixing a bug or adding a new feature, but it took way longer. Do you have some tips in mind?

Hyperledger Fabric: Transitioning from Development to Production

Priyansh Jain — Fri, 15 Jun 2018 15:25:53 +0000

You’ve set up your development environment. Designed your chaincode. Set up the client. Made some decent looking UI as well. Everything works fine locally. All of your tests pass, and all the services are up and running.

All this, and you still aren’t satisfied. You want to do more. You want to scale the network. Simulate a production level environment. Maybe you even want to deploy to production.
You give it an attempt. Add multiple orderers. Add more organizations with their own peers and CAs. You try testing it with multiple machines.

This is where everything you’ve done fails, no matter what you try. You somehow debug everything and come up with some nifty hacks you’re not entirely proud of. Some parts work, some don’t. You wind up the day, still not satisfied.

I don’t know if you can relate to this. I went through this phase, racking my head and taking multiple cups of coffee trying to clear my head and starting the debugging from scratch. Multiple facepalms and “ah, this is how this works” later, I decided to write this article, and be your friendly neighbourhood developer-man(sorry).

The First Steps

Fabric provides two consensus protocols for the network - Solo, and Kafka-Zookeeper.
If you’ve only been working with Solo mode(configurable in configtx.yaml), this is where you change it. When we want our network to be shared among more than 3 peers, it makes sense to have multiple orderers, in the case where one particular, cursed node goes down.

Now by design, orderers are like the postmen in a Fabric network. They fetch and relay the transactions to the peers. Solo mode requires all our orderers to be up, and if one goes down the entire network goes down. So here, the trick is to use a Kafka based ordering service. I’ve comprehensively explained how it works in Fabric here.

Fabric docs provide some great best practices to follow.

So here goes our first step - Solo to Kafka. Don’t forget to specify the brokers in your transaction config files. A sample has been provided below.

configtx.yaml

    ...
    Orderer: &OrdererDefaults
        OrdererType: kafka
        Addresses:
            - orderer0.example.com:7050
            - orderer1.example.com:7050
            - orderer2.example.com:7050
            - orderer3.example.com:7050
        BatchTimeout: 2s
        BatchSize:
            MaxMessageCount: 10
            AbsoluteMaxBytes: 99 MB
            PreferredMaxBytes: 512 KB

        Kafka:
            Brokers:
                - kafka0:9092
                - kafka1:9092
                - kafka2:9092
                - kafka3:9092
    ...

Now that we have our orderers’ dependencies resolved, let’s dive into some network level tips.

A Swarm Of Services

All the images that we use for Hyperledger Fabric are docker images, and the services that we deploy are dockerized. To deploy to production, we’ve two choices. Well, we have a lot of choices, but I’ll only describe the ones that might just not get you fired. For now.

The two choices, are Kubernetes and Docker Swarm. I decided to stick with Docker Swarm because I didn’t really like the hacky docker-in-docker set up for the former. More on this here.

What’s a swarm?

From the Docker docs,

“A swarm consists of multiple Docker hosts which run in swarm mode and act as managers (to manage membership and delegation) and workers (which run swarm services). A given Docker host can be a manager, a worker, or perform both roles”.

It is a cluster management and orchestration method, that is featured in Docker Engine 1.12.

So consider this - the blockchain administrator can have access to these manager nodes, and each potential organization can be a designated worker node. This way, a worker node will only have access to the allowed resources, and nothing gets in the way. Swarm uses the raft consensus algorithm, and the managers and workers accomodate replication of services, thus providing crash tolerance.

DNS - Do Not Sforget_to_give_hostname

Not giving hostnames to your services is a rookie mistake. A hostname for every service is necessary. This is how one service in the Swarm maps the IP address to the host name. This is used extensively for Zookeeper ensembles.

Consider 3 Zookeeper instances. We would have these depend on each other for synchronization. When these are deployed to a swarm, the Swarm gives an internal IP address, which is pretty dynamic.

    ...
      zookeeper0:
        hostname: zookeeper0.example.com
        image: hyperledger/fabric-zookeeper
        environment:
          - ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=zookeeper1:2888:3888 server.3=zookeeper2:2888:3888
    ...

Another Zookeeper instance that would depend on this,

    ...
      zookeeper1:
        hostname: zookeeper1.example.com
        environment:
          - ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=zookeeper1:2888:3888 server.3=zookeeper2:2888:3888
    ...

Restrictions and Constraints

Usually, inside a Swarm, the services are distributed to any node by default. Docker Swarm will always try to improve the performance on the manager nodes. Therefore, it will try to distribute the services to the worker nodes. When a worker node goes down, the manager can redistribute the services inside the swarm.

Now there are some very important services that we would like to keep afloat, like the Kafka-Zookeeper ensemble, since they synchronize transactions among all the orderers. Hence, what we would like to do here is make sure that we don’t suffer any downtime. There also may be a service that holds certificates, and it is important that the certificates don’t leak in the Swarm network. Therefore, we need restrictions on stack deployment in the Swarm.

We can constrain the services deployed to nodes in the swarm. A simple example is shown below.

    ...
    services:

        peer0:
            deploy:
                replicas: 1
                restart_policy:
                    condition: on-failure
                    delay: 5s
                    max_attempts: 3
                placement:
                    constraints:
                        - node.role == Manager
                        - node.hostname == Skcript  
    ...

Note that the deploy section defines the crash handling as well, so use it to your benefit.
When we implement constraints, almost every authentication issue can be resolved.
You might observe that this defeats the purpose of docker Swarm, where it is supposed to maintain the state of the Swarm as much as possible. If such is the case, you’ll have to spend more on another node that can handle downtime and probably extend your constraints.

Protecting Certificates and Crypto-material

Certificates are not supposed to be disclosed to other entities in a network. Ideally, the easiest thing to do, is to supply the certificates of a particular organization to the node hosting it, and install to a location that is common to both a docker container and a simple linux machine, like /var/<network-name>/certs. This way, you mount only the required volumes.
Speaking of volumes, when mounting, be sure to have absolute paths. You can only deploy services to a Docker Swarm from a manager node, and hence it needs to have the certificates at the location in a node hosting it, else the service will shut down.

An example:

docker-compose-peer.yml

    ...
    volumes:
                - /var/run/:/host/var/run/
                - /var/network/certs/crypto-config/peerOrganizations/state.example.com/peers/peer0.org1.example.com/msp:/var/hyperledger/msp
    ....

The /var/network/certs/ directory should be copied in the host worker node before deploying the service.

I don’t recommend a standalone service floating in the swarm, that anyone can access.

Naming the services

docker stack deploy doesn’t allow certain characters for service creation. If you worked directly from the tutorials, you would have the service names like peer0.org1.example.com, orderer0.example.com.

So it is better to name them as peer0_org1 , and orderer0 and so on.

Fetching the names

Docker Swarm usually prefixes the stack name to the service, and the suffix is a SHA256 hash. Hence, to execute any commands, we need the name of the services, given to them by the Swarm.
So for example, if you’ve named your service peer0_org1, and the stack you’ve deployed it to is deadpool, the name that swarm will give it would look like deadpool_peer0_org1.1.sa213adsdaa…..

You can fetch its name by a simple command,

    docker ps --format="{{.Names}}" | grep peer0_org1

PRO TIP: Environment variables are your best friends. Do have a dedicated .env for all your scripts.

Channels and Chaincodes

When you have multiple organizations, and you want custom channels to run amongst these, and install different smart contracts on each channel, this is how it should work.

Channel creation has to be done based on what is defined in the configtx.yaml. You create the channel, and join the respective peers. A sample channel creation command looks like this,

    docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org1.example.com/msp" "$ORG1_PEER_NAME" peer channel create -o "$ORG1_ORDERER_NAME":7050 -c "$ORG1_CHANNEL_NAME" -f "$ORG1_CHANNEL_TX_LOCATION"

Now to join the channel from a seperate organization, fetch is necessary. Note that 0 here means we are fetching the 0th block.

    # fetch the channel block
    docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org1.example.com/msp" "$PEER_NAME" peer channel fetch 0 -o "$ORDERER_NAME":7050 -c "$CHANNEL_NAME"

    # join the channel
    docker exec -e "CORE_PEER_LOCALMSPID=CHMSP" -e "CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@ch.example.com/msp" "$CH_PEER_NAME" peer channel join -b "$CHANNEL_NAME"_0.block

Chaincode creation is similar, however it is interesting to note that you have to instantiate the chaincode only on one peer in the channel. For the other peers, you will have to manually install the chaincode, but instantiating is not required. An invoke command should fetch the most currently instantiated chaincode, and simultaneously execute the transaction as well.

Winding up

So if you’re able to invoke and query successfully, you should be good to go.

Document your configuration well, create some scripts that save time, like ssh-ing into the services and executing the above commands from a manager node itself.

Attach your client to the services above, and enjoy your Production-level Fabric Network set up.
Try deploying your network to the IBM cloud, or AWS.

If you have any queries or suggestions, do comment below.

Can the entire journey of Interstellar be rendered pointless by a supercomputer?

Priyansh Jain — Sat, 14 Apr 2018 11:00:04 +0000

So here's my theory.
In the movie, to solve the unified theory - harnessing gravity - requires quantum data to be transmitted from inside a black hole.
But it is TARS the robot that sends the data.
Assuming they have computational capabilities of a monstrous supercomputer and extreme programming capabilities( cause they programmed the super intelligent TARS), isn't it technically possible to have brute-forced the finite amount of data that TARS is capable of sending that somehow solves the unified theory and hence renders the journey useless?

P.S. Can we apply this theory to a theoretical physicist's research? Instead of arriving at a conclusion, use a supercomputer to generate a conclusion that solves the problem in question?

Let me know your opinions on this in the comments!

Visualising the JavaScript Event Loop with a Pizza Restaurant analogy

Priyansh Jain — Tue, 10 Apr 2018 17:44:26 +0000

Consider a pizza restaurant.
There are two types of orders that we currently have from a single customer - one is an elaborate order, that requires a pizza with an olive topping(1), a cheese filling(2), and a large base(3).
The other one is just a simple one, mayonnaise(a) with garlic bread(b).

The chef on receiving the order starts making the first pizza, by taking a large base(3), adding the filling inside it(2), and then adding the olive toppings(1).
The same chef also has to make garlic bread. The manager suddenly realizes that the restaurant is completely out of mayonnaise. The manager adds the 'get mayonnaise' task to a chart and sends the only available errand boy to go fetch some.

Technically, if orders were to be taken together and delivered together, the customer would have to wait until an errand boy goes to a supermarket five blocks away, get the mayonnaise, and give it to the cook for completing the order. But this is a restaurant, and customers don't need to have the entire order delivered to them all at once.

The chef decides to continue making the pizza, bakes it and sends it to the customer.
When that is done, the errand boy arrives with the mayonnaise, and the cook takes some garlic bread(b), adds the mayonnaise(a) on top, and delivers it to the customer.

What did we learn about Javascript here?

The customer's orders(make Pizza + make GarlicBread) are the functions in JavaScript code.
The order details are simply about how to customize the pizza and the bread, and they can be treated as functions called inside of make Pizza - and the order is taken from top to bottom - the toppings, the filling, and the size. These details are basically a representation of the call-stack, that executes all of these events in reverse order.
The absence of mayonnaise from the restaurant - that was an Event that got triggered and called an asynchronous function, that was to fetch the mayonnaise from a supermarket. Since the restaurant doesn't need to send the entire order together, they complete the tasks in the call stack in a reverse order, as implied above in the story.
The manager here is the event table - his job is to keep track of all the mishaps(Events) that happen, in a chronological order.
The errand boy is the event queue, that is, if he is already asked to fetch something and a new item is requested, the item has to wait until the errand boy fetches the old item.
The chef is the event loop, that is continuously making the orders(executing all the functions).
The restaurant is a browser, that doesn't need to freeze until all the content is loaded and doesn't need to wait for other things to complete before one is completed.(No need of serving the entire order together)

So in essence, the event loop checks if the call stack is empty, and if so, looks into the event queue. If there is something in there, it adds it to the call stack and executes it. The event loop constantly runs until their shift is over(browser content is loaded/ browser is closed). The event table keeps track of all the events that have been triggered and sends them to the event queue to be executed.

This was my attempt at explaining the event loop, let me know if you found this analogy interesting!